You can search for best hyperparameters of your model in DeepPavlov by means of cross-validation of neural evolution.
You can run cross-validation in DeepPavlov to select best parameters of your model. For this purpose you have to run special command 'paramserach'. for example:
python -m deeppavlov.paramsearch path_to_json_config.json --folds 5
Cross validation command have several parameters:
config_path
:- Specify config path, where you model is located.
--folds
:- This parameter shows how many folds you need in cross validation.
Do you want to use leave one out cross validation instead of folds?
Just specify this:
--folds loo
. If you want not to cross-validate just omit this parameter.
--search_type
:- This parameter is optional - default value is "grid" (grid search).
Note
Folds will be created automatically from union of train and validation datasets.
Config file of model should be consist of parameters ranges for search. For example, you try to optimize regularization coefficient in model, so you should add additional parameter in config with suffix '_range'. Let's see example for logistic regression model:
{
"class_name": "faq_logreg_model",
"in": "q_vect",
"fit_on": ["q_vect", "y"],
"c": {"search_choice": [1, 10, 100, 1000]},
"out": ["answer", "score"]
}
In this example parameter "c" described as search_choice, values for grid search:
{"search_choice": [value_0, ..., value_n]}
As a result you'll have new json config with best model parameters. It'll be stored in the same directory as config file and will have suffix '_cvbest.json'. Also you'll see final log messages about best model:
INFO in '__main__'['paramsearch'] at line 169: Best model params: {'C': 10000, 'penalty': 'l1', 'accuracy': 0.81466}
INFO in '__main__'['paramsearch'] at line 184: Best model saved in json-file: path_to_model_config_cvbest.json
Evolution process can be described in the following way:
- Initialize parameters of evolutionary process setting the following
arguments to
evolve.py
: --p_size
- number of individuals (models) per population (Default: 10).--key_main_model
- key of the dictionary in config containing the model being trained (see description below) (Default: "main").--p_cross
- probability of crossover for a parent pair (Default: 0.2).--pow_cross
- crossover power - portion of evolving parameters that will be exchanged between parents during crossover (Default: 0.1).--p_mut
- probability of mutation for a parameter (Default: 1.).--pow_mut
- mutation power - maximal portion of maximal possible value of parameter which can be added or subtracted during mutation (Default: 0.1).--gpus
- available GPUs divided by comma "," (Default: -1 means CPU support). If one runsevolve.py
with assignedCUDA_VISIBLE_DEVICES
, gpus are either ordinal numbers of device within those fromCUDA_VISIBLE_DEVICES
(e.g.CUDA_VISIBLE_DEVICES=3,4,5
and--gpus 1,2
mean running models on4,5
original GPUs) or all devices fromCUDA_VISIBLE_DEVICES
if gpus is not given.--train_partition
- if train file is too big to train (recommended to divide train files if train dataset is more than 100 thousands examples), one can split it intrain_partition
number of files, save it calling "any_name_{0}.any_extension", ..., "any_name_{train_partition
}.any_extension". In dataset_reader "train" field indicate the first one file. Population is trained on the N_{population} %train_partition
part of the dataset (Default: 1).--start_from_population
- the number of population to start from that is needed to restart population (Default: 0 means starts from 0 population).--path_to_population
- path to the directory "population_{start_from_population
}". Should be given ifstart_from_population
is not 0 (Default: "").--elitism_with_weights
- whether to initialize elite models with pre-trained weights from previous population or not (Default: not given means save elite models without weights. If parameter is given, then save elite models with weights).--iterations
- number of iterations to conduct (Default: -1 means infinite number of iterations (while loop)).- Warning:
metrics
can not be evolved because the main metric determines evolutionary process. - Current version allows to evolve any parameter of the config that is an item of some dictionary in config file. One can make a copy of a usual DeepPavlov model config, and reassign parameters that can be tuned during evolution. To evolve some parameter one has to assign it to a dictionary of one of the following type:
{"evolve_range": [min_value, max_value]}
- values uniformly distributed on the given interval,{"evolve_range": [min_value, max_value], "scale": "log"}
- values distributed on the given interval logariphmically,{"evolve_range": [min_value, max_value], "discrete": true}
- discrete values uniformly distributed on the following interval,{"evolve_bool": true}
- bool values,{"evolve_choice": [value_0, ..., value_n]}
- values uniformly taking out of the given values.- The
main
model in the pipe is being evolved. It is obligatory to use the one and only variable for setting paths for ALL fitted and trained models fromconfig["metadata"]["variables"]["MODEL_PATH"]
- this variable should be set to the common folder for current evolutionary process (for example,config["metadata"]["variables"]["MODEL_PATH"] = "{ROOT_PATH}/snips_evolution"
). Changesave_path
andload_path
of all fitted and trained components of the config to the relative paths usingconfig["metadata"]["variables"]["MODEL_PATH"]
(for example,"save_path": "{MODEL_PATH}/classes.dict"
).
That's all you need to change in the config. Now let's move on to the example.
If one prefers to run evolution on some provided by DeepPavlov dataset, firstly, download embeddings and datasets. Consider parameters evolution on SNIPS dataset, download data running the following command providing corresponding name of the config file:
python -m deeppavlov download intents_snips
To evolve the model run the following command providing corresponding name of the config file (see above) :config:`intents_dstc2.json <evolution/evolve_intents_snips.json>`:
python -m deeppavlov.evolve evolve_intents_snips
Folder
~/.deeppavlov/models/classifiers/intents_snips_evolution/intents_snips
will be created. Each population will be saved in a folder~/.deeppavlov/models/classifiers/intents_snips_evolution/intents_snips/population_i
each of which containspopulation_size
foldersmodel_i
consisting of saved model files explicitly, saved files of models from pipe that has a key"fit_on"
,out.txt
anderr.txt
with logs ofpython -m deeppavlov train
script from training each model separately, andconfig.json
with config for this individual.Now one can open iPython Notebook file
deeppavlov/models/evolution/Results_analysis.ipynb
, setCONFIG_FILE
to config file path and run cells to see evolution results.