This section presents some advanced functionalities.
benchmark
hyperparameter-tuning
If you take a set of run from the training set and move it in a separate folder, e.g. validation_set
, you can add it to the configuration runcard under the input
key:
input:
# ...
benchmark_folders:
- validation_set
# ...
Now you have access to a performance assessment procedure based on `closure tests`:
mcnntunes -o output benchmark runcard.yaml
For each run in the benchmark folders, the program will perform the tuning procedure with this run instead of the actual experimental data. Then, the obtained parameters are compared with the exact parameters used for generating the run, and a loss function is computed. For more information about the closure test procedure, see https://arxiv.org/abs/2010.02213.
Deep learning algorithms present many hyperparameters that can heavily influence their performance. MCNNTUNES implements an hyperparameter tuning procedure using HyperOpt. In order to enable the hyperparameter optimization, you need to insert the corresponding settings in the configuration runcard:
hyperparameter_scan:
max_evals: number of max evaluation for the hyperparameter search
model:
architecture:
actfunction:
optimizer:
epochs:
batch_size:
# other model settings...
You need to define the search space under the model
key. It must be defined by means of specific function provided by HyperOpt (see the official documentation). You don't need to use each available key. An example of search space may be:
architecture: "hp.choice(’layers’, [[hp.quniform(f’size_{_}_2’,5,10,1) for _ in range(2)],
[hp.quniform(f’size_{_}_3’,5,10,1) for _ in range(3)],
[hp.quniform(f’size_{_}_4’,5,10,1) for _ in range(4)]])"
optimizer: "hp.choice(’optimizer’, [’sgd’,’rmsprop’,’adagrad’,’adadelta’,’adam’,’adamax’])"
optimizer_lr: "hp.loguniform(’learning_rate’, -10, -1)"
Now you can run the hyperparameter scan with
mcnntunes -o output optimize runcard.yaml
At the end of the optimization procedure, a summary of all evaluations will be presented. Moreover, an in-depth analysis of the results is presented in the HTML report that mcnntunes tune
builds. If you are satisfied with the best configuration, you can reconfigure MCNNTUNES with that configuration and re-execute the full basic work cycle.
Note
Optionally, you can run a parallel search using the features of HyperOpt. This requires the instantiation of a MongoDB database that must be able to communicate with all the workers. Then, you must insert the database URL and an arbitrary exp_key to the configuration runcard:
hyperparameter_scan:
# settings...
cluster:
url: URL
exp_key: EXP_KEY
# settings...
The program will send some work items to the database and wait for their results. These items corresponds to different trials for the hyperparameter scan. In order to evaluate these items, you need to launch some workers using e.g.
hyperopt-mongo-worker --exp-key=EXP_KEY --mongo=<host>[:port]/<db> --workdir=WORK_DIR
See hyperopt-mongo-worker --help
for more options. For more details about the HyperOpt implementation of the parallel search, see the official documentation.