Hyperopt Parameter Search

Before you start, you might want to read the Hyperopt documentation, especially the basic tutorial.

Introduction

We use Hyperopt to optimize the random forest training parameters, such as the number of trees to train, the number of feature candidates to evaluate during training, the number of pixels to sample per image, etc. The parameter search is split into two processes. One process inserts new trial to evaluate in a MongoDB database. The other process fetches a new trial and evaluates a loss function which is minimized by Hyperopt in the long run. The loss function can be specified but is usually the average pixel accuracy of image segmentation or the average per-class accuracy.

The parameter search space is defined by the process which inserts new trials to the database.

Example

In the following we show an example how to run a Hyperopt parameter search for random forests on the NYU Depth v2 dataset.

1. Make sure a MongoDB database is up and running.
2. Adjust the parameter search space in the scripts/hyperopt/hyperopt_search.py script.
3. Install the required python modules, especially hyperopt and run the script: python scripts/hyperopt/hyperopt_search.py
4. Launch the Hyperopt parameter evaluation on your CUDA machines by using curfil_hyperopt. We assume that you have already downloaded and converted the NYU Depth v2 dataset.

curfil_hyperopt --url MONGODB_URL --db DATABASE --experiment EXPERIMENT \
                --trainingFolder NYU_FOLDER/training \
                --testingFolder NYU_FOLDER/testing \
                --ignoreColor '0,0,0' \
                --lossFunction classAccuracyWithoutVoid \
                --subsamplingType classUniform \
                --deviceId GPU_DEVICE_ID