An example for integrating a general machine learning training script with SACRED experimental framework, and HyperOpt (Distributed Asynchronous Hyperparameter Optimization).
Below you can also find full installation and usage instructions.
- Random order grid search, which is easy to config and manage, combining HyperOpt and SACRED.
- Grid-search ranges support logarithmic, linear and categorical ranges.
- Very easy to scale across machines and clusters.
- Avoids running the same experiment twice.
- Experiments are logged on MongoDB with SACRED experimental framework.
- Example notebook for accessing results saved on MongoDB using PANDAS DataFrame.
- An example on how to quickly wrap a commandline based python training script.
mnist_keras.py is a script to train mnist from commandline.
sacred_wrapper.py is a SACRED-HyperOpt wrapper for mnist_keras.
hyperopt_search.py is a distributed scheduler for hyper-params optimization.
hopt_sacred.py contains core classes and functions for this project
mongo_queries.ipynb is a usage example for listing results saved by MongoDB, but
using PANDAS DataFrame API. It also demonstrates manipulating (saving, loading,
deleting) experiment artifacts (model files) using the MongoDB filesystem (GridFS).
yes | conda create -n my_env python=3.6
conda activate my_env
yes | conda install -c anaconda tensorflow-gpu=1.9.0
yes | pip install pymongo
yes | pip install sacred hyperopt h5py
yes | conda install pandas matplotlib ipython jupyter nb_conda
yes | conda install cython
yes | pip install setuptools==39.1.0 # downgrade to meet tf 1.9 req
yes | pip install networkx==1.11 # downgrade to meet hyperopt req
# install sacredboard (optional)
yes | pip install https://github.com/chovanecm/sacredboard/archive/develop.zip
# Other common libraries I use, not required by this example
yes | pip install GitPython markdown-editor
Both SACRED and HyperOpt require a MongoDB host. Installation instruction are based on this page.
Execute the command below and also add it to your ~/.bashrc file
# Important: write the full path. Don't use ~/<...>, instead you can use ${HOME}/<...> .
export MONGO_DIR=<your full path to MongoDB>
Download and extract MongoDB tarball
mkdir $MONGO_DIR
cd $MONGO_DIR
mkdir data
mkdir log
wget https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-rhel70-4.0.1.tgz
tar -zxvf mongodb-linux-x86_64-rhel70-4.0.1.tgz
rm mongodb-linux-x86_64-rhel70-4.0.1.tgz
Execute the commands below and also add them to your ~/.bashrc file
export PATH=$MONGO_DIR/mongodb-linux-x86_64-rhel70-4.0.1/bin:$PATH
# Port forwarding function
# Usage example: pfwd hostname {6000..6009}
function pfwd {
for i in ${@:2}
do
echo Forwarding port $i
ssh -N -L $i:localhost:$i $1 &
done
}
Login to a mongo host machine in <mongo_host_address>
mongod --fork --dbpath=$MONGO_DIR/data --logpath=$MONGO_DIR/log/mongo.log --logappend
Check that mongo server is running by cat $MONGO_DIR/log/mongo.log and verifying that the output contains the line [initandlisten] waiting for connections on port 27017
# Login to client machine
ssh <client_address>
# Forward MongoDB port to mongo host machine
pfwd <mongo_host_address> 27017
# Activate anaconda environment
conda activate my_env
# Execute experiment
cd <project path>
python sacred_wrapper.py
Monitor results with sacred board (optional)
sacredboard -m MNISTdebug
For every client machine, do the following:
# Login to client machine
ssh <client_address>
# Forward MongoDB port to mongo host machine
pfwd <mongo_host_address> 27017
# Activate anaconda environment
conda activate my_env
cd <project path>
Login to client #1, run the commands above, and execute the hyper-params scheduler:
python hyperopt_search.py
For every other client machines (a "worker" machine), execute the worker script:
export GPU_ID=<gpu_id> # select gpu id
PYTHONPATH="./" CUDA_VISIBLE_DEVICES=$GPU_ID hyperopt-mongo-worker --mongo=localhost:27017/hyperopt_mnist --poll-interval=1 --workdir=`mktemp -u -p /tmp/hyperopt/`
This is an extension of my earlier repo https://github.com/yuvalatzmon/SACRED_HYPEROPT_Example, which demonstrated a more minimal example for integrating SACRED and hyperopt.