NOTE https://github.com/yuvalatzmon/SACRED_HyperOpt_v2 is an extension to this repo, with more features and more abstraction.
A minimal example for integrating a general machine learning training script with SACRED experimental framework, and HyperOpt (Distributed Asynchronous Hyperparameter Optimization).
Below you can also find full installation and usage instructions.
mnist_keras.py is a script to train mnist with Keras.
sacred_wrapper.py is a SACRED wrapper for mnist_keras.
hyperopt_search.py is a distributed scheduler for hyper-params optimization.
hyperopt_grid.py is a grid-search extension for HyperOpt
mongo_queries.ipynb is a usage example for listing results saved by MongoDB, but
using PANDAS DataFrame API. It also demonstrates manipulating (saving, loading,
deleting) experiment artifacts (model files) using the MongoDB filesystem (GridFS).
yes | conda create -n my_env python=3.6
conda activate my_env
yes | conda install -c anaconda tensorflow-gpu=1.9.0
yes | pip install pymongo
yes | pip install sacred hyperopt
yes | conda install cython
yes | pip install setuptools==39.1.0 # downgrade to meet tf 1.9 req
yes | pip install networkx==1.11 # downgrade to meet hyperopt req
# install sacredboard (optional)
yes | pip install https://github.com/chovanecm/sacredboard/archive/develop.zip
# Other common libraries I use, not required by this example
yes | pip install GitPython markdown-editor h5py
yes | conda install pandas matplotlib ipython jupyter nb_conda
Both SACRED and HyperOpt require a MongoDB host. Installation instruction are based on this page.
Execute the command below and also add it to your ~/.bashrc file
# Important: write the full path. Don't use ~/<...>, instead you can use ${HOME}/<...> .
export MONGO_DIR=<your full path to MongoDB>
Download and extract MongoDB tarball
mkdir $MONGO_DIR
cd $MONGO_DIR
mkdir data
mkdir log
wget https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-rhel70-4.0.1.tgz
tar -zxvf mongodb-linux-x86_64-rhel70-4.0.1.tgz
rm mongodb-linux-x86_64-rhel70-4.0.1.tgz
Execute the commands below and also add them to your ~/.bashrc file
export PATH=$MONGO_DIR/mongodb-linux-x86_64-rhel70-4.0.1/bin:$PATH
# Port forwarding function
# Usage example: pfwd hostname {6000..6009}
function pfwd {
for i in ${@:2}
do
echo Forwarding port $i
ssh -N -L $i:localhost:$i $1 &
done
}
wget https://download.robomongo.org/1.2.1/linux/robo3t-1.2.1-linux-x86_64-3e50a65.tar.gz
tar xvf robo3t-1.2.1-linux-x86_64-3e50a65.tar.gz
rm robo3t-1.2.1-linux-x86_64-3e50a65.tar.gz
Login to a mongo host machine in <mongo_host_address>
mongod --fork --dbpath=$MONGO_DIR/data --logpath=$MONGO_DIR/log/mongo.log --logappend
Check that mongo server is running by cat $MONGO_DIR/log/mongo.log and verifying that the output contains the line [initandlisten] waiting for connections on port 27017
# Login to client machine
ssh <client_address>
# Forward MongoDB port to mongo host machine
pfwd <mongo_host_address> 27017
# Activate anaconda environment
conda activate my_env
# Execute experiment
cd <project path>
python sacred_wrapper.py
Monitor results with sacred board (optional)
sacredboard -m sacred_mnist
For every client machine, do the following:
# Login to client machine
ssh <client_address>
# Forward MongoDB port to mongo host machine
pfwd <mongo_host_address> 27017
# Activate anaconda environment
conda activate my_env
cd <project path>
Login to client #1, run the commands above, and execute the hyper-params scheduler:
python hyperopt_search.py
For every other client machines (a "worker" machine), login and execute the worker script:
export GPU_ID=<gpu_id> # select gpu id for multi gpu machine
PYTHONPATH="./" CUDA_VISIBLE_DEVICES=$GPU_ID hyperopt-mongo-worker --mongo=localhost:27017/hyperopt --poll-interval=1 --workdir=-workdir=`mktemp -u -p /tmp/hyperopt/`