A shared domain approach to Machine Learning.
It is possible to run an interactive python notebook on a cluster. This is
preferable to some. For this to work, SSH into your system using a
ControlMaster.
On your local computer in a new terminal, execute ssh -O forward -L 8888:localhost:8888 -S "path-to-control/control:gateway.net:22:login@jhu.edu gateway.net. On MARCC load the python-3.6 and gcc5.5 modules. To run a python
notebook, execute bazel run notebooks. Visit the url that the command
provides and change the link from bc-login0x -> localhost.
- locally
ssh marcc - locally
./juypter(Does port forwarding.) - marcc
module load python/3.6-anaconda - marcc
bazel run notebooks - Go to link, change
bcloginXX->localhost - Rejoice
Install Bazel on MARCC, (A precompiled version can be provided on request).
To build this repository on MARCC for HPC use, start an interactive debug
session and run bazel build src/multiencoder. After this, most files
should be cached such that the project may be built from the head node.
Run scripts/checkout.sh. Optionally set ME_EXPERIMENTS beforehand. The name
of the checkout experiment will reflect the git branch by default.
To run a job on Slurm, execute sbatch deploy-slurm.sh. To allow for use on
your particular cluster, optionally set the following environmental variables
globally.
# The location to save outputs and model files. This will default to
# ./experiments/ otherwise.
export ME_EXPERIMENTS=~/scratch/experiments
# Any additional commands that must be run to make your system run. For
# instance, on MARCC, this file might load modules. This may take some tweaking.
export ME_PROVISION=./provision.shPlease start and join a docker swarm.
Train and evaluate on the swarm run ./deploy-docker.sh. To develop and test
on the swarm run docker-compose -f docker/test.yaml up- this will spawn a
notebook and a build/ test service.
You may need to create the folders /tmp/build_root and /tmp/build_output
with root ownership.