Because the complexity of neural networks keeps increasing, the amount of time required to train and evaluate them has become a serious bottleneck. But there has been a second development lately: Cloud Computing. Distributed computing resources have never been cheaper and, when required, additional resources can be spun up at the click of a button. This is where our project comes into play. The NAS Scheduler combines these two advancements, Deep Learning and Cloud Computing, to accelerate the training process and manage the training of several models at a time. We are developing a scheduler for deep learning jobs to provide streamlined resource scaling using Docker and Kubernetes as well as integrating support for the distributed training library, Horovod.
Our final architecture consists of a scheduler for managing resources, a progressor for retrieving status updates, a cluster of workers responsible for performing the actual training work and a command-line client through which a user may interact with the system. The scheduler enables us to efficiently train several deep learning jobs at once. Based on different scheduling algorithms, it decides which jobs will be executed at which point in time and how much resources they may use. The progressor monitors the worker cluster in real-time to inform the user as well as the scheduler about the progress on the running jobs and resources used in the cluster.
pip3 install -r requirements.txt
To launch redis inside the k8s cluster:
kubectl apply -f images/redis-master.yml
First to start the scheduler, run the daemon with
python daemon.py
Then you may interact with the scheduler by submitting commands through the client, for example to run a specific job:
python submit job_repo/experiment-cifar10-resnext110.yaml
See client.py for a more detailed description of the available commands.
You can run the unit tests via
python -m pytest tests --ignore=tests/integration_test.py
and the integration tests via
python -m pytest tests/integration_test.py
To follow our coding conventions, run
black --line-length 120 .
to format your code. Also add type annotations and comments to every function you add to the code base.
If you experience errors, this might be due to the fact that there are too many dangling pods and jobs on the cluster. Run
microk8s kubectl delete pods --all
microk8s kubectl delete jobs --all
to fix this. Another reason might be that there are too many dangling docker images, which drain the server's storage:
docker rm $(docker ps -a -q -f status=exited)
docker image prune