tutorials/ray_aws.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Distributed training on AWS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial we will learn: \n",
    "1. How to start a cluster on AWS for use with Classy Vision; \n",
    "2. How to start a new project on the cluster; \n",
    "3. How to launch training jobs on the cluster;\n",
    "\n",
    "## 1. Setup\n",
    "\n",
    "Make sure you have Classy Vision installed, as described in our [Getting started](https://classyvision.ai/tutorials/getting_started) tutorial. \n",
    "\n",
    "For this tutorial we will also need the Classy Vision sources, you can clone it with this command (on your terminal):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! git clone https://github.com/facebookresearch/ClassyVision.git"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial we'll use [Ray](https://github.com/ray-project/ray) to manage the AWS resources. Install Ray and all its required dependencies with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd ./ClassyVision/examples/ray\n",
    "! pip install -r requirements.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should also set up your AWS CLI and credentials as described [here](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html#cli-quick-configuration). To make sure everything is working, run on your terminal:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! aws ec2 describe-instances"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That should print a JSON file with all your current AWS instances (or empty if you don't have any). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Cluster setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have a sample cluster configuration file stored in the Classy Vision repository, under `./examples/ray/cluster_config.yml`. Let's verify that Ray can start the cluster appropriately:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! ray up cluster_config.yml -y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That will take about 10 minutes, and at the end you should see a message explaining how to connect to the cluster. Assuming everything worked successfully, now tear down the cluster:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! ray down cluster_config.yml -y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now set up an EFS volume to store our code and datasets. Follow [this tutorial](https://aws.amazon.com/getting-started/tutorials/create-network-file-system/) to setup the EFS volume in your AWS account. \n",
    "\n",
    "When you're done with that tutorial, go back to the EFS section in the AWS console, find your filesystem there and click `Manage file system access`. Add the `ray-autoscaler-default` security group to the list of security groups allowed to use your EFS volume. That security group should have been created by the `ray up` command we ran earlier.\n",
    "\n",
    "You should now have an identifier for your EFS volume. Open `cluster_config.yml` in your favorite text editor and replace `{{FileSystemId}}` with your own EFS id. We are now ready to launch our cluster again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! ray up cluster_config.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Create a project\n",
    "\n",
    "When it's done, let's attach to the head node of the cluster:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! ray attach cluster_config.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That will give you an SSH session into the head node, which coordinates all the worker nodes in Ray. In our example configuration file, the head node is a CPU-only machine, and the workers all have GPUs.\n",
    "\n",
    "Both the head node and the worker nodes will have the same EFS volume mounted, so we'll use that to send code from the head to the workers. The following commands are meant to run on the head node (e.g. in the terminal prompt you got from `ray attach`). Let's start a project in the EFS folder:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "$ cd efs\n",
    "$ classy-project my_project"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Start training\n",
    "\n",
    "Classy Vision comes with a launcher analogous to `torch.distributed.launch`, but that launches jobs on multiple machines using Ray. To use it, simply run:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "$ python -m classy_vision.distributed.launch_ray --nnodes=2 --use_env ~/efs/my_project/classy_train.py --config ~/efs/my_project/configs/template_config.json --distributed_backend ddp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Your first time running this you might see logs like `Not enough GPUs available`. That's normal, and it's because the worker nodes are still being set up. The `ray up` command should have printed a command line you can use to follow their progress. But there's no need to do anything, the launcher will wait until the workers are available and execute the command automatically.\n",
    "\n",
    "That's it! When that command is done it should print the folder where the checkpoints are.\n",
    "\n",
    "> Note that we specified the full absolute path for the config in the argument list. That's because the `classy_train.py` command is running on a remote machine and we are relying on the fact that the EFS folder is mounted at exactly the same location on the head and worker nodes. Keep that in mind if you modify this setup.\n",
    "\n",
    "> Remember to tear down the cluster with `ray down cluster_config.yml` when you're done. You will be billed as long as the machines are up, even when not using them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Troubleshooting\n",
    "\n",
    "If you hit an error during this tutorial, here are a few things that might help to debug what is going on:\n",
    "\n",
    "### Make sure all workers have initialized properly\n",
    "\n",
    "When the `ray up` command finishes, it prints a command line to tail the logs. It should look like:\n",
    "```bash\n",
    "ray exec cluster_config.yml 'tail -n 100 -f /tmp/ray/session_*/logs/monitor*'\n",
    "```\n",
    "\n",
    "Run that command and look for any errors. When the workers are done initializing, you should see `-- StandardAutoscaler: 2/2 target nodes (0 pending)` printed repeatedly on the logs.\n",
    "\n",
    "### Make sure EFS volumes are mounted on all machines\n",
    "\n",
    "Sometimes the EFS package fails to install on workers. To verify EFS is working, get the worker node IPs with `ray get-worker-ips cluster_config.yml`, then ssh on them with:\n",
    "```bash\n",
    "ssh -i ~/.ssh/ray-autoscaler_us-west-2.pem ubuntu@<WORKER-IP>\n",
    "```\n",
    "\n",
    "Once in a worker machine, run `df -h` to list all the current mounts. Verify `/home/ubuntu/efs` is on that list. If it's not, look for the EFS setup commands on the `cluster_config.yml` file and run them yourself. That should clarify what the issue is. If you didn't setup the EFS security groups correctly (as described in step 2), the `mount` command will hang for a few minutes then fail.\n",
    "\n",
    "## 6. Conclusion\n",
    "\n",
    "In this tutorial we covered how to start using Classy Vision on AWS using Ray. For more information about Ray, check out their [repository](https://github.com/ray-project/ray). The next tutorials ([[1]](https://classyvision.ai/tutorials/classy_model), [[2]](https://classyvision.ai/tutorials/classy_loss), [[3]](https://classyvision.ai/tutorials/classy_dataset)) will demonstrate how to customize the project created by the `classy-project` utility for your own needs. To learn more about how to train models in Classy Vision and how to use Tensorboard to visualize training progress, check out our [Getting started](https://classyvision.ai/tutorials/getting_started) tutorial."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}