05_run_the_batch_ai_job_from_docker.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Run the Batch AI job from Docker\n",
    "This notebook will walk through the steps to create a Docker image, test it locally, and upload it to Dockerhub to be consumed later in the tutorial when we operaitonalize the workflow.\n",
    "\n",
    "This notebook assumes a basic understanding of Docker and Dockerfiles and how to use the Docker commandline to create an Docker image. To learn more about Docker, see [here.](https://docs.docker.com/get-started/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we need to set up some variables for Docker:\n",
    "- `docker_image_tag` - this is the tag that you wish to name your Docker image\n",
    "- `docker_user` - your Dockerhub username"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "docker_image_tag = \"<docker-image-tag>\"\n",
    "docker_user = \"<docker-username>\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save these variables to the .env file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!dotenv set DOCKER_IMAGE_TAG $docker_image_tag\n",
    "!dotenv set DOCKER_USER $docker_user"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the .env file so you can access these variables as environment variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext dotenv\n",
    "%dotenv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a requirements.txt file that will be used in the docker image. Because the Docker image is only responsible for running the `create_job.py` script, we only need to include the packages that that script requires."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile az/requirements.txt\n",
    "azure==4.0.0\n",
    "azure-mgmt-batchai==2.0.0\n",
    "iteration-utilities==0.7.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Write the Dockerfile that will be used to create the Docker image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile az/Dockerfile\n",
    "FROM python:3.6-jessie\n",
    "WORKDIR /job\n",
    "ADD create_job.py /job\n",
    "ADD util /job/util\n",
    "ADD requirements.txt /job\n",
    "\n",
    "RUN pip install --trusted-host pypi.python.org -r requirements.txt\n",
    "\n",
    "CMD [\"python\", \"create_job.py\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Build the Dockerfile."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! cd az && sudo docker build -t $DOCKER_IMAGE_TAG ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now lets test that the image worked as expected. Run the image in the background (detached) and pass it the environment variables needed to run the `create_job.py` script."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "sudo docker run -d --name $DOCKER_IMAGE_TAG \\\n",
    "    -e AAD_CLIENT_ID=$AAD_CLIENT_ID \\\n",
    "    -e AAD_SECRET=$AAD_SECRET \\\n",
    "    -e AAD_TENANT=$AAD_TENANT \\\n",
    "    -e STORAGE_ACCOUNT_NAME=$STORAGE_ACCOUNT_NAME \\\n",
    "    -e STORAGE_ACCOUNT_KEY=$STORAGE_ACCOUNT_KEY \\\n",
    "    -e ADMIN_USER_NAME=$ADMIN_USER_NAME\\\n",
    "    -e ADMIN_USER_PASSWORD=$ADMIN_USER_PASSWORD \\\n",
    "    -e SUBSCRIPTION_ID=$SUBSCRIPTION_ID \\\n",
    "    -e RESOURCE_GROUP=$RESOURCE_GROUP \\\n",
    "    -e REGION=$REGION \\\n",
    "    -e AZURE_CONTAINER_NAME=$AZURE_CONTAINER_NAME \\\n",
    "    -e FS_SCRIPT=$FS_SCRIPT \\\n",
    "    -e FS_STYLE_IMG=$FS_STYLE_IMG \\\n",
    "    -e FS_CONTENT_DIR=$FS_CONTENT_DIR \\\n",
    "    -e FS_OUTPUT_DIR_PREFIX=$FS_OUTPUT_DIR_PREFIX \\\n",
    "    -e FS_LOGGER_DIR_PREFIX=$FS_LOGGER_DIR_PREFIX \\\n",
    "    -e WORKSPACE=$WORKSPACE \\\n",
    "    -e EXPERIMENT_PREFIX=$EXPERIMENT_PREFIX \\\n",
    "    -e JOB_NAME_PREFIX=$JOB_NAME_PREFIX \\\n",
    "    -e JOB_NODE_COUNT=$JOB_NODE_COUNT \\\n",
    "    -e JOB_BATCH_SIZE=$JOB_BATCH_SIZE \\\n",
    "    -e CLUSTER_NAME=$CLUSTER_NAME \\\n",
    "    -e CLUSTER_CONTAINER_MNT_PATH=$CLUSTER_CONTAINER_MNT_PATH \\\n",
    "    -e STYLE_WEIGHT=$STYLE_WEIGHT \\\n",
    "    -e CONTENT_WEIGHT=$CONTENT_WEIGHT \\\n",
    "    -e NUM_STEPS=$NUM_STEPS \\\n",
    "    -e IMAGE_SIZE=$IMAGE_SIZE \\\n",
    "    $DOCKER_IMAGE_TAG"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets take a look at the logs of the running Docker container to see that the output looks as expected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!sudo docker logs $DOCKER_IMAGE_TAG"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've testing the image locally, lets push it up to Dockerhub so that we can use it when operationalizing this workflow with Logic Apps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!sudo docker tag $DOCKER_IMAGE_TAG $DOCKER_USER/$DOCKER_IMAGE_TAG\n",
    "!sudo docker push $DOCKER_USER/$DOCKER_IMAGE_TAG"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "In this tutorial, we've setup our `create_job.py` script to run inside of a Docker image. We've also pushed the image that we built locally up to Dockerhub to be consumed later on in the tutorial.\n",
    "\n",
    "Next, lets use this Docker image to operationalize this workflow. To do so, we need to [deploy Logic App and trigger the workflow.](06_deploy_and_test_logic_apps.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:batchscoringdl]",
   "language": "python",
   "name": "conda-env-batchscoringdl-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}