Keras_TF_CNN_DeployModel_IoTEdge.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deploy ML Model to Azure IoT Edge\n",
    "\n",
    "\n",
    "In this exercise, we introduce the steps of deploying an ML module through [Azure IoT Edge](https://docs.microsoft.com/en-us/azure/iot-edge/how-iot-edge-works). The purpose is to deploy a trained image classification model to the edge device. When the image data is generated from a particular process pipeline and fed into the edge device, the deployed model is able to make predictions right on the edge device without accessing to the cloud. \n",
    "\n",
    "\n",
    "## Outline<a id=\"BackToTop\"></a>\n",
    "- [Prerequisite](#prerequisite)\n",
    "- [Step 1: Build the trained ML Model into Docker Image](#step1)\n",
    "- [Step 2: Provision and Configure IoT Edge Device](#step2)\n",
    "- [Step 3: Deploy ML Module on IoT Edge Device](#step3)\n",
    "- [Step 4: Test ML Module](#step4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequisite <a id=\"Prerequisite\"></a>\n",
    "\n",
    "Before starting this notebook, you should finish [Keras_TF_CNN_DeployModel.ipynb](Keras_TF_CNN_DeployModel.ipynb) in the same repository (except the last section \"Clean up resources\"). As a recap, we have created following resources in step \"Deploy model as a Web Service\" in this previous execercise:\n",
    "    \n",
    "    - Resource group defined in variable YOUR_RESOURCE_GROUP\n",
    "        * Machine Learning Model Management\n",
    "        * cluster environment (Microsoft.MachineLearningCompute/operationalizationClusters)\n",
    "    - Resource group created during the cluster environment provision (YOUR_RESOURCE_GROUP plus \"-azureml-xxxxx\") \n",
    "        * Container registry\n",
    "        * Container service\n",
    "        * .... a bunch of other automatically provisoned resources\n",
    "\n",
    "In this notebook, we will be using *Machine Learning Model Managment*, and *Container registry* in later instructions. You need to find relevant information about these resources from [Azure portal](https://portal.azure.com). Also keep in mind that they are from two differnt resource group when you locate them from the Azure portal. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Build the trained ML Model into Docker Image <a id=\"step1\"></a>\n",
    "\n",
    "If you have finished [Prerequisite](#prerequisite), you can skip this step. Otherwise, you can follow Section *Create the Azure ML container* in [Deploy Azure Machine Learning as an IoT Edge module - preview](https://docs.microsoft.com/en-us/azure/iot-edge/tutorial-deploy-machine-learning) to deploy your own ML model. The expected output of this step include:\n",
    "\n",
    "    1. A docker image hosted on ACR (Azure Container Registry). This image will be used to create a docker container running on the edge device. \n",
    "    2.  A web service. This web service can be used for testing purpose."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Provision and Configure IoT Edge Device <a id=\"step2\"></a>\n",
    "\n",
    "In [Azure portal](https://portal.azure.com) we need to create following resources in YOUR_RESOURCE_GROUP.\n",
    "\n",
    "Resource 1: An IoT hub. Please follow the sections *Create an IoT hub* and *Register an IoT Edge device* in document [Deploy Azure IoT Edge on a simulated device in Linux or MacOS - preview](https://docs.microsoft.com/en-us/azure/iot-edge/tutorial-simulate-device-linux). After finishing these instructions, copy the value for **Connection string—primary key** from the IoT Edge Device page and save it to *ConnectionString* in the next cell. Notice that we need to have double quote in the command. You will need to execute this command in later instructions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sudo iotedgectl setup --connection-string  \"Your Connection string\"  --nopass\n"
     ]
    }
   ],
   "source": [
    "ConnectionString = '\"Your Connection string\"'\n",
    "print('sudo iotedgectl setup --connection-string ', ConnectionString, ' --nopass')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Resource 2: A Linux VM serving as the edge device. You can use the same Linux deep learning VM you have provisioned for running Keras_TF_CNN_DeployModel.ipynb (Or alternatively, you can use another Linux VM, e.g. Ubuntu server 16.04 LTS). The goal is to configure the VM so that it can run IoT Edge runtime and Docker. To finish the configuration, please follow the Sections *Prerequisites* and *Install and start the IoT Edge runtime* in document [Deploy Azure IoT Edge on a simulated device in Linux or MacOS - preview](https://docs.microsoft.com/en-us/azure/iot-edge/tutorial-simulate-device-linux).\n",
    "\n",
    "   【Tips】: When you are at [this page](https://docs.docker.com/install/linux/docker-ce/ubuntu/#prerequisites), you probably only need to finish sections *Prerequisites - Uninstall old versions* and *Install Docker CE - Install using the repository*. You will also need to use the command generated from the previous cell when following *Install and start the IoT Edge runtime* in document [Deploy Azure IoT Edge on a simulated device in Linux or MacOS - preview](https://docs.microsoft.com/en-us/azure/iot-edge/tutorial-simulate-device-linux)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Deploy ML Module on IoT Edge Device <a id=\"step3\"></a>\n",
    "\n",
    "In an IoT application, modules are function units on an edge device. The output of one module can serve as the input of another module. We often need multiple modules on the same IoT Edge device working together to achieve the realtime data analytics pipeline. ML module is typically one of them. The ML module takes input from a data module, and produces the prediction result as the output.\n",
    "\n",
    "We need following two steps to deploy ML module on IoT Edge device.\n",
    "\n",
    "a: Configure iot edge devce. \n",
    "\n",
    "Since our docker image is hosted on a private container registry (i.e. Azure container registry), please follow Section *Add registry credentials to your Edge device* in document [Deploy Azure Machine Learning as an IoT Edge module - preview](https://docs.microsoft.com/en-us/azure/iot-edge/tutorial-deploy-machine-learning). \n",
    "\n",
    "【Tips】: You need to go to Azure portal to find out information about your container registry. Notice that the name of your resource group is likely to be YOUR_RESOURCE_GROUP plus \"-azureml-xxxxx\". Please see below screenshots for your reference. Save relevant information including *Login server*, *User name*, and *password* into the variables in the next cell. You will need to use the command generated from the next cell to finish this step.\n",
    "\n",
    "![containerregistry](./imgs/containerRegistry.PNG)\n",
    "\n",
    "![containerregistry2](./imgs/containerRegistry2.PNG)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sudo iotedgectl login --address Login_server --username  User_name --password password\n"
     ]
    }
   ],
   "source": [
    "registryLoginServer = \"Login_server\" # replace with the real Login Server\n",
    "registryUsername = \"User_name\" # replace with the real User name\n",
    "registryPassword = \"password\" # replace with the real password\n",
    "# No double quotes are needed in this command.\n",
    "print('sudo iotedgectl login --address', registryLoginServer, '--username ', registryUsername, '--password', registryPassword)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "b: Deploy ML module\n",
    "    \n",
    "Essentially, the objective is to deploy the ML container to the IoT Edge device.\n",
    "\n",
    "1. On the Azure portal, navigate to your IoT hub.\n",
    "2. Click *IoT Edge (preview)* and select your IoT Edge device.\n",
    "3. Select *Set modules*.\n",
    "4. Select *Add IoT Edge Module*.\n",
    "5. In the Name field, enter a name, `yourmodulename`. \n",
    "6. In the Image field, enter your image location; for example `mlcrpacrdf78885275b6.azurecr.io/yanzimgclussrvc:1`.\n",
    "    \n",
    "    【Tips】: You can find the image location in your machine learning account manamgement - `Model Management` - `Images` - click the image you have created - copy the address from `Location` field. Please see below screenshots for your reference.\n",
    "    \n",
    "7. In the *Container Create Options* field, set the following configuration. You can change the HostPort Binding port number to your desired port number. \n",
    "\n",
    "    {\n",
    "      \"HostConfig\": {\n",
    "        \"PortBindings\": {\n",
    "          \"5001/tcp\": [\n",
    "            {\n",
    "              \"HostPort\": \"5001\"\n",
    "            }\n",
    "          ]\n",
    "         }\n",
    "        }\n",
    "       }\n",
    "\n",
    "8.\tClick *Save*.\n",
    "9.\tBack in the *Add Modules* step, click *Next*.\n",
    "10.\tIn the *Specify Routes* step. Put the following:\n",
    "      {}      \n",
    "      \n",
    "11.\tSelect Next.\n",
    "12.\tIn the *Review Deployment* step, click *Submit*.\n",
    "13.\tReturn to the device details page and click *Refresh*. You should see the new `yourmodulename` running.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "![mm](./imgs/modelmanagement.PNG)\n",
    "\n",
    "![mm2](./imgs/modelmanagement2.PNG)\n",
    "    \n",
    "    \n",
    "An alternative example you can reference is the section *Run the solution* in document [Deploy Azure Machine Learning as an IoT Edge module - preview](https://docs.microsoft.com/en-us/azure/iot-edge/tutorial-deploy-machine-learning)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['score.py', 'kerastfmodel.h5', 'automobile8.png']"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check if the image, scoring script and model are in the same folder.\n",
    "os.listdir(model_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['score.py', 'kerastfmodel.h5', 'automobile8.png']"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Change the current working directory to model_path\n",
    "os.chdir(model_path)\n",
    "#list files in current working directory\n",
    "os.listdir(os.curdir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Test ML Module <a id=\"step4\"></a>\n",
    "\n",
    "Now we have deployed your ML model as a module on the IoT edge device. How to test it and make sure it functions correctly?\t\n",
    "\n",
    "First of all, we should test the deployed web service and make sure it works. We assume this step has been completed in step \"Test Web Service\" in our previous exercise [Keras_TF_CNN_DeployModel.ipynb](./Keras_TF_CNN_DeployModel.ipynb). \n",
    "Secondly, we can test to score the deployed ML module from the IoT edge device with the service endpoint url address. We will show detailed instructions for this approach in the remaining of this section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/home/mylogin/notebooks/DeepLearningModelDeployment/o16n/kerastfmodel/o16n/kerastfmodel'"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# make sure you have correct working directory settings\n",
    "import os\n",
    "local_path = os.getcwd()\n",
    "o16n_path = os.path.join(local_path,'o16n')  \n",
    "model_path = os.path.join(o16n_path,'kerastfmodel')\n",
    "model_path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['conda_dependencies.yml', 'score.py', 'kerastfmodel.h5', 'automobile8.png']"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Change the current working directory to model_path\n",
    "os.chdir(model_path)\n",
    "#list files in current working directory\n",
    "os.listdir(os.curdir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# service endpoint url address\n",
    "url = 'Your Scoring URL Here!!'\n",
    "#url = 'http://137.117.32.176:5001/score'\n",
    "#url = 'http://localhost:5001/score'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The service endpoint url address should have the format of \"http://localhost:[port_number]/score\" or \"http://[edge_device_ip]:[port_number]/score\". If your ipython notebook and ML module are running from the same VM, you can use `localhost`. Otherwise you should use the ip address of the edge device VM. The `port_number` is predefined in step b of [Deploy ML Module on IoT Edge Device](#step3)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAIAAAD8GO2jAAAH3UlEQVR4nG1WaZMUxxHNzDq6+pp7Z6/ZA8Na6ERYobBDcsgfbP8F6/86QgopFIAQ1mEdWCwLiJ17eqanj6pKf5iFBUR/6ujoyJfvvcxXhYgIr3sIxZ//8vG1k+uj4XmWLfLSEni066paOyAQ2qMQQJowjvRqvfzm3r2qqn5fTb62OgCQgF9++e9sNu12W91++1p/12g1fPJgORuXte1sbXc6/VAlBfrHvz364d/fa6Xquv59nRcBEOAZPnokXK0W7UbqORKx3j4+YO9rqJTE+z/fL8qHioRIMNjtHnROfvr222K1JiLv/auNvvxOiIJQIAoAFEi2LKbj8floXnlhyWztXmm39rJ5tZpXp7+c3bl9+/GTJ0EU7x0cA5JSioiI6EWhXpIIAREQEYVUUWjiQIfaSJCisJhbaZRgZBfEYdtoGWiV5Uuo2ECws3co9W0Gr7V2zpVl6Zx7nQeIxpjt7e2d3f3tfj9SKpQUsA0DbbIl51UYhXu7u//4598To5MotCjIRGvre/39uNEsljwYHOR5fnp6muf5hsdLAKHWH964+dHHf90ZHIRhiK7mqlo+PYPVAhSRkn+4etQ9PkRkLRAQ5rO8GGZPpjPu77174+Z3d77q9/tVWU7G4zzPX2WACNf299/a3m2STnQQBCLQYaAV9DuSrYkSE0atVqvZboZxFKYpSTH86uv6pzM9ntpm428ffcLrdV0X3tbW2lc9YOY41L1A/fD55yYwW8f7dcnIAIBSKBC6JOG9hzx3zKl1IIRCHH32pfv8VjWZB388Sd85uXHj/S+++Ozevf8sl8vnPl8yaJggqquqWBX5bJkvBenRaGpry8RAAIBEqKTUQh4Ndt9553pgK+ISu4F6vCjOfswGzXa7/d577925c8u+MKyXAAp8JwyiwZ731XKVpVH3tydPf/zxp8rVOjLOuyzLpJBJZP707tvHh7vz+XQZSX39YPHgwXk5AVcENr5yfOXo+Mrdb75+zuByD4hRKGVaTQtcrAsp5cnJSbfb0Vr3tnqDwX6apo1m4+DwMIqjZbYcD8fnoyyzKt46SOMueW+MNsZcuXqCKF7DoJE0Zas9qSshg7qs18Xq6Oiwt9VFBUkSRlEoJYXGtBqNOAoXi4wZt65ebUWx7B3MHvz8K3ktRW3t1WsnSdrIFtOXxpSIjo6v7r319mi1CHWDGLNsOl+kzVZiIkWKhBBx2DVBECiptVwslpHRyXYzDKJ1t8l+gb+dV2VZAW31+91edwPAzBcASqmt3W1vDDObsCkQsjz79cF9IlKKhCClpBBCSWGU0FIorXYPDnv9RoDSVUV9JhndbDZGHYZxnIYGAJifScQASimHfl0USdTs9PqETuRSKaWklEJoIqVUEGgTmMjoKIparfbh8XHSTgUiLaaBlAB2na81oEdOFEoEC4iwYcCMAJ4dEXrny6oMQ5mmqRBCCCkJNZGSUmttjNGB0kEACLPpBCUGgcznc64rIrCutqtMettqplopWzsAvDRZShEEerLMJ8xaEwrYuITAkplISCmVkibQYWjiKE4b6c6y32ml2XzGtgL2AFyWBcbh3t6uDnRerQFfMJkEem/LqqirEtg6dszsvWfvkR0CEpGQIlBSaxUExhjTbac7/a4AXmZz761n6xmcs91OJwzDWZbjcwZEVFXV4yePhrPCO8/eevB88XhgDwjAgAiSME3TqqoQIE1Mu5E0ktgoAQJXq2W7u4PISBho/dIeIFFVFqenp0+na4HovW2121LKs7MzZs98Ee7eey3lp5/+6+7du+fn587WVZE3kvjDD97v9zur1bLTA+ctESmlXg07QBxPJmePR9u9LSWFMcYYMxwOmT2DBwAE8N4nUXT9jTfOz8/v3/8fABTrstPups22tdbamtlbZ40JwtAAMABKBGAA75xlQqmLYg0EjhkR4zgGBGcdbdQBAO8DpdI4bqaJEkSIJkn393bjKJmMz5GpnSa+KtNWR+kAAABBAgMCsOfKcpg0tNZIJIQgomazee3aCTsbBYHWWgeBVhLYx1F0dHD0wc2bg8G+IOp1u6EJJ9OZtSxJlM53Ot3NlvFzkz17x9zpdN98880kSTrtdhInURT97ZNP2HsCZmYhSAgxnc2++OqWZ273+kEUh4EpKjucPB5N52XtG83OsCik1nVVXYz4BsBa+/D0YVVXeZ4Ph8OHpw9xo6CU7D2w28wTkqisQ0QhJABLgSYIiKgsinxdHA4GX966Xa1zT3I0OicAYJZSSa20VPLR40f5OnfWERF75meHBhEy+4tsQYTLKGZEoGe5772Poqj+vlgvl5PJZLnIBAIj4OHgYGdnRwjx7XffLVdLRNRah8ZkWcaetdbdbgcJnXPOO2Z2lwviwTtmX1aVsw4899qdZhq7qhwM9sq6zotyni2ltXa1WiFetLmRqywr9szA1trxZOzZ86Z/hAut8KLtixEHJGZb5r3D3VYjSZK4cm40meVFIdfrNREJIYQQSkkAQETvrVISEREREIVAxM2NDYlZEG4ucIJQKWmCMDCBFioxZqffkwTLdT6azp8OR6PJXEqlTBRqrY1R7N2mqAAkJNoUEiTkJYASF0O8+ZNoE4iITAC4WObrfDVeZOP5YjqblWUt18WaMlJKEjsCfnZS4zP3EBGQ4CJYgS8+Pd9/gGeOoGd21q7XxaqqS+s2t8f/A7cwEQqH36NlAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<PIL.PngImagePlugin.PngImageFile image mode=RGB size=32x32 at 0x7FCFE178C7F0>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from PIL import Image\n",
    "# the test image to predict on\n",
    "test_img_name = 'automobile8.png'\n",
    "Image.open(test_img_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "automobile8.png\n"
     ]
    }
   ],
   "source": [
    "import base64\n",
    "import json\n",
    "\n",
    "img_file_name = os.path.split(test_img_name)[1]\n",
    "print(img_file_name)\n",
    "# prepare a test image\n",
    "with open(test_img_name, 'rb') as file:\n",
    "  encoded = base64.b64encode(file.read())\n",
    "img_dict = {img_file_name: encoded.decode('utf-8')}\n",
    "body = json.dumps(img_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Response [200]>"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import requests\n",
    "\n",
    "# call the web service end point\n",
    "headers = {'Content-Type':'application/json'}\n",
    "response = requests.post(url, headers=headers, data=body)\n",
    "response"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output from above cell should be `<Response [200]>`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'[{\"automobile8.png\": \"automobile\"}]'"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prediction = json.loads(response.content.decode('ascii'))\n",
    "prediction # The firt part is the test image's name, and the second part is the predicted category. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clean up resources <a id=\"cleanup\"></a>\n",
    "\n",
    "When you finish this example, you may want to avoid unnecessary cost by cleaning up the Azure resources you have provisioned. You need to delete two resource groups: YOUR_RESOURCE_GROUP and YOUR_RESOURCE_GROUP plus\"-azureml-xxxxx\". The exact name for the second resource group can be found in your Azure portal. For example, my resource group name is YOUR_RESOURCE_GROUP = \"yanzimgrg\" and the other system created resource group name is \"yanzimgrg-azureml-a0c61\". I then need to execute following commands to delete these two resource groups.\n",
    "\n",
    "    az group delete -n yanzimgrg\n",
    "    az group delete -n yanzimgrg-azureml-a0c61"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "az group delete -n  yanzimgrg\n"
     ]
    }
   ],
   "source": [
    "# Delete resource group. Execute this command in the console. \n",
    "# Execute below command in CLI console, at the prompt type \"y\" (Q: Are you sure you want to perform this operation? (y/n):) \n",
    "# az group delete -n $YOUR_RESOURCE_GROUP\n",
    "print(\"az group delete -n \", YOUR_RESOURCE_GROUP)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}