tutorials/deploy-edge/ase-gpu.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deploying a ML model as web service on Azure Stack\n",
    "This notebook shows the steps to : registering a model, creating an image, provisioning,deploying a service using Iot Edge on Azure Edge."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install --upgrade azureml-contrib-services tensorflow"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Get workspace"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.workspace import Workspace\n",
    "\n",
    "ws = Workspace.from_config()\n",
    "ws"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Download the model\n",
    "\n",
    "Prior to registering the model, you should have a TensorFlow [Saved Model](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/saved_model/README.md) in the `resnet50` directory. This cell will download a [pretrained resnet50](http://download.tensorflow.org/models/official/20181001_resnet/savedmodels/resnet_v1_fp32_savedmodel_NCHW_jpg.tar.gz) and unpack it to that directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import requests\n",
    "import shutil\n",
    "import tarfile\n",
    "import tempfile\n",
    "\n",
    "from io import BytesIO\n",
    "\n",
    "model_url = \"http://download.tensorflow.org/models/official/20181001_resnet/savedmodels/resnet_v1_fp32_savedmodel_NCHW_jpg.tar.gz\"\n",
    "\n",
    "archive_prefix = \"./resnet_v1_fp32_savedmodel_NCHW_jpg/1538686758/\"\n",
    "target_folder = \"resnet50\"\n",
    "\n",
    "if not os.path.exists(target_folder):\n",
    "    response = requests.get(model_url)\n",
    "    archive = tarfile.open(fileobj=BytesIO(response.content))\n",
    "    with tempfile.TemporaryDirectory() as temp_folder:\n",
    "        archive.extractall(temp_folder)\n",
    "        shutil.copytree(\n",
    "            os.path.join(temp_folder, archive_prefix), target_folder\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Register the model\n",
    "Register an existing trained model, add description and tags."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.model import Model\n",
    "\n",
    "model = Model.register(\n",
    "    model_path=\"resnet50\",  # This points to the local directory to upload.\n",
    "    model_name=\"resnet50\",  # This is the name the model is registered as.\n",
    "    tags={\"area\": \"Image classification\", \"type\": \"classification\"},\n",
    "    description=\"Image classification trained on Imagenet Dataset\",\n",
    "    workspace=ws,\n",
    ")\n",
    "\n",
    "print(model.name, model.description, model.version)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deploy the model as a web service to Edge\n",
    "\n",
    "We begin by writing a score.py file that will be invoked by the web service call. The init() function is called once when the container is started so we load the model using the Tensorflow session. The run() function is called when the webservice is invoked for inferencing. After running the code below you should see a score.py file in the same folder as this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile score.py\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import json\n",
    "import os\n",
    "from azureml.contrib.services.aml_request import AMLRequest, rawhttp\n",
    "from azureml.contrib.services.aml_response import AMLResponse\n",
    "\n",
    "\n",
    "def init():\n",
    "    global session\n",
    "    global input_name\n",
    "    global output_name\n",
    "\n",
    "    session = tf.Session()\n",
    "\n",
    "    # AZUREML_MODEL_DIR is an environment variable created during deployment.\n",
    "    # It is the path to the model folder (./azureml-models/$MODEL_NAME/$VERSION)\n",
    "    # For multiple models, it points to the folder containing all deployed models (./azureml-models)\n",
    "    model_path = os.path.join(os.getenv(\"AZUREML_MODEL_DIR\"), \"resnet50\")\n",
    "    model = tf.saved_model.loader.load(session, [\"serve\"], model_path)\n",
    "    if len(model.signature_def[\"serving_default\"].inputs) > 1:\n",
    "        raise ValueError(\"This score.py only supports one input\")\n",
    "    input_name = [\n",
    "        tensor.name\n",
    "        for tensor in model.signature_def[\"serving_default\"].inputs.values()\n",
    "    ][0]\n",
    "    output_name = [\n",
    "        tensor.name\n",
    "        for tensor in model.signature_def[\"serving_default\"].outputs.values()\n",
    "    ]\n",
    "\n",
    "\n",
    "@rawhttp\n",
    "def run(request):\n",
    "    if request.method == \"POST\":\n",
    "        reqBody = request.get_data(False)\n",
    "        resp = score(reqBody)\n",
    "        return AMLResponse(resp, 200)\n",
    "    if request.method == \"GET\":\n",
    "        respBody = str.encode(\"GET is not supported\")\n",
    "        return AMLResponse(respBody, 405)\n",
    "    return AMLResponse(\"bad request\", 500)\n",
    "\n",
    "\n",
    "def score(data):\n",
    "    result = session.run(output_name, {input_name: [data]})\n",
    "    return json.dumps(result[1].tolist())\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    init()\n",
    "    with open(\"test_image.jpg\", \"rb\") as f:\n",
    "        content = f.read()\n",
    "        print(score(content))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now create the deployment configuration objects "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the web service configuration (using default here)\n",
    "from azureml.core.model import InferenceConfig\n",
    "\n",
    "# from azureml.core.webservice import AksWebservice\n",
    "from azureml.core.conda_dependencies import CondaDependencies\n",
    "from azureml.core.environment import Environment, DEFAULT_GPU_IMAGE\n",
    "\n",
    "env = Environment(\"deploytoedgeenv\")\n",
    "# Please see [Azure ML Containers repository](https://github.com/Azure/AzureML-Containers#featured-tags)\n",
    "# for open-sourced GPU base images.\n",
    "env.docker.base_image = DEFAULT_GPU_IMAGE\n",
    "env.python.conda_dependencies = CondaDependencies.create(\n",
    "    conda_packages=[\"tensorflow-gpu==1.12.0\", \"numpy\"],\n",
    "    pip_packages=[\"azureml-contrib-services\", \"azureml-defaults\"],\n",
    ")\n",
    "\n",
    "inference_config = InferenceConfig(entry_script=\"score.py\", environment=env)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create container image in Azure ML\n",
    "Use Azure ML to create the container image. This step will likely take a few minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# provide name of azure contaienr image and tag\n",
    "imagename = \"tfgpu\"\n",
    "imagelabel = \"0.2\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Builds an image in ACR.\n",
    "\n",
    "package = Model.package(\n",
    "    ws,\n",
    "    [model],\n",
    "    inference_config=inference_config,\n",
    "    image_name=imagename,\n",
    "    image_label=imagelabel,\n",
    ")\n",
    "package.wait_for_creation(show_output=True)\n",
    "\n",
    "print(\"ACR:\", package.get_container_registry)\n",
    "print(\"Image:\", package.location)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup Azure Stack Edge \n",
    "\n",
    "Follow [documentation](https://review.docs.microsoft.com/en-us/azure/databox-online/azure-stack-edge-gpu-deploy-sample-module-marketplace?branch=release-preview-ase-gpu) to setup compute and validate if GPU on ASE are up and runing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup Azure IoT Edge device\n",
    "\n",
    "Follow [documentation](https://docs.microsoft.com/en-us/azure/iot-edge/quickstart-linux) to setup a Linux VM as an Azure IoT Edge device"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy container to Azure IoT Edge device"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "acr_name = package.location.split(\"/\")[0]\n",
    "reg_name = acr_name.split(\".\")[0]\n",
    "subscription_id = ws.subscription_id\n",
    "\n",
    "print(\"{}\".format(acr_name))\n",
    "print(\"{}\".format(subscription_id))\n",
    "\n",
    "# TODO: Derive image_location through code.\n",
    "image_location = acr_name + \"/\" + imagename + \":\" + imagelabel\n",
    "\n",
    "print(\"{}\".format(image_location))\n",
    "\n",
    "# Fetch username, password of ACR.\n",
    "from azure.mgmt.containerregistry import ContainerRegistryManagementClient\n",
    "from azure.mgmt import containerregistry\n",
    "\n",
    "client = ContainerRegistryManagementClient(ws._auth, subscription_id)\n",
    "result = client.registries.list_credentials(\n",
    "    ws.resource_group, reg_name, custom_headers=None, raw=False\n",
    ")\n",
    "\n",
    "username = result.username\n",
    "password = result.passwords[0].value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a deployment.json file using the template json. Then push the deployment json file to the IoT Hub, which will then send it to the IoT Edge device. The IoT Edge agent will then pull the Docker images and run them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile iotedge-tf-template-gpu.json\n",
    "{\n",
    "    \"modulesContent\": {\n",
    "        \"$edgeAgent\": {\n",
    "            \"properties.desired\": {\n",
    "                \"schemaVersion\": \"1.0\",\n",
    "                \"runtime\": {\n",
    "                    \"type\": \"docker\",\n",
    "                    \"settings\": {\n",
    "                        \"minDockerVersion\": \"v1.25\",\n",
    "                        \"loggingOptions\": \"\",\n",
    "                        \"registryCredentials\": {\n",
    "                            \"__REGISTRY_NAME\": {\n",
    "                                \"username\": \"__REGISTRY_USER_NAME\",\n",
    "                                \"password\": \"__REGISTRY_PASSWORD\",\n",
    "                                \"address\": \"__REGISTRY_NAME.azurecr.io\",\n",
    "                            }\n",
    "                        },\n",
    "                    },\n",
    "                },\n",
    "                \"systemModules\": {\n",
    "                    \"edgeAgent\": {\n",
    "                        \"type\": \"docker\",\n",
    "                        \"settings\": {\n",
    "                            \"image\": \"mcr.microsoft.com/azureiotedge-agent:1.0\",\n",
    "                            \"createOptions\": \"{}\",\n",
    "                            \"env\": {\"UpstreamProtocol\": {\"value\": \"MQTT\"}},\n",
    "                        },\n",
    "                    },\n",
    "                    \"edgeHub\": {\n",
    "                        \"type\": \"docker\",\n",
    "                        \"status\": \"running\",\n",
    "                        \"restartPolicy\": \"always\",\n",
    "                        \"settings\": {\n",
    "                            \"image\": \"mcr.microsoft.com/azureiotedge-hub:1.0\",\n",
    "                            \"createOptions\": '{\"User\":\"root\",\"HostConfig\":{\"PortBindings\":{\"5671/tcp\":[{\"HostPort\":\"5671\"}], \"8883/tcp\":[{\"HostPort\":\"8883\"}],\"443/tcp\":[{\"HostPort\":\"443\"}]}}}',\n",
    "                            \"env\": {\"UpstreamProtocol\": {\"value\": \"MQTT \"}},\n",
    "                        },\n",
    "                    },\n",
    "                },\n",
    "                \"modules\": {\n",
    "                    \"__MODULE_NAME\": {\n",
    "                        \"version\": \"1.0\",\n",
    "                        \"type\": \"docker\",\n",
    "                        \"status\": \"running\",\n",
    "                        \"restartPolicy\": \"always\",\n",
    "                        \"settings\": {\n",
    "                            \"image\": \"__REGISTRY_IMAGE_LOCATION\",\n",
    "                            \"createOptions\": '{\"HostConfig\":{\"PortBindings\":{\"5001/tcp\":[{\"HostPort\":\"5001\"}], \"8883/tcp\":[{\"HostPort\":\"5002\"}],},\"runtime\":\"nvidia\"},\"WorkingDir\":\"/var/azureml-app\"}',\n",
    "                        },\n",
    "                    }\n",
    "                },\n",
    "            }\n",
    "        },\n",
    "        \"$edgeHub\": {\n",
    "            \"properties.desired\": {\n",
    "                \"schemaVersion\": \"1.0\",\n",
    "                \"routes\": {\n",
    "                    \"machineLearningToIoTHub\": \"FROM /messages/modules/__MODULE_NAME/outputs/amlOutput INTO $upstream\"\n",
    "                },\n",
    "                \"storeAndForwardConfiguration\": {\"timeToLiveSecs\": 7200},\n",
    "            }\n",
    "        },\n",
    "        \"__MODULE_NAME\": {\"properties.desired\": {}},\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "module_name = \"tfgpu\"\n",
    "\n",
    "file = open(\"iotedge-tf-template-gpu.json\")\n",
    "contents = file.read()\n",
    "contents = contents.replace(\"__MODULE_NAME\", module_name)\n",
    "contents = contents.replace(\"__REGISTRY_NAME\", reg_name)\n",
    "contents = contents.replace(\"__REGISTRY_USER_NAME\", username)\n",
    "contents = contents.replace(\"__REGISTRY_PASSWORD\", password)\n",
    "contents = contents.replace(\"__REGISTRY_IMAGE_LOCATION\", image_location)\n",
    "with open(\"deployment_gpu.json\", \"wt\", encoding=\"utf-8\") as output_file:\n",
    "    output_file.write(contents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sending deployment ot the edge device "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## working example !az iot edge set-modules --device-id juanedge --hub-name yadavmAiMLGpu --content deployment_gpu.json\n",
    "\n",
    "# UNCOMMENT TO RUN, once you put your device's info\n",
    "#!az iot edge set-modules --device-id <replace with iot edger device name> --hub-name <repalce with iot hub name> --content deployment_gpu.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Test the web service\n",
    "We test the web sevice by passing the test images content."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# downloading labels for imagenet that resnet model was trained on\n",
    "import requests\n",
    "\n",
    "classes_entries = requests.get(\n",
    "    \"https://raw.githubusercontent.com/Lasagne/Recipes/master/examples/resnet50/imagenet_classes.txt\"\n",
    ").text.splitlines()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "import requests\n",
    "\n",
    "## Run it like so, for example:\n",
    "# do_inference(\"snowleopardgaze.jpg\", \"http://51.141.178.47:5001/score\")\n",
    "\n",
    "\n",
    "def do_inference(myfilename, myscoring_uri):\n",
    "\n",
    "    test_sample = open(myfilename, \"rb\").read()\n",
    "\n",
    "    try:\n",
    "        scoring_uri = (\n",
    "            # You can construct your own, passing only the ip in arguments\n",
    "            # \"http://<replace with yout edge device ip address>:5001/score\"\n",
    "            #\n",
    "            myscoring_uri\n",
    "        )\n",
    "\n",
    "        # Set the content type\n",
    "        headers = {\"Content-Type\": \"application/json\"}\n",
    "\n",
    "        # Make the request\n",
    "        resp = requests.post(scoring_uri, test_sample, headers=headers)\n",
    "\n",
    "        print(\"Found a ::\" + classes_entries[int(resp.text.strip(\"[]\")) - 1])\n",
    "    except KeyError as e:\n",
    "        print(str(e))"
   ]
  }
 ],
 "metadata": {
  "authors": [
   {
    "name": "vaidyas"
   }
  ],
  "kernelspec": {
   "display_name": "Python 3.8",
   "language": "python",
   "name": "python3.8"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}