python-sdk/tutorials/automl-with-azureml/image-object-detection/auto-ml-image-object-detection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright (c) Microsoft Corporation. All rights reserved.\n",
    "\n",
    "Licensed under the MIT License.\n",
    "\n",
    "# Training an Object Detection model using AutoML\n",
    "In this notebook, we go over how you can use AutoML for training an Object Detection model. We will use a small dataset to train the model, demonstrate how you can tune hyperparameters of the model to optimize model performance and deploy the model to use in inference scenarios. For detailed information please refer to the [documentation of AutoML for Images](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-auto-train-image-models)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![img](example_object_detection_predictions.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Important:** This feature is currently in public preview. This preview version is provided without a service-level agreement. Certain features might not be supported or might have constrained capabilities. For more information, see [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/en-us/support/legal/preview-supplemental-terms/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Environment Setup\n",
    "Please follow the [\"Setup a new conda environment\"](https://github.com/Azure/azureml-examples/tree/main/python-sdk/tutorials/automl-with-azureml#3-setup-a-new-conda-environment) instructions to get started."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import azureml.core\n",
    "\n",
    "print(\"This notebook was created using version 1.35.0 of the Azure ML SDK.\")\n",
    "print(\"You are currently using version\", azureml.core.VERSION, \"of the Azure ML SDK.\")\n",
    "assert (\n",
    "    azureml.core.VERSION >= \"1.35\"\n",
    "), \"Please upgrade the Azure ML SDK by running '!pip install --upgrade azureml-sdk' then restart the kernel.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Workspace setup\n",
    "In order to train and deploy models in Azure ML, you will first need to set up a workspace.\n",
    "\n",
    "An [Azure ML Workspace](https://docs.microsoft.com/en-us/azure/machine-learning/concept-azure-machine-learning-architecture#workspace) is an Azure resource that organizes and coordinates the actions of many other Azure resources to assist in executing and sharing machine learning workflows. In particular, an Azure ML Workspace coordinates storage, databases, and compute resources providing added functionality for machine learning experimentation, deployment, inference, and the monitoring of deployed models.\n",
    "\n",
    "Create an Azure ML Workspace within your Azure subscription or load an existing workspace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.workspace import Workspace\n",
    "\n",
    "ws = Workspace.from_config()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compute target setup\n",
    "You will need to provide a [Compute Target](https://docs.microsoft.com/en-us/azure/machine-learning/concept-azure-machine-learning-architecture#computes) that will be used for your AutoML model training. AutoML models for image tasks require [GPU SKUs](https://docs.microsoft.com/en-us/azure/virtual-machines/sizes-gpu) such as the ones from the NC, NCv2, NCv3, ND, NDv2 and NCasT4 series. We recommend using the NCsv3-series (with v100 GPUs) for faster training. Using a compute target with a multi-GPU VM SKU will leverage the multiple GPUs to speed up training. Additionally, setting up a compute target with multiple nodes will allow for faster model training by leveraging parallelism, when tuning hyperparameters for your model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.compute import AmlCompute, ComputeTarget\n",
    "\n",
    "cluster_name = \"gpu-cluster-nc6\"\n",
    "\n",
    "try:\n",
    "    compute_target = ws.compute_targets[cluster_name]\n",
    "    print(\"Found existing compute target.\")\n",
    "except KeyError:\n",
    "    print(\"Creating a new compute target...\")\n",
    "    compute_config = AmlCompute.provisioning_configuration(\n",
    "        vm_size=\"Standard_NC6\",\n",
    "        idle_seconds_before_scaledown=600,\n",
    "        min_nodes=0,\n",
    "        max_nodes=4,\n",
    "    )\n",
    "    compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
    "# Can poll for a minimum number of nodes and for a specific timeout.\n",
    "# If no min_node_count is provided, it will use the scale settings for the cluster.\n",
    "compute_target.wait_for_completion(\n",
    "    show_output=True, min_node_count=None, timeout_in_minutes=20\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Experiment Setup\n",
    "Create an [Experiment](https://docs.microsoft.com/en-us/azure/machine-learning/concept-azure-machine-learning-architecture#experiments) in your workspace to track your model training runs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core import Experiment\n",
    "\n",
    "experiment_name = \"automl-image-object-detection\"\n",
    "experiment = Experiment(ws, name=experiment_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset with input Training Data\n",
    "\n",
    "In order to generate models for computer vision, you will need to bring in labeled image data as input for model training in the form of an [AzureML Tabular Dataset](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.data.tabulardataset). You can either use a dataset that you have exported from a [Data Labeling](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-label-data) project, or create a new Tabular Dataset with your labeled training data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we use a toy dataset called Fridge Objects, which consists of 128 images of 4 classes of beverage container {can, carton, milk bottle, water bottle} photos taken on different backgrounds.\n",
    "\n",
    "All images in this notebook are hosted in [this repository](https://github.com/microsoft/computervision-recipes) and are made available under the [MIT license](https://github.com/microsoft/computervision-recipes/blob/master/LICENSE).\n",
    "\n",
    "We first download and unzip the data locally."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import urllib\n",
    "from zipfile import ZipFile\n",
    "\n",
    "# download data\n",
    "download_url = \"https://cvbp-secondary.z19.web.core.windows.net/datasets/object_detection/odFridgeObjects.zip\"\n",
    "data_file = \"./odFridgeObjects.zip\"\n",
    "urllib.request.urlretrieve(download_url, filename=data_file)\n",
    "\n",
    "# extract files\n",
    "with ZipFile(data_file, \"r\") as zip:\n",
    "    print(\"extracting files...\")\n",
    "    zip.extractall()\n",
    "    print(\"done\")\n",
    "# delete zip file\n",
    "os.remove(data_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a sample image from this dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import Image\n",
    "\n",
    "Image(filename=\"./odFridgeObjects/images/31.jpg\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convert the downloaded data to JSONL\n",
    "In this example, the fridge object dataset is annotated in Pascal VOC format, where each image corresponds to an xml file. Each xml file contains information on where its corresponding image file is located and also contains information about the bounding boxes and the object labels. In order to use this data to create an [AzureML Tabular Dataset](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.data.tabulardataset), we first need to convert it to the required JSONL format. Please refer to the [documentation on how to prepare datasets](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-prepare-datasets-for-automl-images).\n",
    "\n",
    "The following script is creating two .jsonl files (one for training and one for validation) in the parent folder of the dataset. The train / validation ratio corresponds to 20% of the data going into the validation file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import os\n",
    "import xml.etree.ElementTree as ET\n",
    "\n",
    "src = \"./odFridgeObjects/\"\n",
    "train_validation_ratio = 5\n",
    "\n",
    "# Retrieving default datastore that got automatically created when we setup a workspace\n",
    "workspaceblobstore = ws.get_default_datastore().name\n",
    "\n",
    "# Path to the annotations\n",
    "annotations_folder = os.path.join(src, \"annotations\")\n",
    "\n",
    "# Path to the training and validation files\n",
    "train_annotations_file = os.path.join(src, \"train_annotations.jsonl\")\n",
    "validation_annotations_file = os.path.join(src, \"validation_annotations.jsonl\")\n",
    "\n",
    "# sample json line dictionary\n",
    "json_line_sample = {\n",
    "    \"image_url\": \"AmlDatastore://\"\n",
    "    + workspaceblobstore\n",
    "    + \"/\"\n",
    "    + os.path.basename(os.path.dirname(src))\n",
    "    + \"/\"\n",
    "    + \"images\",\n",
    "    \"image_details\": {\"format\": None, \"width\": None, \"height\": None},\n",
    "    \"label\": [],\n",
    "}\n",
    "\n",
    "# Read each annotation and convert it to jsonl line\n",
    "with open(train_annotations_file, \"w\") as train_f:\n",
    "    with open(validation_annotations_file, \"w\") as validation_f:\n",
    "        for i, filename in enumerate(os.listdir(annotations_folder)):\n",
    "            if filename.endswith(\".xml\"):\n",
    "                print(\"Parsing \" + os.path.join(src, filename))\n",
    "\n",
    "                root = ET.parse(os.path.join(annotations_folder, filename)).getroot()\n",
    "\n",
    "                width = int(root.find(\"size/width\").text)\n",
    "                height = int(root.find(\"size/height\").text)\n",
    "\n",
    "                labels = []\n",
    "                for object in root.findall(\"object\"):\n",
    "                    name = object.find(\"name\").text\n",
    "                    xmin = object.find(\"bndbox/xmin\").text\n",
    "                    ymin = object.find(\"bndbox/ymin\").text\n",
    "                    xmax = object.find(\"bndbox/xmax\").text\n",
    "                    ymax = object.find(\"bndbox/ymax\").text\n",
    "                    isCrowd = int(object.find(\"difficult\").text)\n",
    "                    labels.append(\n",
    "                        {\n",
    "                            \"label\": name,\n",
    "                            \"topX\": float(xmin) / width,\n",
    "                            \"topY\": float(ymin) / height,\n",
    "                            \"bottomX\": float(xmax) / width,\n",
    "                            \"bottomY\": float(ymax) / height,\n",
    "                            \"isCrowd\": isCrowd,\n",
    "                        }\n",
    "                    )\n",
    "                # build the jsonl file\n",
    "                image_filename = root.find(\"filename\").text\n",
    "                _, file_extension = os.path.splitext(image_filename)\n",
    "                json_line = dict(json_line_sample)\n",
    "                json_line[\"image_url\"] = json_line[\"image_url\"] + \"/\" + image_filename\n",
    "                json_line[\"image_details\"][\"format\"] = file_extension[1:]\n",
    "                json_line[\"image_details\"][\"width\"] = width\n",
    "                json_line[\"image_details\"][\"height\"] = height\n",
    "                json_line[\"label\"] = labels\n",
    "\n",
    "                if i % train_validation_ratio == 0:\n",
    "                    # validation annotation\n",
    "                    validation_f.write(json.dumps(json_line) + \"\\n\")\n",
    "                else:\n",
    "                    # train annotation\n",
    "                    train_f.write(json.dumps(json_line) + \"\\n\")\n",
    "            else:\n",
    "                print(\"Skipping unknown file: {}\".format(filename))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convert annotation file from COCO to JSONL\n",
    "If you want to try with a dataset in COCO format, the scripts below shows how to convert it to `jsonl` format. The file \"odFridgeObjects_coco.json\" consists of annotation information for the `odFridgeObjects` dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate jsonl file from coco file\n",
    "!python coco2jsonl.py \\\n",
    "--input_coco_file_path \"./odFridgeObjects_coco.json\" \\\n",
    "--output_dir \"./odFridgeObjects\" --output_file_name \"odFridgeObjects_from_coco.jsonl\" \\\n",
    "--task_type \"ObjectDetection\" \\\n",
    "--base_url \"AmlDatastore://workspaceblobstore/odFridgeObjects/images/\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualize bounding boxes\n",
    "Please refer to the \"Visualize data\" section in the following [tutorial](https://docs.microsoft.com/en-us/azure/machine-learning/tutorial-auto-train-image-models#visualize-data) to see how to easily visualize your ground truth bounding boxes before starting to train."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Upload the JSONL file and images to Datastore\n",
    "In order to use the data for training in Azure ML, we upload it to our Azure ML Workspace via a [Datastore](https://docs.microsoft.com/en-us/azure/machine-learning/concept-azure-machine-learning-architecture#datasets-and-datastores). The datastore provides a mechanism for you to upload/download data and interact with it from your remote compute targets. It is an abstraction over Azure Storage."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Retrieving default datastore that got automatically created when we setup a workspace\n",
    "ds = ws.get_default_datastore()\n",
    "ds.upload(src_dir=\"./odFridgeObjects\", target_path=\"odFridgeObjects\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we need to create an [AzureML Tabular Dataset](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.data.tabulardataset) from the data we uploaded to the Datastore. We create one dataset for training and one for validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core import Dataset\n",
    "from azureml.data import DataType\n",
    "\n",
    "# get existing training dataset\n",
    "training_dataset_name = \"odFridgeObjectsTrainingDataset\"\n",
    "if training_dataset_name in ws.datasets:\n",
    "    training_dataset = ws.datasets.get(training_dataset_name)\n",
    "    print(\"Found the training dataset\", training_dataset_name)\n",
    "else:\n",
    "    # create training dataset\n",
    "    training_dataset = Dataset.Tabular.from_json_lines_files(\n",
    "        path=ds.path(\"odFridgeObjects/train_annotations.jsonl\"),\n",
    "        set_column_types={\"image_url\": DataType.to_stream(ds.workspace)},\n",
    "    )\n",
    "    training_dataset = training_dataset.register(\n",
    "        workspace=ws, name=training_dataset_name\n",
    "    )\n",
    "# get existing validation dataset\n",
    "validation_dataset_name = \"odFridgeObjectsValidationDataset\"\n",
    "if validation_dataset_name in ws.datasets:\n",
    "    validation_dataset = ws.datasets.get(validation_dataset_name)\n",
    "    print(\"Found the validation dataset\", validation_dataset_name)\n",
    "else:\n",
    "    # create validation dataset\n",
    "    validation_dataset = Dataset.Tabular.from_json_lines_files(\n",
    "        path=ds.path(\"odFridgeObjects/validation_annotations.jsonl\"),\n",
    "        set_column_types={\"image_url\": DataType.to_stream(ds.workspace)},\n",
    "    )\n",
    "    validation_dataset = validation_dataset.register(\n",
    "        workspace=ws, name=validation_dataset_name\n",
    "    )\n",
    "print(\"Training dataset name: \" + training_dataset.name)\n",
    "print(\"Validation dataset name: \" + validation_dataset.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Validation dataset is optional. If no validation dataset is specified, by default 20% of your training data will be used for validation. You can control the percentage using the `split_ratio` argument - please refer to the [documentation](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-auto-train-image-models#model-agnostic-hyperparameters) for more details.\n",
    "\n",
    "This is what the training dataset looks like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_dataset.to_pandas_dataframe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configuring your AutoML run for image tasks\n",
    "AutoML allows you to easily train models for Image Classification, Object Detection & Instance Segmentation on your image data. You can control the model algorithm to be used, specify hyperparameter values for your model as well as perform a sweep across the hyperparameter space to generate an optimal model. Parameters for configuring your AutoML Image run are specified using the `AutoMLImageConfig` - please refer to the [documentation](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-auto-train-image-models#configure-your-experiment-settings) for the details on the parameters that can be used and their values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When using AutoML for image tasks, you need to specify the model algorithms using the `model_name` parameter. You can either specify a single model or choose to sweep over multiple models. Please refer to the [documentation](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-auto-train-image-models#configure-model-algorithms-and-hyperparameters) for the list of supported model algorithms."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using default hyperparameter values for the specified algorithm\n",
    "Before doing a large sweep to search for the optimal models and hyperparameters, we recommend trying the default values for a given model to get a first baseline. Next, you can explore multiple hyperparameters for the same model before sweeping over multiple models and their parameters. This allows an iterative approach, as with multiple models and multiple hyperparameters for each (as we showcase in the next section), the search space grows exponentially, and  you need more iterations to find optimal configurations.\n",
    "\n",
    "If you wish to use the default hyperparameter values for a given algorithm (say `yolov5`), you can specify the config for your AutoML Image runs as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.automl.core.shared.constants import ImageTask\n",
    "from azureml.train.automl import AutoMLImageConfig\n",
    "from azureml.train.hyperdrive import GridParameterSampling, choice\n",
    "\n",
    "image_config_yolov5 = AutoMLImageConfig(\n",
    "    task=ImageTask.IMAGE_OBJECT_DETECTION,\n",
    "    compute_target=compute_target,\n",
    "    training_data=training_dataset,\n",
    "    validation_data=validation_dataset,\n",
    "    hyperparameter_sampling=GridParameterSampling({\"model_name\": choice(\"yolov5\")}),\n",
    "    iterations=1,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Submitting an AutoML run for Computer Vision tasks\n",
    "Once you've created the config settings for your run, you can submit an AutoML run using the config in order to train a vision model using your training dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "automl_image_run = experiment.submit(image_config_yolov5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "automl_image_run.wait_for_completion(wait_post_processing=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hyperparameter sweeping for your AutoML models for computer vision tasks\n",
    "\n",
    "In this example, we use the AutoMLImageConfig to train an Object Detection model using `yolov5` and `fasterrcnn_resnet50_fpn`, both of which are pretrained on COCO, a large-scale object detection, segmentation, and captioning dataset that contains over 200K labeled images with over 80 label categories.\n",
    "\n",
    "When using AutoML for Images, you can perform a hyperparameter sweep over a defined parameter space to find the optimal model. In this example, we sweep over the hyperparameters for each algorithm, choosing from a range of values for `learning_rate`, `optimizer`, `lr_scheduler`, etc., to generate a model with the optimal primary metric. If hyperparameter values are not specified, then default values are used for the specified algorithm.\n",
    "\n",
    "We use Random Sampling to pick samples from this parameter space and try a total of 10 iterations with these different samples, running 2 iterations at a time on our compute target, which has been previously set up using 4 nodes. Please note that the more parameters the space has, the more iterations you need to find optimal models.\n",
    "\n",
    "We leverage the Bandit early termination policy which will terminate poor performing configs (those that are not within 20% slack of the best performing config), thus significantly saving compute resources.\n",
    "\n",
    "For more details on model and hyperparameter sweeping, please refer to the [documentation](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-tune-hyperparameters)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.automl.core.shared.constants import ImageTask\n",
    "from azureml.train.automl import AutoMLImageConfig\n",
    "from azureml.train.hyperdrive import BanditPolicy, RandomParameterSampling\n",
    "from azureml.train.hyperdrive import choice, uniform\n",
    "\n",
    "parameter_space = {\n",
    "    \"model\": choice(\n",
    "        {\n",
    "            \"model_name\": choice(\"yolov5\"),\n",
    "            \"learning_rate\": uniform(0.0001, 0.01),\n",
    "            \"model_size\": choice(\"small\", \"medium\"),  # model-specific\n",
    "            #'img_size': choice(640, 704, 768), # model-specific; might need GPU with large memory\n",
    "        },\n",
    "        {\n",
    "            \"model_name\": choice(\"fasterrcnn_resnet50_fpn\"),\n",
    "            \"learning_rate\": uniform(0.0001, 0.001),\n",
    "            \"optimizer\": choice(\"sgd\", \"adam\", \"adamw\"),\n",
    "            \"min_size\": choice(600, 800),  # model-specific\n",
    "            #'warmup_cosine_lr_warmup_epochs': choice(0, 3),\n",
    "        },\n",
    "    ),\n",
    "}\n",
    "\n",
    "tuning_settings = {\n",
    "    \"iterations\": 10,\n",
    "    \"max_concurrent_iterations\": 2,\n",
    "    \"hyperparameter_sampling\": RandomParameterSampling(parameter_space),\n",
    "    \"early_termination_policy\": BanditPolicy(\n",
    "        evaluation_interval=2, slack_factor=0.2, delay_evaluation=6\n",
    "    ),\n",
    "}\n",
    "\n",
    "automl_image_config = AutoMLImageConfig(\n",
    "    task=ImageTask.IMAGE_OBJECT_DETECTION,\n",
    "    compute_target=compute_target,\n",
    "    training_data=training_dataset,\n",
    "    validation_data=validation_dataset,\n",
    "    **tuning_settings,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "automl_image_run = experiment.submit(automl_image_config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "automl_image_run.wait_for_completion(wait_post_processing=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When doing a hyperparameter sweep, it can be useful to visualize the different configurations that were tried using the HyperDrive UI. You can navigate to this UI by going to the 'Child runs' tab in the UI of the main `automl_image_run` from above, which is the HyperDrive parent run. Then you can go into the 'Child runs' tab of this HyperDrive parent run. Alternatively, here below you can see directly the HyperDrive parent run and navigate to its 'Child runs' tab:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core import Run\n",
    "\n",
    "hyperdrive_run = Run(experiment=experiment, run_id=automl_image_run.id + \"_HD\")\n",
    "hyperdrive_run"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Register the optimal vision model from the AutoML run\n",
    "Once the run completes, we can register the model that was created from the best run (configuration that resulted in the best primary metric)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Register the model from the best run\n",
    "\n",
    "best_child_run = automl_image_run.get_best_child()\n",
    "model_name = best_child_run.properties[\"model_name\"]\n",
    "model = best_child_run.register_model(\n",
    "    model_name=model_name, model_path=\"outputs/model.pt\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy model as a web service\n",
    "Once you have your trained model, you can deploy the model on Azure. You can deploy your trained model as a web service on Azure Container Instances ([ACI](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-deploy-azure-container-instance)) or Azure Kubernetes Service ([AKS](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-deploy-azure-kubernetes-service)). Please note that ACI only supports small models under 1 GB in size. For testing larger models or for the high-scale production stage, we recommend using AKS.\n",
    "In this tutorial, we will deploy the model as a web service in AKS."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will need to first create an AKS compute cluster or use an existing AKS cluster. You can use either GPU or CPU VM SKUs for your deployment cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.compute import ComputeTarget, AksCompute\n",
    "from azureml.exceptions import ComputeTargetException\n",
    "\n",
    "# Choose a name for your cluster\n",
    "aks_name = \"aks-cpu-od\"\n",
    "# Check to see if the cluster already exists\n",
    "try:\n",
    "    aks_target = ComputeTarget(workspace=ws, name=aks_name)\n",
    "    print(\"Found existing compute target\")\n",
    "except ComputeTargetException:\n",
    "    print(\"Creating a new compute target...\")\n",
    "    # Provision AKS cluster with a CPU machine\n",
    "    prov_config = AksCompute.provisioning_configuration(vm_size=\"STANDARD_D3_V2\")\n",
    "    # Create the cluster\n",
    "    aks_target = ComputeTarget.create(\n",
    "        workspace=ws, name=aks_name, provisioning_configuration=prov_config\n",
    "    )\n",
    "    aks_target.wait_for_completion(show_output=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, you will need to define the [inference configuration](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-auto-train-image-models#update-inference-configuration), that describes how to set up the web-service containing your model. You can use the scoring script and the environment from the training run in your inference config.\n",
    "\n",
    "<b>Note:</b> To change the model's settings, open the downloaded scoring script and modify the model_settings variable <i>before</i> deploying the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.model import InferenceConfig\n",
    "\n",
    "best_child_run.download_file(\n",
    "    \"outputs/scoring_file_v_1_0_0.py\", output_file_path=\"score.py\"\n",
    ")\n",
    "environment = best_child_run.get_environment()\n",
    "inference_config = InferenceConfig(entry_script=\"score.py\", environment=environment)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can then deploy the model as an AKS web service."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Deploy the model from the best run as an AKS web service\n",
    "from azureml.core.webservice import AksWebservice\n",
    "from azureml.core.model import Model\n",
    "\n",
    "aks_config = AksWebservice.deploy_configuration(\n",
    "    autoscale_enabled=True, cpu_cores=1, memory_gb=5, enable_app_insights=True\n",
    ")\n",
    "\n",
    "aks_service = Model.deploy(\n",
    "    ws,\n",
    "    models=[model],\n",
    "    inference_config=inference_config,\n",
    "    deployment_config=aks_config,\n",
    "    deployment_target=aks_target,\n",
    "    name=\"automl-image-test-cpu-od\",\n",
    "    overwrite=True,\n",
    ")\n",
    "aks_service.wait_for_deployment(show_output=True)\n",
    "print(aks_service.state)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test the web service\n",
    "Finally, let's test our deployed web service to predict new images. You can pass in any image. In this case, we'll use a random image from the dataset and pass it to the scoring URI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "# URL for the web service\n",
    "scoring_uri = aks_service.scoring_uri\n",
    "\n",
    "# If the service is authenticated, set the key or token\n",
    "key, _ = aks_service.get_keys()\n",
    "\n",
    "sample_image = \"./test_image.jpg\"\n",
    "\n",
    "# Load image data\n",
    "data = open(sample_image, \"rb\").read()\n",
    "\n",
    "# Set the content type\n",
    "headers = {\"Content-Type\": \"application/octet-stream\"}\n",
    "\n",
    "# If authentication is enabled, set the authorization header\n",
    "headers[\"Authorization\"] = f\"Bearer {key}\"\n",
    "\n",
    "# Make the request and display the response\n",
    "resp = requests.post(scoring_uri, data, headers=headers)\n",
    "print(resp.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize detections\n",
    "Now that we have scored a test image, we can visualize the bounding boxes for this image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.image as mpimg\n",
    "import matplotlib.patches as patches\n",
    "from PIL import Image\n",
    "import numpy as np\n",
    "import json\n",
    "\n",
    "IMAGE_SIZE = (18, 12)\n",
    "plt.figure(figsize=IMAGE_SIZE)\n",
    "img_np = mpimg.imread(sample_image)\n",
    "img = Image.fromarray(img_np.astype(\"uint8\"), \"RGB\")\n",
    "x, y = img.size\n",
    "\n",
    "fig, ax = plt.subplots(1, figsize=(15, 15))\n",
    "# Display the image\n",
    "ax.imshow(img_np)\n",
    "\n",
    "# draw box and label for each detection\n",
    "detections = json.loads(resp.text)\n",
    "for detect in detections[\"boxes\"]:\n",
    "    label = detect[\"label\"]\n",
    "    box = detect[\"box\"]\n",
    "    conf_score = detect[\"score\"]\n",
    "    if conf_score > 0.6:\n",
    "        ymin, xmin, ymax, xmax = (\n",
    "            box[\"topY\"],\n",
    "            box[\"topX\"],\n",
    "            box[\"bottomY\"],\n",
    "            box[\"bottomX\"],\n",
    "        )\n",
    "        topleft_x, topleft_y = x * xmin, y * ymin\n",
    "        width, height = x * (xmax - xmin), y * (ymax - ymin)\n",
    "        print(\n",
    "            \"{}: [{}, {}, {}, {}], {}\".format(\n",
    "                detect[\"label\"],\n",
    "                round(topleft_x, 3),\n",
    "                round(topleft_y, 3),\n",
    "                round(width, 3),\n",
    "                round(height, 3),\n",
    "                round(conf_score, 3),\n",
    "            )\n",
    "        )\n",
    "\n",
    "        color = np.random.rand(3)  #'red'\n",
    "        rect = patches.Rectangle(\n",
    "            (topleft_x, topleft_y),\n",
    "            width,\n",
    "            height,\n",
    "            linewidth=3,\n",
    "            edgecolor=color,\n",
    "            facecolor=\"none\",\n",
    "        )\n",
    "\n",
    "        ax.add_patch(rect)\n",
    "        plt.text(topleft_x, topleft_y - 10, label, color=color, fontsize=20)\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.6 - AzureML",
   "language": "python",
   "name": "python3-azureml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  },
  "nteract": {
   "version": "nteract-front-end@1.0.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}