 archived/Amazon_Tensorflow_Object_Detection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c58dc2d3",
   "metadata": {},
   "source": [
    "# Introduction to SageMaker TensorFlow - Object Detection"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "77551e56",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/introduction_to_amazon_algorithms|object_detection_tensorflow|Amazon_Tensorflow_Object_Detection.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b44a8bb",
   "metadata": {},
   "source": [
    "---\n",
    "Welcome to [Amazon SageMaker Built-in Algorithms](https://sagemaker.readthedocs.io/en/stable/algorithms/index.html)! You can use SageMaker Built-in algorithms to solve many Machine Learning tasks through [SageMaker Python SDK](https://sagemaker.readthedocs.io/en/stable/overview.html). You can also use these algorithms through one-click in SageMaker Studio via [JumpStart](https://docs.aws.amazon.com/sagemaker/latest/dg/studio-jumpstart.html).\n",
    "\n",
    "In this demo notebook, we demonstrate how to use the TensorFlow Object Detection algorithm. Object Detection refers to predicting the objects in an image via bounding boxes.  We demonstrate two use cases of TensorFlow Object Detection models:\n",
    "\n",
    "* How to use a model pre-trained on COCO '17 dataset to predict objects in an image. [COCO Labels](https://github.com/amikelive/coco-labels/blob/master/coco-labels-2014_2017.txt).\n",
    "* How to fine-tune a pre-trained model to a custom dataset, and then run inference on the fine-tuned model.\n",
    "\n",
    "Note: This notebook was tested on ml.t3.medium instance in Amazon SageMaker Studio with Python 3 (Data Science) kernel and in Amazon SageMaker Notebook instance with conda_python3 kernel.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37b18385",
   "metadata": {},
   "source": [
    "1. [Set Up](#1.-Set-Up)\n",
    "2. [Select a pre-trained model](#2.-Select-a-pre-trained-model)\n",
    "3. [Run inference on the pre-trained model](#3.-Run-inference-on-the-pre-trained-model)\n",
    "    * [Retrieve Artifacts & Deploy an Endpoint](#3.1.-Retrieve-Artifacts-&-Deploy-an-Endpoint)\n",
    "    * [Download example images for inference](#3.2.-Download-example-images-for-inference)\n",
    "    * [Query endpoint and parse response](#3.3.-Query-endpoint-and-parse-response)\n",
    "    * [Display Model Predictions](#3.4.-Display-Model-Predictions)\n",
    "    * [Clean up the endpoint](#3.5.-Clean-up-the-endpoint)\n",
    "4. [Fine-tune the pre-trained model on a custom dataset](#4.-Fine-tune-the-pre-trained-model-on-a-custome-dataset)\n",
    "    * [Retrieve Training artifacts](#4.1.-Retrieve-Training-artifacts)\n",
    "    * [Set Training parameters](#4.2.-Set-Training-parameters)\n",
    "    * [Train with Automatic Model Tuning (HPO)](#AMT)\n",
    "    * [Start Training](#4.4.-Start-Training)\n",
    "    * [Extract Training performance metrics](#4.5.-Extract-Training-performance-metrics)\n",
    "    * [Deploy & run Inference on the fine-tuned model](#4.6.-Deploy-&-run-Inference-on-the-fine-tuned-model)\n",
    "    * [Incrementally train the fine-tuned model](#4.7.-Incrementally-train-the-fine-tuned-model)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3709b9d5",
   "metadata": {},
   "source": [
    "## 1. Set Up\n",
    "***\n",
    "Before executing the notebook, there are some initial steps required for setup. This notebook requires latest version of sagemaker and ipywidgets.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "850f69db",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install sagemaker ipywidgets --upgrade --quiet"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6158404f",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "To train and host on Amazon Sagemaker, we need to setup and authenticate the use of AWS services. Here, we use the execution role associated with the current notebook instance as the AWS account role with SageMaker access. It has necessary permissions, including access to your data in S3. \n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68a5afe8",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker, boto3, json\n",
    "from sagemaker.session import Session\n",
    "\n",
    "sagemaker_session = Session()\n",
    "aws_role = sagemaker_session.get_caller_identity_arn()\n",
    "aws_region = boto3.Session().region_name\n",
    "sess = sagemaker.Session()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c69d6e8",
   "metadata": {},
   "source": [
    "## 2. Select a pre-trained model\n",
    "***\n",
    "You can continue with the default model, or can choose a different model from the dropdown generated upon running the next cell. A complete list of SageMaker pre-trained models can also be accessed at [Sagemaker pre-trained Models](https://sagemaker.readthedocs.io/en/stable/doc_utils/pretrainedmodels.html#).\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60727742",
   "metadata": {
    "jumpStartAlterations": [
     "modelIdVersion"
    ]
   },
   "outputs": [],
   "source": [
    "model_id, model_version = \"tensorflow-od1-ssd-resnet50-v1-fpn-640x640-coco17-tpu-8\", \"*\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bffad126",
   "metadata": {},
   "source": [
    "***\n",
    "[Optional] Select a different Sagemaker pre-trained model. Here, we download the model_manifest file from the Built-In Algorithms s3 bucket, filter-out all the Object Detection models and select a model for inference. Note that the models identified with the prefix tensorflow-od1- can be used to run inference or fine-tune on a custom dataset. Models identified with the prefix tensorflow-od- currently only support inference and can not be fine-tuned on a custom dataset.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "057ac669",
   "metadata": {},
   "outputs": [],
   "source": [
    "import IPython\n",
    "import ipywidgets as widgets\n",
    "from sagemaker.jumpstart.notebook_utils import list_jumpstart_models\n",
    "from sagemaker.jumpstart.filters import And, Or\n",
    "\n",
    "# Retrieves all TensorFlow Object Detection models.\n",
    "filter_value = Or(\n",
    "    And(\"task == od1\", \"framework == tensorflow\"), And(\"task == od\", \"framework == tensorflow\")\n",
    ")\n",
    "tensorflow_od_models = list_jumpstart_models(filter=filter_value)\n",
    "# display the model-ids in a dropdown, for user to select a model.\n",
    "dropdown = widgets.Dropdown(\n",
    "    options=tensorflow_od_models,\n",
    "    value=model_id,\n",
    "    description=\"SageMaker Built-In TensorFlow Object Detection Models:\",\n",
    "    style={\"description_width\": \"initial\"},\n",
    "    layout={\"width\": \"max-content\"},\n",
    ")\n",
    "display(IPython.display.Markdown(\"## Select a SageMaker pre-trained model from the dropdown below\"))\n",
    "display(dropdown)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a53e0683",
   "metadata": {},
   "source": [
    "## 3. Run inference on the pre-trained model\n",
    "***\n",
    "Using SageMaker, we can perform inference on the pre-trained model, even without fine-tuning it first on a custom dataset. For this example, that means on an input image, predicting the [objects from one of the 80 classes of the COCO '17 dataset](https://github.com/amikelive/coco-labels/blob/master/coco-labels-2014_2017.txt).\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35066ab7",
   "metadata": {},
   "source": [
    "### 3.1. Retrieve Artifacts & Deploy an Endpoint\n",
    "***\n",
    "We retrieve the deploy_image_uri, deploy_source_uri, and base_model_uri for the pre-trained model. To host the pre-trained base-model, we create an instance of [`sagemaker.model.Model`](https://sagemaker.readthedocs.io/en/stable/api/inference/model.html) and deploy it. This can take up to 15 minutes.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21e5a659",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker import image_uris, model_uris, script_uris\n",
    "from sagemaker.model import Model\n",
    "from sagemaker.predictor import Predictor\n",
    "from sagemaker.utils import name_from_base\n",
    "\n",
    "# model_version=\"*\" fetches the latest version of the model.\n",
    "infer_model_id, infer_model_version = dropdown.value, \"*\"\n",
    "\n",
    "endpoint_name = name_from_base(f\"jumpstart-example-{infer_model_id}\")\n",
    "\n",
    "inference_instance_type = \"ml.p2.xlarge\"\n",
    "\n",
    "# Retrieve the inference docker container uri.\n",
    "deploy_image_uri = image_uris.retrieve(\n",
    "    region=None,\n",
    "    framework=None,\n",
    "    image_scope=\"inference\",\n",
    "    model_id=infer_model_id,\n",
    "    model_version=infer_model_version,\n",
    "    instance_type=inference_instance_type,\n",
    ")\n",
    "# Retrieve the inference script uri.\n",
    "deploy_source_uri = script_uris.retrieve(\n",
    "    model_id=infer_model_id, model_version=infer_model_version, script_scope=\"inference\"\n",
    ")\n",
    "# Retrieve the base model uri.\n",
    "base_model_uri = model_uris.retrieve(\n",
    "    model_id=infer_model_id, model_version=infer_model_version, model_scope=\"inference\"\n",
    ")\n",
    "# Create the SageMaker model instance. Note that we need to pass Predictor class when we deploy model through Model class,\n",
    "# for being able to run inference through the sagemaker API.\n",
    "model = Model(\n",
    "    image_uri=deploy_image_uri,\n",
    "    source_dir=deploy_source_uri,\n",
    "    model_data=base_model_uri,\n",
    "    entry_point=\"inference.py\",\n",
    "    role=aws_role,\n",
    "    predictor_cls=Predictor,\n",
    "    name=endpoint_name,\n",
    ")\n",
    "# deploy the Model.\n",
    "base_model_predictor = model.deploy(\n",
    "    initial_instance_count=1,\n",
    "    instance_type=inference_instance_type,\n",
    "    endpoint_name=endpoint_name,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ef73c70",
   "metadata": {},
   "source": [
    "### 3.2. Download example images for inference\n",
    "***\n",
    "We download example images from a public S3 bucket.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7bb3299",
   "metadata": {},
   "outputs": [],
   "source": [
    "s3_bucket = f\"jumpstart-cache-prod-{aws_region}\"\n",
    "key_prefix = \"inference-notebook-assets\"\n",
    "Naxos_Taverna = \"Naxos_Taverna.jpg\"\n",
    "\n",
    "\n",
    "boto3.client(\"s3\").download_file(s3_bucket, f\"{key_prefix}/{Naxos_Taverna}\", Naxos_Taverna)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8823b8f7",
   "metadata": {},
   "source": [
    "### 3.3. Query endpoint and parse response\n",
    "***\n",
    "Input to the endpoint is a single image in binary format. Response of the endpoint is a set of bounding boxes as well as class names and scores for the bounding boxes. JumpStart allows the flexibility in the number of bounding boxes returned. Below, we show to predict two bounding boxes per image by appending `;n_predictions=5` to `Accept`. To predict xx boxes, one can change it to `;n_predictions=xx` or to get all the predicted boxes, one can remove `;n_predictions=5`.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "01697684",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "\n",
    "def query(model_predictor, image_file_name):\n",
    "    with open(image_file_name, \"rb\") as file:\n",
    "        input_img_rb = file.read()\n",
    "\n",
    "    query_response = model_predictor.predict(\n",
    "        input_img_rb,\n",
    "        {\n",
    "            \"ContentType\": \"application/x-image\",\n",
    "            \"Accept\": \"application/json;verbose;n_predictions=5\",\n",
    "        },\n",
    "    )\n",
    "    return query_response\n",
    "\n",
    "\n",
    "def parse_response(query_response):\n",
    "    model_predictions = json.loads(query_response)\n",
    "    normalized_boxes, classes, scores, labels = (\n",
    "        model_predictions[\"normalized_boxes\"],\n",
    "        model_predictions[\"classes\"],\n",
    "        model_predictions[\"scores\"],\n",
    "        model_predictions[\"labels\"],\n",
    "    )\n",
    "    # Substitute the classes index with the classes name\n",
    "    class_names = [labels[int(idx)] for idx in classes]\n",
    "    return normalized_boxes, class_names, scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fea97d9f",
   "metadata": {},
   "outputs": [],
   "source": [
    "query_response = query(base_model_predictor, Naxos_Taverna)\n",
    "\n",
    "normalized_boxes, classes_names, confidences = parse_response(query_response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed88ab16",
   "metadata": {},
   "source": [
    "### 3.4. Display Model Predictions\n",
    "***\n",
    "Next, we display the bounding boxes overlaid on the original image.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f2eda87",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.patches as patches\n",
    "from matplotlib import pyplot as plt\n",
    "from PIL import Image\n",
    "from PIL import ImageColor\n",
    "import numpy as np\n",
    "\n",
    "\n",
    "def display_predictions(img_jpg, normalized_boxes, classes_names, confidences):\n",
    "    colors = list(ImageColor.colormap.values())\n",
    "    image_np = np.array(Image.open(img_jpg))\n",
    "    plt.figure(figsize=(20, 20))\n",
    "    ax = plt.axes()\n",
    "    ax.imshow(image_np)\n",
    "\n",
    "    for idx in range(len(normalized_boxes)):\n",
    "        left, bot, right, top = normalized_boxes[idx]\n",
    "        x, w = [val * image_np.shape[1] for val in [left, right - left]]\n",
    "        y, h = [val * image_np.shape[0] for val in [bot, top - bot]]\n",
    "        color = colors[hash(classes_names[idx]) % len(colors)]\n",
    "        rect = patches.Rectangle((x, y), w, h, linewidth=3, edgecolor=color, facecolor=\"none\")\n",
    "        ax.add_patch(rect)\n",
    "        ax.text(\n",
    "            x,\n",
    "            y,\n",
    "            \"{} {:.0f}%\".format(classes_names[idx], confidences[idx] * 100),\n",
    "            bbox=dict(facecolor=\"white\", alpha=0.5),\n",
    "        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f675ee75",
   "metadata": {},
   "outputs": [],
   "source": [
    "display_predictions(Naxos_Taverna, normalized_boxes, classes_names, confidences)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "990c9cab",
   "metadata": {},
   "source": [
    "### 3.5. Clean up the endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "269d4234",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Delete the SageMaker endpoint and the attached resources\n",
    "base_model_predictor.delete_model()\n",
    "base_model_predictor.delete_endpoint()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7f19997",
   "metadata": {},
   "source": [
    "## 4. Fine-tune the pre-trained model on a custom dataset\n",
    "***\n",
    "\n",
    "Previously, we saw how to run inference on a pre-trained model. Next, we discuss how a model can be finetuned to a custom dataset with any number of classes.\n",
    "\n",
    "Transfer learning algorithm removes the object detection head of the pre-trained model and attaches a new randomly initialized head with number of classes same as the custom dataset. The fine-tuning step fine-tunes the last layer parameters while keeping the parameters of the rest of the model frozen, and returns the fine-tuned model. The objective during finetuning is to minimize box prediction error on the input data.\n",
    "\n",
    "- **Input** – A directory with sub-directory images and a file annotations.json.\n",
    "\n",
    "- **Output** – A fine-tuned model that can be deployed for inference or can be further trained using incremental training. A file mapping class indexes to class labels is saved along with the models.\n",
    "\n",
    "The input directory should look like below if the training data contains two images. The names of .png files can be anything.\n",
    "\n",
    "The s3 path should look like `s3://bucket_name/input_directory/`. Note the trailing `/` is required.\n",
    "\n",
    "    input_directory\n",
    "        |--images\n",
    "            |--abc.png\n",
    "            |--def.png\n",
    "        |--annotations.json\n",
    "\n",
    "The annotations.json file should have information for bounding_boxes and their class labels. It should have a dictionary with keys \"images\" and \"annotations\". Value for the \"images\" key should be a list of entries, one for each image of the form {\"file_name\": image_name, \"height\": height, \"width\": width, \"id\": image_id}. Value of the 'annotations' key should be a list of entries, one for each bounding box of the form {\"image_id\": image_id, \"bbox\": [xmin, ymin, xmax, ymax], \"category_id\": bbox_label}.\n",
    "\n",
    "We provide pennfudanped dataset as a default dataset for fine-tuning the model. PennFudanPed comprises images of pedestrians. The dataset has been downloaded from here.\n",
    "\n",
    "Citation: @ONLINE {pennfudanped, author = \"Liming Wang1, Jianbo Shi2, Gang Song2, and I-fan Shen1\", title = \"Penn-Fudan Database for Pedestrian Detection and Segmentation\", year = \"2007\", url = \"https://www.cis.upenn.edu/~jshi/ped_html/\" }\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "700b5eaf",
   "metadata": {},
   "source": [
    "### 4.1. Retrieve Training artifacts\n",
    "***\n",
    "Here, for the selected model, we retrieve the training docker container, the training algorithm source, the pre-trained base model, and a python dictionary of the training hyper-parameters that the algorithm accepts with their default values. Note that the model_version=\"*\" fetches the latest model. Also, we do need to specify the training_instance_type to fetch train_image_uri.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ee4c412",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker import image_uris, model_uris, script_uris, hyperparameters\n",
    "\n",
    "model_id, model_version = dropdown.value, \"*\"\n",
    "training_instance_type = \"ml.p3.2xlarge\"\n",
    "\n",
    "# Retrieve the docker image\n",
    "train_image_uri = image_uris.retrieve(\n",
    "    region=None,\n",
    "    framework=None,\n",
    "    model_id=model_id,\n",
    "    model_version=model_version,\n",
    "    image_scope=\"training\",\n",
    "    instance_type=training_instance_type,\n",
    ")\n",
    "\n",
    "# Retrieve the training script\n",
    "train_source_uri = script_uris.retrieve(\n",
    "    model_id=model_id, model_version=model_version, script_scope=\"training\"\n",
    ")\n",
    "\n",
    "# Retrieve the pre-trained model tarball to further fine-tune\n",
    "train_model_uri = model_uris.retrieve(\n",
    "    model_id=model_id, model_version=model_version, model_scope=\"training\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc150b68",
   "metadata": {},
   "source": [
    "### 4.2. Set Training parameters\n",
    "***\n",
    "Now that we are done with all the setup that is needed, we are ready to fine-tune our Object Detection model. To begin, let us create a [``sageMaker.estimator.Estimator``](https://sagemaker.readthedocs.io/en/stable/api/training/estimators.html) object. This estimator will launch the training job. \n",
    "\n",
    "There are two kinds of parameters that need to be set for training. \n",
    "\n",
    "The first one are the parameters for the training job. These include: (i) Training data path. This is S3 folder in which the input data is stored, (ii) Output path: This the s3 folder in which the training output is stored. (iii) Training instance type: This indicates the type of machine on which to run the training. Typically, we use GPU instances for these training. We defined the training instance type above to fetch the correct train_image_uri. \n",
    "\n",
    "The second set of parameters are algorithm specific training hyper-parameters.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7bed3ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sample training data is available in this bucket\n",
    "training_data_bucket = f\"jumpstart-cache-prod-{aws_region}\"\n",
    "training_data_prefix = \"training-datasets/PennFudanPed_COCO_format/\"\n",
    "\n",
    "training_dataset_s3_path = f\"s3://{training_data_bucket}/{training_data_prefix}\"\n",
    "\n",
    "output_bucket = sess.default_bucket()\n",
    "output_prefix = \"jumpstart-example-od-training\"\n",
    "\n",
    "s3_output_location = f\"s3://{output_bucket}/{output_prefix}/output\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52047c35",
   "metadata": {},
   "source": [
    "***\n",
    "For algorithm specific hyper-parameters, we start by fetching python dictionary of the training hyper-parameters that the algorithm accepts with their default values. This can then be overridden to custom values.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d6198903",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker import hyperparameters\n",
    "\n",
    "# Retrieve the default hyper-parameters for fine-tuning the model\n",
    "hyperparameters = hyperparameters.retrieve_default(model_id=model_id, model_version=model_version)\n",
    "\n",
    "# [Optional] Override default hyperparameters with custom values\n",
    "hyperparameters[\"epochs\"] = \"5\"\n",
    "print(hyperparameters)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42462a38",
   "metadata": {},
   "source": [
    "### 4.3. Train with Automatic Model Tuning ([HPO](https://docs.aws.amazon.com/sagemaker/latest/dg/automatic-model-tuning.html)) <a id='AMT'></a>\n",
    "***\n",
    "Amazon SageMaker automatic model tuning, also known as hyperparameter tuning, finds the best version of a model by running many training jobs on your dataset using the algorithm and ranges of hyperparameters that you specify. It then chooses the hyperparameter values that result in a model that performs the best, as measured by a metric that you choose. We will use a [HyperparameterTuner](https://sagemaker.readthedocs.io/en/stable/api/training/tuner.html) object to interact with Amazon SageMaker hyperparameter tuning APIs.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bacb7149",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.tuner import ContinuousParameter\n",
    "\n",
    "# Use AMT for tuning and selecting the best model\n",
    "use_amt = False\n",
    "\n",
    "# Define objective metric per framework, based on which the best model will be selected.\n",
    "amt_metric_definitions = {\n",
    "    \"metrics\": [{\"Name\": \"validation:localization_loss\", \"Regex\": \"Val_localization=([0-9\\\\.]+)\"}],\n",
    "    \"type\": \"Minimize\",\n",
    "}\n",
    "\n",
    "# You can select from the hyperparameters supported by the model, and configure ranges of values to be searched for training the optimal model.(https://docs.aws.amazon.com/sagemaker/latest/dg/automatic-model-tuning-define-ranges.html)\n",
    "hyperparameter_ranges = {\n",
    "    \"learning_rate\": ContinuousParameter(0.000001, 0.5, scaling_type=\"Logarithmic\")\n",
    "}\n",
    "\n",
    "# Increase the total number of training jobs run by AMT, for increased accuracy (and training time).\n",
    "max_jobs = 6\n",
    "# Change parallel training jobs run by AMT to reduce total training time, constrained by your account limits.\n",
    "# if max_jobs=max_parallel_jobs then Bayesian search turns to Random.\n",
    "max_parallel_jobs = 2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2ce0dee",
   "metadata": {},
   "source": [
    "### 4.4. Start Training\n",
    "***\n",
    "We start by creating the estimator object with all the required assets and then launch the training job. This can take up to 30 minutes.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60d10a44",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.estimator import Estimator\n",
    "from sagemaker.utils import name_from_base\n",
    "from sagemaker.tuner import HyperparameterTuner\n",
    "\n",
    "training_job_name = name_from_base(f\"jumpstart-example-{model_id}-transfer-learning\")\n",
    "\n",
    "training_metric_definitions = [\n",
    "    {\"Name\": \"val_localization_loss\", \"Regex\": \"Val_localization=([0-9\\\\.]+)\"},\n",
    "    {\"Name\": \"val_classification_loss\", \"Regex\": \"Val_classification=([0-9\\\\.]+)\"},\n",
    "    {\"Name\": \"train_loss\", \"Regex\": \"loss=([0-9\\\\.]+).\"},\n",
    "]\n",
    "\n",
    "# Create SageMaker Estimator instance\n",
    "od_estimator = Estimator(\n",
    "    role=aws_role,\n",
    "    image_uri=train_image_uri,\n",
    "    source_dir=train_source_uri,\n",
    "    model_uri=train_model_uri,\n",
    "    entry_point=\"transfer_learning.py\",\n",
    "    instance_count=1,\n",
    "    instance_type=training_instance_type,\n",
    "    max_run=360000,\n",
    "    hyperparameters=hyperparameters,\n",
    "    output_path=s3_output_location,\n",
    "    base_job_name=training_job_name,\n",
    "    metric_definitions=training_metric_definitions,\n",
    ")\n",
    "\n",
    "if use_amt:\n",
    "    hp_tuner = HyperparameterTuner(\n",
    "        od_estimator,\n",
    "        amt_metric_definitions[\"metrics\"][0][\"Name\"],\n",
    "        hyperparameter_ranges,\n",
    "        amt_metric_definitions[\"metrics\"],\n",
    "        max_jobs=max_jobs,\n",
    "        max_parallel_jobs=max_parallel_jobs,\n",
    "        objective_type=amt_metric_definitions[\"type\"],\n",
    "        base_tuning_job_name=training_job_name,\n",
    "    )\n",
    "\n",
    "    # Launch a SageMaker Tuning job to search for the best hyperparameters\n",
    "    hp_tuner.fit({\"training\": training_dataset_s3_path})\n",
    "else:\n",
    "    # Launch a SageMaker Training job by passing s3 path of the training data\n",
    "    od_estimator.fit({\"training\": training_dataset_s3_path}, logs=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06a02b09",
   "metadata": {},
   "source": [
    "### 4.5. Extract Training performance metrics\n",
    "***\n",
    "Performance metrics such as training loss and validation localization/classification loss can be accessed through cloudwatch while the training. Code below provides the link to the cloudwatch log where these metrics can be found. \n",
    "\n",
    "Note that default resolution in Amazon Cloudwatch is one minute i.e. it averages the metrics logged within a single minute interval. Amazon CloudWatch also supports [high-resolution custom metrics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/publishingMetrics.html), and its finest resolution is 1 second. However, the finer the resolution, the shorter the lifespan of the CloudWatch metrics. For the 1-second frequency resolution, the CloudWatch metrics are available for 3 hours. For more information about the resolution and the lifespan of the CloudWatch metrics, see [GetMetricStatistics](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_GetMetricStatistics.html) in the Amazon CloudWatch API Reference.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cfc0d12c",
   "metadata": {},
   "outputs": [],
   "source": [
    "if use_amt:\n",
    "    training_job_name = hp_tuner.best_training_job()\n",
    "else:\n",
    "    training_job_name = od_estimator.latest_training_job.job_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "08db96ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "from IPython.core.display import Markdown\n",
    "\n",
    "sagemaker_session = sagemaker.Session()\n",
    "\n",
    "link = (\n",
    "    \"https://console.aws.amazon.com/cloudwatch/home?region=\"\n",
    "    + sagemaker_session.boto_region_name\n",
    "    + \"#metricsV2:query=%7B/aws/sagemaker/TrainingJobs,TrainingJobName%7D%20\"\n",
    "    + training_job_name\n",
    ")\n",
    "display(Markdown(\"CloudWatch metrics: [link](\" + link + \")\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b6ff1e9",
   "metadata": {},
   "source": [
    "***\n",
    "Alternatively, we can also fetch these metrics and analyze them within the notebook.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a2a19979",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker import TrainingJobAnalytics\n",
    "\n",
    "df = TrainingJobAnalytics(training_job_name=training_job_name).dataframe()\n",
    "\n",
    "df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80a87732",
   "metadata": {},
   "source": [
    "***\n",
    "We can filter out different metrics by names as well.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf833bff",
   "metadata": {},
   "outputs": [],
   "source": [
    "metric_names = [metric[\"Name\"] for metric in training_metric_definitions]\n",
    "\n",
    "metrics_df = {\n",
    "    metric_name: df.query(f\"metric_name == '{metric_name}'\") for metric_name in metric_names\n",
    "}\n",
    "\n",
    "metrics_df[\"val_localization_loss\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "511d3787",
   "metadata": {},
   "source": [
    "## 4.6. Deploy & run Inference on the fine-tuned model\n",
    "***\n",
    "A trained model does nothing on its own. We now want to use the model to perform inference. For this example, that means predicting the class label of an image. We follow the same steps as in the [Section 3 - Run inference on the pre-trained model](#3.-Run-inference-on-the-pre-trained-model). We start by retrieving the artifacts for deploying an endpoint. However, instead of base_predictor, we  deploy the `od_estimator` that we fine-tuned.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "016e6ea8",
   "metadata": {},
   "outputs": [],
   "source": [
    "inference_instance_type = \"ml.p2.xlarge\"\n",
    "\n",
    "# Retrieve the inference docker container uri\n",
    "deploy_image_uri = image_uris.retrieve(\n",
    "    region=None,\n",
    "    framework=None,\n",
    "    image_scope=\"inference\",\n",
    "    model_id=model_id,\n",
    "    model_version=model_version,\n",
    "    instance_type=inference_instance_type,\n",
    ")\n",
    "# Retrieve the inference script uri\n",
    "deploy_source_uri = script_uris.retrieve(\n",
    "    model_id=model_id, model_version=model_version, script_scope=\"inference\"\n",
    ")\n",
    "\n",
    "endpoint_name = name_from_base(f\"jumpstart-example-FT-{model_id}-\")\n",
    "\n",
    "# Use the estimator from the previous step to deploy to a SageMaker endpoint\n",
    "finetuned_predictor = (hp_tuner if use_amt else od_estimator).deploy(\n",
    "    initial_instance_count=1,\n",
    "    instance_type=inference_instance_type,\n",
    "    entry_point=\"inference.py\",\n",
    "    image_uri=deploy_image_uri,\n",
    "    source_dir=deploy_source_uri,\n",
    "    endpoint_name=endpoint_name,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea3a4669",
   "metadata": {},
   "source": [
    "---\n",
    "Next, we download example images of pedestrian from the S3 bucket for inference.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c8619b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "s3_bucket = f\"jumpstart-cache-prod-{aws_region}\"\n",
    "key_prefix = \"training-datasets/PennFudanPed_COCO_format/images\"\n",
    "\n",
    "pedestrian_image_file_name = \"FudanPed00001.png\"\n",
    "\n",
    "boto3.client(\"s3\").download_file(\n",
    "    s3_bucket,\n",
    "    f\"{key_prefix}/{pedestrian_image_file_name}\",\n",
    "    pedestrian_image_file_name,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87a50b5b",
   "metadata": {},
   "source": [
    "---\n",
    "Next, we query the finetuned model, parse the response and display the predictions. Functions for these are implemented in sections [3.3. Query endpoint and parse response](#3.3.-Query-endpoint-and-parse-response) and [3.4. Display Model Predictions](#3.4.-Display-Model-Predictions).\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b6c5b01",
   "metadata": {},
   "outputs": [],
   "source": [
    "query_response = query(finetuned_predictor, pedestrian_image_file_name)\n",
    "\n",
    "normalized_boxes, classes_names, confidences = parse_response(query_response)\n",
    "display_predictions(pedestrian_image_file_name, normalized_boxes, classes_names, confidences)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64282829",
   "metadata": {},
   "source": [
    "---\n",
    "Next, we clean up the deployed endpoint.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd5c861d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Delete the SageMaker endpoint and the attached resources\n",
    "finetuned_predictor.delete_model()\n",
    "finetuned_predictor.delete_endpoint()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54c8c4bb",
   "metadata": {},
   "source": [
    "## 4.7. Incrementally train the fine-tuned model\n",
    "\n",
    "***\n",
    "Incremental training allows you to train a new model using an expanded dataset that contains an underlying pattern that was not accounted for in the previous training and which resulted in poor model performance. You can use the artifacts from an existing model and use an expanded dataset to train a new model. Incremental training saves both time and resources as you don’t need to retrain a model from scratch.\n",
    "\n",
    "One may use any dataset (old or new) as long as the dataset format remain the same (set of classes). Incremental training step is similar to the finetuning step discussed above with the following difference: In fine-tuning above, we start with a pre-trained model whereas in incremental training, we start with an existing fine-tuned model.\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85bfc738",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Identify the previously trained model path based on the output location where artifacts are stored previously and the training job name.\n",
    "\n",
    "if use_amt:  # If using amt, select the model for the best training job.\n",
    "    sage_client = boto3.Session().client(\"sagemaker\")\n",
    "    tuning_job_result = sage_client.describe_hyper_parameter_tuning_job(\n",
    "        HyperParameterTuningJobName=hp_tuner._current_job_name\n",
    "    )\n",
    "    last_training_job_name = tuning_job_result[\"BestTrainingJob\"][\"TrainingJobName\"]\n",
    "else:\n",
    "    last_training_job_name = od_estimator._current_job_name\n",
    "\n",
    "last_trained_model_path = f\"{s3_output_location}/{last_training_job_name}/output/model.tar.gz\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "64f50a45",
   "metadata": {},
   "outputs": [],
   "source": [
    "incremental_train_output_prefix = \"jumpstart-example-ic-incremental-training\"\n",
    "\n",
    "incremental_s3_output_location = f\"s3://{output_bucket}/{incremental_train_output_prefix}/output\"\n",
    "\n",
    "incremental_training_job_name = name_from_base(f\"jumpstart-example-{model_id}-incremental-training\")\n",
    "\n",
    "incremental_train_estimator = Estimator(\n",
    "    role=aws_role,\n",
    "    image_uri=train_image_uri,\n",
    "    source_dir=train_source_uri,\n",
    "    model_uri=last_trained_model_path,\n",
    "    entry_point=\"transfer_learning.py\",\n",
    "    instance_count=1,\n",
    "    instance_type=training_instance_type,\n",
    "    max_run=360000,\n",
    "    hyperparameters=hyperparameters,\n",
    "    output_path=incremental_s3_output_location,\n",
    "    base_job_name=incremental_training_job_name,\n",
    "    metric_definitions=training_metric_definitions,\n",
    ")\n",
    "\n",
    "incremental_train_estimator.fit({\"training\": training_dataset_s3_path}, logs=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94374951",
   "metadata": {},
   "source": [
    "Once trained, we can use the same steps as in [Deploy & run Inference on the fine-tuned model](#4.6.-Deploy-&-run-Inference-on-the-fine-tuned-model) to deploy the model."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c92c563c",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/introduction_to_amazon_algorithms|object_detection_tensorflow|Amazon_Tensorflow_Object_Detection.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/introduction_to_amazon_algorithms|object_detection_tensorflow|Amazon_Tensorflow_Object_Detection.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/introduction_to_amazon_algorithms|object_detection_tensorflow|Amazon_Tensorflow_Object_Detection.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/introduction_to_amazon_algorithms|object_detection_tensorflow|Amazon_Tensorflow_Object_Detection.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/introduction_to_amazon_algorithms|object_detection_tensorflow|Amazon_Tensorflow_Object_Detection.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/introduction_to_amazon_algorithms|object_detection_tensorflow|Amazon_Tensorflow_Object_Detection.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/introduction_to_amazon_algorithms|object_detection_tensorflow|Amazon_Tensorflow_Object_Detection.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/introduction_to_amazon_algorithms|object_detection_tensorflow|Amazon_Tensorflow_Object_Detection.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/introduction_to_amazon_algorithms|object_detection_tensorflow|Amazon_Tensorflow_Object_Detection.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/introduction_to_amazon_algorithms|object_detection_tensorflow|Amazon_Tensorflow_Object_Detection.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/introduction_to_amazon_algorithms|object_detection_tensorflow|Amazon_Tensorflow_Object_Detection.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/introduction_to_amazon_algorithms|object_detection_tensorflow|Amazon_Tensorflow_Object_Detection.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/introduction_to_amazon_algorithms|object_detection_tensorflow|Amazon_Tensorflow_Object_Detection.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/introduction_to_amazon_algorithms|object_detection_tensorflow|Amazon_Tensorflow_Object_Detection.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/introduction_to_amazon_algorithms|object_detection_tensorflow|Amazon_Tensorflow_Object_Detection.ipynb)\n"
   ]
  }
 ],
 "metadata": {
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (Data Science 3.0)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/sagemaker-data-science-310-v1"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}