       prepare_data/sm-ground_truth_pretrained_model_labeling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using a Pre-Trained Model for Cost Effective Data Labeling\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/prepare_data|sm-ground_truth_pretrained_model_labeling.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "1. [Introduction](#Introduction)\n",
    "2. [Iteration #1: Create Initial Labeling Job](#Iteration1)\n",
    "3. [Iteration #2: Labeling Job with Pre-Trained Model](#Iteration2)\n",
    "3. [Iteration #3: Second Data Subset Without Pre-Trained Model](#Iteration3)\n",
    "4. [Conclusion](#Conclusion)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "SageMaker Ground Truth is a fully managed service for labeling datasets for machine learning applications. Ground Truth allows you to start a labeling job with a pre-trained model, which is a great way to accelerate the labeling process. If you have a machine learning model that already encodes some domain knowledge about your dataset, you can use it to \"jump start\" Ground Truth's auto-labeling process. \n",
    "\n",
    "This example takes you through an end-to-end workflow to demonstrate the use of pre-trained models with SageMaker Ground Truth. We'll proceed in two parts. First, we'll start with an unlabeled image dataset and acquire labels using SageMaker Ground Truth. Then we'll create a new labeling job with a held-out dataset, and we'll provide the machine learning model that was trained for us in the previous step by SageMkaer Ground Truth. In the end, we'll see how much the second labeling job benefitted from the knowledge acquired during the first labeling job!\n",
    "\n",
    "### Prerequisites\n",
    "This notebook builds off the the examples and lessons from our [beginner notebook for object detection labeling jobs](https://github.com/awslabs/amazon-sagemaker-examples/blob/master/ground_truth_labeling_jobs/ground_truth_object_detection_tutorial/object_detection_tutorial.ipynb). If you haven't already done so, we highly recommend that you familiarize yourself with this notebook before proceeding.\n",
    "\n",
    "To run this notebook, you can simply execute each cell in order. To understand what's happening, you'll need:\n",
    "* An S3 bucket you can write to -- please provide its name in the following cell. The bucket must be in the same region as this SageMaker Notebook instance. You can also change the `EXP_NAME` to any valid S3 prefix. All the files related to this experiment will be stored in that prefix of your bucket. \n",
    "* The S3 bucket that you use for this demo must have a CORS policy attached. To learn more about this requirement, and how to attach a CORS policy to an S3 bucket, see [CORS Permission Requirement](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-cors-update.html).\n",
    "* Familiarity with Python and [numpy](http://www.numpy.org/).\n",
    "* Basic familiarity with [AWS S3](https://docs.aws.amazon.com/s3/index.html).\n",
    "* Basic understanding of [AWS Sagemaker](https://aws.amazon.com/sagemaker/).\n",
    "* Basic familiarity with [AWS Command Line Interface (CLI)](https://aws.amazon.com/cli/) -- ideally, you should have it set up with credentials to access the AWS account you're running this notebook from.\n",
    "\n",
    "This notebook has only been tested on a SageMaker notebook instance. The runtimes given are approximate. We used an `ml.m4.xlarge` instance in our tests. However, you can likely run it on a local instance by first executing the cell below on SageMaker and then copying the `role` string to your local copy of the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import os\n",
    "from collections import namedtuple\n",
    "from collections import defaultdict\n",
    "from collections import Counter\n",
    "from datetime import datetime\n",
    "import itertools\n",
    "import base64\n",
    "import glob\n",
    "import json\n",
    "import random\n",
    "import time\n",
    "import imageio\n",
    "import numpy as np\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "import shutil\n",
    "from matplotlib.backends.backend_pdf import PdfPages\n",
    "from sklearn.metrics import confusion_matrix\n",
    "import boto3\n",
    "import botocore\n",
    "import sagemaker\n",
    "from urllib.parse import urlparse\n",
    "\n",
    "BUCKET = \"<< YOUR S3 BUCKET NAME >>\"\n",
    "EXP_NAME = \"ground-truth-pretrained-model-demo\"\n",
    "EXP_NAME_1 = EXP_NAME + \"/iteration-1\"  # Any valid S3 prefix.\n",
    "EXP_NAME_2 = EXP_NAME + \"/iteration-2\"  # Any valid S3 prefix.\n",
    "EXP_NAME_3 = EXP_NAME + \"/iteration-3\"  # Any valid S3 prefix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make sure the bucket is in the same region as this notebook.\n",
    "role = sagemaker.get_execution_role()\n",
    "region = boto3.session.Session().region_name\n",
    "s3 = boto3.client(\"s3\")\n",
    "bucket_region = s3.head_bucket(Bucket=BUCKET)[\"ResponseMetadata\"][\"HTTPHeaders\"][\n",
    "    \"x-amz-bucket-region\"\n",
    "]\n",
    "assert (\n",
    "    bucket_region == region\n",
    "), \"Your S3 bucket {} and this notebook need to be in the same region.\".format(BUCKET)\n",
    "\n",
    "sagemaker_client = boto3.client(\"sagemaker\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Iteration #1: Create Initial Labeling Job\n",
    "\n",
    "## Setup\n",
    "\n",
    "**This section should take about 4 hours to complete.**\n",
    "\n",
    "We will first run a labeling job. This involves several steps: collecting the images we want annotated, creating instructions, and writing a labeling job specification. Using a public workforce, this section should take about 4 hours. However, this will vary depending on the availability of workers.\n",
    "\n",
    "### Prepare the data\n",
    "We will first download images and labels of a subset of the [Google Open Images Dataset](https://storage.googleapis.com/openimages/web/index.html). These labels were [carefully verified](https://storage.googleapis.com/openimages/web/factsfigures.html). Later, we will compare Ground Truth annotations to these labels. Our dataset will consist of images of various species of bird.\n",
    "\n",
    "This is a diverse dataset of interesting images, and it should be fun for the human annotators to work with. You are free to ask the annotators to annotate any images you wish as long as the images do not contain adult content. In this case, you must adjust the labeling job request this job produces; please check the Ground Truth documentation.\n",
    "\n",
    "We will copy these images to our local `BUCKET` and create a corresponding *input manifest*. The input manifest is a formatted list of the S3 locations of the images we want Ground Truth to annotate. We will upload this manifest to our S3 `BUCKET`.\n",
    "\n",
    "#### Disclosure regarding the Open Images Dataset V4:\n",
    "Open Images Dataset V4 is created by Google Inc. We have not modified the images or the accompanying annotations. You can obtain the images and the annotations [here](https://storage.googleapis.com/openimages/web/download.html). The annotations are licensed by Google Inc. under [CC BY 4.0](https://creativecommons.org/licenses/by/2.0/) license. The images are listed as having a [CC BY 2.0](https://creativecommons.org/licenses/by/2.0/) license. The following paper describes Open Images V4 in depth: from the data collection and annotation to detailed statistics about the data and evaluation of models trained on it.\n",
    "\n",
    "A. Kuznetsova, H. Rom, N. Alldrin, J. Uijlings, I. Krasin, J. Pont-Tuset, S. Kamali, S. Popov, M. Malloci, T. Duerig, and V. Ferrari.\n",
    "*The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale.* arXiv:1811.00982, 2018. ([link to PDF](https://arxiv.org/abs/1811.00982))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download and process the Open Images annotations.\n",
    "!wget https://storage.googleapis.com/openimages/2018_04/test/test-annotations-bbox.csv\n",
    "!wget https://storage.googleapis.com/openimages/2018_04/bbox_labels_600_hierarchy.json\n",
    "\n",
    "with open(\"bbox_labels_600_hierarchy.json\", \"r\") as f:\n",
    "    hierarchy = json.load(f)\n",
    "\n",
    "CLASS_NAME = \"Bird\"\n",
    "CLASS_ID = \"/m/015p6\"\n",
    "NUM_IMGS = 2500\n",
    "\n",
    "# Find all the subclasses of the desired image class (e.g. 'swans' and 'pigeons' etc if CLASS_NAME=='Bird').\n",
    "good_subclasses = set()\n",
    "\n",
    "\n",
    "def get_all_subclasses(hierarchy, good_subtree=False):\n",
    "    if hierarchy[\"LabelName\"] == CLASS_ID:\n",
    "        good_subtree = True\n",
    "    if good_subtree:\n",
    "        good_subclasses.add(hierarchy[\"LabelName\"])\n",
    "    if \"Subcategory\" in hierarchy:\n",
    "        for subcat in hierarchy[\"Subcategory\"]:\n",
    "            get_all_subclasses(subcat, good_subtree=good_subtree)\n",
    "    return good_subclasses\n",
    "\n",
    "\n",
    "good_subclasses = get_all_subclasses(hierarchy)\n",
    "\n",
    "\n",
    "fids2bbs = defaultdict(list)\n",
    "# Skip images with risky content.\n",
    "skip_these_images = [\"251d4c429f6f9c39\", \"065ad49f98157c8d\"]\n",
    "\n",
    "with open(\"test-annotations-bbox.csv\", \"r\") as f:\n",
    "    for line in f.readlines()[1:]:\n",
    "\n",
    "        line = line.strip().split(\",\")\n",
    "        img_id, _, cls_id, conf, xmin, xmax, ymin, ymax, *_ = line\n",
    "        if img_id in skip_these_images:\n",
    "            continue\n",
    "        copy_source = {\"Bucket\": \"open-images-dataset\", \"Key\": \"test/{}.jpg\".format(img_id)}\n",
    "\n",
    "        if cls_id in good_subclasses:\n",
    "            if s3.head_object(**copy_source)[\"ContentLength\"] / 1024 ** 2 > 6:\n",
    "                continue  # skip images larger than 6 MB\n",
    "            fids2bbs[img_id].append([CLASS_NAME, xmin, xmax, ymin, ymax])\n",
    "\n",
    "        if len(fids2bbs) == NUM_IMGS:\n",
    "            break\n",
    "\n",
    "# Copy the images to our local bucket.\n",
    "s3 = boto3.client(\"s3\")\n",
    "for img_id_id, img_id in enumerate(fids2bbs.keys()):\n",
    "    if img_id_id % 100 == 0:\n",
    "        print(\"Copying image {} / {}\".format(img_id_id, NUM_IMGS))\n",
    "    copy_source = {\"Bucket\": \"open-images-dataset\", \"Key\": \"test/{}.jpg\".format(img_id)}\n",
    "    s3.copy(copy_source, BUCKET, \"{}/images/{}.jpg\".format(EXP_NAME, img_id))\n",
    "print(\"Done!\")\n",
    "\n",
    "# Split dataset into two subsets\n",
    "first_iteration_dataset = list(fids2bbs.keys())[: len(fids2bbs.keys()) // 2]\n",
    "second_iteration_dataset = list(fids2bbs.keys())[len(fids2bbs.keys()) // 2 :]\n",
    "\n",
    "manifest_name_iteration_1 = \"iteration-1-input.manifest\"\n",
    "with open(manifest_name_iteration_1, \"w\") as f:\n",
    "    for img in first_iteration_dataset:\n",
    "        img_path = \"s3://{}/{}/images/{}.jpg\".format(BUCKET, EXP_NAME, img)\n",
    "        f.write('{\"source-ref\": \"' + img_path + '\"}\\n')\n",
    "s3.upload_file(manifest_name_iteration_1, BUCKET, EXP_NAME_1 + \"/\" + manifest_name_iteration_1)\n",
    "\n",
    "manifest_name_iteration_2 = \"iteration-2-input.manifest\"\n",
    "with open(manifest_name_iteration_2, \"w\") as f:\n",
    "    for img in second_iteration_dataset:\n",
    "        img_path = \"s3://{}/{}/images/{}.jpg\".format(BUCKET, EXP_NAME, img)\n",
    "        f.write('{\"source-ref\": \"' + img_path + '\"}\\n')\n",
    "s3.upload_file(manifest_name_iteration_2, BUCKET, EXP_NAME_2 + \"/\" + manifest_name_iteration_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After running the cell above, you should be able to go to `s3://BUCKET/EXP_NAME/images` in the [S3 console](https://console.aws.amazon.com/s3/) and see 2500 images. We recommend you inspect these images! You can download them to a local machine using the AWS CLI."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Specify the categories\n",
    "\n",
    "To run an object detection labeling job, you must decide on a set of classes the annotators can choose from. At the moment, Ground Truth only supports annotating one OD class at a time. In our case, the singleton class list is simply `[\"Bird\"]`.  To work with Ground Truth, this list needs to be converted to a .json file and uploaded to the S3 `BUCKET`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "CLASS_LIST = [CLASS_NAME]\n",
    "print(\"Label space is {}\".format(CLASS_LIST))\n",
    "\n",
    "json_body = {\"labels\": [{\"label\": label} for label in CLASS_LIST]}\n",
    "with open(\"class_labels.json\", \"w\") as f:\n",
    "    json.dump(json_body, f)\n",
    "\n",
    "s3.upload_file(\"class_labels.json\", BUCKET, EXP_NAME + \"/class_labels.json\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should now see `class_labels.json` in `s3://BUCKET/EXP_NAME/`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create the instruction template\n",
    "\n",
    "Part or all of your images will be annotated by human annotators. It is **essential** to provide good instructions. Good instructions are:\n",
    "1. Concise. We recommend limiting verbal/textual instruction to two sentences and focusing on clear visuals.\n",
    "2. Visual. In the case of object detection, we recommend providing several labeled examples with different numbers of boxes.\n",
    "\n",
    "When used through the AWS Console, Ground Truth helps you create the instructions using a visual wizard. When using the API, you need to create an HTML template for your instructions. Below, we prepare a very simple but effective template and upload it to your S3 bucket.\n",
    "\n",
    "NOTE: If you use any images in your template (as we do), they need to be publicly accessible. You can enable public access to files in your S3 bucket through the S3 Console, as described in [S3 Documentation](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/set-object-permissions.html). \n",
    "\n",
    "#### Testing your instructions\n",
    "**It is very easy to create broken instructions.** This might cause your labeling job to fail. However, it might also cause your job to complete with meaningless results if, for example, the annotators have no idea what to do or the instructions are misleading. At the moment the only way to test the instructions is to run your job in a private workforce. This is a way to run a mock labeling job for free. We describe how in [Verify your task using a private team [OPTIONAL]](#Verify-your-task-using-a-private-team-[OPTIONAL]).\n",
    "\n",
    "It is helpful to show examples of correctly labeled images in the instructions. The following code block produces several such examples for our dataset and saves them in `s3://BUCKET/EXP_NAME/`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot sample images.\n",
    "def plot_bbs(ax, bbs, img):\n",
    "    '''Add bounding boxes to images.'''\n",
    "    ax.imshow(img)\n",
    "    imh, imw, _ = img.shape\n",
    "    for bb in bbs:\n",
    "        xmin, xmax, ymin, ymax = bb\n",
    "        xmin *= imw\n",
    "        xmax *= imw\n",
    "        ymin *= imh\n",
    "        ymax *= imh\n",
    "        rec = plt.Rectangle((xmin, ymin), xmax-xmin, ymax-ymin, fill=None, lw=4, edgecolor='blue')\n",
    "        ax.add_patch(rec)\n",
    "        \n",
    "plt.figure(facecolor='white', dpi=100, figsize=(3, 7))\n",
    "plt.suptitle('Please draw a box\\n around each {}\\n like the examples below.\\n Thank you!'.format(CLASS_NAME), fontsize=15)\n",
    "for fid_id, (fid, bbs) in enumerate([list(fids2bbs.items())[idx] for idx in [1, 3]]):\n",
    "    !aws s3 cp s3://open-images-dataset/test/{fid}.jpg .\n",
    "    img = imageio.imread(fid + '.jpg')\n",
    "    bbs = [[float(a) for a in annot[1:]] for annot in bbs]\n",
    "    ax = plt.subplot(2, 1, fid_id+1)\n",
    "    plot_bbs(ax, bbs, img)\n",
    "    plt.axis('off')\n",
    "    \n",
    "plt.savefig('instructions.png', dpi=60)\n",
    "with open('instructions.png', 'rb') as instructions:\n",
    "    instructions_uri = base64.b64encode(instructions.read()).decode('utf-8').replace('\\n', '')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.core.display import HTML, display\n",
    "\n",
    "\n",
    "def make_template(test_template=False, save_fname=\"instructions.template\"):\n",
    "    template = r\"\"\"<script src=\"https://assets.crowd.aws/crowd-html-elements.js\"></script>\n",
    "    <crowd-form>\n",
    "      <crowd-bounding-box\n",
    "        name=\"boundingBox\"\n",
    "        src=\"{{{{ task.input.taskObject | grant_read_access }}}}\"\n",
    "        header=\"Dear Annotator, please draw a tight box around each {class_name} you see (if there are more than 8 birds, draw boxes around at least 8). Thank you!\"\n",
    "        labels=\"{labels_str}\"\n",
    "      >\n",
    "        <full-instructions header=\"Please annotate each {class_name}.\">\n",
    "\n",
    "    <ol>\n",
    "        <li><strong>Inspect</strong> the image</li>\n",
    "        <li><strong>Determine</strong> if the specified label is/are visible in the picture.</li>\n",
    "        <li><strong>Outline</strong> each instance of the specified label in the image using the provided “Box” tool.</li>\n",
    "    </ol>\n",
    "    <ul>\n",
    "        <li>Boxes should fit tight around each object</li>\n",
    "        <li>Do not include parts of the object are overlapping or that cannot be seen, even though you think you can interpolate the whole shape.</li>\n",
    "        <li>Avoid including shadows.</li>\n",
    "        <li>If the target is off screen, draw the box up to the edge of the image.</li>\n",
    "    </ul>\n",
    "\n",
    "        </full-instructions>\n",
    "        <short-instructions>\n",
    "        <img src=\"data:image/png;base64,{instructions_uri}\" style=\"max-width:100%\">\n",
    "        </short-instructions>\n",
    "      </crowd-bounding-box>\n",
    "    </crowd-form>\n",
    "    \"\"\".format(\n",
    "        class_name=CLASS_NAME,\n",
    "        instructions_uri=instructions_uri,\n",
    "        labels_str=str(CLASS_LIST)\n",
    "        if test_template\n",
    "        else \"{{ task.input.labels | to_json | escape }}\",\n",
    "    )\n",
    "    with open(save_fname, \"w\") as f:\n",
    "        f.write(template)\n",
    "\n",
    "\n",
    "make_template(test_template=True, save_fname=\"instructions.html\")\n",
    "make_template(test_template=False, save_fname=\"instructions.template\")\n",
    "s3.upload_file(\"instructions.template\", BUCKET, EXP_NAME + \"/instructions.template\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should now be able to find your template in `s3://BUCKET/EXP_NAME/instructions.template`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "private_workteam_arn = (\n",
    "    \"<< your private workteam ARN here >>\"  # If you want to use a private workteam\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define pre-built lambda functions for use in the labeling job \n",
    "\n",
    "Before we submit the request, we need to define the ARNs for four key components of the labeling job: 1) the workteam, 2) the annotation consolidation Lambda function, 3) the pre-labeling task Lambda function, and 4) the machine learning algorithm to perform auto-annotation. These functions are defined by strings with region names and AWS service account numbers, so we will define a mapping below that will enable you to run this notebook in any of our supported regions. \n",
    "\n",
    "See the official documentation for the available ARNs:\n",
    "* [Documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/API_HumanTaskConfig.html#SageMaker-Type-HumanTaskConfig-PreHumanTaskLambdaArn) for available pre-human ARNs for other workflows.\n",
    "* [Documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/API_AnnotationConsolidationConfig.html#SageMaker-Type-AnnotationConsolidationConfig-AnnotationConsolidationLambdaArn) for available annotation consolidation ANRs for other workflows.\n",
    "* [Documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/API_LabelingJobAlgorithmsConfig.html#SageMaker-Type-LabelingJobAlgorithmsConfig-LabelingJobAlgorithmSpecificationArn) for available auto-labeling ARNs for other workflows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Specify ARNs for resources needed to run an object detection job.\n",
    "ac_arn_map = {\n",
    "    \"us-west-2\": \"081040173940\",\n",
    "    \"us-east-1\": \"432418664414\",\n",
    "    \"us-east-2\": \"266458841044\",\n",
    "    \"eu-west-1\": \"568282634449\",\n",
    "    \"ap-northeast-1\": \"477331159723\",\n",
    "}\n",
    "\n",
    "prehuman_arn = \"arn:aws:lambda:{}:{}:function:PRE-BoundingBox\".format(region, ac_arn_map[region])\n",
    "acs_arn = \"arn:aws:lambda:{}:{}:function:ACS-BoundingBox\".format(region, ac_arn_map[region])\n",
    "labeling_algorithm_specification_arn = \"arn:aws:sagemaker:{}:027400017018:labeling-job-algorithm-specification/object-detection\".format(\n",
    "    region\n",
    ")\n",
    "workteam_arn = \"arn:aws:sagemaker:{}:394669845002:workteam/public-crowd/default\".format(region)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Create Labeling Job\n",
    "\n",
    "We will start Ground Truth labeling job by submitting a request through the SageMaker API. The request contains the \n",
    "full configuration of the annotation task, and allows you to modify the fine details of\n",
    "the job that are fixed to default values when you use the AWS Console. The parameters that make up the request are described in more detail in the [SageMaker Ground Truth documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/API_CreateLabelingJob.html).\n",
    "\n",
    "After you submit the request, you should be able to see the job in your AWS Console, at `Amazon SageMaker > Labeling Jobs`.\n",
    "You can track the progress of the job there. This job will take several hours to complete. If your job\n",
    "is larger (say 100,000 images), the speed and cost benefit of auto-labeling should be larger.\n",
    "\n",
    "### Verify your task using a private team [OPTIONAL]\n",
    "If you chose to follow the steps in [Create a private team](#Create-a-private-team-to-test-your-task-[OPTIONAL]), you can first verify that your task runs as expected. To do this:\n",
    "1. Set VERIFY_USING_PRIVATE_WORKFORCE to True in the cell below.\n",
    "2. Run the next two cells. This will define the task and submit it to the private workforce (you).\n",
    "3. After a few minutes, you should be able to see your task in your private workforce interface [Create a private team](#Create-a-private-team-to-test-your-task-[OPTIONAL]).\n",
    "Please verify that the task appears as you want it to appear.\n",
    "4. If everything is in order, change `VERIFY_USING_PRIVATE_WORKFORCE` to `False` and rerun the cell below to start the real annotation task!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "VERIFY_USING_PRIVATE_WORKFORCE = False\n",
    "\n",
    "\n",
    "task_description = \"Dear Annotator, please draw a box around each {}. Thank you!\".format(CLASS_NAME)\n",
    "task_keywords = [\"image\", \"object\", \"detection\"]\n",
    "task_title = \"Please draw a box around each {}.\".format(CLASS_NAME)\n",
    "job_name = \"pretrained-model-iteration-1\" + str(int(time.time()))\n",
    "\n",
    "human_task_config = {\n",
    "    \"AnnotationConsolidationConfig\": {\n",
    "        \"AnnotationConsolidationLambdaArn\": acs_arn,\n",
    "    },\n",
    "    \"PreHumanTaskLambdaArn\": prehuman_arn,\n",
    "    \"MaxConcurrentTaskCount\": 200,  # 200 images will be sent at a time to the workteam.\n",
    "    \"NumberOfHumanWorkersPerDataObject\": 3,  # We will obtain and consolidate 3 human annotations for each image.\n",
    "    \"TaskAvailabilityLifetimeInSeconds\": 21600,  # Your workteam has 6 hours to complete all pending tasks.\n",
    "    \"TaskDescription\": task_description,\n",
    "    \"TaskKeywords\": task_keywords,\n",
    "    \"TaskTimeLimitInSeconds\": 300,  # Each image must be labeled within 5 minutes.\n",
    "    \"TaskTitle\": task_title,\n",
    "    \"UiConfig\": {\n",
    "        \"UiTemplateS3Uri\": \"s3://{}/{}/instructions.template\".format(BUCKET, EXP_NAME),\n",
    "    },\n",
    "}\n",
    "\n",
    "if not VERIFY_USING_PRIVATE_WORKFORCE:\n",
    "    human_task_config[\"PublicWorkforceTaskPrice\"] = {\n",
    "        \"AmountInUsd\": {\n",
    "            \"Dollars\": 0,\n",
    "            \"Cents\": 3,\n",
    "            \"TenthFractionsOfACent\": 6,\n",
    "        }\n",
    "    }\n",
    "    human_task_config[\"WorkteamArn\"] = workteam_arn\n",
    "else:\n",
    "    human_task_config[\"WorkteamArn\"] = private_workteam_arn\n",
    "\n",
    "ground_truth_request = {\n",
    "    \"InputConfig\": {\n",
    "        \"DataSource\": {\n",
    "            \"S3DataSource\": {\n",
    "                \"ManifestS3Uri\": \"s3://{}/{}/{}\".format(\n",
    "                    BUCKET, EXP_NAME_1, manifest_name_iteration_1\n",
    "                ),\n",
    "            }\n",
    "        },\n",
    "        \"DataAttributes\": {\n",
    "            \"ContentClassifiers\": [\"FreeOfPersonallyIdentifiableInformation\", \"FreeOfAdultContent\"]\n",
    "        },\n",
    "    },\n",
    "    \"OutputConfig\": {\n",
    "        \"S3OutputPath\": \"s3://{}/{}/output/\".format(BUCKET, EXP_NAME_1),\n",
    "    },\n",
    "    \"HumanTaskConfig\": human_task_config,\n",
    "    \"LabelingJobName\": job_name,\n",
    "    \"RoleArn\": role,\n",
    "    \"LabelAttributeName\": \"category\",\n",
    "    \"LabelCategoryConfigS3Uri\": \"s3://{}/{}/class_labels.json\".format(BUCKET, EXP_NAME),\n",
    "}\n",
    "\n",
    "\n",
    "ground_truth_request[\"LabelingJobAlgorithmsConfig\"] = {\n",
    "    \"LabelingJobAlgorithmSpecificationArn\": labeling_algorithm_specification_arn\n",
    "}\n",
    "\n",
    "sagemaker_client.create_labeling_job(**ground_truth_request)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Monitor job progress\n",
    "A Ground Truth job can take a few hours to complete (if your dataset is larger than 10000 images, it can take much longer than that!). One way to monitor the job's progress is through AWS Console. In this notebook, we will use Ground Truth output files and Cloud Watch logs in order to monitor the progress.\n",
    "\n",
    "You can re-evaluate the next cell repeatedly. It sends a `describe_labeling_job` request which should tell you whether the job is completed or not. If it is, then 'LabelingJobStatus' will be 'Completed'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sagemaker_client = boto3.client(\"sagemaker\")\n",
    "sagemaker_client.describe_labeling_job(LabelingJobName=job_name)[\"LabelingJobStatus\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next cell extracts detailed information on how your job is doing. You can re-evaluate it at any time. It should give you:\n",
    "* The number of human and machine-annotated images across the iterations of your labeling job.\n",
    "* The training curves of any neural network training jobs launched by Ground Truth.\n",
    "* The cost of the human- and machine-annotated labels.\n",
    "\n",
    "To understand the pricing, study [this document](https://aws.amazon.com/sagemaker/groundtruth/pricing/) carefully. In our case, each human label costs `$0.08 + 5 * $0.036 = $0.26` and each auto-label costs `$0.08`. If you set `RUN_FULL_AL_DEMO=True`, there is also the added cost of using SageMaker instances for neural net training and inference during auto-labeling. However, this should be insignificant compared to the other costs.\n",
    "\n",
    "If `RUN_FULL_AL_DEMO==True`, then the job will proceed in multiple iterations. \n",
    "* Iteration 1: Ground Truth will send out 10 images as 'probes' for human annotation. If these are successfully annotated, proceed to Iteration 2.\n",
    "* Iteration 2: Send out a batch of `MaxConcurrentTaskCount - 10` (in our case, 190) images for human annotation to obtain an active learning training batch.\n",
    "* Iteration 3: Send out another batch of 200 images for human annotation to obtain an active learning validation set.\n",
    "* Iteration 4a: Train a neural net to do auto-labeling. Auto-label as many data points as possible. \n",
    "* Iteration 4b: If there is any data leftover, send out at most 200 images for human annotation.\n",
    "* Repeat Iteration 4a and 4b until all data is annotated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#job_name = 'pretrained-model-iteration-11554764290'\n",
    "\n",
    "HUMAN_PRICE = 0.26\n",
    "AUTO_PRICE = 0.08\n",
    "\n",
    "try:\n",
    "    os.makedirs('od_output_data/', exist_ok=False)\n",
    "except FileExistsError:\n",
    "    shutil.rmtree('od_output_data/')\n",
    "    \n",
    "S3_OUTPUT = boto3.client('sagemaker').describe_labeling_job(LabelingJobName=job_name)[\n",
    "    'OutputConfig']['S3OutputPath'] + job_name\n",
    "\n",
    "# Count number of human annotations in each class each iteration.\n",
    "!aws s3 cp {S3_OUTPUT + '/annotations/consolidated-annotation/consolidation-response'} od_output_data/consolidation-response --recursive --quiet\n",
    "consolidated_nboxes = defaultdict(int)\n",
    "consolidated_nims = defaultdict(int)\n",
    "consolidation_times = {}\n",
    "consolidated_cost_times = []\n",
    "obj_ids = set()\n",
    "\n",
    "for consolidated_fname in glob.glob('od_output_data/consolidation-response/**', recursive=True):\n",
    "    if consolidated_fname.endswith('json'):\n",
    "        iter_id = int(consolidated_fname.split('/')[-2][-1])\n",
    "        # Store the time of the most recent consolidation event as iteration time.\n",
    "        iter_time = datetime.strptime(consolidated_fname.split('/')[-1], '%Y-%m-%d_%H:%M:%S.json')\n",
    "        if iter_id in consolidation_times:\n",
    "            consolidation_times[iter_id] = max(consolidation_times[iter_id], iter_time)\n",
    "        else:\n",
    "            consolidation_times[iter_id] = iter_time\n",
    "        consolidated_cost_times.append(iter_time)\n",
    "                                      \n",
    "        with open(consolidated_fname, 'r') as f:\n",
    "            consolidated_data = json.load(f)\n",
    "        for consolidation in consolidated_data:\n",
    "            obj_id = consolidation['datasetObjectId']\n",
    "            n_boxes = len(consolidation['consolidatedAnnotation']['content'][\n",
    "                'category']['annotations'])\n",
    "            if obj_id not in obj_ids:\n",
    "                obj_ids.add(obj_id)\n",
    "                consolidated_nims[iter_id] += 1            \n",
    "                consolidated_nboxes[iter_id] += n_boxes\n",
    "            \n",
    "total_human_labels = sum(consolidated_nims.values())\n",
    "            \n",
    "# Count the number of machine iterations in each class each iteration.\n",
    "!aws s3 cp {S3_OUTPUT + '/activelearning'} od_output_data/activelearning --recursive --quiet\n",
    "auto_nboxes = defaultdict(int)\n",
    "auto_nims = defaultdict(int)\n",
    "auto_times = {}\n",
    "auto_cost_times = []\n",
    "\n",
    "for auto_fname in glob.glob('od_output_data/activelearning/**', recursive=True):\n",
    "    if auto_fname.endswith('auto_annotator_output.txt'):\n",
    "        iter_id = int(auto_fname.split('/')[-3])\n",
    "        with open(auto_fname, 'r') as f:\n",
    "            annots = [' '.join(l.split()[1:]) for l in f.readlines()]\n",
    "        auto_nims[iter_id] += len(annots)\n",
    "        for annot in annots:\n",
    "            annot = json.loads(annot)\n",
    "            time_str = annot['category-metadata']['creation-date']\n",
    "            auto_time = datetime.strptime(time_str, '%Y-%m-%dT%H:%M:%S.%f')\n",
    "            n_boxes = len(annot['category']['annotations'])\n",
    "            auto_nboxes[iter_id] += n_boxes\n",
    "            if iter_id in auto_times:\n",
    "                auto_times[iter_id] = max(auto_times[iter_id], auto_time)\n",
    "            else:\n",
    "                auto_times[iter_id] = auto_time\n",
    "            auto_cost_times.append(auto_time)\n",
    "                \n",
    "total_auto_labels = sum(auto_nims.values())\n",
    "n_iters = max(len(auto_times), len(consolidation_times))\n",
    "\n",
    "# Get plots for auto-annotation neural-net training.\n",
    "def get_training_job_data(training_job_name):\n",
    "    logclient = boto3.client('logs')\n",
    "    log_group_name = '/aws/sagemaker/TrainingJobs'\n",
    "    log_stream_name = logclient.describe_log_streams(logGroupName=log_group_name,\n",
    "        logStreamNamePrefix=training_job_name)['logStreams'][0]['logStreamName']\n",
    "    train_log = logclient.get_log_events(\n",
    "        logGroupName=log_group_name,\n",
    "        logStreamName=log_stream_name,\n",
    "        startFromHead=True\n",
    "    )\n",
    "    events = train_log['events']\n",
    "    next_token = train_log['nextForwardToken']\n",
    "    while True:\n",
    "        train_log = logclient.get_log_events(\n",
    "            logGroupName=log_group_name,\n",
    "            logStreamName=log_stream_name,\n",
    "            startFromHead=True,\n",
    "            nextToken=next_token\n",
    "        )\n",
    "        if train_log['nextForwardToken'] == next_token:\n",
    "            break\n",
    "        events = events + train_log['events']\n",
    "        next_token = train_log['nextForwardToken']\n",
    "\n",
    "    mAPs = []\n",
    "    for event in events:\n",
    "        msg = event['message']\n",
    "        if 'Final configuration' in msg:\n",
    "            num_samples = int(msg.split('num_training_samples\\': u\\'')[1].split('\\'')[0])\n",
    "        elif 'validation mAP <score>=(' in msg:\n",
    "            mAPs.append(float(msg.split('validation mAP <score>=(')[1][:-1]))\n",
    "\n",
    "    return num_samples, mAPs\n",
    "\n",
    "training_data = !aws s3 ls {S3_OUTPUT + '/training/'} --recursive\n",
    "training_sizes = []\n",
    "training_mAPs = []\n",
    "training_iters = []\n",
    "for line in training_data:\n",
    "    if line.split('/')[-1] == 'model.tar.gz':\n",
    "        training_job_name = line.split('/')[-3]\n",
    "        n_samples, mAPs = get_training_job_data(training_job_name)\n",
    "        training_sizes.append(n_samples)\n",
    "        training_mAPs.append(mAPs)\n",
    "        training_iters.append(int(line.split('/')[-5]))\n",
    "        \n",
    "plt.figure(facecolor='white', figsize=(14, 5), dpi=100)\n",
    "ax = plt.subplot(131)\n",
    "total_human = 0\n",
    "total_auto = 0\n",
    "for iter_id in range(1, n_iters + 1):\n",
    "    cost_human = consolidated_nims[iter_id] * HUMAN_PRICE\n",
    "    cost_auto = auto_nims[iter_id] * AUTO_PRICE\n",
    "    total_human += cost_human\n",
    "    total_auto += cost_auto\n",
    "    \n",
    "    plt.bar(iter_id, cost_human, width=.8, color='C0',\n",
    "            label='human' if iter_id==1 else None)\n",
    "    plt.bar(iter_id, cost_auto, bottom=cost_human,\n",
    "            width=.8, color='C1', label='auto' if iter_id==1 else None)\n",
    "plt.title('Total annotation costs:\\n\\${:.2f} human, \\${:.2f} auto'.format(\n",
    "    total_human, total_auto))\n",
    "plt.xlabel('Iter')\n",
    "plt.ylabel('Cost in dollars')\n",
    "plt.legend()\n",
    "\n",
    "plt.subplot(132)\n",
    "plt.title('Total annotation counts:\\nHuman: {} ims, {} boxes\\nMachine: {} ims, {} boxes'.format(\n",
    "    sum(consolidated_nims.values()), sum(consolidated_nboxes.values()), sum(auto_nims.values()), sum(auto_nboxes.values())))\n",
    "for iter_id in consolidated_nims.keys():\n",
    "    plt.bar(iter_id, auto_nims[iter_id], color='C1', width=.4, label='ims, auto' if iter_id==1 else None)\n",
    "    plt.bar(iter_id, consolidated_nims[iter_id],\n",
    "            bottom=auto_nims[iter_id], color='C0', width=.4, label='ims, human' if iter_id==1 else None)\n",
    "    plt.bar(iter_id + .4, auto_nboxes[iter_id], color='C1', alpha=.4, width=.4, label='boxes, auto' if iter_id==1 else None)\n",
    "    plt.bar(iter_id + .4, consolidated_nboxes[iter_id],\n",
    "            bottom=auto_nboxes[iter_id], color='C0', width=.4, alpha=.4, label='boxes, human' if iter_id==1 else None)\n",
    "\n",
    "tick_labels_boxes = ['Iter {}, boxes'.format(iter_id + 1) for iter_id in range(n_iters)]\n",
    "tick_labels_images = ['Iter {}, images'.format(iter_id + 1) for iter_id in range(n_iters)]\n",
    "tick_locations_images = np.arange(n_iters) + 1\n",
    "tick_locations_boxes = tick_locations_images + .4\n",
    "tick_labels = np.concatenate([[tick_labels_boxes[idx], tick_labels_images[idx]] for idx in range(n_iters)])\n",
    "tick_locations = np.concatenate([[tick_locations_boxes[idx], tick_locations_images[idx]] for idx in range(n_iters)])\n",
    "plt.xticks(tick_locations, tick_labels, rotation=90)\n",
    "plt.legend()\n",
    "plt.ylabel('Count')\n",
    "\n",
    "if len(training_sizes) > 0:\n",
    "    plt.subplot(133)\n",
    "    plt.title('Active learning training curves')\n",
    "    plt.grid(True)\n",
    "\n",
    "    cmap = plt.get_cmap('coolwarm')\n",
    "    n_all = len(training_sizes)\n",
    "    for iter_id_id, (iter_id, size, mAPs) in enumerate(zip(training_iters, training_sizes, training_mAPs)):\n",
    "        plt.plot(mAPs, label='Iter {}, auto'.format(iter_id + 1), color=cmap(iter_id_id / max(1, (n_all-1))))\n",
    "        plt.legend()\n",
    "\n",
    "    plt.xlabel('Training epoch')\n",
    "    plt.ylabel('Validation mAP')\n",
    "\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wait for Completion of Job\n",
    "\n",
    "Confirm that the labeling job status returns \"Completed\" before proceeding to the next section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sagemaker_client.describe_labeling_job(LabelingJobName=job_name)[\"LabelingJobStatus\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Iteration #2: Labeling Job with Pre-Trained Model <a class=\"anchor\" id=\"Iteration2\"></a>\n",
    "\n",
    "Now we'll use the model trained during the first labeling job to help label the second subset of our original dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fetch the SageMaker Model ARN of the machine learning model trained in the final iteration of the previous labeling job.\n",
    "pretrained_model_arn = sagemaker_client.describe_labeling_job(LabelingJobName=job_name)[\n",
    "    \"LabelingJobOutput\"\n",
    "][\"FinalActiveLearningModelArn\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "VERIFY_USING_PRIVATE_WORKFORCE = False\n",
    "\n",
    "task_description = \"Dear Annotator, please draw a box around each {}. Thank you!\".format(CLASS_NAME)\n",
    "task_keywords = [\"image\", \"object\", \"detection\"]\n",
    "task_title = \"Please draw a box around each {}.\".format(CLASS_NAME)\n",
    "job_name = \"pretrained-model-iteration-2\" + str(int(time.time()))\n",
    "\n",
    "human_task_config = {\n",
    "    \"AnnotationConsolidationConfig\": {\n",
    "        \"AnnotationConsolidationLambdaArn\": acs_arn,\n",
    "    },\n",
    "    \"PreHumanTaskLambdaArn\": prehuman_arn,\n",
    "    \"MaxConcurrentTaskCount\": 200,  # 200 images will be sent at a time to the workteam.\n",
    "    \"NumberOfHumanWorkersPerDataObject\": 3,  # We will obtain and consolidate 3 human annotations for each image.\n",
    "    \"TaskAvailabilityLifetimeInSeconds\": 21600,  # Your workteam has 6 hours to complete all pending tasks.\n",
    "    \"TaskDescription\": task_description,\n",
    "    \"TaskKeywords\": task_keywords,\n",
    "    \"TaskTimeLimitInSeconds\": 300,  # Each image must be labeled within 5 minutes.\n",
    "    \"TaskTitle\": task_title,\n",
    "    \"UiConfig\": {\n",
    "        \"UiTemplateS3Uri\": \"s3://{}/{}/instructions.template\".format(BUCKET, EXP_NAME),\n",
    "    },\n",
    "}\n",
    "\n",
    "if not VERIFY_USING_PRIVATE_WORKFORCE:\n",
    "    human_task_config[\"PublicWorkforceTaskPrice\"] = {\n",
    "        \"AmountInUsd\": {\n",
    "            \"Dollars\": 0,\n",
    "            \"Cents\": 3,\n",
    "            \"TenthFractionsOfACent\": 6,\n",
    "        }\n",
    "    }\n",
    "    human_task_config[\"WorkteamArn\"] = workteam_arn\n",
    "else:\n",
    "    human_task_config[\"WorkteamArn\"] = private_workteam_arn\n",
    "\n",
    "ground_truth_request = {\n",
    "    \"InputConfig\": {\n",
    "        \"DataSource\": {\n",
    "            \"S3DataSource\": {\n",
    "                \"ManifestS3Uri\": \"s3://{}/{}/{}\".format(\n",
    "                    BUCKET, EXP_NAME_2, manifest_name_iteration_2\n",
    "                ),\n",
    "            }\n",
    "        },\n",
    "        \"DataAttributes\": {\n",
    "            \"ContentClassifiers\": [\"FreeOfPersonallyIdentifiableInformation\", \"FreeOfAdultContent\"]\n",
    "        },\n",
    "    },\n",
    "    \"OutputConfig\": {\n",
    "        \"S3OutputPath\": \"s3://{}/{}/output/\".format(BUCKET, EXP_NAME_2),\n",
    "    },\n",
    "    \"HumanTaskConfig\": human_task_config,\n",
    "    \"LabelingJobName\": job_name,\n",
    "    \"RoleArn\": role,\n",
    "    \"LabelAttributeName\": \"category\",\n",
    "    \"LabelCategoryConfigS3Uri\": \"s3://{}/{}/class_labels.json\".format(BUCKET, EXP_NAME),\n",
    "}\n",
    "\n",
    "\n",
    "ground_truth_request[\"LabelingJobAlgorithmsConfig\"] = {\n",
    "    \"InitialActiveLearningModelArn\": pretrained_model_arn,\n",
    "    \"LabelingJobAlgorithmSpecificationArn\": labeling_algorithm_specification_arn,\n",
    "}\n",
    "\n",
    "sagemaker_client = boto3.client(\"sagemaker\")\n",
    "sagemaker_client.create_labeling_job(**ground_truth_request)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Monitor Job Progress"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sagemaker_client.describe_labeling_job(LabelingJobName=job_name)[\"LabelingJobStatus\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "HUMAN_PRICE = 0.26\n",
    "AUTO_PRICE = 0.08\n",
    "\n",
    "try:\n",
    "    !rm -rf od_output_data\n",
    "    os.makedirs('od_output_data/', exist_ok=False)\n",
    "except FileExistsError:\n",
    "    shutil.rmtree('od_output_data/')\n",
    "    \n",
    "S3_OUTPUT = boto3.client('sagemaker').describe_labeling_job(LabelingJobName=job_name)[\n",
    "    'OutputConfig']['S3OutputPath'] + job_name\n",
    "\n",
    "# Count number of human annotations in each class each iteration.\n",
    "!aws s3 cp {S3_OUTPUT + '/annotations/consolidated-annotation/consolidation-response'} od_output_data/consolidation-response --recursive --quiet\n",
    "consolidated_nboxes = defaultdict(int)\n",
    "consolidated_nims = defaultdict(int)\n",
    "consolidation_times = {}\n",
    "consolidated_cost_times = []\n",
    "obj_ids = set()\n",
    "\n",
    "for consolidated_fname in glob.glob('od_output_data/consolidation-response/**', recursive=True):\n",
    "    if consolidated_fname.endswith('json'):\n",
    "        iter_id = int(consolidated_fname.split('/')[-2][-1])\n",
    "        # Store the time of the most recent consolidation event as iteration time.\n",
    "        iter_time = datetime.strptime(consolidated_fname.split('/')[-1], '%Y-%m-%d_%H:%M:%S.json')\n",
    "        if iter_id in consolidation_times:\n",
    "            consolidation_times[iter_id] = max(consolidation_times[iter_id], iter_time)\n",
    "        else:\n",
    "            consolidation_times[iter_id] = iter_time\n",
    "        consolidated_cost_times.append(iter_time)\n",
    "                                      \n",
    "        with open(consolidated_fname, 'r') as f:\n",
    "            consolidated_data = json.load(f)\n",
    "        for consolidation in consolidated_data:\n",
    "            obj_id = consolidation['datasetObjectId']\n",
    "            n_boxes = len(consolidation['consolidatedAnnotation']['content'][\n",
    "                'category']['annotations'])\n",
    "            if obj_id not in obj_ids:\n",
    "                obj_ids.add(obj_id)\n",
    "                consolidated_nims[iter_id] += 1            \n",
    "                consolidated_nboxes[iter_id] += n_boxes\n",
    "            \n",
    "total_human_labels = sum(consolidated_nims.values())\n",
    "            \n",
    "# Count the number of machine iterations in each class each iteration.\n",
    "!aws s3 cp {S3_OUTPUT + '/activelearning'} od_output_data/activelearning --recursive --quiet\n",
    "auto_nboxes = defaultdict(int)\n",
    "auto_nims = defaultdict(int)\n",
    "auto_times = {}\n",
    "auto_cost_times = []\n",
    "\n",
    "for auto_fname in glob.glob('od_output_data/activelearning/**', recursive=True):\n",
    "    if auto_fname.endswith('auto_annotator_output.txt'):\n",
    "        iter_id = int(auto_fname.split('/')[-3])\n",
    "        with open(auto_fname, 'r') as f:\n",
    "            annots = [' '.join(l.split()[1:]) for l in f.readlines()]\n",
    "        auto_nims[iter_id] += len(annots)\n",
    "        for annot in annots:\n",
    "            annot = json.loads(annot)\n",
    "            time_str = annot['category-metadata']['creation-date']\n",
    "            auto_time = datetime.strptime(time_str, '%Y-%m-%dT%H:%M:%S.%f')\n",
    "            n_boxes = len(annot['category']['annotations'])\n",
    "            auto_nboxes[iter_id] += n_boxes\n",
    "            if iter_id in auto_times:\n",
    "                auto_times[iter_id] = max(auto_times[iter_id], auto_time)\n",
    "            else:\n",
    "                auto_times[iter_id] = auto_time\n",
    "            auto_cost_times.append(auto_time)\n",
    "                \n",
    "total_auto_labels = sum(auto_nims.values())\n",
    "n_iters = max(len(auto_times), len(consolidation_times))\n",
    "\n",
    "# Get plots for auto-annotation neural-net training.\n",
    "def get_training_job_data(training_job_name):\n",
    "    logclient = boto3.client('logs')\n",
    "    log_group_name = '/aws/sagemaker/TrainingJobs'\n",
    "    log_stream_name = logclient.describe_log_streams(logGroupName=log_group_name,\n",
    "        logStreamNamePrefix=training_job_name)['logStreams'][0]['logStreamName']\n",
    "    train_log = logclient.get_log_events(\n",
    "        logGroupName=log_group_name,\n",
    "        logStreamName=log_stream_name,\n",
    "        startFromHead=True\n",
    "    )\n",
    "    events = train_log['events']\n",
    "    next_token = train_log['nextForwardToken']\n",
    "    while True:\n",
    "        train_log = logclient.get_log_events(\n",
    "            logGroupName=log_group_name,\n",
    "            logStreamName=log_stream_name,\n",
    "            startFromHead=True,\n",
    "            nextToken=next_token\n",
    "        )\n",
    "        if train_log['nextForwardToken'] == next_token:\n",
    "            break\n",
    "        events = events + train_log['events']\n",
    "        next_token = train_log['nextForwardToken']\n",
    "\n",
    "    mAPs = []\n",
    "    for event in events:\n",
    "        msg = event['message']\n",
    "        if 'Final configuration' in msg:\n",
    "            num_samples = int(msg.split('num_training_samples\\': u\\'')[1].split('\\'')[0])\n",
    "        elif 'validation mAP <score>=(' in msg:\n",
    "            mAPs.append(float(msg.split('validation mAP <score>=(')[1][:-1]))\n",
    "\n",
    "    return num_samples, mAPs\n",
    "\n",
    "training_data = !aws s3 ls {S3_OUTPUT + '/training/'} --recursive\n",
    "training_sizes = []\n",
    "training_mAPs = []\n",
    "training_iters = []\n",
    "for line in training_data:\n",
    "    if line.split('/')[-1] == 'model.tar.gz':\n",
    "        training_job_name = line.split('/')[-3]\n",
    "        n_samples, mAPs = get_training_job_data(training_job_name)\n",
    "        training_sizes.append(n_samples)\n",
    "        training_mAPs.append(mAPs)\n",
    "        training_iters.append(int(line.split('/')[-5]))\n",
    "        \n",
    "plt.figure(facecolor='white', figsize=(14, 5), dpi=100)\n",
    "ax = plt.subplot(131)\n",
    "total_human = 0\n",
    "total_auto = 0\n",
    "for iter_id in range(1, n_iters + 1):\n",
    "    cost_human = consolidated_nims[iter_id] * HUMAN_PRICE\n",
    "    cost_auto = auto_nims[iter_id] * AUTO_PRICE\n",
    "    total_human += cost_human\n",
    "    total_auto += cost_auto\n",
    "    \n",
    "    plt.bar(iter_id, cost_human, width=.8, color='C0',\n",
    "            label='human' if iter_id==1 else None)\n",
    "    plt.bar(iter_id, cost_auto, bottom=cost_human,\n",
    "            width=.8, color='C1', label='auto' if iter_id==1 else None)\n",
    "    \n",
    "plt.title('Total annotation costs:\\n\\${:.2f} human, \\${:.2f} auto'.format(\n",
    "    total_human, total_auto))\n",
    "plt.xlabel('Iter')\n",
    "plt.ylabel('Cost in dollars')\n",
    "plt.legend()\n",
    "\n",
    "plt.subplot(132)\n",
    "plt.title('Total annotation counts:\\nHuman: {} ims, {} boxes\\nMachine: {} ims, {} boxes'.format(\n",
    "    sum(consolidated_nims.values()), sum(consolidated_nboxes.values()), sum(auto_nims.values()), sum(auto_nboxes.values())))\n",
    "for iter_id in consolidated_nims.keys():\n",
    "    plt.bar(iter_id, auto_nims[iter_id], color='C1', width=.4, label='ims, auto' if iter_id==1 else None)\n",
    "    plt.bar(iter_id, consolidated_nims[iter_id],\n",
    "            bottom=auto_nims[iter_id], color='C0', width=.4, label='ims, human' if iter_id==1 else None)\n",
    "    plt.bar(iter_id + .4, auto_nboxes[iter_id], color='C1', alpha=.4, width=.4, label='boxes, auto' if iter_id==1 else None)\n",
    "    plt.bar(iter_id + .4, consolidated_nboxes[iter_id],\n",
    "            bottom=auto_nboxes[iter_id], color='C0', width=.4, alpha=.4, label='boxes, human' if iter_id==1 else None)\n",
    "\n",
    "tick_labels_boxes = ['Iter {}, boxes'.format(iter_id + 1) for iter_id in range(n_iters)]\n",
    "tick_labels_images = ['Iter {}, images'.format(iter_id + 1) for iter_id in range(n_iters)]\n",
    "tick_locations_images = np.arange(n_iters) + 1\n",
    "tick_locations_boxes = tick_locations_images + .4\n",
    "tick_labels = np.concatenate([[tick_labels_boxes[idx], tick_labels_images[idx]] for idx in range(n_iters)])\n",
    "tick_locations = np.concatenate([[tick_locations_boxes[idx], tick_locations_images[idx]] for idx in range(n_iters)])\n",
    "plt.xticks(tick_locations, tick_labels, rotation=90)\n",
    "plt.legend()\n",
    "plt.ylabel('Count')\n",
    "\n",
    "if len(training_sizes) > 0:\n",
    "    plt.subplot(133)\n",
    "    plt.title('Active learning training curves')\n",
    "    plt.grid(True)\n",
    "\n",
    "    cmap = plt.get_cmap('coolwarm')\n",
    "    n_all = len(training_sizes)\n",
    "    for iter_id_id, (iter_id, size, mAPs) in enumerate(zip(training_iters, training_sizes, training_mAPs)):\n",
    "        plt.plot(mAPs, label='Iter {}, auto'.format(iter_id + 1), color=cmap(iter_id_id / max(1, (n_all-1))))\n",
    "        plt.legend()\n",
    "\n",
    "    plt.xlabel('Training epoch')\n",
    "    plt.ylabel('Validation mAP')\n",
    "\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wait for Completion of Job\n",
    "\n",
    "Confirm that the labeling job status returns \"Completed\" before proceeding to the next section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sagemaker_client.describe_labeling_job(LabelingJobName=job_name)[\"LabelingJobStatus\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Iteration #3: Second Data Subset Without Pre-Trained Model <a class=\"anchor\" id=\"Iteration3\"></a>\n",
    "\n",
    "This time, we'll create a new labeling job using the second subset of the data (the one we just used in the previous labeling job), but we'll start it without the pre-trained model. In the previous step, we saw some significant improvements in cost and labeling time by leveraging a pre-trained model, but some of the differences might be due to the fact that the first and second labeling jobs used different datasets. This third labeling job will provide a more fair comparison, since it is identical to the second labeling job without the pre-trained model specification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "VERIFY_USING_PRIVATE_WORKFORCE = False\n",
    "USE_AUTO_LABELING = True\n",
    "\n",
    "task_description = \"Dear Annotator, please draw a box around each {}. Thank you!\".format(CLASS_NAME)\n",
    "task_keywords = [\"image\", \"object\", \"detection\"]\n",
    "task_title = \"Please draw a box around each {}.\".format(CLASS_NAME)\n",
    "job_name = \"pretrained-model-iteration-3EX\" + str(int(time.time()))\n",
    "\n",
    "human_task_config = {\n",
    "    \"AnnotationConsolidationConfig\": {\n",
    "        \"AnnotationConsolidationLambdaArn\": acs_arn,\n",
    "    },\n",
    "    \"PreHumanTaskLambdaArn\": prehuman_arn,\n",
    "    \"MaxConcurrentTaskCount\": 200,  # 200 images will be sent at a time to the workteam.\n",
    "    \"NumberOfHumanWorkersPerDataObject\": 3,  # We will obtain and consolidate 3 human annotations for each image.\n",
    "    \"TaskAvailabilityLifetimeInSeconds\": 43200,  # Your workteam has 6 hours to complete all pending tasks.\n",
    "    \"TaskDescription\": task_description,\n",
    "    \"TaskKeywords\": task_keywords,\n",
    "    \"TaskTimeLimitInSeconds\": 300,  # Each image must be labeled within 5 minutes.\n",
    "    \"TaskTitle\": task_title,\n",
    "    \"UiConfig\": {\n",
    "        \"UiTemplateS3Uri\": \"s3://{}/{}/instructions.template\".format(BUCKET, EXP_NAME),\n",
    "    },\n",
    "}\n",
    "\n",
    "if not VERIFY_USING_PRIVATE_WORKFORCE:\n",
    "    human_task_config[\"PublicWorkforceTaskPrice\"] = {\n",
    "        \"AmountInUsd\": {\n",
    "            \"Dollars\": 0,\n",
    "            \"Cents\": 3,\n",
    "            \"TenthFractionsOfACent\": 6,\n",
    "        }\n",
    "    }\n",
    "    human_task_config[\"WorkteamArn\"] = workteam_arn\n",
    "else:\n",
    "    human_task_config[\"WorkteamArn\"] = private_workteam_arn\n",
    "\n",
    "ground_truth_request = {\n",
    "    \"InputConfig\": {\n",
    "        \"DataSource\": {\n",
    "            \"S3DataSource\": {\n",
    "                \"ManifestS3Uri\": \"s3://{}/{}/{}\".format(\n",
    "                    BUCKET, EXP_NAME_2, manifest_name_iteration_2\n",
    "                ),\n",
    "            }\n",
    "        },\n",
    "        \"DataAttributes\": {\n",
    "            \"ContentClassifiers\": [\"FreeOfPersonallyIdentifiableInformation\", \"FreeOfAdultContent\"]\n",
    "        },\n",
    "    },\n",
    "    \"OutputConfig\": {\n",
    "        \"S3OutputPath\": \"s3://{}/{}/output/\".format(BUCKET, EXP_NAME_3),\n",
    "    },\n",
    "    \"HumanTaskConfig\": human_task_config,\n",
    "    \"LabelingJobName\": job_name,\n",
    "    \"RoleArn\": role,\n",
    "    \"LabelAttributeName\": \"category\",\n",
    "    \"LabelCategoryConfigS3Uri\": \"s3://{}/{}/class_labels.json\".format(BUCKET, EXP_NAME),\n",
    "}\n",
    "\n",
    "if USE_AUTO_LABELING:\n",
    "    ground_truth_request[\"LabelingJobAlgorithmsConfig\"] = {\n",
    "        \"LabelingJobAlgorithmSpecificationArn\": labeling_algorithm_specification_arn\n",
    "    }\n",
    "\n",
    "sagemaker_client = boto3.client(\"sagemaker\")\n",
    "sagemaker_client.create_labeling_job(**ground_truth_request)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Monitor Job Progress"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sagemaker_client.describe_labeling_job(LabelingJobName=job_name)[\"LabelingJobStatus\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Post-Process Job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "HUMAN_PRICE = 0.26\n",
    "AUTO_PRICE = 0.08\n",
    "\n",
    "try:\n",
    "    !rm -rf od_output_data\n",
    "    os.makedirs('od_output_data/', exist_ok=False)\n",
    "except FileExistsError:\n",
    "    shutil.rmtree('od_output_data/')\n",
    "    \n",
    "S3_OUTPUT = boto3.client('sagemaker').describe_labeling_job(LabelingJobName=job_name)[\n",
    "    'OutputConfig']['S3OutputPath'] + job_name\n",
    "\n",
    "# Count number of human annotations in each class each iteration.\n",
    "!aws s3 cp {S3_OUTPUT + '/annotations/consolidated-annotation/consolidation-response'} od_output_data/consolidation-response --recursive --quiet\n",
    "consolidated_nboxes = defaultdict(int)\n",
    "consolidated_nims = defaultdict(int)\n",
    "consolidation_times = {}\n",
    "consolidated_cost_times = []\n",
    "obj_ids = set()\n",
    "\n",
    "for consolidated_fname in glob.glob('od_output_data/consolidation-response/**', recursive=True):\n",
    "    if consolidated_fname.endswith('json'):\n",
    "        iter_id = int(consolidated_fname.split('/')[-2][-1])\n",
    "        # Store the time of the most recent consolidation event as iteration time.\n",
    "        iter_time = datetime.strptime(consolidated_fname.split('/')[-1], '%Y-%m-%d_%H:%M:%S.json')\n",
    "        if iter_id in consolidation_times:\n",
    "            consolidation_times[iter_id] = max(consolidation_times[iter_id], iter_time)\n",
    "        else:\n",
    "            consolidation_times[iter_id] = iter_time\n",
    "        consolidated_cost_times.append(iter_time)\n",
    "                                      \n",
    "        with open(consolidated_fname, 'r') as f:\n",
    "            consolidated_data = json.load(f)\n",
    "        for consolidation in consolidated_data:\n",
    "            obj_id = consolidation['datasetObjectId']\n",
    "            n_boxes = len(consolidation['consolidatedAnnotation']['content'][\n",
    "                'category']['annotations'])\n",
    "            if obj_id not in obj_ids:\n",
    "                obj_ids.add(obj_id)\n",
    "                consolidated_nims[iter_id] += 1            \n",
    "                consolidated_nboxes[iter_id] += n_boxes\n",
    "            \n",
    "total_human_labels = sum(consolidated_nims.values())\n",
    "            \n",
    "# Count the number of machine iterations in each class each iteration.\n",
    "!aws s3 cp {S3_OUTPUT + '/activelearning'} od_output_data/activelearning --recursive --quiet\n",
    "auto_nboxes = defaultdict(int)\n",
    "auto_nims = defaultdict(int)\n",
    "auto_times = {}\n",
    "auto_cost_times = []\n",
    "\n",
    "for auto_fname in glob.glob('od_output_data/activelearning/**', recursive=True):\n",
    "    if auto_fname.endswith('auto_annotator_output.txt'):\n",
    "        iter_id = int(auto_fname.split('/')[-3])\n",
    "        with open(auto_fname, 'r') as f:\n",
    "            annots = [' '.join(l.split()[1:]) for l in f.readlines()]\n",
    "        auto_nims[iter_id] += len(annots)\n",
    "        for annot in annots:\n",
    "            annot = json.loads(annot)\n",
    "            time_str = annot['category-metadata']['creation-date']\n",
    "            auto_time = datetime.strptime(time_str, '%Y-%m-%dT%H:%M:%S.%f')\n",
    "            n_boxes = len(annot['category']['annotations'])\n",
    "            auto_nboxes[iter_id] += n_boxes\n",
    "            if iter_id in auto_times:\n",
    "                auto_times[iter_id] = max(auto_times[iter_id], auto_time)\n",
    "            else:\n",
    "                auto_times[iter_id] = auto_time\n",
    "            auto_cost_times.append(auto_time)\n",
    "                \n",
    "total_auto_labels = sum(auto_nims.values())\n",
    "n_iters = max(len(auto_times), len(consolidation_times))\n",
    "\n",
    "# Get plots for auto-annotation neural-net training.\n",
    "def get_training_job_data(training_job_name):\n",
    "    logclient = boto3.client('logs')\n",
    "    log_group_name = '/aws/sagemaker/TrainingJobs'\n",
    "    log_stream_name = logclient.describe_log_streams(logGroupName=log_group_name,\n",
    "        logStreamNamePrefix=training_job_name)['logStreams'][0]['logStreamName']\n",
    "    train_log = logclient.get_log_events(\n",
    "        logGroupName=log_group_name,\n",
    "        logStreamName=log_stream_name,\n",
    "        startFromHead=True\n",
    "    )\n",
    "    events = train_log['events']\n",
    "    next_token = train_log['nextForwardToken']\n",
    "    while True:\n",
    "        train_log = logclient.get_log_events(\n",
    "            logGroupName=log_group_name,\n",
    "            logStreamName=log_stream_name,\n",
    "            startFromHead=True,\n",
    "            nextToken=next_token\n",
    "        )\n",
    "        if train_log['nextForwardToken'] == next_token:\n",
    "            break\n",
    "        events = events + train_log['events']\n",
    "        next_token = train_log['nextForwardToken']\n",
    "\n",
    "    mAPs = []\n",
    "    for event in events:\n",
    "        msg = event['message']\n",
    "        if 'Final configuration' in msg:\n",
    "            num_samples = int(msg.split('num_training_samples\\': u\\'')[1].split('\\'')[0])\n",
    "        elif 'validation mAP <score>=(' in msg:\n",
    "            mAPs.append(float(msg.split('validation mAP <score>=(')[1][:-1]))\n",
    "\n",
    "    return num_samples, mAPs\n",
    "\n",
    "training_data = !aws s3 ls {S3_OUTPUT + '/training/'} --recursive\n",
    "training_sizes = []\n",
    "training_mAPs = []\n",
    "training_iters = []\n",
    "for line in training_data:\n",
    "    if line.split('/')[-1] == 'model.tar.gz':\n",
    "        training_job_name = line.split('/')[-3]\n",
    "        print('About to get training job logs')\n",
    "        n_samples, mAPs = get_training_job_data(training_job_name)\n",
    "        training_sizes.append(n_samples)\n",
    "        training_mAPs.append(mAPs)\n",
    "        training_iters.append(int(line.split('/')[-5]))\n",
    "        \n",
    "plt.figure(facecolor='white', figsize=(14, 5), dpi=100)\n",
    "ax = plt.subplot(131)\n",
    "total_human = 0\n",
    "total_auto = 0\n",
    "for iter_id in range(1, n_iters + 1):\n",
    "    cost_human = consolidated_nims[iter_id] * HUMAN_PRICE\n",
    "    cost_auto = auto_nims[iter_id] * AUTO_PRICE\n",
    "    total_human += cost_human\n",
    "    total_auto += cost_auto\n",
    "    \n",
    "    plt.bar(iter_id, cost_human, width=.8, color='C0',\n",
    "            label='human' if iter_id==1 else None)\n",
    "    plt.bar(iter_id, cost_auto, bottom=cost_human,\n",
    "            width=.8, color='C1', label='auto' if iter_id==1 else None)   \n",
    "\n",
    "plt.title('Total annotation costs:\\n\\${:.2f} human, \\${:.2f} auto'.format(\n",
    "    total_human, total_auto))\n",
    "plt.xlabel('Iter')\n",
    "plt.ylabel('Cost in dollars')\n",
    "plt.legend()\n",
    "\n",
    "plt.subplot(132)\n",
    "plt.title('Total annotation counts:\\nHuman: {} ims, {} boxes\\nMachine: {} ims, {} boxes'.format(\n",
    "    sum(consolidated_nims.values()), sum(consolidated_nboxes.values()), sum(auto_nims.values()), sum(auto_nboxes.values())))\n",
    "for iter_id in consolidated_nims.keys():\n",
    "    plt.bar(iter_id, auto_nims[iter_id], color='C1', width=.4, label='ims, auto' if iter_id==1 else None)\n",
    "    plt.bar(iter_id, consolidated_nims[iter_id],\n",
    "            bottom=auto_nims[iter_id], color='C0', width=.4, label='ims, human' if iter_id==1 else None)\n",
    "    plt.bar(iter_id + .4, auto_nboxes[iter_id], color='C1', alpha=.4, width=.4, label='boxes, auto' if iter_id==1 else None)\n",
    "    plt.bar(iter_id + .4, consolidated_nboxes[iter_id],\n",
    "            bottom=auto_nboxes[iter_id], color='C0', width=.4, alpha=.4, label='boxes, human' if iter_id==1 else None)\n",
    "\n",
    "tick_labels_boxes = ['Iter {}, boxes'.format(iter_id + 1) for iter_id in range(n_iters)]\n",
    "tick_labels_images = ['Iter {}, images'.format(iter_id + 1) for iter_id in range(n_iters)]\n",
    "tick_locations_images = np.arange(n_iters) + 1\n",
    "tick_locations_boxes = tick_locations_images + .4\n",
    "tick_labels = np.concatenate([[tick_labels_boxes[idx], tick_labels_images[idx]] for idx in range(n_iters)])\n",
    "tick_locations = np.concatenate([[tick_locations_boxes[idx], tick_locations_images[idx]] for idx in range(n_iters)])\n",
    "plt.xticks(tick_locations, tick_labels, rotation=90)\n",
    "plt.legend()\n",
    "plt.ylabel('Count')\n",
    "\n",
    "if len(training_sizes) > 0:\n",
    "    plt.subplot(133)\n",
    "    plt.title('Active learning training curves')\n",
    "    plt.grid(True)\n",
    "\n",
    "    cmap = plt.get_cmap('coolwarm')\n",
    "    n_all = len(training_sizes)\n",
    "    for iter_id_id, (iter_id, size, mAPs) in enumerate(zip(training_iters, training_sizes, training_mAPs)):\n",
    "        plt.plot(mAPs, label='Iter {}, auto'.format(iter_id + 1), color=cmap(iter_id_id / max(1, (n_all-1))))\n",
    "        plt.legend()\n",
    "\n",
    "    plt.xlabel('Training epoch')\n",
    "    plt.ylabel('Validation mAP')\n",
    "\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion <a class=\"anchor\" id=\"Conclusion\"></a>\n",
    "\n",
    "This marks the conclusion of our sample notebook demonstrating the use of pre-trained models to accelerate labeling jobs. Let's review what we covered.\n",
    "\n",
    "* We gathered a dataset consisting of 2500 images of birds from the Open Images dataset.\n",
    "* We split this dataset into two halves. \n",
    "* We created a labeling job for the first dataset of 1250 images and saw that approximately 48% of the dataset was machine-labeled. \n",
    "* We created a second labeling job for the second dataset, and we specified the machine learning model that was trained during the first labeling job. This time we found that approximately 80% of the dataset was machine-labeled.\n",
    "* As a final benchmark, we re-ran the second labeling job without specifying the pre-trained model. Now we found that approximately 60% of the dataset was machine-labeled. \n",
    "\n",
    "That's it! If we were to acquire a new unlabeled dataset in this domain (e.g., object detection of birds), we could setup another labeling job, and specify the model trained in our second labeling job. The use of pre-trained machine learning models allows you to run labeling jobs in succession, with each job improving from the predictive ability gained through the previous job.  Remember that the capability to use pre-trained models is only available through the SageMaker Ground Truth API."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/prepare_data|sm-ground_truth_pretrained_model_labeling.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/prepare_data|sm-ground_truth_pretrained_model_labeling.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/prepare_data|sm-ground_truth_pretrained_model_labeling.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/prepare_data|sm-ground_truth_pretrained_model_labeling.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/prepare_data|sm-ground_truth_pretrained_model_labeling.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/prepare_data|sm-ground_truth_pretrained_model_labeling.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/prepare_data|sm-ground_truth_pretrained_model_labeling.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/prepare_data|sm-ground_truth_pretrained_model_labeling.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/prepare_data|sm-ground_truth_pretrained_model_labeling.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/prepare_data|sm-ground_truth_pretrained_model_labeling.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/prepare_data|sm-ground_truth_pretrained_model_labeling.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/prepare_data|sm-ground_truth_pretrained_model_labeling.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/prepare_data|sm-ground_truth_pretrained_model_labeling.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/prepare_data|sm-ground_truth_pretrained_model_labeling.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/prepare_data|sm-ground_truth_pretrained_model_labeling.ipynb)\n"
   ]
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   },
   {
    "_defaultOrder": 55,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 56,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 1152,
    "name": "ml.p4de.24xlarge",
    "vcpuNum": 96
   }
  ],
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (Data Science 3.0)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/sagemaker-data-science-310-v1"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}