aws_marketplace/using_model_packages/amazon_augmented_ai_with_aws_marketplace_ml_models/amazon_augmented_ai_with_aws_marketplace_ml_models.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Amazon Augmented AI(A2I) Integrated with AWS Marketplace ML Models"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n",
    "\n",
    "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/aws_marketplace|using_model_packages|amazon_augmented_ai_with_aws_marketplace_ml_models|amazon_augmented_ai_with_aws_marketplace_ml_models.ipynb)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes, for some payloads, machine learning (ML) model predictions are just not confident enough and you want more than a machine. Furthermore, training a model can be complicated, time-consuming, and expensive. This is where [AWS Marketplace](https://aws.amazon.com/marketplace/b/6297422012?page=1&filters=FulfillmentOptionType&FulfillmentOptionType=SageMaker&ref_=sa_campaign_pbrao) and [Amazon Augmented AI](https://aws.amazon.com/augmented-ai/) (Amazon A2I) come in. By combining a pretrained ML model in AWS Marketplace with Amazon Augmented AI, you can quickly reap the benefits of pretrained models with validating and augmenting the model's accuracy with human intelligence.\n",
    "\n",
    "AWS Marketplace contains over 400 pretrained ML models. Some models are general purpose. For example, the [GluonCV SSD Object Detector](https://aws.amazon.com/marketplace/pp/prodview-ggbuxlnrm2lh4?qid=1605041213915&sr=0-5&ref_=sa_campaign_pbrao) can detect objects in an image and place bounding boxes around the objects. AWS Marketplace also offers many purpose-built models such as a [Background Noise Classifier](https://aws.amazon.com/marketplace/pp/prodview-vpd6qdjm4d7u4?applicationId=AWS-Sagemaker-Console&ref_=sa_campaign_pbrao), a [Hard Hat Detector for Worker Safety](https://aws.amazon.com/marketplace/pp/prodview-jd5tj2egpxxum?applicationId=AWS-Sagemaker-Console&ref_=sa_campaign_pbrao), and a [Person in Water](https://aws.amazon.com/marketplace/pp/prodview-wlndemzv5pxhw?applicationId=AWS-Sagemaker-Console&ref_=sa_campaign_pbrao).\n",
    "\n",
    "Amazon A2I provides a human-in-loop workflow to review ML predictions. Its configurable human-review workflow solution and customizable user-review console enable you to focus on ML tasks and increase the accuracy of the predictions with human input."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview\n",
    "\n",
    "In this notebook, you will use a pre-trained AWS Marketplace machine learning model with Amazon A2I to detect images as well as trigger a human-in-loop workflow to review, update and add additional labeled objects to an individual image. Furthermore, you can specify configurable threshold criteria for triggering the human-in-loop workflow in Amazon A2I. For example, you can trigger a human-in-loop workflow if there are no objects that are detected with an accuracy of 90% or greater.\n",
    "\n",
    "The following diagram shows the AWS services that are used in this notebook and the steps that you will perform. Here are the high level steps in this notebook:\n",
    "1.\tConfigure the human-in-loop review using Amazon A2I\n",
    "1.\tSelect, deploy, and invoke an AWS Marketplace ML model\n",
    "1.\tTrigger the human review workflow in Amazon A2I.\n",
    "1.\tThe private workforce that was created in Amazon SageMaker Ground Truth reviews and edits the objects detected in the image.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img style=\"float: center;\" src=\"./img/a2i_diagram.png\" width=\"700\" height=\"500\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contents\n",
    "\n",
    "* [Prerequisites](#Prerequisites)\n",
    "* [Step 1 Configure Amazon A2I service](#step1)\n",
    "    * [Step 1.1 Creating human review Workteam or Workforce](#step1_1)\n",
    "    * [Step 1.2 Create Human Task UI](#step1_2)\n",
    "    * [Step 1.3 Create the Flow Definition](#step1_3)\n",
    "* [Step 2 Deploy and invoke AWS Marketplace model](#step2)\n",
    "    * [Step 2.1 Create an endpoint](#step2_1)\n",
    "    * [Step 2.2 Create input payload](#step2_2)\n",
    "    * [Step 2.3 Perform real-time inference](#step2_3)\n",
    "* [Step3 Starting Human Loops](#step3)\n",
    "    * [Step 3.1 View Task Results](#step3_1)\n",
    "* [Step 4 Next steps](#step4)\n",
    "    * [Step 4.1 Additional resources](#step4_1)\n",
    "* [Step 5 Cleanup Resources](#step5)\n",
    "\n",
    "### Usage instructions\n",
    "You can run this notebook one cell at a time (By using Shift+Enter for running a cell)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequisites <a class=\"anchor\" id=\"prerequisites\"></a>\n",
    "\n",
    "This sample notebook requires a subscription to **[GluonCV SSD Object Detector](https://aws.amazon.com/marketplace/pp/prodview-ggbuxlnrm2lh4?ref_=sa_campaign_pbrao)**, a pre-trained machine learning model package from AWS Marketplace. \n",
    "If your AWS account has not been subscribed to this listing, here is the process you can follow: \n",
    "1. Open the [listing](https://aws.amazon.com/marketplace/pp/prodview-ggbuxlnrm2lh4?ref_=sa_campaign_pbrao) from AWS Marketplace\n",
    "1. Read the Highlights section and then product overview section of the listing.\n",
    "1. View usage information and then additional resources.\n",
    "1. Note the supported instance types.\n",
    "1. Next, click on **Continue to subscribe.**\n",
    "1. Review End-user license agreement, support terms, as well as pricing information.\n",
    "1. The **Accept Offer** button needs to be selected if your organization agrees with EULA, pricing information as well as support terms.  If the Continue to configuration button is active, it means your account already has a subscription to this listing. Once you select the **Continue to configuration** button and then choose **region**, you will see that a Product Arn will appear. This is the **model package ARN** that you need to specify in the following cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_package_arn = \"arn:aws:sagemaker:us-east-1:865070037744:model-package/gluoncv-ssd-resnet501547760463-0f9e6796d2438a1d64bb9b15aac57bc0\"  # Update as needed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "8. This notebook requires the IAM role associated with this notebook to have *AmazonSageMakerFullAccess* IAM permission.\n",
    "\n",
    "8. Note: If you want to run this notebook on AWS SageMaker Studio - please use Classic Jupyter mode to be able correctly render visualization. Pick instance type **'ml.m4.xlarge'** or larger. Set kernel to **'Data Science'**.\n",
    "\n",
    "    <img style=\"float: left;\" src=\"./img/classicjupyter.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Installing Dependencies\n",
    "\n",
    "Import the libraries that are needed for this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import necessary libraries\n",
    "import boto3\n",
    "import json\n",
    "import pandas as pd\n",
    "import pprint\n",
    "import requests\n",
    "import sagemaker\n",
    "import shutil\n",
    "import time\n",
    "import uuid\n",
    "import PIL.Image\n",
    "from IPython.display import Image\n",
    "from IPython.display import Markdown as md\n",
    "from sagemaker import get_execution_role\n",
    "from sagemaker import ModelPackage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup Variables, Bucket and Paths"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Setting Role to the default SageMaker Execution Role\n",
    "role = get_execution_role()\n",
    "\n",
    "# Instantiate the SageMaker session and client that will be used throughout the notebook\n",
    "sagemaker_session = sagemaker.Session()\n",
    "sagemaker_client = sagemaker_session.sagemaker_client\n",
    "\n",
    "# Fetch the region\n",
    "region = sagemaker_session.boto_region_name\n",
    "\n",
    "# Create S3 and A2I clients\n",
    "s3 = boto3.client(\"s3\", region)\n",
    "a2i = boto3.client(\"sagemaker-a2i-runtime\", region)\n",
    "\n",
    "# Retrieve the current timestamp\n",
    "timestamp = time.strftime(\"%Y-%m-%d-%H-%M-%S\", time.gmtime())\n",
    "\n",
    "# endpoint_name = '<ENDPOINT_NAME>'\n",
    "endpoint_name = \"gluoncv-object-detector\"\n",
    "\n",
    "# content_type='<CONTENT_TYPE>'\n",
    "content_type = \"image/jpeg\"\n",
    "\n",
    "# Instance size type to be used for making real-time predictions\n",
    "real_time_inference_instance_type = \"ml.m4.xlarge\"\n",
    "\n",
    "# Task UI name - this value is unique per account and region. You can also provide your own value here.\n",
    "# task_ui_name = '<TASK_UI_NAME>'\n",
    "task_ui_name = \"ui-aws-marketplace-gluon-model-\" + timestamp\n",
    "\n",
    "# Flow definition name - this value is unique per account and region. You can also provide your own value here.\n",
    "flow_definition_name = \"fd-aws-marketplace-gluon-model-\" + timestamp\n",
    "\n",
    "# Name of the image file that will be used in object detection\n",
    "image_file_name = \"image.jpg\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the sub-directory in the default S3 bucket\n",
    "# that will store the results of the human-in-loop A2I review\n",
    "bucket = sagemaker_session.default_bucket()\n",
    "key = \"a2i-results\"\n",
    "s3.put_object(Bucket=bucket, Key=(key + \"/\"))\n",
    "output_path = f\"s3://{bucket}/a2i-results\"\n",
    "print(f\"Results of A2I will be stored in {output_path}.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1 Configure Amazon A2I service<a class=\"anchor\" id=\"step1\"></a>\n",
    "In this section, you will create 3 resources:\n",
    "1. Private workforce\n",
    "2. Human-in-loop Console UI\n",
    "3. Workflow definition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1.1 Creating human review Workteam or Workforce <a class=\"anchor\" id=\"step1_1\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you have already created a private work team, replace <WORKTEAM_ARN> with the ARN of your work team. If you have never created a private work team, use the instructions below to create one. To learn more about using and managing private work teams, see [Use a Private Workforce](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-workforce-private.html)).\n",
    "\n",
    "1. In the Amazon SageMaker console in the left sidebar under the Ground Truth heading, open the **Labeling Workforces**.\n",
    "1. Choose **Private**, and then choose **Create private team**.\n",
    "1. If you are a new user of SageMaker workforces, it is recommended you select **Create a private work team with AWS Cognito**.\n",
    "1. For team name, enter \"MyTeam\".\n",
    "1. To add workers by email, select **Invite new workers by email** and paste or type a list of up to 50 email addresses, separated by commas, into the email addresses box. If you are following this notebook, specify an email account that you have access to. The system sends an invitation email, which allows users to authenticate and set up their profile for performing human-in-loop review.\n",
    "1. Enter an organization name - this will be used to customize emails sent to your workers.\n",
    "1. For contact email, enter an email address you have access to.\n",
    "1. Select **Create private team**.\n",
    "\n",
    "This will bring you back to the Private tab under labeling workforces, where you can view and manage your private teams and workers.\n",
    "\n",
    "**IMPORTANT: After you have created your workteam, from the Team summary section copy the value of the ARN and uncomment and replace `<WORKTEAM_ARN>` below:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# workteam_arn = '<WORKTEAM_ARN>'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1.2 Create Human Task UI <a class=\"anchor\" id=\"step1_2\"></a>\n",
    "\n",
    "Create a human task UI resource, giving a UI template in liquid HTML. This template will be rendered to the human workers whenever human loop is required.\n",
    "\n",
    "For additional UI templates, check out this repository:  https://github.com/aws-samples/amazon-a2i-sample-task-uis.\n",
    "\n",
    "You will be using a slightly modified version of the [object detection UI](https://github.com/aws-samples/amazon-a2i-sample-task-uis/blob/master/images/bounding-box.liquid.html) that provides support for the `initial-value` and `labels` variables in the template."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create task UI\n",
    "\n",
    "# Read in the template from a local file\n",
    "template = open(\"./src/worker-task-template.html\").read()\n",
    "\n",
    "human_task_ui_response = sagemaker_client.create_human_task_ui(\n",
    "    HumanTaskUiName=task_ui_name, UiTemplate={\"Content\": template}\n",
    ")\n",
    "\n",
    "human_task_ui_arn = human_task_ui_response[\"HumanTaskUiArn\"]\n",
    "print(human_task_ui_arn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1.3 Create the Flow Definition <a class=\"anchor\" id=\"step1_3\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, you will create a flow definition. Flow Definitions allow you to specify:\n",
    "\n",
    "* The workforce that your tasks will be sent to.\n",
    "* The instructions that your workforce will receive. This is called a worker task template.\n",
    "* The configuration of your worker tasks, including the number of workers that receive a task and time limits to complete tasks.\n",
    "* Where your output data will be stored.\n",
    "\n",
    "For more details and instructions, see: https://docs.aws.amazon.com/sagemaker/latest/dg/a2i-create-flow-definition.html."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "create_workflow_definition_response = sagemaker_client.create_flow_definition(\n",
    "    FlowDefinitionName=flow_definition_name,\n",
    "    RoleArn=role,\n",
    "    HumanLoopConfig={\n",
    "        \"WorkteamArn\": workteam_arn,\n",
    "        \"HumanTaskUiArn\": human_task_ui_arn,\n",
    "        \"TaskCount\": 1,\n",
    "        \"TaskDescription\": \"Identify and locate the object in an image.\",\n",
    "        \"TaskTitle\": \"Object detection Amazon A2I demo\",\n",
    "    },\n",
    "    OutputConfig={\"S3OutputPath\": output_path},\n",
    ")\n",
    "flow_definition_arn = create_workflow_definition_response[\n",
    "    \"FlowDefinitionArn\"\n",
    "]  # let's save this ARN for future use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "# Describe flow definition - status should be active\n",
    "for x in range(60):\n",
    "    describe_flow_definition_response = sagemaker_client.describe_flow_definition(\n",
    "        FlowDefinitionName=flow_definition_name\n",
    "    )\n",
    "    print(describe_flow_definition_response[\"FlowDefinitionStatus\"])\n",
    "    if describe_flow_definition_response[\"FlowDefinitionStatus\"] == \"Active\":\n",
    "        print(\"Flow Definition is active\")\n",
    "        break\n",
    "    time.sleep(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2 Deploy and invoke AWS Marketplace model <a class=\"anchor\" id=\"step2\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, you will stand up an Amazon SageMaker endpoint. Each endpoint must have a unique name which you can use for performing inference. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2.1 Create an Endpoint <a class=\"anchor\" id=\"step2_1\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "# Create a deployable model from the model package.\n",
    "model = ModelPackage(\n",
    "    role=role,\n",
    "    model_package_arn=model_package_arn,\n",
    "    sagemaker_session=sagemaker_session,\n",
    "    predictor_cls=sagemaker.predictor.Predictor,\n",
    ")\n",
    "\n",
    "# Deploy the model\n",
    "predictor = model.deploy(\n",
    "    initial_instance_count=1,\n",
    "    instance_type=real_time_inference_instance_type,\n",
    "    endpoint_name=endpoint_name,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It will take anywhere between 5 to 10 minutes to create the endpoint. Once the endpoint has been created, you would be able to perform real-time inference."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2.2 Create input payload <a class=\"anchor\" id=\"step2_2\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this step, you will prepare a payload to perform a prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download the image file\n",
    "# Open the url image, set stream to True, this will return the stream content.\n",
    "r = requests.get(\"https://images.pexels.com/photos/763398/pexels-photo-763398.jpeg\", stream=True)\n",
    "\n",
    "# Open a local file with wb ( write binary ) permission to save it locally.\n",
    "with open(image_file_name, \"wb\") as f:\n",
    "    shutil.copyfileobj(r.raw, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Resize the image and upload the file to S3 so that the image can be referenced from the worker console UI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the image\n",
    "image = PIL.Image.open(image_file_name)\n",
    "# Resize the image\n",
    "resized_image = image.resize((600, 400))\n",
    "\n",
    "# Save the resized image file locally\n",
    "resized_image.save(image_file_name)\n",
    "\n",
    "# Save file to S3\n",
    "s3 = boto3.client(\"s3\")\n",
    "with open(image_file_name, \"rb\") as f:\n",
    "    s3.upload_fileobj(f, bucket, image_file_name)\n",
    "\n",
    "# Display the image\n",
    "from IPython.core.display import Image, display\n",
    "\n",
    "Image(filename=image_file_name, width=600, height=400)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2.3 Perform real-time inference <a class=\"anchor\" id=\"step2_3\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Submit the image file to the model and it will detect the objects in the image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(image_file_name, \"rb\") as f:\n",
    "    payload = f.read()\n",
    "\n",
    "response = sagemaker_session.sagemaker_runtime_client.invoke_endpoint(\n",
    "    EndpointName=endpoint_name, ContentType=content_type, Accept=\"json\", Body=payload\n",
    ")\n",
    "\n",
    "result = json.loads(response[\"Body\"].read().decode())\n",
    "\n",
    "# Convert list to JSON\n",
    "json_result = json.dumps(result)\n",
    "df = pd.read_json(json_result)\n",
    "\n",
    "# Display confidence scores < 0.90\n",
    "df = df[df.score < 0.90]\n",
    "print(df.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3 Starting Human Loops <a class=\"anchor\" id=\"step3\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In a previous step, you have already submitted your image to the model for prediction and stored the output in JSON format in the `result` variable. You simply need to modify the X, Y coordinates of the bounding boxes. Additionally, you can filter out all predictions that are less than 90% accurate before submitting it to your human-in-loop review. This will insure that your model's predictions are highly accurate and any additional detections of objects will be made by a human."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Helper function to update X,Y coordinates and labels for the bounding boxes\n",
    "def fix_boundingboxes(prediction_results, threshold=0.8):\n",
    "\n",
    "    bounding_boxes = []\n",
    "    labels = set()\n",
    "\n",
    "    for data in prediction_results:\n",
    "        label = data[\"id\"]\n",
    "        labels.add(label)\n",
    "\n",
    "        if data[\"score\"] > threshold:\n",
    "            width = data[\"right\"] - data[\"left\"]\n",
    "            height = data[\"bottom\"] - data[\"top\"]\n",
    "            top = data[\"top\"]\n",
    "            left = data[\"left\"]\n",
    "            bounding_boxes.append(\n",
    "                {\"height\": height, \"width\": width, \"top\": top, \"left\": left, \"label\": label}\n",
    "            )\n",
    "\n",
    "    return bounding_boxes, list(labels)\n",
    "\n",
    "\n",
    "bounding_boxes, labels = fix_boundingboxes(result, threshold=0.9)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the content that is passed into the human-in-loop workflow and console\n",
    "human_loop_name = str(uuid.uuid4())\n",
    "input_content = {\n",
    "    \"initialValue\": bounding_boxes,  # the bounding box values that have been detected by model prediction\n",
    "    \"taskObject\": f\"s3://{bucket}/\"\n",
    "    + image_file_name,  # the s3 object will be passed to the worker task UI to render\n",
    "    \"labels\": labels,  # the labels that are displayed in the legend\n",
    "}\n",
    "\n",
    "# Trigger the human-in-loop workflow\n",
    "start_loop_response = a2i.start_human_loop(\n",
    "    HumanLoopName=human_loop_name,\n",
    "    FlowDefinitionArn=flow_definition_arn,\n",
    "    HumanLoopInput={\"InputContent\": json.dumps(input_content)},\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the human-in-loop review has been triggered, you can log into the worker console to work on the task and make edits and additions to the object detection bounding boxes from the image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fetch the URL for the worker console UI\n",
    "workteam_name = workteam_arn.split(\"/\")[-1]\n",
    "my_workteam = sagemaker_session.sagemaker_client.list_workteams(NameContains=workteam_name)\n",
    "worker_console_url = \"https://\" + my_workteam[\"Workteams\"][0][\"SubDomain\"]\n",
    "\n",
    "md(\n",
    "    \"### Click on the [Worker Console]({}) to begin reviewing the object detection\".format(\n",
    "        worker_console_url\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The below image shows the objects that were detected for the sample image that was used in this notebook by your model and displayed in the worker console. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./img/rain_biker_bb.png' align='center' height=600 width=800/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can now make edits to the image to detect other objects. For example, in the image above, the model failed to detect the bicycle in the foreground with an accuracy of 90% or greater. However, as a human reviewer, you can clearly see the bicycle and can make a bounding box around it. Once you have finished with your edits, you can submit the result."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3.1 View Task Results <a class=\"anchor\" id=\"step3_1\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once work is completed, Amazon A2I stores results in your S3 bucket and sends a Cloudwatch event. Your results should be available in the S3 `output_path` that you specified when all work is completed. Note that the human answer, the label and the bounding box, is returned and saved in the JSON file.\n",
    "\n",
    "**NOTE: You must edit/submit the image in the Worker console so that its status is `Completed`.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fetch the details about the human loop review in order to locate the JSON output on S3\n",
    "resp = a2i.describe_human_loop(HumanLoopName=human_loop_name)\n",
    "\n",
    "# Wait for the human-in-loop review to be completed\n",
    "while True:\n",
    "    resp = a2i.describe_human_loop(HumanLoopName=human_loop_name)\n",
    "    print(\"-\", sep=\"\", end=\"\", flush=True)\n",
    "    if resp[\"HumanLoopStatus\"] == \"Completed\":\n",
    "        print(\"!\")\n",
    "        break\n",
    "    time.sleep(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once its status is `Completed`, you can execute the below cell to view the JSON output that is stored in S3. Under `annotatedResult`, any new bounding boxes will be included along with those that the model predicted, will be included. To learn more about the output data schema, please refer to the documentation about [Output Data From Custom Task Types](https://docs.aws.amazon.com/sagemaker/latest/dg/a2i-output-data.html#sms-output-data-custom)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Once the image has been submitted, display the JSON output that was sent to S3\n",
    "\n",
    "bucket, key = resp[\"HumanLoopOutput\"][\"OutputS3Uri\"].replace(\"s3://\", \"\").split(\"/\", 1)\n",
    "response = s3.get_object(Bucket=bucket, Key=key)\n",
    "\n",
    "content = response[\"Body\"].read()\n",
    "json_output = json.loads(content)\n",
    "\n",
    "print(json.dumps(json_output, indent=1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4 Next Steps <a class=\"anchor\" id=\"step4\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4.1 Additional Resources <a class=\"anchor\" id=\"step4_1\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* You can explore additional machine learning models in [AWS Marketplace - Machine Learning](https://aws.amazon.com/marketplace/b/c3714653-8485-4e34-b35b-82c2203e81c1?page=1&filters=FulfillmentOptionType&FulfillmentOptionType=SageMaker&ref_=sa_campaign_pbrao). \n",
    "* Learn more about [Amazon Augmented AI](https://aws.amazon.com/augmented-ai/)\n",
    "* Other AWS blogs that may be of interest are:\n",
    "    * [Using AWS Marketplace for machine learning workloads](https://aws.amazon.com/blogs/awsmarketplace/using-aws-marketplace-for-machine-learning-workloads/)\n",
    "    * [Adding AI to your applications with ready-to-use models from AWS Marketplace](https://aws.amazon.com/blogs/machine-learning/adding-ai-to-your-applications-with-ready-to-use-models-from-aws-marketplace/)\n",
    "    * [Building an end-to-end intelligent document processing solution using AWS](https://aws.amazon.com/blogs/machine-learning/building-an-end-to-end-intelligent-document-processing-solution-using-aws/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5 Clean up resources <a class=\"anchor\" id=\"step5\"></a>\n",
    "In order to clean up the resources from this notebook,simply execute the below cells."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Delete Workflow definition\n",
    "sagemaker_client.delete_flow_definition(FlowDefinitionName=flow_definition_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Delete Human Task UI\n",
    "sagemaker_client.delete_human_task_ui(HumanTaskUiName=task_ui_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Delete Endpoint\n",
    "predictor.delete_endpoint()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Delete Model\n",
    "predictor.delete_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Cancel AWS Marketplace subscription (Optional)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, if you subscribed to AWS Marketplace model for an experiment and would like to unsubscribe, you can follow the steps below. Before you cancel the subscription, ensure that you do not have any [deployable model](https://console.aws.amazon.com/sagemaker/home#/models) created from the model package or using the algorithm. Note - You can find this information by looking at the container name associated with the model.\n",
    "\n",
    "**Steps to unsubscribe from the product on AWS Marketplace:**\n",
    "\n",
    "Navigate to Machine Learning tab on Your [Software subscriptions page](https://aws.amazon.com/marketplace/ai/library?productType=ml&ref_=lbr_tab_ml).\n",
    "Locate the listing that you would need to cancel, and click Cancel Subscription."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notebook CI Test Results\n",
    "\n",
    "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n",
    "\n",
    "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/aws_marketplace|using_model_packages|amazon_augmented_ai_with_aws_marketplace_ml_models|amazon_augmented_ai_with_aws_marketplace_ml_models.ipynb)\n",
    "\n",
    "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/aws_marketplace|using_model_packages|amazon_augmented_ai_with_aws_marketplace_ml_models|amazon_augmented_ai_with_aws_marketplace_ml_models.ipynb)\n",
    "\n",
    "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/aws_marketplace|using_model_packages|amazon_augmented_ai_with_aws_marketplace_ml_models|amazon_augmented_ai_with_aws_marketplace_ml_models.ipynb)\n",
    "\n",
    "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/aws_marketplace|using_model_packages|amazon_augmented_ai_with_aws_marketplace_ml_models|amazon_augmented_ai_with_aws_marketplace_ml_models.ipynb)\n",
    "\n",
    "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/aws_marketplace|using_model_packages|amazon_augmented_ai_with_aws_marketplace_ml_models|amazon_augmented_ai_with_aws_marketplace_ml_models.ipynb)\n",
    "\n",
    "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/aws_marketplace|using_model_packages|amazon_augmented_ai_with_aws_marketplace_ml_models|amazon_augmented_ai_with_aws_marketplace_ml_models.ipynb)\n",
    "\n",
    "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/aws_marketplace|using_model_packages|amazon_augmented_ai_with_aws_marketplace_ml_models|amazon_augmented_ai_with_aws_marketplace_ml_models.ipynb)\n",
    "\n",
    "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/aws_marketplace|using_model_packages|amazon_augmented_ai_with_aws_marketplace_ml_models|amazon_augmented_ai_with_aws_marketplace_ml_models.ipynb)\n",
    "\n",
    "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/aws_marketplace|using_model_packages|amazon_augmented_ai_with_aws_marketplace_ml_models|amazon_augmented_ai_with_aws_marketplace_ml_models.ipynb)\n",
    "\n",
    "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/aws_marketplace|using_model_packages|amazon_augmented_ai_with_aws_marketplace_ml_models|amazon_augmented_ai_with_aws_marketplace_ml_models.ipynb)\n",
    "\n",
    "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/aws_marketplace|using_model_packages|amazon_augmented_ai_with_aws_marketplace_ml_models|amazon_augmented_ai_with_aws_marketplace_ml_models.ipynb)\n",
    "\n",
    "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/aws_marketplace|using_model_packages|amazon_augmented_ai_with_aws_marketplace_ml_models|amazon_augmented_ai_with_aws_marketplace_ml_models.ipynb)\n",
    "\n",
    "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/aws_marketplace|using_model_packages|amazon_augmented_ai_with_aws_marketplace_ml_models|amazon_augmented_ai_with_aws_marketplace_ml_models.ipynb)\n",
    "\n",
    "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/aws_marketplace|using_model_packages|amazon_augmented_ai_with_aws_marketplace_ml_models|amazon_augmented_ai_with_aws_marketplace_ml_models.ipynb)\n",
    "\n",
    "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/aws_marketplace|using_model_packages|amazon_augmented_ai_with_aws_marketplace_ml_models|amazon_augmented_ai_with_aws_marketplace_ml_models.ipynb)\n"
   ]
  }
 ],
 "metadata": {
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (Data Science)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/datascience-1.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}