Amazon Augmented AI (A2I) and Comprehend DetectSentiment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Amazon Augmented AI (Amazon A2I) integration with Amazon Comprehend [Example]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visit https://github.com/aws-samples/amazon-a2i-sample-jupyter-notebooks for all A2I Sample Notebooks\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. [Introduction](#Introduction)\n",
    "2. [Prerequisites](#Prerequisites)\n",
    "    2. [Workteam](#Workteam)\n",
    "    3. [Permissions](#Notebook-Permission)\n",
    "3. [Client Setup](#Client-Setup)\n",
    "4. [Create Control Plane Resources](#Create-Control-Plane-Resources)\n",
    "    1. [Create Human Task UI](#Create-Human-Task-UI)\n",
    "    2. [Create Flow Definition](#Create-Flow-Definition)\n",
    "5. [Starting Human Loops](#Scenario-1-:-When-Activation-Conditions-are-met-,-and-HumanLoop-is-created)\n",
    "    1. [Wait For Workers to Complete Task](#Wait-For-Workers-to-Complete-Task)\n",
    "    2. [Check Status of Human Loop](#Check-Status-of-Human-Loop)\n",
    "    3. [View Task Results](#View-Task-Results)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "Amazon Augmented AI (Amazon A2I) makes it easy to build the workflows required for human review of ML predictions. Amazon A2I brings human review to all developers, removing the undifferentiated heavy lifting associated with building human review systems or managing large numbers of human reviewers. \n",
    "\n",
    "You can create your own workflows for ML models built on Amazon SageMaker or any other tools. Using Amazon A2I, you can allow human reviewers to step in when a model is unable to make a high confidence prediction or to audit its predictions on an on-going basis. \n",
    "\n",
    "Learn more here: https://aws.amazon.com/augmented-ai/\n",
    "\n",
    "In this tutorial, we will show how you can use **Amazon A2I with AWS Comprehend's Detect Sentiment API.**\n",
    "\n",
    "For more in depth instructions, visit https://docs.aws.amazon.com/sagemaker/latest/dg/a2i-getting-started.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To incorporate Amazon A2I into your human review workflows, you need three resources:\n",
    "\n",
    "* A **worker task template** to create a worker UI. The worker UI displays your input data, such as documents or images, and instructions to workers. It also provides interactive tools that the worker uses to complete your tasks. For more information, see https://docs.aws.amazon.com/sagemaker/latest/dg/a2i-instructions-overview.html\n",
    "\n",
    "* A **human review workflow**, also referred to as a flow definition. You use the flow definition to configure your human workforce and provide information about how to accomplish the human review task. You can create a flow definition in the Amazon Augmented AI console or with Amazon A2I APIs. To learn more about both of these options, see https://docs.aws.amazon.com/sagemaker/latest/dg/a2i-create-flow-definition.html\n",
    "\n",
    "* A **human loop** to start your human review workflow. When you use one of the built-in task types, the corresponding AWS service creates and starts a human loop on your behalf when the conditions specified in your flow definition are met or for each object if no conditions were specified. When a human loop is triggered, human review tasks are sent to the workers as specified in the flow definition.\n",
    "\n",
    "When using a custom task type, as this tutorial will show, you start a human loop using the Amazon Augmented AI Runtime API. When you call StartHumanLoop in your custom application, a task is sent to human reviewers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Install Latest SDK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First, let's get the latest installations of our dependencies\n",
    "!pip install --upgrade pip\n",
    "!pip install boto3 --upgrade\n",
    "!pip install -U botocore"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "We need to set up the following data:\n",
    "* `region` - Region to call A2I\n",
    "* `bucket` - A S3 bucket accessible by the given role\n",
    "    * Used to store the sample images & output results\n",
    "    * Must be within the same region A2I is called from\n",
    "* `role` - The IAM role used as part of StartHumanLoop. By default, this notebook will use the execution role\n",
    "* `workteam` - Group of people to send the work to"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Region\n",
    "REGION = '<REGION>'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Setup Bucket and Paths"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "import botocore\n",
    "\n",
    "BUCKET = '<YOUR_BUCKET>'\n",
    "OUTPUT_PATH = f's3://{BUCKET}/a2i-results'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Role and Permissions\n",
    "\n",
    "The AWS IAM Role used to execute the notebook needs to have the following permissions:\n",
    "\n",
    "* ComprehendFullAccess\n",
    "* SagemakerFullAccess\n",
    "* S3 Read/Write Access to the BUCKET listed above\n",
    "* AmazonSageMakerMechanicalTurkAccess (if using MechanicalTurk as your Workforce)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker import get_execution_role\n",
    "\n",
    "# Setting Role to the default SageMaker Execution Role\n",
    "ROLE = get_execution_role()\n",
    "display(ROLE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Workteam or Workforce"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A workforce is the group of workers that you have selected to label your dataset. You can choose either the Amazon Mechanical Turk workforce, a vendor-managed workforce, or you can create your own private workforce for human reviews. Whichever workforce type you choose, Amazon Augmented AI takes care of sending tasks to workers. \n",
    "\n",
    "When you use a private workforce, you also create work teams, a group of workers from your workforce that are assigned to Amazon Augmented AI human review tasks. You can have multiple work teams and can assign one or more work teams to each job."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To create your Workteam, visit the instructions here: https://docs.aws.amazon.com/sagemaker/latest/dg/sms-workforce-management.html\n",
    "\n",
    "After you have created your workteam, replace YOUR_WORKTEAM_ARN below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "WORKTEAM_ARN= \"<YOUR_WORKTEAM>\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visit: https://docs.aws.amazon.com/sagemaker/latest/dg/a2i-permissions-security.html to add the necessary permissions to your role"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Client Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we are going to setup the rest of our clients. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import io\n",
    "import json\n",
    "import uuid\n",
    "import time\n",
    "import boto3\n",
    "import botocore\n",
    "\n",
    "# Amazon SageMaker client\n",
    "sagemaker = boto3.client('sagemaker', REGION)\n",
    "\n",
    "# Amazon Comprehend client\n",
    "comprehend = boto3.client('comprehend', REGION)\n",
    "\n",
    "# Amazon Augment AI (A2I) client\n",
    "a2i = boto3.client('sagemaker-a2i-runtime')\n",
    "\n",
    "s3 = boto3.client('s3', REGION)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comprehend helper method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Will help us parse Detect Sentiment API responses \n",
    "def capsToCamel(all_caps_string):\n",
    "    if all_caps_string == 'POSITIVE':\n",
    "        return 'Positive'\n",
    "    elif all_caps_string == 'NEGATIVE': \n",
    "        return 'Negative'\n",
    "    elif all_caps_string == 'NEUTRAL':\n",
    "        return 'Neutral'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Control Plane Resources"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Human Task UI\n",
    "\n",
    "Create a human task UI resource, giving a UI template in liquid html. This template will be rendered to the human workers whenever human loop is required.\n",
    "\n",
    "Below we've provided a simple demo template that is compatible with AWS Comprehend's Detect Sentiment API input and response.\n",
    "\n",
    "For over 70 pre built UIs, check: https://github.com/aws-samples/amazon-a2i-sample-task-uis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "template = r\"\"\"\n",
    "<script src=\"https://assets.crowd.aws/crowd-html-elements.js\"></script>\n",
    "\n",
    "<crowd-form>\n",
    "    <crowd-classifier\n",
    "      name=\"sentiment\"\n",
    "      categories=\"['Positive', 'Negative', 'Neutral', 'Mixed']\"\n",
    "      initial-value=\"{{ task.input.initialValue }}\"\n",
    "      header=\"What sentiment does this text convey?\"\n",
    "    >\n",
    "      <classification-target>\n",
    "        {{ task.input.taskObject }}\n",
    "      </classification-target>\n",
    "      \n",
    "      <full-instructions header=\"Sentiment Analysis Instructions\">\n",
    "        <p><strong>Positive</strong> sentiment include: joy, excitement, delight</p>\n",
    "        <p><strong>Negative</strong> sentiment include: anger, sarcasm, anxiety</p>\n",
    "        <p><strong>Neutral</strong>: neither positive or negative, such as stating a fact</p>\n",
    "        <p><strong>Mixed</strong>: when the sentiment is mixed</p>\n",
    "      </full-instructions>\n",
    "\n",
    "      <short-instructions>\n",
    "       Choose the primary sentiment that is expressed by the text. \n",
    "      </short-instructions>\n",
    "    </crowd-classifier>\n",
    "</crowd-form>\n",
    "\"\"\"\n",
    "\n",
    "def create_task_ui():\n",
    "    '''\n",
    "    Creates a Human Task UI resource.\n",
    "\n",
    "    Returns:\n",
    "    struct: HumanTaskUiArn\n",
    "    '''\n",
    "    response = sagemaker.create_human_task_ui(\n",
    "        HumanTaskUiName=taskUIName,\n",
    "        UiTemplate={'Content': template})\n",
    "    return response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Task UI name - this value is unique per account and region. You can also provide your own value here.\n",
    "taskUIName = 'ui-comprehend-' + str(uuid.uuid4()) \n",
    "\n",
    "# Create task UI\n",
    "humanTaskUiResponse = create_task_ui()\n",
    "humanTaskUiArn = humanTaskUiResponse['HumanTaskUiArn']\n",
    "print(humanTaskUiArn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating the Flow Definition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, we're going to create a flow definition definition. Flow Definitions allow us to specify:\n",
    "\n",
    "* The workforce that your tasks will be sent to.\n",
    "* The instructions that your workforce will receive. This is called a worker task template.\n",
    "* The configuration of your worker tasks, including the number of workers that receive a task and time limits to complete tasks.\n",
    "* Where your output data will be stored.\n",
    "\n",
    "This demo is going to use the API, but you can optionally create this workflow definition in the console as well. \n",
    "\n",
    "For more details and instructions, see: https://docs.aws.amazon.com/sagemaker/latest/dg/a2i-create-flow-definition.html."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Flow definition name - this value is unique per account and region. You can also provide your own value here.\n",
    "flowDefinitionName = 'fd-comprehend-demo-' + str(uuid.uuid4()) \n",
    "\n",
    "create_workflow_definition_response = sagemaker.create_flow_definition(\n",
    "        FlowDefinitionName= flowDefinitionName,\n",
    "        RoleArn= ROLE,\n",
    "        HumanLoopConfig= {\n",
    "            \"WorkteamArn\": WORKTEAM_ARN,\n",
    "            \"HumanTaskUiArn\": humanTaskUiArn,\n",
    "            \"TaskCount\": 1,\n",
    "            \"TaskDescription\": \"Identify the sentiment of the provided text\",\n",
    "            \"TaskTitle\": \"Detect Sentiment of Text\"\n",
    "        },\n",
    "        OutputConfig={\n",
    "            \"S3OutputPath\" : OUTPUT_PATH\n",
    "        }\n",
    "    )\n",
    "flowDefinitionArn = create_workflow_definition_response['FlowDefinitionArn'] # let's save this ARN for future use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Describe flow definition - status should be active\n",
    "for x in range(60):\n",
    "    describeFlowDefinitionResponse = sagemaker.describe_flow_definition(FlowDefinitionName=flowDefinitionName)\n",
    "    print(describeFlowDefinitionResponse['FlowDefinitionStatus'])\n",
    "    if (describeFlowDefinitionResponse['FlowDefinitionStatus'] == 'Active'):\n",
    "        print(\"Flow Definition is active\")\n",
    "        break\n",
    "    time.sleep(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Human Loops"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Detect Sentiment with AWS Comprehend"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have setup our Flow Definition, we are ready to call AWS Comprehend and start our human loops. In this tutorial, we are interested in starting a HumanLoop only if the SentimentScore returned by AWS Comprehend is less than 99%. \n",
    "\n",
    "So, with a bit of logic, we can check the response for each call to Detect Sentiment, and if the SentimentScore is less than 99%, we will kick off a HumanLoop to engage our workforce for a human review. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Sample Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_detect_sentiment_blurbs = ['I enjoy this product', 'I am unhappy with this product', 'It is okay', 'sometimes it works']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "human_loops_started = []\n",
    "SENTIMENT_SCORE_THRESHOLD = .99\n",
    "for blurb in sample_detect_sentiment_blurbs:\n",
    "    # Call AWS Comprehend's Detect Sentiment API\n",
    "    response = comprehend.detect_sentiment(Text=blurb, LanguageCode='en')\n",
    "    sentiment = response['Sentiment']\n",
    "    \n",
    "    print(f'Processing blurb: \\\"{blurb}\\\"')\n",
    "    \n",
    "    # Our condition for when we want to engage a human for review\n",
    "    if (response['SentimentScore'][capsToCamel(sentiment)]< SENTIMENT_SCORE_THRESHOLD):\n",
    "    \n",
    "        humanLoopName = str(uuid.uuid4())\n",
    "        inputContent = {\n",
    "            \"initialValue\": sentiment.title(),\n",
    "            \"taskObject\": blurb\n",
    "        }\n",
    "        start_loop_response = a2i.start_human_loop(\n",
    "            HumanLoopName=humanLoopName,\n",
    "            FlowDefinitionArn=flowDefinitionArn,\n",
    "            HumanLoopInput={\n",
    "                \"InputContent\": json.dumps(inputContent)\n",
    "            }\n",
    "        )\n",
    "        human_loops_started.append(humanLoopName)\n",
    "        print(f'SentimentScore of {response[\"SentimentScore\"][capsToCamel(sentiment)]} is less than the threshold of {SENTIMENT_SCORE_THRESHOLD}')\n",
    "        print(f'Starting human loop with name: {humanLoopName}  \\n')\n",
    "    else:\n",
    "        print(f'SentimentScore of {response[\"SentimentScore\"][capsToCamel(sentiment)]} is above threshold of {SENTIMENT_SCORE_THRESHOLD}')\n",
    "        print('No human loop created. \\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check Status of Human Loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "completed_human_loops = []\n",
    "for human_loop_name in human_loops_started:\n",
    "    resp = a2i.describe_human_loop(HumanLoopName=human_loop_name)\n",
    "    print(f'HumanLoop Name: {human_loop_name}')\n",
    "    print(f'HumanLoop Status: {resp[\"HumanLoopStatus\"]}')\n",
    "    print(f'HumanLoop Output Destination: {resp[\"HumanLoopOutput\"]}')\n",
    "    print('\\n')\n",
    "    \n",
    "    if resp[\"HumanLoopStatus\"] == \"Completed\":\n",
    "        completed_human_loops.append(resp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Wait For Workers to Complete Task"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "workteamName = WORKTEAM_ARN[WORKTEAM_ARN.rfind('/') + 1:]\n",
    "print(\"Navigate to the private worker portal and do the tasks. Make sure you've invited yourself to your workteam!\")\n",
    "print('https://' + sagemaker.describe_workteam(WorkteamName=workteamName)['Workteam']['SubDomain'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check Status of Human Loop Again"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "completed_human_loops = []\n",
    "for human_loop_name in human_loops_started:\n",
    "    resp = a2i.describe_human_loop(HumanLoopName=human_loop_name)\n",
    "    print(f'HumanLoop Name: {human_loop_name}')\n",
    "    print(f'HumanLoop Status: {resp[\"HumanLoopStatus\"]}')\n",
    "    print(f'HumanLoop Output Destination: {resp[\"HumanLoopOutput\"]}')\n",
    "    print('\\n')\n",
    "    \n",
    "    if resp[\"HumanLoopStatus\"] == \"Completed\":\n",
    "        completed_human_loops.append(resp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View Task Results  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once work is completed, Amazon A2I stores results in your S3 bucket and sends a Cloudwatch event. Your results should be available in the S3 OUTPUT_PATH when all work is completed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "import pprint\n",
    "\n",
    "pp = pprint.PrettyPrinter(indent=4)\n",
    "\n",
    "for resp in completed_human_loops:\n",
    "    splitted_string = re.split('s3://' +  BUCKET + '/', resp['HumanLoopOutput']['OutputS3Uri'])\n",
    "    output_bucket_key = splitted_string[1]\n",
    "\n",
    "    response = s3.get_object(Bucket=BUCKET, Key=output_bucket_key)\n",
    "    content = response[\"Body\"].read()\n",
    "    json_output = json.loads(content)\n",
    "    pp.pprint(json_output)\n",
    "    print('\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The End!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}