Amazon Augmented AI (Amazon A2I) Integration with tabular data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Amazon Augmented AI (Amazon A2I) integration with Tabular Data [Example]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. [Introduction](#Introduction)\n",
    "2. [Prerequisites](#Prerequisites)\n",
    "    1. [Workteam](#Workteam)\n",
    "    2. [Permissions](#Notebook-Permission)\n",
    "3. [Client Setup](#Client-Setup)\n",
    "4. [Create Control Plane Resources](#Create-Control-Plane-Resources)\n",
    "    1. [Create Human Task UI](#Create-Human-Task-UI)\n",
    "    2. [Create Flow Definition](#Create-Flow-Definition)\n",
    "5. [Starting Human Loops](#Scenario-1-:-When-Activation-Conditions-are-met-,-and-HumanLoop-is-created)\n",
    "    1. [Wait For Workers to Complete Task](#Wait-For-Workers-to-Complete-Task)\n",
    "    2. [Check Status of Human Loop](#Check-Status-of-Human-Loop)\n",
    "    3. [View Task Results](#View-Task-Results)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "Amazon Augmented AI (Amazon A2I) makes it easy to build the workflows required for human review of ML predictions. Amazon A2I brings human review to all developers, removing the undifferentiated heavy lifting associated with building human review systems or managing large numbers of human reviewers. \n",
    "\n",
    "You can create your own workflows for ML models built on Amazon SageMaker or any other tools. Using Amazon A2I, you can allow human reviewers to step in when a model is unable to make a high confidence prediction or to audit its predictions on an on-going basis. \n",
    "\n",
    "Learn more here: https://aws.amazon.com/augmented-ai/\n",
    "\n",
    "In this tutorial, we will show how you can use **Amazon A2I with Tabular data.** Tabular data is the most common form of data used by data scientists today for generating models. Use cases include, fraud detection, building customer propensity models, forecasting sales using regression etc. In many cases, data scientists often convert unstructured data such as text or images into structured tables that they then use for training models. \n",
    "\n",
    "Here we will first train a model and use the outputs of the trained model to build a human loop for review.\n",
    "\n",
    "For more in depth instructions, visit https://docs.aws.amazon.com/sagemaker/latest/dg/a2i-getting-started.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To incorporate Amazon A2I into your human review workflows, you need three resources:\n",
    "\n",
    "* A **worker task template** to create a worker UI. The worker UI displays your input data, such as documents or images, and instructions to workers. It also provides interactive tools that the worker uses to complete your tasks. For more information, see https://docs.aws.amazon.com/sagemaker/latest/dg/a2i-instructions-overview.html\n",
    "\n",
    "* A **human review workflow**, also referred to as a flow definition. You use the flow definition to configure your human workforce and provide information about how to accomplish the human review task. You can create a flow definition in the Amazon Augmented AI console or with Amazon A2I APIs. To learn more about both of these options, see https://docs.aws.amazon.com/sagemaker/latest/dg/a2i-create-flow-definition.html\n",
    "\n",
    "* A **human loop** to start your human review workflow. When you use one of the built-in task types, the corresponding AWS service creates and starts a human loop on your behalf when the conditions specified in your flow definition are met or for each object if no conditions were specified. When a human loop is triggered, human review tasks are sent to the workers as specified in the flow definition.\n",
    "\n",
    "When using a custom task type, as this tutorial will show, you start a human loop using the Amazon Augmented AI Runtime API. When you call `start_human_loop()` in your custom application, a task is sent to human reviewers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Install Latest SDK\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First, let's get the latest installations of our dependencies\n",
    "!pip install --upgrade pip\n",
    "!pip install boto3 --upgrade\n",
    "!pip install -U botocore"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "We need to set up the following data:\n",
    "* `region` - Region to call A2I.\n",
    "* `BUCKET` - A S3 bucket accessible by the given role\n",
    "    * Used to store the sample images & output results\n",
    "    * Must be within the same region A2I is called from\n",
    "* `role` - The IAM role used as part of StartHumanLoop. By default, this notebook will use the execution role\n",
    "* `workteam` - Group of people to send the work to"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Role and Permissions\n",
    "\n",
    "The AWS IAM Role used to execute the notebook needs to have the following permissions:\n",
    "\n",
    "* SagemakerFullAccess\n",
    "* AmazonSageMakerMechanicalTurkAccess (if using MechanicalTurk as your Workforce)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker import get_execution_role\n",
    "import sagemaker\n",
    "\n",
    "# Setting Role to the default SageMaker Execution Role\n",
    "role = get_execution_role()\n",
    "display(role)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import boto3\n",
    "import botocore\n",
    "\n",
    "sess = sagemaker.Session() \n",
    "\n",
    "#bucket\n",
    "BUCKET = sess.default_bucket() # or use a custom bucket if you created one. \n",
    "PREFIX = 'a2i-data'\n",
    "\n",
    "#specify output path for artifacts\n",
    "OUTPUT_PATH = f's3://{BUCKET}/a2i-results'\n",
    "\n",
    "# Region \n",
    "region = boto3.session.Session().region_name\n",
    "print(region)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tabular data with Amazon SageMaker\n",
    "\n",
    "Before creating the template, we will load a tabular dataset, split the data into train and test, store the test data in Amazon S3, and train a machine learning model. The dataset we use is on Breast Cancer prediction and can be found here: [1] Dua, D. and Graff, C. (2019). UCI Machine Learning Repository [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California, School of Information and Computer Science.\n",
    "\n",
    "Based on the input features, we will first train a model to detect a benign or malignant label. \n",
    "\n",
    "Once the model is trained, we will create an endpoint, and generate some model predictions. We will then create a WorkerUI to load in our immutable test dataset as a table, and dynamically modify the verify and change predictions if needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn.datasets import load_breast_cancer\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "def generatedf(split_ratio):\n",
    "    \"\"\"Loads the dataset into a dataframe and generates train/test splits\"\"\"\n",
    "    data = load_breast_cancer()\n",
    "    df = pd.DataFrame(data.data, columns = data.feature_names)\n",
    "    df['label'] = data.target\n",
    "    cols = list(df.columns)\n",
    "    cols = cols[-1:] + cols[:-1]\n",
    "    df = df[cols]\n",
    "    train, test = train_test_split(df, test_size=split_ratio, random_state=42)\n",
    "    return train, test\n",
    "\n",
    "train_data, test_data = generatedf(0.2)\n",
    "train_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#store the datasets locally\n",
    "train_data.to_csv('train.csv',index = None, header=None)\n",
    "test_data.to_csv('test.csv', index = None, header=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the data into S3\n",
    "sess.upload_data('train.csv', bucket=BUCKET, key_prefix=os.path.join(PREFIX, 'train'))\n",
    "sess.upload_data('test.csv', bucket=BUCKET, key_prefix=os.path.join(PREFIX, 'test'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#load the train and test data filenames from Amazon S3\n",
    "s3_input_train = sagemaker.s3_input(s3_data='s3://{}/{}/train'.format(BUCKET, PREFIX), content_type='csv')\n",
    "s3_input_validation = sagemaker.s3_input(s3_data='s3://{}/{}/test/'.format(BUCKET, PREFIX), content_type='csv')\n",
    "                                        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train and Deploy the model\n",
    "\n",
    "SageMaker will set up the instance types needed and copy the data over to train the model. This may take about **3** minutes to complete training. Once the model is trained, we will deploy the model as an endpoint. Again, SageMaker will set up the instance required, copy the inference image and the inference code and create a HTTPS endpoint. This may take **4-5** minutes. For more details on how SageMaker creates an endpoint, visit: https://docs.aws.amazon.com/sagemaker/latest/dg/how-it-works-hosting.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.amazon.amazon_estimator import get_image_uri\n",
    "\n",
    "container = get_image_uri(region, 'xgboost', '0.90-1')\n",
    "\n",
    "xgb = sagemaker.estimator.Estimator(container,\n",
    "                                    role, \n",
    "                                    train_instance_count=1, \n",
    "                                    train_instance_type='ml.m5.xlarge',\n",
    "                                    output_path=OUTPUT_PATH,\n",
    "                                    sagemaker_session=sess)\n",
    "\n",
    "xgb.set_hyperparameters(max_depth=2,\n",
    "                        eta=0.2,\n",
    "                        gamma=4,\n",
    "                        min_child_weight=6,\n",
    "                        subsample=0.8,\n",
    "                        silent=0,\n",
    "                        objective='binary:logistic',\n",
    "                        num_round=100,\n",
    "                        eval_metric='auc')\n",
    "\n",
    "xgb.fit({'train': s3_input_train, \n",
    "         'validation': s3_input_validation})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb_predictor = xgb.deploy(initial_instance_count = 1, instance_type = 'ml.m5.xlarge')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.predictor import csv_serializer\n",
    "xgb_predictor.content_type = 'text/csv'\n",
    "xgb_predictor.serializer = csv_serializer\n",
    "xgb_predictor.deserializer = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Lets now run predictions on our test set and use it to create a table containing our outputs.\n",
    "import numpy as np\n",
    "\n",
    "def predict(data, model, rows=500):\n",
    "    split_array = np.array_split(data, int(data.shape[0] / float(rows) + 1))\n",
    "    predictions = ''\n",
    "    for array in split_array:\n",
    "        predictions = ','.join([predictions, model.predict(array).decode('utf-8')])\n",
    "\n",
    "    return np.round(np.fromstring(predictions[1:], sep=','))\n",
    "\n",
    "## Generate predictions on the test set for the difference models\n",
    "\n",
    "predictions = predict(test_data[list(test_data.columns)[1:]].values, xgb_predictor)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating human review Workteam or Workforce"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A workforce is the group of workers that you have selected to label your dataset. You can choose either the Amazon Mechanical Turk workforce, a vendor-managed workforce, or you can create your own private workforce for human reviews. Whichever workforce type you choose, Amazon Augmented AI takes care of sending tasks to workers. \n",
    "\n",
    "When you use a private workforce, you also create work teams, a group of workers from your workforce that are assigned to Amazon Augmented AI human review tasks. You can have multiple work teams and can assign one or more work teams to each job."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To create your Workteam, visit the instructions here: https://docs.aws.amazon.com/sagemaker/latest/dg/sms-workforce-management.html\n",
    "\n",
    "After you have created your workteam, replace YOUR_WORKTEAM_ARN below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "WORKTEAM_ARN = 'arn:aws:sagemaker:us-east-2:{account_num}:workteam/private-crowd/stefan-team'#'YOUR_WORKTEAM_ARN'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visit: https://docs.aws.amazon.com/sagemaker/latest/dg/a2i-permissions-security.html to add the necessary permissions to your role"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Client Setup\n",
    "\n",
    "Here we are going to setup the rest of our clients. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import io\n",
    "import uuid\n",
    "import time\n",
    "\n",
    "timestamp = time.strftime(\"%Y-%m-%d-%H-%M-%S\", time.gmtime())\n",
    "# Amazon SageMaker client\n",
    "sagemaker_client = boto3.client('sagemaker', region)\n",
    "\n",
    "# Amazon Augment AI (A2I) client\n",
    "a2i = boto3.client('sagemaker-a2i-runtime')\n",
    "\n",
    "# Amazon S3 client \n",
    "s3 = boto3.client('s3', region)\n",
    "\n",
    "# Flow definition name - this value is unique per account and region. You can also provide your own value here.\n",
    "flowDefinitionName = 'fd-sagemaker-tabular-data-demo-' + timestamp\n",
    "\n",
    "# Task UI name - this value is unique per account and region. You can also provide your own value here.\n",
    "taskUIName = 'ui-sagemaker-tabular-data-demo-' + timestamp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Control Plane Resources"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Human Task UI\n",
    "\n",
    "Create a human task UI resource, giving a UI template in liquid html. This template will be rendered to the human workers whenever human loop is required.\n",
    "\n",
    "For over 70 pre built UIs, check: https://github.com/aws-samples/amazon-a2i-sample-task-uis.\n",
    "\n",
    "We will use the following template to render both the test dataset, as well as the model predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "template = r\"\"\"\n",
    "<script src=\"https://assets.crowd.aws/crowd-html-elements.js\"></script>\n",
    "\n",
    "<style>\n",
    "  table, tr, th, td {\n",
    "    border: 1px solid black;\n",
    "    border-collapse: collapse;\n",
    "    padding: 5px;\n",
    "  }\n",
    "</style>\n",
    "\n",
    "<crowd-form>\n",
    "    <div>\n",
    "        <h1>Instructions</h1>\n",
    "        <p>Please review the predictions in the Predictions table based on the input data table below, and make corrections where appropriate. </p>\n",
    "        <p> Here are the labels: </p>\n",
    "        <p> 0: Benign </p>\n",
    "        <p> 1: Malignant </p>\n",
    "    </div>\n",
    "    <div>\n",
    "      <h3> Breast cancer dataset </h3>\n",
    "      <div id=\"my_table\"> {{ task.input.table | skip_autoescape }} </div>\n",
    "   </div>\n",
    "    <br>\n",
    "    <h1> Predictions Table </h1>\n",
    "    <table>\n",
    "      <tr>\n",
    "        <th>ROW NUMBER</th>\n",
    "        <th>MODEL PREDICTION</th>\n",
    "        <th>AGREE/DISAGREE WITH ML RATING?</th>\n",
    "        <th>YOUR PREDICTION</th>\n",
    "        <th>CHANGE REASON </th>\n",
    "      </tr>\n",
    "\n",
    "      {% for pair in task.input.Pairs %}\n",
    "\n",
    "        <tr>\n",
    "          <td>{{ pair.row }}</td>\n",
    "          <td><crowd-text-area name=\"predicted{{ forloop.index }}\" value=\"{{ pair.prediction }}\"></crowd-text-area></td>\n",
    "          <td>\n",
    "            <p>\n",
    "              <input type=\"radio\" id=\"agree{{ forloop.index }}\" name=\"rating{{ forloop.index }}\" value=\"agree\" required>\n",
    "              <label for=\"agree{{ forloop.index }}\">Agree</label>\n",
    "            </p>\n",
    "            <p>\n",
    "              <input type=\"radio\" id=\"disagree{{ forloop.index }}\" name=\"rating{{ forloop.index }}\" value=\"disagree\" required>\n",
    "              <label for=\"disagree{{ forloop.index }}\">Disagree</label>       \n",
    "            </p> \n",
    "          </td>\n",
    "          <td>\n",
    "            <p>\n",
    "            <input type=\"text\" name=\"True Prediction\" placeholder=\"Enter your Prediction\" />\n",
    "            </p>\n",
    "           </td>\n",
    "           <td>\n",
    "            <p>\n",
    "            <input type=\"text\" name=\"Change Reason\" placeholder=\"Explain why you changed the prediction\" />\n",
    "            </p>\n",
    "           </td>\n",
    "        </tr>\n",
    "\n",
    "      {% endfor %}\n",
    "\n",
    "    </table>\n",
    "</crowd-form>\n",
    "\"\"\"\n",
    "\n",
    "def create_task_ui():\n",
    "    '''\n",
    "    Creates a Human Task UI resource.\n",
    "\n",
    "    Returns:\n",
    "    struct: HumanTaskUiArn\n",
    "    '''\n",
    "    response = sagemaker_client.create_human_task_ui(\n",
    "        HumanTaskUiName=taskUIName,\n",
    "        UiTemplate={'Content': template})\n",
    "    return response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create task UI\n",
    "humanTaskUiResponse = create_task_ui()\n",
    "humanTaskUiArn = humanTaskUiResponse['HumanTaskUiArn']\n",
    "print(humanTaskUiArn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the Flow Definition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, we're going to create a flow definition definition. Flow Definitions allow us to specify:\n",
    "\n",
    "* The workforce that your tasks will be sent to.\n",
    "* The instructions that your workforce will receive. This is called a worker task template.\n",
    "* The configuration of your worker tasks, including the number of workers that receive a task and time limits to complete tasks.\n",
    "* Where your output data will be stored.\n",
    "\n",
    "This demo is going to use the API, but you can optionally create this workflow definition in the console as well. \n",
    "\n",
    "For more details and instructions, see: https://docs.aws.amazon.com/sagemaker/latest/dg/a2i-create-flow-definition.html."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "create_workflow_definition_response = sagemaker_client.create_flow_definition(\n",
    "        FlowDefinitionName= flowDefinitionName,\n",
    "        RoleArn= role,\n",
    "        HumanLoopConfig= {\n",
    "            \"WorkteamArn\": WORKTEAM_ARN,\n",
    "            \"HumanTaskUiArn\": humanTaskUiArn,\n",
    "            \"TaskCount\": 1,\n",
    "            \"TaskDescription\": \"Make sure the labels are correct\",\n",
    "            \"TaskTitle\": \"tabular data a2i demo\"\n",
    "        },\n",
    "        OutputConfig={\n",
    "            \"S3OutputPath\" : OUTPUT_PATH\n",
    "        }\n",
    "    )\n",
    "flowDefinitionArn = create_workflow_definition_response['FlowDefinitionArn'] # let's save this ARN for future use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Describe flow definition - status should be active\n",
    "for x in range(60):\n",
    "    describeFlowDefinitionResponse = sagemaker_client.describe_flow_definition(FlowDefinitionName=flowDefinitionName)\n",
    "    print(describeFlowDefinitionResponse['FlowDefinitionStatus'])\n",
    "    if (describeFlowDefinitionResponse['FlowDefinitionStatus'] == 'Active'):\n",
    "        print(\"Flow Definition is active\")\n",
    "        break\n",
    "    time.sleep(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Human Loops\n",
    "\n",
    "Now that we have setup our Flow Definition, we are ready to start the human loop to have the reviewers asynchronously review the outputs generated by our model. First we need to create a dictionary containing our model outputs, so we can load it dynamically"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "item_list = [{'row': \"ROW_{}\".format(x), 'prediction': predictions[x]} for x in range(5)]\n",
    "item_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ip_content = {\"table\": test_data.reset_index().drop(columns = ['index', 'label']).head().to_html(), \n",
    "              'Pairs': item_list\n",
    "             }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "humanLoopName = str(uuid.uuid4())\n",
    "\n",
    "start_loop_response = a2i.start_human_loop(\n",
    "            HumanLoopName=humanLoopName,\n",
    "            FlowDefinitionArn=flowDefinitionArn,\n",
    "            HumanLoopInput={\n",
    "                \"InputContent\": json.dumps(ip_content)\n",
    "            }\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check Status of Human Loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "resp = a2i.describe_human_loop(HumanLoopName=humanLoopName)\n",
    "print(f'HumanLoop Name: {humanLoopName}')\n",
    "print(f'HumanLoop Status: {resp[\"HumanLoopStatus\"]}')\n",
    "print(f'HumanLoop Output Destination: {resp[\"HumanLoopOutput\"]}')\n",
    "print('\\n')\n",
    "    \n",
    "if resp[\"HumanLoopStatus\"] == \"Completed\":\n",
    "    completed_human_loops.append(resp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Wait For Workers to Complete Task\n",
    "Since we are using private workteam, we should go to the labling UI to perform the inspection ourselves."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "workteamName = WORKTEAM_ARN[WORKTEAM_ARN.rfind('/') + 1:]\n",
    "print(\"Navigate to the private worker portal and do the tasks. Make sure you've invited yourself to your workteam!\")\n",
    "print('https://' + sagemaker_client.describe_workteam(WorkteamName=workteamName)['Workteam']['SubDomain'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check Status of Human Loop Again"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "resp = a2i.describe_human_loop(HumanLoopName=humanLoopName)\n",
    "print(f'HumanLoop Name: {humanLoopName}')\n",
    "print(f'HumanLoop Status: {resp[\"HumanLoopStatus\"]}')\n",
    "print(f'HumanLoop Output Destination: {resp[\"HumanLoopOutput\"]}')\n",
    "print('\\n')\n",
    "    \n",
    "if resp[\"HumanLoopStatus\"] == \"Completed\":\n",
    "    completed_human_loops.append(resp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View Task Results  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "import pprint\n",
    "\n",
    "pp = pprint.PrettyPrinter(indent=4)\n",
    "\n",
    "for resp in completed_human_loops:\n",
    "    splitted_string = re.split('s3://' +  BUCKET + '/', resp['HumanLoopOutput']['OutputS3Uri'])\n",
    "    output_bucket_key = splitted_string[1]\n",
    "\n",
    "    response = s3.get_object(Bucket=BUCKET, Key=output_bucket_key)\n",
    "    content = response[\"Body\"].read()\n",
    "    json_output = json.loads(content)\n",
    "    pp.pprint(json_output)\n",
    "    print('\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Delete Resources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a2i.stop_human_loop(HumanLoopName=humanLoopName)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a2i.delete_human_loop(HumanLoopName=humanLoopName)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xgb_predictor.delete_endpoint()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}