aws_marketplace/using_model_packages/improving_industrial_workplace_safety/improving_industrial_workplace_safety.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Demonstrating Industrial Workplace Safety using Pre-trained Machine Learning Models\n",
    "\n",
    "### Introduction\n",
    "\n",
    "This sample notebook shows how to use pre-trained model packages from [AWS Marketplace](https://aws.amazon.com/marketplace/search/results?page=1&filters=FulfillmentOptionType&FulfillmentOptionType=SageMaker&ref_=mlmp_gitdemo_indust) to detect industrial workspace safety related object labels, such as hard-hat, personal protective equipment, construction machinery, and construction worker in an image. The notebook also shows an approach to perform inference on a video by taking snapshots from the video file to generate an activity/status log. At the end of this you will become familiar on steps to integrate inferences from pre-trained models into your application. This notebook is intended for demonstration, we highly recommend you to evaluate the accuracy of machine learning models to see if they meet your expectations.\n",
    "\n",
    "\n",
    "### Pre-requisites:\n",
    "This sample notebook requires you to subscribe to pre-trained machine learning model packages. Follow the following steps to subscribe to the listings:\n",
    "\n",
    "1. Open the following model package product detail pages, in separate tabs, in your web browser. \n",
    "\n",
    "  1.  [Construction Worker Detection](https://aws.amazon.com/marketplace/pp/prodview-6utmzaproaqhs?qid=1563547984309&sr=0-5&ref_=mlmp_gitdemo_indust) to identify construction workers in an image. \n",
    "\n",
    "  1. [Hard Hat Detector for Worker Safety](https://aws.amazon.com/marketplace/pp/prodview-jd5tj2egpxxum?qid=1563547984309&sr=0-2&ref_=mlmp_gitdemo_indust) model to infer if construction workers are wearing hard hats.\n",
    "\n",
    "  1. [Personal Protective Equipments](https://aws.amazon.com/marketplace/pp/prodview-2inbkii6o24k4?qid=1563547984309&sr=0-6&ref_=mlmp_gitdemo_indust) to infer if a person is wearing a high visibility safety vest. \n",
    "\n",
    "  1. [Construction Machines Detector](https://aws.amazon.com/marketplace/pp/prodview-fuukizaiq5o7c?qid=1563549078039&sr=0-1&ref_=mlmp_gitdemo_indust) to identify construction machines in an image. \n",
    "\n",
    "2. For each of the model packages, follow these steps: \n",
    "  1. Review the information available on the product details page including **Support Terms** .\n",
    "  1. Click on **\"Continue to Subscribe\"**. You will now see the **\"Subscribe to this software\"** page. \n",
    "  1. Review **End User License Agreement** and **Pricing Terms**.\n",
    "  1. **\"Accept Offer\"** button needs to be clicked if your organization agrees with EULA, pricing information and support terms.\n",
    " \n",
    "\n",
    "Notes: \n",
    "  1. Once you click on **Continue to configuration** button and then choose a region, you will see a **Product Arn** displayed. This is the model package ARN that you need to specify while creating a deployable model using Boto3.  However, for this notebook, the model ARNs have been specified in **src/model_package_arns.py** file and you need not specify them explicitly. The configuration page also provides a **\"View in SageMaker\"** button to navigate to Amazon SageMaker to deploy via Amazon SageMaker Console. \n",
    "  1. Products with **Free Trials**, do not incur hourly software charges during free trial period, but AWS infrastructure charges still apply. Free Trials will automatically convert to a paid hourly subscription upon expiration. We have included steps below to cancel subscription at the end of this exercise. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install opencv-python-headless"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Set up environment and view sample images\n",
    "\n",
    "### Step 1.1: Set up environment\n",
    "\n",
    "In this section, we will first import necessary libraries and define variables such as an S3 bucket, an IAM role, and an Amazon SageMaker session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import necessary libraries and declare variables\n",
    "import json\n",
    "from sagemaker import ModelPackage\n",
    "from src.model_package_arns import ModelPackageArnProvider\n",
    "import sagemaker as sage\n",
    "from sagemaker import get_execution_role\n",
    "from sagemaker import ModelPackage\n",
    "import boto3\n",
    "from IPython.display import Image\n",
    "import matplotlib.pyplot as plt\n",
    "import cv2\n",
    "import numpy as np\n",
    "\n",
    "role = get_execution_role()\n",
    "\n",
    "sagemaker_session = sage.Session()\n",
    "bucket = sagemaker_session.default_bucket()\n",
    "region = sagemaker_session.boto_region_name\n",
    "runtime = boto3.client(\"runtime.sagemaker\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will create utility functions to:\n",
    "1. Draw a bounding box on an image based on coordinates.\n",
    "2. Display an image.\n",
    "3. Perform a prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This function accepts an image, bounding box co-ordinates, label, label probability,\n",
    "# and returns the image that has the bounding box along with the label and its probability.\n",
    "def draw_bounding_box(img, x1, y1, x2, y2, class_name, probability):\n",
    "    # truncate probability to two decimal places\n",
    "    img = cv2.rectangle(img, (x1, y1), (x2, y2), (0, 215, 255), 2)\n",
    "    if probability is not None:\n",
    "        img = cv2.putText(\n",
    "            img,\n",
    "            \"{} {}\".format(class_name, float(str(probability)[:4])),\n",
    "            (x1, y1 - 10),\n",
    "            cv2.FONT_HERSHEY_SIMPLEX,\n",
    "            1,\n",
    "            (127, 255, 0),\n",
    "            2,\n",
    "        )\n",
    "\n",
    "    else:\n",
    "        img = cv2.putText(\n",
    "            img,\n",
    "            \"{}\".format(class_name),\n",
    "            (x1, y1 - 10),\n",
    "            cv2.FONT_HERSHEY_SIMPLEX,\n",
    "            1,\n",
    "            (127, 255, 0),\n",
    "            2,\n",
    "        )\n",
    "    return img"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This function accepts image along with a title and displays the same.\n",
    "def show_image(img, title):\n",
    "    rgb_img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n",
    "    figure = plt.figure(figsize=(12, 18))\n",
    "    axis = figure.add_subplot(111)\n",
    "    axis.imshow(rgb_img, interpolation=\"none\")\n",
    "    axis.set_title(title)\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict(endpoint_name, payload):\n",
    "    # Perform a prediction\n",
    "    response = runtime.invoke_endpoint(\n",
    "        EndpointName=endpoint_name, ContentType=\"image/jpeg\", Body=payload\n",
    "    )\n",
    "\n",
    "    # Perform a prediction\n",
    "    result = response[\"Body\"].read().decode()\n",
    "\n",
    "    return result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1.2: View sample images"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now view the sample images used to perform an inference. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1.2.1:  View construction site image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the next cell to view an image of a construction site with workers and a truck. The workers are wearing personal protective equipment - hard hat, and safety vest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "construction_image = {\"path\": \"img/construction-2578410_640.jpg\"}\n",
    "\n",
    "with open(construction_image[\"path\"], \"rb\") as image:\n",
    "    construction_image[\"byte_array\"] = bytearray(image.read())\n",
    "\n",
    "Image(url=construction_image[\"path\"], width=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Courtesy - https://pixabay.com/photos/construction-worker-safety-2578410"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1.2.2:  View an image with a worker and a person at a workplace"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following image shows two people, a worker wearing a high-visibility vest and a person."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "workers_image = {\"path\": \"img/two-employees.jpg\"}\n",
    "\n",
    "with open(workers_image[\"path\"], \"rb\") as image:\n",
    "    workers_image[\"byte_array\"] = bytearray(image.read())\n",
    "\n",
    "Image(url=workers_image[\"path\"], width=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Courtesy - https://www.pexels.com/photo/two-men-wearing-white-hard-hat-901941"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1.2.3: View an image with an excavator and a truck at work"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following image shows a truck and an excavator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "machines_image = {\"path\": \"img/earth-2579434_1280.jpg\"}\n",
    "\n",
    "with open(machines_image[\"path\"], \"rb\") as image:\n",
    "    machines_image[\"byte_array\"] = bytearray(image.read())\n",
    "\n",
    "Image(url=machines_image[\"path\"], width=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Courtesy - https://pixabay.com/photos/earth-390f-hydraulic-excavators-2579434/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will deploy pre-trained models to generate inferences using sample images."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Deploy construction worker detection model\n",
    "\n",
    "In this step, you will deploy the [Construction Worker Detection](https://aws.amazon.com/marketplace/pp/prodview-6utmzaproaqhs?qid=1563547984309&sr=0-5&ref_=mlmp_gitdemo_indust) model package and perform an inference using sample images."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2.1: Deploy the model for performing real-time inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "construction_worker_detection_model_name = \"construction-worker-detection-endpoint\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the model_package_arn.\n",
    "construction_worker_detection_modelpackage_arn = (\n",
    "    ModelPackageArnProvider.get_construction_worker_model_package_arn(region)\n",
    ")\n",
    "\n",
    "# create a deployable model for damage inspection model package.\n",
    "construction_worker_detection_model = ModelPackage(\n",
    "    role=role,\n",
    "    model_package_arn=construction_worker_detection_modelpackage_arn,\n",
    "    sagemaker_session=sagemaker_session,\n",
    ")\n",
    "\n",
    "# Deploy the model.\n",
    "predictor_construction_worker_detection = construction_worker_detection_model.deploy(\n",
    "    1, \"ml.c5.xlarge\", endpoint_name=construction_worker_detection_model_name\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While the model is deploying, review the **Usage Information** and **Additional Resources** section from the [model package detail page](https://aws.amazon.com/marketplace/pp/prodview-6utmzaproaqhs?qid=1563547984309&sr=0-5&ref_=mlmp_gitdemo_indust) to understand the I/O interface of the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2.2: Perform a prediction (Test 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this step, we will perform a prediction using the construction-site image. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform a prediction.\n",
    "construction_worker_detection_result_1 = json.loads(\n",
    "    predict(construction_worker_detection_model_name, construction_image[\"byte_array\"])\n",
    ")\n",
    "# Un-comment the following line to view the result returned by the model.\n",
    "# print(json.dumps(construction_worker_detection_result,indent=2))\n",
    "\n",
    "# Read original image.\n",
    "image = cv2.imread(construction_image[\"path\"])\n",
    "\n",
    "# Plot the inference on the image\n",
    "for output in construction_worker_detection_result_1[\"output\"]:\n",
    "    x1 = int(output[\"bbox\"][0])\n",
    "    y1 = int(output[\"bbox\"][1])\n",
    "    x2 = int(output[\"bbox\"][2])\n",
    "    y2 = int(output[\"bbox\"][3])\n",
    "    image = draw_bounding_box(image, x1, y1, x2, y2, output[\"class\"], None)\n",
    "\n",
    "show_image(image, \"Worker detection Test 1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see that the model recognized all the three workers found in the image."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2.3: Perform a prediction (Test 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us perform inference using the worker/person image and see how the model can identify a worker and a person."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform a prediction\n",
    "construction_worker_detection_result_2 = json.loads(\n",
    "    predict(construction_worker_detection_model_name, workers_image[\"byte_array\"])\n",
    ")\n",
    "\n",
    "# Un-comment the following line to view the result returned by the model.\n",
    "# print(json.dumps(construction_worker_detection_result,indent=2))\n",
    "\n",
    "# Read original image.\n",
    "\n",
    "image = cv2.imread(workers_image[\"path\"])\n",
    "\n",
    "# Plot the inference on the image\n",
    "for output in construction_worker_detection_result_2[\"output\"]:\n",
    "    x1 = int(output[\"bbox\"][0])\n",
    "    y1 = int(output[\"bbox\"][1])\n",
    "    x2 = int(output[\"bbox\"][2])\n",
    "    y2 = int(output[\"bbox\"][3])\n",
    "    image = draw_bounding_box(image, x1, y1, x2, y2, output[\"class\"], None)\n",
    "\n",
    "show_image(image, \"Worker detection Test 2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see that the model can differentiate between a construction worker (on the left) and a non-construction worker (person on the right). \n",
    "\n",
    "AWS Marketplace also contains another model you may want to try for [construction worker detection](https://aws.amazon.com/marketplace/pp/prodview-labdyzgb3z6fe?qid=1563562334851&sr=0-2&ref_=mlmp_gitdemo_indust)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Deploy the hard-hat detection model.\n",
    "\n",
    "In this step, we will deploy the [Hard Hat Detector for Worker Safety](https://aws.amazon.com/marketplace/pp/prodview-jd5tj2egpxxum?qid=1563547984309&sr=0-2&ref_=mlmp_gitdemo_indust) model to identify whether people in the image are wearing [hard hats](https://en.wikipedia.org/wiki/Hard_hat)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3.1:  Deploy the model for performing real-time inference.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hard_hat_detection_model_name = \"hardhat-detection-endpoint\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the model_package_arn\n",
    "hard_hat_detection_modelpackage_arn = (\n",
    "    ModelPackageArnProvider.get_hard_hat_detection_model_package_arn(region)\n",
    ")\n",
    "\n",
    "# create a deployable model.\n",
    "hard_hat_detection_model = ModelPackage(\n",
    "    role=role,\n",
    "    model_package_arn=hard_hat_detection_modelpackage_arn,\n",
    "    sagemaker_session=sagemaker_session,\n",
    ")\n",
    "\n",
    "# Deploy the model\n",
    "predictor_hard_hat_detection = hard_hat_detection_model.deploy(\n",
    "    1, \"ml.p2.xlarge\", endpoint_name=hard_hat_detection_model_name\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3.2: Perform real-time inference on the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform a prediction\n",
    "hard_hat_detection_result = json.loads(\n",
    "    predict(hard_hat_detection_model_name, construction_image[\"byte_array\"])\n",
    ")\n",
    "# Un-comment the following line to view the result returned by the model.\n",
    "# print(json.dumps(hard_hat_detection_result,indent=2))\n",
    "\n",
    "# Read original image.\n",
    "image = cv2.imread(construction_image[\"path\"])\n",
    "\n",
    "# Plot the inference on the image\n",
    "width = image.shape[1]\n",
    "height = image.shape[0]\n",
    "\n",
    "for i in range(len(hard_hat_detection_result[\"boxes\"])):\n",
    "    output = hard_hat_detection_result[\"boxes\"][i]\n",
    "    x1 = int(round(output[0] * width, 2))\n",
    "    y1 = int(round(output[1] * height, 2))\n",
    "    x2 = int(round(output[2] * width, 2))\n",
    "    y2 = int(round(output[3] * height, 2))\n",
    "    image = draw_bounding_box(\n",
    "        image, x1, y1, x2, y2, \"hard-hat\", hard_hat_detection_result[\"scores\"][i]\n",
    "    )\n",
    "# Display result\n",
    "show_image(image, \"hard-hat detection\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note, the pre-trained model could identify all three hard-hats found in the picture with high probabilities."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4. Deploy the Personal Protective Equipment (PPE) detection model\n",
    "\n",
    "Next, we will deploy [Personal Protective Equipment](https://aws.amazon.com/marketplace/pp/prodview-2inbkii6o24k4?qid=1563547984309&sr=0-6&ref_=mlmp_gitdemo_indust) machine learning model to identify whether the person in the image is wearing PPE such as a high visibility vest. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4.1:  Deploy the model for performing real-time inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ppe_detection_model_name = \"personal-protective-equip-detection-endpoint\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the model_package_arn.\n",
    "ppe_detection_modelpackage_arn = ModelPackageArnProvider.get_ppe_detection_model_package_arn(region)\n",
    "\n",
    "# create a deployable model.\n",
    "ppe_detection_model = ModelPackage(\n",
    "    role=role, model_package_arn=ppe_detection_modelpackage_arn, sagemaker_session=sagemaker_session\n",
    ")\n",
    "\n",
    "# Deploy the model.\n",
    "predictor_ppe_detection = ppe_detection_model.deploy(\n",
    "    1, \"ml.c5.xlarge\", endpoint_name=ppe_detection_model_name\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4.2: Perform real-time inference on the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform a prediction\n",
    "ppe_detection_result = json.loads(\n",
    "    predict(ppe_detection_model_name, construction_image[\"byte_array\"])\n",
    ")\n",
    "\n",
    "# Un-comment the following line to view the result returned by the model.\n",
    "# print(json.dumps(ppe_detection_result,indent=2))\n",
    "\n",
    "# Read original image.\n",
    "image = cv2.imread(construction_image[\"path\"])\n",
    "\n",
    "# Plot inference result on the image\n",
    "for output in ppe_detection_result[\"output\"]:\n",
    "\n",
    "    x1 = int(output[\"bbox\"][0])\n",
    "    y1 = int(output[\"bbox\"][1])\n",
    "    x2 = int(output[\"bbox\"][2])\n",
    "    y2 = int(output[\"bbox\"][3])\n",
    "    image = draw_bounding_box(image, x1, y1, x2, y2, \"PPE\", output[\"score\"])\n",
    "\n",
    "# Display result\n",
    "show_image(image, \"Personal protective equipments\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note how the pre-trained model could identify the PPEs in the image with high probabilities."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5. Deploy the Construction Machines detection model\n",
    "\n",
    "Next, you will deploy [Construction Machines Detector](https://aws.amazon.com/marketplace/pp/prodview-fuukizaiq5o7c?qid=1563549078039&sr=0-1&ref_=mlmp_gitdemo_indust) to identify construction machines from an image. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 5.1:  Deploy the model for performing real-time inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "machine_detection_model_name = \"machine-detection-endpoint\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the model_package_arn\n",
    "machine_detection_modelpackage_arn = (\n",
    "    ModelPackageArnProvider.get_machine_detection_model_package_arn(region)\n",
    ")\n",
    "\n",
    "# create a deployable model.\n",
    "machine_detection_model = ModelPackage(\n",
    "    role=role,\n",
    "    model_package_arn=machine_detection_modelpackage_arn,\n",
    "    sagemaker_session=sagemaker_session,\n",
    ")\n",
    "\n",
    "# Deploy the model\n",
    "predictor_machine_detection = machine_detection_model.deploy(\n",
    "    1, \"ml.p3.2xlarge\", endpoint_name=machine_detection_model_name\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 5.2: Perform real-time inference on the model (Test 1)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform a prediction\n",
    "machine_detection_result = json.loads(\n",
    "    predict(machine_detection_model_name, construction_image[\"byte_array\"])\n",
    ")\n",
    "# Un-comment the following line to view the result returned by the model.\n",
    "# print(json.dumps(machine_detection_result,indent=2))\n",
    "\n",
    "# Read original image.\n",
    "image = cv2.imread(construction_image[\"path\"])\n",
    "\n",
    "# Plot inference result on the image\n",
    "for output in machine_detection_result[\"outputs\"][\"detections\"]:\n",
    "    x1 = output[0]\n",
    "    y1 = output[1]\n",
    "    x2 = output[2]\n",
    "    y2 = output[3]\n",
    "    image = draw_bounding_box(image, x1, y1, x2, y2, output[4], output[5])\n",
    "\n",
    "# Display result\n",
    "show_image(image, \"Construction machines test 1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Model did not detect any construction machines since there were none. We will now perform inference on one more image that shows construction machinery such as an excavator."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 5.3: Perform real-time inference on the model (Test 2)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform a prediction\n",
    "machine_detection_result = json.loads(\n",
    "    predict(machine_detection_model_name, machines_image[\"byte_array\"])\n",
    ")\n",
    "# print(json.dumps(machine_detection_result,indent=2))\n",
    "\n",
    "# Read original image.\n",
    "image = cv2.imread(machines_image[\"path\"])\n",
    "\n",
    "# Plot the inference on the image\n",
    "for output in machine_detection_result[\"outputs\"][\"detections\"]:\n",
    "    x1 = output[0]\n",
    "    y1 = output[1]\n",
    "    x2 = output[2]\n",
    "    y2 = output[3]\n",
    "    image = draw_bounding_box(image, x1, y1, x2, y2, output[4], output[5])\n",
    "\n",
    "# Display result\n",
    "show_image(image, \"Construction machines Test 2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note how the pre-trained model could detect both, a truck, and an excavator from the picture."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Step 6. Generate actionable insights on video input"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The pre-trained models demonstrated above accept images as an input. However, the input data can also be in the form of a video. In this section, you will see how to extract actionable insights from a video by performing inference on snapshots."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import HTML\n",
    "\n",
    "video_path = \"./video/construction-video.mp4\"\n",
    "\n",
    "HTML(\n",
    "    '<iframe width=\"560\" height=\"315\" src=\"'\n",
    "    + video_path\n",
    "    + '?rel=0&amp;controls=0&amp;showinfo=0\" frameborder=\"0\" allowfullscreen></iframe>'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Courtesy - https://pixabay.com/videos/construction-road-excavator-worker-26239/ (Edited)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Analyzing hours of video footage can be tedious. Status summary reports and rules can help detect non-compliance to trigger alarms.  In this section, we will generate following status summary from the video:\n",
    "\n",
    "__Sample Summary report__<br>\n",
    "No Alarm : 1 truck(s), 1 excavator(s), no workers found.<br>\n",
    "No Alarm : 2 truck(s), 1 excavator(s), 1 workers found.<br>\n",
    "No Alarm : 1 truck(s), 1 excavator(s), 1 workers found.<br>\n",
    "No Alarm : 1 truck(s), 1 excavator(s), no workers found.<br>\n",
    "__ALARM__    : 1 worker(s) wearing PPE but 0 wearing hard hats, 1 truck(s), 1 excavator(s) found.<br>\n",
    "No Alarm : 1 truck(s), 1 excavator(s), no workers found.<br>\n",
    "__ALARM__   : 1 worker(s) wearing PPE but 0 wearing hard hats, 1 truck(s), 1 excavator(s) found.<br><br>\n",
    "\n",
    "\n",
    "__Note__: There are couple instances in the video when the worker is not visible because of an obstruction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this task, we will take a snapshot from the video every 1.5 seconds and then perform inference on each snapshot to identify actionable insights. The snapshot images from the video enable you to to generate inferences from model packages that only support image payloads. In some cases, this approach may help you scale usage of endpoints.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "capture = cv2.VideoCapture(video_path)\n",
    "\n",
    "# Get number of frames from the video.\n",
    "framecount = int(capture.get(cv2.CAP_PROP_FRAME_COUNT))\n",
    "\n",
    "# Take snapshot every 1.5 second(s)\n",
    "num_seconds = 1.5\n",
    "\n",
    "skip_frames = capture.get(cv2.CAP_PROP_FPS) * num_seconds\n",
    "\n",
    "num_snapshots = int(framecount / skip_frames)\n",
    "\n",
    "# For this experiment, we extract an image every second so that we can utilize the endpoints more efficiently.\n",
    "for i in range(num_snapshots):\n",
    "    flag, frame = capture.read()\n",
    "    if flag:\n",
    "        path = \"./video/snapshots/frame\" + str(i) + \".jpg\"\n",
    "        print(\"Creating snapshot on path - \" + path)\n",
    "        cv2.imwrite(path, frame)\n",
    "        capture.set(cv2.CAP_PROP_POS_FRAMES, ((i + 1) * skip_frames))\n",
    "capture.release()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have created snapshots from the video, let us create a utility function that generates status summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The following method accepts path of an image, performs inference on\n",
    "# construction machines detector, PPE, hard-hat detector models and generates a status summary.\n",
    "def generate_status_summary(image_path):\n",
    "    image_byte_array = []\n",
    "\n",
    "    num_trucks = 0\n",
    "    num_excavator = 0\n",
    "    num_ppe = 0\n",
    "    num_hard_hat = 0\n",
    "\n",
    "    # Open the image.\n",
    "    with open(image_path, \"rb\") as image:\n",
    "        image_byte_array = bytearray(image.read())\n",
    "\n",
    "    # Count number of machines\n",
    "    machine_detection_result = json.loads(predict(machine_detection_model_name, image_byte_array))\n",
    "    for output in machine_detection_result[\"outputs\"][\"detections\"]:\n",
    "        if output[5] > 0.65:\n",
    "            if output[4] == \"TRUCK\":\n",
    "                num_trucks += 1\n",
    "            if output[4] == \"EXCAVATOR\":\n",
    "                num_excavator += 1\n",
    "\n",
    "    # Count number of personal protective equipments(PPEs)\n",
    "    ppe_detection_result = json.loads(predict(ppe_detection_model_name, image_byte_array))\n",
    "    for output in ppe_detection_result[\"output\"]:\n",
    "        if output[\"score\"] > 0.5:\n",
    "            num_ppe += 1\n",
    "\n",
    "    # Count number of hard-hats\n",
    "    hard_hat_detection_result = json.loads(predict(hard_hat_detection_model_name, image_byte_array))\n",
    "    for i in range(len(hard_hat_detection_result[\"boxes\"])):\n",
    "        if hard_hat_detection_result[\"scores\"][i] > 0.5:\n",
    "            num_hard_hat += 1\n",
    "\n",
    "    # Create and return the summary.\n",
    "    if num_ppe == num_hard_hat == 0:\n",
    "        current_status = (\n",
    "            \"No Alarm : \"\n",
    "            + str(num_trucks)\n",
    "            + \" truck(s), \"\n",
    "            + str(num_excavator)\n",
    "            + \" excavator(s), no workers found.\"\n",
    "        )\n",
    "    elif num_ppe == num_hard_hat:\n",
    "        current_status = (\n",
    "            \"No Alarm : \"\n",
    "            + str(num_trucks)\n",
    "            + \" truck(s), \"\n",
    "            + str(num_excavator)\n",
    "            + \" excavator(s), \"\n",
    "            + str(num_ppe)\n",
    "            + \" workers found.\"\n",
    "        )\n",
    "    elif num_ppe > num_hard_hat:\n",
    "        current_status = (\n",
    "            \"ALARM    : \"\n",
    "            + str(num_ppe)\n",
    "            + \" worker(s) wearing PPE but \"\n",
    "            + str(num_hard_hat)\n",
    "            + \" wearing hard hats, \"\n",
    "            + str(num_trucks)\n",
    "            + \" truck(s), \"\n",
    "            + str(num_excavator)\n",
    "            + \" excavator(s) found.\"\n",
    "        )\n",
    "    elif num_hard_hat > num_ppe:\n",
    "        current_status = (\n",
    "            \"ALARM    : \"\n",
    "            + str(num_hard_hat)\n",
    "            + \" worker(s) wearing hard hats but \"\n",
    "            + str(num_ppe)\n",
    "            + \" workers wearing PPE, \"\n",
    "            + str(num_trucks)\n",
    "            + \" truck(s), and \"\n",
    "            + str(num_excavator)\n",
    "            + \" excavator(s) found.\"\n",
    "        )\n",
    "    return current_status"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will run the utility function on each snapshot to generate status summary log from the video."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize start-time with timestamp for first entry.\n",
    "start_time = \"00:00:{:0>3d}\".format(0)\n",
    "previous_status = \"\"\n",
    "\n",
    "print(\"(Start)HH:mm:SSS-(End)HH:mm:SSS : Alarm/No alarm : Status Details\")\n",
    "print(\"---------------------------------------------------------------\")\n",
    "\n",
    "# next, we loop on each of the screenshot and extract summary. If summary for a screenshot\n",
    "# matches with summary of previous screenshot, then we simply record the duration instead of\n",
    "# adding a duplicate summary record.\n",
    "\n",
    "for j in range(num_snapshots):\n",
    "\n",
    "    image_path = \"./video/snapshots/frame\" + str(j) + \".jpg\"\n",
    "    current_status = generate_status_summary(image_path)\n",
    "\n",
    "    if previous_status == \"\":\n",
    "        # For first record, populate the previous_status as current_status.\n",
    "        previous_status = current_status\n",
    "\n",
    "    # This means that summary status of the picture has changed. print the previous status and\n",
    "    # start tracking new status.\n",
    "    elif previous_status != current_status:\n",
    "\n",
    "        # map j to seconds value.\n",
    "        end_time = \"00:00:{:0>3d}\".format(int(j * num_seconds * 10))\n",
    "\n",
    "        # print the previous status.\n",
    "        print(start_time + \"-\" + (end_time) + \" : \" + previous_status)\n",
    "\n",
    "        # Update end-time\n",
    "        start_time = end_time\n",
    "        previous_status = current_status\n",
    "\n",
    "# Print the final summary.\n",
    "print(start_time + \"-\" + \"End\" + \" : \" + previous_status)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Step 7. Explore other relevant models!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You just learnt how pre-trained machine learning models can  identify metadata from workplace pictures (or snapshots of a video). This metadata can be used to set up alarms to detect non-compliance. \n",
    "\n",
    "Checkout these additional relevant models:\n",
    "1. [Person and Truck Detector](https://aws.amazon.com/marketplace/pp/prodview-mxkmbwcmojzg4?qid=1563549078039&sr=0-5&ref_=mlmp_gitdemo_indust) to identify trucks and people from an image.\n",
    "2. [Modjoul Geo Fence model](https://aws.amazon.com/marketplace/pp/prodview-bspkbdfyfj42e?qid=1567887787959&sr=0-4&ref_=mlmp_gitdemo_indust) informs an organization of employee and equipment location and the activities and movements within that location.\n",
    "3. [Modjoul Automotive Telematics Model](https://aws.amazon.com/marketplace/pp/prodview-cj46uchjavfa6?qid=1567887787959&sr=0-6&ref_=mlmp_gitdemo_indust) can identify aggressive events such as hard braking and hard acceleration, duration of driving and distance of driving.\n",
    "4. [Modjoul Asset Utilization Model](https://aws.amazon.com/marketplace/pp/prodview-6ay5xkpc6lqbi?qid=1567887787959&sr=0-1&ref_=mlmp_gitdemo_indust) to understand the utilization of heavy equipments such as back hoes, generators, dump trucks, etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 8. Cleanup "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, clean-up deployable models as well as endpoints from your account."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "construction_worker_detection_model.sagemaker_session.delete_endpoint(\n",
    "    construction_worker_detection_model_name\n",
    ")\n",
    "construction_worker_detection_model.sagemaker_session.delete_endpoint_config(\n",
    "    construction_worker_detection_model_name\n",
    ")\n",
    "construction_worker_detection_model.delete_model()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hard_hat_detection_model.sagemaker_session.delete_endpoint(hard_hat_detection_model_name)\n",
    "hard_hat_detection_model.sagemaker_session.delete_endpoint_config(hard_hat_detection_model_name)\n",
    "hard_hat_detection_model.delete_model()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ppe_detection_model.sagemaker_session.delete_endpoint(ppe_detection_model_name)\n",
    "ppe_detection_model.sagemaker_session.delete_endpoint_config(ppe_detection_model_name)\n",
    "ppe_detection_model.delete_model()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "machine_detection_model.sagemaker_session.delete_endpoint(machine_detection_model_name)\n",
    "machine_detection_model.sagemaker_session.delete_endpoint_config(machine_detection_model_name)\n",
    "machine_detection_model.delete_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you would like to unsubscribe to the model, follow these steps. Before you cancel the subscription, ensure that you do not have any [deployable model](https://console.aws.amazon.com/sagemaker/home#/models) created from the model package or using the algorithm. Note - You can find this information by looking at the container name associated with the model. \n",
    "\n",
    "**Steps to unsubscribe to product from AWS Marketplace**:\n",
    "1. Navigate to __Machine Learning__ tab on [__Your Software subscriptions page__](https://aws.amazon.com/marketplace/ai/library?productType=ml&ref_=mlmp_gitdemo_indust)\n",
    "2. Locate the listing that you would need to cancel subscription for, and then choose __Cancel Subscription__  to cancel the subscription.\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (Data Science)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/datascience-1.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}