A2I-Video-Transcription-with-Amazon-Transcribe.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Improving speech-to-text transcripts from Amazon Transcribe using custom vocabularies and Amazon Augmented AI\n",
    "\n",
    "\n",
    "\n",
    "This notebook accompanies the blog [Improving speech-to-text transcripts from Amazon Transcribe using custom vocabularies and Amazon Augmented AI](https://aws.amazon.com/blogs/machine-learning/improving-speech-to-text-transcripts-from-amazon-transcribe-using-custom-vocabularies-and-amazon-augmented-ai/).\n",
    "\n",
    "## Introduction\n",
    "When transcribing speech containing domain-specific terminologies in fields such as legal, financial, construction, higher education, or engineering, [Amazon Transcribe’s](https://aws.amazon.com/transcribe/) [custom vocabularies](https://docs.aws.amazon.com/transcribe/latest/dg/how-vocabulary.html) feature can improve transcription quality, especially on key technical terms. \n",
    "\n",
    "To use custom vocabularies with Amazon Transcribe, you need a list of domain-specific terms. Using a collection of videos or audio files (i.e., your dataset) that you want transcribed with high accuracy, you can send a portion of your dataset to Amazon Transcribe to identify terms it has difficulty with, indicated by low-confidence scores. You can use [Amazon Augmented AI (A2I)](https://aws.amazon.com/a2i/) to send these low-confidence predictions directly to a human to manually review and transcribe the terms. This walkthrough will demonstrate how you can process the results obtained from Amazon A2I to quickly to build a custom vocabulary, at scale.\n",
    "\n",
    "In summary, in this walkthrough you will:\n",
    "* Send a subset of videos to Amazon Transcribe to find terms that are difficult to transcribe.\n",
    "* Set up a human review workflow using Amazon A2I to send low-confidence predictions to your human workforce for manual review and transcription.\n",
    "* Create a custom vocabulary using the results obtained from human workers.\n",
    "* Test Amazon Transcribe on another subset of videos to assess the improvement in transcription quality. \n",
    "\n",
    "![mistranscription_before](https://aws-ml-blog.s3.amazonaws.com/artifacts/a2i-transcribe-custom-demo/images/mistranscription_custom_before.jpg)\n",
    "\n",
    "*Example of mis-transcribed annotation of the technical term, “an EC2 instance”. This term was transcribed as “Annecy two instance”.*\n",
    "\n",
    "![mistranscription_after](https://aws-ml-blog.s3.amazonaws.com/artifacts/a2i-transcribe-custom-demo/images/mistranscription_custom_after.jpg)\n",
    "\n",
    "*Example of correctly transcribed annotation of the technical term “an EC2 instance” after using Amazon A2I to build an Amazon Transcribe custom vocabulary and re-transcribing the video.*\n",
    "\n",
    "This walkthrough focuses on transcribing video content. You can modify the code provided to use audio files (such as MP3 files) by doing the following:\n",
    "\n",
    "* Upload audio files to your [Amazon Simple Storage Service](https://aws.amazon.com/s3/) (Amazon S3) bucket and using them in place of the video files provided.\n",
    "* Modify the button text and instructions in the worker task template provided in this walkthrough and tell workers to listen to and transcribe audio clips.\n",
    "\n",
    "## Solution Overview:\n",
    "\n",
    "The following diagram presents the solution architecture:\n",
    "\n",
    "![solution architecture](https://aws-ml-blog.s3.amazonaws.com/artifacts/a2i-transcribe-custom-demo/images/a2i_transcribe_solution_architecture.png)\n",
    "\n",
    "## Prerequisites\n",
    "\n",
    "To run this notebook, you can simply execute each cell in order. Before beginning, you'll need:\n",
    "\n",
    "* An AWS account.\n",
    "* An S3 bucket you can write to - please provide its name in `BUCKET`. The bucket must be in the same region as this SageMaker Notebook instance. All the video files involved in this demo will be stored in `s3://BUCKET/a2i_transcribe_demo`. All A2I results will be stored in `s3://BUCKET/a2i-results/`. All Amazon Trasncribe results will be stored in `s3://BUCKET/transcribe-results/`.\n",
    "\n",
    "To understand this use case, the following are also recommended:\n",
    "1. Basic understanding of AWS services [Amazon Transcribe](https://docs.aws.amazon.com/transcribe/latest/dg/what-is-transcribe.html) and Transcribe [custom vocabularies](https://docs.aws.amazon.com/transcribe/latest/dg/how-vocabulary.html), and the core components and workflow used by [Amazon A2I](https://docs.aws.amazon.com/sagemaker/latest/dg/a2i-use-augmented-ai-a2i-human-review-loops.html). The notebook uses the [AWS SDK for Python (Boto3)](https://aws.amazon.com/sdk-for-python/) to interact with these services.  \n",
    "2. Familiarity with Python and numpy.\n",
    "3. Basic familiarity with [AWS S3](https://docs.aws.amazon.com/AmazonS3/latest/dev/Welcome.html).\n",
    "\n",
    "This notebook has been tested on a SageMaker notebook instance. The runtimes given are approximated on an ml.t2.medium instance. You can run it on a local instance by first executing the cell below on SageMaker and then copying the name of the role to your local copy of the notebook.\n",
    "\n",
    "For more sample notebooks using A2I, visit this [Github repository](https://github.com/aws-samples/amazon-a2i-sample-jupyter-notebooks).\n",
    "\n",
    "Let's get started!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Install Latest SDKs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# First, let's get the latest installations of our dependencies\n",
    "!pip install --upgrade pip --quiet\n",
    "!pip install --upgrade awscli --quiet\n",
    "!pip install -U botocore --quiet\n",
    "!pip install boto3 --upgrade --quiet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import io\n",
    "import uuid\n",
    "import botocore\n",
    "import boto3\n",
    "import re\n",
    "import time\n",
    "import pprint\n",
    "import subprocess\n",
    "import json\n",
    "import sagemaker\n",
    "from sagemaker import get_execution_role\n",
    "from datetime import datetime, timezone"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Region, Bucket, and Paths\n",
    "Make sure all your resources are stored in the same region. You'll be using the same bucket for this entire walkthrough."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "BUCKET = '' #Add your bucket name here\n",
    "\n",
    "if(BUCKET==''):\n",
    "    BUCKET = sagemaker.Session().default_bucket()\n",
    "print(f'The S3 bucket used in this demo will be: {BUCKET}')\n",
    "\n",
    "OUTPUT_PATH_A2I = f's3://{BUCKET}/a2i-results'\n",
    "OUTPUT_PATH_TRANSCRIBE = f's3://{BUCKET}/transcribe-results'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sess = sagemaker.session.Session()\n",
    "role = sagemaker.get_execution_role()\n",
    "region = boto3.session.Session().region_name\n",
    "\n",
    "# Amazon S3 (S3) client\n",
    "s3 = boto3.client('s3', region)\n",
    "bucket_region = s3.head_bucket(Bucket=BUCKET)['ResponseMetadata']['HTTPHeaders']['x-amz-bucket-region']\n",
    "assert bucket_region == region, \"Your S3 bucket {} and this notebook need to be in the same region.\".format(BUCKET)\n",
    "\n",
    "# Amazon SageMaker client\n",
    "sagemaker_client = boto3.client('sagemaker')\n",
    "\n",
    "# Amazon Augment AI (A2I) client\n",
    "a2i = boto3.client('sagemaker-a2i-runtime')\n",
    "\n",
    "# Amazon Transcribe client\n",
    "transcribe_client = boto3.client(\"transcribe\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Roles and Permissions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the execution role that will be used to call Amazon Transcribe and Amazon A2I. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker import get_execution_role\n",
    "\n",
    "ROLE = get_execution_role()\n",
    "display(ROLE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add the following policies to this role in IAM:\n",
    "* AmazonAugmentedAIFullAccess\n",
    "* AmazonTranscribeFullAccess\n",
    "\n",
    "Your execution role has the AmazonSageMakerFullAccess policy attached. This gives Amazon SageMaker permission to access your resources in S3 if the bucket or objects have the word `sagemaker` in the name. If your S3 bucket listed in `BUCKET` does not have sagemaker in the name, you will need to add an S3 policy to your execution role to give your role permissions to access your data objects in S3. The following is an example of an S3 policy:\n",
    "\n",
    "```\n",
    "{\n",
    "    \"Version\": \"2012-10-17\",\n",
    "    \"Statement\": [\n",
    "        {\n",
    "            \"Effect\": \"Allow\",\n",
    "            \"Action\": [\n",
    "                \"s3:GetObject\"\n",
    "            ],\n",
    "            \"Resource\": [\n",
    "                \"arn:aws:s3:::my_input_bucket/*\"\n",
    "            ]\n",
    "        },\n",
    "        {\n",
    "            \"Effect\": \"Allow\",\n",
    "            \"Action\": [\n",
    "                \"s3:PutObject\"\n",
    "            ],\n",
    "            \"Resource\": [\n",
    "                \"arn:aws:s3:::my_output_bucket/*\"\n",
    "            ]\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: your default **Maximum CLI/API session duration** (MaxSessionDuration) for this execution role is 1 hour. It is recommended that you [increase MaxSessionDuration](https://docs.aws.amazon.com/IAM/latest/UserGuide/roles-managingrole-editing-console.html#roles-modify_max-session-duration) to 2 hours. If you take more than `MaxSessionDuration`-hours to complete the notebook, you may have to re-run previous code cells, and all code cells that define functions below. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Upload Sample Video to S3\n",
    "We'll be analyzing videos from the official AWS playlist on introductory SageMaker videos, also available on [YouTube](https://www.youtube.com/watch?v=uQc8Itd4UTs&list=PLhr1KZpdzukcOr_6j_zmSrvYnLUtgqsZz&index=1). Steps 1 and 2 of this demo walk through transcribing and viewing Amazon A2I tasks for a single video about Amazon SageMaker Jupyter Notebook instances. If you want to experiment with additional videos from this playlist in Steps 1 and 2, change the S3 URI used to download the videos. \n",
    "\n",
    "In Steps 3 and 4, you will analyze results for a larger dataset of four videos. The following table outlines the videos that are used in the notebook.\n",
    "\n",
    "| Video # | Video Title | File Name | How is it used in this demo? |\n",
    "|------------|-------|-----------------------------------------|------------------------------------|\n",
    "| 1 | Fully-Managed Notebook Instances with Amazon SageMaker - a Deep Dive      |   Fully-Managed Notebook Instances with Amazon SageMaker - a Deep Dive.mp4    | Perform the initial transcription and viewing sample Amazon A2I jobs in Steps 1 and 2. Build a custom vocabulary in Step 3  |\n",
    "| 2 | Built-in Machine Learning Algorithms with Amazon SageMaker - a Deep Dive   |  Built-in Machine Learning Algorithms with Amazon SageMaker - a Deep Dive.mp4  | Test transcription with the custom vocabulary in Step 4 |\n",
    "| 3 | Bring Your Own Custom ML Models with Amazon SageMaker         | Bring Your Own Custom ML Models with Amazon SageMaker.mp4 | Build a custom vocabulary in Step 3 |\n",
    "| 4 | Train Your ML Models Accurately with Amazon SageMaker | Train Your ML Models Accurately with Amazon SageMaker.mp4  | Test transcription with the custom vocabulary in Step 4  |\n",
    "\n",
    "\n",
    "Note that in Step 4, we’ll be referring to videos numbered one and three to be the **in-sample** videos, i.e., the videos that were used to build the custom vocabulary. Videos numbered two and four are the **out-sample** videos, i.e., videos that our workflow has not “seen” before and are used to test how well our methodology can generalize to (i.e., identify technical terms from) new videos."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -rf ./a2i_transcribe_demo*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to experiment with additional videos in Steps 1 and 2, change the S3 URI below to `s3://aws-ml-blog/artifacts/a2i-transcribe-custom-demo/additional-videos/`. This will download 4 videos."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# If you want to experiment with additional videos, change the S3 URI below from\n",
    "# \"s3://aws-ml-blog/artifacts/a2i-transcribe-custom-demo/transcribe-notebook-demo/\" to\n",
    "# \"s3://aws-ml-blog/artifacts/a2i-transcribe-custom-demo/additional-videos/\".\n",
    "# This will download 4 additional videos.\n",
    "!rm -rf ./a2i_transcribe_demo\n",
    "!mkdir ./a2i_transcribe_demo\n",
    "!aws s3 sync s3://aws-ml-blog/artifacts/a2i-transcribe-custom-demo/transcribe-notebook-demo/ ./a2i_transcribe_demo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash -s \"$BUCKET\"\n",
    "aws s3 cp ./a2i_transcribe_demo/ s3://$1/a2i_transcribe_demo/ --recursive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash -s \"$BUCKET\"\n",
    "aws s3 ls s3://$1/a2i_transcribe_demo/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture cap_out --no-stderr\n",
    "%%bash -s \"$BUCKET\"\n",
    "aws s3 ls s3://$1/a2i_transcribe_demo/ --recursive"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following cell will define the variable `all_videos`. This will determine the number of videos that you use to run this demo. If you want to change the number of videos used in this demo, copy and paste the output from this cell, and re-define `all_videos`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "VIDEOS = cap_out.stdout\n",
    "all_videos = re.findall(\"a2i_transcribe_demo/.*\",VIDEOS)\n",
    "for i in range(len(all_videos)):\n",
    "    all_videos[i] = all_videos[i].split('a2i_transcribe_demo/')[-1]\n",
    "all_videos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_videos = len(all_videos)\n",
    "num_videos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# You can give each transcription job any name. We attach a timestamp to each job name here\n",
    "# to prevent conflicting job names in case we need to re-run any jobs.\n",
    "now = datetime.now()\n",
    "time_now = now.strftime(\"%H.%M.%S\")\n",
    "\n",
    "# Path to folder\n",
    "folder_path = f\"s3://{BUCKET}/a2i_transcribe_demo/\"\n",
    "\n",
    "job_names = []\n",
    "for i in range(num_videos):\n",
    "    job_names.append(\"AWS-sage-vid-\" + str(i) + \"-\" + str(time_now))\n",
    "\n",
    "job_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll be transcribing the [video](https://www.youtube.com/watch?v=uQc8Itd4UTs) in the following cells. Feel free to experiment with additional videos we've provided, or your own content."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Here is a transcribe function\n",
    "def transcribe(job_name, job_uri, out_bucket, format=\"mp4\", vocab_name=None):\n",
    "    \"\"\"Transcribe a .wav or .mp4 file to text.\n",
    "    Args:\n",
    "        job_name (str): the name of the job that you specify;\n",
    "                        the output json will be job_name.json\n",
    "        job_uri (str): input path (in s3) to the file being transcribed\n",
    "        out_bucket (str): s3 bucket name that you want the output json\n",
    "                          to be placed in\n",
    "        format (str): mp4 or wav for input file format;\n",
    "                      defaults to mp4\n",
    "        vocab_name (str): name of custom vocabulary used;\n",
    "                          optional, defaults to None\n",
    "    \"\"\"\n",
    "    \n",
    "    if format not in ['mp3','mp4','wav','flac']:\n",
    "        print(\"Invalid format\")\n",
    "        return\n",
    "\n",
    "    try:\n",
    "        print(\"------\" + format)\n",
    "        if vocab_name is None:\n",
    "            transcribe_client.start_transcription_job(\n",
    "                TranscriptionJobName=job_name,\n",
    "                Media={\"MediaFileUri\": job_uri},\n",
    "                MediaFormat=format,\n",
    "                LanguageCode=\"en-US\",\n",
    "                OutputBucketName=out_bucket,\n",
    "            )\n",
    "        else:\n",
    "            transcribe_client.start_transcription_job(\n",
    "                TranscriptionJobName=job_name,\n",
    "                Media={\"MediaFileUri\": job_uri},\n",
    "                MediaFormat=format,\n",
    "                LanguageCode=\"en-US\",\n",
    "                OutputBucketName=out_bucket,\n",
    "                Settings={'VocabularyName': vocab_name}\n",
    "            )\n",
    "        \n",
    "        time.sleep(2)\n",
    "        \n",
    "        print(transcribe_client.get_transcription_job(TranscriptionJobName=job_name))\n",
    "\n",
    "    except Exception as e:\n",
    "        print(e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Start a transcription job\n",
    "for i in range(len(job_names)):\n",
    "    transcribe(job_names[i], folder_path+all_videos[i], BUCKET)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check transcription job statuses\n",
    "\n",
    "Wait until the status displays `COMPLETED` before moving on to the next cells. A transcription job for a 10-15 minute video typically takes roughly 5 minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Wait for the status of the transcription job to finish\n",
    "while True:\n",
    "    response = transcribe_client.get_transcription_job(\n",
    "        TranscriptionJobName=job_names[0] # You can change this job index if you have more jobs\n",
    "    )\n",
    "    status = response['TranscriptionJob']['TranscriptionJobStatus']\n",
    "    if status in ['COMPLETED', 'FAILED']:\n",
    "        print(status)\n",
    "        break\n",
    "    print(\"Not ready yet...\")\n",
    "    time.sleep(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Retrieve and Parse Transcription Results\n",
    "\n",
    "When the transcription job finishes, the results will be stored in your specified S3 bucket as an output JSON file called “YOUR_JOB_NAME.json.” You can use the following function to retrieve your results, and parse them into sentences with time stamps, confidence scores, and other useful representations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_transcript_text_and_timestamps(bucket_name, file_name):\n",
    "    \"\"\"take json file from S3 bucket and returns a tuple of:\n",
    "       entire transcript, list object of tuples of timestamp and individual sentences\n",
    "    \n",
    "    Args:\n",
    "        bucket_name (str): name of s3 bucket\n",
    "        file_name (str): name of file\n",
    "    Returns:\n",
    "        (\n",
    "        entire_transcript: str,\n",
    "        sentences_and_times: [ {start_time (sec) : float,\n",
    "                                end_time (sec)   : float,\n",
    "                                sentence         : str,\n",
    "                                min_confidence   : float (minimum confidence score of that sentence)\n",
    "                                } ],\n",
    "        confidences:  [ {start_time (sec) : float,\n",
    "                         end_time (sec)   : float,\n",
    "                         content          : str, (single word/phrase)\n",
    "                         confidence       : float (confidence score of the word/phrase)\n",
    "                         } ],\n",
    "        scores: list of confidence scores\n",
    "        )\n",
    "    \"\"\"\n",
    "    s3_clientobj = s3.get_object(Bucket=bucket_name, Key=file_name)\n",
    "    s3_clientdata = s3_clientobj[\"Body\"].read().decode(\"utf-8\")\n",
    "\n",
    "    original = json.loads(s3_clientdata)\n",
    "    items = original[\"results\"][\"items\"]\n",
    "    entire_transcript = original[\"results\"][\"transcripts\"]\n",
    "\n",
    "    sentences_and_times = []\n",
    "    temp_sentence = \"\"\n",
    "    temp_start_time = 0\n",
    "    temp_min_confidence = 1.0\n",
    "    newSentence = True\n",
    "    \n",
    "    confidences = []\n",
    "    scores = []\n",
    "\n",
    "    i = 0\n",
    "    for item in items:\n",
    "        # always add the word\n",
    "        if item[\"type\"] == \"punctuation\":\n",
    "            temp_sentence = (\n",
    "                temp_sentence.strip() + item[\"alternatives\"][0][\"content\"] + \" \"\n",
    "            )\n",
    "        else:\n",
    "            temp_sentence = temp_sentence + item[\"alternatives\"][0][\"content\"] + \" \"\n",
    "            temp_min_confidence = min(temp_min_confidence,\n",
    "                                      float(item[\"alternatives\"][0][\"confidence\"]))\n",
    "            confidences.append({\"start_time\": float(item[\"start_time\"]),\n",
    "                                \"end_time\": float(item[\"end_time\"]),\n",
    "                                \"content\": item[\"alternatives\"][0][\"content\"],\n",
    "                                \"confidence\": float(item[\"alternatives\"][0][\"confidence\"])\n",
    "                               })\n",
    "            scores.append(float(item[\"alternatives\"][0][\"confidence\"]))\n",
    "\n",
    "        # if this is a new sentence, and it starts with a word, save the time\n",
    "        if newSentence == True:\n",
    "            if item[\"type\"] == \"pronunciation\":\n",
    "                temp_start_time = float(item[\"start_time\"])\n",
    "            newSentence = False\n",
    "        # else, keep going until you hit a punctuation\n",
    "        else:\n",
    "            if (\n",
    "                item[\"type\"] == \"punctuation\"\n",
    "                and item[\"alternatives\"][0][\"content\"] != \",\"\n",
    "            ):\n",
    "                # end time of sentence is end_time of previous word\n",
    "                end_time = items[i-1][\"end_time\"] if i-1 >= 0 else items[0][\"end_time\"]\n",
    "                sentences_and_times.append(\n",
    "                    {\"start_time\": temp_start_time,\n",
    "                     \"end_time\": end_time,\n",
    "                     \"sentence\": temp_sentence.strip(),\n",
    "                     \"min_confidence\": temp_min_confidence\n",
    "                    }\n",
    "                )\n",
    "                # reset the temp sentence and relevant variables\n",
    "                newSentence = True\n",
    "                temp_sentence = \"\"\n",
    "                temp_min_confidence = 1.0\n",
    "                \n",
    "        i = i + 1\n",
    "        \n",
    "    sentences_and_times.append(\n",
    "                    {\"start_time\": temp_start_time,\n",
    "                     \"end_time\": confidences[-1][\"end_time\"],\n",
    "                     \"sentence\": temp_sentence.strip(),\n",
    "                     \"min_confidence\": temp_min_confidence\n",
    "                    }\n",
    "                )\n",
    "    return entire_transcript, sentences_and_times, confidences, scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's parse each video and store the results. Note that there may be a slight time delay before the JSON files are available in your S3 bucket. If the following cell fails, please wait 5 seconds and try rerunning it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_entire_transcript = []\n",
    "all_sentences_and_times = []\n",
    "all_confidences = []\n",
    "all_scores = []\n",
    "for i in range(num_videos):\n",
    "    print(f\"{i}: Parsing {job_names[i]}.json\")\n",
    "    entire_transcript_1, sentences_and_times_1, confidences_1, scores_1 = get_transcript_text_and_timestamps(BUCKET,job_names[i]+\".json\")\n",
    "    all_entire_transcript.append(entire_transcript_1)\n",
    "    all_sentences_and_times.append(sentences_and_times_1)\n",
    "    all_confidences.append(confidences_1)\n",
    "    all_scores.append(scores_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sanity check -- the first word transcribed should be \"Hi.\" and corresponding printed output should look something like:\n",
    "# \"0: {'start_time': 0.54, 'end_time': '1.03', 'sentence': 'Hi.', 'min_confidence': 1.0}\"\n",
    "for i in range(num_videos):\n",
    "    print(f\"{i}: {all_sentences_and_times[i][0]}\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save the original transcripts to txt files\n",
    "Let's save the full transcripts, as we'll be using this later for comparisons."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "i = 1\n",
    "for sentences_times in all_sentences_and_times:\n",
    "    file0 = open(f\"original_transcript_\"+str(uuid.uuid4())+\".txt\",\"w\") \n",
    "    for tup in sentences_times:\n",
    "        file0.write(tup['sentence'] + \"\\n\") \n",
    "    file0.close()\n",
    "    i = i + 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%%bash -s \"$BUCKET\"\n",
    "rm -rf transcript_text_files\n",
    "mkdir transcript_text_files\n",
    "cp original_transcript_*.txt transcript_text_files/\n",
    "aws s3 cp ./transcript_text_files s3://$1/transcribe-results/ --recursive"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Histogram of confidence scores\n",
    "Let’s take a look at the distribution of confidence scores. A majority of words in the video were transcribe with a confidence score greater than .90. You may have to run the following cell twice to see the plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "plt.style.use('ggplot')\n",
    "\n",
    "flat_scores_list = [j for sub in all_scores for j in sub] \n",
    "\n",
    "plt.xlim([min(flat_scores_list)-0.1, max(flat_scores_list)+0.1])\n",
    "plt.hist(flat_scores_list, bins=20, alpha=0.5)\n",
    "plt.title('Distribution of confidence scores')\n",
    "plt.xlabel('Confidence score')\n",
    "plt.ylabel('Frequency')\n",
    "\n",
    "plt.show()\n",
    "# If you don't see a plot initially, you may need to rerun this cell."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Histogram of low confidence scores\n",
    "Let’s filter out the high confidence scores to take a closer look at the lower ones. You can experiment with different thresholds to see how many words fall below that threshold. For this demo, we use a threshold of 0.4 which corresponds to 16 words below this threshold. Sequences of words with a term under this threshold will be sent to human review.\n",
    "\n",
    "As you experiment with different thresholds and observe the number of tasks it created in the A2I workflow, you'll notice that there will be a tradeoff between the number of mis-transcriptions you want to catch and the amount of time you're willing to devote to corrections. In other words, using a higher threshold would capture a greater percentage of mis-transcriptions, but it would also increase the number of false positives -- low-confidence transriptions that don't actually contain any important technical term mis-transcriptions. The good news is that you can use this workflow to quickly experiment with as many different threshold values as you'd like, before sending it to your workforce for human review."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "THRESHOLD = 0.4\n",
    "\n",
    "# Filter scores that are less than THRESHOLD\n",
    "all_bad_scores = [i for i in flat_scores_list if i < THRESHOLD]\n",
    "print(f\"There are {len(all_bad_scores)} words that have confidence score less than {THRESHOLD}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.xlim([min(all_bad_scores)-0.1, max(all_bad_scores)+0.1])\n",
    "plt.hist(all_bad_scores, bins=20, alpha=0.5)\n",
    "plt.title(f'Distribution of confidence scores less than {THRESHOLD}')\n",
    "plt.xlabel('Confidence score')\n",
    "plt.ylabel('Frequency')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that there is a nontrivial number of words classified with low confidence. As we’ll see later, technical terms are more often mis-transcribed, so it’s important that we correct those mistakes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Create human review workflows with A2I\n",
    "\n",
    "Our next step is create a human review workflow that sends low confidence scores to human reviewers and then retrieves the corrected transcription they provide. This section contains the following steps:\n",
    "\n",
    "1. Create a work task template that will be displayed to workers for every task. The template will be rendered with input data you provide, instructions to workers, and interactive tools to help workers complete your tasks.\n",
    "2. Create a human review workflow, also called a flow definition. You use the flow definition to configure details about your human workforce and the human tasks they are assigned.\n",
    "3. Create a human loop to start the human review workflow, sending data for human review as needed. In this example, you use a custom task type and start human loop tasks using the [Amazon A2I Runtime API](https://docs.aws.amazon.com/augmented-ai/2019-11-07/APIReference/Welcome.html). Each time `StartHumanLoop` is called, a task is sent to human reviewers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a Private Work Team \n",
    "\n",
    "\n",
    "A workforce is the group of workers that you have selected to label your dataset. You can choose either the Amazon Mechanical Turk workforce, a vendor-managed workforce, or you can create your own private workforce for human reviews. Whichever workforce type you choose, Amazon Augmented AI takes care of sending tasks to workers.\n",
    "\n",
    "For this demo, it is recommended that you create a private workforce and add yourself to the work team. This will allow you to send the A2I human review tasks to yourself so that you can preview the worker UI and complete this demo. \n",
    "\n",
    "To create and manage your private workforce, you can use the Labeling workforces page in the Amazon SageMaker console. When following the instructions below, you will have the option to create a private workforce by entering worker emails or importing a pre-existing workforce from an Amazon Cognito user pool. To import a workforce, see Create a Private Workforce (Amazon Cognito Console). \n",
    "\n",
    "### To add yourself to an existing private workforce\n",
    "\n",
    "If you have already created a private workforce in the same AWS Region as this notebook instance, [add yourself to a private team](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-workforce-management-private-console.html#add-remove-workers-sm) and copy and paste that work team ARN to set `WORKTEAM_ARN`.\n",
    "\n",
    "Once you add yourself to the private workteam, you will recieve an email notifying you that you've been invited to the work team. Use the link in that email to reset your password and sign in to the worker portal. The worker portal is where the A2I human review tasks that you create in the following cells will appear. \n",
    "\n",
    "### To create a private workforce using worker emails:\n",
    "\n",
    "You can create a private work team in the Amazon SageMaker console (approximately 15 minutes). \n",
    "\n",
    "* Open the **Labeling workforces page** page in the Amazon SageMaker console: https://console.aws.amazon.com/sagemaker/groundtruth#/labeling-workforces.\n",
    "\n",
    "\n",
    "* If this notebook instance is not in N. Virginia (us-east-1), [change your AWS Region](https://docs.aws.amazon.com/awsconsolehelpdocs/latest/gsg/getting-started.html#select-region) to the same region that this notebook instance is in. \n",
    "\n",
    "\n",
    "* In the navigation pane, choose **Labeling workforces**.\n",
    "\n",
    "\n",
    "* Choose **Private**, then choose **Create private team**.\n",
    "\n",
    "\n",
    "* Choose **Invite new workers by email**.\n",
    "\n",
    "\n",
    "* Add your email address and any others that you want to recieve the A2I human review tasks created in this demo. \n",
    "    * You will recieve an email notifying you that you've been invited to the work team. Use the link in that email to reset your password and sign in to the worker portal. The worker portal is where the A2I human review tasks that you create in the following cells will appear. \n",
    "\n",
    "\n",
    "* Enter an organization name and contact email.\n",
    "\n",
    "\n",
    "* (Optionally) choose an SNS topic to subscribe the team to so workers are notified by email when new Ground Truth labeling jobs become available.\n",
    "\n",
    "\n",
    "* Click the Create private team button.\n",
    "\n",
    "After you import your private workforce, refresh the page. On the Private workforce summary page, you'll see your work team ARN. Enter this ARN in the following cell to set `WORKTEAM_ARN`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "WORKTEAM_ARN= \"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Resources for an A2I Human Review\n",
    "Now let's create the resources we'll need to build our human review workflow."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Human Task UI\n",
    "\n",
    "Amazon A2I uses Liquid, an open-source template language that can be used to “inject” data dynamically into HTML files.\n",
    "\n",
    "In this walkthrough, we want for each task to enable a human reviewer to watch a section of the video and transcribe the speech they hear. The HTML template consists of three main parts:\n",
    "\n",
    "1. A video player with a replay button that only allows the reviewer to play the specific subsection\n",
    "2. A form for the reviewer to type and submit what they hear\n",
    "3. Logic written in JavaScript to give the replay button its intended functionality\n",
    "\n",
    "For over 60 other pre-built UIs, check out this [repository](https://github.com/aws-samples/amazon-a2i-sample-task-uis).\n",
    "\n",
    "Here’s the template you’ll be using (skip ahead to the \"Wait For Workers to Complete Task\" section to preview what this template looks like):\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "template = r\"\"\"\n",
    "<head>\n",
    "    <style>\n",
    "        h1 {\n",
    "            color: black;\n",
    "            font-family: verdana;\n",
    "            font-size: 150%;\n",
    "        }\n",
    "    </style>\n",
    "</head>\n",
    "<script src=\"https://assets.crowd.aws/crowd-html-elements.js\"></script>\n",
    "\n",
    "<crowd-form>\n",
    "    <video id=\"this_vid\">\n",
    "        <source src=\"{{ task.input.filePath | grant_read_access }}\"\n",
    "            type=\"audio/mp4\">\n",
    "        Your browser does not support the audio element.\n",
    "    </video>\n",
    "    <br />\n",
    "    <br />\n",
    "    <crowd-button onclick=\"onClick(); return false;\"><h1> Click to play video section!</h1></crowd-button>\n",
    "    <br />\n",
    "    Video title: <strong>{{ task.input.video_title }}</strong>\n",
    "    <br />\n",
    "\n",
    "    <h3>Instructions</h3>\n",
    "    <p>Transcribe the audio clip </p>\n",
    "    <p>The original transcript is <strong>\"{{ task.input.original_words }}\"</strong>.\n",
    "    If the text matches the audio, you can copy and paste the same transcription.</p>\n",
    "    <p>Ignore \"umms\", \"hmms\", \"uhs\" and other non-textual phrases.\n",
    "    If a word is cut off in the beginning or end of the video clip, you do NOT need to transcribe that word.\n",
    "    You also do NOT need to transcribe punctuation at the end of clauses or sentences.\n",
    "    However, apostrophes and punctuation used in technical terms should still be included, such as \"Denny's\" or \"file_name.txt\"</p>\n",
    "    <p><strong>Important:</strong> If you encounter a technical term that has multiple words,\n",
    "    please <strong>hyphenate</strong> those words together. For example, \"k nearest neighbors\" should be transcribed as \"k-nearest-neighbors.\"</p>\n",
    "    <p>Click the space below to start typing.</p>\n",
    "    <crowd-text-area name=\"transcription\" rows=\"2\" label=\"Your transcription\" placeholder=\"Please enter the transcribed text.\"></crowd-text-area>\n",
    "\n",
    "    <full-instructions header=\"Transcription Instructions\">\n",
    "        <h2>Instructions</h2>\n",
    "        <p>Click the play button and listen carefully to the audio clip. Type what you hear in the box\n",
    "            below. Replay the clip by clicking the button again, as many times as needed.</p>\n",
    "    </full-instructions>\n",
    "\n",
    "</crowd-form>\n",
    "\n",
    "<script>\n",
    "    var video = document.getElementById('this_vid');\n",
    "    video.onloadedmetadata = function() {\n",
    "        video.currentTime = {{ task.input.start_time }};\n",
    "    };\n",
    "    function onClick() {\n",
    "        video.pause();\n",
    "        video.currentTime = {{ task.input.start_time }};\n",
    "        video.play();\n",
    "        video.ontimeupdate = function () {\n",
    "            if (video.currentTime >= {{ task.input.end_time }}) {\n",
    "                video.pause()\n",
    "            }\n",
    "        }\n",
    "    }\n",
    "</script>\n",
    "\"\"\"\n",
    "\n",
    "def create_task_ui():\n",
    "    '''\n",
    "    Creates a Human Task UI resource.\n",
    "\n",
    "    Returns:\n",
    "    struct: HumanTaskUiArn\n",
    "    '''\n",
    "    response = sagemaker_client.create_human_task_ui(\n",
    "        HumanTaskUiName=taskUIName,\n",
    "        UiTemplate={'Content': template})\n",
    "    return response"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `{{ task.input.filePath | grant_read_access }}` field grants access to and display a video using a path to the video’s location in an S3 bucket. To prevent the reviewer from navigating to irrelevant sections of the video, the `controls` parameter is omitted from the video tag and a single replay button is included to control which section can be replayed.\n",
    "\n",
    "Below the video player, the `<crowd-text-area>` HTML tag creates a submission form that your reviewer will use to type and submit.\n",
    "\n",
    "At the end of the HTML snippet, the `<script>` tag contains the logic for the replay button. The `{{ task.input.start_time }}` and `{{ task.input.end_time }}` fields allow you to inject the start and end times of the video subsection you want transcribed for the current task.\n",
    "\n",
    "Now let's create a Human Task UI resource."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Task UI name - this value is unique per account and region. You can also provide your own value here.\n",
    "taskUIName = 'ui-transcribe-' + str(uuid.uuid4()) \n",
    "\n",
    "# Create task UI\n",
    "humanTaskUiResponse = create_task_ui()\n",
    "humanTaskUiArn = humanTaskUiResponse['HumanTaskUiArn']\n",
    "print(humanTaskUiArn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Flow Definition\n",
    "\n",
    "In this section, we're going to create a flow definition. Flow Definitions allow us to specify:\n",
    "\n",
    "* The workforce that your tasks will be sent to.\n",
    "* The instructions that your workforce will receive. This is called a worker task template.\n",
    "* The configuration of your worker tasks, including the number of workers that receive a task and time limits to complete tasks.\n",
    "* Where your output data will be stored.\n",
    "\n",
    "This demo is going to use the API, but you can optionally create this workflow definition in the console as well.\n",
    "\n",
    "For more details and instructions, see [Create a Flow Definition](https://docs.aws.amazon.com/sagemaker/latest/dg/a2i-create-flow-definition.html).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Flow definition name - this value is unique per account and region. You can also provide your own value here.\n",
    "flowDefinitionName = 'demo-flowdef-transcribe-' + str(uuid.uuid4()) \n",
    "\n",
    "create_workflow_definition_response = sagemaker_client.create_flow_definition(\n",
    "        FlowDefinitionName= flowDefinitionName,\n",
    "        RoleArn= ROLE,\n",
    "        HumanLoopConfig= {\n",
    "            \"WorkteamArn\": WORKTEAM_ARN,\n",
    "            \"HumanTaskUiArn\": humanTaskUiArn,\n",
    "            \"TaskCount\": 1,\n",
    "            \"TaskDescription\": \"Identify the word(s) spoken in the provided audio clip\",\n",
    "            \"TaskTitle\": \"DEMO: Determine Words/Phrases of Audio Clip: \" + str(datetime.now())\n",
    "        },\n",
    "        OutputConfig={\n",
    "            \"S3OutputPath\" : OUTPUT_PATH_A2I\n",
    "        }\n",
    "    )\n",
    "flowDefinitionArn = create_workflow_definition_response['FlowDefinitionArn'] # let's save this ARN for future use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Describe flow definition - status should be active\n",
    "for x in range(60):\n",
    "    describeFlowDefinitionResponse = sagemaker_client.describe_flow_definition(FlowDefinitionName=flowDefinitionName)\n",
    "    print(describeFlowDefinitionResponse['FlowDefinitionStatus'])\n",
    "    if (describeFlowDefinitionResponse['FlowDefinitionStatus'] == 'Active'):\n",
    "        print(\"Flow Definition is active\")\n",
    "        break\n",
    "    time.sleep(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Human Loops\n",
    "### Sending sequences of words/phrases of low confidence for review\n",
    "After setting up our Flow Definition, we're ready to use Amazon Transcribe and initiate human loops. While iterating through the list of transcribed words and their confidence scores, we create a HumanLoop task whenever the confidence score is below some threshold, `CONFIDENCE_SCORE_THRESHOLD`.\n",
    "\n",
    "An important thing to consider is how we deal with a low-confidence word that is part of a phrase that was also mis-transcribed. To handle these cases, let’s write a function that gets the sequence of words centered about a given index, and the sequence's starting and ending timestamps:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use this function to get the words near a word with poor confidence,\n",
    "# since it is possible that the transcription also mis-transcribed nearby words/phrases\n",
    "def get_word_neighbors(words, index, margin):\n",
    "    \"\"\"\n",
    "    gets the words transcribe found at most `margin` away from the input index\n",
    "    Returns:\n",
    "        list: words at most 3 away from the input index\n",
    "        int: starting time of the first word in the list\n",
    "        int: ending time of the last word in the list\n",
    "    \"\"\"\n",
    "    i = max(0, index - margin)\n",
    "    j = min(len(words) - 1, index + margin)\n",
    "    return words[i: j + 1], words[i][\"start_time\"], words[j][\"end_time\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, for every word we encounter with low confidence, we send its associated sequence of neighboring words for human review:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sample data, human loop started\n",
    "human_loops_started = []\n",
    "CONFIDENCE_SCORE_THRESHOLD = THRESHOLD\n",
    "MARGIN = 3\n",
    "\n",
    "count = 0\n",
    "for index in range(num_videos):\n",
    "    this_uri = folder_path+all_videos[index]\n",
    "    this_confidences = all_confidences[index]\n",
    "    \n",
    "    print(\"========= \" + all_videos[index] + \" =========\")\n",
    "    \n",
    "    i = 0\n",
    "    while i < len(this_confidences):\n",
    "        word = this_confidences[i][\"content\"]\n",
    "        neighbors, start_time, end_time = get_word_neighbors(this_confidences, i, MARGIN)\n",
    "\n",
    "        # Our condition for when we want to engage a human for review\n",
    "        if (this_confidences[i][\"confidence\"] < CONFIDENCE_SCORE_THRESHOLD):\n",
    "\n",
    "            # get the original sequence of words\n",
    "            sequence = \"\"\n",
    "            for block in neighbors:\n",
    "                sequence += block['content'] + \" \"\n",
    "\n",
    "            humanLoopName = str(uuid.uuid4())\n",
    "            # \"initialValue\": word,\n",
    "            inputContent = {\n",
    "                \"filePath\": this_uri,\n",
    "                \"start_time\": start_time,\n",
    "                \"end_time\": end_time,\n",
    "                \"original_words\": sequence,\n",
    "                \"video_title\": all_videos[index]\n",
    "            }\n",
    "            start_loop_response = a2i.start_human_loop(\n",
    "                HumanLoopName=humanLoopName,\n",
    "                FlowDefinitionArn=flowDefinitionArn,\n",
    "                HumanLoopInput={\n",
    "                    \"InputContent\": json.dumps(inputContent)\n",
    "                }\n",
    "            )\n",
    "            human_loops_started.append(humanLoopName)\n",
    "            # print(f'Confidence score of {obj[\"confidence\"]} is less than the threshold of {CONFIDENCE_SCORE_THRESHOLD}')\n",
    "            # print(f'Starting human loop with name: {humanLoopName}')\n",
    "            # print(f'Sending words from times {start_time} to {end_time} to review')\n",
    "            print(f'The original transcription is \"{sequence}\"\\n')\n",
    "            \n",
    "            count = count + 1\n",
    "            \n",
    "            # Advance to next word after the margin away from the low-confidence word\n",
    "            i = i + MARGIN + 1\n",
    "        else:\n",
    "            # No human loop created, advance to next word.\n",
    "            i = i + 1\n",
    "            # print(f'Confidence score of {obj[\"confidence\"]} is above threshold of {CONFIDENCE_SCORE_THRESHOLD}')\n",
    "            # print('No human loop created. \\n')\n",
    "        \n",
    "\n",
    "print(f'Number of tasks sent to review: {count}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's also save the name of each human loop, in case we need to retrieve them later after shutting down this notebook instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "file_hl = open(\"human_loops_names.txt\",\"w\") \n",
    "for name in human_loops_started:\n",
    "    file_hl.write(name + \"\\n\") \n",
    "file_hl.close()\n",
    "!cat human_loops_names.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check Status of Human Loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for human_loop_name in human_loops_started:\n",
    "    resp = a2i.describe_human_loop(HumanLoopName=human_loop_name)\n",
    "    print(f'HumanLoop Name: {human_loop_name}')\n",
    "    print(f'HumanLoop Status: {resp[\"HumanLoopStatus\"]}')\n",
    "    # print(f'HumanLoop Output Destination: {resp[\"HumanLoopOutput\"]}')\n",
    "    # print('\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Wait For Workers to Complete Task\n",
    "We display the link to the private worker portal here for convenience. Check your email (or the email your worker provided) for the login information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Wait For Workers to Complete Task\n",
    "workteamName = WORKTEAM_ARN[WORKTEAM_ARN.rfind('/') + 1:]\n",
    "print(\"Navigate to the private worker portal and do the tasks. Make sure you've invited yourself to your workteam!\")\n",
    "print('https://' + sagemaker_client.describe_workteam(WorkteamName=workteamName)['Workteam']['SubDomain'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's an example of how a transcription human review task would appear to a worker:\n",
    "\n",
    "![example task](https://aws-ml-blog.s3.amazonaws.com/artifacts/a2i-transcribe-custom-demo/images/a2i_transcribe_human_review_task_example.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check Status of Human Loop Again\n",
    "Human loop statuses that are completed will display `Completed` below. Note that it is not required to complete all human review tasks before continuing. Having 3-5 completed tasks is typically sufficient to see how technical terms can be extracted from the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "completed_human_loops = []\n",
    "for human_loop_name in human_loops_started:\n",
    "    resp = a2i.describe_human_loop(HumanLoopName=human_loop_name)\n",
    "    print(f'HumanLoop Name: {human_loop_name}')\n",
    "    print(f'HumanLoop Status: {resp[\"HumanLoopStatus\"]}')\n",
    "    # print(f'HumanLoop Output Destination: {resp[\"HumanLoopOutput\"]}')\n",
    "    # print('\\n')\n",
    "    \n",
    "    if resp[\"HumanLoopStatus\"] == \"Completed\":\n",
    "        completed_human_loops.append(resp)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View Task Results\n",
    "\n",
    "Once work is completed, Amazon A2I stores results in your S3 bucket and sends a Cloudwatch event. Your results should be available in the S3 `OUTPUT_PATH` when all work is completed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "import pprint\n",
    "\n",
    "pp = pprint.PrettyPrinter(indent=4)\n",
    "\n",
    "for resp in completed_human_loops:\n",
    "    splitted_string = re.split('s3://' +  BUCKET + '/', resp['HumanLoopOutput']['OutputS3Uri'])\n",
    "    output_bucket_key = splitted_string[1]\n",
    "\n",
    "    response = s3.get_object(Bucket=BUCKET, Key=output_bucket_key)\n",
    "    content = response[\"Body\"].read()\n",
    "    json_output = json.loads(content)\n",
    "    pp.pprint(json_output['humanAnswers'][0]['answerContent'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Build custom vocabularies using A2I results\n",
    "\n",
    "Using the corrected transcriptions from our human reviewers, let’s parse through these results to identify the domain-specific terms that we want to add to a custom vocabulary. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Retrieve A2I results\n",
    "To get the technical terms identified by human review, we first accumulate all human-reviewed words into a list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from string import punctuation\n",
    "\n",
    "corrected_words = []\n",
    "\n",
    "for resp in completed_human_loops:\n",
    "    splitted_string = re.split('s3://' +  BUCKET + '/', resp['HumanLoopOutput']['OutputS3Uri'])\n",
    "    output_bucket_key = splitted_string[1]\n",
    "\n",
    "    response = s3.get_object(Bucket=BUCKET, Key=output_bucket_key)\n",
    "    content = response[\"Body\"].read()\n",
    "    json_output = json.loads(content)\n",
    "    \n",
    "    # check if a transcription was submitted\n",
    "    if 'transcription' in json_output['humanAnswers'][0]['answerContent']:\n",
    "        # add the human-reviewed answers split by spaces\n",
    "        corrected_words += [word.strip(punctuation).lower() for word in json_output['humanAnswers'][0]['answerContent']['transcription'].split(\" \")]\n",
    "\n",
    "# Sanity check!\n",
    "print(corrected_words[0:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Filtering out common English words\n",
    "Now, we want to parse through these words and look for “uncommon” English words. An easy way to do this is to use a large English corpus and verify whether each of our human-reviewed words exists in this corpus:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create dictionary of English words\n",
    "# Note that this corpus of words is not 100% exhaustive\n",
    "import nltk\n",
    "nltk.download('words')\n",
    "\n",
    "from nltk.corpus import words\n",
    "my_dict=set(words.words()) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function for removing contractions\n",
    "# Source: https://en.wikipedia.org/wiki/Wikipedia:List_of_English_contractions\n",
    "contractions = { \n",
    "\"ain't\": \"am not / are not / is not / has not / have not\",\n",
    "\"aren't\": \"are not / am not\",\n",
    "\"can't\": \"cannot\",\n",
    "\"can't've\": \"cannot have\",\n",
    "\"'cause\": \"because\",\n",
    "\"could've\": \"could have\",\n",
    "\"couldn't\": \"could not\",\n",
    "\"couldn't've\": \"could not have\",\n",
    "\"didn't\": \"did not\",\n",
    "\"doesn't\": \"does not\",\n",
    "\"don't\": \"do not\",\n",
    "\"hadn't\": \"had not\",\n",
    "\"hadn't've\": \"had not have\",\n",
    "\"hasn't\": \"has not\",\n",
    "\"haven't\": \"have not\",\n",
    "\"he'd\": \"he had / he would\",\n",
    "\"he'd've\": \"he would have\",\n",
    "\"he'll\": \"he shall / he will\",\n",
    "\"he'll've\": \"he shall have / he will have\",\n",
    "\"he's\": \"he has / he is\",\n",
    "\"how'd\": \"how did\",\n",
    "\"how'd'y\": \"how do you\",\n",
    "\"how'll\": \"how will\",\n",
    "\"how's\": \"how has / how is / how does\",\n",
    "\"I'd\": \"I had / I would\",\n",
    "\"I'd've\": \"I would have\",\n",
    "\"I'll\": \"I shall / I will\",\n",
    "\"I'll've\": \"I shall have / I will have\",\n",
    "\"I'm\": \"I am\",\n",
    "\"I've\": \"I have\",\n",
    "\"isn't\": \"is not\",\n",
    "\"it'd\": \"it had / it would\",\n",
    "\"it'd've\": \"it would have\",\n",
    "\"it'll\": \"it shall / it will\",\n",
    "\"it'll've\": \"it shall have / it will have\",\n",
    "\"it's\": \"it has / it is\",\n",
    "\"let's\": \"let us\",\n",
    "\"ma'am\": \"madam\",\n",
    "\"mayn't\": \"may not\",\n",
    "\"might've\": \"might have\",\n",
    "\"mightn't\": \"might not\",\n",
    "\"mightn't've\": \"might not have\",\n",
    "\"must've\": \"must have\",\n",
    "\"mustn't\": \"must not\",\n",
    "\"mustn't've\": \"must not have\",\n",
    "\"needn't\": \"need not\",\n",
    "\"needn't've\": \"need not have\",\n",
    "\"o'clock\": \"of the clock\",\n",
    "\"oughtn't\": \"ought not\",\n",
    "\"oughtn't've\": \"ought not have\",\n",
    "\"shan't\": \"shall not\",\n",
    "\"sha'n't\": \"shall not\",\n",
    "\"shan't've\": \"shall not have\",\n",
    "\"she'd\": \"she had / she would\",\n",
    "\"she'd've\": \"she would have\",\n",
    "\"she'll\": \"she shall / she will\",\n",
    "\"she'll've\": \"she shall have / she will have\",\n",
    "\"she's\": \"she has / she is\",\n",
    "\"should've\": \"should have\",\n",
    "\"shouldn't\": \"should not\",\n",
    "\"shouldn't've\": \"should not have\",\n",
    "\"so've\": \"so have\",\n",
    "\"so's\": \"so as / so is\",\n",
    "\"that'd\": \"that would / that had\",\n",
    "\"that'd've\": \"that would have\",\n",
    "\"that's\": \"that has / that is\",\n",
    "\"there'd\": \"there had / there would\",\n",
    "\"there'd've\": \"there would have\",\n",
    "\"there's\": \"there has / there is\",\n",
    "\"they'd\": \"they had / they would\",\n",
    "\"they'd've\": \"they would have\",\n",
    "\"they'll\": \"they shall / they will\",\n",
    "\"they'll've\": \"they shall have / they will have\",\n",
    "\"they're\": \"they are\",\n",
    "\"they've\": \"they have\",\n",
    "\"to've\": \"to have\",\n",
    "\"wasn't\": \"was not\",\n",
    "\"we'd\": \"we had / we would\",\n",
    "\"we'd've\": \"we would have\",\n",
    "\"we'll\": \"we will\",\n",
    "\"we'll've\": \"we will have\",\n",
    "\"we're\": \"we are\",\n",
    "\"we've\": \"we have\",\n",
    "\"weren't\": \"were not\",\n",
    "\"what'll\": \"what shall / what will\",\n",
    "\"what'll've\": \"what shall have / what will have\",\n",
    "\"what're\": \"what are\",\n",
    "\"what's\": \"what has / what is\",\n",
    "\"what've\": \"what have\",\n",
    "\"when's\": \"when has / when is\",\n",
    "\"when've\": \"when have\",\n",
    "\"where'd\": \"where did\",\n",
    "\"where's\": \"where has / where is\",\n",
    "\"where've\": \"where have\",\n",
    "\"who'll\": \"who shall / who will\",\n",
    "\"who'll've\": \"who shall have / who will have\",\n",
    "\"who's\": \"who has / who is\",\n",
    "\"who've\": \"who have\",\n",
    "\"why's\": \"why has / why is\",\n",
    "\"why've\": \"why have\",\n",
    "\"will've\": \"will have\",\n",
    "\"won't\": \"will not\",\n",
    "\"won't've\": \"will not have\",\n",
    "\"would've\": \"would have\",\n",
    "\"wouldn't\": \"would not\",\n",
    "\"wouldn't've\": \"would not have\",\n",
    "\"y'all\": \"you all\",\n",
    "\"y'all'd\": \"you all would\",\n",
    "\"y'all'd've\": \"you all would have\",\n",
    "\"y'all're\": \"you all are\",\n",
    "\"y'all've\": \"you all have\",\n",
    "\"you'd\": \"you had / you would\",\n",
    "\"you'd've\": \"you would have\",\n",
    "\"you'll\": \"you shall / you will\",\n",
    "\"you'll've\": \"you shall have / you will have\",\n",
    "\"you're\": \"you are\",\n",
    "\"you've\": \"you have\"\n",
    "}\n",
    "def remove_contractions(word_list):\n",
    "    return [word for word in word_list if word not in contractions]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View Technical/Uncommon Words\n",
    "After removing contractions, human-reviewed words that are not in the English language corpus are likely to be the technical terms we are looking for."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "word_set = set([])\n",
    "for word in remove_contractions(corrected_words):\n",
    "    if word:\n",
    "        if word.lower() not in my_dict:\n",
    "            if word.endswith('s') and word[:-1] in my_dict:\n",
    "                print(\"\")\n",
    "            elif word.endswith(\"'s\") and word[:-2] in my_dict:\n",
    "                print(\"\")\n",
    "            else:\n",
    "                word_set.add(word)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for word in word_set:\n",
    "    print(word)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Custom Vocabulary\n",
    "Using the technical terms identified above, it is now easier to manually create a custom vocabulary of those terms that we want Transcribe to be able to recognize. A custom vocabulary table enables options to tell Amazon Transcribe how each technical term is pronounced and how it should be displayed. More details on how to form a custom vocabulary table can be found at [Create a Custom Vocabulary Using a Table](https://docs.aws.amazon.com/transcribe/latest/dg/how-vocabulary.html#create-vocabulary-table)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that as you process additional videos on the same topic, you can keep updating this list, and the number of new technical terms you'll have to add will likely decrease each time you get a new video."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've built in advance a custom vocabulary (below) using parsed A2I results from the first and third videos with a 0.5 `THRESHOLD` confidence value. You can use this vocabulary for the rest of the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "finalized_words=[['Phrase','IPA','SoundsLike','DisplayAs'], # This top line denote the column headers of the text file.\n",
    "                 ['machine-learning','','','machine learning'],\n",
    "                 ['amazon','','am-uh-zon','Amazon'],\n",
    "                 ['boto-three','','boe-toe-three','Boto3'],\n",
    "                 ['T.-three','','tee-three','T3'],\n",
    "                 ['Sarab','','suh-rob','Sarab'],\n",
    "                 ['E.C.R.','','ee-see-are','ECR'],\n",
    "                 ['E.B.S.','','ee-bee-ess','EBS'],\n",
    "                 ['jupyter','','joo-pih-ter','Jupyter'],\n",
    "                 ['opt-M.L.','','opt-em-ell','/opt/ml'],\n",
    "                 ['desktop','','desk-top','desktop'],\n",
    "                 ['S.-Three','','ess-three','S3'],\n",
    "                 ['S.D.K.','','ess-dee-kay','SDK'],\n",
    "                 ['sagemaker','','sage-may-ker','SageMaker'],\n",
    "                 ['mars-dot-r','','mars-dot-are','mars.R'],\n",
    "                 ['I.A.M.','','eye-ay-em','IAM'],\n",
    "                 ['V.P.C.','','vee-pee-see','VPC'],\n",
    "                 ['E.C.-Two','','ee-see-too','EC2'],\n",
    "                 ['blazing-text','','blay-zing-text','BlazingText'],\n",
    "                ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example, let's take a look at several phrases. The term \"machine learning\" in hyphenated in the first column to indicate that these two distinct words are pronounced. The `SoundsLike` column is left blank since both \"machine\" and \"learning\" are common words by themselves that Transcribe likely already knows how to pronounce.\n",
    "\n",
    "The terms ECR, EBS, SDK, and other acronyms all should have periods after each letter in the `Phrase` column to denote that the letters are pronounced individually by name. This is reinforced in the `SoundsLike` column.\n",
    "\n",
    "You'll also notice that hybrid acronym phrases like \"opt-M.L.\" and \"E.C.-Two\" follow a combination of both conventions above.\n",
    "\n",
    "Finally, the column corresponding to the IPA pronunciation is left blank for every word. The International Phonetic Alphabet (IPA) is the official standardized representation of the sounds of spoken language and may require some familiarity with phonetic notation. If you want to give Transcribe an extremely precise and unambiguous pronunciation guide, you can include IPA pronunciations instead of `SoundsLike` pronunciations by using an [IPA pronunciation key](https://en.wiktionary.org/wiki/Wiktionary:IPA_pronunciation_key) or looking them up in a dictionary."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Write the Custom Vocabulary Table to a Txt File"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "custom_vocab_file_name = \"customvocab4.txt\"\n",
    "file1 = open(custom_vocab_file_name,\"w\")\n",
    "template = '{}\\t{}\\t{}\\t{}\\n'\n",
    "for line in finalized_words:\n",
    "    file1.write(template.format(line[0],\n",
    "                                line[1],\n",
    "                                line[2],\n",
    "                                line[3])\n",
    "               )\n",
    "file1.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Upload Custom Vocabulary File to S3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "import boto3\n",
    "from botocore.exceptions import ClientError\n",
    "\n",
    "\n",
    "def upload_file(file_name, bucket, object_name=None):\n",
    "    \"\"\"Upload a file to an S3 bucket\n",
    "\n",
    "    :param file_name: File to upload\n",
    "    :param bucket: Bucket to upload to\n",
    "    :param object_name: S3 object name. If not specified then file_name is used\n",
    "    :return: True if file was uploaded, else False\n",
    "    \"\"\"\n",
    "\n",
    "    # If S3 object_name was not specified, use file_name\n",
    "    if object_name is None:\n",
    "        object_name = file_name\n",
    "\n",
    "    # Upload the file\n",
    "    s3_client = boto3.client('s3')\n",
    "    try:\n",
    "        response = s3_client.upload_file(file_name, bucket, object_name)\n",
    "    except ClientError as e:\n",
    "        logging.error(e)\n",
    "        return False\n",
    "    return True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "upload_file(custom_vocab_file_name, BUCKET)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the Custom Vocabulary for Amazon Transcribe\n",
    "After saving your custom vocabulary table to a text file and uploading it to an S3 bucket, create your custom vocabulary with a specified name so that Amazon Transcribe can use it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The name of your custom vocabulary must be unique!\n",
    "vocab_improved='sagemaker-custom-vocab-' + str(uuid.uuid4())\n",
    "vocab_improved"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "transcribe_client = boto3.client(\"transcribe\")\n",
    "response = transcribe_client.create_vocabulary(\n",
    "    VocabularyName=vocab_improved,\n",
    "    LanguageCode='en-US',\n",
    "    VocabularyFileUri='s3://' + BUCKET + '/' + custom_vocab_file_name\n",
    ")\n",
    "pp.pprint(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wait until the `VocabularyState` displays `READY` before continuing. This typically takes a few minutes or less."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Wait for the status of the vocab you created to finish\n",
    "while True:\n",
    "    response = transcribe_client.get_vocabulary(\n",
    "        VocabularyName=vocab_improved\n",
    "    )\n",
    "    status = response['VocabularyState']\n",
    "    if status in ['READY', 'FAILED']:\n",
    "        print(status)\n",
    "        break\n",
    "    print(\"Not ready yet...\")\n",
    "    time.sleep(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Improve transcription using custom vocabulary\n",
    "\n",
    "### Re-transcribe using the Custom Vocabulary\n",
    "Let's re-transcribe the video using our custom vocabulary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# New job names\n",
    "# In-sample videos\n",
    "job_name_custom_vid_0='AWS-custom-0-using-' + vocab_improved + str(time_now)\n",
    "job_names_custom = [job_name_custom_vid_0]\n",
    "job_names_custom"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Start another transcription job using your custom vocabulary.\n",
    "transcribe(job_name_custom_vid_0, folder_path+all_videos[0], BUCKET, vocab_name=vocab_improved)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point, it may have been some time since you've run the first few cells of this notebook. If you get the error `TypeError: 'TranscribeService' object is not callable`, your execution role may have expired. Refreshing the role connection using the cell below (after uncommenting) is typically able to resolve this issue. You may also need to re-run the cells containing the definition of the function `transcribe` (the first code cell of Step 1)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from sagemaker import get_execution_role\n",
    "\n",
    "# ROLE = get_execution_role()\n",
    "# display(ROLE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the status of your transcription job. Wait until the status displays `COMPLETED`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "while True:\n",
    "    response = transcribe_client.get_transcription_job(\n",
    "        TranscriptionJobName=job_name_custom_vid_0\n",
    "    )\n",
    "    status = response['TranscriptionJob']['TranscriptionJobStatus']\n",
    "    if status in ['COMPLETED', 'FAILED']:\n",
    "        print(status)\n",
    "        break\n",
    "    print(\"Not ready yet...\")\n",
    "    time.sleep(5)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_entire_transcript_custom = []\n",
    "all_sentences_and_times_custom = []\n",
    "all_confidences_custom = []\n",
    "all_scores_custom = []\n",
    "for i in range(num_videos):\n",
    "    entire_transcript_1, sentences_and_times_1, confidences_1, scores_1 = get_transcript_text_and_timestamps(BUCKET,job_names_custom[i]+\".json\")\n",
    "    all_entire_transcript_custom.append(entire_transcript_1)\n",
    "    all_sentences_and_times_custom.append(sentences_and_times_1)\n",
    "    all_confidences_custom.append(confidences_1)\n",
    "    all_scores_custom.append(scores_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# View several sentences\n",
    "print(all_sentences_and_times_custom[0][0:20])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Write the Improved Transcripts to Txt File"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the improved transcripts\n",
    "i = 1\n",
    "for list_ in all_sentences_and_times_custom:   \n",
    "    file = open(f\"improved_transcript_{i}.txt\",\"w\")\n",
    "    for tup in list_:\n",
    "        file.write(tup['sentence'] + \"\\n\") \n",
    "    file.close()\n",
    "    i = i + 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Results and Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To analyze metrics on a larger sample size for this workflow, we've generated in advance a ground truth transcript, a transcription before custom vocabulary, and a transcription after custom vocabulary for each of the first four videos of the playlist. The first and third videos are the in-sample videos used to build the custom vocabulary you saw earlier. The second and fourth videos are used as out-sample videos to test Transcribe again after building the custom vocabulary."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following cells to import the transcripts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -rf a2i_transcribe_demo_results/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash -s \"$BUCKET\"\n",
    "mkdir a2i_transcribe_demo_results\n",
    "aws s3 sync s3://aws-ml-blog/artifacts/a2i-transcribe-custom-demo/demo-txt-files ./a2i_transcribe_demo_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Calculating Word Error Rate (WER)\n",
    "\n",
    "The most common metric for speech recognition accuracy is called word error rate (WER), which can be roughly defined to be the proportion of transcription errors relative to the number of words that were actually said. More details can be found [here](https://en.wikipedia.org/wiki/Word_error_rate).\n",
    "\n",
    "We'll be using a lightweight open-source Python library called JiWER for calculating WER between transcripts.\n",
    "\n",
    "For more details, see the open-source [description](https://pypi.org/project/jiwer/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install jiwer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from jiwer import wer\n",
    "import jiwer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Small example\n",
    "ground_truth = \"hello world\"\n",
    "hypothesis = \"hello duck\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wer(ground_truth, hypothesis)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a transformation function to preprocess transcript\n",
    "transformation = jiwer.Compose([\n",
    "    jiwer.ToLowerCase(),\n",
    "    jiwer.RemoveMultipleSpaces(),\n",
    "    jiwer.RemovePunctuation(),\n",
    "    jiwer.RemoveWhiteSpace(replace_by_space=True),\n",
    "    jiwer.SentencesToListOfWords(),\n",
    "    jiwer.SentencesToListOfWords(word_delimiter=\" \"),\n",
    "    jiwer.RemoveEmptyStrings()\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### In-sample video metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"===== In-sample videos =====\")\n",
    "for index in [1,3]:\n",
    "    print(f\"Processing video #{index}\")\n",
    "    # Original transcript\n",
    "    hypothesis_original = \"\"\n",
    "    f1 = open(f\"./a2i_transcribe_demo_results/original_transcript_{index}.txt\", \"r\")\n",
    "    for line in f1:\n",
    "        hypothesis_original += (line.strip() + \" \")\n",
    "    f1.close()\n",
    "    \n",
    "    # Transcript after custom vocabulary\n",
    "    hypothesis_2 = \"\"\n",
    "    f2 = open(f\"./a2i_transcribe_demo_results/improved_transcript_{index}.txt\", \"r\")\n",
    "    for line in f2:\n",
    "        hypothesis_2 += (line.strip() + \" \")\n",
    "    f2.close()\n",
    "    \n",
    "    # Ground truth transcript\n",
    "    ground_truth = \"\"\n",
    "    f3 = open(f\"./a2i_transcribe_demo_results/ground_truth_{index}.txt\", \"r\")\n",
    "    for line in f3:\n",
    "        ground_truth += (line.strip() + \" \")\n",
    "    f3.close()\n",
    "    \n",
    "    # Calculate baseline accuracy\n",
    "    baseline_accuracy = jiwer.wer(\n",
    "        ground_truth, \n",
    "        hypothesis_original, \n",
    "        truth_transform=transformation, \n",
    "        hypothesis_transform=transformation\n",
    "    )\n",
    "    \n",
    "    print(f\"The baseline WER (before using custom vocabularies) is {'{:.2%}'.format(baseline_accuracy)}.\")\n",
    "    \n",
    "    # Calculate new accuracy after custom vocabulary\n",
    "    new_accuracy = jiwer.wer(\n",
    "        ground_truth,\n",
    "        hypothesis_2, \n",
    "        truth_transform=transformation, \n",
    "        hypothesis_transform=transformation\n",
    "    )\n",
    "    \n",
    "    print(f\"The WER (after using custom vocabularies) is {'{:.2%}'.format(new_accuracy)}.\")\n",
    "    \n",
    "    print(f\"The percentage change in WER score is {'{:.1%}'.format((new_accuracy - baseline_accuracy)/baseline_accuracy)}.\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Out-sample video metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"===== Out-sample videos =====\")\n",
    "for index in [2,4]:\n",
    "    print(f\"Processing video #{index}\")\n",
    "    # Original transcript\n",
    "    hypothesis_original = \"\"\n",
    "    f1 = open(f\"./a2i_transcribe_demo_results/original_transcript_{index}.txt\", \"r\")\n",
    "    for line in f1:\n",
    "        hypothesis_original += (line.strip() + \" \")\n",
    "    f1.close()\n",
    "    \n",
    "    # Transcript after custom vocabulary\n",
    "    hypothesis_2 = \"\"\n",
    "    f2 = open(f\"./a2i_transcribe_demo_results/improved_transcript_{index}.txt\", \"r\")\n",
    "    for line in f2:\n",
    "        hypothesis_2 += (line.strip() + \" \")\n",
    "    f2.close()\n",
    "    \n",
    "    ground_truth = \"\"\n",
    "    f3 = open(f\"./a2i_transcribe_demo_results/ground_truth_{index}.txt\", \"r\")\n",
    "    for line in f3:\n",
    "        ground_truth += (line.strip() + \" \")\n",
    "    f3.close()\n",
    "    \n",
    "    # Calculate baseline accuracy\n",
    "    baseline_accuracy = jiwer.wer(\n",
    "        ground_truth, \n",
    "        hypothesis_original, \n",
    "        truth_transform=transformation, \n",
    "        hypothesis_transform=transformation\n",
    "    )\n",
    "    \n",
    "    print(f\"The baseline WER (before using custom vocabularies) is {'{:.2%}'.format(baseline_accuracy)}.\")\n",
    "    \n",
    "    # Calculate new accuracy after custom vocabulary\n",
    "    new_accuracy = jiwer.wer(\n",
    "        ground_truth,\n",
    "        hypothesis_2, \n",
    "        truth_transform=transformation, \n",
    "        hypothesis_transform=transformation\n",
    "    )\n",
    "    \n",
    "    print(f\"The WER (after using custom vocabularies) is {'{:.2%}'.format(new_accuracy)}.\")\n",
    "    \n",
    "    print(f\"The percentage change in WER score is {'{:.1%}'.format((new_accuracy - baseline_accuracy)/baseline_accuracy)}.\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Review Results\n",
    "\n",
    "We've provided a table summarizing the changes in WER scores below.\n",
    "\n",
    "| In- or out-sample | Video | Baseline WER (before custom vocabulary) | New WER (after custom vocabulary) | Percentage change |\n",
    "|------------|-------|-----------------------------------------|------------------------------------|-------------------|\n",
    "| In-sample  | #1     | 5.18%                                   | 2.62%                              | -49%              |\n",
    "| In-sample  | #3     | 11.94%                                  | 7.84%                              | -34%              |\n",
    "| Out-sample | #2     | 7.55%                                   | 6.56%                              | -13%              |\n",
    "| Out-sample | #4     | 10.91%                                  | 8.98%                              | -18%              |\n",
    "\n",
    "If we consider absolute WER scores, the initial WER of 5.18%, for instance, might be sufficiently low for some use cases—that's only around 1 in 20 words that are mis-transcribed! However, this rate can be insufficient for other purposes, because domain-specific terms are often the least common words spoken (relative to frequent words such as “to,” “and,” or “I”) but the most commonly mis-transcribed. For applications like search engine optimization (SEO) and video organization by topic, you may want to ensure that these technical terms are transcribed correctly. Let’s take a look at how our custom vocabulary impacted the transcription rates of several important technical terms:\n",
    "\n",
    "### Metrics for specific technical terms\n",
    "\n",
    "Note that “ground truth” refers to the true transcript as transcribed by hand, “original transcript” refers to the transcription before applying the custom vocabulary, and “new transcript” refers to the transcription after applying the custom vocabulary.\n",
    "\n",
    "#### In-sample videos:\n",
    "\n",
    "Video #1:\n",
    "\n",
    "| Technical term | Ground truth mentions | Original transcript mentions | New transcript mentions | Percentage point change |\n",
    "|----------------|-----------------------|------------------------------|-------------------------|-------------------------|\n",
    "| SageMaker      | 22                    | 4 (18%)                      | 22 (100%)               | +82%                    |\n",
    "| EC2            | 15                    | 1 (7%)                       | 15 (100%)               | +93%                    |\n",
    "| EBS            | 11                    | 7 (64%)                      | 11 (100%)                | +36%                    |\n",
    "| Jupyter        | 5                     | 0 (0%)                       | 5 (100%)                | +100%                   |\n",
    "| S3             | 3                     | 0 (0%)                       | 3 (100%)                | +100%                   |\n",
    "| SDK            | 2                     | 0 (0%)                        | 2 (100%)                | +100%                 |\n",
    "| BlazingText    | 2                     | 0 (0%)                        | 2 (100%)                | +100%                 |\n",
    "| IAM            | 1                     | 0 (0%)                        | 1 (100%)                | +100%                 |\n",
    "| **Total**      | **61**               | **12 (20%)**                  | **61 (100%)**           | **+80%**               |\n",
    "\n",
    "Video #3:\n",
    "\n",
    "| Technical term | Ground truth mentions | Original transcript mentions | New transcript mentions | Percentage point change |\n",
    "|----------------|-----------------------|------------------------------|-------------------------|-------------------------|\n",
    "| SageMaker      | 17                    | 4 (24%)                      | 17 (100%)               | +76%                    |\n",
    "| ECR            | 7                     | 0 (0%)                       | 7 (100%)                | +100%                    |\n",
    "| /opt/ml        | 6                     | 0 (0%)                       | 6 (100%)                | +100%                    |\n",
    "| mars.R         | 3                     | 0 (0%)                       | 3 (100%)                | +100%                   |\n",
    "| S3             | 1                     | 0 (0%)                       | 1 (100%)                | +100%                   |\n",
    "| **Total**      | **34**               | **4 (12%)**                  | **34 (100%)**           | **+88%**               |\n",
    "\n",
    "\n",
    "#### Out-sample videos:\n",
    "\n",
    "Video #2:\n",
    "\n",
    "| Technical term | Ground truth mentions | Original transcript mentions | New transcript mentions | Percentage point change |\n",
    "|----------------|-----------------------|------------------------------|-------------------------|-------------------------|\n",
    "| SageMaker      | 12                    | 3 (25%)                      | 12 (100%)               | +75%                    |\n",
    "| BlazingText    | 3                     | 0 (0%)                       | 3 (100%)                | +100%                   |\n",
    "| ECR            | 1                     | 0 (0%)                       | 1 (100%)                | +100%                   |\n",
    "| **Total**      | **16**               | **3 (19%)**                  | **16 (100%)**           | **+81%**               |\n",
    "\n",
    "Video #4:\n",
    "\n",
    "| Technical term | Ground truth mentions | Original transcript mentions | New transcript mentions | Percentage point change |\n",
    "|----------------|-----------------------|------------------------------|-------------------------|-------------------------|\n",
    "| SageMaker      | 21                    | 4 (19%)                      | 20 (95%)                | +75%                    |\n",
    "| EC2            | 11                    | 0 (0%)                       | 11 (100%)               | +100%                    |\n",
    "| S3             | 7                     | 0 (0%)                       | 6 (86%)                 | +86%                    |\n",
    "| ECR            | 2                     | 0 (0%)                       | 2 (100%)                | +100%                   |\n",
    "| SDK            | 1                     | 0 (0%)                       | 1 (100%)                | +100%                   |\n",
    "| EBS            | 1                     | 1 (100%)                     | 1 (100%)                | +0%                     |\n",
    "| **Total**      | **43**               | **3 (12%)**                  | **41 (95%)**           | **+83%**               |\n",
    "\n",
    "We see that using custom vocabularies resulted in a 80-percentage point or more increase in the number of correctly transcribed technical terms over the number of times those technical terms were used. A majority of the time, using a custom vocabulary resulted in 100% accuracy in transcribing these domain-specific terms. Now it does look like using custom vocabularies was worth the effort after all!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cleaning up\n",
    "To avoid incurring unnecessary charges, delete resources when not in use."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Recap\n",
    "\n",
    "In this post, we walked through an example of how you can improve transcripts from Amazon Transcribe using custom vocabularies and an Amazon A2I human review workflow. This allows you to quickly identify domain-specific terms using your own private workforce and review workflows, and use these terms to build a custom vocabulary so that future mentions of term are transcribed with greater accuracy, at scale. Transcribing key technical terms correctly can be important for doing SEO, enabling highly specific textual queries, and grouping large quantities of video or audio files by important technical terms.\n",
    "\n",
    "The full proof-of-concept Jupyter notebook can be found at this Github repository. Check out other blog posts covering integrations of Amazon A2I, such as [Using Amazon Textract with Amazon Augmented AI for processing critical documents](https://aws.amazon.com/blogs/machine-learning/using-amazon-textract-with-amazon-augmented-ai-for-processing-critical-documents/) and [Designing human review workflows with Amazon Translate and Amazon Augmented AI](https://aws.amazon.com/blogs/machine-learning/designing-human-review-workflows-with-amazon-translate-and-amazon-augmented-ai/).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The End!\n",
    "For a more detailed discussion with additional visuals, check out the accompanying blog post."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.7.6 64-bit ('base': conda)",
   "language": "python",
   "name": "python_defaultSpec_1596068428258"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6-final"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}