Chapter 16/Improve-accuracy-of-pdf-processing-with-Amazon-Textract-and-Amazon-A2I-forGitHub.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e9946ed1",
   "metadata": {},
   "source": [
    "# Improve accuracy of pdf batch processing with Amazon Textract and Amazon A2I\n",
    "\n",
    "In this chapter and this accompanying notebook learn with an example on how you can use Amazon Textract in asynchronous mode by extracting content from multiple PDF files in batch, and sending specific content from these PDF documents to an Amazon A2I human review loop to review and modify the values, and send them to an Amazon DynamoDB table for downstream processing. \n",
    "\n",
    "**Important Note:** This is an accompanying notebook for Chapter 16 - Improve accuracy of pdf batch processing with Amazon Textract and Amazon A2I from the Natural Language Processing with AWS AI Services book. Please make sure to read the instructions provided in the book prior to attempting this notebook. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb8d93ff",
   "metadata": {},
   "source": [
    "### Step 0 - Create a private human review workforce\n",
    "\n",
    "This step requires you to use the AWS Console. However, we highly recommend that you follow it, especially when creating your own task with a custom template we will use for this notebook. We will create a private workteam and add only one user (you) to it.\n",
    "\n",
    "To create a private team:\n",
    "\n",
    "   1. Go to AWS Console > Amazon SageMaker > Labeling workforces\n",
    "   1. Click \"Private\" and then \"Create private team\".\n",
    "   1. Enter the desired name for your private workteam.\n",
    "   1. Enter your own email address in the \"Email addresses\" section.\n",
    "   1. Enter the name of your organization and a contact email to administer the private workteam.\n",
    "   1. Click \"Create Private Team\".\n",
    "   1. The AWS Console should now return to AWS Console > Amazon SageMaker > Labeling workforces. Your newly created team should be visible under \"Private teams\". Next to it you will see an ARN which is a long string that looks like arn:aws:sagemaker:region-name-123456:workteam/private-crowd/team-name. Please copy this ARN to paste in the cell below.\n",
    "   1. You should get an email from no-reply@verificationemail.com that contains your workforce username and password.\n",
    "   1. In AWS Console > Amazon SageMaker > Labeling workforces, click on the URL in Labeling portal sign-in URL. Use the email/password combination from Step 8 to log in (you will be asked to create a new, non-default password).\n",
    "   1. This is your private worker's interface. When we create a verification task in Verify your task using a private team below, your task should appear in this window. You can invite your colleagues to participate in the labeling job by clicking the \"Invite new workers\" button.\n",
    "\n",
    "Please refer to the [Amazon SageMaker documentation](https://docs.aws.amazon.com/sagemaker/latest/dg/sms-workforce-management.html) if you need more details."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2671a17",
   "metadata": {},
   "source": [
    "### Step 1 - Import libraries and initiliaze variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "01a7f5ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 1 - Cell 1\n",
    "import urllib\n",
    "import boto3\n",
    "import os\n",
    "import json\n",
    "import time\n",
    "import uuid\n",
    "import sagemaker\n",
    "import pandas as pd\n",
    "from sagemaker import get_execution_role\n",
    "from sagemaker.s3 import S3Uploader, S3Downloader\n",
    "\n",
    "textract = boto3.client('textract')\n",
    "s3 = boto3.resource('s3')\n",
    "bucket = \"<S3-bucket-name>\"\n",
    "prefix = 'chapter16/input'\n",
    "# Enter the Workteam ARN you created from point 7 in Step 0 above\n",
    "WORKTEAM_ARN= '<your-private-workteam-arn>'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "12392239",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 1 - Cell 2\n",
    "# Upload the SEC registration documents\n",
    "s3_client = boto3.client('s3')\n",
    "for secfile in os.listdir():\n",
    "    if secfile.endswith('pdf'):\n",
    "        response = s3_client.upload_file(secfile, bucket, prefix+'/'+secfile)\n",
    "        print(\"Uploaded {} to S3 bucket {} in folder {}\".format(secfile, bucket, prefix))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e32b9d1e",
   "metadata": {},
   "source": [
    "### Step 2 - Start Amazon Textract Text Detection Job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bed70cfc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 2 - Cell 1\n",
    "input_bucket = s3.Bucket(bucket)\n",
    "jobids = {}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "44058663",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 2 - Cell 2\n",
    "for doc in input_bucket.objects.all():\n",
    "    if doc.key.startswith(prefix) and doc.key.endswith('pdf'): \n",
    "        tres = textract.start_document_text_detection(\n",
    "            DocumentLocation={\n",
    "                \"S3Object\": {\n",
    "                    \"Bucket\": bucket,\n",
    "                    \"Name\": doc.key\n",
    "                }\n",
    "            }\n",
    "        )\n",
    "        jobids[doc.key.split('/')[2]] = tres['JobId']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da68b5d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 2 - Cell 3\n",
    "for j in jobids:\n",
    "    print(\"Textract detection Job ID for {} is {}\".format(j,str(jobids[j])))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fef7055f",
   "metadata": {},
   "source": [
    "### Step 3 - Get Amazon Textract Text Detection Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc80d2f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 3 - Cell 1\n",
    "\n",
    "class TextExtractor():\n",
    "    def extract_text(self, jobId):\n",
    "        \"\"\" Extract text from document corresponding to jobId and\n",
    "        generate a list of pages containing the text\n",
    "        \"\"\"\n",
    "\n",
    "        textract_result = self.__get_textract_result(jobId)\n",
    "        pages = {}\n",
    "        self.__extract_all_pages(jobId, textract_result, pages, [])\n",
    "        return pages\n",
    "\n",
    "    def __get_textract_result(self, jobId):\n",
    "        \"\"\" retrieve textract result with job Id \"\"\"\n",
    "\n",
    "        result = textract.get_document_text_detection(\n",
    "            JobId=jobId\n",
    "        )\n",
    "        return result\n",
    "\n",
    "    def __extract_all_pages(self, jobId, textract_result, pages, page_numbers):\n",
    "        \"\"\" extract page content: build the pages array,\n",
    "        recurse if response is too big (when NextToken is provided by textract)\n",
    "        \"\"\"\n",
    "        blocks = [x for x in textract_result['Blocks'] if x['BlockType'] == \"LINE\"]\n",
    "        content = {}\n",
    "        line = 0\n",
    "        for block in blocks:\n",
    "            line += 1\n",
    "            content['Text'+str(line)] = block['Text']\n",
    "            content['Confidence'+str(line)] = block['Confidence']\n",
    "            if block['Page'] not in page_numbers:\n",
    "                page_numbers.append(block['Page'])\n",
    "                pages[block['Page']] = {\n",
    "                    \"Number\": block['Page'],\n",
    "                    \"Content\": content\n",
    "                }\n",
    "            else:\n",
    "                pages[block['Page']]['Content'] = content\n",
    "        nextToken = textract_result.get(\"NextToken\", \"\")\n",
    "        if nextToken != '':\n",
    "            textract_result = textract.get_document_text_detection(\n",
    "                JobId=jobId,\n",
    "                NextToken=nextToken\n",
    "            )\n",
    "            self.__extract_all_pages(jobId,\n",
    "                                     textract_result,\n",
    "                                     pages,\n",
    "                                     page_numbers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10c09635",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 3 - Cell 2\n",
    "text_extractor = TextExtractor()\n",
    "indoc = {}\n",
    "df_indoc = pd.DataFrame(columns = ['DocName','LineNr','DetectedText','Confidence', 'CorrectedText', 'Comments'])\n",
    "for x in jobids:\n",
    "    pages = text_extractor.extract_text(jobids[x])\n",
    "    contdict =pages[1]['Content']\n",
    "    for row in range(1,(int(len(contdict)/2))+1):\n",
    "        df_indoc.loc[len(df_indoc.index)] = [x, row, contdict['Text'+str(row)], round(contdict['Confidence'+str(row)],1),'','']\n",
    "# Uncomment the line below if you want to review the contents of this dataframe\n",
    "#df_indoc.to_csv('extract.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c6bbaa65",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 3 - Cell 3\n",
    "# The lines in each document that are of importance for the human loop to review\n",
    "bounding_dict = {'lines': '9:11:12:13:15:16:17:18:19:20:21:22:23:24:25'}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "746f149c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 3 - Cell 4\n",
    "# Let us now create a new dataframe that only contains the subset of lines we need from the bounding_dict\n",
    "df_newdoc = pd.DataFrame(columns = ['DocName','LineNr','DetectedText','Confidence','CorrectedText','Comments'])\n",
    "for idx, row in df_indoc.iterrows():\n",
    "    if str(row['LineNr']) in bounding_dict['lines'].split(':'):\n",
    "        df_newdoc.loc[len(df_newdoc.index)] = [row['DocName'],row['LineNr'], row['DetectedText'], row['Confidence'], row['CorrectedText'],row['Comments']]\n",
    "df_newdoc"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4fc6dc9",
   "metadata": {},
   "source": [
    "### Step 4 - Create the Amazon A2I human review Task UI\n",
    "We will customize a sample tabular template from the Amazon A2I sample Task UI template page - https://github.com/aws-samples/amazon-a2i-sample-task-uis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "04919103",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 4 - Cell 1\n",
    "# Initialize A2I variables\n",
    "a2i_prefix = \"chapter16/a2i-results\"\n",
    "\n",
    "# Define IAM role\n",
    "role = get_execution_role()\n",
    "print(\"RoleArn: {}\".format(role))\n",
    "\n",
    "timestamp = time.strftime(\"%Y-%m-%d-%H-%M-%S\", time.gmtime())\n",
    "# Amazon SageMaker client\n",
    "sagemaker_client = boto3.client('sagemaker')\n",
    "\n",
    "# Amazon Augment AI (A2I) client\n",
    "a2i = boto3.client('sagemaker-a2i-runtime')\n",
    "\n",
    "# Flow definition name - this value is unique per account and region. You can also provide your own value here.\n",
    "flowDefinitionName = 'fd-pdf-docs-' + timestamp\n",
    "\n",
    "# Task UI name - this value is unique per account and region. You can also provide your own value here.\n",
    "taskUIName = 'ui-pdf-docs-' + timestamp\n",
    "\n",
    "# Flow definition outputs\n",
    "OUTPUT_PATH = f's3://' + bucket + '/' + a2i_prefix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f3d7dc9e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 4 - Cell 2\n",
    "# We will use the tabular liquid template and customize it for our requirements\n",
    "\n",
    "template = r\"\"\"\n",
    "<script src=\"https://assets.crowd.aws/crowd-html-elements.js\"></script>\n",
    "\n",
    "<style>\n",
    "  table, tr, th, td {\n",
    "    border: 1px solid black;\n",
    "    border-collapse: collapse;\n",
    "    padding: 5px;\n",
    "  }\n",
    "</style>\n",
    "\n",
    "<crowd-form>\n",
    "    <div>\n",
    "        <h1>Instructions</h1>\n",
    "        <p>Please review the SEC registration form inputs, and make corrections where appropriate. </p>\n",
    "    </div>\n",
    "   <div>\n",
    "      <h3>Original Registration Form - Page 1</h3>\n",
    "      <classification-target>\n",
    "        <img style=\"width: 70%; max-height: 40%; margin-bottom: 10px\" src=\"{{ task.input.image | grant_read_access }}\"/>        \n",
    "      </classification-target>     \n",
    "   </div>\n",
    "    <br>\n",
    "    <h1> Please enter your modifications below </h1>\n",
    "    <table>\n",
    "    <tr>\n",
    "        <th>Line Nr</th>\n",
    "        <th style=\"width:500px\">Detected Text</th>\n",
    "        <th style=\"width:500px\">Confidence</th>\n",
    "        <th>Change Required</th>\n",
    "        <th style=\"width:500px\">Corrected Text</th>\n",
    "        <th>Comments</th>\n",
    "    </tr>\n",
    "    {% for pair in task.input.document %}\n",
    "\n",
    "        <tr>\n",
    "          <td>{{ pair.linenr }}</td>\n",
    "          <td><crowd-text-area name=\"predicteddoc{{ pair.linenr }}\" value=\"{{ pair.detectedtext }}\"></crowd-text-area></td>\n",
    "          <td><crowd-text-area name=\"confidence{{ pair.linenr }}\" value=\"{{ pair.confidence }}\"></crowd-text-area></td>\n",
    "          <td>\n",
    "            <p>\n",
    "              <input type=\"radio\" id=\"agree{{ pair.linenr }}\" name=\"rating{{ pair.linenr }}\" value=\"agree\" required>\n",
    "              <label for=\"agree{{ pair.linenr }}\">Correct</label>\n",
    "            </p>\n",
    "            <p>\n",
    "              <input type=\"radio\" id=\"disagree{{ pair.linenr }}\" name=\"rating{{ pair.linenr }}\" value=\"disagree\" required>\n",
    "              <label for=\"disagree{{ pair.linenr }}\">Incorrect</label>\n",
    "            </p>\n",
    "          </td>\n",
    "          <td>\n",
    "            <p>\n",
    "            <input style=\"width:500px\" rows=\"3\" type=\"text\" name=\"correcteddoc{{ pair.linenr }}\" value=\"{{pair.detectedtext}}\" required/>\n",
    "            </p>\n",
    "           </td>\n",
    "           <td>\n",
    "            <p>\n",
    "            <input style=\"width:500px\" rows=\"3\" type=\"text\" name=\"comments{{ pair.linenr }}\" placeholder=\"Explain why you changed the value\"/>\n",
    "            </p>\n",
    "           </td>\n",
    "        </tr>\n",
    "\n",
    "      {% endfor %}\n",
    "    </table>\n",
    "    <br>\n",
    "    <br>\n",
    "</crowd-form>\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "16e231e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 4 - Cell 3\n",
    "# Define the method to initialize and create the Task UI\n",
    "def create_task_ui():\n",
    "    response = sagemaker_client.create_human_task_ui(\n",
    "        HumanTaskUiName=taskUIName,\n",
    "        UiTemplate={'Content': template})\n",
    "    return response\n",
    "\n",
    "# create Task UI\n",
    "humanTaskUiResponse = create_task_ui()\n",
    "humanTaskUiArn = humanTaskUiResponse['HumanTaskUiArn']\n",
    "print(humanTaskUiArn)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd305e90",
   "metadata": {},
   "source": [
    "### Step 5 - Create the Amazon A2I flow definition\n",
    "\n",
    "In this section, we're going to create a flow definition definition. Flow Definitions allow us to specify:\n",
    "\n",
    "* The workforce that your tasks will be sent to.\n",
    "* The instructions that your workforce will receive. This is called a worker task template.\n",
    "* Where your output data will be stored.\n",
    "\n",
    "This notebook is going to use the API, but you can optionally create this workflow definition in the console as well.\n",
    "For more details and instructions, see: https://docs.aws.amazon.com/sagemaker/latest/dg/a2i-create-flow-definition.html."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2cd363ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 5 - Cell 1\n",
    "create_workflow_definition_response = sagemaker_client.create_flow_definition(\n",
    "        FlowDefinitionName=flowDefinitionName,\n",
    "        RoleArn=role,\n",
    "        HumanLoopConfig= {\n",
    "            \"WorkteamArn\": WORKTEAM_ARN,\n",
    "            \"HumanTaskUiArn\": humanTaskUiArn,\n",
    "            \"TaskCount\": 1,\n",
    "            \"TaskDescription\": \"Review the contents and correct values as indicated\",\n",
    "            \"TaskTitle\": \"SEC Registration Form Review\"\n",
    "        },\n",
    "        OutputConfig={\n",
    "            \"S3OutputPath\" : OUTPUT_PATH\n",
    "        }\n",
    "    )\n",
    "flowDefinitionArn = create_workflow_definition_response['FlowDefinitionArn'] # let's save this ARN for future use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2abee2f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 5 - Cell 2\n",
    "for x in range(60):\n",
    "    describeFlowDefinitionResponse = sagemaker_client.describe_flow_definition(FlowDefinitionName=flowDefinitionName)\n",
    "    print(describeFlowDefinitionResponse['FlowDefinitionStatus'])\n",
    "    if (describeFlowDefinitionResponse['FlowDefinitionStatus'] == 'Active'):\n",
    "        print(\"Flow Definition is active\")\n",
    "        break\n",
    "    time.sleep(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5fba494",
   "metadata": {},
   "source": [
    "### Step 6 - Activate the Amazon A2I flow definition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7a22fd89",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 6 - Cell 1\n",
    "# We will display the PDF first page for reference on what is being edited by the human loop\n",
    "reg_images = {}\n",
    "for image in os.listdir():\n",
    "    if image.endswith('png'):\n",
    "        reg_images[image.split('_')[0]] = S3Uploader.upload(image, 's3://{}/{}'.format(bucket, prefix))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf58248b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 6 - Cell 2\n",
    "# Activate human loops for all the three documents. These will be delivered for review sequentially in the Task UI.\n",
    "# We will also send only low confidence detections to A2I so the human team can update the text for what is should actually be\n",
    "humanLoopName = {}\n",
    "docs = df_newdoc.DocName.unique()\n",
    "# confidence threshold\n",
    "confidence_threshold = 95\n",
    "for doc in docs:\n",
    "    doc_list = []\n",
    "    humanLoopName[doc] = str(uuid.uuid4())\n",
    "    for idx, line in df_newdoc.iterrows():\n",
    "        # Send only those lines whose confidence score is less than threshold\n",
    "        if line['DocName'] == doc and line['Confidence'] <= confidence_threshold:\n",
    "            doc_list.append({'linenr': line['LineNr'], 'detectedtext': line['DetectedText'], 'confidence':line['Confidence']})\n",
    "    ip_content = {\"document\": doc_list,\n",
    "              'image': reg_images[doc.split('.')[0]]\n",
    "             }                \n",
    "    start_loop_response = a2i.start_human_loop(\n",
    "            HumanLoopName=humanLoopName[doc],\n",
    "            FlowDefinitionArn=flowDefinitionArn,\n",
    "            HumanLoopInput={\n",
    "                \"InputContent\": json.dumps(ip_content)\n",
    "            }\n",
    "        )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "031e463a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 6 - Cell 3\n",
    "completed_human_loops = []\n",
    "for doc in humanLoopName:\n",
    "    resp = a2i.describe_human_loop(HumanLoopName=humanLoopName[doc])\n",
    "    print(f'HumanLoop Name: {humanLoopName[doc]}')\n",
    "    print(f'HumanLoop Status: {resp[\"HumanLoopStatus\"]}')\n",
    "    print(f'HumanLoop Output Destination: {resp[\"HumanLoopOutput\"]}')\n",
    "    print('\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6160baf3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 6 - Cell 4\n",
    "workteamName = WORKTEAM_ARN[WORKTEAM_ARN.rfind('/') + 1:]\n",
    "print(\"Navigate to the private worker portal and do the tasks. Make sure you've invited yourself to your workteam!\")\n",
    "print('https://' + sagemaker_client.describe_workteam(WorkteamName=workteamName)['Workteam']['SubDomain'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "22fccc5c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 6 - Cell 5\n",
    "completed_human_loops = []\n",
    "for doc in humanLoopName:\n",
    "    resp = a2i.describe_human_loop(HumanLoopName=humanLoopName[doc])\n",
    "    print(f'HumanLoop Name: {humanLoopName[doc]}')\n",
    "    print(f'HumanLoop Status: {resp[\"HumanLoopStatus\"]}')\n",
    "    print(f'HumanLoop Output Destination: {resp[\"HumanLoopOutput\"]}')\n",
    "    print('\\n')\n",
    "    if resp[\"HumanLoopStatus\"] == \"Completed\":\n",
    "          completed_human_loops.append(resp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4fa80ee9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 6 - Cell 7\n",
    "import re\n",
    "import pandas as pd\n",
    "\n",
    "for resp in completed_human_loops:\n",
    "    splitted_string = re.split('s3://' + bucket  + '/', resp['HumanLoopOutput']['OutputS3Uri'])\n",
    "    output_bucket_key = splitted_string[1]\n",
    "    response = s3_client.get_object(Bucket=bucket, Key=output_bucket_key)\n",
    "    content = response[\"Body\"].read()\n",
    "    json_output = json.loads(content)\n",
    "    loop_name = json_output['humanLoopName']\n",
    "    for i in json_output['humanAnswers']:\n",
    "        x = i['answerContent']\n",
    "        docname = list(humanLoopName.keys())[list(humanLoopName.values()).index(loop_name)]\n",
    "        for i, r in df_newdoc.iterrows():\n",
    "            if r['DocName'] == docname:\n",
    "                df_newdoc.at[i,'CorrectedText'] = x['correcteddoc'+str(r['LineNr'])] if 'correcteddoc'+str(r['LineNr']) in x else ''\n",
    "                df_newdoc.at[i,'Comments'] = x['comments'+str(r['LineNr'])] if 'comments'+str(r['LineNr']) in x else ''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c00b33ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 6 - Cell 8\n",
    "df_newdoc.head(30)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d8e0dc7",
   "metadata": {},
   "source": [
    "### Step 7 - Save changes to Amazon DynamoDB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8999a7b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 7 - Cell 1\n",
    "# Create the Amazon DynamoDB table - note that a new DynamoDB table is created everytime you execute this cell\n",
    "\n",
    "# Get the service resource.\n",
    "dynamodb = boto3.resource('dynamodb')\n",
    "tablename = \"SEC-registration-\"+str(uuid.uuid4())\n",
    "\n",
    "# Create the DynamoDB table.\n",
    "table = dynamodb.create_table(\n",
    "    TableName=tablename,\n",
    "    KeySchema=[\n",
    "        {\n",
    "            'AttributeName': 'row_nr',\n",
    "            'KeyType': 'HASH'\n",
    "        }\n",
    "    ],\n",
    "    AttributeDefinitions=[\n",
    "        {\n",
    "            'AttributeName': 'row_nr',\n",
    "            'AttributeType': 'N'\n",
    "        },\n",
    "    ],\n",
    "    ProvisionedThroughput={\n",
    "        'ReadCapacityUnits': 5,\n",
    "        'WriteCapacityUnits': 5\n",
    "    }\n",
    ")\n",
    "\n",
    "# Wait until the table exists, this will take a minute or so\n",
    "table.meta.client.get_waiter('table_exists').wait(TableName=tablename)\n",
    "\n",
    "# Print out some data about the table.\n",
    "print(\"Table successfully created\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "40565631",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 7 - Cell 2\n",
    "# Load the Amazon DynamoDB table\n",
    "\n",
    "for idx, row in df_newdoc.iterrows():\n",
    "    table.put_item(\n",
    "       Item={\n",
    "        'row_nr': idx,\n",
    "        'doc_name': str(row['DocName']) ,\n",
    "        'line_nr': str(row['LineNr']),\n",
    "        'detected_line': str(row['DetectedText']),\n",
    "        'confidence': str(row['Confidence']),   \n",
    "        'corrected_line': str(row['CorrectedText']),\n",
    "        'change_comments': str(row['Comments'])   \n",
    "        }\n",
    "    )\n",
    "\n",
    "print(\"Items were successfully created in DynamoDB table\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d14df917",
   "metadata": {},
   "source": [
    "### End of Notebook\n",
    "Please go back to Chapter 16 - Improve accuracy of pdf batch processing with Amazon Textract and Amazon A2I from the Natural Language Processing with AWS AI Services book to proceed further. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}