samples/python/extraction/vision/document-extraction-gpt-vision.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Document Extraction with Azure OpenAI GPT-4o (Vision Only)\n",
    "\n",
    "This sample demonstrates how to extract structured data from any document using Azure OpenAI's GPT-4o model with vision capabilities.\n",
    "\n",
    "![Data Extraction](../../../../images/extraction-openai.png)\n",
    "\n",
    "This is achieved by the following process:\n",
    "\n",
    "- Construct a system prompt that defines the instruction for extracting structured data from documents.\n",
    "- Construct a user prompt that includes the specific extraction instruction for the type of document, and each document page as a base64 encoded image.\n",
    "- Use the Azure OpenAI chat completions API with the GPT-4o model to generate a structured output from the content.\n",
    "\n",
    "## Objectives\n",
    "\n",
    "By the end of this sample, you will have learned how to:\n",
    "\n",
    "- Convert a document into a set of base64 encoded images for processing by GPT-4o.\n",
    "- Use prompt engineering techniques to instruct GPT-4o to extract structured data from a type of document.\n",
    "- Use the [Structured Outputs feature](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/structured-outputs?tabs=python-secure) to extract structured data from the document page images using Azure OpenAI's GPT-4o model.\n",
    "- Use the [logprobs](https://learn.microsoft.com/en-us/azure/ai-services/openai/reference#request-body:~:text=False-,logprobs,-integer) parameter in an OpenAI request to determine the confidence of the extracted structured output.\n",
    "\n",
    "## Useful Tips\n",
    "\n",
    "- Combine this technique with a [page classification](../../classification/README.md) approach to reduce the number of pages to extract from to only those that match your criteria for extraction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import modules\n",
    "\n",
    "This sample takes advantage of the following Python dependencies:\n",
    "\n",
    "- **pdf2image** for converting a PDF file into a set of images per page.\n",
    "- **openai** to interface with the Azure OpenAI chat completions API to generate structured extraction outputs using the GPT-4o model.\n",
    "- **azure-identity** to securely authenticate with deployed Azure Services using Microsoft Entra ID credentials.\n",
    "\n",
    "The following local components are also used:\n",
    "\n",
    "- [**vehicle_insurance_policy**](../../modules/samples/models/vehicle_insurance_policy.py) to provide the expected structured output JSON schema for vehicle insurance policy documents.\n",
    "- [**accuracy_evaluator**](../../modules/samples/evaluation/accuracy_evaluator.py) to evaluate the output of the extraction process with expected results.\n",
    "- [**openai_confidence**](../../modules/samples/confidence/openai_confidence.py) to calculate the confidence of the extraction process based on the `logprobs` response from the OpenAI API request.\n",
    "- [**document_processing_result**](../../modules/samples/models/document_processing_result.py) to store the results of the extraction process as a file.\n",
    "- [**stopwatch**](../../modules/samples/utils/stopwatch.py) to measure the end-to-end execution time for the extraction process.\n",
    "- [**app_settings**](../../modules/samples/app_settings.py) to access environment variables from the `.env` file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append('../../modules/') # Import local modules\n",
    "\n",
    "from IPython.display import display\n",
    "import os\n",
    "import pandas as pd\n",
    "from dotenv import dotenv_values\n",
    "import base64\n",
    "import io\n",
    "import json\n",
    "from openai import AzureOpenAI\n",
    "from azure.identity import DefaultAzureCredential, get_bearer_token_provider\n",
    "from concurrent.futures import ThreadPoolExecutor\n",
    "from pdf2image import convert_from_bytes\n",
    "\n",
    "from samples.app_settings import AppSettings\n",
    "from samples.utils.stopwatch import Stopwatch\n",
    "from samples.utils.storage_utils import create_json_file\n",
    "from samples.models.document_processing_result import DataExtractionResult\n",
    "\n",
    "from samples.models.vehicle_insurance_policy import VehicleInsurancePolicy\n",
    "from samples.confidence.confidence_utils import merge_confidence_values\n",
    "from samples.confidence.openai_confidence import evaluate_confidence as evaluate_openai_confidence\n",
    "from samples.evaluation.accuracy_evaluator import AccuracyEvaluator\n",
    "from samples.evaluation.comparison import get_extraction_comparison"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure the Azure services\n",
    "\n",
    "To use Azure OpenAI, the SDK is used to create a client instance using a deployed endpoint and authentication credentials.\n",
    "\n",
    "For this sample, the credentials of the Azure CLI are used to authenticate with the deployed services."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the working directory to the root of the repo\n",
    "working_dir = os.path.abspath('../../../../')\n",
    "settings = AppSettings(dotenv_values(f\"{working_dir}/.env\"))\n",
    "sample_path = f\"{working_dir}/samples/python/extraction/vision\"\n",
    "sample_name = \"document-extraction-gpt-vision\"\n",
    "\n",
    "# Configure the default credential for accessing Azure services using Azure CLI credentials\n",
    "credential = DefaultAzureCredential(\n",
    "    exclude_workload_identity_credential=True,\n",
    "    exclude_developer_cli_credential=True,\n",
    "    exclude_environment_credential=True,\n",
    "    exclude_managed_identity_credential=True,\n",
    "    exclude_powershell_credential=True,\n",
    "    exclude_shared_token_cache_credential=True,\n",
    "    exclude_interactive_browser_credential=True\n",
    ")\n",
    "\n",
    "openai_token_provider = get_bearer_token_provider(credential, 'https://cognitiveservices.azure.com/.default')\n",
    "\n",
    "openai_client = AzureOpenAI(\n",
    "    azure_endpoint=settings.openai_endpoint,\n",
    "    azure_ad_token_provider=openai_token_provider,\n",
    "    api_version=\"2024-12-01-preview\" # Requires the latest API version for structured outputs.\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Establish the expected output\n",
    "\n",
    "To compare the accuracy of the extraction process, the expected output of the extraction process has been defined in the following code block based on each page of a [Vehicle Insurance Policy](../../../assets/vehicle_insurance/policy_1.pdf).\n",
    "\n",
    "> **Note**: More insurance policy examples can be found in the [assets folder](../../../assets/vehicle_insurance). These examples include the PDF file and an associated JSON metadata file that provides the expected structured output. You can add your own scenarios by following the same structure.\n",
    "\n",
    "The expected output has been defined by a human evaluating the document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = f\"{working_dir}/samples/assets/vehicle_insurance/\"\n",
    "metadata_fname = \"policy_5.json\" # Change this to the file you want to evaluate\n",
    "metadata_fpath = f\"{path}{metadata_fname}\"\n",
    "\n",
    "with open(metadata_fpath, 'r') as f:\n",
    "    data = json.load(f)\n",
    "    \n",
    "expected = VehicleInsurancePolicy(**data['expected'])\n",
    "pdf_fname = data['fname']\n",
    "pdf_fpath = f\"{path}{pdf_fname}\"\n",
    "\n",
    "insurance_policy_evaluator = AccuracyEvaluator(match_keys=[])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extract data from the document\n",
    "\n",
    "The following code block executes the data extraction process using Azure OpenAI's GPT-4o model using vision capabilities.\n",
    "\n",
    "It performs the following steps:\n",
    "\n",
    "1. Get the document bytes from the provided file path. _Note: In this example, we are processing a local document, however, you can use any document storage location of your choice, such as Azure Blob Storage._\n",
    "2. Use pdf2image to convert the document's pages into images per page as base64 strings.\n",
    "3. Using Azure OpenAI's GPT-4o model and its [Structured Outputs feature](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/structured-outputs?tabs=python-secure), extract a structured data transfer object (DTO) from the content of the images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "system_prompt = f\"\"\"You are an AI assistant that extracts data from documents.\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prepare the user content for the OpenAI API including any specific details for processing this type of document and the document page images.\n",
    "user_content = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_text_prompt = \"\"\"Extract the data from this insurance policy. \n",
    "- If a value is not present, provide null.\n",
    "- Some values must be inferred based on the rules defined in the policy.\n",
    "- Dates should be in the format YYYY-MM-DD.\"\"\"\n",
    "\n",
    "user_content.append({\n",
    "    \"type\": \"text\",\n",
    "    \"text\": user_text_prompt\n",
    "})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def encode_page(page):\n",
    "    byte_io = io.BytesIO()\n",
    "    page.save(byte_io, format='PNG')\n",
    "    base64_data = base64.b64encode(byte_io.getvalue()).decode('utf-8')\n",
    "    return {\n",
    "        \"type\": \"image_url\",\n",
    "        \"image_url\": {\n",
    "            \"url\": f\"data:image/png;base64,{base64_data}\"\n",
    "        }\n",
    "    }\n",
    "\n",
    "with Stopwatch() as image_stopwatch:\n",
    "    with open(pdf_fpath, \"rb\") as f:\n",
    "        document_bytes = f.read()\n",
    "\n",
    "    pages = convert_from_bytes(document_bytes)\n",
    "    \n",
    "    # Process each page in parallel using multiple processes\n",
    "    with ThreadPoolExecutor() as executor:\n",
    "        results = list(executor.map(encode_page, pages))\n",
    "        user_content.extend(results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "with Stopwatch() as oai_stopwatch:\n",
    "    completion = openai_client.beta.chat.completions.parse(\n",
    "        model=settings.gpt4o_model_deployment_name,\n",
    "        messages=[\n",
    "            {\n",
    "                \"role\": \"system\",\n",
    "                \"content\": system_prompt,\n",
    "            },\n",
    "            {\n",
    "                \"role\": \"user\",\n",
    "                \"content\": user_content\n",
    "            }\n",
    "        ],\n",
    "        response_format=VehicleInsurancePolicy,\n",
    "        max_tokens=4096,\n",
    "        temperature=0.1,\n",
    "        top_p=0.1,\n",
    "        logprobs=True # Enabled to determine the confidence of the response.\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Understanding the Structured Outputs JSON schema\n",
    "\n",
    "Using [Pydantic's JSON schema feature](https://docs.pydantic.dev/latest/concepts/json_schema/), the [Insurance Policy](../../modules/samples/models/vehicle_insurance_policy.py) data model is automatically converted to a JSON schema when applied to the `response_format` parameter of the OpenAI chat completions request.\n",
    "\n",
    "The JSON schema is used to instruct the GPT-4o model to generate a strict output that adheres to the structure defined. The approach using Pydantic makes it easier for developers to manage the data structure in code, with helpful descriptions and examples that will be included in the final JSON schema.\n",
    "\n",
    "Demonstrated below, you can see how the Insurance Policy data model is understood by the OpenAI request:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"$defs\": {\n",
      "    \"VehicleInsuranceCostDetails\": {\n",
      "      \"description\": \"A class representing the cost details of a vehicle insurance policy.\\n\\nAttributes:\\n    annual_total: The annual total cost of the vehicle insurance policy.\\n    payable_by_date: The date by which the vehicle insurance policy must be paid.\",\n",
      "      \"properties\": {\n",
      "        \"annual_total\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"number\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"The annual total cost of the vehicle insurance policy.\",\n",
      "          \"title\": \"Annual Total\"\n",
      "        },\n",
      "        \"payable_by_date\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"The date by which the vehicle insurance policy must be paid.\",\n",
      "          \"title\": \"Payable By Date\"\n",
      "        }\n",
      "      },\n",
      "      \"required\": [\n",
      "        \"annual_total\",\n",
      "        \"payable_by_date\"\n",
      "      ],\n",
      "      \"title\": \"VehicleInsuranceCostDetails\",\n",
      "      \"type\": \"object\"\n",
      "    },\n",
      "    \"VehicleInsuranceExcessDetails\": {\n",
      "      \"description\": \"A class representing the excess details of a vehicle insurance policy.\\n\\nAttributes:\\n    compulsory: The compulsory excess amount.\\n    voluntary: The voluntary excess amount.\\n    unapproved_repair_penalty: The penalty amount for repairs by unapproved repairers.\",\n",
      "      \"properties\": {\n",
      "        \"compulsory\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"integer\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"The compulsory excess amount.\",\n",
      "          \"title\": \"Compulsory\"\n",
      "        },\n",
      "        \"voluntary\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"integer\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"The voluntary excess amount.\",\n",
      "          \"title\": \"Voluntary\"\n",
      "        },\n",
      "        \"unapproved_repair_penalty\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"integer\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"The penalty amount for repairs by unapproved repairers.\",\n",
      "          \"title\": \"Unapproved Repair Penalty\"\n",
      "        }\n",
      "      },\n",
      "      \"required\": [\n",
      "        \"compulsory\",\n",
      "        \"voluntary\",\n",
      "        \"unapproved_repair_penalty\"\n",
      "      ],\n",
      "      \"title\": \"VehicleInsuranceExcessDetails\",\n",
      "      \"type\": \"object\"\n",
      "    },\n",
      "    \"VehicleInsurancePersonDetails\": {\n",
      "      \"description\": \"A class representing the person details of a vehicle insurance policy.\\n\\nAttributes:\\n    first_name: The first name of the person.\\n    last_name: The last name of the person.\\n    date_of_birth: The date of birth of the person.\\n    address: The current address of the person.\\n    email_address: The email address of the person.\\n    total_years_of_residence_in_uk: The total years the person has resided in the UK.\\n    driving_license_number: The driving license number of the person.\",\n",
      "      \"properties\": {\n",
      "        \"first_name\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"The first name of the person.\",\n",
      "          \"title\": \"First Name\"\n",
      "        },\n",
      "        \"last_name\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"The last name or surname of the person.\",\n",
      "          \"title\": \"Last Name\"\n",
      "        },\n",
      "        \"date_of_birth\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"The date of birth of the person.\",\n",
      "          \"title\": \"Date Of Birth\"\n",
      "        },\n",
      "        \"address\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"The current address of the person.\",\n",
      "          \"title\": \"Address\"\n",
      "        },\n",
      "        \"email_address\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"The email address of the person.\",\n",
      "          \"title\": \"Email Address\"\n",
      "        },\n",
      "        \"total_years_of_residence_in_uk\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"integer\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"The total years the person has resided in the UK.\",\n",
      "          \"title\": \"Total Years Of Residence In Uk\"\n",
      "        },\n",
      "        \"driving_license_number\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"The driving license number of the person.\",\n",
      "          \"title\": \"Driving License Number\"\n",
      "        }\n",
      "      },\n",
      "      \"required\": [\n",
      "        \"first_name\",\n",
      "        \"last_name\",\n",
      "        \"date_of_birth\",\n",
      "        \"address\",\n",
      "        \"email_address\",\n",
      "        \"total_years_of_residence_in_uk\",\n",
      "        \"driving_license_number\"\n",
      "      ],\n",
      "      \"title\": \"VehicleInsurancePersonDetails\",\n",
      "      \"type\": \"object\"\n",
      "    },\n",
      "    \"VehicleInsuranceRenewalDetails\": {\n",
      "      \"description\": \"A class representing the renewal details of a vehicle insurance policy.\\n\\nAttributes:\\n    renewal_notification_date: The date on which the renewal notification is sent.\\n    last_date_to_renew: The last date before renewal is required.\",\n",
      "      \"properties\": {\n",
      "        \"renewal_notification_date\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"The date on which the renewal notification is sent.\",\n",
      "          \"title\": \"Renewal Notification Date\"\n",
      "        },\n",
      "        \"last_date_to_renew\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"The last date before renewal is required.\",\n",
      "          \"title\": \"Last Date To Renew\"\n",
      "        }\n",
      "      },\n",
      "      \"required\": [\n",
      "        \"renewal_notification_date\",\n",
      "        \"last_date_to_renew\"\n",
      "      ],\n",
      "      \"title\": \"VehicleInsuranceRenewalDetails\",\n",
      "      \"type\": \"object\"\n",
      "    },\n",
      "    \"VehicleInsuranceVehicleDetails\": {\n",
      "      \"description\": \"A class representing the vehicle details of a vehicle insurance policy.\\n\\nAttributes:\\n    registration_number: The registration number of the vehicle.\\n    make: The make of the vehicle.\\n    model: The model of the vehicle.\\n    year: The year the vehicle was manufactured.\\n    value: The current value of the vehicle.\",\n",
      "      \"properties\": {\n",
      "        \"registration_number\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"The registration number of the vehicle.\",\n",
      "          \"title\": \"Registration Number\"\n",
      "        },\n",
      "        \"make\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"The make of the vehicle.\",\n",
      "          \"title\": \"Make\"\n",
      "        },\n",
      "        \"model\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"The model of the vehicle.\",\n",
      "          \"title\": \"Model\"\n",
      "        },\n",
      "        \"year\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"integer\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"The year the vehicle was manufactured.\",\n",
      "          \"title\": \"Year\"\n",
      "        },\n",
      "        \"value\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"number\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"The current value of the vehicle.\",\n",
      "          \"title\": \"Value\"\n",
      "        }\n",
      "      },\n",
      "      \"required\": [\n",
      "        \"registration_number\",\n",
      "        \"make\",\n",
      "        \"model\",\n",
      "        \"year\",\n",
      "        \"value\"\n",
      "      ],\n",
      "      \"title\": \"VehicleInsuranceVehicleDetails\",\n",
      "      \"type\": \"object\"\n",
      "    }\n",
      "  },\n",
      "  \"description\": \"A class representing a vehicle insurance policy.\\n\\nAttributes:\\n    policy_number: The policy number of the vehicle insurance policy.\\n    cost: The cost details of the vehicle insurance policy.\\n    renewal: The renewal details of the vehicle insurance policy.\\n    effective_from: The effective date from which the vehicle insurance policy is valid.\\n    effective_to: The effective date to which the vehicle insurance policy is valid.\\n    last_date_to_cancel: The last date to cancel the vehicle insurance policy.\\n    policyholder: The person details of the policyholder.\\n    vehicle: The vehicle details of the policy.\\n    accident_excess: The excess costs for accidents.\\n    fire_and_theft_excess: The excess costs for fire and theft.\",\n",
      "  \"properties\": {\n",
      "    \"policy_number\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"type\": \"string\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"The policy number of the vehicle insurance policy.\",\n",
      "      \"title\": \"Policy Number\"\n",
      "    },\n",
      "    \"cost\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"$ref\": \"#/$defs/VehicleInsuranceCostDetails\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"The cost details of the vehicle insurance policy.\"\n",
      "    },\n",
      "    \"renewal\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"$ref\": \"#/$defs/VehicleInsuranceRenewalDetails\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"The renewal details of the vehicle insurance policy.\"\n",
      "    },\n",
      "    \"effective_from\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"type\": \"string\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"The effective date from which the vehicle insurance policy is valid.\",\n",
      "      \"title\": \"Effective From\"\n",
      "    },\n",
      "    \"effective_to\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"type\": \"string\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"The effective date to which the vehicle insurance policy is valid.\",\n",
      "      \"title\": \"Effective To\"\n",
      "    },\n",
      "    \"last_date_to_cancel\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"type\": \"string\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"The last date to cancel the vehicle insurance policy.\",\n",
      "      \"title\": \"Last Date To Cancel\"\n",
      "    },\n",
      "    \"policyholder\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"$ref\": \"#/$defs/VehicleInsurancePersonDetails\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"The person details of the policyholder.\"\n",
      "    },\n",
      "    \"vehicle\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"$ref\": \"#/$defs/VehicleInsuranceVehicleDetails\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"The vehicle details of the policy.\"\n",
      "    },\n",
      "    \"accident_excess\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"$ref\": \"#/$defs/VehicleInsuranceExcessDetails\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"The excess costs for accidents.\"\n",
      "    },\n",
      "    \"fire_and_theft_excess\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"$ref\": \"#/$defs/VehicleInsuranceExcessDetails\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"The excess costs for fire and theft.\"\n",
      "    }\n",
      "  },\n",
      "  \"required\": [\n",
      "    \"policy_number\",\n",
      "    \"cost\",\n",
      "    \"renewal\",\n",
      "    \"effective_from\",\n",
      "    \"effective_to\",\n",
      "    \"last_date_to_cancel\",\n",
      "    \"policyholder\",\n",
      "    \"vehicle\",\n",
      "    \"accident_excess\",\n",
      "    \"fire_and_theft_excess\"\n",
      "  ],\n",
      "  \"title\": \"VehicleInsurancePolicy\",\n",
      "  \"type\": \"object\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "# Highlight the schema sent to the OpenAI model\n",
    "print(json.dumps(VehicleInsurancePolicy.model_json_schema(), indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize the outputs\n",
    "\n",
    "To provide context for the execution of the code, the following code blocks visualize the outputs of the data extraction process.\n",
    "\n",
    "This includes:\n",
    "\n",
    "- The accuracy of the structured data extraction comparing the expected output with the output generated by Azure OpenAI's GPT-4o model.\n",
    "- The confidence score of the structured data extraction based on the log probability of the output generated by Azure OpenAI's GPT-4o model.\n",
    "- The execution time of the end-to-end process.\n",
    "- The total number of tokens consumed by the GPT-4o model.\n",
    "- The side-by-side comparison of the expected output and the output generated by Azure OpenAI's GPT-4o model.\n",
    "\n",
    "### Understanding Accuracy vs Confidence\n",
    "\n",
    "When using AI to extract structured data, both confidence and accuracy are essential for different but complementary reasons.\n",
    "\n",
    "- **Accuracy** measures how close the AI model's output is to a ground truth or expected output. It reflects how well the model's predictions align with reality.\n",
    "  - Accuracy ensures consistency in the extraction process, which is crucial for downstream tasks using the data.\n",
    "- **Confidence** represents the AI model's internal assessment of how certain it is about its predictions.\n",
    "  - Confidence indicates that the model is certain about its predictions, which can be a useful indicator for human reviewers to step in for manual verification.\n",
    "\n",
    "High accuracy and high confidence are ideal, but in practice, there is often a trade-off between the two. While accuracy cannot always be self-assessed, confidence scores can and should be used to prioritize manual verification of low-confidence predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Gets the parsed VehicleInsurancePolicy object from the completion response.\n",
    "insurance_policy = completion.choices[0].message.parsed\n",
    "\n",
    "expected_dict = expected.model_dump()\n",
    "insurance_policy_dict = insurance_policy.model_dump()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Determines the accuracy of the extracted data against the expected values.\n",
    "accuracy = insurance_policy_evaluator.evaluate(expected=expected_dict, actual=insurance_policy_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Determines the confidence of the extracted data using the log probabilities of the OpenAI completion response.\n",
    "confidence = evaluate_openai_confidence(insurance_policy_dict, completion.choices[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Gets the total execution time of the data extraction process.\n",
    "total_elapsed = image_stopwatch.elapsed + oai_stopwatch.elapsed\n",
    "\n",
    "# Gets the prompt tokens and completion tokens from the completion response.\n",
    "prompt_tokens = completion.usage.prompt_tokens\n",
    "completion_tokens = completion.usage.completion_tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the output of the data extraction result.\n",
    "extraction_result = DataExtractionResult(insurance_policy_dict, confidence, accuracy, prompt_tokens, completion_tokens, total_elapsed)\n",
    "\n",
    "create_json_file(f\"{sample_path}/{sample_name}.{pdf_fname}.json\", extraction_result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>Confidence</th>\n",
       "      <th>Execution Time</th>\n",
       "      <th>Image Pre-processing Execution Time</th>\n",
       "      <th>OpenAI Execution Time</th>\n",
       "      <th>Prompt Tokens</th>\n",
       "      <th>Completion Tokens</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>88.46%</td>\n",
       "      <td>96.90%</td>\n",
       "      <td>23.25 seconds</td>\n",
       "      <td>2.64 seconds</td>\n",
       "      <td>20.61 seconds</td>\n",
       "      <td>9848</td>\n",
       "      <td>241</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Accuracy Confidence Execution Time Image Pre-processing Execution Time  \\\n",
       "0   88.46%     96.90%  23.25 seconds                        2.64 seconds   \n",
       "\n",
       "  OpenAI Execution Time  Prompt Tokens  Completion Tokens  \n",
       "0         20.61 seconds           9848                241  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_62385_row0_col0, #T_62385_row0_col1, #T_62385_row0_col2, #T_62385_row0_col3, #T_62385_row0_col4, #T_62385_row1_col0, #T_62385_row1_col1, #T_62385_row1_col2, #T_62385_row1_col3, #T_62385_row1_col4, #T_62385_row2_col0, #T_62385_row2_col1, #T_62385_row2_col2, #T_62385_row2_col3, #T_62385_row2_col4, #T_62385_row3_col0, #T_62385_row3_col1, #T_62385_row3_col2, #T_62385_row3_col3, #T_62385_row3_col4, #T_62385_row5_col0, #T_62385_row5_col1, #T_62385_row5_col2, #T_62385_row5_col3, #T_62385_row5_col4, #T_62385_row6_col0, #T_62385_row6_col1, #T_62385_row6_col2, #T_62385_row6_col3, #T_62385_row6_col4, #T_62385_row7_col0, #T_62385_row7_col1, #T_62385_row7_col2, #T_62385_row7_col3, #T_62385_row7_col4, #T_62385_row8_col0, #T_62385_row8_col1, #T_62385_row8_col2, #T_62385_row8_col3, #T_62385_row8_col4, #T_62385_row9_col0, #T_62385_row9_col1, #T_62385_row9_col2, #T_62385_row9_col3, #T_62385_row9_col4, #T_62385_row10_col0, #T_62385_row10_col1, #T_62385_row10_col2, #T_62385_row10_col3, #T_62385_row10_col4, #T_62385_row11_col0, #T_62385_row11_col1, #T_62385_row11_col2, #T_62385_row11_col3, #T_62385_row11_col4, #T_62385_row12_col0, #T_62385_row12_col1, #T_62385_row12_col2, #T_62385_row12_col3, #T_62385_row12_col4, #T_62385_row13_col0, #T_62385_row13_col1, #T_62385_row13_col2, #T_62385_row13_col3, #T_62385_row13_col4, #T_62385_row15_col0, #T_62385_row15_col1, #T_62385_row15_col2, #T_62385_row15_col3, #T_62385_row15_col4, #T_62385_row16_col0, #T_62385_row16_col1, #T_62385_row16_col2, #T_62385_row16_col3, #T_62385_row16_col4, #T_62385_row17_col0, #T_62385_row17_col1, #T_62385_row17_col2, #T_62385_row17_col3, #T_62385_row17_col4, #T_62385_row18_col0, #T_62385_row18_col1, #T_62385_row18_col2, #T_62385_row18_col3, #T_62385_row18_col4, #T_62385_row19_col0, #T_62385_row19_col1, #T_62385_row19_col2, #T_62385_row19_col3, #T_62385_row19_col4, #T_62385_row21_col0, #T_62385_row21_col1, #T_62385_row21_col2, #T_62385_row21_col3, #T_62385_row21_col4, #T_62385_row22_col0, #T_62385_row22_col1, #T_62385_row22_col2, #T_62385_row22_col3, #T_62385_row22_col4, #T_62385_row23_col0, #T_62385_row23_col1, #T_62385_row23_col2, #T_62385_row23_col3, #T_62385_row23_col4, #T_62385_row24_col0, #T_62385_row24_col1, #T_62385_row24_col2, #T_62385_row24_col3, #T_62385_row24_col4, #T_62385_row25_col0, #T_62385_row25_col1, #T_62385_row25_col2, #T_62385_row25_col3, #T_62385_row25_col4 {\n",
       "  background-color: #66ff33;\n",
       "}\n",
       "#T_62385_row4_col0, #T_62385_row4_col1, #T_62385_row4_col2, #T_62385_row4_col3, #T_62385_row4_col4, #T_62385_row14_col0, #T_62385_row14_col1, #T_62385_row14_col2, #T_62385_row14_col3, #T_62385_row14_col4, #T_62385_row20_col0, #T_62385_row20_col1, #T_62385_row20_col2, #T_62385_row20_col3, #T_62385_row20_col4 {\n",
       "  background-color: #ff9999;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_62385\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_62385_level0_col0\" class=\"col_heading level0 col0\" >Field</th>\n",
       "      <th id=\"T_62385_level0_col1\" class=\"col_heading level0 col1\" >Expected</th>\n",
       "      <th id=\"T_62385_level0_col2\" class=\"col_heading level0 col2\" >Extracted</th>\n",
       "      <th id=\"T_62385_level0_col3\" class=\"col_heading level0 col3\" >Confidence</th>\n",
       "      <th id=\"T_62385_level0_col4\" class=\"col_heading level0 col4\" >Accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_62385_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "      <td id=\"T_62385_row0_col0\" class=\"data row0 col0\" >accident_excess_compulsory</td>\n",
       "      <td id=\"T_62385_row0_col1\" class=\"data row0 col1\" >300</td>\n",
       "      <td id=\"T_62385_row0_col2\" class=\"data row0 col2\" >300</td>\n",
       "      <td id=\"T_62385_row0_col3\" class=\"data row0 col3\" >99.92%</td>\n",
       "      <td id=\"T_62385_row0_col4\" class=\"data row0 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_62385_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
       "      <td id=\"T_62385_row1_col0\" class=\"data row1 col0\" >accident_excess_unapproved_repair_penalty</td>\n",
       "      <td id=\"T_62385_row1_col1\" class=\"data row1 col1\" >250</td>\n",
       "      <td id=\"T_62385_row1_col2\" class=\"data row1 col2\" >250</td>\n",
       "      <td id=\"T_62385_row1_col3\" class=\"data row1 col3\" >98.16%</td>\n",
       "      <td id=\"T_62385_row1_col4\" class=\"data row1 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_62385_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
       "      <td id=\"T_62385_row2_col0\" class=\"data row2 col0\" >accident_excess_voluntary</td>\n",
       "      <td id=\"T_62385_row2_col1\" class=\"data row2 col1\" >200</td>\n",
       "      <td id=\"T_62385_row2_col2\" class=\"data row2 col2\" >200</td>\n",
       "      <td id=\"T_62385_row2_col3\" class=\"data row2 col3\" >99.98%</td>\n",
       "      <td id=\"T_62385_row2_col4\" class=\"data row2 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_62385_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
       "      <td id=\"T_62385_row3_col0\" class=\"data row3 col0\" >cost_annual_total</td>\n",
       "      <td id=\"T_62385_row3_col1\" class=\"data row3 col1\" >750.000000</td>\n",
       "      <td id=\"T_62385_row3_col2\" class=\"data row3 col2\" >750.000000</td>\n",
       "      <td id=\"T_62385_row3_col3\" class=\"data row3 col3\" >87.28%</td>\n",
       "      <td id=\"T_62385_row3_col4\" class=\"data row3 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_62385_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
       "      <td id=\"T_62385_row4_col0\" class=\"data row4 col0\" >cost_payable_by_date</td>\n",
       "      <td id=\"T_62385_row4_col1\" class=\"data row4 col1\" >2024-05-22</td>\n",
       "      <td id=\"T_62385_row4_col2\" class=\"data row4 col2\" >None</td>\n",
       "      <td id=\"T_62385_row4_col3\" class=\"data row4 col3\" >0.00%</td>\n",
       "      <td id=\"T_62385_row4_col4\" class=\"data row4 col4\" >Mismatch</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_62385_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
       "      <td id=\"T_62385_row5_col0\" class=\"data row5 col0\" >effective_from</td>\n",
       "      <td id=\"T_62385_row5_col1\" class=\"data row5 col1\" >2024-05-12</td>\n",
       "      <td id=\"T_62385_row5_col2\" class=\"data row5 col2\" >2024-05-12</td>\n",
       "      <td id=\"T_62385_row5_col3\" class=\"data row5 col3\" >100.00%</td>\n",
       "      <td id=\"T_62385_row5_col4\" class=\"data row5 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_62385_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
       "      <td id=\"T_62385_row6_col0\" class=\"data row6 col0\" >effective_to</td>\n",
       "      <td id=\"T_62385_row6_col1\" class=\"data row6 col1\" >2025-05-11</td>\n",
       "      <td id=\"T_62385_row6_col2\" class=\"data row6 col2\" >2025-05-11</td>\n",
       "      <td id=\"T_62385_row6_col3\" class=\"data row6 col3\" >99.99%</td>\n",
       "      <td id=\"T_62385_row6_col4\" class=\"data row6 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_62385_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
       "      <td id=\"T_62385_row7_col0\" class=\"data row7 col0\" >fire_and_theft_excess_compulsory</td>\n",
       "      <td id=\"T_62385_row7_col1\" class=\"data row7 col1\" >250</td>\n",
       "      <td id=\"T_62385_row7_col2\" class=\"data row7 col2\" >250</td>\n",
       "      <td id=\"T_62385_row7_col3\" class=\"data row7 col3\" >99.99%</td>\n",
       "      <td id=\"T_62385_row7_col4\" class=\"data row7 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_62385_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
       "      <td id=\"T_62385_row8_col0\" class=\"data row8 col0\" >fire_and_theft_excess_unapproved_repair_penalty</td>\n",
       "      <td id=\"T_62385_row8_col1\" class=\"data row8 col1\" >250</td>\n",
       "      <td id=\"T_62385_row8_col2\" class=\"data row8 col2\" >250</td>\n",
       "      <td id=\"T_62385_row8_col3\" class=\"data row8 col3\" >99.50%</td>\n",
       "      <td id=\"T_62385_row8_col4\" class=\"data row8 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_62385_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
       "      <td id=\"T_62385_row9_col0\" class=\"data row9 col0\" >fire_and_theft_excess_voluntary</td>\n",
       "      <td id=\"T_62385_row9_col1\" class=\"data row9 col1\" >150</td>\n",
       "      <td id=\"T_62385_row9_col2\" class=\"data row9 col2\" >150</td>\n",
       "      <td id=\"T_62385_row9_col3\" class=\"data row9 col3\" >99.99%</td>\n",
       "      <td id=\"T_62385_row9_col4\" class=\"data row9 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_62385_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
       "      <td id=\"T_62385_row10_col0\" class=\"data row10 col0\" >last_date_to_cancel</td>\n",
       "      <td id=\"T_62385_row10_col1\" class=\"data row10 col1\" >2024-05-26</td>\n",
       "      <td id=\"T_62385_row10_col2\" class=\"data row10 col2\" >2024-05-26</td>\n",
       "      <td id=\"T_62385_row10_col3\" class=\"data row10 col3\" >91.61%</td>\n",
       "      <td id=\"T_62385_row10_col4\" class=\"data row10 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_62385_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
       "      <td id=\"T_62385_row11_col0\" class=\"data row11 col0\" >policy_number</td>\n",
       "      <td id=\"T_62385_row11_col1\" class=\"data row11 col1\" >GB34567890123</td>\n",
       "      <td id=\"T_62385_row11_col2\" class=\"data row11 col2\" >GB34567890123</td>\n",
       "      <td id=\"T_62385_row11_col3\" class=\"data row11 col3\" >100.00%</td>\n",
       "      <td id=\"T_62385_row11_col4\" class=\"data row11 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_62385_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
       "      <td id=\"T_62385_row12_col0\" class=\"data row12 col0\" >policyholder_address</td>\n",
       "      <td id=\"T_62385_row12_col1\" class=\"data row12 col1\" >18 Elm Grove, Bristol, BS1 5HQ</td>\n",
       "      <td id=\"T_62385_row12_col2\" class=\"data row12 col2\" >18 Elm Grove, Bristol, BS1 5HQ</td>\n",
       "      <td id=\"T_62385_row12_col3\" class=\"data row12 col3\" >99.91%</td>\n",
       "      <td id=\"T_62385_row12_col4\" class=\"data row12 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_62385_level0_row13\" class=\"row_heading level0 row13\" >13</th>\n",
       "      <td id=\"T_62385_row13_col0\" class=\"data row13 col0\" >policyholder_date_of_birth</td>\n",
       "      <td id=\"T_62385_row13_col1\" class=\"data row13 col1\" >1990-09-08</td>\n",
       "      <td id=\"T_62385_row13_col2\" class=\"data row13 col2\" >1990-09-08</td>\n",
       "      <td id=\"T_62385_row13_col3\" class=\"data row13 col3\" >93.75%</td>\n",
       "      <td id=\"T_62385_row13_col4\" class=\"data row13 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_62385_level0_row14\" class=\"row_heading level0 row14\" >14</th>\n",
       "      <td id=\"T_62385_row14_col0\" class=\"data row14 col0\" >policyholder_driving_license_number</td>\n",
       "      <td id=\"T_62385_row14_col1\" class=\"data row14 col1\" >BENNEL008099JJ9IT</td>\n",
       "      <td id=\"T_62385_row14_col2\" class=\"data row14 col2\" >BENNEL008099J9IT</td>\n",
       "      <td id=\"T_62385_row14_col3\" class=\"data row14 col3\" >91.75%</td>\n",
       "      <td id=\"T_62385_row14_col4\" class=\"data row14 col4\" >Mismatch</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_62385_level0_row15\" class=\"row_heading level0 row15\" >15</th>\n",
       "      <td id=\"T_62385_row15_col0\" class=\"data row15 col0\" >policyholder_email_address</td>\n",
       "      <td id=\"T_62385_row15_col1\" class=\"data row15 col1\" >LBennett@me.com</td>\n",
       "      <td id=\"T_62385_row15_col2\" class=\"data row15 col2\" >LBennett@me.com</td>\n",
       "      <td id=\"T_62385_row15_col3\" class=\"data row15 col3\" >99.77%</td>\n",
       "      <td id=\"T_62385_row15_col4\" class=\"data row15 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_62385_level0_row16\" class=\"row_heading level0 row16\" >16</th>\n",
       "      <td id=\"T_62385_row16_col0\" class=\"data row16 col0\" >policyholder_first_name</td>\n",
       "      <td id=\"T_62385_row16_col1\" class=\"data row16 col1\" >Laura</td>\n",
       "      <td id=\"T_62385_row16_col2\" class=\"data row16 col2\" >Laura</td>\n",
       "      <td id=\"T_62385_row16_col3\" class=\"data row16 col3\" >99.96%</td>\n",
       "      <td id=\"T_62385_row16_col4\" class=\"data row16 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_62385_level0_row17\" class=\"row_heading level0 row17\" >17</th>\n",
       "      <td id=\"T_62385_row17_col0\" class=\"data row17 col0\" >policyholder_last_name</td>\n",
       "      <td id=\"T_62385_row17_col1\" class=\"data row17 col1\" >Bennett</td>\n",
       "      <td id=\"T_62385_row17_col2\" class=\"data row17 col2\" >Bennett</td>\n",
       "      <td id=\"T_62385_row17_col3\" class=\"data row17 col3\" >100.00%</td>\n",
       "      <td id=\"T_62385_row17_col4\" class=\"data row17 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_62385_level0_row18\" class=\"row_heading level0 row18\" >18</th>\n",
       "      <td id=\"T_62385_row18_col0\" class=\"data row18 col0\" >policyholder_total_years_of_residence_in_uk</td>\n",
       "      <td id=\"T_62385_row18_col1\" class=\"data row18 col1\" >29</td>\n",
       "      <td id=\"T_62385_row18_col2\" class=\"data row18 col2\" >29</td>\n",
       "      <td id=\"T_62385_row18_col3\" class=\"data row18 col3\" >90.92%</td>\n",
       "      <td id=\"T_62385_row18_col4\" class=\"data row18 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_62385_level0_row19\" class=\"row_heading level0 row19\" >19</th>\n",
       "      <td id=\"T_62385_row19_col0\" class=\"data row19 col0\" >renewal_last_date_to_renew</td>\n",
       "      <td id=\"T_62385_row19_col1\" class=\"data row19 col1\" >2025-05-04</td>\n",
       "      <td id=\"T_62385_row19_col2\" class=\"data row19 col2\" >2025-05-04</td>\n",
       "      <td id=\"T_62385_row19_col3\" class=\"data row19 col3\" >82.07%</td>\n",
       "      <td id=\"T_62385_row19_col4\" class=\"data row19 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_62385_level0_row20\" class=\"row_heading level0 row20\" >20</th>\n",
       "      <td id=\"T_62385_row20_col0\" class=\"data row20 col0\" >renewal_renewal_notification_date</td>\n",
       "      <td id=\"T_62385_row20_col1\" class=\"data row20 col1\" >2025-04-20</td>\n",
       "      <td id=\"T_62385_row20_col2\" class=\"data row20 col2\" >None</td>\n",
       "      <td id=\"T_62385_row20_col3\" class=\"data row20 col3\" >0.00%</td>\n",
       "      <td id=\"T_62385_row20_col4\" class=\"data row20 col4\" >Mismatch</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_62385_level0_row21\" class=\"row_heading level0 row21\" >21</th>\n",
       "      <td id=\"T_62385_row21_col0\" class=\"data row21 col0\" >vehicle_make</td>\n",
       "      <td id=\"T_62385_row21_col1\" class=\"data row21 col1\" >Honda</td>\n",
       "      <td id=\"T_62385_row21_col2\" class=\"data row21 col2\" >Honda</td>\n",
       "      <td id=\"T_62385_row21_col3\" class=\"data row21 col3\" >99.85%</td>\n",
       "      <td id=\"T_62385_row21_col4\" class=\"data row21 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_62385_level0_row22\" class=\"row_heading level0 row22\" >22</th>\n",
       "      <td id=\"T_62385_row22_col0\" class=\"data row22 col0\" >vehicle_model</td>\n",
       "      <td id=\"T_62385_row22_col1\" class=\"data row22 col1\" >Civic</td>\n",
       "      <td id=\"T_62385_row22_col2\" class=\"data row22 col2\" >Civic</td>\n",
       "      <td id=\"T_62385_row22_col3\" class=\"data row22 col3\" >99.27%</td>\n",
       "      <td id=\"T_62385_row22_col4\" class=\"data row22 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_62385_level0_row23\" class=\"row_heading level0 row23\" >23</th>\n",
       "      <td id=\"T_62385_row23_col0\" class=\"data row23 col0\" >vehicle_registration_number</td>\n",
       "      <td id=\"T_62385_row23_col1\" class=\"data row23 col1\" >IJ90MNO</td>\n",
       "      <td id=\"T_62385_row23_col2\" class=\"data row23 col2\" >IJ90MNO</td>\n",
       "      <td id=\"T_62385_row23_col3\" class=\"data row23 col3\" >93.03%</td>\n",
       "      <td id=\"T_62385_row23_col4\" class=\"data row23 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_62385_level0_row24\" class=\"row_heading level0 row24\" >24</th>\n",
       "      <td id=\"T_62385_row24_col0\" class=\"data row24 col0\" >vehicle_value</td>\n",
       "      <td id=\"T_62385_row24_col1\" class=\"data row24 col1\" >11000.000000</td>\n",
       "      <td id=\"T_62385_row24_col2\" class=\"data row24 col2\" >11000.000000</td>\n",
       "      <td id=\"T_62385_row24_col3\" class=\"data row24 col3\" >98.91%</td>\n",
       "      <td id=\"T_62385_row24_col4\" class=\"data row24 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_62385_level0_row25\" class=\"row_heading level0 row25\" >25</th>\n",
       "      <td id=\"T_62385_row25_col0\" class=\"data row25 col0\" >vehicle_year</td>\n",
       "      <td id=\"T_62385_row25_col1\" class=\"data row25 col1\" >2019</td>\n",
       "      <td id=\"T_62385_row25_col2\" class=\"data row25 col2\" >2019</td>\n",
       "      <td id=\"T_62385_row25_col3\" class=\"data row25 col3\" >100.00%</td>\n",
       "      <td id=\"T_62385_row25_col4\" class=\"data row25 col4\" >Match</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7ff159de81d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Display the outputs of the data extraction process.\n",
    "df = pd.DataFrame([\n",
    "    {\n",
    "        \"Accuracy\": f\"{accuracy['overall'] * 100:.2f}%\",\n",
    "        \"Confidence\": f\"{confidence['_overall'] * 100:.2f}%\",\n",
    "        \"Execution Time\": f\"{total_elapsed:.2f} seconds\",\n",
    "        \"Image Pre-processing Execution Time\": f\"{image_stopwatch.elapsed:.2f} seconds\",\n",
    "        \"OpenAI Execution Time\": f\"{oai_stopwatch.elapsed:.2f} seconds\",\n",
    "        \"Prompt Tokens\": prompt_tokens,\n",
    "        \"Completion Tokens\": completion_tokens\n",
    "    }\n",
    "])\n",
    "\n",
    "display(df)\n",
    "display(get_extraction_comparison(expected_dict, insurance_policy_dict, confidence, accuracy['accuracy']))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}