samples/python/extraction/multimodal/document-extraction-gpt-text-and-vision.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Document Extraction using Multi-Modal (Text and Vision) Capabilities combining Azure AI Document Intelligence and Azure OpenAI's GPT-4o\n",
    "\n",
    "This sample demonstrates how to build a comprehensive process to extract structured data from any document using Azure AI Document Intelligence and Azure OpenAI's GPT-4o model with vision capabilities.\n",
    "\n",
    "![Data Extraction](../../../../images/extraction-comprehensive.png)\n",
    "\n",
    "This is achieved by the following process:\n",
    "\n",
    "- Analyze a document using Azure AI Document Intelligence's `prebuilt-layout` model to extract the structure as Markdown.\n",
    "- Construct a system prompt that defines the instruction for extracting structured data from documents.\n",
    "- Construct a user prompt that includes the specific extraction instruction for the type of document, the text content, and each document page as a base64 encoded image.\n",
    "- Use the Azure OpenAI chat completions API with the GPT-4o model to generate a structured output from the content.\n",
    "\n",
    "## Objectives\n",
    "\n",
    "By the end of this sample, you will have learned how to:\n",
    "\n",
    "- Convert a document to Markdown format using Azure AI Document Intelligence.\n",
    "- Convert a document into a set of base64 encoded images for processing by GPT-4o.\n",
    "- Use prompt engineering techniques to instruct GPT-4o to extract structured data from a type of document.\n",
    "- Use the [Structured Outputs feature](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/structured-outputs?tabs=python-secure) to extract structured data from the document page images using Azure OpenAI's GPT-4o model.\n",
    "- Use the analysis result from Azure AI Document Intelligence to determine the confidence of the extracted structured output.\n",
    "- Use the [logprobs](https://learn.microsoft.com/en-us/azure/ai-services/openai/reference#request-body:~:text=False-,logprobs,-integer) parameter in an OpenAI request to determine the confidence of the extracted structured output.\n",
    "\n",
    "## Useful Tips\n",
    "\n",
    "- Combine this technique with a [page classification](../../classification/README.md) approach to reduce the number of pages to extract from to only those that match your criteria for extraction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import modules\n",
    "\n",
    "This sample takes advantage of the following Python dependencies:\n",
    "\n",
    "- **pdf2image** for converting a PDF file into a set of images per page.\n",
    "- **azure-ai-documentintelligence** to interface with the Azure AI Document Intelligence API for analyzing documents.\n",
    "- **openai** to interface with the Azure OpenAI chat completions API to generate structured extraction outputs using the GPT-4o model.\n",
    "- **azure-identity** to securely authenticate with deployed Azure Services using Microsoft Entra ID credentials.\n",
    "\n",
    "The following local components are also used:\n",
    "\n",
    "- [**vehicle_insurance_policy**](../../modules/samples/models/vehicle_insurance_policy.py) to provide the expected structured output JSON schema for vehicle insurance policy documents.\n",
    "- [**accuracy_evaluator**](../../modules/samples/evaluation/accuracy_evaluator.py) to evaluate the output of the classification process with expected results.\n",
    "- [**document_intelligence_confidence**](../../modules/samples/confidence/document_intelligence_confidence.py) to calculate the confidence of the extraction process based on the analysis result from the Azure AI Document Intelligence API.\n",
    "- [**openai_confidence**](../../modules/samples/confidence/openai_confidence.py) to calculate the confidence of the extraction process based on the `logprobs` response from the OpenAI API request.\n",
    "- [**document_processing_result**](../../modules/samples/models/document_processing_result.py) to store the results of the extraction process as a file.\n",
    "- [**stopwatch**](../../modules/samples/utils/stopwatch.py) to measure the end-to-end execution time for the extraction process.\n",
    "- [**app_settings**](../../modules/samples/app_settings.py) to access environment variables from the `.env` file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append('../../modules/') # Import local modules\n",
    "\n",
    "from IPython.display import display, Markdown\n",
    "import os\n",
    "import pandas as pd\n",
    "from dotenv import dotenv_values\n",
    "import base64\n",
    "import io\n",
    "import json\n",
    "from openai import AzureOpenAI\n",
    "from azure.ai.documentintelligence import DocumentIntelligenceClient\n",
    "from azure.ai.documentintelligence.models import AnalyzeResult, DocumentContentFormat\n",
    "from azure.identity import DefaultAzureCredential, get_bearer_token_provider\n",
    "from concurrent.futures import ThreadPoolExecutor\n",
    "from pdf2image import convert_from_bytes\n",
    "\n",
    "from samples.app_settings import AppSettings\n",
    "from samples.utils.stopwatch import Stopwatch\n",
    "from samples.utils.storage_utils import create_json_file\n",
    "from samples.models.document_processing_result import DataExtractionResult\n",
    "\n",
    "from samples.models.vehicle_insurance_policy import VehicleInsurancePolicy\n",
    "from samples.confidence.confidence_utils import merge_confidence_values\n",
    "from samples.confidence.openai_confidence import evaluate_confidence as evaluate_openai_confidence\n",
    "from samples.confidence.document_intelligence_confidence import evaluate_confidence as evaluate_di_confidence\n",
    "from samples.evaluation.accuracy_evaluator import AccuracyEvaluator\n",
    "from samples.evaluation.comparison import get_extraction_comparison"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure the Azure services\n",
    "\n",
    "To use Azure AI Document Intelligence and Azure OpenAI, their SDKs are used to create client instances using a deployed endpoint and authentication credentials.\n",
    "\n",
    "For this sample, the credentials of the Azure CLI are used to authenticate with the deployed services."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the working directory to the root of the repo\n",
    "working_dir = os.path.abspath('../../../../')\n",
    "settings = AppSettings(dotenv_values(f\"{working_dir}/.env\"))\n",
    "sample_path = f\"{working_dir}/samples/python/extraction/multimodal\"\n",
    "sample_name = \"document-extraction-gpt-text-and-vision\"\n",
    "\n",
    "# Configure the default credential for accessing Azure services using Azure CLI credentials\n",
    "credential = DefaultAzureCredential(\n",
    "    exclude_workload_identity_credential=True,\n",
    "    exclude_developer_cli_credential=True,\n",
    "    exclude_environment_credential=True,\n",
    "    exclude_managed_identity_credential=True,\n",
    "    exclude_powershell_credential=True,\n",
    "    exclude_shared_token_cache_credential=True,\n",
    "    exclude_interactive_browser_credential=True\n",
    ")\n",
    "\n",
    "openai_token_provider = get_bearer_token_provider(credential, 'https://cognitiveservices.azure.com/.default')\n",
    "\n",
    "openai_client = AzureOpenAI(\n",
    "    azure_endpoint=settings.openai_endpoint,\n",
    "    azure_ad_token_provider=openai_token_provider,\n",
    "    api_version=\"2024-12-01-preview\" # Requires the latest API version for structured outputs.\n",
    ")\n",
    "\n",
    "document_intelligence_client = DocumentIntelligenceClient(\n",
    "    endpoint=settings.ai_services_endpoint,\n",
    "    credential=credential\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Establish the expected output\n",
    "\n",
    "To compare the accuracy of the extraction process, the expected output of the extraction process has been defined in the following code block based on each page of a [Vehicle Insurance Policy](../../../assets/vehicle_insurance/policy_1.pdf).\n",
    "\n",
    "> **Note**: More insurance policy examples can be found in the [assets folder](../../../assets/vehicle_insurance). These examples include the PDF file and an associated JSON metadata file that provides the expected structured output. You can add your own scenarios by following the same structure.\n",
    "\n",
    "The expected output has been defined by a human evaluating the document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = f\"{working_dir}/samples/assets/vehicle_insurance/\"\n",
    "metadata_fname = \"policy_5.json\" # Change this to the file you want to evaluate\n",
    "metadata_fpath = f\"{path}{metadata_fname}\"\n",
    "\n",
    "with open(metadata_fpath, 'r') as f:\n",
    "    data = json.load(f)\n",
    "    \n",
    "expected = VehicleInsurancePolicy(**data['expected'])\n",
    "pdf_fname = data['fname']\n",
    "pdf_fpath = f\"{path}{pdf_fname}\"\n",
    "\n",
    "insurance_policy_evaluator = AccuracyEvaluator(match_keys=[])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extract data from the document\n",
    "\n",
    "The following code block executes the data extraction process using Azure OpenAI's GPT-4o model using vision capabilities.\n",
    "\n",
    "It performs the following steps:\n",
    "\n",
    "1. Get the document bytes from the provided file path. _Note: In this example, we are processing a local document, however, you can use any document storage location of your choice, such as Azure Blob Storage._\n",
    "2. Use Azure AI Document Intelligence to analyze the structure of the document and convert it to Markdown format using the pre-built layout model.\n",
    "3. Use pdf2image to convert the document's pages into images per page as base64 strings.\n",
    "4. Using Azure OpenAI's GPT-4o model and its [Structured Outputs feature](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/structured-outputs?tabs=python-secure), extract a structured data transfer object (DTO) from the content of the images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "with Stopwatch() as di_stopwatch:\n",
    "    with open(pdf_fpath, \"rb\") as f:\n",
    "        poller = document_intelligence_client.begin_analyze_document(\n",
    "            model_id=\"prebuilt-layout\",\n",
    "            body=f,\n",
    "            output_content_format=DocumentContentFormat.MARKDOWN,\n",
    "            content_type=\"application/pdf\"\n",
    "        )\n",
    "    \n",
    "    result: AnalyzeResult = poller.result()\n",
    "\n",
    "markdown = result.content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "system_prompt = f\"\"\"You are an AI assistant that extracts data from documents.\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prepare the user content for the OpenAI API including any specific details for processing this type of document, text, and the document page images.\n",
    "user_content = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_text_prompt = \"\"\"Extract the data from this insurance policy. \n",
    "- If a value is not present, provide null.\n",
    "- Some values must be inferred based on the rules defined in the policy.\n",
    "- Dates should be in the format YYYY-MM-DD.\"\"\"\n",
    "\n",
    "user_content.append({\n",
    "    \"type\": \"text\",\n",
    "    \"text\": user_text_prompt\n",
    "})\n",
    "\n",
    "user_content.append({\n",
    "    \"type\": \"text\",\n",
    "    \"text\": markdown\n",
    "})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def encode_page(page):\n",
    "    byte_io = io.BytesIO()\n",
    "    page.save(byte_io, format='PNG')\n",
    "    base64_data = base64.b64encode(byte_io.getvalue()).decode('utf-8')\n",
    "    return {\n",
    "        \"type\": \"image_url\",\n",
    "        \"image_url\": {\n",
    "            \"url\": f\"data:image/png;base64,{base64_data}\"\n",
    "        }\n",
    "    }\n",
    "\n",
    "with Stopwatch() as image_stopwatch:\n",
    "    with open(pdf_fpath, \"rb\") as f:\n",
    "        document_bytes = f.read()\n",
    "\n",
    "    pages = convert_from_bytes(document_bytes)\n",
    "    \n",
    "    # Process each page in parallel using multiple processes\n",
    "    with ThreadPoolExecutor() as executor:\n",
    "        results = list(executor.map(encode_page, pages))\n",
    "        user_content.extend(results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "with Stopwatch() as oai_stopwatch:\n",
    "    completion = openai_client.beta.chat.completions.parse(\n",
    "        model=settings.gpt4o_model_deployment_name,\n",
    "        messages=[\n",
    "            {\n",
    "                \"role\": \"system\",\n",
    "                \"content\": system_prompt,\n",
    "            },\n",
    "            {\n",
    "                \"role\": \"user\",\n",
    "                \"content\": user_content\n",
    "            }\n",
    "        ],\n",
    "        response_format=VehicleInsurancePolicy,\n",
    "        max_tokens=4096,\n",
    "        temperature=0.1,\n",
    "        top_p=0.1,\n",
    "        logprobs=True # Enabled to determine the confidence of the response.\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Understanding the Structured Outputs JSON schema\n",
    "\n",
    "Using [Pydantic's JSON schema feature](https://docs.pydantic.dev/latest/concepts/json_schema/), the [Insurance Policy](../../modules/samples/models/vehicle_insurance_policy.py) data model is automatically converted to a JSON schema when applied to the `response_format` parameter of the OpenAI chat completions request.\n",
    "\n",
    "The JSON schema is used to instruct the GPT-4o model to generate a strict output that adheres to the structure defined. The approach using Pydantic makes it easier for developers to manage the data structure in code, with helpful descriptions and examples that will be included in the final JSON schema.\n",
    "\n",
    "Demonstrated below, you can see how the Insurance Policy data model is understood by the OpenAI request:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"$defs\": {\n",
      "    \"VehicleInsuranceCostDetails\": {\n",
      "      \"description\": \"A class representing the cost details of a vehicle insurance policy.\\n\\nAttributes:\\n    annual_total: The annual total cost of the vehicle insurance policy.\\n    payable_by_date: The date by which the vehicle insurance policy must be paid.\",\n",
      "      \"properties\": {\n",
      "        \"annual_total\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"number\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"The annual total cost of the vehicle insurance policy.\",\n",
      "          \"title\": \"Annual Total\"\n",
      "        },\n",
      "        \"payable_by_date\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"The date by which the vehicle insurance policy must be paid.\",\n",
      "          \"title\": \"Payable By Date\"\n",
      "        }\n",
      "      },\n",
      "      \"required\": [\n",
      "        \"annual_total\",\n",
      "        \"payable_by_date\"\n",
      "      ],\n",
      "      \"title\": \"VehicleInsuranceCostDetails\",\n",
      "      \"type\": \"object\"\n",
      "    },\n",
      "    \"VehicleInsuranceExcessDetails\": {\n",
      "      \"description\": \"A class representing the excess details of a vehicle insurance policy.\\n\\nAttributes:\\n    compulsory: The compulsory excess amount.\\n    voluntary: The voluntary excess amount.\\n    unapproved_repair_penalty: The penalty amount for repairs by unapproved repairers.\",\n",
      "      \"properties\": {\n",
      "        \"compulsory\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"integer\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"The compulsory excess amount.\",\n",
      "          \"title\": \"Compulsory\"\n",
      "        },\n",
      "        \"voluntary\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"integer\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"The voluntary excess amount.\",\n",
      "          \"title\": \"Voluntary\"\n",
      "        },\n",
      "        \"unapproved_repair_penalty\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"integer\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"The penalty amount for repairs by unapproved repairers.\",\n",
      "          \"title\": \"Unapproved Repair Penalty\"\n",
      "        }\n",
      "      },\n",
      "      \"required\": [\n",
      "        \"compulsory\",\n",
      "        \"voluntary\",\n",
      "        \"unapproved_repair_penalty\"\n",
      "      ],\n",
      "      \"title\": \"VehicleInsuranceExcessDetails\",\n",
      "      \"type\": \"object\"\n",
      "    },\n",
      "    \"VehicleInsurancePersonDetails\": {\n",
      "      \"description\": \"A class representing the person details of a vehicle insurance policy.\\n\\nAttributes:\\n    first_name: The first name of the person.\\n    last_name: The last name of the person.\\n    date_of_birth: The date of birth of the person.\\n    address: The current address of the person.\\n    email_address: The email address of the person.\\n    total_years_of_residence_in_uk: The total years the person has resided in the UK.\\n    driving_license_number: The driving license number of the person.\",\n",
      "      \"properties\": {\n",
      "        \"first_name\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"The first name of the person.\",\n",
      "          \"title\": \"First Name\"\n",
      "        },\n",
      "        \"last_name\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"The last name or surname of the person.\",\n",
      "          \"title\": \"Last Name\"\n",
      "        },\n",
      "        \"date_of_birth\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"The date of birth of the person.\",\n",
      "          \"title\": \"Date Of Birth\"\n",
      "        },\n",
      "        \"address\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"The current address of the person.\",\n",
      "          \"title\": \"Address\"\n",
      "        },\n",
      "        \"email_address\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"The email address of the person.\",\n",
      "          \"title\": \"Email Address\"\n",
      "        },\n",
      "        \"total_years_of_residence_in_uk\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"integer\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"The total years the person has resided in the UK.\",\n",
      "          \"title\": \"Total Years Of Residence In Uk\"\n",
      "        },\n",
      "        \"driving_license_number\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"The driving license number of the person.\",\n",
      "          \"title\": \"Driving License Number\"\n",
      "        }\n",
      "      },\n",
      "      \"required\": [\n",
      "        \"first_name\",\n",
      "        \"last_name\",\n",
      "        \"date_of_birth\",\n",
      "        \"address\",\n",
      "        \"email_address\",\n",
      "        \"total_years_of_residence_in_uk\",\n",
      "        \"driving_license_number\"\n",
      "      ],\n",
      "      \"title\": \"VehicleInsurancePersonDetails\",\n",
      "      \"type\": \"object\"\n",
      "    },\n",
      "    \"VehicleInsuranceRenewalDetails\": {\n",
      "      \"description\": \"A class representing the renewal details of a vehicle insurance policy.\\n\\nAttributes:\\n    renewal_notification_date: The date on which the renewal notification is sent.\\n    last_date_to_renew: The last date before renewal is required.\",\n",
      "      \"properties\": {\n",
      "        \"renewal_notification_date\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"The date on which the renewal notification is sent.\",\n",
      "          \"title\": \"Renewal Notification Date\"\n",
      "        },\n",
      "        \"last_date_to_renew\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"The last date before renewal is required.\",\n",
      "          \"title\": \"Last Date To Renew\"\n",
      "        }\n",
      "      },\n",
      "      \"required\": [\n",
      "        \"renewal_notification_date\",\n",
      "        \"last_date_to_renew\"\n",
      "      ],\n",
      "      \"title\": \"VehicleInsuranceRenewalDetails\",\n",
      "      \"type\": \"object\"\n",
      "    },\n",
      "    \"VehicleInsuranceVehicleDetails\": {\n",
      "      \"description\": \"A class representing the vehicle details of a vehicle insurance policy.\\n\\nAttributes:\\n    registration_number: The registration number of the vehicle.\\n    make: The make of the vehicle.\\n    model: The model of the vehicle.\\n    year: The year the vehicle was manufactured.\\n    value: The current value of the vehicle.\",\n",
      "      \"properties\": {\n",
      "        \"registration_number\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"The registration number of the vehicle.\",\n",
      "          \"title\": \"Registration Number\"\n",
      "        },\n",
      "        \"make\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"The make of the vehicle.\",\n",
      "          \"title\": \"Make\"\n",
      "        },\n",
      "        \"model\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"The model of the vehicle.\",\n",
      "          \"title\": \"Model\"\n",
      "        },\n",
      "        \"year\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"integer\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"The year the vehicle was manufactured.\",\n",
      "          \"title\": \"Year\"\n",
      "        },\n",
      "        \"value\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"number\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"The current value of the vehicle.\",\n",
      "          \"title\": \"Value\"\n",
      "        }\n",
      "      },\n",
      "      \"required\": [\n",
      "        \"registration_number\",\n",
      "        \"make\",\n",
      "        \"model\",\n",
      "        \"year\",\n",
      "        \"value\"\n",
      "      ],\n",
      "      \"title\": \"VehicleInsuranceVehicleDetails\",\n",
      "      \"type\": \"object\"\n",
      "    }\n",
      "  },\n",
      "  \"description\": \"A class representing a vehicle insurance policy.\\n\\nAttributes:\\n    policy_number: The policy number of the vehicle insurance policy.\\n    cost: The cost details of the vehicle insurance policy.\\n    renewal: The renewal details of the vehicle insurance policy.\\n    effective_from: The effective date from which the vehicle insurance policy is valid.\\n    effective_to: The effective date to which the vehicle insurance policy is valid.\\n    last_date_to_cancel: The last date to cancel the vehicle insurance policy.\\n    policyholder: The person details of the policyholder.\\n    vehicle: The vehicle details of the policy.\\n    accident_excess: The excess costs for accidents.\\n    fire_and_theft_excess: The excess costs for fire and theft.\",\n",
      "  \"properties\": {\n",
      "    \"policy_number\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"type\": \"string\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"The policy number of the vehicle insurance policy.\",\n",
      "      \"title\": \"Policy Number\"\n",
      "    },\n",
      "    \"cost\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"$ref\": \"#/$defs/VehicleInsuranceCostDetails\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"The cost details of the vehicle insurance policy.\"\n",
      "    },\n",
      "    \"renewal\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"$ref\": \"#/$defs/VehicleInsuranceRenewalDetails\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"The renewal details of the vehicle insurance policy.\"\n",
      "    },\n",
      "    \"effective_from\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"type\": \"string\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"The effective date from which the vehicle insurance policy is valid.\",\n",
      "      \"title\": \"Effective From\"\n",
      "    },\n",
      "    \"effective_to\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"type\": \"string\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"The effective date to which the vehicle insurance policy is valid.\",\n",
      "      \"title\": \"Effective To\"\n",
      "    },\n",
      "    \"last_date_to_cancel\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"type\": \"string\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"The last date to cancel the vehicle insurance policy.\",\n",
      "      \"title\": \"Last Date To Cancel\"\n",
      "    },\n",
      "    \"policyholder\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"$ref\": \"#/$defs/VehicleInsurancePersonDetails\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"The person details of the policyholder.\"\n",
      "    },\n",
      "    \"vehicle\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"$ref\": \"#/$defs/VehicleInsuranceVehicleDetails\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"The vehicle details of the policy.\"\n",
      "    },\n",
      "    \"accident_excess\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"$ref\": \"#/$defs/VehicleInsuranceExcessDetails\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"The excess costs for accidents.\"\n",
      "    },\n",
      "    \"fire_and_theft_excess\": {\n",
      "      \"anyOf\": [\n",
      "        {\n",
      "          \"$ref\": \"#/$defs/VehicleInsuranceExcessDetails\"\n",
      "        },\n",
      "        {\n",
      "          \"type\": \"null\"\n",
      "        }\n",
      "      ],\n",
      "      \"description\": \"The excess costs for fire and theft.\"\n",
      "    }\n",
      "  },\n",
      "  \"required\": [\n",
      "    \"policy_number\",\n",
      "    \"cost\",\n",
      "    \"renewal\",\n",
      "    \"effective_from\",\n",
      "    \"effective_to\",\n",
      "    \"last_date_to_cancel\",\n",
      "    \"policyholder\",\n",
      "    \"vehicle\",\n",
      "    \"accident_excess\",\n",
      "    \"fire_and_theft_excess\"\n",
      "  ],\n",
      "  \"title\": \"VehicleInsurancePolicy\",\n",
      "  \"type\": \"object\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "# Highlight the schema sent to the OpenAI model\n",
    "print(json.dumps(VehicleInsurancePolicy.model_json_schema(), indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize the outputs\n",
    "\n",
    "To provide context for the execution of the code, the following code blocks visualize the outputs of the data extraction process.\n",
    "\n",
    "This includes:\n",
    "\n",
    "- The accuracy of the structured data extraction comparing the expected output with the output generated by Azure OpenAI's GPT-4o model.\n",
    "- The confidence score of the structured data extraction based on combining the confidence scores of the Azure AI Document Intelligence layout analysis and the log probability of the output generated by Azure OpenAI's GPT-4o model.\n",
    "- The execution time of the end-to-end process.\n",
    "- The total number of tokens consumed by the GPT-4o model.\n",
    "- The side-by-side comparison of the expected output and the output generated by Azure OpenAI's GPT-4o model.\n",
    "\n",
    "### Understanding Accuracy vs Confidence\n",
    "\n",
    "When using AI to extract structured data, both confidence and accuracy are essential for different but complementary reasons.\n",
    "\n",
    "- **Accuracy** measures how close the AI model's output is to a ground truth or expected output. It reflects how well the model's predictions align with reality.\n",
    "  - Accuracy ensures consistency in the extraction process, which is crucial for downstream tasks using the data.\n",
    "- **Confidence** represents the AI model's internal assessment of how certain it is about its predictions.\n",
    "  - Confidence indicates that the model is certain about its predictions, which can be a useful indicator for human reviewers to step in for manual verification.\n",
    "\n",
    "High accuracy and high confidence are ideal, but in practice, there is often a trade-off between the two. While accuracy cannot always be self-assessed, confidence scores can and should be used to prioritize manual verification of low-confidence predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "NexGen\n",
       "\n",
       "<!-- PageHeader=\"Innovation drives progress\" -->\n",
       "\n",
       "Miss Laura Bennett\n",
       "18 Elm Grove\n",
       "Bristol\n",
       "BS1 5HQ\n",
       "\n",
       "10th May 2024\n",
       "\n",
       "Policy number\n",
       "\n",
       "GB34567890123\n",
       "\n",
       "Manage your account\n",
       "NexGen.com/my-account\n",
       "\n",
       "Hello Miss Bennett,\n",
       "Welcome to your new car insurance\n",
       "\n",
       "Thanks for choosing us. Your new car insurance is effective from 12th May 2024. Please check the summary table below\n",
       "and let us know if there's anything incorrect.\n",
       "\n",
       "\n",
       "<table>\n",
       "<tr>\n",
       "<th>Details</th>\n",
       "<th>Option</th>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>HONDA CIVIC 1.8L RWD (2019) IJ90MNO</td>\n",
       "<td></td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Class Of Use</td>\n",
       "<td>Social, Domestic &amp; Pleasure</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Purchase Date</td>\n",
       "<td>05/02/2020</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Costs of insurance</td>\n",
       "<td>£750.00 per year or £62.50 per month via Direct Debit</td>\n",
       "</tr>\n",
       "</table>\n",
       "\n",
       "\n",
       "Your new documents\n",
       "You'll find your new documents in the app or MyAccount.\n",
       "\n",
       "· Schedule of insurance\n",
       "\n",
       "· Statement of insurance\n",
       "\n",
       "· Certificate of insurance\n",
       "\n",
       "· Cover summary\n",
       "\n",
       "We're here to help\n",
       "\n",
       "If you have any questions about this, there are lots of ways you can get in touch. Please visit the Help Centre on our\n",
       "website for details.\n",
       "\n",
       "Thanks,\n",
       "Jim Taylor\n",
       "Operations Director\n",
       "\n",
       "<!-- PageNumber=\"Insurance Policy | Page 1\" -->\n",
       "<!-- PageBreak -->\n",
       "\n",
       "NexGen\n",
       "\n",
       "Document issued: 10th May\n",
       "2024\n",
       "Reason for issue: New contract\n",
       "Policy number: GB34567890123\n",
       "\n",
       "\n",
       "# Your statement of insurance\n",
       "\n",
       "Please read and check the information shown in this document carefully. It represents the answers given during the initial quotation\n",
       "process, and any changes that have been made to the policy since then. It forms part of your contract for motor insurance.\n",
       "\n",
       "If any of the details are incorrect or have changed, please contact us immediately.\n",
       "\n",
       "You can do this:\n",
       "\n",
       "· Through MyAccount\n",
       "\n",
       "· By calling 0123 321 1234\n",
       "\n",
       "We'll tell you if this changes your premium, terms or insurer.\n",
       "\n",
       "Under the Consumer Insurance (Disclosure and Representations) Act 2012, you have a duty to take reasonable care to answer all\n",
       "questions fully and accurately. If you volunteer information over and above that requested, you must do so honestly and carefully. If\n",
       "you don't answer all questions fully and accurately, it could invalidate your insurance cover and result in all, or part of a claim not\n",
       "being paid.\n",
       "\n",
       "\n",
       "<table>\n",
       "<tr>\n",
       "<td colspan=\"2\">Policyholder details</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Policyholder</td>\n",
       "<td>Miss Laura Bennett</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Email address</td>\n",
       "<td>LBennett@me.com</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Have any drivers on this policy ever had an insurance policy declined, cancelled, voided or had any special terms imposed?</td>\n",
       "<td>No</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Total vehicles in household</td>\n",
       "<td>1</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Total number of drivers in household</td>\n",
       "<td>1</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Your address</td>\n",
       "<td>18 Elm Grove, Bristol, BS1 5HQ</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Years at address</td>\n",
       "<td>3</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Homeowner</td>\n",
       "<td>No</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Children under 16</td>\n",
       "<td>No</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Vehicle key at address overnight</td>\n",
       "<td>Yes</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Kept overnight</td>\n",
       "<td>On street</td>\n",
       "</tr>\n",
       "</table>\n",
       "\n",
       "\n",
       "<!-- PageNumber=\"Insurance Policy | Page 2\" -->\n",
       "<!-- PageBreak -->\n",
       "\n",
       "\n",
       "<table>\n",
       "<tr>\n",
       "<td colspan=\"2\">Insured drivers</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Name</td>\n",
       "<td>Laura Bennett</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Date of birth</td>\n",
       "<td>08/09/1990</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Marital status</td>\n",
       "<td>Single</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Permanent resident in the UK since</td>\n",
       "<td>15/03/1995</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Employment status</td>\n",
       "<td>Full-time</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Primary occupation</td>\n",
       "<td>Marketing Manager</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Industry</td>\n",
       "<td>Advertising &amp; PR</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Driving licence number</td>\n",
       "<td>BENNEL008099JJ9IT</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Medical conditions reportable to the DVLA/DVANI</td>\n",
       "<td>No</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Use of other vehicle</td>\n",
       "<td>No</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Unspent non-motoring convictions</td>\n",
       "<td>No</td>\n",
       "</tr>\n",
       "</table>\n",
       "\n",
       "\n",
       "<table>\n",
       "<tr>\n",
       "<td colspan=\"2\">Your policy details</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Cover type</td>\n",
       "<td>Comprehensive</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Class of use</td>\n",
       "<td>Social, Domestic &amp; Pleasure</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Your no claims discount</td>\n",
       "<td>You told us you have 3 or more years of no claims discount, but we haven't confirmed this with the previous insurer. If you had a non-recoverable claim in this policy period, we'll adjust your no claims discount at renewal.</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>How no claims discount was earned</td>\n",
       "<td>Private car discount</td>\n",
       "</tr>\n",
       "</table>\n",
       "\n",
       "\n",
       "## Incident history\n",
       "\n",
       "For all drivers named in this policy, we need to know of any incident, claim or damage involving any motor vehicle in the past five\n",
       "years, including windscreen damage. This applies whether a claim was made, and regardless of blame.\n",
       "\n",
       "None disclosed\n",
       "\n",
       "\n",
       "## Conviction history\n",
       "\n",
       "For all drivers named in this policy, we need to know of any driving related convictions, endorsements, fixed penalties,\n",
       "disqualifications or bans in the past five years.\n",
       "\n",
       "\n",
       "<table>\n",
       "<tr>\n",
       "<th>Driver</th>\n",
       "<th>Date</th>\n",
       "<th>Type</th>\n",
       "<th>Licence points</th>\n",
       "<th>Disqualification</th>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Laura Bennett</td>\n",
       "<td>-</td>\n",
       "<td>-</td>\n",
       "<td>-</td>\n",
       "<td>-</td>\n",
       "</tr>\n",
       "</table>\n",
       "\n",
       "\n",
       "<!-- PageFooter=\"Insurance Policy | Page 3\" -->\n",
       "<!-- PageBreak -->\n",
       "\n",
       "\n",
       "<table>\n",
       "<tr>\n",
       "<td>Insured vehicle details</td>\n",
       "<td></td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Registration number</td>\n",
       "<td>IJ90MNO</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Annual mileage</td>\n",
       "<td>9,000</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Make &amp; model</td>\n",
       "<td>Honda Civic</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Body type</td>\n",
       "<td>HATCHBACK</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Year registered</td>\n",
       "<td>2019</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Transmission</td>\n",
       "<td>Manual</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Engine size</td>\n",
       "<td>1.8L</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Number of seats</td>\n",
       "<td>5</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Number of doors</td>\n",
       "<td>5</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Date purchased</td>\n",
       "<td>02/2020</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Imported</td>\n",
       "<td>No</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Left hand drive</td>\n",
       "<td>No</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Declared value</td>\n",
       "<td>£11,000.00</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Registered keeper &amp; legal owner</td>\n",
       "<td>Yes</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Security devices or immobiliser</td>\n",
       "<td>Factory fitted Alarm + Immobiliser</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Tracker fitted</td>\n",
       "<td>No</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Modifications</td>\n",
       "<td>No</td>\n",
       "</tr>\n",
       "</table>\n",
       "\n",
       "\n",
       "What's a modification?\n",
       "\n",
       "A modification is any alteration to your car from the manufacturer's standard specification. This includes, but isn't limited to:\n",
       "\n",
       "· Changes to the bodywork, such as spoilers or body kit\n",
       "\n",
       "· Changes to suspension or brakes\n",
       "\n",
       "· Alloy wheels\n",
       "\n",
       "· Audio/entertainment system\n",
       "\n",
       "· Changes affecting performance, such as to the engine management system or exhaust system.\n",
       "\n",
       "You must also update your Policy if you intend to alter or modify your Car/s from the manufacturer's standard specification\n",
       "If you don't tell us about a modification, we may cancel your Policy from its start date, apply additional premium or add new terms to\n",
       "your Policy. If you make a claim your insurer may reject the claim or only provide partial payment for it.\n",
       "We may need to check your vehicle for modifications and accessories at the time of your claim.\n",
       "\n",
       "Please note: optional extras fitted at the point of manufacture aren't classed as modifications.\n",
       "\n",
       "<!-- PageNumber=\"Insurance Policy | Page 4\" -->\n",
       "<!-- PageBreak -->\n",
       "\n",
       "NexGen\n",
       "\n",
       "Cover effective dates:\n",
       "\n",
       "Reason for issue:\n",
       "\n",
       "Policy number:\n",
       "\n",
       "Insurer:\n",
       "\n",
       "All mentions to \"you\" or \"your\" refer to:\n",
       "\n",
       "14.12 on 12th May 2024 to\n",
       "11th May 2025\n",
       "\n",
       "New contract\n",
       "GB34567890123\n",
       "NexGen\n",
       "Miss Laura Bennett\n",
       "\n",
       "\n",
       "### Schedule of insurance\n",
       "\n",
       "You have comprehensive insurance. This means sections 1 to 8 of the policy apply.\n",
       "\n",
       "\n",
       "#### Excesses\n",
       "\n",
       "Your excess is the part of a claim you must pay, even if the damage or loss isn't your fault. It also applies if a named driver was in\n",
       "charge of the car.\n",
       "\n",
       "\n",
       "<table>\n",
       "<tr>\n",
       "<th>Accidental damage</th>\n",
       "<th>Compulsory</th>\n",
       "<th>Voluntary</th>\n",
       "<th>Total</th>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Laura Bennett</td>\n",
       "<td>£300</td>\n",
       "<td>£200</td>\n",
       "<td>£500</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>Fire and theft</td>\n",
       "<td>Compulsory</td>\n",
       "<td>Voluntary</td>\n",
       "<td>Total</td>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>All drivers</td>\n",
       "<td>£250</td>\n",
       "<td>£150</td>\n",
       "<td>£400</td>\n",
       "</tr>\n",
       "</table>\n",
       "\n",
       "\n",
       "<table>\n",
       "<tr>\n",
       "<th>Windscreen/glass</th>\n",
       "<th>Repair</th>\n",
       "<th>Replacement</th>\n",
       "</tr>\n",
       "<tr>\n",
       "<td>All drivers</td>\n",
       "<td>£20</td>\n",
       "<td>£110</td>\n",
       "</tr>\n",
       "</table>\n",
       "\n",
       "\n",
       "These excesses are based on you using one of our approved repairers.\n",
       "You can use your own repairer for an accidental damage or fire and theft claim but you'll still have to pay the excess shown above,\n",
       "plus an additional excess of £250.\n",
       "\n",
       "<!-- PageNumber=\"Insurance Policy | Page 5\" -->\n",
       "<!-- PageBreak -->\n",
       "\n",
       "\n",
       "# Certificate of motor insurance\n",
       "\n",
       "1\\. Registration mark of vehicle: IJ90MNO\n",
       "\n",
       "Courtesy (or replacement) vehicles (\"Replacement Vehicle\") are covered on your policy when supplied to you or any named\n",
       "driver(s) (set out in number 5 of this certificate) by your Insurer's nominated suppliers either:\n",
       "\n",
       "a) whilst your vehicle is being repaired;\n",
       "b) at your Insurer's request.\n",
       "\n",
       "An exception to this is if the nominated supplier of the courtesy or replacement vehicle provides their own insurance cover\n",
       "to you. This will be set out in your agreement with them.\n",
       "\n",
       "If your Replacement Vehicle is covered by your policy, but your policy expiries or is cancelled, we'll automatically extend\n",
       "this insurance, for the Replacement Vehicle only, at no cost to you, until you return it to the supplier on the date agreed\n",
       "(\"Extended Cover\"). During the Extended Cover (i) you'll only be insured to drive the Replacement Vehicle and no other\n",
       "vehicles (including those set out in number 5 of this certificate); and (ii) if there's a claim involving the Replacement Vehicle,\n",
       "you'll need to pay any excess set out in your policy.\n",
       "\n",
       "2\\. Name of policyholder: Miss Laura Bennett\n",
       "\n",
       "3\\. Effective date of the commencement of insurance for the purpose of the relevant law\n",
       "Your insurance starts at\n",
       "14:12 on 12th May 2024\n",
       "\n",
       "4\\. Date of expiry of insurance\n",
       "Your insurance ends at\n",
       "23:59 on 11th May 2025\n",
       "\n",
       "5\\. Persons or classes of persons entitled to drive:\n",
       "\n",
       "· Miss Laura Bennett\n",
       "\n",
       "Cover is provided only if the person driving:\n",
       "\n",
       "· Holds a valid licence\n",
       "\n",
       "· Isn't disqualified from holding or obtaining a valid licence.\n",
       "\n",
       "Driving other cars:\n",
       "\n",
       "As the policyholder, you're covered to drive someone else's car if:\n",
       "\n",
       "· You have the owner's permission\n",
       "\n",
       "· It has valid insurance\n",
       "\n",
       "· It's not owned by you or your partner or business partner/employer\n",
       "\n",
       "· It's not under a hire purchase, self-drive hire or lease agreement of any kind.\n",
       "\n",
       "Named drivers do not have this cover and other conditions apply - see your policy details.\n",
       "\n",
       "\n",
       "## 6. Limitations as to use\n",
       "\n",
       "What's covered:\n",
       "\n",
       "· Social, domestic and pleasure use\n",
       "\n",
       "· Travelling to and from one or more permanent place(s) of business or study\n",
       "\n",
       "· Use for your private business\n",
       "\n",
       "What's not covered:\n",
       "\n",
       "· Commercial travel, soliciting for orders, any use connected with the motor trade and hire or reward\n",
       "\n",
       "· For racing on a public highway\n",
       "\n",
       "· Use at any event where your vehicle is driven:\n",
       "\n",
       "o On a motor-racing track, including de-restricted toll roads such as the Nürburgring\n",
       "☐\n",
       "\n",
       "o On a prepared course\n",
       "☐\n",
       "\n",
       "o At any off-road event, such as a 4x4 event\n",
       "☐\n",
       "\n",
       "o At an airfield\n",
       "☐\n",
       "\n",
       "<!-- PageNumber=\"Insurance Policy | Page 6\" -->\n",
       "<!-- PageBreak -->\n",
       "\n",
       "NexGen\n",
       "\n",
       "<!-- PageHeader=\"Innovation drives progress\" -->\n",
       "\n",
       "\n",
       "# LEGAL STATEMENTS\n",
       "\n",
       "\n",
       "## 1. POLICYHOLDER OF A CAR POLICY\n",
       "\n",
       "A policyholder of a car policy enters two separate contracts when taking out a policy through us.\n",
       "\n",
       "(i)\n",
       "The first contract is between the policyholder and the insurer:\n",
       "\n",
       "a. The insurer's name is shown on the policyholder's current certificate of motor insurance. This can be\n",
       "found on the policyholder's certificate of motor insurance in the app and MyAccount\n",
       "\n",
       "b. The insurer is the company providing the policyholder's motor insurance\n",
       "\n",
       "c. It is the policyholder's responsibility to be aware of these terms and conditions and to make sure that\n",
       "named drivers are also aware of them\n",
       "\n",
       "d. The policyholder is the only individual able to cancel the car policy (as set out in more detail in\n",
       "'Cancellations')\n",
       "\n",
       "e. The policyholder can make claims under the car policy on their own behalf and on behalf of any named\n",
       "drivers in accordance with the terms of this policy\n",
       "\n",
       "f.\n",
       "This contract does not give rise to any rights under the Contracts (Rights of Third Parties) Act 1999 to\n",
       "enforce any term of the contract\n",
       "\n",
       "g. This contract will be governed by and interpreted in accordance with English law\n",
       "\n",
       "h. The insurer will communicate in English throughout the course of this contract.\n",
       "\n",
       "(ii)\n",
       "The second contract is between the policyholder and NexGen (\"we/us/our\"):\n",
       "\n",
       "a. We are an insurance broker, and we arrange and administer the policyholder's car policy on behalf of the\n",
       "insurer\n",
       "\n",
       "b. We are the policyholder's first point of contact\n",
       "\n",
       "c. This contract does not give rise to any rights under the Contracts (Rights of Third Parties) Act 1999 to\n",
       "enforce any term of the contract\n",
       "\n",
       "d. This contract will be governed by and interpreted in accordance with English law\n",
       "\n",
       "e. We will communicate in English throughout the course of this contract.\n",
       "\n",
       "The policyholder of a car policy is covered by the insurer for the period of cover when the policyholder:\n",
       "\n",
       "· Agrees to the terms and conditions offered; and\n",
       "\n",
       "· Pays, or has offered to pay, the costs of insurance for the car policy.\n",
       "\n",
       "The policyholder is required to take reasonable care not to make a misrepresentation when providing information to the\n",
       "insurer and/or us.\n",
       "\n",
       "\n",
       "## 2. PAYMENT\n",
       "\n",
       "The costs of insurance may be paid:\n",
       "\n",
       "(i)\n",
       "In full by debit or credit card; or\n",
       "\n",
       "(ii)\n",
       "By instalments as agreed in a credit agreement (subject to eligibility)\n",
       "\n",
       "If the costs of insurance are paid in one single payment, that payment may be made by a policyholder or a third party.\n",
       "\n",
       "A credit agreement can only be with a policyholder. The instalments may be paid by anyone.\n",
       "\n",
       "\n",
       "## 3. LEGAL OBLIGATIONS\n",
       "\n",
       "It's an offence under the road traffic act to make a false statement or to withhold material information to get motor\n",
       "insurance.\n",
       "\n",
       "Under the Consumer Insurance (Disclosure and Representation) Act 2012, when you apply for insurance, you have a duty\n",
       "to take reasonable care to answer all questions as fully and as accurately as possible.\n",
       "\n",
       "<!-- PageFooter=\"Insurance Policy | Page 7\" -->\n",
       "<!-- PageBreak -->\n",
       "\n",
       "If you do not take reasonable care to answer all questions fully and accurately or if you deliberately make a false statement\n",
       "or withhold material information, there could be serious consequences. In certain circumstances your insurer, or we acting\n",
       "on your insurer's behalf, have the right to treat your policy as if it never existed, without giving you notice or refunding your\n",
       "premium. This will only apply:\n",
       "\n",
       ". If you make a careless misrepresentation and your insurer would not have offered you insurance had it known the\n",
       "true facts; or\n",
       "\n",
       "· if you make a misrepresentation which is deliberate and/or reckless.\n",
       "\n",
       "If the policy is treated as if it never existed, any claims made before or during this time will be declined and not paid and\n",
       "you may have to make a payment to anybody involved in an incident, or to your insurer if it is obliged by law to make a\n",
       "payment to anybody involved in an incident.\n",
       "\n",
       "If the details on your documents are wrong, please contact our customer services team as soon as possible. Their details\n",
       "are at the front of this document.\n",
       "\n",
       "If you want to make a change to your policy, please contact our customer service team. Customers with online policies\n",
       "should tell us about the changes on the app or MyAccount website.\n",
       "\n",
       "\n",
       "### SECTION 1\n",
       "\n",
       "\n",
       "#### DAMAGE TO YOUR CAR/S (EXCEPT THAT CAUSED BY FIRE OR THEFT)\n",
       "\n",
       "You are covered for accident, vandalism and malicious damage\n",
       "\n",
       "If your car/s is damaged or lost because of an accident, vandalism or malicious damage there are four ways your policy\n",
       "can help you get back on the road again. Your insurer will do one of the following:\n",
       "\n",
       "· Pay for any necessary repairs (including damage to charging cables and batteries for electric and hybrid vehicles)\n",
       "\n",
       ". Replace your car\n",
       "\n",
       "· Repair the damage\n",
       "\n",
       "· Pay the market value of your car immediately before the loss.\n",
       "\n",
       "Accessories are also covered while they are in, or on, your car or in your private garage.\n",
       "\n",
       "\n",
       "#### SECTION 2\n",
       "\n",
       "\n",
       "##### DAMAGE OR LOSS CAUSED BY FIRE OR THEFT\n",
       "\n",
       "You are covered for fire, theft, attempted theft or lightning damage to your car/s\n",
       "\n",
       "If your car is damaged or lost because of theft, attempted theft, fire or lightning there are four ways your policy can help\n",
       "you get back on the road again. Your insurer will do one of the following:\n",
       "\n",
       "· Pay for any necessary repairs\n",
       "\n",
       "· Replace your car\n",
       "\n",
       "· Repair the damage\n",
       "\n",
       "· Pay the market value of your car immediately before the loss.\n",
       "\n",
       "Accessories are also covered while they are in, or on, your car or in your private garage.\n",
       "\n",
       "WHAT ISN'T COVERED UNDER SECTIONS 1 AND 2\n",
       "\n",
       "You are not covered for:\n",
       "\n",
       "· Mechanical, electrical, electronic or computer failures (including failures caused by a computer virus or cyber-\n",
       "attack) or breakdowns or breakages\n",
       "\n",
       "· Loss, damage or corruption of data caused by failure to install and/or accept vehicle manufacturer software\n",
       "updates\n",
       "\n",
       ". The excesses shown on your schedule of insurance - you will have to pay these if you make a claim\n",
       "\n",
       ". Loss of use of your car (if you are out of pocket because you can't use your car, including the cost of hiring\n",
       "another vehicle)\n",
       "\n",
       "<!-- PageNumber=\"Insurance Policy | Page 8\" -->\n",
       "<!-- PageBreak -->\n",
       "\n",
       "· Wear and tear, deterioration, depreciation, or any loss or damage that happens gradually\n",
       "\n",
       "· Failures, breakdowns or breakage of mechanical, electrical, electronic or computer equipment\n",
       "\n",
       "· Damage to tyres caused by braking, punctures, cuts or bursts\n",
       "\n",
       "· Loss of value following repair\n",
       "\n",
       "· Theft of or damage, if the car keys were left in or on the car or if the car is left unattended with the engine running\n",
       "\n",
       "· Replacement of locks, if the car keys were left in or on the car or if the car is left unattended with the engine\n",
       "running.\n",
       "\n",
       "· Loss or damage if someone claiming to be a buyer or agent takes possession of your car deceitfully\n",
       "\n",
       "· Your car being repossessed by its rightful owner or having to pay compensation to the owner\n",
       "\n",
       "· Any amount greater than the manufacturer's last list price for replacing any part or accessories lost or damaged\n",
       "\n",
       "· Repairs or replacements unrelated to your claim that improve the condition of your car\n",
       "\n",
       "· Loss or damage because of your car being driven or used without your permission by a member of your family or\n",
       "household unless the incident is reported to the police, and you send us the crime reference number\n",
       "\n",
       "· Loss or damage caused by an inappropriate type or grade of fuel being used\n",
       "\n",
       "· Loss or damage because of malicious damage or vandalism, where the police refuse to issue a crime reference\n",
       "number. Please note that having a crime reference number doesn't guarantee we will settle a claim\n",
       "\n",
       "· Any additional damage resulting from your car being moved by anyone insured under your policy after an\n",
       "accident, fire or theft\n",
       "\n",
       "· Loss or damage resulting from the legal confiscation of your car by HM Revenue and Customs, the police, a local\n",
       "authority or any other government authority.\n",
       "\n",
       "\n",
       "## HOW YOUR CLAIMS ARE SETTLED FOR SECTIONS 1 AND 2\n",
       "\n",
       "How the insurer will deal with your claim for accident, vandalism, malicious damage, theft, attempted theft, fire or\n",
       "lightning\n",
       "\n",
       "If your car is damaged, your insurer will arrange the transportation of your car to the nearest suitable nominated repairer\n",
       "or a place of storage. Where appropriate they will also return it after repair to the address shown on your schedule.\n",
       "Alternatively, they will cover the reasonable cost of doing this.\n",
       "\n",
       "If your car is stolen and not recovered then as soon as a total loss settlement is agreed and paid by your insurer, your\n",
       "insurer will thereby take ownership of your car (where it is entitled to do so) and should your car be subsequently\n",
       "recovered, any salvage shall become your insurer's property.\n",
       "\n",
       "Replacement of locks and stolen keys\n",
       "\n",
       "Provided it can be established to your insurer's reasonable satisfaction that the identity or garaging address of your car is\n",
       "known to any person who may have stolen or found your keys and the value of your claim does not exceed the market\n",
       "value of your car, your insurer will pay up to a maximum of £500 after deducting any excess, towards the cost of replacing:\n",
       "\n",
       ". The door locks and/or boot lock\n",
       "\n",
       "· The ignition/steering lock\n",
       "\n",
       ". The lock transmitter and central locking interface.\n",
       "\n",
       "You are not covered for stolen keys if they were left in or on your car while it was unattended or unoccupied.\n",
       "\n",
       "\n",
       "### Audio visual equipment\n",
       "\n",
       "Your insurer will cover the loss or damage to in-car audio, television, DVD, phone, games-console, dashboard camera,\n",
       "electronic navigation or radar detection equipment permanently fitted to your car. This cover is unlimited if the equipment\n",
       "was fitted by the manufacturer and was part of the specification of your car when first registered. If the equipment wasn't\n",
       "originally part of your car, the most your insurer will pay is £300. Your insurer will settle a claim for audio visual equipment\n",
       "by repairing it, replacing it with a similar piece of equipment or providing a cash payment.\n",
       "\n",
       "\n",
       "### Costs you must pay\n",
       "\n",
       "The total excess that applies to your claim. Your compulsory and voluntary excesses are based on you using your\n",
       "insurer's nominated repairer. If you use a different repairer, there will be an additional excess to pay as shown in your\n",
       "schedule of insurance.\n",
       "\n",
       "<!-- PageNumber=\"Insurance Policy | Page 9\" -->\n",
       "<!-- PageBreak -->\n",
       "\n",
       "\n",
       "### Costs you may have to pay\n",
       "\n",
       ". If your insurer accepts your claim, and finds your details or circumstances have changed since you took your\n",
       "policy out, you may have to pay any additional costs and associated fees (see General Conditions for more\n",
       "details)\n",
       "\n",
       ". If your claim is settled on a total loss basis and you pay by instalments under a loan arrangement with us, we may\n",
       "take all outstanding payments from the claims settlement or ask you to pay the outstanding amount (see total loss\n",
       "section for more details)\n",
       "\n",
       ". If your insurer doesn't accept your claim, you may have to pay any costs already incurred. These may include (but\n",
       "are not limited to) engineers' fees, vehicle recovery and storage charges.\n",
       "\n",
       "Total loss - if your car can't be repaired\n",
       "\n",
       "If your car can't be repaired or your insurer deems your car to be unsafe or the cost of repair to be uneconomical, your\n",
       "car will be declared a total loss (sometimes called a 'write-off'). If your car/s are a total loss, your insurer may put it in\n",
       "storage until your claim is settled. As soon as a total loss settlement is agreed and paid by your insurer, your insurer will\n",
       "thereby take possession and ownership of your car (where it is entitled to do so) and any salvage shall become your\n",
       "insurer's property.\n",
       "\n",
       "If you are paying for your policy by instalments under a loan arrangement with us and your insurer settles a total loss claim\n",
       "under these sections, your consumer credit agreement with us may entitle us to do one of the following:\n",
       "\n",
       ". Take the outstanding amount due for your consumer credit agreement out of the claims settlement\n",
       "\n",
       ". Require you to pay the outstanding amount due for the car in question.\n",
       "\n",
       "If your car is declared a total loss, and you have already paid the premium in full, no refund will be made for the car in\n",
       "question, even if the cover for the car is later cancelled. This may not apply if your insurer is able to recover all losses from\n",
       "a third party. In this case insurers may sometimes refund the premium paid and, if they do, we will pass that refund on to\n",
       "you.\n",
       "\n",
       "If your claim is settled on a total loss basis and you don't replace your car within 30 days of being issued the settlement\n",
       "payment, we will cancel your policy. The provisions above relating to loan agreements and refunds will still apply.\n",
       "\n",
       "\n",
       "## SECTION 3\n",
       "\n",
       "\n",
       "### LEGAL RESPONSIBILITY TO OTHERS (THIRD PARTIES)\n",
       "\n",
       "What's covered\n",
       "\n",
       "Your insurer will pay all sums you are liable for in respect of (1) death or injury to other people or (2) (up to £20,000,000\n",
       "plus up to £5,000,000 for costs and expenses) damage to someone else's property caused by or arising from the use of\n",
       "your car/s (or any other vehicle your policy covers you to drive - see your certificate of motor insurance).\n",
       "\n",
       "This cover also applies to accidents involving a trailer, caravan or broken-down vehicle being towed (as long as you hold\n",
       "the correct entitlement on your driving licence to do so).\n",
       "\n",
       "Legal costs\n",
       "\n",
       "Following a claim covered by this policy and if your insurer agrees it's in their interest to do so, which is entirely their\n",
       "decision, they will pay reasonable legal costs and expenses for:\n",
       "\n",
       "· Solicitors' fees for representing anyone insured at a coroner's inquest, fatal accident inquiry or court\n",
       "\n",
       "· Reasonable legal services, which they will arrange, to defend a charge of manslaughter or causing death by\n",
       "dangerous or reckless driving\n",
       "\n",
       ". Any other legal costs and expenses if agreed in writing beforehand.\n",
       "\n",
       "You must get your insurer's consent in writing before incurring these sorts of fees and costs.\n",
       "\n",
       "What's not covered under section 3\n",
       "\n",
       ". Anyone who has any other insurance covering the same liability\n",
       "\n",
       "<!-- PageFooter=\"Insurance Policy | Page 10\" -->\n",
       "<!-- PageBreak -->\n",
       "\n",
       "· Liability for the death of or injury to anyone while they are working with, or for, the driver of the car except to the\n",
       "extent that the driver is required to be covered for such a liability by the road traffic act\n",
       "\n",
       "· Any damage to personal property owned by the person driving your car at the time of the incident\n",
       "\n",
       ". Loss of, or damage to, any trailer, caravan or vehicle (or their contents) while being towed by or attached to any\n",
       "vehicle covered by this section\n",
       "\n",
       ". Loss or damage to property of more than £20,000,000 for any one incident or series of incidents and costs and\n",
       "expenses over £5,000,000.\n",
       "\n",
       "\n",
       "## SECTION 4\n",
       "\n",
       "\n",
       "### PERSONAL ACCIDENT\n",
       "\n",
       "Your insurer will pay £5,000 if you or your partner are accidentally injured while travelling in or while getting into, or\n",
       "getting out of, your car/s and within 90 days if this injury is the sole cause of:\n",
       "\n",
       "· Death\n",
       "\n",
       "· Permanent loss of sight in one or both eyes\n",
       "\n",
       ". Total physical loss of a limb at or above the ankle or wrist. Your insurer will pay the injured person or their legal\n",
       "representative.\n",
       "\n",
       "What's not covered under section 4\n",
       "\n",
       "· Death or injury resulting from suicide or attempted suicide\n",
       "\n",
       "· Death or injury to anyone not wearing a seat belt when required by law\n",
       "\n",
       "· Death or injury because the driver was unfit to drive because of alcohol, drugs or other substances, whether\n",
       "prescribed or otherwise\n",
       "\n",
       "· Any disablement, whether temporary, permanent, partial or total, except those listed above\n",
       "\n",
       "· Injury caused by a pre-existing disease or physical weakness\n",
       "\n",
       "· Anything excluded by the general exceptions listed later in this document.\n",
       "\n",
       "\n",
       "# SECTION 5\n",
       "\n",
       "\n",
       "## MEDICAL EXPENSES\n",
       "\n",
       "If you, or anyone in your car, is injured in an accident, your insurer will pay medical expenses of up to £500 for each\n",
       "injured person.\n",
       "\n",
       "\n",
       "# SECTION 6\n",
       "\n",
       "\n",
       "## PERSONAL BELONGINGS\n",
       "\n",
       "What's covered\n",
       "\n",
       "If you have comprehensive cover and you are making a claim under sections 1 or 2 of this policy, your insurer will pay up\n",
       "to £300 for any one claim for personal belongings in your car, to you or the owner of the items.\n",
       "\n",
       "What's not covered under section 6\n",
       "\n",
       "· Money, stamps, jewellery, watches, tickets, credit or debit cards, vouchers, documents or securities (such as\n",
       "share and premium bond certificates)\n",
       "\n",
       "· Laptops, mobile phones, tablet computers or electronic navigational equipment\n",
       "\n",
       "· Goods, samples or tools carried in connection with any trade or business\n",
       "\n",
       "· Property insured under any other insurance policy\n",
       "\n",
       "· Theft or attempted theft of personal belongings not kept out of sight in the glove compartment or locked boot\n",
       "\n",
       "· Theft or attempted theft if the car was left unlocked while unoccupied or unattended\n",
       "\n",
       "· Anything excluded by the general exceptions listed later in this document.\n",
       "\n",
       "Your insurer may require documentary evidence to confirm your claim and/or may ask to see the damaged item.\n",
       "\n",
       "<!-- PageFooter=\"Insurance Policy | Page 11\" -->\n",
       "<!-- PageBreak -->\n",
       "\n",
       "\n",
       "# SECTION 7\n",
       "\n",
       "\n",
       "## WINDSCREEN DAMAGE\n",
       "\n",
       "\n",
       "### What's covered\n",
       "\n",
       "If you have comprehensive cover your insurer will pay to replace, or repair broken glass in the windscreen or windows of\n",
       "your car and repair any scratching to the bodywork caused by the broken glass.\n",
       "\n",
       "Making a claim under this section won't affect your no claims discount, if you are not also claiming for any other loss or\n",
       "damage to your car.\n",
       "\n",
       "Your insurer's nominated repairer may use parts or accessories that aren't made or supplied by your car's manufacturer\n",
       "but are of an equivalent type and quality to those being replaced.\n",
       "\n",
       "What's not covered under section 7\n",
       "\n",
       "· Any other glass forming part of your car including sunroofs, panoramic roofs or panoramic sunroofs, where the\n",
       "roof glass is a separate unit to the windscreen glass\n",
       "\n",
       "· Any windscreens or windows not made of glass\n",
       "\n",
       "· Replacement of the hood/roof structure of a convertible or cabriolet car\n",
       "\n",
       ". A repair or replacement cost that's more than the market value of your car at the time of loss (less any excess).\n",
       "\n",
       "Additional charges or limited cover may apply if the repair isn't carried out by your insurer's nominated repairer.\n",
       "\n",
       "You must pay an excess for windscreen, windows and glass repairs or replacement - see your schedule of insurance and\n",
       "certificate of motor insurance for more details.\n",
       "\n",
       "Any repairs to a windscreen will only be guaranteed for 30 days.\n",
       "\n",
       "\n",
       "## SECTION 8\n",
       "\n",
       "\n",
       "### UNINSURED DRIVER PROMISE\n",
       "\n",
       "Your insurer promises that if you are involved in an accident that isn't your fault and the driver of the vehicle that hits you\n",
       "doesn't have motor insurance:\n",
       "\n",
       ". You won't lose your no claims discount\n",
       "\n",
       "· You won't have to pay any excess/es.\n",
       "\n",
       "To benefit from this promise, you must send us the make, model and registration number of the vehicle that caused\n",
       "damage to your car and, when possible, tell us the other driver's name and address.\n",
       "\n",
       "When you make a claim, you may initially have to pay your excess/es. If investigations are still taking place when your\n",
       "renewal is due, you may lose your no claims discount temporarily, as explained above. Once your insurer has confirmed\n",
       "the accident was the fault of an identified uninsured driver, your insurer will refund your excess, restore your no claims\n",
       "discount and refund any extra premium you have paid.\n",
       "\n",
       "\n",
       "## PAYMENTS\n",
       "\n",
       "The total price of your insurance is shown in your documents and includes insurance premium tax. For legal purposes, we\n",
       "must tell you that in future other taxes or costs may apply that are not paid for or imposed by us. However, at present, we\n",
       "are not aware of any other taxes or costs payable.\n",
       "\n",
       "If the policy is paid annually, the costs of insurance will be due within 10 days after the effective date of the\n",
       "commencement for the policy. After this date, the policy will be cancelled.\n",
       "\n",
       "<!-- PageFooter=\"Insurance Policy | Page 12\" -->\n",
       "<!-- PageBreak -->\n",
       "\n",
       "If the policy is paid monthly, a monthly direct debit for continuous payment is required within 10 days after the effective\n",
       "date of the commencement for the policy. We will use the direct debit details to continue to collect the costs of insurance\n",
       "as agreed previously.\n",
       "\n",
       "You must contact us as soon as possible to pay the costs of insurance.\n",
       "\n",
       "\n",
       "# RENEWING YOUR POLICY\n",
       "\n",
       "This section contains important information about renewing your policy.\n",
       "\n",
       "\n",
       "## RENEWAL PERIOD\n",
       "\n",
       "21 days before your policy ends, we will send a renewal notice that the insurance cover is due to expire. In most cases,\n",
       "this notice will include an offer to renew the insurance for another year. The renewal notice will include important facts\n",
       "about the policy, any changes to the policy terms and a price.\n",
       "\n",
       "In a small number of cases, the insurer may not renew the policy.\n",
       "\n",
       "We automatically renew most policies. This means that, unless you tell us otherwise, your new insurance cover will start\n",
       "on your renewal date. If we intend to automatically renew the policy, we will tell you this in your renewal notice.\n",
       "\n",
       "You must contact us at least 7 days before the insurance policy expires to (i) cancel the renewal or (ii) negotiate the\n",
       "renewal payment.\n",
       "\n",
       "\n",
       "# CANCELLATIONS\n",
       "\n",
       "This section contains important information about your rights, plus our and your insurer's rights of cancellation.\n",
       "\n",
       "\n",
       "## YOUR RIGHTS TO CANCEL THIS POLICY\n",
       "\n",
       "You have the right to cancel this policy within the first 14 days without incurring a penalty and without giving a reason. This\n",
       "is known as \"the 14-day cooling off period\" and starts on (i) the day this policy is entered into or (ii) the day on which you\n",
       "receive these terms and conditions, whichever is later.\n",
       "\n",
       "\n",
       "## POLICYHOLDER OF A CAR POLICY\n",
       "\n",
       "Cancellation can only be authorised by the policyholder and it's your responsibility to notify any other drivers named on\n",
       "the policy that they are no longer insured.\n",
       "\n",
       "\n",
       "## WHAT HAPPENS WHEN THE POLICY IS CANCELLED\n",
       "\n",
       "If the policy is cancelled, any fees, such as the arrangement fee, incurred before cancellation are non-refundable, as is the\n",
       "cost of your insurance for the number of days you have been insured.\n",
       "\n",
       "If the policy is cancelled, your insurer won't refund a premium for any car where a non-recoverable claim has been made\n",
       "on the car or any replacement car during the period of cover. Where instalments are being paid under a credit agreement,\n",
       "the balance of the annual premium and the cancellation fee (if it's 14 days or more since your policy started) will need to\n",
       "be paid.\n",
       "\n",
       "Following the cancellation, we will calculate the refund as follows: If the cancellation of the policy is before the cover starts:\n",
       "\n",
       ". The person who paid the costs of insurance will be entitled to a full refund of the insurer premium minus our non-\n",
       "refundable fees as shown on your cover summary document\n",
       "\n",
       "If the cancellation of the policy is within the 14-day cooling off period:\n",
       "\n",
       ". The person who paid the costs of insurance will be entitled to a premium refund on a pro-rata basis for the period\n",
       "of cover that hasn't been used minus our non-refundable fees as shown on your cover summary document.\n",
       "\n",
       "<!-- PageFooter=\"Insurance Policy | Page 13\" -->\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Displays the output of the Azure AI Document Intelligence pre-built layout analysis in Markdown format.\n",
    "display(Markdown(markdown))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Gets the parsed VehicleInsurancePolicy object from the completion response.\n",
    "insurance_policy = completion.choices[0].message.parsed\n",
    "\n",
    "expected_dict = expected.model_dump()\n",
    "insurance_policy_dict = insurance_policy.model_dump()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Determines the accuracy of the extracted data against the expected values.\n",
    "accuracy = insurance_policy_evaluator.evaluate(expected=expected_dict, actual=insurance_policy_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Determines the confidence of the extracted data using both the OpenAI and Azure Document Intelligence responses.\n",
    "di_confidence = evaluate_di_confidence(insurance_policy_dict, result)\n",
    "oai_confidence = evaluate_openai_confidence(insurance_policy_dict, completion.choices[0])\n",
    "\n",
    "confidence = merge_confidence_values(di_confidence, oai_confidence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Gets the total execution time of the data extraction process.\n",
    "total_elapsed = di_stopwatch.elapsed + image_stopwatch.elapsed + oai_stopwatch.elapsed\n",
    "\n",
    "# Gets the prompt tokens and completion tokens from the completion response.\n",
    "prompt_tokens = completion.usage.prompt_tokens\n",
    "completion_tokens = completion.usage.completion_tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the output of the data extraction result.\n",
    "extraction_result = DataExtractionResult(insurance_policy_dict, confidence, accuracy, prompt_tokens, completion_tokens, total_elapsed)\n",
    "\n",
    "create_json_file(f\"{sample_path}/{sample_name}.{pdf_fname}.json\", extraction_result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>Confidence</th>\n",
       "      <th>Execution Time</th>\n",
       "      <th>Document Intelligence Execution Time</th>\n",
       "      <th>Image Pre-processing Execution Time</th>\n",
       "      <th>OpenAI Execution Time</th>\n",
       "      <th>Prompt Tokens</th>\n",
       "      <th>Completion Tokens</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>100.00%</td>\n",
       "      <td>92.26%</td>\n",
       "      <td>34.54 seconds</td>\n",
       "      <td>8.15 seconds</td>\n",
       "      <td>1.88 seconds</td>\n",
       "      <td>24.51 seconds</td>\n",
       "      <td>17646</td>\n",
       "      <td>252</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Accuracy Confidence Execution Time Document Intelligence Execution Time  \\\n",
       "0  100.00%     92.26%  34.54 seconds                         8.15 seconds   \n",
       "\n",
       "  Image Pre-processing Execution Time OpenAI Execution Time  Prompt Tokens  \\\n",
       "0                        1.88 seconds         24.51 seconds          17646   \n",
       "\n",
       "   Completion Tokens  \n",
       "0                252  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_38daf_row0_col0, #T_38daf_row0_col1, #T_38daf_row0_col2, #T_38daf_row0_col3, #T_38daf_row0_col4, #T_38daf_row1_col0, #T_38daf_row1_col1, #T_38daf_row1_col2, #T_38daf_row1_col3, #T_38daf_row1_col4, #T_38daf_row2_col0, #T_38daf_row2_col1, #T_38daf_row2_col2, #T_38daf_row2_col3, #T_38daf_row2_col4, #T_38daf_row3_col0, #T_38daf_row3_col1, #T_38daf_row3_col2, #T_38daf_row3_col3, #T_38daf_row3_col4, #T_38daf_row4_col0, #T_38daf_row4_col1, #T_38daf_row4_col2, #T_38daf_row4_col3, #T_38daf_row4_col4, #T_38daf_row5_col0, #T_38daf_row5_col1, #T_38daf_row5_col2, #T_38daf_row5_col3, #T_38daf_row5_col4, #T_38daf_row6_col0, #T_38daf_row6_col1, #T_38daf_row6_col2, #T_38daf_row6_col3, #T_38daf_row6_col4, #T_38daf_row7_col0, #T_38daf_row7_col1, #T_38daf_row7_col2, #T_38daf_row7_col3, #T_38daf_row7_col4, #T_38daf_row8_col0, #T_38daf_row8_col1, #T_38daf_row8_col2, #T_38daf_row8_col3, #T_38daf_row8_col4, #T_38daf_row9_col0, #T_38daf_row9_col1, #T_38daf_row9_col2, #T_38daf_row9_col3, #T_38daf_row9_col4, #T_38daf_row10_col0, #T_38daf_row10_col1, #T_38daf_row10_col2, #T_38daf_row10_col3, #T_38daf_row10_col4, #T_38daf_row11_col0, #T_38daf_row11_col1, #T_38daf_row11_col2, #T_38daf_row11_col3, #T_38daf_row11_col4, #T_38daf_row12_col0, #T_38daf_row12_col1, #T_38daf_row12_col2, #T_38daf_row12_col3, #T_38daf_row12_col4, #T_38daf_row13_col0, #T_38daf_row13_col1, #T_38daf_row13_col2, #T_38daf_row13_col3, #T_38daf_row13_col4, #T_38daf_row14_col0, #T_38daf_row14_col1, #T_38daf_row14_col2, #T_38daf_row14_col3, #T_38daf_row14_col4, #T_38daf_row15_col0, #T_38daf_row15_col1, #T_38daf_row15_col2, #T_38daf_row15_col3, #T_38daf_row15_col4, #T_38daf_row16_col0, #T_38daf_row16_col1, #T_38daf_row16_col2, #T_38daf_row16_col3, #T_38daf_row16_col4, #T_38daf_row17_col0, #T_38daf_row17_col1, #T_38daf_row17_col2, #T_38daf_row17_col3, #T_38daf_row17_col4, #T_38daf_row18_col0, #T_38daf_row18_col1, #T_38daf_row18_col2, #T_38daf_row18_col3, #T_38daf_row18_col4, #T_38daf_row19_col0, #T_38daf_row19_col1, #T_38daf_row19_col2, #T_38daf_row19_col3, #T_38daf_row19_col4, #T_38daf_row20_col0, #T_38daf_row20_col1, #T_38daf_row20_col2, #T_38daf_row20_col3, #T_38daf_row20_col4, #T_38daf_row21_col0, #T_38daf_row21_col1, #T_38daf_row21_col2, #T_38daf_row21_col3, #T_38daf_row21_col4, #T_38daf_row22_col0, #T_38daf_row22_col1, #T_38daf_row22_col2, #T_38daf_row22_col3, #T_38daf_row22_col4, #T_38daf_row23_col0, #T_38daf_row23_col1, #T_38daf_row23_col2, #T_38daf_row23_col3, #T_38daf_row23_col4, #T_38daf_row24_col0, #T_38daf_row24_col1, #T_38daf_row24_col2, #T_38daf_row24_col3, #T_38daf_row24_col4, #T_38daf_row25_col0, #T_38daf_row25_col1, #T_38daf_row25_col2, #T_38daf_row25_col3, #T_38daf_row25_col4 {\n",
       "  background-color: #66ff33;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_38daf\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th id=\"T_38daf_level0_col0\" class=\"col_heading level0 col0\" >Field</th>\n",
       "      <th id=\"T_38daf_level0_col1\" class=\"col_heading level0 col1\" >Expected</th>\n",
       "      <th id=\"T_38daf_level0_col2\" class=\"col_heading level0 col2\" >Extracted</th>\n",
       "      <th id=\"T_38daf_level0_col3\" class=\"col_heading level0 col3\" >Confidence</th>\n",
       "      <th id=\"T_38daf_level0_col4\" class=\"col_heading level0 col4\" >Accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_38daf_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "      <td id=\"T_38daf_row0_col0\" class=\"data row0 col0\" >accident_excess_compulsory</td>\n",
       "      <td id=\"T_38daf_row0_col1\" class=\"data row0 col1\" >300</td>\n",
       "      <td id=\"T_38daf_row0_col2\" class=\"data row0 col2\" >300</td>\n",
       "      <td id=\"T_38daf_row0_col3\" class=\"data row0 col3\" >91.10%</td>\n",
       "      <td id=\"T_38daf_row0_col4\" class=\"data row0 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_38daf_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
       "      <td id=\"T_38daf_row1_col0\" class=\"data row1 col0\" >accident_excess_unapproved_repair_penalty</td>\n",
       "      <td id=\"T_38daf_row1_col1\" class=\"data row1 col1\" >250</td>\n",
       "      <td id=\"T_38daf_row1_col2\" class=\"data row1 col2\" >250</td>\n",
       "      <td id=\"T_38daf_row1_col3\" class=\"data row1 col3\" >96.30%</td>\n",
       "      <td id=\"T_38daf_row1_col4\" class=\"data row1 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_38daf_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
       "      <td id=\"T_38daf_row2_col0\" class=\"data row2 col0\" >accident_excess_voluntary</td>\n",
       "      <td id=\"T_38daf_row2_col1\" class=\"data row2 col1\" >200</td>\n",
       "      <td id=\"T_38daf_row2_col2\" class=\"data row2 col2\" >200</td>\n",
       "      <td id=\"T_38daf_row2_col3\" class=\"data row2 col3\" >98.90%</td>\n",
       "      <td id=\"T_38daf_row2_col4\" class=\"data row2 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_38daf_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
       "      <td id=\"T_38daf_row3_col0\" class=\"data row3 col0\" >cost_annual_total</td>\n",
       "      <td id=\"T_38daf_row3_col1\" class=\"data row3 col1\" >750.000000</td>\n",
       "      <td id=\"T_38daf_row3_col2\" class=\"data row3 col2\" >750.000000</td>\n",
       "      <td id=\"T_38daf_row3_col3\" class=\"data row3 col3\" >94.34%</td>\n",
       "      <td id=\"T_38daf_row3_col4\" class=\"data row3 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_38daf_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
       "      <td id=\"T_38daf_row4_col0\" class=\"data row4 col0\" >cost_payable_by_date</td>\n",
       "      <td id=\"T_38daf_row4_col1\" class=\"data row4 col1\" >2024-05-22</td>\n",
       "      <td id=\"T_38daf_row4_col2\" class=\"data row4 col2\" >2024-05-22</td>\n",
       "      <td id=\"T_38daf_row4_col3\" class=\"data row4 col3\" >99.68%</td>\n",
       "      <td id=\"T_38daf_row4_col4\" class=\"data row4 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_38daf_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
       "      <td id=\"T_38daf_row5_col0\" class=\"data row5 col0\" >effective_from</td>\n",
       "      <td id=\"T_38daf_row5_col1\" class=\"data row5 col1\" >2024-05-12</td>\n",
       "      <td id=\"T_38daf_row5_col2\" class=\"data row5 col2\" >2024-05-12</td>\n",
       "      <td id=\"T_38daf_row5_col3\" class=\"data row5 col3\" >100.00%</td>\n",
       "      <td id=\"T_38daf_row5_col4\" class=\"data row5 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_38daf_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
       "      <td id=\"T_38daf_row6_col0\" class=\"data row6 col0\" >effective_to</td>\n",
       "      <td id=\"T_38daf_row6_col1\" class=\"data row6 col1\" >2025-05-11</td>\n",
       "      <td id=\"T_38daf_row6_col2\" class=\"data row6 col2\" >2025-05-11</td>\n",
       "      <td id=\"T_38daf_row6_col3\" class=\"data row6 col3\" >100.00%</td>\n",
       "      <td id=\"T_38daf_row6_col4\" class=\"data row6 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_38daf_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
       "      <td id=\"T_38daf_row7_col0\" class=\"data row7 col0\" >fire_and_theft_excess_compulsory</td>\n",
       "      <td id=\"T_38daf_row7_col1\" class=\"data row7 col1\" >250</td>\n",
       "      <td id=\"T_38daf_row7_col2\" class=\"data row7 col2\" >250</td>\n",
       "      <td id=\"T_38daf_row7_col3\" class=\"data row7 col3\" >96.30%</td>\n",
       "      <td id=\"T_38daf_row7_col4\" class=\"data row7 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_38daf_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
       "      <td id=\"T_38daf_row8_col0\" class=\"data row8 col0\" >fire_and_theft_excess_unapproved_repair_penalty</td>\n",
       "      <td id=\"T_38daf_row8_col1\" class=\"data row8 col1\" >250</td>\n",
       "      <td id=\"T_38daf_row8_col2\" class=\"data row8 col2\" >250</td>\n",
       "      <td id=\"T_38daf_row8_col3\" class=\"data row8 col3\" >96.30%</td>\n",
       "      <td id=\"T_38daf_row8_col4\" class=\"data row8 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_38daf_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
       "      <td id=\"T_38daf_row9_col0\" class=\"data row9 col0\" >fire_and_theft_excess_voluntary</td>\n",
       "      <td id=\"T_38daf_row9_col1\" class=\"data row9 col1\" >150</td>\n",
       "      <td id=\"T_38daf_row9_col2\" class=\"data row9 col2\" >150</td>\n",
       "      <td id=\"T_38daf_row9_col3\" class=\"data row9 col3\" >99.20%</td>\n",
       "      <td id=\"T_38daf_row9_col4\" class=\"data row9 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_38daf_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
       "      <td id=\"T_38daf_row10_col0\" class=\"data row10 col0\" >last_date_to_cancel</td>\n",
       "      <td id=\"T_38daf_row10_col1\" class=\"data row10 col1\" >2024-05-26</td>\n",
       "      <td id=\"T_38daf_row10_col2\" class=\"data row10 col2\" >2024-05-26</td>\n",
       "      <td id=\"T_38daf_row10_col3\" class=\"data row10 col3\" >96.32%</td>\n",
       "      <td id=\"T_38daf_row10_col4\" class=\"data row10 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_38daf_level0_row11\" class=\"row_heading level0 row11\" >11</th>\n",
       "      <td id=\"T_38daf_row11_col0\" class=\"data row11 col0\" >policy_number</td>\n",
       "      <td id=\"T_38daf_row11_col1\" class=\"data row11 col1\" >GB34567890123</td>\n",
       "      <td id=\"T_38daf_row11_col2\" class=\"data row11 col2\" >GB34567890123</td>\n",
       "      <td id=\"T_38daf_row11_col3\" class=\"data row11 col3\" >99.40%</td>\n",
       "      <td id=\"T_38daf_row11_col4\" class=\"data row11 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_38daf_level0_row12\" class=\"row_heading level0 row12\" >12</th>\n",
       "      <td id=\"T_38daf_row12_col0\" class=\"data row12 col0\" >policyholder_address</td>\n",
       "      <td id=\"T_38daf_row12_col1\" class=\"data row12 col1\" >18 Elm Grove, Bristol, BS1 5HQ</td>\n",
       "      <td id=\"T_38daf_row12_col2\" class=\"data row12 col2\" >18 Elm Grove, Bristol, BS1 5HQ</td>\n",
       "      <td id=\"T_38daf_row12_col3\" class=\"data row12 col3\" >96.30%</td>\n",
       "      <td id=\"T_38daf_row12_col4\" class=\"data row12 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_38daf_level0_row13\" class=\"row_heading level0 row13\" >13</th>\n",
       "      <td id=\"T_38daf_row13_col0\" class=\"data row13 col0\" >policyholder_date_of_birth</td>\n",
       "      <td id=\"T_38daf_row13_col1\" class=\"data row13 col1\" >1990-09-08</td>\n",
       "      <td id=\"T_38daf_row13_col2\" class=\"data row13 col2\" >1990-09-08</td>\n",
       "      <td id=\"T_38daf_row13_col3\" class=\"data row13 col3\" >99.85%</td>\n",
       "      <td id=\"T_38daf_row13_col4\" class=\"data row13 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_38daf_level0_row14\" class=\"row_heading level0 row14\" >14</th>\n",
       "      <td id=\"T_38daf_row14_col0\" class=\"data row14 col0\" >policyholder_driving_license_number</td>\n",
       "      <td id=\"T_38daf_row14_col1\" class=\"data row14 col1\" >BENNEL008099JJ9IT</td>\n",
       "      <td id=\"T_38daf_row14_col2\" class=\"data row14 col2\" >BENNEL008099JJ9IT</td>\n",
       "      <td id=\"T_38daf_row14_col3\" class=\"data row14 col3\" >96.00%</td>\n",
       "      <td id=\"T_38daf_row14_col4\" class=\"data row14 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_38daf_level0_row15\" class=\"row_heading level0 row15\" >15</th>\n",
       "      <td id=\"T_38daf_row15_col0\" class=\"data row15 col0\" >policyholder_email_address</td>\n",
       "      <td id=\"T_38daf_row15_col1\" class=\"data row15 col1\" >LBennett@me.com</td>\n",
       "      <td id=\"T_38daf_row15_col2\" class=\"data row15 col2\" >LBennett@me.com</td>\n",
       "      <td id=\"T_38daf_row15_col3\" class=\"data row15 col3\" >97.30%</td>\n",
       "      <td id=\"T_38daf_row15_col4\" class=\"data row15 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_38daf_level0_row16\" class=\"row_heading level0 row16\" >16</th>\n",
       "      <td id=\"T_38daf_row16_col0\" class=\"data row16 col0\" >policyholder_first_name</td>\n",
       "      <td id=\"T_38daf_row16_col1\" class=\"data row16 col1\" >Laura</td>\n",
       "      <td id=\"T_38daf_row16_col2\" class=\"data row16 col2\" >Laura</td>\n",
       "      <td id=\"T_38daf_row16_col3\" class=\"data row16 col3\" >30.00%</td>\n",
       "      <td id=\"T_38daf_row16_col4\" class=\"data row16 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_38daf_level0_row17\" class=\"row_heading level0 row17\" >17</th>\n",
       "      <td id=\"T_38daf_row17_col0\" class=\"data row17 col0\" >policyholder_last_name</td>\n",
       "      <td id=\"T_38daf_row17_col1\" class=\"data row17 col1\" >Bennett</td>\n",
       "      <td id=\"T_38daf_row17_col2\" class=\"data row17 col2\" >Bennett</td>\n",
       "      <td id=\"T_38daf_row17_col3\" class=\"data row17 col3\" >30.00%</td>\n",
       "      <td id=\"T_38daf_row17_col4\" class=\"data row17 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_38daf_level0_row18\" class=\"row_heading level0 row18\" >18</th>\n",
       "      <td id=\"T_38daf_row18_col0\" class=\"data row18 col0\" >policyholder_total_years_of_residence_in_uk</td>\n",
       "      <td id=\"T_38daf_row18_col1\" class=\"data row18 col1\" >29</td>\n",
       "      <td id=\"T_38daf_row18_col2\" class=\"data row18 col2\" >29</td>\n",
       "      <td id=\"T_38daf_row18_col3\" class=\"data row18 col3\" >98.25%</td>\n",
       "      <td id=\"T_38daf_row18_col4\" class=\"data row18 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_38daf_level0_row19\" class=\"row_heading level0 row19\" >19</th>\n",
       "      <td id=\"T_38daf_row19_col0\" class=\"data row19 col0\" >renewal_last_date_to_renew</td>\n",
       "      <td id=\"T_38daf_row19_col1\" class=\"data row19 col1\" >2025-05-04</td>\n",
       "      <td id=\"T_38daf_row19_col2\" class=\"data row19 col2\" >2025-05-04</td>\n",
       "      <td id=\"T_38daf_row19_col3\" class=\"data row19 col3\" >95.06%</td>\n",
       "      <td id=\"T_38daf_row19_col4\" class=\"data row19 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_38daf_level0_row20\" class=\"row_heading level0 row20\" >20</th>\n",
       "      <td id=\"T_38daf_row20_col0\" class=\"data row20 col0\" >renewal_renewal_notification_date</td>\n",
       "      <td id=\"T_38daf_row20_col1\" class=\"data row20 col1\" >2025-04-20</td>\n",
       "      <td id=\"T_38daf_row20_col2\" class=\"data row20 col2\" >2025-04-20</td>\n",
       "      <td id=\"T_38daf_row20_col3\" class=\"data row20 col3\" >97.96%</td>\n",
       "      <td id=\"T_38daf_row20_col4\" class=\"data row20 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_38daf_level0_row21\" class=\"row_heading level0 row21\" >21</th>\n",
       "      <td id=\"T_38daf_row21_col0\" class=\"data row21 col0\" >vehicle_make</td>\n",
       "      <td id=\"T_38daf_row21_col1\" class=\"data row21 col1\" >Honda</td>\n",
       "      <td id=\"T_38daf_row21_col2\" class=\"data row21 col2\" >Honda</td>\n",
       "      <td id=\"T_38daf_row21_col3\" class=\"data row21 col3\" >98.76%</td>\n",
       "      <td id=\"T_38daf_row21_col4\" class=\"data row21 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_38daf_level0_row22\" class=\"row_heading level0 row22\" >22</th>\n",
       "      <td id=\"T_38daf_row22_col0\" class=\"data row22 col0\" >vehicle_model</td>\n",
       "      <td id=\"T_38daf_row22_col1\" class=\"data row22 col1\" >Civic</td>\n",
       "      <td id=\"T_38daf_row22_col2\" class=\"data row22 col2\" >Civic</td>\n",
       "      <td id=\"T_38daf_row22_col3\" class=\"data row22 col3\" >98.90%</td>\n",
       "      <td id=\"T_38daf_row22_col4\" class=\"data row22 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_38daf_level0_row23\" class=\"row_heading level0 row23\" >23</th>\n",
       "      <td id=\"T_38daf_row23_col0\" class=\"data row23 col0\" >vehicle_registration_number</td>\n",
       "      <td id=\"T_38daf_row23_col1\" class=\"data row23 col1\" >IJ90MNO</td>\n",
       "      <td id=\"T_38daf_row23_col2\" class=\"data row23 col2\" >IJ90MNO</td>\n",
       "      <td id=\"T_38daf_row23_col3\" class=\"data row23 col3\" >94.10%</td>\n",
       "      <td id=\"T_38daf_row23_col4\" class=\"data row23 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_38daf_level0_row24\" class=\"row_heading level0 row24\" >24</th>\n",
       "      <td id=\"T_38daf_row24_col0\" class=\"data row24 col0\" >vehicle_value</td>\n",
       "      <td id=\"T_38daf_row24_col1\" class=\"data row24 col1\" >11000.000000</td>\n",
       "      <td id=\"T_38daf_row24_col2\" class=\"data row24 col2\" >11000.000000</td>\n",
       "      <td id=\"T_38daf_row24_col3\" class=\"data row24 col3\" >99.14%</td>\n",
       "      <td id=\"T_38daf_row24_col4\" class=\"data row24 col4\" >Match</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_38daf_level0_row25\" class=\"row_heading level0 row25\" >25</th>\n",
       "      <td id=\"T_38daf_row25_col0\" class=\"data row25 col0\" >vehicle_year</td>\n",
       "      <td id=\"T_38daf_row25_col1\" class=\"data row25 col1\" >2019</td>\n",
       "      <td id=\"T_38daf_row25_col2\" class=\"data row25 col2\" >2019</td>\n",
       "      <td id=\"T_38daf_row25_col3\" class=\"data row25 col3\" >99.20%</td>\n",
       "      <td id=\"T_38daf_row25_col4\" class=\"data row25 col4\" >Match</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7ff43b692ab0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Display the outputs of the data extraction process.\n",
    "df = pd.DataFrame([\n",
    "    {\n",
    "        \"Accuracy\": f\"{accuracy['overall'] * 100:.2f}%\",\n",
    "        \"Confidence\": f\"{confidence['_overall'] * 100:.2f}%\",\n",
    "        \"Execution Time\": f\"{total_elapsed:.2f} seconds\",\n",
    "        \"Document Intelligence Execution Time\": f\"{di_stopwatch.elapsed:.2f} seconds\",\n",
    "        \"Image Pre-processing Execution Time\": f\"{image_stopwatch.elapsed:.2f} seconds\",\n",
    "        \"OpenAI Execution Time\": f\"{oai_stopwatch.elapsed:.2f} seconds\",\n",
    "        \"Prompt Tokens\": prompt_tokens,\n",
    "        \"Completion Tokens\": completion_tokens\n",
    "    }\n",
    "])\n",
    "\n",
    "display(df)\n",
    "display(get_extraction_comparison(expected_dict, insurance_policy_dict, confidence, accuracy['accuracy']))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}