samples/python/classification/document-classification-gpt-vision.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Document Classification with Azure OpenAI's GPT-4o Vision Capabilities\n",
    "\n",
    "This sample demonstrates how to classify a document using Azure OpenAI's GPT-4o model with vision capabilities.\n",
    "\n",
    "![Data Classification](../../../images/classification-openai.png)\n",
    "\n",
    "This is achieved by the following process:\n",
    "\n",
    "- Define a list of classifications, with descriptions and keywords.\n",
    "- Construct a system prompt that defines the instruction for classifying document pages.\n",
    "- Construct a user prompt that includes the defined classifications, and each document page as an base64 encoded image.\n",
    "- Use the Azure OpenAI chat completions API with the GPT-4o model to generate a classification for each document page as a structured output.\n",
    "\n",
    "## Objectives\n",
    "\n",
    "By the end of this sample, you will have learned how to:\n",
    "\n",
    "- Convert a document into a set of base64 encoded images for processing by GPT-4o.\n",
    "- Use prompt engineering techniques to instruct GPT-4o to classify a document's pages into predefined categories.\n",
    "\n",
    "## Useful Tips\n",
    "\n",
    "- Combine this technique with a [page extraction](../extraction/README.md) approach to ensure that you extract the most relevant data from a document's pages."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import modules\n",
    "\n",
    "This sample takes advantage of the following Python dependencies:\n",
    "\n",
    "- **pdf2image** for converting a PDF file into a set of images per page.\n",
    "- **openai** to interface with the Azure OpenAI chat completions API to generate structured classification outputs using the GPT-4o model.\n",
    "- **azure-identity** to securely authenticate with deployed Azure Services using Microsoft Entra ID credentials.\n",
    "\n",
    "The following local components are also used:\n",
    "\n",
    "- [**classification**](../modules/samples/models/classification.py) to define the classifications.\n",
    "- [**accuracy_evaluator**](../modules/samples/evaluation/accuracy_evaluator.py) to evaluate the output of the classification process with expected results.\n",
    "- [**openai_confidence**](../modules/samples/confidence/openai_confidence.py) to calculate the confidence of the classification process based on the `logprobs` response from the OpenAI API request.\n",
    "- [**document_processing_result**](../modules/samples/models/document_processing_result.py) to store the results of the classification process as a file.\n",
    "- [**stopwatch**](../modules/samples/utils/stopwatch.py) to measure the end-to-end execution time for the classification process.\n",
    "- [**app_settings**](../modules/samples/app_settings.py) to access environment variables from the `.env` file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append('../modules/') # Import local modules\n",
    "\n",
    "from IPython.display import display\n",
    "import os\n",
    "import pandas as pd\n",
    "from dotenv import dotenv_values\n",
    "import base64\n",
    "import io\n",
    "import json\n",
    "from openai import AzureOpenAI\n",
    "from azure.identity import DefaultAzureCredential, get_bearer_token_provider\n",
    "from concurrent.futures import ThreadPoolExecutor\n",
    "from pdf2image import convert_from_bytes\n",
    "\n",
    "from samples.app_settings import AppSettings\n",
    "from samples.utils.stopwatch import Stopwatch\n",
    "from samples.utils.storage_utils import create_json_file\n",
    "from samples.models.document_processing_result import DataClassificationResult\n",
    "\n",
    "from samples.models.classification import Classifications, Classification\n",
    "from samples.confidence.openai_confidence import evaluate_confidence\n",
    "from samples.evaluation.accuracy_evaluator import AccuracyEvaluator\n",
    "from samples.evaluation.comparison import get_classification_comparison"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure the Azure services\n",
    "\n",
    "To use Azure OpenAI, the SDK is used to create a client instance using a deployed endpoint and authentication credentials.\n",
    "\n",
    "For this sample, the credentials of the Azure CLI are used to authenticate with the deployed services."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the working directory to the root of the repo\n",
    "working_dir = os.path.abspath('../../../')\n",
    "settings = AppSettings(dotenv_values(f\"{working_dir}/.env\"))\n",
    "sample_path = f\"{working_dir}/samples/python/classification/\"\n",
    "sample_name = \"document-classification-gpt-vision\"\n",
    "\n",
    "# Configure the default credential for accessing Azure services using Azure CLI credentials\n",
    "credential = DefaultAzureCredential(\n",
    "    exclude_workload_identity_credential=True,\n",
    "    exclude_developer_cli_credential=True,\n",
    "    exclude_environment_credential=True,\n",
    "    exclude_managed_identity_credential=True,\n",
    "    exclude_powershell_credential=True,\n",
    "    exclude_shared_token_cache_credential=True,\n",
    "    exclude_interactive_browser_credential=True\n",
    ")\n",
    "\n",
    "openai_token_provider = get_bearer_token_provider(credential, 'https://cognitiveservices.azure.com/.default')\n",
    "\n",
    "openai_client = AzureOpenAI(\n",
    "    azure_endpoint=settings.openai_endpoint,\n",
    "    azure_ad_token_provider=openai_token_provider,\n",
    "    api_version=\"2024-12-01-preview\" # Requires the latest API version for structured outputs.\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Establish the expected output\n",
    "\n",
    "To compare the accuracy of the classification process, the expected output of the classification process has been defined in the following code block based on each page of a [Vehicle Insurance Policy](../../assets/vehicle_insurance/policy_1.pdf).\n",
    "\n",
    "The expected output has been defined by a human evaluating the document.\n",
    "\n",
    "> **Note**: Only the `page_number` and `classification` are used in the accuracy evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = f\"{working_dir}/samples/assets/vehicle_insurance/\"\n",
    "pdf_fname = \"policy_1.pdf\"\n",
    "pdf_fpath = f\"{path}{pdf_fname}\"\n",
    "\n",
    "expected = Classifications(classifications=[\n",
    "    Classification(page_number=1, classification=\"Insurance Policy\", similarity=1),\n",
    "    Classification(page_number=2, classification=\"Insurance Policy\", similarity=1),\n",
    "    Classification(page_number=3, classification=\"Insurance Policy\", similarity=1),\n",
    "    Classification(page_number=4, classification=\"Insurance Policy\", similarity=1),\n",
    "    Classification(page_number=5, classification=\"Insurance Policy\", similarity=1),\n",
    "    Classification(page_number=6, classification=\"Insurance Certificate\", similarity=1),\n",
    "    Classification(page_number=7, classification=\"Terms and Conditions\", similarity=1),\n",
    "    Classification(page_number=8, classification=\"Terms and Conditions\", similarity=1),\n",
    "    Classification(page_number=9, classification=\"Terms and Conditions\", similarity=1),\n",
    "    Classification(page_number=10, classification=\"Terms and Conditions\", similarity=1),\n",
    "    Classification(page_number=11, classification=\"Terms and Conditions\", similarity=1),\n",
    "    Classification(page_number=12, classification=\"Terms and Conditions\", similarity=1),\n",
    "    Classification(page_number=13, classification=\"Terms and Conditions\", similarity=1)\n",
    "])\n",
    "\n",
    "classification_evaluator = AccuracyEvaluator(match_keys=[\"page_number\"], ignore_keys=[\"similarity\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define classifications\n",
    "\n",
    "The following code block defines the classifications for a document. Each classification has a name, description, and keywords that will be used to classify the document's pages.\n",
    "\n",
    "> **Note**, the classifications have been defined based on expected content in a specific type of document, in this example, [a Vehicle Insurance Policy](../../assets/vehicle_insurance/policy_1.pdf)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "classifications = [\n",
    "    {\n",
    "        \"classification\": \"Insurance Policy\",\n",
    "        \"description\": \"Specific information related to an insurance policy, such as coverage, limits, premiums, and terms, often used for reference or clarification purposes.\",\n",
    "        \"keywords\": [\n",
    "            \"welcome letter\",\n",
    "            \"personal details\",\n",
    "            \"vehicle details\",\n",
    "            \"insured driver details\",\n",
    "            \"policy details\",\n",
    "            \"incident/conviction history\",\n",
    "            \"schedule of insurance\",\n",
    "            \"vehicle damage excesses\"\n",
    "        ]\n",
    "    },\n",
    "    {\n",
    "        \"classification\": \"Insurance Certificate\",\n",
    "        \"description\": \"A document that serves as proof of insurance coverage, often required for legal, regulatory, or contractual purposes.\",\n",
    "        \"keywords\": [\n",
    "            \"certificate of vehicle insurance\",\n",
    "            \"effective date of insurance\",\n",
    "            \"entitlement to drive\",\n",
    "            \"limitations of use\"\n",
    "        ]\n",
    "    },\n",
    "    {\n",
    "        \"classification\": \"Terms and Conditions\",\n",
    "        \"description\": \"The rules, requirements, or obligations that govern an agreement or contract, often related to insurance policies, financial products, or legal documents.\",\n",
    "        \"keywords\": [\n",
    "            \"terms and conditions\",\n",
    "            \"legal statements\",\n",
    "            \"payment instructions\",\n",
    "            \"legal obligations\",\n",
    "            \"covered for\",\n",
    "            \"claim settlement\",\n",
    "            \"costs to pay\",\n",
    "            \"legal responsibility\",\n",
    "            \"personal accident coverage\",\n",
    "            \"medical expense coverage\",\n",
    "            \"personal liability coverage\",\n",
    "            \"windscreen damage coverage\",\n",
    "            \"uninsured motorist protection\",\n",
    "            \"renewal instructions\",\n",
    "            \"cancellation instructions\"\n",
    "        ]\n",
    "    }\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classify the document pages\n",
    "\n",
    "The following code block runs the classification process using Azure OpenAI's GPT-4o model using vision capabilities.\n",
    "\n",
    "It performs the following steps:\n",
    "\n",
    "1. Get the document bytes from the provided file path. _Note: In this example, we are processing a local document, however, you can use any document storage location of your choice, such as Azure Blob Storage._\n",
    "2. Use pdf2image to convert the document to a list of images per page as base64 strings.\n",
    "3. Use Azure OpenAI's GPT-4o model and the classification definitions to provide a classification for each page of the document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "system_prompt = f\"\"\"Using the classifications provided, classify each page of the following document into one of the classifications.\n",
    "\n",
    "- If a page contains multiple classifications, choose the most relevant one. \n",
    "- If a page does not fit any of the classifications, use the classification 'Unclassified'.\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prepare the user content for the OpenAI API including the classifications and the document page images.\n",
    "user_content = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_text_prompt = f\"\"\"Classifications:\n",
    "\n",
    "{json.dumps(classifications)}\"\"\"\n",
    "\n",
    "user_content.append({\n",
    "    \"type\": \"text\",\n",
    "    \"text\": user_text_prompt\n",
    "})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def encode_page(page):\n",
    "    byte_io = io.BytesIO()\n",
    "    page.save(byte_io, format='PNG')\n",
    "    base64_data = base64.b64encode(byte_io.getvalue()).decode('utf-8')\n",
    "    return {\n",
    "        \"type\": \"image_url\",\n",
    "        \"image_url\": {\n",
    "            \"url\": f\"data:image/png;base64,{base64_data}\"\n",
    "        }\n",
    "    }\n",
    "\n",
    "with Stopwatch() as image_stopwatch:\n",
    "    with open(pdf_fpath, \"rb\") as f:\n",
    "        document_bytes = f.read()\n",
    "\n",
    "    pages = convert_from_bytes(document_bytes)\n",
    "    \n",
    "    # Process each page in parallel using multiple processes\n",
    "    with ThreadPoolExecutor() as executor:\n",
    "        results = list(executor.map(encode_page, pages))\n",
    "        user_content.extend(results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with Stopwatch() as oai_stopwatch:\n",
    "    completion = openai_client.beta.chat.completions.parse(\n",
    "        model=settings.gpt4o_model_deployment_name,\n",
    "        messages=[\n",
    "            {\n",
    "                \"role\": \"system\",\n",
    "                \"content\": system_prompt,\n",
    "            },\n",
    "            {\n",
    "                \"role\": \"user\",\n",
    "                \"content\": user_content\n",
    "            }\n",
    "        ],\n",
    "        response_format=Classifications,\n",
    "        max_tokens=4096,\n",
    "        temperature=0.1,\n",
    "        top_p=0.1,\n",
    "        logprobs=True # Enabled to determine the confidence of the response.\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Understanding the Structured Outputs JSON schema\n",
    "\n",
    "Using [Pydantic's JSON schema feature](https://docs.pydantic.dev/latest/concepts/json_schema/), the [Classification](../modules/samples/models/classification.py) data model is automatically converted to a JSON schema when applied to the `response_format` parameter of the OpenAI chat completions request.\n",
    "\n",
    "The JSON schema is used to instruct the GPT-4o model to generate a strict output that adheres to the structure defined. The approach using Pydantic makes it easier for developers to manage the data structure in code, with helpful descriptions and examples that will be included in the final JSON schema.\n",
    "\n",
    "Demonstrated below, you can see how the Classification data model is understood by the OpenAI request:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"$defs\": {\n",
      "    \"Classification\": {\n",
      "      \"description\": \"A class representing a classification of a page.\\n\\nAttributes:\\n    page_number: The page number of the classification.\\n    classification: The classification of the page.\\n    similarity: The similarity of the classification from 0 to 100.\",\n",
      "      \"properties\": {\n",
      "        \"page_number\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"integer\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"The page number of the classification.\",\n",
      "          \"title\": \"Page Number\"\n",
      "        },\n",
      "        \"classification\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"string\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"The classification of the page.\",\n",
      "          \"title\": \"Classification\"\n",
      "        },\n",
      "        \"similarity\": {\n",
      "          \"anyOf\": [\n",
      "            {\n",
      "              \"type\": \"number\"\n",
      "            },\n",
      "            {\n",
      "              \"type\": \"null\"\n",
      "            }\n",
      "          ],\n",
      "          \"description\": \"The similarity of the classification from 0 to 100.\",\n",
      "          \"title\": \"Similarity\"\n",
      "        }\n",
      "      },\n",
      "      \"required\": [\n",
      "        \"page_number\",\n",
      "        \"classification\",\n",
      "        \"similarity\"\n",
      "      ],\n",
      "      \"title\": \"Classification\",\n",
      "      \"type\": \"object\"\n",
      "    }\n",
      "  },\n",
      "  \"description\": \"A class representing a list of classifications.\\n\\nAttributes:\\n    classifications: The list of Classification objects.\",\n",
      "  \"properties\": {\n",
      "    \"classifications\": {\n",
      "      \"description\": \"The list of Classification objects.\",\n",
      "      \"items\": {\n",
      "        \"$ref\": \"#/$defs/Classification\"\n",
      "      },\n",
      "      \"title\": \"Classifications\",\n",
      "      \"type\": \"array\"\n",
      "    }\n",
      "  },\n",
      "  \"required\": [\n",
      "    \"classifications\"\n",
      "  ],\n",
      "  \"title\": \"Classifications\",\n",
      "  \"type\": \"object\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "# Highlight the schema sent to the OpenAI model\n",
    "print(json.dumps(Classifications.model_json_schema(), indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calculate the accuracy\n",
    "\n",
    "The following code block calculates the accuracy of the classification process by comparing the actual classifications with the predicted classifications."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Gets the parsed Classifications object from the completion response.\n",
    "document_classifications = completion.choices[0].message.parsed\n",
    "\n",
    "expected_dict = expected.model_dump()\n",
    "classifications_dict = document_classifications.model_dump()\n",
    "\n",
    "accuracy = classification_evaluator.evaluate(expected=expected_dict, actual=classifications_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize the outputs\n",
    "\n",
    "To provide context for the execution of the code, the following code blocks visualize the outputs of the classification process.\n",
    "\n",
    "This includes:\n",
    "\n",
    "- The accuracy of the classification process comparing the expected output with the output generated by Azure OpenAI's GPT-4o model.\n",
    "- The confidence score of the classification process based on the log probability of the predicted classification.\n",
    "- The execution time of the end-to-end process.\n",
    "- The total number of tokens consumed by the GPT-4o model.\n",
    "- The classification results for each page of the document.\n",
    "\n",
    "### Understanding Accuracy vs Confidence\n",
    "\n",
    "When using AI to classify data, both confidence and accuracy are essential for different but complementary reasons.\n",
    "\n",
    "- **Accuracy** measures how close the AI model's output is to a ground truth or expected output. It reflects how well the model's predictions align with reality.\n",
    "  - Accuracy ensures consistency in the classification process, which is crucial for downstream tasks using the data.\n",
    "- **Confidence** represents the AI model's internal assessment of how certain it is about its predictions.\n",
    "  - Confidence indicates that the model is certain about its predictions, which can be a useful indicator for human reviewers to step in for manual verification.\n",
    "\n",
    "High accuracy and high confidence are ideal, but in practice, there is often a trade-off between the two. While accuracy cannot always be self-assessed, confidence scores can and should be used to prioritize manual verification of low-confidence predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Determines the confidence of the classifications using the log probabilities of the completion response.\n",
    "confidence = evaluate_confidence(classifications_dict, completion.choices[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Gets the total execution time of the classification process.\n",
    "total_elapsed = image_stopwatch.elapsed + oai_stopwatch.elapsed\n",
    "\n",
    "# Gets the prompt tokens and completion tokens from the completion response.\n",
    "prompt_tokens = completion.usage.prompt_tokens\n",
    "completion_tokens = completion.usage.completion_tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the output of the data classification result.\n",
    "classification_result = DataClassificationResult(\n",
    "    classification=classifications_dict,\n",
    "    accuracy=accuracy,\n",
    "    execution_time=total_elapsed\n",
    ")\n",
    "\n",
    "create_json_file(f\"{sample_path}/{sample_name}.{pdf_fname}.json\", classification_result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>Confidence</th>\n",
       "      <th>Execution Time</th>\n",
       "      <th>Image Pre-processing Execution Time</th>\n",
       "      <th>OpenAI Execution Time</th>\n",
       "      <th>Prompt Tokens</th>\n",
       "      <th>Completion Tokens</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>100.00%</td>\n",
       "      <td>99.63%</td>\n",
       "      <td>26.68 seconds</td>\n",
       "      <td>1.79 seconds</td>\n",
       "      <td>24.89 seconds</td>\n",
       "      <td>8721</td>\n",
       "      <td>208</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Accuracy Confidence Execution Time Image Pre-processing Execution Time  \\\n",
       "0  100.00%     99.63%  26.68 seconds                        1.79 seconds   \n",
       "\n",
       "  OpenAI Execution Time  Prompt Tokens  Completion Tokens  \n",
       "0         24.89 seconds           8721                208  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th style=\"text-align: right;\">  Page</th><th>Expected             </th><th>Extracted            </th><th style=\"text-align: right;\">  Similarity</th><th style=\"text-align: right;\">  Confidence</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td style=\"text-align: right;\">     1</td><td>Insurance Policy     </td><td>Insurance Policy     </td><td style=\"text-align: right;\">          95</td><td style=\"text-align: right;\">    0.999941</td></tr>\n",
       "<tr><td style=\"text-align: right;\">     2</td><td>Insurance Policy     </td><td>Insurance Policy     </td><td style=\"text-align: right;\">          95</td><td style=\"text-align: right;\">    0.999993</td></tr>\n",
       "<tr><td style=\"text-align: right;\">     3</td><td>Insurance Policy     </td><td>Insurance Policy     </td><td style=\"text-align: right;\">          95</td><td style=\"text-align: right;\">    0.999963</td></tr>\n",
       "<tr><td style=\"text-align: right;\">     4</td><td>Insurance Policy     </td><td>Insurance Policy     </td><td style=\"text-align: right;\">          95</td><td style=\"text-align: right;\">    0.99959 </td></tr>\n",
       "<tr><td style=\"text-align: right;\">     5</td><td>Insurance Policy     </td><td>Insurance Policy     </td><td style=\"text-align: right;\">          95</td><td style=\"text-align: right;\">    0.999884</td></tr>\n",
       "<tr><td style=\"text-align: right;\">     6</td><td>Insurance Certificate</td><td>Insurance Certificate</td><td style=\"text-align: right;\">          90</td><td style=\"text-align: right;\">    0.99893 </td></tr>\n",
       "<tr><td style=\"text-align: right;\">     7</td><td>Terms and Conditions </td><td>Terms and Conditions </td><td style=\"text-align: right;\">          90</td><td style=\"text-align: right;\">    0.997379</td></tr>\n",
       "<tr><td style=\"text-align: right;\">     8</td><td>Terms and Conditions </td><td>Terms and Conditions </td><td style=\"text-align: right;\">          90</td><td style=\"text-align: right;\">    0.99954 </td></tr>\n",
       "<tr><td style=\"text-align: right;\">     9</td><td>Terms and Conditions </td><td>Terms and Conditions </td><td style=\"text-align: right;\">          90</td><td style=\"text-align: right;\">    0.99993 </td></tr>\n",
       "<tr><td style=\"text-align: right;\">    10</td><td>Terms and Conditions </td><td>Terms and Conditions </td><td style=\"text-align: right;\">          90</td><td style=\"text-align: right;\">    0.999649</td></tr>\n",
       "<tr><td style=\"text-align: right;\">    11</td><td>Terms and Conditions </td><td>Terms and Conditions </td><td style=\"text-align: right;\">          90</td><td style=\"text-align: right;\">    0.999429</td></tr>\n",
       "<tr><td style=\"text-align: right;\">    12</td><td>Terms and Conditions </td><td>Terms and Conditions </td><td style=\"text-align: right;\">          90</td><td style=\"text-align: right;\">    0.99993 </td></tr>\n",
       "<tr><td style=\"text-align: right;\">    13</td><td>Terms and Conditions </td><td>Terms and Conditions </td><td style=\"text-align: right;\">          90</td><td style=\"text-align: right;\">  nan       </td></tr>\n",
       "</tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "'<table>\\n<thead>\\n<tr><th style=\"text-align: right;\">  Page</th><th>Expected             </th><th>Extracted            </th><th style=\"text-align: right;\">  Similarity</th><th style=\"text-align: right;\">  Confidence</th></tr>\\n</thead>\\n<tbody>\\n<tr><td style=\"text-align: right;\">     1</td><td>Insurance Policy     </td><td>Insurance Policy     </td><td style=\"text-align: right;\">          95</td><td style=\"text-align: right;\">    0.999941</td></tr>\\n<tr><td style=\"text-align: right;\">     2</td><td>Insurance Policy     </td><td>Insurance Policy     </td><td style=\"text-align: right;\">          95</td><td style=\"text-align: right;\">    0.999993</td></tr>\\n<tr><td style=\"text-align: right;\">     3</td><td>Insurance Policy     </td><td>Insurance Policy     </td><td style=\"text-align: right;\">          95</td><td style=\"text-align: right;\">    0.999963</td></tr>\\n<tr><td style=\"text-align: right;\">     4</td><td>Insurance Policy     </td><td>Insurance Policy     </td><td style=\"text-align: right;\">          95</td><td style=\"text-align: right;\">    0.99959 </td></tr>\\n<tr><td style=\"text-align: right;\">     5</td><td>Insurance Policy     </td><td>Insurance Policy     </td><td style=\"text-align: right;\">          95</td><td style=\"text-align: right;\">    0.999884</td></tr>\\n<tr><td style=\"text-align: right;\">     6</td><td>Insurance Certificate</td><td>Insurance Certificate</td><td style=\"text-align: right;\">          90</td><td style=\"text-align: right;\">    0.99893 </td></tr>\\n<tr><td style=\"text-align: right;\">     7</td><td>Terms and Conditions </td><td>Terms and Conditions </td><td style=\"text-align: right;\">          90</td><td style=\"text-align: right;\">    0.997379</td></tr>\\n<tr><td style=\"text-align: right;\">     8</td><td>Terms and Conditions </td><td>Terms and Conditions </td><td style=\"text-align: right;\">          90</td><td style=\"text-align: right;\">    0.99954 </td></tr>\\n<tr><td style=\"text-align: right;\">     9</td><td>Terms and Conditions </td><td>Terms and Conditions </td><td style=\"text-align: right;\">          90</td><td style=\"text-align: right;\">    0.99993 </td></tr>\\n<tr><td style=\"text-align: right;\">    10</td><td>Terms and Conditions </td><td>Terms and Conditions </td><td style=\"text-align: right;\">          90</td><td style=\"text-align: right;\">    0.999649</td></tr>\\n<tr><td style=\"text-align: right;\">    11</td><td>Terms and Conditions </td><td>Terms and Conditions </td><td style=\"text-align: right;\">          90</td><td style=\"text-align: right;\">    0.999429</td></tr>\\n<tr><td style=\"text-align: right;\">    12</td><td>Terms and Conditions </td><td>Terms and Conditions </td><td style=\"text-align: right;\">          90</td><td style=\"text-align: right;\">    0.99993 </td></tr>\\n<tr><td style=\"text-align: right;\">    13</td><td>Terms and Conditions </td><td>Terms and Conditions </td><td style=\"text-align: right;\">          90</td><td style=\"text-align: right;\">  nan       </td></tr>\\n</tbody>\\n</table>'"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Display the outputs of the classification process.\n",
    "df = pd.DataFrame([\n",
    "    {\n",
    "        \"Accuracy\": f\"{accuracy['overall'] * 100:.2f}%\",\n",
    "        \"Confidence\": f\"{confidence['_overall'] * 100:.2f}%\",\n",
    "        \"Execution Time\": f\"{total_elapsed:.2f} seconds\",\n",
    "        \"Image Pre-processing Execution Time\": f\"{image_stopwatch.elapsed:.2f} seconds\",\n",
    "        \"OpenAI Execution Time\": f\"{oai_stopwatch.elapsed:.2f} seconds\",\n",
    "        \"Prompt Tokens\": prompt_tokens,\n",
    "        \"Completion Tokens\": completion_tokens\n",
    "    }\n",
    "])\n",
    "\n",
    "display(df)\n",
    "display(get_classification_comparison(expected, document_classifications, confidence))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}