samples/dotnet/classification/document-classification-text-embeddings.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Document Classification with Azure AI Document Intelligence and Text Embeddings\n",
    "\n",
    "This sample demonstrates how to classify a document using Azure AI Document Intelligence and text embeddings.\n",
    "\n",
    "![Data Classification](../../../images/classification-embeddings.png)\n",
    "\n",
    "This is achieved by the following process:\n",
    "\n",
    "- Define a list of classifications, with descriptions and keywords.\n",
    "- Create text embeddings for each of the classifications.\n",
    "- Analyze a document using Azure AI Document Intelligence's `prebuilt-layout` model to extract the text from each page.\n",
    "- For each page:\n",
    "  - Create text embeddings.\n",
    "  - Compare the embeddings with the embeddings of each classification.\n",
    "  - Assign the page to the classification with the highest similarity that exceeds a given threshold.\n",
    "\n",
    "## Objectives\n",
    "\n",
    "By the end of this sample, you will have learned how to:\n",
    "\n",
    "- Convert text to embeddings using Azure OpenAI's `text-embedding-3-large` model.\n",
    "- Convert a document's pages to Markdown format using Azure AI Document Intelligence.\n",
    "- Use cosine similarity to compare embeddings of classifications with document pages to classify them.\n",
    "\n",
    "## Useful Tips\n",
    "\n",
    "- Combine this technique with a [page extraction](../extraction/README.md) approach to ensure that you extract the most relevant data from a document's pages."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import modules\n",
    "\n",
    "This sample takes advantage of the following .NET dependencies:\n",
    "\n",
    "- **System.Numerics.Tensors** to perform cosine similiarity between embeddings.\n",
    "- **Azure.AI.DocumentIntelligence** to interface with the Azure AI Document Intelligence API for analyzing documents.\n",
    "- **Azure.AI.OpenAI** to interface with the Azure OpenAI chat completions API to generate structured classification outputs using the GPT-4o model.\n",
    "- **Azure.Identity** to securely authenticate with deployed Azure Services using Microsoft Entra ID credentials.\n",
    "\n",
    "The following local components are also used:\n",
    "\n",
    "- [**Classification**](../modules/samples/models/Classification.csx) to define the classifications.\n",
    "- [**AccuracyEvaluator**](../modules/samples/evaluation/AccuracyEvaluator.csx) to evaluate the output of the classification process with expected results.\n",
    "- [**DocumentProcessingResult**](../modules/samples/models/DocumentProcessingResult.csx) to store the results of the classification process as a file.\n",
    "- [**AppSettings**](../modules/samples/AppSettings.csx) to access environment variables from the `.env` file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><div></div><div></div><div><strong>Installed Packages</strong><ul><li><span>Azure.AI.DocumentIntelligence, 1.0.0</span></li><li><span>Azure.AI.OpenAI, 2.1.0</span></li><li><span>Azure.Identity, 1.13.2</span></li><li><span>DotNetEnv, 3.1.1</span></li><li><span>System.Numerics.Tensors, 9.0.3</span></li></ul></div></div>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#r \"nuget: System.Numerics.Tensors, 9.0.3\"\n",
    "#r \"nuget: Azure.Identity, 1.13.2\"\n",
    "#r \"nuget: Azure.AI.OpenAI, 2.1.0\"\n",
    "#r \"nuget: Azure.AI.DocumentIntelligence, 1.0.0\"\n",
    "#r \"nuget: DotNetEnv, 3.1.1\"\n",
    "\n",
    "#!import ../modules/samples/AppSettings.csx\n",
    "#!import ../modules/samples/helpers/StopwatchContext.csx\n",
    "#!import ../modules/samples/models/Classification.csx\n",
    "#!import ../modules/samples/models/DocumentProcessingResult.csx\n",
    "#!import ../modules/samples/evaluation/AccuracyEvaluator.csx\n",
    "\n",
    "using System;\n",
    "using System.IO;\n",
    "using System.Numerics.Tensors;\n",
    "using System.Text.Json;\n",
    "using Azure;\n",
    "using Azure.Core;\n",
    "using Azure.Identity;\n",
    "using Azure.AI.OpenAI;\n",
    "using Azure.AI.DocumentIntelligence;\n",
    "using DotNetEnv;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure the Azure services\n",
    "\n",
    "To use Azure AI Document Intelligence and Azure OpenAI, their SDKs are used to create a client instance using a deployed endpoint and authentication credentials.\n",
    "\n",
    "For this sample, the credentials of the Azure CLI are used to authenticate with the deployed services."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [],
   "source": [
    "string workingDir = Path.GetFullPath(\"../../../\");\n",
    "AppSettings settings = new AppSettings(new Dictionary<string, string>(Env.Load(Path.Combine(workingDir, \".env\"))));\n",
    "string samplePath = Path.Combine(workingDir, \"samples/dotnet/classification/\");\n",
    "string sampleName = \"document-classification-text-embeddings\";\n",
    "\n",
    "DefaultAzureCredential credential = new DefaultAzureCredential(\n",
    "    new DefaultAzureCredentialOptions { \n",
    "        ExcludeWorkloadIdentityCredential = true,\n",
    "        ExcludeAzureDeveloperCliCredential = true,\n",
    "        ExcludeEnvironmentCredential = true,\n",
    "        ExcludeManagedIdentityCredential = true,\n",
    "        ExcludeAzurePowerShellCredential = true,\n",
    "        ExcludeSharedTokenCacheCredential = true,\n",
    "        ExcludeInteractiveBrowserCredential = true\n",
    "    }\n",
    ");\n",
    "\n",
    "AzureOpenAIClient openaiClient = new AzureOpenAIClient(\n",
    "    new Uri(settings.OpenAIEndpoint),\n",
    "    credential\n",
    ");\n",
    "\n",
    "var documentIntelligenceClient = new DocumentIntelligenceClient(\n",
    "    new Uri(settings.AIServicesEndpoint),\n",
    "    credential\n",
    ");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Establish the expected output\n",
    "\n",
    "To compare the accuracy of the classification process, the expected output of the classification process has been defined in the following code block based on each page of a [Vehicle Insurance Policy](../../assets/vehicle_insurance/policy_1.pdf).\n",
    "\n",
    "The expected output has been defined by a human evaluating the document.\n",
    "\n",
    "> **Note**: Only the `PageNumber` and `Classification` are used in the accuracy evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [],
   "source": [
    "string path = Path.Combine(workingDir, \"samples/assets/vehicle_insurance/\");\n",
    "string pdfFName = \"policy_1.pdf\";\n",
    "string pdfFPath = Path.Combine(path, pdfFName);\n",
    "\n",
    "ClassificationsModel expected = new ClassificationsModel()\n",
    "{\n",
    "    Classifications = new List<ClassificationModel>()\n",
    "    {\n",
    "        new ClassificationModel() { PageNumber = 1, Classification = \"Insurance Policy\" },\n",
    "        new ClassificationModel() { PageNumber = 2, Classification = \"Insurance Policy\" },\n",
    "        new ClassificationModel() { PageNumber = 3, Classification = \"Insurance Policy\" },\n",
    "        new ClassificationModel() { PageNumber = 4, Classification = \"Insurance Policy\" },\n",
    "        new ClassificationModel() { PageNumber = 5, Classification = \"Insurance Policy\" },\n",
    "        new ClassificationModel() { PageNumber = 6, Classification = \"Insurance Certificate\" },\n",
    "        new ClassificationModel() { PageNumber = 7, Classification = \"Terms and Conditions\" },\n",
    "        new ClassificationModel() { PageNumber = 8, Classification = \"Terms and Conditions\" },\n",
    "        new ClassificationModel() { PageNumber = 9, Classification = \"Terms and Conditions\" },\n",
    "        new ClassificationModel() { PageNumber = 10, Classification = \"Terms and Conditions\" },\n",
    "        new ClassificationModel() { PageNumber = 11, Classification = \"Terms and Conditions\" },\n",
    "        new ClassificationModel() { PageNumber = 12, Classification = \"Terms and Conditions\" },\n",
    "        new ClassificationModel() { PageNumber = 13, Classification = \"Terms and Conditions\" }\n",
    "    }\n",
    "};\n",
    "\n",
    "AccuracyEvaluator<ClassificationsModel> classificationEvaluator = new AccuracyEvaluator<ClassificationsModel>(matchKeys: new List<string>() { \"PageNumber\" }, ignoreKeys: new List<string>() { \"PageNumber\", \"Similarity\" });"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define classifications\n",
    "\n",
    "The following code block defines the classifications for a document. Each classification has a name, description, and keywords that will be used to classify the document's pages.\n",
    "\n",
    "> **Note**, the classifications have been defined based on expected content in a specific type of document, in this example, [a Vehicle Insurance Policy](../../assets/vehicle_insurance/policy_1.pdf)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [],
   "source": [
    "List<ClassificationDefinitionModel> classifications = new List<ClassificationDefinitionModel>()\n",
    "{\n",
    "    new ClassificationDefinitionModel() { \n",
    "        Classification = \"Insurance Policy\", \n",
    "        Description = \"Specific information related to an insurance policy, such as coverage, limits, premiums, and terms, often used for reference or clarification purposes.\",\n",
    "        Keywords = new List<string>() {\n",
    "            \"welcome letter\",\n",
    "            \"personal details\",\n",
    "            \"vehicle details\",\n",
    "            \"insured driver details\",\n",
    "            \"policy details\",\n",
    "            \"incident/conviction history\",\n",
    "            \"schedule of insurance\",\n",
    "            \"vehicle damage excesses\"\n",
    "        }\n",
    "    },\n",
    "    new ClassificationDefinitionModel() { \n",
    "        Classification = \"Insurance Certificate\", \n",
    "        Description = \"A document that serves as proof of insurance coverage, often required for legal, regulatory, or contractual purposes.\",\n",
    "        Keywords = new List<string>() {\n",
    "            \"certificate of vehicle insurance\",\n",
    "            \"effective date of insurance\",\n",
    "            \"entitlement to drive\",\n",
    "            \"limitations of use\"\n",
    "        }\n",
    "    },\n",
    "    new ClassificationDefinitionModel() { \n",
    "        Classification = \"Terms and Conditions\", \n",
    "        Description = \"The rules, requirements, or obligations that govern an agreement or contract, often related to insurance policies, financial products, or legal documents.\",\n",
    "        Keywords = new List<string>() {\n",
    "            \"terms and conditions\",\n",
    "            \"legal statements\",\n",
    "            \"payment instructions\",\n",
    "            \"legal obligations\",\n",
    "            \"covered for\",\n",
    "            \"claim settlement\",\n",
    "            \"costs to pay\",\n",
    "            \"legal responsibility\",\n",
    "            \"personal accident coverage\",\n",
    "            \"medical expense coverage\",\n",
    "            \"personal liability coverage\",\n",
    "            \"windscreen damage coverage\",\n",
    "            \"uninsured motorist protection\",\n",
    "            \"renewal instructions\",\n",
    "            \"cancellation instructions\"\n",
    "        }\n",
    "    }\n",
    "};"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convert the document pages to Markdown\n",
    "\n",
    "To classify the document pages using embeddings, the text from each page must first be extracted.\n",
    "\n",
    "The following code block converts the document pages to Markdown format using Azure AI Document Intelligence's `prebuilt-layout` model.\n",
    "\n",
    "For the purposes of this sample, we will be classifying each page. The benefit of using Azure AI Document Intelligence for this extraction is that it provides a page-by-page analysis result of the document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [],
   "source": [
    "AnalyzeResult result;\n",
    "\n",
    "StopwatchContext diSw;\n",
    "\n",
    "using (diSw = new StopwatchContext())\n",
    "{\n",
    "    var pollerResult = await documentIntelligenceClient.AnalyzeDocumentAsync(\n",
    "        WaitUntil.Completed,\n",
    "        modelId: \"prebuilt-layout\",\n",
    "        bytesSource: BinaryData.FromBytes(File.ReadAllBytes(pdfFPath))\n",
    "    );\n",
    "\n",
    "    result = pollerResult.Value;\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [],
   "source": [
    "var pagesContent = new List<string>();\n",
    "\n",
    "foreach (var page in result.Pages)\n",
    "{\n",
    "    // Extract the entire content for each page of the document based on the span offsets and lengths\n",
    "    var pageContent = result.Content.Substring(page.Spans[0].Offset, page.Spans[0].Length);\n",
    "    pagesContent.Add(pageContent);\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create embeddings\n",
    "\n",
    "With the text extracted from the document and the classifications defined, the next step is to create embeddings for each page and classification."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Retrieving embeddings for text\n",
    "\n",
    "The following helper function retrieves embeddings for a given piece of text using Azure OpenAI's `text-embedding-3-large` model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [],
   "source": [
    "private async Task<float[]> GetEmbeddingAsync(string text)\n",
    "{\n",
    "    var embeddingClient = openaiClient.GetEmbeddingClient(settings.TextEmbeddingModelDeploymentName);\n",
    "    var embedding = await embeddingClient.GenerateEmbeddingAsync(text);\n",
    "    return embedding.Value.ToFloats().ToArray();\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convert the classifications to embeddings\n",
    "\n",
    "The following code block takes each classification and generates the embeddings for the keywords."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [],
   "source": [
    "StopwatchContext ceSw;\n",
    "\n",
    "using (ceSw = new StopwatchContext())\n",
    "{\n",
    "    foreach (var classification in classifications)\n",
    "    {\n",
    "        var classificationEmbeddings = await GetEmbeddingAsync(string.Join(\", \", classification.Keywords));\n",
    "        classification.Embedding = classificationEmbeddings;\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convert the document pages to embeddings\n",
    "\n",
    "The following code block takes each page of the document and generates the embeddings for the text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [],
   "source": [
    "var pageEmbeddings = new List<float[]>();\n",
    "\n",
    "StopwatchContext deSw;\n",
    "\n",
    "using (deSw = new StopwatchContext())\n",
    "{\n",
    "    foreach (var pageContent in pagesContent)\n",
    "    {\n",
    "        var pageEmbedding = await GetEmbeddingAsync(pageContent);\n",
    "        pageEmbeddings.Add(pageEmbedding);\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classify the document pages\n",
    "\n",
    "The following code block runs the classification process using cosine similarity to compare the embeddings of the document pages with the embeddings of the predefined categories.\n",
    "\n",
    "It performs the following steps iteratively for each page in the document:\n",
    "\n",
    "1. Calculates the cosine similarity between the embeddings of the page and the matrix of embeddings of the predefined categories.\n",
    "2. Finds the best match for the page based on the maximum cosine similarity score.\n",
    "3. If the cosine similarity score is above a certain threshold, the page is classified under the best match category. Otherwise, the page is classified as \"Unclassified\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [],
   "source": [
    "var similarityThreshold = 0.6f;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [],
   "source": [
    "ClassificationsModel documentClassifications;\n",
    "\n",
    "StopwatchContext classifySw;\n",
    "\n",
    "using (classifySw = new StopwatchContext())\n",
    "{\n",
    "    documentClassifications = new ClassificationsModel()\n",
    "    {\n",
    "        Classifications = new List<ClassificationModel>()\n",
    "    };\n",
    "\n",
    "    for (int i = 0; i < pageEmbeddings.Count; i++)\n",
    "    {\n",
    "        var pageEmbedding = pageEmbeddings[i];\n",
    "        var pageContent = pagesContent[i];\n",
    "\n",
    "        var similarities = new List<float>();\n",
    "\n",
    "        foreach (var c in classifications)\n",
    "        {\n",
    "            var similarity = TensorPrimitives.CosineSimilarity(new ReadOnlySpan<float>(pageEmbedding), new ReadOnlySpan<float>(c.Embedding));\n",
    "            similarities.Add(similarity);\n",
    "        }\n",
    "\n",
    "        var maxSimilarity = similarities.Max();\n",
    "        var classificationIndex = similarities.IndexOf(maxSimilarity);\n",
    "        var classification = classifications[classificationIndex];\n",
    "\n",
    "        documentClassifications.Classifications.Add(new ClassificationModel()\n",
    "        {\n",
    "            PageNumber = i + 1,\n",
    "            Classification = maxSimilarity >= similarityThreshold ? classification.Classification : $\"Unclassified ({classification.Classification})\",\n",
    "            Similarity = maxSimilarity,\n",
    "        });\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calculate the accuracy\n",
    "\n",
    "The following code block calculates the accuracy of the classification process by comparing the actual classifications with the predicted classifications."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [],
   "source": [
    "var accuracy = classificationEvaluator.Evaluate(expected, documentClassifications);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize the outputs\n",
    "\n",
    "To provide context for the execution of the code, the following code blocks visualize the outputs of the classification process.\n",
    "\n",
    "This includes:\n",
    "\n",
    "- The accuracy of the classification process comparing the expected output with the result of comparing the embeddings.\n",
    "- The execution time of the end-to-end process.\n",
    "- The classification results for each page in the document.\n",
    "\n",
    "### Understanding Similarity\n",
    "\n",
    "Cosine similarity is a metric used to measure how similar two vectors are. Embeddings are numerical representations of text. By converting a document page and classification keywords to embeddings, we can compare the similarity between the two using this technique.\n",
    "\n",
    "Similarity scores close to 1 indicate that the two vectors share similar characteristics, while scores closer to 0 or negative values indicate that the two vectors are dissimilar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [],
   "source": [
    "// Gets the total execution time of the classification process.\n",
    "var totalElapsed = diSw.Elapsed + ceSw.Elapsed + deSw.Elapsed + classifySw.Elapsed;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [],
   "source": [
    "// Save the output of the data classification result\n",
    "var classificationResult = new DataProcessingResult<ClassificationsModel>(\n",
    "    documentClassifications,\n",
    "    accuracy,\n",
    "    null,\n",
    "    null,\n",
    "    null,\n",
    "    totalElapsed\n",
    ");\n",
    "\n",
    "var classificationResultJson = JsonSerializer.Serialize(classificationResult, new JsonSerializerOptions { WriteIndented = true });\n",
    "var classificationResultFPath = Path.Combine(samplePath, $\"{sampleName}.{pdfFName}.json\");\n",
    "\n",
    "await File.WriteAllTextAsync(classificationResultFPath, classificationResultJson);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<details open=\"open\" class=\"dni-treeview\"><summary><span class=\"dni-code-hint\"><code>{ Accuracy = 58.97%, ExecutionTime = 15.67 seconds, DocumentIntelligenceExecutionTime = 9.41 seconds, ClassificationEmbeddingExecutionTime = 1.97 seconds, DocumentEmbeddingExecutionTime = 4.29 seconds, ClassificationExecutionTime = 0.00 seconds }</code></span></summary><div><table><thead><tr></tr></thead><tbody><tr><td>Accuracy</td><td><div class=\"dni-plaintext\"><pre>58.97%</pre></div></td></tr><tr><td>ExecutionTime</td><td><div class=\"dni-plaintext\"><pre>15.67 seconds</pre></div></td></tr><tr><td>DocumentIntelligenceExecutionTime</td><td><div class=\"dni-plaintext\"><pre>9.41 seconds</pre></div></td></tr><tr><td>ClassificationEmbeddingExecutionTime</td><td><div class=\"dni-plaintext\"><pre>1.97 seconds</pre></div></td></tr><tr><td>DocumentEmbeddingExecutionTime</td><td><div class=\"dni-plaintext\"><pre>4.29 seconds</pre></div></td></tr><tr><td>ClassificationExecutionTime</td><td><div class=\"dni-plaintext\"><pre>0.00 seconds</pre></div></td></tr></tbody></table></div></details><style>\r\n",
       ".dni-code-hint {\r\n",
       "    font-style: italic;\r\n",
       "    overflow: hidden;\r\n",
       "    white-space: nowrap;\r\n",
       "}\r\n",
       ".dni-treeview {\r\n",
       "    white-space: nowrap;\r\n",
       "}\r\n",
       ".dni-treeview td {\r\n",
       "    vertical-align: top;\r\n",
       "    text-align: start;\r\n",
       "}\r\n",
       "details.dni-treeview {\r\n",
       "    padding-left: 1em;\r\n",
       "}\r\n",
       "table td {\r\n",
       "    text-align: start;\r\n",
       "}\r\n",
       "table tr { \r\n",
       "    vertical-align: top; \r\n",
       "    margin: 0em 0px;\r\n",
       "}\r\n",
       "table tr td pre \r\n",
       "{ \r\n",
       "    vertical-align: top !important; \r\n",
       "    margin: 0em 0px !important;\r\n",
       "} \r\n",
       "table th {\r\n",
       "    text-align: start;\r\n",
       "}\r\n",
       "</style>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<details open=\"open\" class=\"dni-treeview\"><summary><span class=\"dni-code-hint\"><code>Submission#23+ClassificationsModel</code></span></summary><div><table><thead><tr></tr></thead><tbody><tr><td>Classifications</td><td><table><thead><tr><th><i>index</i></th><th>value</th></tr></thead><tbody><tr><td>0</td><td><details class=\"dni-treeview\"><summary><span class=\"dni-code-hint\"><code>Submission#23+ClassificationModel</code></span></summary><div><table><thead><tr></tr></thead><tbody><tr><td>PageNumber</td><td><div class=\"dni-plaintext\"><pre>1</pre></div></td></tr><tr><td>Classification</td><td><div class=\"dni-plaintext\"><pre>Unclassified (Insurance Policy)</pre></div></td></tr><tr><td>Similarity</td><td><div class=\"dni-plaintext\"><pre>0.5702783</pre></div></td></tr></tbody></table></div></details></td></tr><tr><td>1</td><td><details class=\"dni-treeview\"><summary><span class=\"dni-code-hint\"><code>Submission#23+ClassificationModel</code></span></summary><div><table><thead><tr></tr></thead><tbody><tr><td>PageNumber</td><td><div class=\"dni-plaintext\"><pre>2</pre></div></td></tr><tr><td>Classification</td><td><div class=\"dni-plaintext\"><pre>Unclassified (Insurance Policy)</pre></div></td></tr><tr><td>Similarity</td><td><div class=\"dni-plaintext\"><pre>0.5684335</pre></div></td></tr></tbody></table></div></details></td></tr><tr><td>2</td><td><details class=\"dni-treeview\"><summary><span class=\"dni-code-hint\"><code>Submission#23+ClassificationModel</code></span></summary><div><table><thead><tr></tr></thead><tbody><tr><td>PageNumber</td><td><div class=\"dni-plaintext\"><pre>3</pre></div></td></tr><tr><td>Classification</td><td><div class=\"dni-plaintext\"><pre>Insurance Policy</pre></div></td></tr><tr><td>Similarity</td><td><div class=\"dni-plaintext\"><pre>0.6199012</pre></div></td></tr></tbody></table></div></details></td></tr><tr><td>3</td><td><details class=\"dni-treeview\"><summary><span class=\"dni-code-hint\"><code>Submission#23+ClassificationModel</code></span></summary><div><table><thead><tr></tr></thead><tbody><tr><td>PageNumber</td><td><div class=\"dni-plaintext\"><pre>4</pre></div></td></tr><tr><td>Classification</td><td><div class=\"dni-plaintext\"><pre>Unclassified (Insurance Policy)</pre></div></td></tr><tr><td>Similarity</td><td><div class=\"dni-plaintext\"><pre>0.5451388</pre></div></td></tr></tbody></table></div></details></td></tr><tr><td>4</td><td><details class=\"dni-treeview\"><summary><span class=\"dni-code-hint\"><code>Submission#23+ClassificationModel</code></span></summary><div><table><thead><tr></tr></thead><tbody><tr><td>PageNumber</td><td><div class=\"dni-plaintext\"><pre>5</pre></div></td></tr><tr><td>Classification</td><td><div class=\"dni-plaintext\"><pre>Unclassified (Insurance Policy)</pre></div></td></tr><tr><td>Similarity</td><td><div class=\"dni-plaintext\"><pre>0.59131694</pre></div></td></tr></tbody></table></div></details></td></tr><tr><td>5</td><td><details class=\"dni-treeview\"><summary><span class=\"dni-code-hint\"><code>Submission#23+ClassificationModel</code></span></summary><div><table><thead><tr></tr></thead><tbody><tr><td>PageNumber</td><td><div class=\"dni-plaintext\"><pre>6</pre></div></td></tr><tr><td>Classification</td><td><div class=\"dni-plaintext\"><pre>Insurance Certificate</pre></div></td></tr><tr><td>Similarity</td><td><div class=\"dni-plaintext\"><pre>0.6859927</pre></div></td></tr></tbody></table></div></details></td></tr><tr><td>6</td><td><details class=\"dni-treeview\"><summary><span class=\"dni-code-hint\"><code>Submission#23+ClassificationModel</code></span></summary><div><table><thead><tr></tr></thead><tbody><tr><td>PageNumber</td><td><div class=\"dni-plaintext\"><pre>7</pre></div></td></tr><tr><td>Classification</td><td><div class=\"dni-plaintext\"><pre>Terms and Conditions</pre></div></td></tr><tr><td>Similarity</td><td><div class=\"dni-plaintext\"><pre>0.6170646</pre></div></td></tr></tbody></table></div></details></td></tr><tr><td>7</td><td><details class=\"dni-treeview\"><summary><span class=\"dni-code-hint\"><code>Submission#23+ClassificationModel</code></span></summary><div><table><thead><tr></tr></thead><tbody><tr><td>PageNumber</td><td><div class=\"dni-plaintext\"><pre>8</pre></div></td></tr><tr><td>Classification</td><td><div class=\"dni-plaintext\"><pre>Unclassified (Terms and Conditions)</pre></div></td></tr><tr><td>Similarity</td><td><div class=\"dni-plaintext\"><pre>0.5881937</pre></div></td></tr></tbody></table></div></details></td></tr><tr><td>8</td><td><details class=\"dni-treeview\"><summary><span class=\"dni-code-hint\"><code>Submission#23+ClassificationModel</code></span></summary><div><table><thead><tr></tr></thead><tbody><tr><td>PageNumber</td><td><div class=\"dni-plaintext\"><pre>9</pre></div></td></tr><tr><td>Classification</td><td><div class=\"dni-plaintext\"><pre>Unclassified (Terms and Conditions)</pre></div></td></tr><tr><td>Similarity</td><td><div class=\"dni-plaintext\"><pre>0.5563077</pre></div></td></tr></tbody></table></div></details></td></tr><tr><td>9</td><td><details class=\"dni-treeview\"><summary><span class=\"dni-code-hint\"><code>Submission#23+ClassificationModel</code></span></summary><div><table><thead><tr></tr></thead><tbody><tr><td>PageNumber</td><td><div class=\"dni-plaintext\"><pre>10</pre></div></td></tr><tr><td>Classification</td><td><div class=\"dni-plaintext\"><pre>Unclassified (Terms and Conditions)</pre></div></td></tr><tr><td>Similarity</td><td><div class=\"dni-plaintext\"><pre>0.5892735</pre></div></td></tr></tbody></table></div></details></td></tr><tr><td>10</td><td><details class=\"dni-treeview\"><summary><span class=\"dni-code-hint\"><code>Submission#23+ClassificationModel</code></span></summary><div><table><thead><tr></tr></thead><tbody><tr><td>PageNumber</td><td><div class=\"dni-plaintext\"><pre>11</pre></div></td></tr><tr><td>Classification</td><td><div class=\"dni-plaintext\"><pre>Terms and Conditions</pre></div></td></tr><tr><td>Similarity</td><td><div class=\"dni-plaintext\"><pre>0.60119116</pre></div></td></tr></tbody></table></div></details></td></tr><tr><td>11</td><td><details class=\"dni-treeview\"><summary><span class=\"dni-code-hint\"><code>Submission#23+ClassificationModel</code></span></summary><div><table><thead><tr></tr></thead><tbody><tr><td>PageNumber</td><td><div class=\"dni-plaintext\"><pre>12</pre></div></td></tr><tr><td>Classification</td><td><div class=\"dni-plaintext\"><pre>Terms and Conditions</pre></div></td></tr><tr><td>Similarity</td><td><div class=\"dni-plaintext\"><pre>0.64805645</pre></div></td></tr></tbody></table></div></details></td></tr><tr><td>12</td><td><details class=\"dni-treeview\"><summary><span class=\"dni-code-hint\"><code>Submission#23+ClassificationModel</code></span></summary><div><table><thead><tr></tr></thead><tbody><tr><td>PageNumber</td><td><div class=\"dni-plaintext\"><pre>13</pre></div></td></tr><tr><td>Classification</td><td><div class=\"dni-plaintext\"><pre>Unclassified (Terms and Conditions)</pre></div></td></tr><tr><td>Similarity</td><td><div class=\"dni-plaintext\"><pre>0.59910923</pre></div></td></tr></tbody></table></div></details></td></tr></tbody></table></td></tr></tbody></table></div></details><style>\r\n",
       ".dni-code-hint {\r\n",
       "    font-style: italic;\r\n",
       "    overflow: hidden;\r\n",
       "    white-space: nowrap;\r\n",
       "}\r\n",
       ".dni-treeview {\r\n",
       "    white-space: nowrap;\r\n",
       "}\r\n",
       ".dni-treeview td {\r\n",
       "    vertical-align: top;\r\n",
       "    text-align: start;\r\n",
       "}\r\n",
       "details.dni-treeview {\r\n",
       "    padding-left: 1em;\r\n",
       "}\r\n",
       "table td {\r\n",
       "    text-align: start;\r\n",
       "}\r\n",
       "table tr { \r\n",
       "    vertical-align: top; \r\n",
       "    margin: 0em 0px;\r\n",
       "}\r\n",
       "table tr td pre \r\n",
       "{ \r\n",
       "    vertical-align: top !important; \r\n",
       "    margin: 0em 0px !important;\r\n",
       "} \r\n",
       "table th {\r\n",
       "    text-align: start;\r\n",
       "}\r\n",
       "</style>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "// Display the outputs of the classification process.\n",
    "var output = new\n",
    "{\n",
    "    Accuracy = $\"{float.Parse(accuracy[\"overall\"].ToString()) * 100:0.00}%\",\n",
    "    ExecutionTime = $\"{totalElapsed.TotalSeconds:0.00} seconds\",\n",
    "    DocumentIntelligenceExecutionTime = $\"{diSw.Elapsed.TotalSeconds:0.00} seconds\",\n",
    "    ClassificationEmbeddingExecutionTime = $\"{ceSw.Elapsed.TotalSeconds:0.00} seconds\",\n",
    "    DocumentEmbeddingExecutionTime = $\"{deSw.Elapsed.TotalSeconds:0.00} seconds\",\n",
    "    ClassificationExecutionTime = $\"{classifySw.Elapsed.TotalSeconds:0.00} seconds\",\n",
    "};\n",
    "\n",
    "display(output);\n",
    "display(documentClassifications);"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".NET (C#)",
   "language": "C#",
   "name": ".net-csharp"
  },
  "language_info": {
   "name": "python"
  },
  "polyglot_notebook": {
   "kernelInfo": {
    "defaultKernelName": "csharp",
    "items": [
     {
      "aliases": [],
      "name": "csharp"
     }
    ]
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}