docs/source/llms/rag/notebooks/retriever-evaluation-tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Retriever Evaluation with MLflow"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "<a href=\"https://raw.githubusercontent.com/mlflow/mlflow/master/docs/source/llms/rag/notebooks/retriever-evaluation-tutorial.ipynb\" class=\"notebook-download-btn\"><i class=\"fas fa-download\"></i>Download this Notebook</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "f8938de9-7fae-41cd-ad6b-7ee26c288eab",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "In MLflow 2.8.0, we introduced a new model type \"retriever\" to the `mlflow.evaluate()` API. It helps you to evaluate the retriever in a RAG application. It contains two built-in metrics `precision_at_k` and `recall_at_k`. In MLflow 2.9.0, `ndcg_at_k` is available.\n",
    "\n",
    "This notebook illustrates how to use `mlflow.evaluate()` to evaluate the retriever in a RAG application. It has the following steps:\n",
    "\n",
    "* Step 1: Install and Load Packages\n",
    "* Step 2: Evaluation Dataset Preparation\n",
    "* Step 3: Calling `mlflow.evaluate()`\n",
    "* Step 4: Result Analysis and Visualization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Install and Load Packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "5bf12edb-2498-4edd-aeff-b4844451850f",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "%pip install mlflow==2.9.0 langchain==0.0.339 openai faiss-cpu gensim nltk pyLDAvis tiktoken"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "414eb948-7f7a-411b-8308-facadb0bdde8",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "import ast\n",
    "import os\n",
    "import pprint\n",
    "from typing import List\n",
    "\n",
    "import pandas as pd\n",
    "from langchain.docstore.document import Document\n",
    "from langchain.embeddings.openai import OpenAIEmbeddings\n",
    "from langchain.text_splitter import CharacterTextSplitter\n",
    "from langchain.vectorstores import FAISS\n",
    "\n",
    "import mlflow\n",
    "\n",
    "os.environ[\"OPENAI_API_KEY\"] = \"<redacted>\"\n",
    "\n",
    "CHUNK_SIZE = 1000\n",
    "\n",
    "# Assume running from https://github.com/mlflow/mlflow/blob/master/examples/llms/rag\n",
    "OUTPUT_DF_PATH = \"question_answer_source.csv\"\n",
    "SCRAPPED_DOCS_PATH = \"mlflow_docs_scraped.csv\"\n",
    "EVALUATION_DATASET_PATH = \"static_evaluation_dataset.csv\"\n",
    "DB_PERSIST_DIR = \"faiss_index\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "eebcf0d9-6634-47d9-808d-e79c5a50fbbf",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "## Step 2: Evaluation Dataset Preparation\n",
    "The evaluation dataset should contain three columns: questions, ground truth doc IDs, retrieved relevant doc IDs. A \"doc ID\" is a unique string identifier of the documents in you RAG application. For example, it could be the URL of a documentation web page, or the file path of a PDF document.\n",
    "\n",
    "If you have a list of questions that you would like to evaluate, please see 1.1 Manual Preparation. If you do not have a question list yet, please see 1.2 Generate the Evaluation Dataset.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "f8a690cc-7672-4f24-8518-8faabfc9afea",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "### Manual Preparation\n",
    "\n",
    "When evaluating a retriever, it's recommended to save the retrieved document IDs into a static dataset represented by a Pandas Dataframe or an MLflow Pandas Dataset containing the input queries, retrieved relevant document IDs, and the ground-truth document IDs for the evaluation.\n",
    "\n",
    "#### Concepts\n",
    "\n",
    "A \"document ID\" is a string that identifies a document.\n",
    "\n",
    "A list of \"retrieved relevant document IDs\" are the output of the retriever for a specific input query and a `k` value.\n",
    "\n",
    "A list of \"ground-truth document IDs\" are the labeled relevant documents for a specific input query.\n",
    "\n",
    "#### Expected Data Format\n",
    "\n",
    "For each row, the retrieved relevant document IDs and the ground-truth relevant document IDs should be provided as a tuple of document ID strings.\n",
    "\n",
    "The column name of the retrieved relevant document IDs should be specified by the `predictions` parameter, and the column name of the ground-truth relevant document IDs should be specified by the `targets` parameter.\n",
    "\n",
    "Here is a simple example dataset that illustrates the expected data format. The doc IDs are the paths of the documentation pages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "1a61b1b2-582e-49d5-864d-b58d2b6c3392",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "data = pd.DataFrame(\n",
    "    {\n",
    "        \"questions\": [\n",
    "            \"What is MLflow?\",\n",
    "            \"What is Databricks?\",\n",
    "            \"How to serve a model on Databricks?\",\n",
    "            \"How to enable MLflow Autologging for my workspace by default?\",\n",
    "        ],\n",
    "        \"retrieved_context\": [\n",
    "            [\n",
    "                \"mlflow/index.html\",\n",
    "                \"mlflow/quick-start.html\",\n",
    "            ],\n",
    "            [\n",
    "                \"introduction/index.html\",\n",
    "                \"getting-started/overview.html\",\n",
    "            ],\n",
    "            [\n",
    "                \"machine-learning/model-serving/index.html\",\n",
    "                \"machine-learning/model-serving/model-serving-intro.html\",\n",
    "            ],\n",
    "            [],\n",
    "        ],\n",
    "        \"ground_truth_context\": [\n",
    "            [\"mlflow/index.html\"],\n",
    "            [\"introduction/index.html\"],\n",
    "            [\n",
    "                \"machine-learning/model-serving/index.html\",\n",
    "                \"machine-learning/model-serving/llm-optimized-model-serving.html\",\n",
    "            ],\n",
    "            [\"mlflow/databricks-autologging.html\"],\n",
    "        ],\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "f740b47c-71ee-4633-944c-172887ff5081",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "### Generate the Evaluation Dataset\n",
    "There are two steps to generate the evaluation dataset: generate questions with ground truth doc IDs and retrieve relevant doc IDs. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "f6beddae-85e2-44e7-8ec6-7ca2f02bc16b",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "\n",
    "#### Generate Questions with Ground Truth Doc IDs\n",
    "If you don't have a list of questions to evaluate, you can generate them using LLMs. The [Question Generation Notebook](https://mlflow.org/docs/latest/llms/rag/notebooks/question-generation-retrieval-evaluation.html) provides an example way to do it. Here is the result of running that notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "98bf55c7-3e58-4fff-bc0e-1af58d64839f",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "generated_df = pd.read_csv(OUTPUT_DF_PATH)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "17baa097-457f-46df-9e25-56061972785f",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question</th>\n",
       "      <th>answer</th>\n",
       "      <th>chunk</th>\n",
       "      <th>chunk_id</th>\n",
       "      <th>source</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>What is the purpose of the MLflow Model Registry?</td>\n",
       "      <td>The purpose of the MLflow Model Registry is to...</td>\n",
       "      <td>Documentation MLflow Model Registry MLflow Mod...</td>\n",
       "      <td>0</td>\n",
       "      <td>model-registry.html</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>What is the purpose of registering a model wit...</td>\n",
       "      <td>The purpose of registering a model with the Mo...</td>\n",
       "      <td>logged, this model can then be registered with...</td>\n",
       "      <td>1</td>\n",
       "      <td>model-registry.html</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>What can you do with registered models and mod...</td>\n",
       "      <td>With registered models and model versions, you...</td>\n",
       "      <td>associate with registered models and model ver...</td>\n",
       "      <td>2</td>\n",
       "      <td>model-registry.html</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            question  \\\n",
       "0  What is the purpose of the MLflow Model Registry?   \n",
       "1  What is the purpose of registering a model wit...   \n",
       "2  What can you do with registered models and mod...   \n",
       "\n",
       "                                              answer  \\\n",
       "0  The purpose of the MLflow Model Registry is to...   \n",
       "1  The purpose of registering a model with the Mo...   \n",
       "2  With registered models and model versions, you...   \n",
       "\n",
       "                                               chunk  chunk_id  \\\n",
       "0  Documentation MLflow Model Registry MLflow Mod...         0   \n",
       "1  logged, this model can then be registered with...         1   \n",
       "2  associate with registered models and model ver...         2   \n",
       "\n",
       "                source  \n",
       "0  model-registry.html  \n",
       "1  model-registry.html  \n",
       "2  model-registry.html  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "generated_df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "93165dc5-aff9-46f9-83ab-e6dbfcbbc32b",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question</th>\n",
       "      <th>source</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>What is the purpose of the MLflow Model Registry?</td>\n",
       "      <td>[model-registry.html]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>What is the purpose of registering a model wit...</td>\n",
       "      <td>[model-registry.html]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>What can you do with registered models and mod...</td>\n",
       "      <td>[model-registry.html]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            question                 source\n",
       "0  What is the purpose of the MLflow Model Registry?  [model-registry.html]\n",
       "1  What is the purpose of registering a model wit...  [model-registry.html]\n",
       "2  What can you do with registered models and mod...  [model-registry.html]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Prepare dataframe `data` with the required format\n",
    "data = pd.DataFrame({})\n",
    "data[\"question\"] = generated_df[\"question\"].copy(deep=True)\n",
    "data[\"source\"] = generated_df[\"source\"].apply(lambda x: [x])\n",
    "data.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "3eabe651-28be-45bb-94ad-58e6bc582137",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "#### Retrieve Relevant Doc IDs\n",
    "\n",
    "Once we have a list of questions with ground truth doc IDs from 1.1, we can collect the retrieved relevant doc IDs. In this tutorial, we use a LangChain retriever. You can plug in your own retriever as needed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "9817f671-f2fd-4b2e-abe9-3bc9afd9ce3c",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "First, we build a FAISS retriever from the docs saved at https://github.com/mlflow/mlflow/blob/master/examples/llms/question_generation/mlflow_docs_scraped.csv. See the [Question Generation Notebook](https://mlflow.org/docs/latest/llms/rag/notebooks/question-generation-retrieval-evaluation.html) for how to create this csv file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "178b45b4-11f9-47ca-9564-c8caa32d2504",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "embeddings = OpenAIEmbeddings()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "e5a113bb-11b8-4d1a-a21b-b59b523f3525",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "scrapped_df = pd.read_csv(SCRAPPED_DOCS_PATH)\n",
    "list_of_documents = [\n",
    "    Document(page_content=row[\"text\"], metadata={\"source\": row[\"source\"]})\n",
    "    for i, row in scrapped_df.iterrows()\n",
    "]\n",
    "text_splitter = CharacterTextSplitter(chunk_size=CHUNK_SIZE, chunk_overlap=0)\n",
    "docs = text_splitter.split_documents(list_of_documents)\n",
    "db = FAISS.from_documents(docs, embeddings)\n",
    "\n",
    "# Save the db to local disk\n",
    "db.save_local(DB_PERSIST_DIR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "bace7c63-e3d5-42f3-bf6a-00ef1842baae",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "# Load the db from local disk\n",
    "db = FAISS.load_local(DB_PERSIST_DIR, embeddings)\n",
    "retriever = db.as_retriever()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "c06bcb3c-58c8-454c-bf5b-e29ec227991f",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Test the retriever with a query\n",
    "retrieved_docs = retriever.get_relevant_documents(\n",
    "    \"What is the purpose of the MLflow Model Registry?\"\n",
    ")\n",
    "len(retrieved_docs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "2ec7b458-c248-4ec0-9d85-0e447d6b4ecd",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "After building a retriever, we define a function that takes a question string as input and returns a list of relevant doc ID strings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "bc688e4b-3389-4804-b7bf-159bce4f9db8",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "# Define a function to return a list of retrieved doc ids\n",
    "def retrieve_doc_ids(question: str) -> List[str]:\n",
    "    docs = retriever.get_relevant_documents(question)\n",
    "    return [doc.metadata[\"source\"] for doc in docs]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "330a3336-6ca2-455f-a1ae-5bd842a4d2bb",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "We can store the retrieved doc IDs in the dataframe as a column \"retrieved_doc_ids\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "f96ec69b-bea3-4023-8cd3-6bee1e327ff0",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question</th>\n",
       "      <th>source</th>\n",
       "      <th>retrieved_doc_ids</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>What is the purpose of the MLflow Model Registry?</td>\n",
       "      <td>[model-registry.html]</td>\n",
       "      <td>[model-registry.html, introduction/index.html,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>What is the purpose of registering a model wit...</td>\n",
       "      <td>[model-registry.html]</td>\n",
       "      <td>[model-registry.html, models.html, introductio...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>What can you do with registered models and mod...</td>\n",
       "      <td>[model-registry.html]</td>\n",
       "      <td>[model-registry.html, models.html, deployment/...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            question                 source  \\\n",
       "0  What is the purpose of the MLflow Model Registry?  [model-registry.html]   \n",
       "1  What is the purpose of registering a model wit...  [model-registry.html]   \n",
       "2  What can you do with registered models and mod...  [model-registry.html]   \n",
       "\n",
       "                                   retrieved_doc_ids  \n",
       "0  [model-registry.html, introduction/index.html,...  \n",
       "1  [model-registry.html, models.html, introductio...  \n",
       "2  [model-registry.html, models.html, deployment/...  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[\"retrieved_doc_ids\"] = data[\"question\"].apply(retrieve_doc_ids)\n",
    "data.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "5e5c4cd1-38c3-4709-8d41-6e319fb8a924",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "# Persist the static evaluation dataset to disk\n",
    "data.to_csv(EVALUATION_DATASET_PATH, index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "deabd8f0-44cf-409f-a27b-e82dd4d99940",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question</th>\n",
       "      <th>source</th>\n",
       "      <th>retrieved_doc_ids</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>What is the purpose of the MLflow Model Registry?</td>\n",
       "      <td>[model-registry.html]</td>\n",
       "      <td>[model-registry.html, introduction/index.html,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>What is the purpose of registering a model wit...</td>\n",
       "      <td>[model-registry.html]</td>\n",
       "      <td>[model-registry.html, models.html, introductio...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>What can you do with registered models and mod...</td>\n",
       "      <td>[model-registry.html]</td>\n",
       "      <td>[model-registry.html, models.html, deployment/...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            question                 source  \\\n",
       "0  What is the purpose of the MLflow Model Registry?  [model-registry.html]   \n",
       "1  What is the purpose of registering a model wit...  [model-registry.html]   \n",
       "2  What can you do with registered models and mod...  [model-registry.html]   \n",
       "\n",
       "                                   retrieved_doc_ids  \n",
       "0  [model-registry.html, introduction/index.html,...  \n",
       "1  [model-registry.html, models.html, introductio...  \n",
       "2  [model-registry.html, models.html, deployment/...  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load the static evaluation dataset from disk and deserialize the source and retrieved doc ids\n",
    "data = pd.read_csv(EVALUATION_DATASET_PATH)\n",
    "data[\"source\"] = data[\"source\"].apply(ast.literal_eval)\n",
    "data[\"retrieved_doc_ids\"] = data[\"retrieved_doc_ids\"].apply(ast.literal_eval)\n",
    "data.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "c62b5306-9c0d-4ce8-8c4a-23cb1ecc7f66",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "## Step 3: Calling `mlflow.evaluate()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "bdeebdcc-b4e7-4f9d-8fdc-366f9c13ed20",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "### Metrics Definition\n",
    "\n",
    "There are three built-in metrics provided for the retriever model type. Click the metric name below to see the metrics definitions.\n",
    "\n",
    "1.  [mlflow.metrics.precision_at_k(k)](https://mlflow.org/docs/latest/python_api/mlflow.metrics.html#mlflow.metrics.precision_at_k)\n",
    "1.  [mlflow.metrics.recall_at_k(k)](https://mlflow.org/docs/latest/python_api/mlflow.metrics.html#mlflow.metrics.recall_at_k)\n",
    "1.  [mlflow.metrics.ndcg_at_k(k)](https://mlflow.org/docs/latest/python_api/mlflow.metrics.html#mlflow.metrics.ndcg_at_k) \n",
    "\n",
    "All metrics compute a score between 0 and 1 for each row representing the corresponding metric of the retriever model at the given `k` value.\n",
    "\n",
    "The `k` parameter should be a positive integer representing the number of retrieved documents\n",
    "to evaluate for each row. `k` defaults to 3.\n",
    "\n",
    "When the model type is `\"retriever\"`, these metrics will be calculated automatically with the\n",
    "default `k` value of 3.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "25a32237-b2ef-4f4a-9e5a-4537e7e43012",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "### Basic usage\n",
    "\n",
    "There are two supported ways to specify the retriever's output:\n",
    "\n",
    "* Case 1: Save the retriever's output to a static evaluation dataset\n",
    "* Case 2: Wrap the retriever in a function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "0390728a-a6cf-4c84-867a-0c6832114471",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023/11/22 14:39:59 WARNING mlflow.data.pandas_dataset: Failed to infer schema for Pandas dataset. Exception: Unable to map 'object' type to MLflow DataType. object can be mapped iff all values have identical data type which is one of (string, (bytes or byterray),  int, float).\n",
      "2023/11/22 14:39:59 INFO mlflow.models.evaluation.base: Evaluating the model with the default evaluator.\n",
      "2023/11/22 14:39:59 INFO mlflow.models.evaluation.default_evaluator: Testing metrics on first row...\n",
      "2023/11/22 14:39:59 INFO mlflow.models.evaluation.default_evaluator: Evaluating builtin metrics: precision_at_3\n",
      "2023/11/22 14:39:59 INFO mlflow.models.evaluation.default_evaluator: Evaluating builtin metrics: recall_at_3\n",
      "2023/11/22 14:39:59 INFO mlflow.models.evaluation.default_evaluator: Evaluating builtin metrics: ndcg_at_3\n"
     ]
    }
   ],
   "source": [
    "# Case 1: Evaluating a static evaluation dataset\n",
    "with mlflow.start_run() as run:\n",
    "    evaluate_results = mlflow.evaluate(\n",
    "        data=data,\n",
    "        model_type=\"retriever\",\n",
    "        targets=\"source\",\n",
    "        predictions=\"retrieved_doc_ids\",\n",
    "        evaluators=\"default\",\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "70aa6719-f69d-4fda-8a67-ac4e0d8ea6d8",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question</th>\n",
       "      <th>source</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>What is the purpose of the MLflow Model Registry?</td>\n",
       "      <td>[model-registry.html]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>What is the purpose of registering a model wit...</td>\n",
       "      <td>[model-registry.html]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>What can you do with registered models and mod...</td>\n",
       "      <td>[model-registry.html]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            question                 source\n",
       "0  What is the purpose of the MLflow Model Registry?  [model-registry.html]\n",
       "1  What is the purpose of registering a model wit...  [model-registry.html]\n",
       "2  What can you do with registered models and mod...  [model-registry.html]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "question_source_df = data[[\"question\", \"source\"]]\n",
    "question_source_df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "00672280-3dfc-4c00-9ae2-bea50732ef8b",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023/11/22 14:09:12 WARNING mlflow.data.pandas_dataset: Failed to infer schema for Pandas dataset. Exception: Unable to map 'object' type to MLflow DataType. object can be mapped iff all values have identical data type which is one of (string, (bytes or byterray),  int, float).\n",
      "2023/11/22 14:09:12 INFO mlflow.models.evaluation.base: Evaluating the model with the default evaluator.\n",
      "2023/11/22 14:09:12 INFO mlflow.models.evaluation.default_evaluator: Computing model predictions.\n",
      "2023/11/22 14:09:24 INFO mlflow.models.evaluation.default_evaluator: Testing metrics on first row...\n",
      "2023/11/22 14:09:24 INFO mlflow.models.evaluation.default_evaluator: Evaluating builtin metrics: precision_at_3\n",
      "2023/11/22 14:09:24 INFO mlflow.models.evaluation.default_evaluator: Evaluating builtin metrics: recall_at_3\n",
      "2023/11/22 14:09:24 INFO mlflow.models.evaluation.default_evaluator: Evaluating builtin metrics: ndcg_at_3\n"
     ]
    }
   ],
   "source": [
    "# Case 2: Evaluating a function\n",
    "def retriever_model_function(question_df: pd.DataFrame) -> pd.Series:\n",
    "    return question_df[\"question\"].apply(retrieve_doc_ids)\n",
    "\n",
    "\n",
    "with mlflow.start_run() as run:\n",
    "    evaluate_results = mlflow.evaluate(\n",
    "        model=retriever_model_function,\n",
    "        data=question_source_df,\n",
    "        model_type=\"retriever\",\n",
    "        targets=\"source\",\n",
    "        evaluators=\"default\",\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "cb24318b-6149-4703-ad06-731c8a75866f",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{   'ndcg_at_3/mean': 0.7530888125490431,\n",
      "    'ndcg_at_3/p90': 1.0,\n",
      "    'ndcg_at_3/variance': 0.1209151911325433,\n",
      "    'precision_at_3/mean': 0.26785714285714285,\n",
      "    'precision_at_3/p90': 0.3333333333333333,\n",
      "    'precision_at_3/variance': 0.017538265306122448,\n",
      "    'recall_at_3/mean': 0.8035714285714286,\n",
      "    'recall_at_3/p90': 1.0,\n",
      "    'recall_at_3/variance': 0.15784438775510204}\n"
     ]
    }
   ],
   "source": [
    "pp = pprint.PrettyPrinter(indent=4)\n",
    "pp.pprint(evaluate_results.metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "7a9b83c5-1544-4b0e-81f6-7abc1fafa258",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "### Try different k values\n",
    "To use another `k` value, use the `evaluator_config` parameter\n",
    "in the `mlflow.evaluate()` API as follows: `evaluator_config={\"retriever_k\": <k_value>}`.\n",
    "\n",
    "\n",
    "```python\n",
    "# Case 1: Specifying the model type\n",
    "evaluate_results = mlflow.evaluate(\n",
    "    data=data,\n",
    "    model_type=\"retriever\",\n",
    "    targets=\"ground_truth_context\",\n",
    "    predictions=\"retrieved_context\",\n",
    "    evaluators=\"default\",\n",
    "    evaluator_config={\"retriever_k\": 5}\n",
    "  )\n",
    "```\n",
    "\n",
    "Alternatively, you can directly specify the desired metrics\n",
    "in the `extra_metrics` parameter of the `mlflow.evaluate()` API without specifying a model\n",
    "type. In this case, the `k` value specified in the `evaluator_config` parameter will be\n",
    "ignored.\n",
    "\n",
    "\n",
    "```python\n",
    "# Case 2: Specifying the extra_metrics\n",
    "evaluate_results = mlflow.evaluate(\n",
    "    data=data,\n",
    "    targets=\"ground_truth_context\",\n",
    "    predictions=\"retrieved_context\",\n",
    "    extra_metrics=[\n",
    "      mlflow.metrics.precision_at_k(4),\n",
    "      mlflow.metrics.precision_at_k(5)\n",
    "    ],\n",
    "  )\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "4b7174aa-0aa2-497d-aaa5-842121fcf270",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023/11/22 14:40:22 WARNING mlflow.data.pandas_dataset: Failed to infer schema for Pandas dataset. Exception: Unable to map 'object' type to MLflow DataType. object can be mapped iff all values have identical data type which is one of (string, (bytes or byterray),  int, float).\n",
      "2023/11/22 14:40:22 INFO mlflow.models.evaluation.base: Evaluating the model with the default evaluator.\n",
      "2023/11/22 14:40:22 INFO mlflow.models.evaluation.default_evaluator: Testing metrics on first row...\n",
      "2023/11/22 14:40:22 INFO mlflow.models.evaluation.default_evaluator: Evaluating metrics: precision_at_1\n",
      "2023/11/22 14:40:22 INFO mlflow.models.evaluation.default_evaluator: Evaluating metrics: precision_at_2\n",
      "2023/11/22 14:40:22 INFO mlflow.models.evaluation.default_evaluator: Evaluating metrics: precision_at_3\n",
      "2023/11/22 14:40:22 INFO mlflow.models.evaluation.default_evaluator: Evaluating metrics: recall_at_1\n",
      "2023/11/22 14:40:22 INFO mlflow.models.evaluation.default_evaluator: Evaluating metrics: recall_at_2\n",
      "2023/11/22 14:40:22 INFO mlflow.models.evaluation.default_evaluator: Evaluating metrics: recall_at_3\n",
      "2023/11/22 14:40:22 INFO mlflow.models.evaluation.default_evaluator: Evaluating metrics: ndcg_at_1\n",
      "2023/11/22 14:40:22 INFO mlflow.models.evaluation.default_evaluator: Evaluating metrics: ndcg_at_2\n",
      "2023/11/22 14:40:22 INFO mlflow.models.evaluation.default_evaluator: Evaluating metrics: ndcg_at_3\n"
     ]
    }
   ],
   "source": [
    "with mlflow.start_run() as run:\n",
    "    evaluate_results = mlflow.evaluate(\n",
    "        data=data,\n",
    "        targets=\"source\",\n",
    "        predictions=\"retrieved_doc_ids\",\n",
    "        evaluators=\"default\",\n",
    "        extra_metrics=[\n",
    "            mlflow.metrics.precision_at_k(1),\n",
    "            mlflow.metrics.precision_at_k(2),\n",
    "            mlflow.metrics.precision_at_k(3),\n",
    "            mlflow.metrics.recall_at_k(1),\n",
    "            mlflow.metrics.recall_at_k(2),\n",
    "            mlflow.metrics.recall_at_k(3),\n",
    "            mlflow.metrics.ndcg_at_k(1),\n",
    "            mlflow.metrics.ndcg_at_k(2),\n",
    "            mlflow.metrics.ndcg_at_k(3),\n",
    "        ],\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "d57c201b-3718-43af-b8c2-ef22bfa2c15b",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAHHCAYAAABDUnkqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAAB4f0lEQVR4nO3dd1zV9f4H8NfZ7D1FBBRwM1VUXJkjc6Td0hy5NXf+zDIzLbX0lpnmvm4zV64sNbVIU5wloLhFUFxM2Ztzvr8/jhw5MgQFDhxez8eDe/N7vuf7fZ8jcl58pkgQBAFEREREekKs6wKIiIiIKhLDDREREekVhhsiIiLSKww3REREpFcYboiIiEivMNwQERGRXmG4ISIiIr3CcENERER6heGGiIiI9ArDDVEZbd68GSKRCHfv3tV1KbXGiRMnIBKJcOLECV2XUq2V9D5t3boVjRo1gkwmg4WFheb4okWLUL9+fUgkEvj4+FRprURVgeGGqp2CECESiRAcHFzkcUEQ4OzsDJFIhF69er3UPVatWoXNmze/YqVV48SJE3j77bfh4OAAuVwOOzs79O7dG/v27dN1aVRGCxYswC+//FKmc+/evav5/heJRJDJZLCxsUHbtm3x2WefITo6ukzXuXHjBoYPH44GDRpg3bp1WLt2LQDg2LFj+OSTTxAYGIhNmzZhwYIFL/uyKt2ZM2fw5ZdfIjk5uUznDx8+HCYmJkWOX758GTY2NnB1deUvJ7WFQFTNbNq0SQAgGBgYCOPHjy/y+PHjxwUAgkKhEHr27PlS92jatKnQsWPHcj0nPz9fyMrKElQq1Uvd82XMmTNHACB4eHgIc+bMETZs2CB8++23QqdOnQQAwrZt26qsFl1QKpVCVlaWoFQqdV3KKzE2NhaGDRtWpnOjoqIEAMLAgQOFrVu3Clu2bBGWLl0qDB48WDA0NBSMjIyEHTt2aD2nuPdp9erVAgDh9u3bWufOmDFDEIvFQk5Oziu/rsq2aNEiAYAQFRVVpvOHDRsmGBsbax0LDw8XbGxshHr16gmRkZGVUCVVR1Id5iqiUr355pvYvXs3li1bBqn02bfq9u3b4e/vj4SEhCqpIyMjA8bGxpBIJJBIJFVyTwDYs2cP5s2bh3feeQfbt2+HTCbTPPbxxx/j6NGjyMvLq7J6qlJ2djbkcjnEYjEMDAx0XY5O+Pn5YciQIVrH7t27h27dumHYsGFo3LgxvL29AaDY9ykuLg4AtLqjCo4bGhpCLpdXWK2ZmZkwMjKqsOtVlKtXr6Jz584wNDTE8ePH4ebmpuuSqKroOl0RPa+g5Wb37t2CSCQSDh8+rHksJydHsLS0FBYvXiy4uLgUablRKpXCkiVLhCZNmggKhUKws7MTxo4dKzx58kRzjouLiwBA66ugFafg3idOnBDGjx8v2NraChYWFlqPPf9b5OHDh4UOHToIJiYmgqmpqdCiRQutFpVbt24Jb7/9tmBvby8oFArByclJGDBggJCcnFzq+9CoUSPByspKSE1NLdP7FhsbK4wcOVKws7MTFAqF4OXlJWzevFnrnIJWgUWLFgkrVqwQ3NzcBENDQ6Fr165CdHS0oFKphHnz5glOTk6CgYGB0KdPHyExMVHrGgXv+9GjRwVvb29BoVAIjRs3Fvbu3at1XmJiovDRRx8JzZo1E4yNjQVTU1PhjTfeEMLCwrTOK2iJ27FjhzBr1iyhTp06gkgkEpKSkjSPHT9+vFzvZ15enjBv3jyhfv36glwuF1xcXISZM2cK2dnZxb6WU6dOCS1bthQUCoXg5uYmbNmypUzv+aJFi4Q2bdoIVlZWgoGBgeDn5yfs3r1b65znv9cAlNqKU/jvqDhnzpwRAAiDBg0q8h4WvE/FfY9/8cUXxdayadMmzXW2bt0q+Pn5CQYGBoKlpaUwYMAAITo6Wuv+HTt2FJo2bSr8+++/Qvv27QVDQ0Phww8/FARBELKzs4U5c+YIDRo0EORyuVC3bl3h448/LvK+AxAmTpwo7N+/X2jatKkgl8uFJk2aCL///rvmnJLqLa0Vp3DLzbVr1wR7e3uhbt26QkRERJFz//nnH6Fbt26CtbW1YGBgILi6ugojRowo8dpUs7DlhqotV1dXtGnTBjt27ECPHj0AAL///jtSUlLw3nvvYdmyZUWe88EHH2Dz5s0YMWIEpkyZgqioKKxYsQKhoaE4ffo0ZDIZli5dismTJ8PExASzZs0CANjb22tdZ8KECbC1tcWcOXOQkZFRYo2bN2/GyJEj0bRpU8ycORMWFhYIDQ3FkSNHMGjQIOTm5qJ79+7IycnB5MmT4eDggIcPH+LgwYNITk6Gubl5sde9ffs2bty4gZEjR8LU1PSF71VWVhY6deqEiIgITJo0CW5ubti9ezeGDx+O5ORkfPjhh1rnb9u2Dbm5uZg8eTKePHmCb7/9Fv3790fnzp1x4sQJzJgxAxEREVi+fDmmT5+OjRs3FqlvwIABGDduHIYNG4ZNmzbh3XffxZEjR9C1a1cAQGRkJH755Re8++67cHNzQ2xsLP73v/+hY8eOuHbtGurUqaN1zfnz50Mul2P69OnIyckptmWhrO/n6NGjsWXLFrzzzjv46KOPcP78eSxcuBDXr1/H/v37ta4ZERGBd955B6NGjcKwYcOwceNGDB8+HP7+/mjatGmp7/sPP/yAPn36YPDgwcjNzcXOnTvx7rvv4uDBg+jZsycA9aDe0aNHo1WrVhg7diwAoEGDBi/6Ky1RmzZt0KBBA/zxxx8lnrN06VL8+OOP2L9/P1avXg0TExN4eXnB3d0da9euxYULF7B+/XoAQNu2bQEAX3/9NWbPno3+/ftj9OjRiI+Px/Lly9GhQweEhoZqtQAlJiaiR48eeO+99zBkyBDY29tDpVKhT58+CA4OxtixY9G4cWOEh4djyZIluHXrVpExR8HBwdi3bx8mTJgAU1NTLFu2DP/5z38QHR0Na2trvP3227h16xZ27NiBJUuWwMbGBgBga2v7wvfo5s2b6Ny5M6RSKY4fP17k/Y6Li0O3bt1ga2uLTz/9FBYWFrh79y7HsekTXacroucVtJD8888/wooVKwRTU1MhMzNTEARBePfdd4XXXntNEAShSMvNqVOnih2HcuTIkSLHSxpzU3Dvdu3aCfn5+cU+VvCbY3JysmBqaioEBAQIWVlZWucWjMsJDQ3VtEKVx4EDBwQAwpIlS8p0/tKlSwUAwk8//aQ5lpubK7Rp00YwMTHRtP4UtArY2tpqtXTMnDlTACB4e3sLeXl5muMDBw4U5HK51m/eBa0ChVtqUlJSBEdHR8HX11dzLDs7u8hYmaioKEGhUAjz5s3THCtodahfv77m7/n5xwpaJMryfoaFhQkAhNGjR2sdnz59ugBA+Ouvv4q8lpMnT2qOxcXFCQqFQvjoo49KvEeB5+vNzc0VmjVrJnTu3Fnr+MuMuSmp5UYQBOGtt94SAAgpKSmCIBR9nwThWctHfHy81nOLG5dy9+5dQSKRCF9//bXW8fDwcEEqlWod79ixowBAWLNmjda5W7duFcRisXDq1Cmt42vWrBEACKdPn9YcAyDI5XKtFpVLly4JAITly5drjr3MmBuZTCY4OjoKderUEW7dulXsefv379f8jCH9xNlSVK31798fWVlZOHjwINLS0nDw4EEMGjSo2HN3794Nc3NzdO3aFQkJCZovf39/mJiY4Pjx42W+75gxY144vuaPP/5AWloaPv300yLjHUQiEQBoWhKOHj2KzMzMMt8/NTUVAMrUagMAhw8fhoODAwYOHKg5JpPJMGXKFKSnp+Pvv//WOv/dd9/VajUKCAgAAAwZMkRrfFNAQAByc3Px8OFDrefXqVMH/fr10/zZzMwMQ4cORWhoKGJiYgAACoUCYrH6R4xSqURiYiJMTEzQsGFDhISEFHkNw4YNg6GhYamvsyzv5+HDhwEA06ZN0zr+0UcfAQAOHTqkdbxJkyZo37695s+2trZo2LAhIiMjS60FgFa9SUlJSElJQfv27Yt9fRWpYEZQWlpahVxv3759UKlU6N+/v9a/HQcHB3h4eBT5t6NQKDBixAitY7t370bjxo3RqFEjrWt07twZAIpco0uXLlotKl5eXjAzMyvT+14apVKJhIQEWFlZaVp7nlfQCnXw4EG9HbdW2zHcULVma2uLLl26YPv27di3bx+USiXeeeedYs+9ffs2UlJSYGdnB1tbW62v9PR0zQDLsijLwMM7d+4AAJo1a1bqdaZNm4b169fDxsYG3bt3x8qVK5GSklLqtc3MzACU/cPr3r178PDw0ISJAo0bN9Y8Xli9evW0/lwQGpydnYs9npSUpHXc3d1dE+AKeHp6AoBmqq1KpcKSJUvg4eEBhUIBGxsb2Nra4vLly8W+/rK852V5P+/duwexWAx3d3et5zo4OMDCwuKF7wUAWFpaFnnNxTl48CBat24NAwMDWFlZwdbWFqtXr37h3++rSk9PB1D28Psit2/fhiAI8PDwKPJv5/r160X+7Tg5ORXpNrx9+zauXr1a5PkF3xfPX+NV3vfSGBoa4scff8S1a9fQs2fPYruVO3bsiP/85z+YO3cubGxs8NZbb2HTpk3Iycl5pXtT9cExN1TtDRo0CGPGjEFMTAx69OhRZPZHAZVKBTs7O2zbtq3Yx8vSV1/gRS0I5bF48WIMHz4cBw4cwLFjxzBlyhQsXLgQ586dQ926dYt9TqNGjQAA4eHhFVZHYSW1SpV0XBCEct9jwYIFmD17NkaOHIn58+fDysoKYrEYU6dOhUqlKnJ+Wd/zsr6fz4evkrzsaz516hT69OmDDh06YNWqVXB0dIRMJsOmTZuwffv2Mt37ZV25cgV2dnaaEPyqVCoVRCIRfv/992Lfj+fXjinu70qlUqF58+b4/vvvi73H88G5Ir/Xnvfee+8hKSkJEyZMwNtvv43ffvtNK4yJRCLs2bMH586dw2+//YajR49i5MiRWLx4Mc6dO1fsWjlUszDcULXXr18/fPDBBzh37hx27dpV4nkNGjTAn3/+icDAwBd+UJb1g680BU3qV65cKdJK8LzmzZujefPm+Pzzz3HmzBkEBgZizZo1+Oqrr4o939PTEw0bNsSBAwfwww8/vPCHrYuLCy5fvgyVSqXVenPjxg3N4xUpIiICgiBovY+3bt0CoB4IDqinsr/22mvYsGGD1nOTk5NL7C4oq9LeTxcXF6hUKty+fVvTcgUAsbGxSE5OrrD3Yu/evTAwMMDRo0ehUCg0xzdt2lTk3Ir4fitw9uxZ3Llzp8g08VfRoEEDCIIANzc3TUvLy1zj0qVLeP311yvs9b7KdcaPH48nT57g888/x5AhQ7Bz584iLZutW7dG69at8fXXX2P79u0YPHgwdu7cidGjR79q6aRj7Jaias/ExASrV6/Gl19+id69e5d4Xv/+/aFUKjF//vwij+Xn52utcmpsbFzmVU9L0q1bN5iammLhwoXIzs7Weqzgt8/U1FTk5+drPda8eXOIxeIXNoHPnTsXiYmJGD16dJFrAOqVZg8ePAhAvSZQTEyMVvjLz8/H8uXLYWJigo4dO77UayzJo0ePtGYdpaam4scff4SPjw8cHBwAqH8zf/638N27dxcZv1MeZXk/33zzTQDqGUOFFbQoFMxielUSiQQikQhKpVJz7O7du8WuRFwR32+Austt+PDhkMvl+Pjjj1/5egXefvttSCQSzJ07t8jfmSAISExMfOE1+vfvj4cPH2LdunVFHsvKyip11mFJjI2NAeCl37tZs2bh//7v/7B792588MEHmuNJSUlFXmfBNhTsmtIPbLmhGmHYsGEvPKdjx4744IMPsHDhQoSFhaFbt26QyWS4ffs2du/ejR9++EEzXsff3x+rV6/GV199BXd3d9jZ2WkGPpaVmZkZlixZgtGjR6Nly5YYNGgQLC0tcenSJWRmZmLLli3466+/MGnSJLz77rvw9PREfn4+tm7dColEgv/85z+lXn/AgAEIDw/H119/jdDQUAwcOBAuLi5ITEzEkSNHEBQUpOn+GDt2LP73v/9h+PDhuHjxIlxdXbFnzx6cPn0aS5curbCxGQU8PT0xatQo/PPPP7C3t8fGjRsRGxur1WrRq1cvzJs3DyNGjEDbtm0RHh6Obdu2oX79+i9937K8n97e3hg2bBjWrl2L5ORkdOzYERcuXMCWLVvQt29fvPbaa6/8+gF1SPr+++/xxhtvYNCgQYiLi8PKlSvh7u6Oy5cva53r7++PP//8E99//z3q1KkDNzc3zSDukoSEhOCnn36CSqVCcnIy/vnnH+zduxcikQhbt26Fl5dXhbwOQN3q8tVXX2HmzJm4e/cu+vbtC1NTU0RFRWH//v0YO3Yspk+fXuo13n//ffz8888YN24cjh8/jsDAQCiVSty4cQM///wzjh49ihYtWpSrLn9/fwDqkPLee+9BJpOhd+/emtBTFosXL0ZSUhLWr18PKysrfPPNN9iyZQtWrVqFfv36oUGDBkhLS8O6detgZmamCcdUw+lolhZRiQpPBS9NcYv4CYIgrF27VvD39xcMDQ0FU1NToXnz5sInn3wiPHr0SHNOTEyM0LNnT8HU1LTYRfyKu3dJi/j9+uuvQtu2bQVDQ0PBzMxMaNWqlWZ5/MjISGHkyJFCgwYNBAMDA8HKykp47bXXhD///LPM70dQUJDw1ltvCXZ2doJUKhVsbW2F3r17CwcOHNA6LzY2VhgxYoRgY2MjyOVyoXnz5loLtAlCydOMC6YSPz/Furj3o/Aifl5eXoJCoRAaNWpU5LnZ2dnCRx99JDg6OgqGhoZCYGCgcPbsWaFjx45a0/BLunfhxwqmOJf1/czLyxPmzp0ruLm5CTKZTHB2di51Eb/nPV9jSTZs2CB4eHho3oNNmzZppmAXduPGDaFDhw6CoaFhmRfxK/iSSqWClZWVEBAQIMycOVO4d+/eC98nQSjfVPACe/fuFdq1aycYGxsLxsbGQqNGjYSJEycKN2/e1HpvmjZtWuzzc3NzhW+++UZo2rSpoFAoBEtLS8Hf31+YO3euZtq6IDxbxO95Li4uRd6b+fPnC05OToJYLC7XIn6F5efnC3379hUACAsXLhRCQkKEgQMHCvXq1dMs9tmrVy/h33//LfHaVLOIBKECRm8RUa3h6uqKZs2aabrEiIiqG465ISIiIr3CcENERER6heGGiIiI9ArH3BAREZFeYcsNERER6RWGGyIiItIrtW4RP5VKhUePHsHU1LRCl0QnIiKiyiMIAtLS0lCnTp0iW2k8r9aFm0ePHhXZwI2IiIhqhvv375e46XCBWhduCpahv3//foXtqEtERESVKzU1Fc7OzmXaTqbWhZuCrigzMzOGGyIiohqmLENKOKCYiIiI9ArDDREREekVhhsiIiLSK7VuzE1ZKZVK5OXl6boMqgZkMhkkEomuyyAiojJiuHmOIAiIiYlBcnKyrkuhasTCwgIODg5cG4mIqAZguHlOQbCxs7ODkZERP8xqOUEQkJmZibi4OACAo6OjjisiIqIXYbgpRKlUaoKNtbW1rsuhasLQ0BAAEBcXBzs7O3ZRERFVcxxQXEjBGBsjIyMdV0LVTcH3BMdhERFVfww3xWBXFD2P3xNERDUHww0RERHpFYYbemknTpyASCQq08yy8pxb2apTLUREVPEYbuiltW3bFo8fP4a5uXmFnvsyoqOjMX36dHh7e8PGxgb169fHO++8gyNHjlTK/YiIqPrSebhZuXIlXF1dYWBggICAAFy4cKHU85cuXYqGDRvC0NAQzs7O+L//+z9kZ2dXUbX6Izc395WvIZfLy7z2S3nOLa+tW7eiWbNmePjwIb788ksEBQVhx44daN26NcaOHYuhQ4dCqVRW+H2JiKgYaTFAQoROS9BpuNm1axemTZuGL774AiEhIfD29kb37t01a4o8b/v27fj000/xxRdf4Pr169iwYQN27dqFzz77rIorr346deqESZMmYdKkSTA3N4eNjQ1mz54NQRAAAK6urpg/fz6GDh0KMzMzjB07FgAQHByM9u3ba8LilClTkJGRobluTk4OZsyYAWdnZygUCri7u2PDhg0Ainbv3Lt3D71794alpSWMjY3RtGlTHD58uNhzAWDv3r1o2rQpFAoFXF1dsXjxYq3X5OrqigULFmDkyJEwNTVFvXr1sHbtWq1zfvvtN3z88cc4duwYduzYgX79+sHb2xsBAQGYPn06rl+/jri4OEydOrXE9y4zMxM9evRAYGAgu6qIiMoj9TFw83fg+EJg+wDgu4bA4obAkU91WpZO17n5/vvvMWbMGIwYMQIAsGbNGhw6dAgbN27Ep58WfWPOnDmDwMBADBo0CID6w2/gwIE4f/58pdUoCAKy8nTzW7+hTFKulo4tW7Zg1KhRuHDhAv7991+MHTsW9erVw5gxYwAA3333HebMmYMvvvgCAHDnzh288cYb+Oqrr7Bx40bEx8drAtKmTZsAAEOHDsXZs2exbNkyeHt7IyoqCgkJCcXef+LEicjNzcXJkydhbGyMa9euwcTEpNhzL168iP79++PLL7/EgAEDcObMGUyYMAHW1tYYPny45rzFixdj/vz5+Oyzz7Bnzx6MHz8eHTt2RMOGDZGbm4tJkyZh8+bNaN26NYKDgzF16lTcv38f/fr1Q2ZmJrp3745t27bB09MTU6dORYMGDbTqSE5ORs+ePWFiYoI//viDywAQERVHEIDUR8DjMOBRGPD4kvq/02OLnisSA/m67VHRWbjJzc3FxYsXMXPmTM0xsViMLl264OzZs8U+p23btvjpp59w4cIFtGrVCpGRkTh8+DDef//9Eu+Tk5ODnJwczZ9TU1PLVWdWnhJN5hwt13MqyrV53WEkL/tfkbOzM5YsWQKRSISGDRsiPDwcS5Ys0YSbzp0746OPPtKcP3r0aAwePFjTquHh4YFly5ahY8eOWL16NaKjo/Hzzz/jjz/+QJcuXQAA9evXL/H+0dHR+M9//oPmzZu/8Nzvv/8er7/+OmbPng0A8PT0xLVr17Bo0SKtcPPmm29iwoQJAIAZM2ZgyZIlOH78OBo2bIi///4btra2eOONN5CcnIy33noLkyZNQr9+/bBnzx7897//RefOnWFtbY0333wTf/zxh1a4iYmJwYABA+Dh4YHt27dDLpeX+b0mItJbggCkPHgWYB6Fqf8/I77ouSIxYNsIcPQGHH2AOj6AQ3NAblylJT9PZ+EmISEBSqUS9vb2Wsft7e1x48aNYp8zaNAgJCQkoF27dhAEAfn5+Rg3blyp3VILFy7E3LlzK7T26qp169ZaLT1t2rTB4sWLNeNNWrRooXX+pUuXcPnyZWzbtk1zTBAEqFQqREVFITw8HBKJBB07dizT/adMmYLx48fj2LFj6NKlC/7zn//Ay8ur2HOvX7+Ot956S+tYYGAgli5dCqVSqVkFuPDzRSIRHBwcNN2W4eHhaNu2LQB1q561tbXm79rHxwe7du3SPNfR0RFJSUla9+vatStatWqFXbt2cdVhIqqdBAFIuf8swBS0ymQW00IvkqiDTB2fZ0HGvhkgr34t3jVq+4UTJ05gwYIFWLVqFQICAhAREYEPP/wQ8+fP17QAPG/mzJmYNm2a5s+pqalwdnYu8z0NZRJcm9f9lWt/GYayiv3ANTbWTtLp6en44IMPMGXKlCLn1qtXDxER5RsQNnr0aHTv3h2HDh3CsWPHsHDhQixevBiTJ09+6ZplMpnWn0UiEVQqFQAgPz9fszVCbm5ukddXuEssJCQEH3zwgdbjPXv2xN69e3Ht2jVNaxMRkd4SBCD53rMg8/iS+r+znhQ9VywFbBsDdZ62yDj6AA7NAJlhlZb8snQWbmxsbCCRSBAbq91fFxsbCwcHh2KfM3v2bLz//vsYPXo0AKB58+bIyMjA2LFjMWvWLIjFRcdHKxQKKBSKl65TJBKVq2tIl54fe3Tu3Dl4eHiU2Crh5+eHa9euwd3dvdjHmzdvDpVKhb///lvTLfUizs7OGDduHMaNG4eZM2di3bp1xYabxo0b4/Tp01rHTp8+DU9PzzK3ori7u+PEiRMAgJYtW+LGjRs4cOAAevfujd9++w2XLl1CVlYWFi1ahPv376NPnz5az//vf/8LExMTvP766zhx4gSaNGlSpvsSEVV7ggAkRT0LMAVhJiup6LliKWDXRN21VMcHcPQF7JsCMoMqLrri6OxTWy6Xw9/fH0FBQejbty8AQKVSISgoCJMmTSr2OZmZmUUCTMEHYcGsoNosOjoa06ZNwwcffICQkBAsX768yAykwmbMmIHWrVtj0qRJGD16tGYQ8B9//IEVK1bA1dUVw4YNw8iRIzUDiu/du4e4uDj079+/yPWmTp2KHj16wNPTE0lJSTh+/DgaN25c7L0/+ugjtGzZEvPnz8eAAQNw9uxZrFixAqtWrSrz6+3SpQvGjBmDW7duwdPTEytXrsTAgQORm5uLli1bonv37vjwww/Ro0cPBAUFFRtyv/vuOyiVSnTu3BknTpxAo0aNynx/IqJqQRCAJ5Ha42MeXwKyU4qeK5YB9k2edSs5+qiDjPTlGwGqI502SUybNg3Dhg1DixYt0KpVKyxduhQZGRma2VNDhw6Fk5MTFi5cCADo3bs3vv/+e/j6+mq6pWbPno3evXtzzATU71dWVhZatWoFiUSCDz/8UDPluzheXl74+++/MWvWLLRv3x6CIKBBgwYYMGCA5pzVq1fjs88+w4QJE5CYmIh69eqVOMZJqVRi4sSJePDgAczMzPDGG29gyZIlxZ7r5+eHn3/+GXPmzMH8+fPh6OiIefPmaQ0mfhEzMzPMmDED/fv3R1BQEEaOHIkhQ4YgMTERjo6OSExMhJGRkabrqiRLlizRCjienp5lroGIqEqpVM+CjCbMXAZyigkyErk6uBQOMnaN9S7IFEck6LjJY8WKFVi0aBFiYmLg4+ODZcuWISAgAIB67RZXV1ds3rwZgHqMxddff42tW7fi4cOHsLW1Re/evfH111/DwsKiTPdLTU2Fubk5UlJSYGZmpvVYdnY2oqKi4ObmBgODmtUc16lTJ/j4+GDp0qW6LqVKCYKACRMm4ODBg5gzZw769u0LW1tbZGRk4MiRI5g/fz7Wr19fZDB1edXk7w0iqqFUKuDJHe3BvjGXgZxiZv1KFOoxMYVnLdk2BqT6Mwu0tM/v5+k83FQ1hhv99Ouvv+Lbb7/F2bNnIZVKkZ+fjxYtWuDjjz/GO++888rXr8nfG0RUA6iUQGJE0SCTm170XKmBepZS4VlLto0AiazouXqkPOGmZoyUJXqBPn36oE+fPsjKykJCQgIsLCxgamqq67KIiIpSKYGEW9qL4T2+DORlFD1XaqheN6ZwkLFpCEj48V0avjt6omDWUG1XsI0EEVG1oMxXB5nCg31jwoG8zKLnyozUQabwGBkbTwaZl8B3jIiIqCIo84H4G88FmStAflbRc2XGgKPXc0HGAxBzckxFYLghIiIqL2UeEHddezG82CvF76kkN9Ee6OvoA1g3YJCpRAw3REREpcnPBeKvaw/2jb0KKHOKnqswexpkCoUZqwZAMYvMUuVhuCEiIiqQnwPEXdMOMnHXAGVu0XMV5uquJc1gX1/A0o1BphpguCEiotopP0fdlVR4i4LYa4Aqr+i5Buba3UqO3oBVfaDQZsVUfTDcEBGR/svLVnclPQ59FmTirgOq/KLnGlo+CzAFYcbSlUGmBmG4oZf25Zdf4pdffkFYWBgAYPjw4UhOTsYvv/zyStd1dXXF1KlTMXXq1FeukYhqobws9SylwrOW4q4DgrLouYZW2mvIOPoAFvUYZGo4hhuqdHl5edi0aRN+/vlnXL9+HUqlEvXr18fbb7+NCRMmwMjISNclElFNlZupXjemYDG8R2Hq6djFBRkjm+eCjDdg7swgo4cYbvRUbm4u5HLd7ykSGRmJt956C2KxGOPHj4eXlxdMTExw48YNbNq0CStXrsTRo0e5WSURvVhuhjrIFB7sm3ATEFRFzzW2exZgCsKMmRODTC3BcKMnOnXqhGbNmkEqleKnn35C8+bNsXz5cnz88cc4deoUjI2N0a1bNyxZsgQ2NjYAAJVKhe+++w5r167F/fv3YW9vjw8++ACzZs0CAMyYMQP79+/HgwcP4ODggMGDB2POnDmQycq2f0lKSgq6d++OgQMHYu7cuRAV+qHi5eWF/v37Y926dejWrRtCQ0NhaWlZ7HXWr1+P6dOnY+/evXj99ddf8Z0iohohJ129t5JWkLkFoJjtEE3stbuV6vgApo4MMrUYw82LCELxy2RXBZlRuf5xbtmyBePHj8fp06eRnJyMzp07Y/To0ViyZAmysrIwY8YM9O/fH3/99RcAYObMmVi3bh2WLFmCdu3a4fHjx7hx44bmeqampti8eTPq1KmD8PBwjBkzBqampvjkk0/KVM9///tf+Pv7Y968eUhOTsbEiRMRFBSE+vXr47333sPvv/+O33//HSdPnsTSpUsxd+7cItf49ttv8e233+LYsWNo1apVmd8LIqpBslPVQabwrKWE2yg2yJg6Fp21ZOZYldVSDcBw8yJ5mcCCOrq592ePALlxmU/38PDAt99+CwD46quv4OvriwULFmge37hxI5ydnXHr1i04Ojrihx9+wIoVKzBs2DAAQIMGDdCuXTvN+Z9//rnmv11dXTF9+nTs3LmzzOFm69atOHLkCADgo48+QlRUFA4cOIC4uDiMHTsWDRs2BKAeiDxr1qwi4WbGjBnYunUr/v77bzRt2rTM7wMRVWPZKepNIgsP9k28g2KDjJlT0VlLpvZVWCzVVAw3esTf31/z35cuXcLx48dhYmJS5Lw7d+4gOTkZOTk5pXbz7Nq1C8uWLcOdO3eQnp6O/Pz8F24zX+DJkydIS0tDs2bNAAC//fYbfvnlFwQEBAAAJk2ahD/++AMA4OjoiKSkJK3nL168GBkZGfj3339Rv379Mt2TiKqZrGTtgb6PLwFP7hR/rlndooN9TeyqqlLSMww3LyIzUreg6Ore5WBs/KyVJz09Hb1798Y333xT5DxHR0dERkaWeq2zZ89i8ODBmDt3Lrp37w5zc3Ps3LkTixcvLlMt+fn5MDAw0Pw5NzdXq77CoSskJATu7u5az2/fvj0OHTqEn3/+GZ9++mmZ7klEOpSV9CzAFISZpKjizzWvB9R5bq8lY5uqqpRqAYabFxGJytU1VF34+flh7969cHV1hVRa9K/Zw8MDhoaGCAoKwujRo4s8fubMGbi4uGgGFwPAvXv3ynx/Gxsb5ObmIjY2Fvb29mjXrh2+/fZbrF+/Hk+ePMG6detgY2ODM2fOYNasWdi4caPW81u1aoVJkybhjTfegFQqxfTp08vx6omoUmU+0e5WehQGJJfw88HCRXvWkqMPYGxdRYVSbcVwo6cmTpyIdevWYeDAgfjkk09gZWWFiIgI7Ny5E+vXr4eBgQFmzJiBTz75BHK5HIGBgYiPj8fVq1cxatQoeHh4IDo6Gjt37kTLli1x6NAh7N+/v8z3F4vF6NOnD1atWoW5c+fihx9+QO/evWFiYgJzc3MMGzYMS5cuxciRI/HDDz8U2z3Wtm1bHD58GD169IBUKuWifkS6kJGovarv40tAcnTx51q6Fh3sa2RVVZUSaTDc6Kk6derg9OnTmDFjBrp164acnBy4uLjgjTfegPjppm6zZ8+GVCrFnDlz8OjRIzg6OmLcuHEAgD59+uD//u//MGnSJOTk5KBnz56YPXs2vvzyyzLXMGfOHLRq1QqtW7dGjx49cO3aNcTExMDS0hIqlQqzZs3STEsvSbt27XDo0CG8+eabkEgkmDx58ku/J0T0AunxT7uVQp91MaXcL/5cq/rPBRkv9bYFRNWASBCEYoao66/U1FSYm5sjJSWlyODY7OxsREVFwc3NTWu8CL28Y8eO4b333sOQIUMwZswYzayn8PBwfPfdd7C1tcX333+v4ypfjN8bpHfS47S7lR6HAakPiz/X2l17MTwHL8DQoooKJVIr7fP7eWy5oUrVrVs3XLx4EfPmzUP79u2Rnp4OALCzs8OwYcMwc+ZMHVdIVAukxWh3Kz0KA9KKmyghUgeZwrOWHLwAg7LNkiSqLhhuqNK5ublh06ZN2LBhA2JjYyEWi2Fvz7UqiCqcIABpj7UXw3sUBqTHFHOyCLDxfC7INAcUplVYMNVk+ap8pOamIjknGak56v9PzklGSk4KrAys0LtBb53VxnBDVUYsFsPRkSuJElUIQQBSHxWdtZQRV/RckRiwaai9GJ5Dc0BRdB0sqn0EQUB6XrommKTkpGiCilZoyU1BSnaK5ry0vLQSr+lj68NwQ0REpRAEIOWB9mJ4j8OAjPii54rEgG0j7cG+Ds1q5JIWVH7Z+dnFh5TcVCRnP2tZSclN0TpPWdwu6mVkKjOFucIcFgoLmCvMYa4wRwOLBhX4qsqP4YaIqDoRBPVUa62VfcOAzMSi54okgF1j7SBj3xSQl28BUKp+8lX5muCRkptSajAp3MqSrcx+6XsaSAyKhJSC/37+/wv+20xuBqm4+kWJ6lcREVFtIQjqxe+0Zi1dArKeFD1XLH0WZBy9gTq+6iAjM6zamqlcBEFAWl7as+6cUoJJ4TEr6XnpL31PqUgKM4VZkSBSJLTItQOMgVR/ZoIy3BARVQVBUG9HULhb6fEl9bYFzxPL1EGm8GBfu6aATH8+fGqirPwsTSAp/P+FQ0rhP6fkpCA1N/XVunzkps+CiEGhkCIvuWXFWGYMkUhUga+85mG4ISKqaCqVOsgU7lZ6fEm9I/bzJHLArslzQaYJIFVUZcW1Sp4q71mXTzFhpUj3T7b6v3OUOS99T0OpYYktJiWFFFO5abXs8qkJ+K4REb0KlQp4Evk0yIQ+bZW5DOQUF2QU6q6kwnst2TUBpPIqLlo/qAQV0nLTSg4m2UXHqCTnJCMjL+Ol7ykVSUsMJiV2/yjMoZAwrFYlhptaTCQSYf/+/ejbt69O6+jUqRN8fHywdOlSndZB9EIqFZAYob0YXsxlICe16LkShXqWUuHBvnaNAYmsSkuuCQRB0O7yea7FpKRuoNTcVKgE1UvdUwTRsy6f54JJwXiV57uDzOXm7PKpIRhuqMKpVCrs3r0bP/30Ey5duoSsrCy4uLigV69emDx5MqytuSMw1QAqpTrIFB7sG3MZyC1moKfUQL1uTOEgY9uwVgaZPGXes1aTUoLJ88dyVbkvfU9DqWHxIUX+NKQYaHcFFXT5SMSSCnzlVJ0w3FCFSkhIwDvvvIP79+9j4sSJ+Pjjj2FlZYXIyEhs374dTZo0wf79+9G2bVtdl0r0jEoJJNx6LsiEA8V1X8iMngWZgkXxbBoCEv36cVrQ5VPSQNmSjmXmZ770PaViaZmCiVbLisIccgm79Uibfv1rrMU6deoELy8vGBgYYP369ZDL5Rg3bpxmF+/bt29j1KhRuHDhAurXr48ffvihyDUePHiAjz/+GEePHkVOTg4aN26MlStXIiAgAADw1VdfYdmyZcjKysKAAQNgY2ODI0eOICwsDACQn5+PPn36oHHjxvjjjz8gkz37rbVZs2bo06cPDh06hH79+uHs2bOoX79+sa/l0KFDGDRoEFatWoXBgwdX7BtFpMwHEm5qz1qKCQfyivlQlhmp91YqPNjXxhOoQb/xF3T5vCikaFagLejyyUmFgJfbV1kE0bOpyCUMni0upBhJjdjlQxWC4eYFCn4w6IKh1LBc/9C3bNmCadOm4fz58zh79iyGDx+OwMBAvP7663j77bdhb2+P8+fPIyUlBVOnTtV6bnp6Ojp27AgnJyf8+uuvcHBwQEhICFQqdX/2tm3b8PXXX2PVqlUIDAzEzp07sXjxYri5uWmusWHDBohEIqxduxb5+fmYPHky9u3bB1tbW0yZMgWLFy/G1atXMXbsWMydOxdbtmwp8hq2b9+OcePGYfv27ejVq9fLvXFEBZR5QPxN7VlLMVeA4v5Ny02KBhlr92oVZPKUecWGFK1gkq0dUlJyUpCnynvpexpJjco1eNZCYQETmQm7fEinGG5eICs/CwHbA3Ry7/ODzsNIVvaVRr28vPDFF18AADw8PLBixQoEBQVBEATcuHEDR48eRZ06dQAACxYsQI8ePTTP3b59O+Lj4/HPP//AysoKAODu7q55fPny5Rg1ahRGjBgBAJgzZw6OHTum2eUbAH788UfMnDkTEokECxYswLFjx7Bt2zYIgoAJEyYgK0v9gVIQup63cuVKzJo1C7/99hs6duxY5tdNBEC9jkzCLeD++WdBJvYqkF/Miq1y00L7LD2dtWTtDojFVVKqUqXU6vIp2Hzw+WDy/OJur/KLlkwsK9vg2YLHDNRdQ+zyoZqI4UaPeHl5af3Z0dERcXFxuH79OpydnTXBBgDatGmjdW5YWBh8fX01weZ5N2/exIQJE7SOtWrVCn/99Zfmz+Hh4ZqxNL/99hu++OILdOrUCQDw+eefY9asWZq6kpK0Fy7bs2cP4uLicPr0abRs2bIcr5pqLUFQt8rcPQXcDQbunS5+ryWF2dMA83RVX0cfwKp+hQQZQRCQmZ9ZYpdPSSElLTftpbt8xCKxZjxK4WCiGaOisIC5gTnM5dotK+VtCSaqyRhuXsBQaojzg87r7N7lUXiMC6Ce6l3QrfTCexm++hLu+fn5muvk5ubC2PjZRn0mJs92Hw4JCdFqFQIAX19fhISEYOPGjWjRogV/CFNRKhUQf+NpkAkG7p4GMhO0z5EaAHVbqkNMQfeSpVuZgkyuMrfELp/nl8YvPEYlX5X/0i/JWGZcJJg8Pw7l+e4fU7kpxKKqaWEiqqkYbl5AJBKVq2uoOmrcuDHu37+Px48fw9HREQBw7tw5rXO8vLywfv16PHnypNjWm4YNG+Kff/7B0KFDNcf++ecfrXPc3d0RHh6OVq1aoV27dvjhhx/QoUMHANAMYL569SrGjx+Pjz/+WOu5DRo0wOLFi9GpUydIJBKsWLHi1V841WwqFRB3Td0ic/eUOsw8v+eS1BCoFwC4tANc2wFOflCKpZpunpScFKQ8PFVyy0qhqcqv0uUjF8tfGEyKPCY3h6wWThUnqgoMN7VAly5d4OnpiWHDhmHRokVITU3VdBEVGDhwIBYsWIC+ffti4cKFcHR0RGhoKOrUqYM2bdpg8uTJGDNmDFq0aIG2bdti165duHz5staMp379+mHlypVo1aoVvvzyS/Tt2xfW1tYwNDTElClTcPz4cbzxxhuYPXs2hg8fXqROT09PHD9+HJ06dYJUKuWifrWNSgXEXVW3zBR0Mz2/75LMCHAOUAcZ1/YQHH3wIDsOoXGhCHl4FGGh3yAyJfKVunwKxp2UZYZPQasLu3yIqheGm1pALBZj//79GDVqFFq1agVXV1csW7YMb7zxhuYcuVyOY8eO4aOPPsKbb76J/Px8NGnSBCtXrgQADB48GJGRkZg+fTqys7PRv39/DB8+HBcuXNBcY+rUqWjevDnWr1+P0aNHIzg4GPHx8TA2NoZcLsfUqVNhb29faq0NGzbEX3/9pWnBWbx4ceW8KaR7KiUQe+VpmDmtDjPZydrnyIyBeq2fhpl2yHdojpupkQiNDUVo5M8IPTcT8VnFjLMBYCIzKRpS5OawMNBuPSkcWtjlQ6QfRIIgvNyvODVUamoqzM3NkZKSAjMzM63HsrOzERUVBTc3NxgYcPfdF+natSscHBywdetWzbHQ0FD07NkTnTt3xpQpU+Dn5wepVIrbt29j+fLliImJwc8//6zDql8OvzcqgEqpXuH37umnLTNniu6/JDcB6rUBXAMB1/bItPHApSfXEBYXhpC4EFyKv1Sk+0gqlqKJdRP42fnB184XTa2bwsrQCjIxu3yI9Elpn9/PY8sNlUlmZibWrFmD7t27QyKRYMeOHfjzzz/xxx9/aJ3n6+uLsLAwfP311+jduzcSEhIgFothZmaGAQMGYNmyZTp6BVTllPlPw8zTbqbos0X3YJKbAi5tNC0z8WZ1EJoYru5mCluEm09uQikotZ5iKjOFt523Jsw0s2kGAykDJxE9w5abQvjbecmysrLQu3dvhIaGIjs7Gw0bNsTnn3+Ot99+u8TnCIKA+Ph45Ofnw8HBAeIqWkOkMvB7owyU+eoVf++eUncx3TsL5KZpn6MwA1zaAq7toKrXBneNzBCScFkdZmJD8CD9QZHLOho7wtfOF352fvCx84G7hTsXiCOqhdhyQxXO0NAQf/75Z7meIxKJYGdnV0kVkc4p89SL5d0raJk5V3RTSQNzwCUQcAlEbr0AXJOKEZJwST1mJngXUp7rlhJBBE9LT/ja+Wq+HE0cq+41EZFeYLghorJR5gGPQp8tmhd9vujGkgYW6jDj2g4pTr64hGyExF9CaNw/uHJ8U5Gdnw0kBmhu2xw+tj7ws/eDt603TOWmVfeaiEgvMdwUo5b11FEZ1Mrvifxc4FHIszVm7p8vurmkoSXgEgjBpR0eOTREiDIdofFhCI37AxG3Vhe5pJWBlSbI+Nr5orFVY671QkQVjuGmkIIVfjMzMytkxV7SH5mZ6g/151eB1iv5OcDDi09nM50C7l8ousGkkTXgEgilS1vcsnZGSF6yOsw82IO4W3FFLuli5qIZL+Nr5wsXMxeuB0NElY7hphCJRAILCwvExal/SBsZGfEHcS0nCAIyMzMRFxcHCwsLSCR6NJA1Lxt4+O+zMPPgn6KbTBrZAK6ByKzXGuHmtgjNSUBofBguRW5Bxk3tLimpSIrG1o01Ycbbzhs2hjZV+IKIiNQYbp7j4OAAAJqAQwQAFhYWmu+NGisvWx1gClb/vX8BUOZon2NsC7i2Q0JdX4QZWyAkOxahcaG4fmttkSnZxjJj+Nj6qMOMvR+a2TQr935oRESVgeHmOSKRCI6OjrCzs0NeXp6uy6FqQCaT1cwWm7wsdYC593TRvAf/Fg0zJvYQXAJx17EJQo2MEZL5CKFxoYi+tb7I5eyM7OBv5w9fe3XLDKdkE1F1xXBTAolEUjM/0Kj2ys1UD/otCDMPLwJK7dlJMHFAnktbXLN3R6hCjpCM+wiLC0PSnX+1ThNBBHdLd81YGV87XzgaO7KblohqBIYbopoqN0MdZgr2Znp4EVA919poWgepLq1xyaYeQmVihKRF4UpCOHKitMOMXCxHc9vmmoXyvG29Ya4wr8IXQ0RUcapFuFm5ciUWLVqEmJgYeHt7Y/ny5WjVqlWx53bq1Al///13keNvvvkmDh06VNmlEulOTjpw/9yzvZkehQCqfO1zzJzwuF4rhFg5IFSsQmjqHdxOugghQzvMWCgstBbKa2LdBHKJvApfDBFR5dF5uNm1axemTZuGNWvWICAgAEuXLkX37t1x8+bNYle33bdvH3JznzW1JyYmwtvbG++++25Vlk1U+XLS1AvlFSya9ygUeG5Qr9LcGRHOvggxt0WoKAehSbcQk3EReG5tvXqm9eBj56PuZrL3hZuZG7uYiEhv6XxvqYCAALRs2RIrVqwAAKhUKjg7O2Py5Mn49NNPX/j8pUuXYs6cOXj8+DGMjY1feH559qYgqlLZqeotDAr2ZnoUViTMZFnUw5W6zRFqYokQIQOXkm4iPU97ywOJSIJGVo00s5h87Xw5JZuIarwas7dUbm4uLl68iJkzZ2qOicVidOnSBWfPni3TNTZs2ID33nuvxGCTk5ODnJxnM0RSU1OLPY+oymWnqDeXLNib6fElQFBpnfLEygWhjo0RamyK0PxkXEu+g/yMcK2WGSOpEbxtvTWzmJrbNIeRzKiKXwwRUfWh03CTkJAApVIJe3t7reP29va4cePGC59/4cIFXLlyBRs2bCjxnIULF2Lu3LmvXCvRK8tKBqLPPh0AfAqICdcKMwKAezZuCLVzR6ihAUJzE3A3/SGQeQ0otOuBnaEdfO19NYvleVh6QCrWeQ8zEVG1UaN/Im7YsAHNmzcvcfAxAMycORPTpk3T/Dk1NRXOzs5VUR7VdplPCoWZYHWYwbNe4DwAN2zdEGLrilC5FKFZMXiSmwJk3QQK7XrgbuGuNfjXycSJ42WIiEqh03BjY2MDiUSC2NhYreOxsbEvXA02IyMDO3fuxLx580o9T6FQQKFQvHKtRC+U+eTpGjNPZzPFXkHhMJMuEuGSbQOEWDshVAqEZz1GtjIHyLqjCTMysQzNbZprgoyPnQ+nZBMRlZNOw41cLoe/vz+CgoLQt29fAOoBxUFBQZg0aVKpz929ezdycnIwZMiQKqiUqBgZic8WzLsbDMRd1Xo4RiJBqK0rQizsESbOx62sOKiQB2Tf1ZxjrjCHr62vppupiXUTKCQM40REr0Ln3VLTpk3DsGHD0KJFC7Rq1QpLly5FRkYGRowYAQAYOnQonJycsHDhQq3nbdiwAX379oW1tbUuyqbaKD3+WZi5dxqIu6Z5SAUgQiZDqE09hJrbIFTIwqPcZAB5QM4DzXl1TerCz95PMy3bzdwNYpG4yl8KEZE+03m4GTBgAOLj4zFnzhzExMTAx8cHR44c0Qwyjo6Ohlis/cP/5s2bCA4OxrFjx3RRMtUW6XHPgszdYCD+2SD3bJEIVwwUCLN2RoixGcJU6UhTZkMdZh4DAMQisWZKdsGXnVHRtZuIiKhi6Xydm6rGdW6oRGmxz6Zl3w0GEm5pHkoSixFmoECoZR2EGBnjqjIN+c+tQWMoNYSXrZdmPyYvWy8Yy1689hIREb1YjVnnhkinUh8/bZU5pR4EnHgbgHoI8AOpFCEmxgi1dECIQo4oZcHCMnlAfjIAwMbQRjMd29feFw0tG3JKNhFRNcCfxFR7pDzUHgD85A4AIB/ATbkcIWamCLWwQ4hMjERVwcKPeYBSvRllffP6Wqv+1jWpyynZRETVEMMN6a+UB0+nZT/dmykpCgCQIRLhkkKBUEsLhJpa4bJUQJamiykPUAFSsRTNrJtpVv31sfWBhYGFzl4KERGVHcMN6Y/k6GdrzNwLBpLuAgDiJBKEGCgQam2JUBML3BQr8Wxd4HxAAEzlploDf5taN4WB1EBHL4SIiF4Fww3VXEn3Cs1mOgUkR0MFIFImQ4iBAmG2NggxNsFDUeH9mtQtNE4mTlphpoFFA07JJiLSEww3VDMIgrolpvCYmZT7yBEB1+RydcuMvS1CDY2QKio8AVAFsUiMhpYNNWvL+Nj5wMG49BWwiYio5mK4oepJENRjZAqCzN3TQOoDpIjFCFM8DTOO9riiUCBPa0yvAEOpoWYLAz87P3jZesFEbqKrV0JERFWM4YaqB0EAnkQ+m5Z9NxhC2iM8lEoQaqBAiEKBUCdH3JHLijzVysBKs7aMn70fGlo1hExc9DwiIqodGG5INwQBSIwo1DITjPz0GNySy9RhxkCBUAsnxEslRZ7qauaqmY7tZ+cHZ1NnTskmIiINhhuqGoIAJNx+Ni373mlkZsThskKuDjMmCly2rovM57bakIqlaGLdRNMy42PnAysDKx29CCIiqgkYbqhyCAIQf1MdZu6dBu6eRnx2AkIVCoQaKBBqpsANm7pQPtfiYiozhbedtybMNLNpxinZRERULgw3VDFUKvXGkk+nZQt3TyMqL1ndvaRQINRSgfuyukWe5mjsqOle8rHzgbuFOyTiol1RREREZcVwQy9HpQLirz8dL3MKuffO4JoyTTP4N8xGgWRJHa2niCCCp6Wn1voyjiaOOnoBRESkrxhuqGxUKiDuqmbwb0r0aVwSsjSDf6/YGiFXrL0DtoFEgea2XvCx9YGfvR+8bb1hKjfV0QsgIqLaguGGiqdSArFXgLunIUSdwuOHZxGCHE2YibA3BaAdVKwUlvAptLFkY6vGkEk4JZuIiKoWww2pqZRATDhwNxjKqFO4/fg8QkR5mjATZ1c0zLiY1oOvvZ9m8K+LmQunZBMRkc4x3NRWynwg5jJwNxiZd0/hSsy/CJHkI1ShwCUDBTJstYOMVCRBY6vGmjDjbecNG0MbHRVPRERUMoab2kKZDzy+BNwLRkLUCYTFhSFUqkKoQoHrCjnybbS3JzCWGMDHzk8dZuz90MymGQylhjoqnoiIqOwYbvSVMg94fAlC1EncvXscYYlXECIFQg0UuCeTAdbaYcZOYQl/xwBNywynZBMRUU3FcKMvlHnAo1DkRZ3A9bsnEJp0AyEyEcIMFHgikQCWz8KMCIC7iTP86rSGr72/ekq2sSPHyxARkV5guKmp8nOBRyFIi/wLl+6dQEjqHYTKxAhXyJEjFgMWz6Zly0USNLfwgJ9TIHyeTsk2V5jrsHgiIqLKw3BTU+TnAA8vIibiGEIenERIWjRC5RLclssgiESAmZHmVAuJAXytm8G3bnv42vuhiXUTyCVyHRZPRERUdRhuqqv8HCjvX0DE7UMIfXwWIRmPEKqQIkb69K/M9Nng3noyc/VCefU6wdfBD25mbuxiIiKiWovhprrIy0b2vdMIjziI0Jh/EZIdi8tyGdIkT3fJNlZvHikB0MjADr4OLeDn8jp87f04JZuIiKgQhhtdycvCkztBCL1zGKHxYQjNTcQ1uQz5IpF6xK+hAgBgBDG8jevCt04b+Ll2QXNbLxjJjEq/NhERUS3GcFNFhJwMREccQsidIwhLvIoQZSruyp6+/SIACvWYGFuRDH6mburxMm5d4WnVEFIx/5qIiIjKip+alSQvOwU3buxD6N0ghCbdRIiQoZ6SDQBiAE8Di7vYEL7mHvCt9xp863eHk2ldjpchIiJ6BQw3FSQzIx7rfv8f8pWhuJZ5F+HIRpb46XgZMQBIIBMENJeawceyMfzcusKn/hswN7DQYdVERET6h+GmguwIWon1GXvVfxCr/8dMJcBXZgVfWy/4NeiBJq6vQyE10GWZREREeo/hpoK09x2IXUf3wDbLBMosVzSv3wcfv/Uu5DKZrksjIiKqVcS6LkBfeDo3xKERl9HIbSPOPXkf6/41x9BN/yIuLVvXpREREdUqDDcVSCYRY3avJlg5yA/GcgnORT5Br2XBuBD1RNelERER1RoMN5Wgp5cjfp3cDp72JohLy8HAdeew9uQdCIKg69KIiIj0HsNNJWlga4JfJgain68TlCoBCw7fwLifLiI1O0/XpREREek1hptKZCSX4vv+3viqbzPIJWIcvRqLPsuDcf1xqq5LIyIi0lsMN5VMJBJhSGsX7B7XBk4WhribmIl+q05jz8UHui6NiIhILzHcVBFvZwscnNwOnRraIjtPhem7L2HmvsvIzlPqujQiIiK9wnBThSyN5dg4rCWmdfWESATsuHAf76w5g/tPMnVdGhERkd5guKliYrEIU173wI8jW8HKWI4rD1PRc9kpBF2P1XVpREREeoHhRkfae9ji4OR28K1ngdTsfIza8i++PXID+UqVrksjIiKq0RhudKiOhSF2jW2D4W1dAQCrTtzB+xsuID4tR7eFERER1WAMNzoml4rxZZ+mWD7QF0ZyCc5GJqLX8lP45y5XNSYiInoZDDfVRG/vOvh1UiDc7UwQm5qD99aew/pTkVzVmIiIqJwYbqoRdztTHJgYiD7edaBUCfjq0HVM3B6CNK5qTEREVGYMN9WMsUKKH97zwby3mkImEeFweAzeWnEaN2K4qjEREVFZMNxUQyKRCEPbuOLnD9qgjrkBIhMy0HflaewL4arGREREL8JwU4351rPEwSnt0cFTvarxtJ8v4bP94VzVmIiIqBQMN9WclbEcm4a3xNQuHhCJgO3no/HumrNc1ZiIiKgEDDc1gEQswtQuntg8ohUsjWQIf5iCXsuDcfxGnK5LIyIiqnYYbmqQjp62ODilPbydLZCSlYcRm//B4mM3oVRxujgREVEBhpsaxsnCED9/0BpD27gAAJb/FYFhGy8gMZ2rGhMREQHVINysXLkSrq6uMDAwQEBAAC5cuFDq+cnJyZg4cSIcHR2hUCjg6emJw4cPV1G11YNCKsG8t5rhh/d8YCiTIDgiAT2XBePivSRdl0ZERKRzOg03u3btwrRp0/DFF18gJCQE3t7e6N69O+Liih9Lkpubi65du+Lu3bvYs2cPbt68iXXr1sHJyamKK68e3vJxwq+TAtHA1hgxqdkY8L+z2BgcxVWNiYioVhMJOvwkDAgIQMuWLbFixQoAgEqlgrOzMyZPnoxPP/20yPlr1qzBokWLcOPGDchkspe6Z2pqKszNzZGSkgIzM7NXqr+6SM/Jx6d7L+Pg5ccAgJ5ejvjmP14wUUh1XBkREVHFKM/n9yu13GRnZ7/0c3Nzc3Hx4kV06dLlWTFiMbp06YKzZ88W+5xff/0Vbdq0wcSJE2Fvb49mzZphwYIFUCpr97ovJgoplg/0xZe9m0AqFuHQ5cfosyIYt2LTdF0aERFRlSt3uFGpVJg/fz6cnJxgYmKCyMhIAMDs2bOxYcOGMl8nISEBSqUS9vb2Wsft7e0RExNT7HMiIyOxZ88eKJVKHD58GLNnz8bixYvx1VdflXifnJwcpKaman3pI5FIhOGBbtj1QRs4mhsgMj4Db604jV9CH+q6NCIioipV7nDz1VdfYfPmzfj2228hl8s1x5s1a4b169dXaHHPU6lUsLOzw9q1a+Hv748BAwZg1qxZWLNmTYnPWbhwIczNzTVfzs7OlVqjrvm7WOLg5HZo526DrDwlpu4Kw+xfriAnv3a3bhERUe1R7nDz448/Yu3atRg8eDAkEonmuLe3N27cuFHm69jY2EAikSA2NlbreGxsLBwcHIp9jqOjIzw9PbXu27hxY8TExCA3N7fY58ycORMpKSmar/v375e5xprK2kSBLSNbYUpndwDA1nP30H/NWTxI4qrGRESk/8odbh4+fAh3d/cix1UqFfLy8sp8HblcDn9/fwQFBWldIygoCG3atCn2OYGBgYiIiIBKpdIcu3XrFhwdHbVakQpTKBQwMzPT+qoNJGIRpnVriE0jWsLCSIZLD9SrGp+4yVWNiYhIv5U73DRp0gSnTp0qcnzPnj3w9fUt17WmTZuGdevWYcuWLbh+/TrGjx+PjIwMjBgxAgAwdOhQzJw5U3P++PHj8eTJE3z44Ye4desWDh06hAULFmDixInlfRm1xmsN7XBwcjt41TVHcqZ6VePv/7jFVY2JiEhvlXuu8Jw5czBs2DA8fPgQKpUK+/btw82bN/Hjjz/i4MGD5brWgAEDEB8fjzlz5iAmJgY+Pj44cuSIZpBxdHQ0xOJn+cvZ2RlHjx7F//3f/8HLywtOTk748MMPMWPGjPK+jFqlrqURdo9rg/kHr+Gnc9FYFnQbodFJ+OE9X1gZF9/iRUREVFO91Do3p06dwrx583Dp0iWkp6fDz88Pc+bMQbdu3Sqjxgqlj+vclMf+0Af4bN8VZOUpUcfcACsH+8G3nqWuyyIiIipVeT6/dbqIny7U9nADADdj0jD+p4uITMiATCLC5z2bYGgbF4hEIl2XRkREVKwqW8SPaqaGDqY4MCkQbzZ3QJ5SwBe/XsWUnWHIyMnXdWlERESvrNzhRiwWQyKRlPhFNYOpgQwrB/lhdi/1qsa/XXqEt1aeRkQcVzUmIqKardwDivfv36/157y8PISGhmLLli2YO3duhRVGlU8kEmFUOzd41zXHxO0hiIhLR58Vp/Hf/3ihj3cdXZdHRET0UipszM327duxa9cuHDhwoCIuV2k45qZ4Cek5mLIjFGfuJAIAhrVxwayeTSCXsueSiIh0Tydjblq3bq21IB/VLDYmCmwdFYBJr6kXaNxy9h76/+8sHiVn6bgyIiKi8qmQcJOVlYVly5bBycmpIi5HOiIRizC9e0NsGNYCZgZShN1PRs9lp3DyVryuSyMiIiqzco+5sbS01JoyLAgC0tLSYGRkhJ9++qlCiyPdeL2xPQ5NaY/x2y7iysNUDNt0AVNf98Tkzu4QizldnIiIqrdyj7nZvHmzVrgRi8WwtbVFQEAALC2r/2JwHHNTdtl5Ssz97Rp2XIgGAHT0tMXSAT6w5KrGRERUxbiIXykYbspv78UHmPVLOLLzVHCyMMTKwX7wcbbQdVlERFSLVHi4uXz5cplv7uXlVeZzdYHh5uVcf5yKCdtCEPV0VeM5vZpgSGuuakxERFWjwsONWCyGSCTCi04ViURQKpXlq7aKMdy8vNTsPHyy+zKOXI0BAPT1qYMFbzeHkbzcQ7eIiIjKpTyf32X6VIqKiqqQwqhmMzOQYfUQP6w/FYX/HrmBX8Ie4eqjVKwe4g93OxNdl0dERASAY250XU6NdSHqCSZtD0FcWg6M5RJ8+443eno56rosIiLSU1UyoPjatWuIjo5Gbm6u1vE+ffq8zOWqDMNNxYlLy8aUHaE4F/kEADAi0BUzezTmqsZERFThKjXcREZGol+/fggPD9cah1MwsJRjbmqXfKUKi/+4hdUn7gAA/OpZYOVgPziaG+q4MiIi0ieVuv3Chx9+CDc3N8TFxcHIyAhXr17FyZMn0aJFC5w4ceJla6YaSioRY8YbjbBuaAuYGkgREp2MXsuCEXw7QdelERFRLVXucHP27FnMmzcPNjY2EIvFEIvFaNeuHRYuXIgpU6ZURo1UA3RtYo+Dk9uhiaMZEjNy8f7G81jx122oVLVqSBcREVUD5Q43SqUSpqamAAAbGxs8evQIAODi4oKbN29WbHVUo7hYG2PfhLYY0MIZggB8d+wWRm35B8mZuS9+MhERUQUpd7hp1qwZLl26BAAICAjAt99+i9OnT2PevHmoX79+hRdINYuBTIJv3vHCt+94QSEV4/jNePRcFozLD5J1XRoREdUS5Q43n3/+OVQqFQBg3rx5iIqKQvv27XH48GEsW7aswgukmql/C2fsm9AWLtZGeJichXdWn8W28/deuBAkERHRqyrzbKkWLVpg9OjRGDRoUJFRyk+ePCmyW3h1xdlSVSslKw8f776EY9diAQBv+zrh637NYSiX6LgyIiKqSSpltpS3tzc++eQTODo6YujQoVozo6ysrGpEsKGqZ24ow//e98fMHo0gEYuwL/Qh+q48jcj4dF2XRkREeqrM4WbDhg2IiYnBypUrER0djddffx3u7u5YsGABHj58WJk1Ug0nEonwQccG2DY6ALamCtyMTUOfFafxe/hjXZdGRER66KVXKL5z5w42bdqErVu34tGjR+jWrRtGjRqFt99+u6JrrFDsltKtuNRsTNoRigtR6lWNR7Vzw6c9GkEm4arGRERUsirZfqGAIAjYu3cvPvjgAyQnJ3OFYnqhfKUKi47exP9ORgIAWrhYYsUgPziYG+i4MiIiqq4qdYXiwk6cOIHhw4dj+PDhUCqVGDNmzKtcjmoJqUSMmW82xv/e94epQop/7yWh1/JTOHOHqxoTEdGrK3e4efDgAb766iu4u7ujc+fOuHv3LlatWoXHjx9jzZo1lVEj6anuTR3w2+R2aORgioT0XAxZfx4rj0dwVWMiInolZe6W+vnnn7Fx40YEBQXBzs4Ow4YNw8iRI+Hu7l7ZNVYodktVP9l5Ssz+5Qp2X3wAAHi9kR2+7+8DcyOZjisjIqLqolLG3MjlcvTs2ROjRo3Cm2++CbG4Zg4AZbipvnb9E43ZB64iN18FZytDrB7sj2ZO5roui4iIqoFKCTdxcXGws7OrkAJ1ieGmervyMAXjt13E/SdZkEvFmNenKQa0dOY6SkREtVylDCjWh2BD1V8zJ3McnNQeXRrbITdfhU/3hePjPZeRlVu9Z+EREVH1UTP7lkivmRvJsPb9FvjkjYYQi4A9Fx+g36rTuJuQoevSiIioBmC4oWpJLBZhQid3/DQ6ADYmctyISUPv5cE4ciVG16UREVE1x3BD1VrbBjY4NKU9WrpaIi0nH+N+uogFh68jX6nSdWlERFRNlTvc/PPPPzh//nyR4+fPn8e///5bIUURFWZvZoDtY1pjTHs3AMDak5EYtO484lKzdVwZERFVR+UONxMnTsT9+/eLHH/48CEmTpxYIUURPU8mEWNWzyZYPdgPJgopLtx9gjeXBeNcZKKuSyMiomqm3OHm2rVr8PPzK3Lc19cX165dq5CiiErSo7kjfp0U+HRV4xwMWncOq0/cwStukUZERHqk3OFGoVAgNja2yPHHjx9DKpVWSFFEpalva4L9EwLxtp8TVALwzZEbGLv1IlKy8nRdGhERVQPlDjfdunXDzJkzkZKSojmWnJyMzz77DF27dq3Q4ohKYiiXYPG73ljQrznkEjH+uBaLPiuCcfVRyoufTEREeq3MKxQXePjwITp06IDExET4+voCAMLCwmBvb48//vgDzs7OlVJoReEKxfon/IF6VeMHSVlQSMWY/1Yz9G9Zvb8PiYiofCpl+4XCMjIysG3bNly6dAmGhobw8vLCwIEDIZNV/40OGW70U3JmLqb9fAl/3YgDAPRvURfz3moGA5lEx5UREVFFqPRwU5Mx3OgvlUrA6r/vYPGxm1AJQBNHM6we4gcXa2Ndl0ZERK+owsPNr7/+ih49ekAmk+HXX38t9dw+ffqUr9oqxnCj/05HJGDKjlAkZuTC1ECKxe96o1tTB12XRUREr6DCw41YLEZMTAzs7OwgFpc8BlkkEkGprN4bHDLc1A4xKdmYuD0EF+8lAQA+6FgfH3drCKmEi3ITEdVEFb4ruEql0uwKrlKpSvyq7sGGag8HcwPsHNsaIwPVqxr/7+9IDNlwHnFpXNWYiEjflevX2Ly8PLz++uu4fft2ZdVDVGFkEjHm9G6ClYP8YCyX4FzkE/RaFowLUU90XRoREVWicoUbmUyGy5cvV1YtRJWip5cjfp3cDp72JohLy8HAdeew9iRXNSYi0lflHoAwZMgQbNiwoTJqIao0DWxN8MvEQPTzdYJSJWDB4RsY99NFpGZzVWMiIn1T7v0S8vPzsXHjRvz555/w9/eHsbH2NNvvv/++woojqkhGcim+7+8NPxdLzP/tGo5ejcXNmGCsHuKPxo4cXE5EpC/Kvc7Na6+9Vurjx48ff6WCKhtnSxEAXLqfjAnbQvAwOQsGMjG+6tsc7/jX1XVZRERUAi7iVwqGGyqQlJGLqbvC8PeteADAwFbO+KJ3U65qTERUDVX4VPDCRo4cibS0tCLHMzIyMHLkyPJeDgCwcuVKuLq6wsDAAAEBAbhw4UKJ527evBkikUjry8DA4KXuS7WbpbEcm4a3xLSunhCJgB0X7uOdNWdw/0mmrksjIqJXUO5ws2XLFmRlZRU5npWVhR9//LHcBezatQvTpk3DF198gZCQEHh7e6N79+6Ii4sr8TlmZmZ4/Pix5uvevXvlvi8RAIjFIkx53QM/jmwFSyMZrjxMRc9lpxB0PVbXpRER0Usqc7hJTU1FSkoKBEFAWloaUlNTNV9JSUk4fPiwZqG/8vj+++8xZswYjBgxAk2aNMGaNWtgZGSEjRs3lvgckUgEBwcHzZe9vX2570tUWHsPWxya0h6+9SyQmp2PUVv+xbdHbiBfqdJ1aUREVE5lDjcWFhawsrKCSCSCp6cnLC0tNV82NjYYOXIkJk6cWK6b5+bm4uLFi+jSpcuzgsRidOnSBWfPni3xeenp6XBxcYGzszPeeustXL16tcRzc3JytIJYampquWqk2qOOhSF2jW2D4W1dAQCrTtzB+xsuID4tR7eFERFRuZR5Kvjx48chCAI6d+6MvXv3wsrKSvOYXC6Hi4sL6tSpU66bJyQkQKlUFml5sbe3x40bN4p9TsOGDbFx40Z4eXkhJSUF3333Hdq2bYurV6+ibt2is10WLlyIuXPnlqsuqr3kUjG+7NMUfi6W+HTvZZyNTESv5aewYpAfWrpavfgCRESkc+WeLXXv3j3Uq1cPIpHolW/+6NEjODk54cyZM2jTpo3m+CeffIK///4b58+ff+E18vLy0LhxYwwcOBDz588v8nhOTg5ycp795p2amgpnZ2fOlqIXiohLw7ifQhARlw6JWISZPRphVDu3CvneJyKi8qnU2VIuLi4IDg7GkCFD0LZtWzx8+BAAsHXrVgQHB5frWjY2NpBIJIiN1R68GRsbCwcHhzJdQyaTwdfXFxEREcU+rlAoYGZmpvVFVBbudqY4MDEQfbzrQKkS8NWh65iwLQRpXNWYiKhaK3e42bt3L7p37w5DQ0OEhIRoWkVSUlKwYMGCcl1LLpfD398fQUFBmmMqlQpBQUFaLTmlUSqVCA8Ph6OjY7nuTVQWxgopfnjPB/PeagqZRITfr8Sgz4rTuBHDsVtERNVVucPNV199hTVr1mDdunWQyWSa44GBgQgJCSl3AdOmTcO6deuwZcsWXL9+HePHj0dGRgZGjBgBABg6dChmzpypOX/evHk4duwYIiMjERISgiFDhuDevXsYPXp0ue9NVBYikQhD27ji5w/aoI65AaISMtB35WnsC3mg69KIiKgY5d5b6ubNm+jQoUOR4+bm5khOTi53AQMGDEB8fDzmzJmDmJgY+Pj44MiRI5pBxtHR0RCLn2WwpKQkjBkzBjExMbC0tIS/vz/OnDmDJk2alPveROXhW88SB6e0x4c7Q3HqdgKm/XwJ/95LwpxeTbiqMRFRNVLuAcX169fH2rVr0aVLF5iamuLSpUuoX78+fvzxR/z3v//FtWvXKqvWCsHtF+hVKVUClgXdxrK/bkMQgOZO5lg12A/OVka6Lo2ISG9V6oDiMWPG4MMPP8T58+chEonw6NEjbNu2DdOnT8f48eNfumiimkIiFuH/unpi0/CWsDCSIfxhCnotD8bxGyWvqk1ERFWn3C03giBgwYIFWLhwITIz1XvwKBQKTJ8+vdip2NUNW26oIj1MzsKEbSG4dD8ZADC5szumdvGERMzp4kREFalKdgXPzc1FREQE0tPT0aRJE5iYmLxUsVWN4YYqWk6+El8fuo4fz6r3OGvnboMf3vOBtYlCx5UREemPKgk3NRXDDVWWA2EP8enecGTlKeFgZoCVg/3g72Kp67KIiPRCpYSbkSNHlunmpW14WR0w3FBluhWbhnE/XURkfAakYhE+e7MxRgS6clVjIqJXVCnhRiwWw8XFBb6+vijtKfv37y9ftVWM4YYqW3pOPmbsvYxDlx8DAHp6OeKb/3jBRFHulReIiOip8nx+l/mn7fjx47Fjxw5ERUVhxIgRGDJkiNbmmUSkZqKQYsVAX7RwscTXh67j0OXHuP44FWuG+MPT3lTX5RER6b0yTwVfuXIlHj9+jE8++QS//fYbnJ2d0b9/fxw9erTUlhyi2kgkEmFEoBt2fdAGDmYGiIzPwFsrTuOX0Ie6Lo2ISO+99IDie/fuYfPmzfjxxx+Rn5+Pq1ev1ogZU+yWoqqWmJ6DD3eGITgiAQDwfmsXfN6rMRRSrmpMRFRWlbqIn+aJYjFEIhEEQYBSqXzZyxDpPWsTBbaMbIUpnd0BAFvP3UP/NWfxIClTx5UREemncoWbnJwc7NixA127doWnpyfCw8OxYsUKREdH14hWGyJdkYhFmNatITYNbwlzQxkuPVCvanziJlc1JiKqaGXulpowYQJ27twJZ2dnjBw5EoMHD4aNjU1l11fh2C1Funb/SSYmbg/B5QcpEImAyZ098OHrHlzVmIioFJU2FbxevXrw9fUtdc2Offv2la/aKsZwQ9VBTr4S8367hm3nowEA7T1s8MN7vrAyluu4MiKi6qlSpoIPHTqUC5ERVRCFVIKv+zWHv4slPtsfjlO3E9Bz2SmsHOwHv3pc1ZiI6FVw+wUiHbsZk4bxP11EZEIGZBIRZr3ZGMPaclVjIqLCqmS2FBFVjIYOpjgwKRBvNndAnlLAl79dw5SdYcjIydd1aURENRLDDVE1YGogw8pBfpjdqwmkYhF+u/QIb608jYi4NF2XRkRU4zDcEFUTIpEIo9q5YefY1rA3UyAiLh19VpzGr5ce6bo0IqIaheGGqJpp4WqFQ1Pao20Da2TmKjFlRyi+OHAFufkqXZdGRFQjMNwQVUM2JgpsHRWAia81AABsOXsP/f93Fo+Ss3RcGRFR9cdwQ1RNScQifNy9ETYMawEzAynC7iej57JTOHkrXtelERFVaww3RNXc643tcWhKezRzMkNSZh6GbbqAH/68DZWqVq3iQERUZgw3RDWAs5UR9oxri4GtnCEIwJI/b2HE5n+QlJGr69KIiKodhhuiGsJAJsHCt73w3bveUEjF+PtWPHotD0bY/WRdl0ZEVK0w3BDVMO/418UvEwPham2Eh8lZeHfNGWw9exe1bLFxIqISMdwQ1UCNHc3w6+R26N7UHnlKAbMPXMX/7QpDZi5XNSYiYrghqqHMDGRYM8Qfs95sDIlYhF/CHuGtFacREZeu69KIiHSK4YaoBhOJRBjToT52jGkNO1MFbsel460VwTh4masaE1HtxXBDpAdauVnh4JR2aF3fChm5SkzaHoq5v13lqsZEVCsx3BDpCTtTA/w0KgDjOqpXNd50+i7eW3sWj1O4qjER1S4MN0R6RCoR49MejbBuaAuYGkgREp2MnsuCEXw7QdelERFVGYYbIj3UtYk9Dk5uhyaOZniSkYv3N57Hir+4qjER1Q4MN0R6ysXaGPsmtMWAFupVjb87dgujtvyD5EyuakxE+o3hhkiPGcgk+OYdL3z7jhcUUjGO34xHz2XBuPwgWdelERFVGoYbolqgfwtn7JvQFvWs1Ksav7P6LLadv8dVjYlILzHcENUSTeuY47fJ7dC1iT1ylSrM2n8FH/18CVm5Sl2XRkRUoRhuiGoRc0MZ1r7vj097NIJYBOwLfYi+K08jMp6rGhOR/mC4IaplRCIRxnVsgO1jWsPGRIGbsWnos+I0fg9/rOvSiIgqBMMNUS3Vur41Dk9ph1auVkjPycf4bSGYf/Aa8pRc1ZiIajaGG6JazM7MANvHBOCDDvUBABuCozBw7TnEpGTruDIiopfHcENUy0klYsx8szHWDPGHqUKKf+8lodfyUzhzh6saE1HNxHBDRACAN5o54NfJ7dDIwRQJ6bkYsv48Vh6P4KrGRFTjMNwQkYabjTH2TwjEO/51oRKARUdvYsyP/yIlM0/XpRERlRnDDRFpMZRLsOgdL/z37eaQS8UIuhGHXitO4crDFF2XRkRUJgw3RFSESCTCe63qYd/4tnC2MsT9J1l4e/UZ7LgQzVWNiajaY7ghohI1czLHwUnt0aWxHXLzVZi5LxzTd1/mqsZEVK0x3BBRqcyNZFj7fgt83L0hxCJgb8gD9Ft1GlEJGboujYioWAw3RPRCYrEIE19zx0+jA2BjIseNmDT0WR6MI1didF0aEVERDDdEVGZtG9jg4OT2aOFiibScfIz76SIWHL6OfK5qTETVCMMNEZWLg7kBdoxtjdHt3AAAa09GYtC684hL5arGRFQ9MNwQUbnJJGJ83qsJVg/2g4lCigt3n+DNZcE4F5mo69KIiBhuiOjl9WjuiF8nBaKhvSkS0nMwaN05rD5xh9PFiUinqkW4WblyJVxdXWFgYICAgABcuHChTM/buXMnRCIR+vbtW7kFElGJ6tuaYP/Etnjb1wkqAfjmyA2M3XoRKVlc1ZiIdEPn4WbXrl2YNm0avvjiC4SEhMDb2xvdu3dHXFxcqc+7e/cupk+fjvbt21dRpURUEiO5FIv7e2NBv+aQS8T441os+qwIxtVHXNWYiKqezsPN999/jzFjxmDEiBFo0qQJ1qxZAyMjI2zcuLHE5yiVSgwePBhz585F/fr1q7BaIiqJSCTCoIB62DO+DZwsDHEvMRNvrzqDn/+5r+vSiKiW0Wm4yc3NxcWLF9GlSxfNMbFYjC5duuDs2bMlPm/evHmws7PDqFGjXniPnJwcpKaman0RUeXxqmuBQ1Pa4bWGtsjJV+GTvZfxyZ5LyM7jqsZEVDV0Gm4SEhKgVCphb2+vddze3h4xMcUvDhYcHIwNGzZg3bp1ZbrHwoULYW5urvlydnZ+5bqJqHQWRnJsGNYS07t5QiwCfv73Ad5edQb3ErmqMRFVPp13S5VHWloa3n//faxbtw42NjZles7MmTORkpKi+bp/n03kRFVBLBZhUmcPbB0VAGtjOa49TkWv5cE4dpWrGhNR5ZLq8uY2NjaQSCSIjY3VOh4bGwsHB4ci59+5cwd3795F7969NcdUKvXKqFKpFDdv3kSDBg20nqNQKKBQKCqheiIqi0B3Gxyc0g4Tt4UgJDoZY7dexAcd6+Pjbg0hldSo36+IqIbQ6U8WuVwOf39/BAUFaY6pVCoEBQWhTZs2Rc5v1KgRwsPDERYWpvnq06cPXnvtNYSFhbHLiaiacjQ3xM6xbTAi0BUA8L+/IzF4/XnEpXFVYyKqeDptuQGAadOmYdiwYWjRogVatWqFpUuXIiMjAyNGjAAADB06FE5OTli4cCEMDAzQrFkzredbWFgAQJHjRFS9yKVifNG7KfxdLDFjz2Wcj3qCnsuCsXKQH1q5Wem6PCLSIzoPNwMGDEB8fDzmzJmDmJgY+Pj44MiRI5pBxtHR0RCL2XRNpC96edVBIwczTNh2Ebdi0zFw3TnMeKMhxrSvD5FIpOvyiEgPiIRatk56amoqzM3NkZKSAjMzM12XQ1RrZebm47N94fgl7BEAoHtTeyx61xtmBjIdV0ZE1VF5Pr/ZJEJEOmEkl2LJAB/M79sMcokYR6/Gos/yYFx/zLWoiOjVMNwQkc6IRCK839oFu8epVzW+m5iJfqtOY8/FB7oujYhqMIYbItI5b2cLHJzcDh09bZGdp8L03Zcwc99lrmpMRC+F4YaIqgVLYzk2DW+JaV09IRIBOy7cxztrzuD+k0xdl0ZENQzDDRFVG2KxCFNe98CWEa1gaSTDlYep6LnsFIKux774yURETzHcEFG108HTFoemtIePswVSs/Mxasu/+PbIDeQrVboujYhqAIYbIqqW6lgY4ucP2mB4W1cAwKoTd/D+hguIT8vRbWFEVO0x3BBRtSWXivFln6ZYNtAXRnIJzkYmotfyU/jn7hNdl0ZE1RjDDRFVe3286+DXSYFwtzNBbGoO3lt7DutPRaKWrUFKRGXEcENENYK7nSkOTAxEb+86UKoEfHXoOiZsC0Fadp6uSyOiaobhhohqDGOFFMve88HcPk0hk4jw+5UY9FlxGjdiuKoxET3DcENENYpIJMKwtq7Y9UEb1DE3QFRCBvquPI19IVzVmIjUGG6IqEbyq2eJg1Pao72HDbLzVJj28yV8tj+cqxoTEcMNEdVcVsZybB7RCh++7gGRCNh+Phr9Vp3B5tNRuBOfzgHHRLWUSKhl//rLs2U6EdUcJ27GYequMCRnPhtg7GRhiPYeNmjvYYtAd2tYGMl1WCERvYryfH4z3BCR3ohNzca+kIcIjojHP1FJyC20orFIBHjVtUCHp2HHt54FZBI2XhPVFAw3pWC4IaodsnKVOB+ViFO3E3DqdjxuxaZrPW4sl6BNA2u097BFew8buNkYQyQS6ahaInoRhptSMNwQ1U4xKdk4dTsep24nIDgiAU8ycrUed7IwRAfPp11YDWxgbiTTUaVEVByGm1Iw3BCRSiXg2uNUnLwdj1O3EvDvvSfIUz77USgu3IXlaQsfZ3ZhEekaw00pGG6I6HmZufk4H/kEJ2/HI/h2Am7HaXdhmSikaNPAWjNex8XaiF1YRFWM4aYUDDdE9CKPU7KejtVJQPDteCRlam/x4GxliPYetujgYYM2DWxgbsguLKLKxnBTCoYbIioPlUrA1UdPu7Bux+PivaQiXVg+zhbqsONpA++6FpCyC4uowjHclILhhoheRUZOPs5HJeLkLfUsrDvxGVqPmz7twmrvqW7ZcbE21lGlRPqF4aYUDDdEVJEeJmch+HY8Tt5OwOmIBK1FBAGgnpWRZiHBtu7WMDNgFxbRy2C4KQXDDRFVFqVKwJWHKTj1NOyE3EtCvurZj1iJWPS0C0sddrzrmrMLi6iMGG5KwXBDRFUlPScf5+4kqtfXiUhA5PNdWAZSBDawQXtPG3TwsIWzlZGOKiWq/hhuSsFwQ0S68iApE8EFs7AiEpCSpd2F5WptpFkxuU0Da5iyC4tIg+GmFAw3RFQdKFUCwh+m4NQt9arJIdFFu7B8n87Cau9pAy8ndmFR7cZwUwqGGyKqjtKy83Au8olmi4ioBO0uLDMDKQLdbTQtO+zCotqG4aYUDDdEVBPcf5Kp2fTzdEQCUrPztR53szHWDExuXd+KXVik9xhuSsFwQ0Q1Tb5ShcsPU56O14lHSHQylIW6sKRiEfzqWarDjqctmjuZQyLm9hCkXxhuSsFwQ0Q1XWp23tNZWOqwczcxU+txc0MZ2rnboJ2HDdp72KCuJbuwqOZjuCkFww0R6ZvoxEycilDvcH76TgLSnuvCql+4C6uBNUwUUh1VSvTyGG5KwXBDRPosX6nCpQcpmoHJYfeL6cJysdTscN6MXVhUQzDclILhhohqk5SsPJwtWEjwdgKin2h3YVkYyRDoboMOHjZo52ELJwtDHVVKVDqGm1Iw3BBRbXYvMUMzVudMRCLScrS7sBrYGmt2OA9ws4Yxu7CommC4KQXDDRGRmroLK1mzw3nY/WQU6sGCTKKehdXBU722TrM65hCzC4t0hOGmFAw3RETFU3dhJeDk7QScvBWPB0lZWo9barqwbNHOwwZ12IVFVYjhphQMN0RELyYIAu4lZmp2OD97JxHpz3VhuduZoL2HOuwE1LeCkZxdWFR5GG5KwXBDRFR+eUoVwu4n49Qtddi5/KBoF1YLFyvNDudNHM3YhUUViuGmFAw3RESvLiUzD2cKdWE9TNbuwrIylqOdu41mfR0HcwMdVUr6guGmFAw3REQVSxAERCVkIDgiASdvJeDsnQRk5Cq1zvGwM9HscB7gxi4sKj+Gm1Iw3BARVa48pQqh0cma8TqXHySj8CeNXCJGC1dLzQ7n7MKismC4KQXDDRFR1UrKyMWZpwsJnrwVj0cp2VqPWxvLn+6DpQ479mbswqKiGG5KwXBDRKQ7giAgMiEDp26pV0w+G5mIzOe6sBram2p2OG/lagVDuURH1VJ1wnBTCoYbIqLqIzdfhZDoJM32EOEPU7S7sKRitHK10gxMbuRgyi6sWorhphQMN0RE1VdSRi5O30nAqVsJOHk7Ho+f68KyMVE8DTo2aOduAzt2YdUaDDelYLghIqoZBEHAnfgMTavO2TuJyMrT7sJq5GCqadVp5WYFAxm7sPQVw00pGG6IiGqmnHwlQu4la8LOlUdFu7AC3LS7sEQidmHpC4abUjDcEBHph8T0HJy+k6gZnByTqt2FZWuqQHt3G7T3tEGguw3sTNmFVZMx3JSC4YaISP8IgoCIuHScvJ2A4NvxOBf5pEgXVmNHM3R42qrTwtWSXVg1TI0LNytXrsSiRYsQExMDb29vLF++HK1atSr23H379mHBggWIiIhAXl4ePDw88NFHH+H9998v070YboiI9F9OvhIX7yXh1O0EnLodjysPU7UeV0jFCKhvjQ4eNmjnYYOG9uzCqu5qVLjZtWsXhg4dijVr1iAgIABLly7F7t27cfPmTdjZ2RU5/8SJE0hKSkKjRo0gl8tx8OBBfPTRRzh06BC6d+/+wvsx3BAR1T4J6Tk4HZGgCTuxqTlaj9uZKtDu6Q7nge42sDVV6KhSKkmNCjcBAQFo2bIlVqxYAQBQqVRwdnbG5MmT8emnn5bpGn5+fujZsyfmz5//wnMZboiIajdBEHA7Lh0nn47VOR+ViOw8ldY5TRzNNDuc+7uwC6s6KM/nt053LsvNzcXFixcxc+ZMzTGxWIwuXbrg7NmzL3y+IAj466+/cPPmTXzzzTeVWSoREekJkUgET3tTeNqbYnT7+sjOU3dhnbwdj1O3EnDtcarm639/R8JAJkaAmzXae9igg6ctPOxM2IVVzek03CQkJECpVMLe3l7ruL29PW7cuFHi81JSUuDk5IScnBxIJBKsWrUKXbt2LfbcnJwc5OQ8a35MTU0t9jwiIqqdDGQSBLqrZ1TN7AHEp6m7sE4+nXIen5aDv2/F4+9b8cCh67A3U2j2wWrnbgNrE3ZhVTc1cs95U1NThIWFIT09HUFBQZg2bRrq16+PTp06FTl34cKFmDt3btUXSURENZKtqQJ9fZ3Q19cJgiDgZmwaTt1KwKmIBJyPTERsag72XHyAPRcfAACaOZmpw467DfxdLaGQsgtL13Q65iY3NxdGRkbYs2cP+vbtqzk+bNgwJCcn48CBA2W6zujRo3H//n0cPXq0yGPFtdw4OztzzA0REZVbdp4S/95V74V18nYCrj/W7g0wlEkQUN8K7T1s0cHDBu7swqowNWbMjVwuh7+/P4KCgjThRqVSISgoCJMmTSrzdVQqlVaAKUyhUEChYJMhERG9OgOZBO2eTh+fCSAuLVs9C+tWAk7eTkBCeg5O3IzHiZvxAAAHMwPNDuft3G1gZSzX7QuoJXTeLTVt2jQMGzYMLVq0QKtWrbB06VJkZGRgxIgRAIChQ4fCyckJCxcuBKDuZmrRogUaNGiAnJwcHD58GFu3bsXq1at1+TKIiKgWsjM1QD/fuujnWxeCIOBGTJpme4jzUU8Qk5qN3RcfYPfFBxCJgGZ1zDXbQ/i7WEIuFev6JeglnYebAQMGID4+HnPmzEFMTAx8fHxw5MgRzSDj6OhoiMXP/vIzMjIwYcIEPHjwAIaGhmjUqBF++uknDBgwQFcvgYiICCKRCI0dzdDY0QxjOzRAdp4SF6KeaMLOjZg0hD9MQfjDFKw6cQdGcgla17fWhJ0GtsbswqogOl/npqpxnRsiItKFuNRsnLqdgOAI9UKCCem5Wo/XMTdAu6dBp527DSzZhaWlRi3iV9UYboiISNdUKu0urAt3nyA3/9lCgiIR0NzpWReWXz12YTHclILhhoiIqpusXCUu3H2i2eH8Zmya1uNGcgnaFHRhedqivk3t68JiuCkFww0REVV3sU+7sE7djkfw7QQkZmh3YTlZGGpadQLdrWFhpP9dWAw3pWC4ISKimkSlEnDtcaom7Px7Nwm5Su0uLK+6FujwNOz41rOATKJ/XVgMN6VguCEioposMzf/6Swsddi5FZuu9bixXII2Daw1W0S46UkXFsNNKRhuiIhIn8SkZGsGJgdHJOBJMV1YHTyfdmE1sIG5kUxHlb4ahptSMNwQEZG+KujCKtjh/N97T5CnfPYxLy7cheVpCx/nmtOFxXBTCoYbIiKqLTJz83E+8olmh/OIOO0uLBOFFG0aWGvG67hYG1XbLiyGm1Iw3BARUW31KDkLwbcTcPJ2PE5HJCApM0/rcWcrQ82mn20a2MDcsPp0YTHclILhhoiISN2FdfWRugvr5K14hEQnFenC8nG2UIcdTxt417WAVIddWAw3pWC4ISIiKiojJx/noxJx8pZ6Ftad+Aytx02fdmG191S37LhYG1dpfQw3pWC4ISIierGHyVkIvh2Pk7cTcDoiAcnPdWHVszLSLCTY1t0aZgaV24XFcFMKhhsiIqLyUaoEXHmYglNPw07IvSTkq57FB4lY9LQLSx12vOuaV3gXFsNNKRhuiIiIXk16Tj7O3UnUrK8TmaDdheVqbYTj0ztV6Myr8nx+SyvsrkRERFQrmCik6NLEHl2a2AMA7j/JRHDEs72wmte10OmUcrbcEBERUYVRqgSkZedV+Gae5fn8rhnLEhIREVGNIBGLdL5LOcMNERER6RWGGyIiItIrDDdERESkVxhuiIiISK8w3BAREZFeYbghIiIivcJwQ0RERHqF4YaIiIj0CsMNERER6RWGGyIiItIrDDdERESkVxhuiIiISK8w3BAREZFekeq6gKomCAIA9dbpREREVDMUfG4XfI6XptaFm7S0NACAs7OzjishIiKi8kpLS4O5uXmp54iEskQgPaJSqfDo0SOYmppCJBJV6LVTU1Ph7OyM+/fvw8zMrEKvTUQvxn+DRLpXWf8OBUFAWloa6tSpA7G49FE1ta7lRiwWo27dupV6DzMzM/5gJdIh/hsk0r3K+Hf4ohabAhxQTERERHqF4YaIiIj0CsNNBVIoFPjiiy+gUCh0XQpRrcR/g0S6Vx3+Hda6AcVERESk39hyQ0RERHqF4YaIiIj0CsMNERER6RWGGyIiItIrDDcV4OTJk+jduzfq1KkDkUiEX375RdclEdUqCxcuRMuWLWFqago7Ozv07dsXN2/e1HVZRLXG6tWr4eXlpVm4r02bNvj99991Vg/DTQXIyMiAt7c3Vq5cqetSiGqlv//+GxMnTsS5c+fwxx9/IC8vD926dUNGRoauSyOqFerWrYv//ve/uHjxIv7991907twZb731Fq5evaqTejgVvIKJRCLs378fffv21XUpRLVWfHw87Ozs8Pfff6NDhw66LoeoVrKyssKiRYswatSoKr93rdtbioj0X0pKCgD1D1ciqlpKpRK7d+9GRkYG2rRpo5MaGG6ISK+oVCpMnToVgYGBaNasma7LIao1wsPD0aZNG2RnZ8PExAT79+9HkyZNdFILww0R6ZWJEyfiypUrCA4O1nUpRLVKw4YNERYWhpSUFOzZswfDhg3D33//rZOAw3BDRHpj0qRJOHjwIE6ePIm6devquhyiWkUul8Pd3R0A4O/vj3/++Qc//PAD/ve//1V5LQw3RFTjCYKAyZMnY//+/Thx4gTc3Nx0XRJRradSqZCTk6OTezPcVID09HRERERo/hwVFYWwsDBYWVmhXr16OqyMqHaYOHEitm/fjgMHDsDU1BQxMTEAAHNzcxgaGuq4OiL9N3PmTPTo0QP16tVDWloatm/fjhMnTuDo0aM6qYdTwSvAiRMn8NprrxU5PmzYMGzevLnqCyKqZUQiUbHHN23ahOHDh1dtMUS10KhRoxAUFITHjx/D3NwcXl5emDFjBrp27aqTehhuiIiISK9whWIiIiLSKww3REREpFcYboiIiEivMNwQERGRXmG4ISIiIr3CcENERER6heGGiIiI9ArDDRHVeJ06dcLUqVN1XQYRVRMMN0RERKRXGG6IiIhIrzDcEJHeOXToEMzNzbFt2zZdl0JEOsBdwYlIr2zfvh3jxo3D9u3b0atXL12XQ0Q6wJYbItIbK1euxIQJE/Dbb78x2BDVYmy5ISK9sGfPHsTFxeH06dNo2bKlrsshIh1iyw0R6QVfX1/Y2tpi48aNEARB1+UQkQ4x3BCRXmjQoAGOHz+OAwcOYPLkybouh4h0iN1SRKQ3PD09cfz4cXTq1AlSqRRLly7VdUlEpAMMN0SkVxo2bIi//voLnTp1gkQiweLFi3VdEhFVMZHAzmkiIiLSIxxzQ0RERHqF4YaIiIj0CsMNERER6RWGGyIiItIrDDdERESkVxhuiIiISK8w3BAREZFeYbghIiIivcJwQ0RERHqF4YaIiIj0CsMNERER6RWGGyIiItIr/w8fdGIz5LM/lAAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Plotting each metric\n",
    "for metric_name in [\"precision\", \"recall\", \"ndcg\"]:\n",
    "    y = [evaluate_results.metrics[f\"{metric_name}_at_{k}/mean\"] for k in range(1, 4)]\n",
    "    plt.plot([1, 2, 3], y, label=f\"{metric_name}@k\")\n",
    "\n",
    "# Adding labels and title\n",
    "plt.xlabel(\"k\")\n",
    "plt.ylabel(\"Metric Value\")\n",
    "plt.title(\"Metrics Comparison at Different Ks\")\n",
    "# Setting x-axis ticks\n",
    "plt.xticks([1, 2, 3])\n",
    "plt.legend()\n",
    "\n",
    "# Display the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "cac23d4b-bece-4274-836f-9ca2b7c3860d",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "### Corner case handling\n",
    "\n",
    "There are a few corner cases handle specially for each built-in metric."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "e05a4ede-db44-46d2-bce8-752b0ce5d807",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "#### Empty retrieved document IDs\n",
    "\n",
    "When no relevant docs are retrieved:\n",
    "\n",
    "- `mlflow.metrics.precision_at_k(k)` is defined as:\n",
    "  * 0 if the ground-truth doc IDs is non-empty\n",
    "  * 1 if the ground-truth doc IDs is also empty\n",
    "\n",
    "- `mlflow.metrics.ndcg_at_k(k)` is defined as:\n",
    "  * 0 if the ground-truth doc IDs is non-empty\n",
    "  * 1 if the ground-truth doc IDs is also empty"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "931a32e7-29cb-4a22-b94e-ea2bf4f0b1a7",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "#### Empty ground-truth document IDs\n",
    "\n",
    "When no ground-truth document IDs are provided:\n",
    "\n",
    "- `mlflow.metrics.recall_at_k(k)` is defined as:\n",
    "  * 0 if the retrieved doc IDs is non-empty\n",
    "  * 1 if the retrieved doc IDs is also empty\n",
    "\n",
    "- `mlflow.metrics.ndcg_at_k(k)` is defined as:\n",
    "  * 0 if the retrieved doc IDs is non-empty\n",
    "  * 1 if the retrieved doc IDs is also empty"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "5a1453f6-a62d-43da-b230-955841c66651",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "#### Duplicate retreived document IDs\n",
    "\n",
    "It is a common case for the retriever in a RAG system to retrieve multiple chunks in the same document for a given query. In this case, `mlflow.metrics.ndcg_at_k(k)` is calculated as follows:\n",
    "\n",
    "If the duplicate doc IDs are in the ground truth,\n",
    "       they will be treated as different docs. For example, if the ground truth doc IDs are\n",
    "       [1, 2] and the retrieved doc IDs are [1, 1, 1, 3], the score will be equavalent to\n",
    "       ground truth doc IDs [10, 11, 12, 2] and retrieved doc IDs [10, 11, 12, 3].\n",
    "\n",
    "If the duplicate doc IDs are not in the ground truth, the ndcg score is calculated as normal."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "525ccc10-3a60-4dc9-804e-083cfa313349",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "## Step 4: Result Analysis and Visualization\n",
    "\n",
    "You can view the per-row scores in the logged \"eval_results_table.json\" in artifacts by either loading it to a pandas dataframe (shown below) or visiting the MLflow run comparison UI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "32f3d5b3-245c-46b7-87ce-d85e261eac28",
     "showTitle": true,
     "title": ""
    }
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ee4bfb1998174c558e537ebb1dd737d9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading artifacts:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question</th>\n",
       "      <th>source</th>\n",
       "      <th>retrieved_doc_ids</th>\n",
       "      <th>precision_at_1/score</th>\n",
       "      <th>precision_at_2/score</th>\n",
       "      <th>precision_at_3/score</th>\n",
       "      <th>recall_at_1/score</th>\n",
       "      <th>recall_at_2/score</th>\n",
       "      <th>recall_at_3/score</th>\n",
       "      <th>ndcg_at_1/score</th>\n",
       "      <th>ndcg_at_2/score</th>\n",
       "      <th>ndcg_at_3/score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>What is the purpose of the MLflow Model Registry?</td>\n",
       "      <td>[model-registry.html]</td>\n",
       "      <td>[model-registry.html, introduction/index.html,...</td>\n",
       "      <td>1</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.919721</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>What is the purpose of registering a model wit...</td>\n",
       "      <td>[model-registry.html]</td>\n",
       "      <td>[model-registry.html, models.html, introductio...</td>\n",
       "      <td>1</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>What can you do with registered models and mod...</td>\n",
       "      <td>[model-registry.html]</td>\n",
       "      <td>[model-registry.html, models.html, deployment/...</td>\n",
       "      <td>1</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>How can you add, modify, update, or delete a m...</td>\n",
       "      <td>[model-registry.html]</td>\n",
       "      <td>[model-registry.html, models.html, deployment/...</td>\n",
       "      <td>1</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>How can you deploy and organize models in the ...</td>\n",
       "      <td>[model-registry.html]</td>\n",
       "      <td>[model-registry.html, deployment/index.html, d...</td>\n",
       "      <td>1</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.919721</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            question                 source  \\\n",
       "0  What is the purpose of the MLflow Model Registry?  [model-registry.html]   \n",
       "1  What is the purpose of registering a model wit...  [model-registry.html]   \n",
       "2  What can you do with registered models and mod...  [model-registry.html]   \n",
       "3  How can you add, modify, update, or delete a m...  [model-registry.html]   \n",
       "4  How can you deploy and organize models in the ...  [model-registry.html]   \n",
       "\n",
       "                                   retrieved_doc_ids  precision_at_1/score  \\\n",
       "0  [model-registry.html, introduction/index.html,...                     1   \n",
       "1  [model-registry.html, models.html, introductio...                     1   \n",
       "2  [model-registry.html, models.html, deployment/...                     1   \n",
       "3  [model-registry.html, models.html, deployment/...                     1   \n",
       "4  [model-registry.html, deployment/index.html, d...                     1   \n",
       "\n",
       "   precision_at_2/score  precision_at_3/score  recall_at_1/score  \\\n",
       "0                   0.5              0.333333                  1   \n",
       "1                   0.5              0.333333                  1   \n",
       "2                   0.5              0.333333                  1   \n",
       "3                   0.5              0.333333                  1   \n",
       "4                   0.5              0.333333                  1   \n",
       "\n",
       "   recall_at_2/score  recall_at_3/score  ndcg_at_1/score  ndcg_at_2/score  \\\n",
       "0                  1                  1                1              1.0   \n",
       "1                  1                  1                1              1.0   \n",
       "2                  1                  1                1              1.0   \n",
       "3                  1                  1                1              1.0   \n",
       "4                  1                  1                1              1.0   \n",
       "\n",
       "   ndcg_at_3/score  \n",
       "0         0.919721  \n",
       "1         1.000000  \n",
       "2         1.000000  \n",
       "3         1.000000  \n",
       "4         0.919721  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "eval_results_table = evaluate_results.tables[\"eval_results_table\"]\n",
    "eval_results_table.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "cf18dd29-1017-4245-9f3b-923dbd46f742",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "With the evaluate results table, you can further visualize the well-answered questions and poorly-answered questions using topical analysis techniques."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "b1d9e40a-ccf6-4d6a-b24c-8cf41bbfa005",
     "showTitle": true,
     "title": "Utilitity functions"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to\n",
      "[nltk_data]     /Users/liang.zhang/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n",
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     /Users/liang.zhang/nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "import pyLDAvis.gensim_models as gensimvis\n",
    "from gensim import corpora, models\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.tokenize import word_tokenize\n",
    "\n",
    "# Initialize NLTK resources\n",
    "nltk.download(\"punkt\")\n",
    "nltk.download(\"stopwords\")\n",
    "\n",
    "\n",
    "def topical_analysis(questions: List[str]):\n",
    "    stop_words = set(stopwords.words(\"english\"))\n",
    "\n",
    "    # Tokenize and remove stop words\n",
    "    tokenized_data = []\n",
    "    for question in questions:\n",
    "        tokens = word_tokenize(question.lower())\n",
    "        filtered_tokens = [word for word in tokens if word not in stop_words and word.isalpha()]\n",
    "        tokenized_data.append(filtered_tokens)\n",
    "\n",
    "    # Create a dictionary and corpus\n",
    "    dictionary = corpora.Dictionary(tokenized_data)\n",
    "    corpus = [dictionary.doc2bow(text) for text in tokenized_data]\n",
    "\n",
    "    # Apply LDA model\n",
    "    lda_model = models.LdaModel(corpus, num_topics=5, id2word=dictionary, passes=15)\n",
    "\n",
    "    # Get topic distribution for each question\n",
    "    topic_distribution = []\n",
    "    for i, ques in enumerate(questions):\n",
    "        bow = dictionary.doc2bow(tokenized_data[i])\n",
    "        topics = lda_model.get_document_topics(bow)\n",
    "        topic_distribution.append(topics)\n",
    "        print(f\"Question: {ques}\\nTopic: {topics}\")\n",
    "\n",
    "    # Print all topics\n",
    "    print(\"\\nTopics found are:\")\n",
    "    for idx, topic in lda_model.print_topics(-1):\n",
    "        print(f\"Topic: {idx} \\nWords: {topic}\\n\")\n",
    "    return lda_model, corpus, dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "e892d804-a4d8-468c-93e2-acc4a5fbcf2c",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "filtered_df = eval_results_table[eval_results_table[\"precision_at_1/score\"] == 1]\n",
    "hit_questions = filtered_df[\"question\"].tolist()\n",
    "filtered_df = eval_results_table[eval_results_table[\"precision_at_1/score\"] == 0]\n",
    "miss_questions = filtered_df[\"question\"].tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "7c178b69-37d4-4a6b-9737-b93e7f3d75c5",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: What is the purpose of the MLflow Model Registry?\n",
      "Topic: [(0, 0.0400703), (1, 0.040002838), (2, 0.040673085), (3, 0.04075462), (4, 0.8384991)]\n",
      "Question: What is the purpose of registering a model with the Model Registry?\n",
      "Topic: [(0, 0.0334267), (1, 0.033337697), (2, 0.033401005), (3, 0.033786207), (4, 0.8660484)]\n",
      "Question: What can you do with registered models and model versions?\n",
      "Topic: [(0, 0.04019648), (1, 0.04000775), (2, 0.040166058), (3, 0.8391777), (4, 0.040452003)]\n",
      "Question: How can you add, modify, update, or delete a model in the Model Registry?\n",
      "Topic: [(0, 0.025052568), (1, 0.025006149), (2, 0.025024023), (3, 0.025236268), (4, 0.899681)]\n",
      "Question: How can you deploy and organize models in the Model Registry?\n",
      "Topic: [(0, 0.033460867), (1, 0.033337582), (2, 0.033362914), (3, 0.8659808), (4, 0.033857808)]\n",
      "Question: What method do you use to create a new registered model?\n",
      "Topic: [(0, 0.028867528), (1, 0.028582651), (2, 0.882546), (3, 0.030021703), (4, 0.029982116)]\n",
      "Question: How can you deploy and organize models in the Model Registry?\n",
      "Topic: [(0, 0.033460878), (1, 0.033337586), (2, 0.033362918), (3, 0.8659798), (4, 0.03385884)]\n",
      "Question: How can you fetch a list of registered models in the MLflow registry?\n",
      "Topic: [(0, 0.0286206), (1, 0.028577656), (2, 0.02894385), (3, 0.88495284), (4, 0.028905064)]\n",
      "Question: What is the default channel logged for models using MLflow v1.18 and above?\n",
      "Topic: [(0, 0.02862059), (1, 0.028577654), (2, 0.028883327), (3, 0.8851736), (4, 0.028744776)]\n",
      "Question: What information is stored in the conda.yaml file?\n",
      "Topic: [(0, 0.050020963), (1, 0.051287953), (2, 0.051250603), (3, 0.7968765), (4, 0.05056402)]\n",
      "Question: How can you save a model with a manually specified conda environment?\n",
      "Topic: [(0, 0.02862434), (1, 0.02858204), (2, 0.02886313), (3, 0.8851747), (4, 0.028755778)]\n",
      "Question: What are inference params and how are they used during model inference?\n",
      "Topic: [(0, 0.86457103), (1, 0.03353862), (2, 0.033417325), (3, 0.034004394), (4, 0.034468662)]\n",
      "Question: What is the purpose of model signatures in MLflow?\n",
      "Topic: [(0, 0.040070876), (1, 0.04000346), (2, 0.040688124), (3, 0.040469088), (4, 0.8387685)]\n",
      "Question: What is the API used to set signatures on models?\n",
      "Topic: [(0, 0.033873636), (1, 0.033508822), (2, 0.033337757), (3, 0.035357967), (4, 0.8639218)]\n",
      "Question: What components are used to generate the final time series?\n",
      "Topic: [(0, 0.028693806), (1, 0.8853218), (2, 0.028573763), (3, 0.02862714), (4, 0.0287835)]\n",
      "Question: What functionality does the configuration DataFrame submitted to the pyfunc flavor provide?\n",
      "Topic: [(0, 0.02519801), (1, 0.025009492), (2, 0.025004204), (3, 0.025004204), (4, 0.8997841)]\n",
      "Question: What is a common configuration for lowering the total memory pressure for pytorch models within transformers pipelines?\n",
      "Topic: [(0, 0.93316424), (1, 0.016669936), (2, 0.016668117), (3, 0.016788227), (4, 0.016709473)]\n",
      "Question: What does the save_model() function do?\n",
      "Topic: [(0, 0.10002145), (1, 0.59994656), (2, 0.10001026), (3, 0.10001026), (4, 0.10001151)]\n",
      "Question: What is an MLflow Project?\n",
      "Topic: [(0, 0.06667001), (1, 0.06667029), (2, 0.7321751), (3, 0.06711196), (4, 0.06737265)]\n",
      "Question: What are the entry points in a MLproject file and how can you specify parameters for them?\n",
      "Topic: [(0, 0.02857626), (1, 0.88541776), (2, 0.02868285), (3, 0.028626908), (4, 0.02869626)]\n",
      "Question: What are the project environments supported by MLflow?\n",
      "Topic: [(0, 0.040009078), (1, 0.040009864), (2, 0.839655), (3, 0.040126894), (4, 0.040199146)]\n",
      "Question: What is the purpose of specifying a Conda environment in an MLflow project?\n",
      "Topic: [(0, 0.028579442), (1, 0.028580135), (2, 0.8841217), (3, 0.028901232), (4, 0.029817443)]\n",
      "Question: What is the purpose of the MLproject file?\n",
      "Topic: [(0, 0.05001335), (1, 0.052611485), (2, 0.050071735), (3, 0.05043289), (4, 0.7968705)]\n",
      "Question: How can you pass runtime parameters to the entry point of an MLflow Project?\n",
      "Topic: [(0, 0.025007373), (1, 0.025498485), (2, 0.8993807), (3, 0.02504522), (4, 0.025068246)]\n",
      "Question: How does MLflow run a Project on Kubernetes?\n",
      "Topic: [(0, 0.04000677), (1, 0.040007353), (2, 0.83931196), (3, 0.04012452), (4, 0.04054937)]\n",
      "Question: What fields are replaced when MLflow creates a Kubernetes Job for an MLflow Project?\n",
      "Topic: [(0, 0.022228329), (1, 0.022228856), (2, 0.023192631), (3, 0.02235802), (4, 0.90999216)]\n",
      "Question: What is the syntax for searching runs using the MLflow UI and API?\n",
      "Topic: [(0, 0.025003674), (1, 0.02500399), (2, 0.02527212), (3, 0.89956146), (4, 0.025158761)]\n",
      "Question: What is the syntax for searching runs using the MLflow UI and API?\n",
      "Topic: [(0, 0.025003672), (1, 0.025003988), (2, 0.025272164), (3, 0.8995614), (4, 0.025158769)]\n",
      "Question: What are the key parts of a search expression in MLflow?\n",
      "Topic: [(0, 0.03334423), (1, 0.03334517), (2, 0.8662702), (3, 0.033611353), (4, 0.033429127)]\n",
      "Question: What are the key attributes for the model with the run_id 'a1b2c3d4' and run_name 'my-run'?\n",
      "Topic: [(0, 0.05017508), (1, 0.05001634), (2, 0.05058142), (3, 0.7985237), (4, 0.050703418)]\n",
      "Question: What information does each run record in MLflow Tracking?\n",
      "Topic: [(0, 0.03333968), (1, 0.033340227), (2, 0.86639804), (3, 0.03349555), (4, 0.033426523)]\n",
      "Question: What are the two components used by MLflow for storage?\n",
      "Topic: [(0, 0.0334928), (1, 0.033938777), (2, 0.033719826), (3, 0.03357158), (4, 0.86527705)]\n",
      "Question: What interfaces does the MLflow client use to record MLflow entities and artifacts when running MLflow on a local machine with a SQLAlchemy-compatible database?\n",
      "Topic: [(0, 0.014289577), (1, 0.014289909), (2, 0.94276434), (3, 0.014325481), (4, 0.014330726)]\n",
      "Question: What is the default backend store used by MLflow?\n",
      "Topic: [(0, 0.033753525), (1, 0.03379533), (2, 0.033777602), (3, 0.86454684), (4, 0.0341267)]\n",
      "Question: What information does autologging capture when launching short-lived MLflow runs?\n",
      "Topic: [(0, 0.028579954), (1, 0.02858069), (2, 0.8851724), (3, 0.029027484), (4, 0.028639426)]\n",
      "Question: What is the purpose of the --serve-artifacts flag?\n",
      "Topic: [(0, 0.06670548), (1, 0.066708855), (2, 0.067003354), (3, 0.3969311), (4, 0.40265122)]\n",
      "\n",
      "Topics found are:\n",
      "Topic: 0 \n",
      "Words: 0.059*\"inference\" + 0.032*\"models\" + 0.032*\"used\" + 0.032*\"configuration\" + 0.032*\"common\" + 0.032*\"transformers\" + 0.032*\"total\" + 0.032*\"within\" + 0.032*\"pytorch\" + 0.032*\"pipelines\"\n",
      "\n",
      "Topic: 1 \n",
      "Words: 0.036*\"file\" + 0.035*\"mlproject\" + 0.035*\"used\" + 0.035*\"components\" + 0.035*\"entry\" + 0.035*\"parameters\" + 0.035*\"specify\" + 0.035*\"final\" + 0.035*\"points\" + 0.035*\"time\"\n",
      "\n",
      "Topic: 2 \n",
      "Words: 0.142*\"mlflow\" + 0.066*\"project\" + 0.028*\"information\" + 0.028*\"use\" + 0.028*\"record\" + 0.028*\"run\" + 0.015*\"key\" + 0.015*\"running\" + 0.015*\"artifacts\" + 0.015*\"client\"\n",
      "\n",
      "Topic: 3 \n",
      "Words: 0.066*\"models\" + 0.066*\"model\" + 0.066*\"mlflow\" + 0.041*\"using\" + 0.041*\"registry\" + 0.028*\"api\" + 0.028*\"registered\" + 0.028*\"runs\" + 0.028*\"syntax\" + 0.028*\"searching\"\n",
      "\n",
      "Topic: 4 \n",
      "Words: 0.089*\"model\" + 0.074*\"purpose\" + 0.074*\"mlflow\" + 0.046*\"registry\" + 0.031*\"used\" + 0.031*\"signatures\" + 0.017*\"kubernetes\" + 0.017*\"fields\" + 0.017*\"job\" + 0.017*\"replaced\"\n",
      "\n"
     ]
    }
   ],
   "source": [
    "lda_model, corpus, dictionary = topical_analysis(hit_questions)\n",
    "vis_data = gensimvis.prepare(lda_model, corpus, dictionary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "a0587a0f-b35d-488d-9054-55435a9585bf",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "# Uncomment the following line to render the interactive widget\n",
    "# pyLDAvis.display(vis_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "1375250d-9818-4503-87ec-f14020d87c81",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: What is the purpose of the mlflow.sklearn.log_model() method?\n",
      "Topic: [(0, 0.0669118), (1, 0.06701085), (2, 0.06667974), (3, 0.73235476), (4, 0.06704286)]\n",
      "Question: How can you fetch a specific model version?\n",
      "Topic: [(0, 0.83980393), (1, 0.040003464), (2, 0.04000601), (3, 0.040101767), (4, 0.040084846)]\n",
      "Question: How can you fetch the latest model version in a specific stage?\n",
      "Topic: [(0, 0.88561153), (1, 0.028575428), (2, 0.028578365), (3, 0.0286214), (4, 0.028613236)]\n",
      "Question: What can you do to promote MLflow Models across environments?\n",
      "Topic: [(0, 0.8661927), (1, 0.0333396), (2, 0.03362743), (3, 0.033428304), (4, 0.033411972)]\n",
      "Question: What is the name of the model and its version details?\n",
      "Topic: [(0, 0.83978903), (1, 0.04000637), (2, 0.04001106), (3, 0.040105395), (4, 0.040088095)]\n",
      "Question: What is the purpose of saving the model in pickled format?\n",
      "Topic: [(0, 0.033948876), (1, 0.03339717), (2, 0.033340737), (3, 0.86575514), (4, 0.033558063)]\n",
      "Question: What is an MLflow Model and what is its purpose?\n",
      "Topic: [(0, 0.7940762), (1, 0.05068333), (2, 0.050770763), (3, 0.053328265), (4, 0.05114142)]\n",
      "Question: What are the flavors defined in the MLmodel file for the mlflow.sklearn library?\n",
      "Topic: [(0, 0.86628276), (1, 0.033341788), (2, 0.03334801), (3, 0.03368498), (4, 0.033342462)]\n",
      "Question: What command can be used to package and deploy models to AWS SageMaker?\n",
      "Topic: [(0, 0.89991224), (1, 0.025005225), (2, 0.025009066), (3, 0.025006713), (4, 0.025066752)]\n",
      "Question: What is the purpose of the --build-image flag when running mlflow run?\n",
      "Topic: [(0, 0.033957016), (1, 0.033506736), (2, 0.034095332), (3, 0.034164555), (4, 0.86427635)]\n",
      "Question: What is the relative path to the python_env YAML file within the MLflow project's directory?\n",
      "Topic: [(0, 0.02243), (1, 0.02222536), (2, 0.022470985), (3, 0.9105873), (4, 0.02228631)]\n",
      "Question: What are the additional local volume mounted and environment variables in the docker container?\n",
      "Topic: [(0, 0.022225259), (1, 0.9110914), (2, 0.02222932), (3, 0.022227468), (4, 0.022226628)]\n",
      "Question: What are some examples of entity names that contain special characters?\n",
      "Topic: [(0, 0.028575381), (1, 0.88568854), (2, 0.02858065), (3, 0.028578246), (4, 0.028577149)]\n",
      "Question: What type of constant does the RHS need to be if LHS is a metric?\n",
      "Topic: [(0, 0.028575381), (1, 0.8856886), (2, 0.028580645), (3, 0.028578239), (4, 0.028577147)]\n",
      "Question: How can you get all active runs from experiments IDs 3, 4, and 17 that used a CNN model with 10 layers and had a prediction accuracy of 94.5% or higher?\n",
      "Topic: [(0, 0.015563371), (1, 0.015387185), (2, 0.015389071), (3, 0.015427767), (4, 0.9382326)]\n",
      "Question: What is the purpose of the 'experimentIds' variable in the given paragraph?\n",
      "Topic: [(0, 0.040206533), (1, 0.8384999), (2, 0.040013183), (3, 0.040967643), (4, 0.040312726)]\n",
      "Question: What is the MLflow Tracking component used for?\n",
      "Topic: [(0, 0.8390845), (1, 0.04000697), (2, 0.040462855), (3, 0.04014182), (4, 0.040303845)]\n",
      "Question: How can you create an experiment in MLflow?\n",
      "Topic: [(0, 0.050333958), (1, 0.0500024), (2, 0.7993825), (3, 0.050153885), (4, 0.05012722)]\n",
      "Question: How can you create an experiment using MLflow?\n",
      "Topic: [(0, 0.04019285), (1, 0.04000254), (2, 0.8396381), (3, 0.040091105), (4, 0.04007539)]\n",
      "Question: What is the architecture depicted in this example scenario?\n",
      "Topic: [(0, 0.04000523), (1, 0.040007014), (2, 0.040012203), (3, 0.04000902), (4, 0.83996654)]\n",
      "\n",
      "Topics found are:\n",
      "Topic: 0 \n",
      "Words: 0.078*\"model\" + 0.059*\"mlflow\" + 0.059*\"version\" + 0.041*\"models\" + 0.041*\"fetch\" + 0.041*\"specific\" + 0.041*\"used\" + 0.022*\"command\" + 0.022*\"deploy\" + 0.022*\"sagemaker\"\n",
      "\n",
      "Topic: 1 \n",
      "Words: 0.030*\"local\" + 0.030*\"container\" + 0.030*\"variables\" + 0.030*\"docker\" + 0.030*\"mounted\" + 0.030*\"environment\" + 0.030*\"volume\" + 0.030*\"additional\" + 0.030*\"special\" + 0.030*\"names\"\n",
      "\n",
      "Topic: 2 \n",
      "Words: 0.096*\"experiment\" + 0.096*\"create\" + 0.096*\"mlflow\" + 0.051*\"using\" + 0.009*\"purpose\" + 0.009*\"model\" + 0.009*\"method\" + 0.009*\"file\" + 0.009*\"version\" + 0.009*\"used\"\n",
      "\n",
      "Topic: 3 \n",
      "Words: 0.071*\"purpose\" + 0.039*\"file\" + 0.039*\"mlflow\" + 0.039*\"yaml\" + 0.039*\"directory\" + 0.039*\"relative\" + 0.039*\"within\" + 0.039*\"path\" + 0.039*\"project\" + 0.039*\"format\"\n",
      "\n",
      "Topic: 4 \n",
      "Words: 0.032*\"purpose\" + 0.032*\"used\" + 0.032*\"model\" + 0.032*\"prediction\" + 0.032*\"get\" + 0.032*\"accuracy\" + 0.032*\"active\" + 0.032*\"layers\" + 0.032*\"higher\" + 0.032*\"experiments\"\n",
      "\n"
     ]
    }
   ],
   "source": [
    "lda_model, corpus, dictionary = topical_analysis(miss_questions)\n",
    "vis_data = gensimvis.prepare(lda_model, corpus, dictionary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "724db985-5382-43a6-ada5-0ac1c2d49c18",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "# Uncomment the following line to render the interactive widget\n",
    "# pyLDAvis.display(vis_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "31945151-7cf9-4f25-af30-d9b9bd526e7b",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "application/vnd.databricks.v1+notebook": {
   "dashboards": [],
   "language": "python",
   "notebookMetadata": {
    "pythonIndentUnit": 4
   },
   "notebookName": "retriever-evaluation-tutorial",
   "widgets": {}
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.17"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}