training/tfidf_model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bug Classifcation using Natural Language Processing\n",
    "The following project uses a Natural Language Processing model to read in bug descriptions submitted by the Global Customer Support Desk, internal Pega employees, and pega customers, and assign these bugs to the relevant backlogs in Agile Studio.\n",
    "\n",
    "To perform bug assignment, we first pre-process bug descriptions and subject lines. Then, we convert each bug description to a TF/ IDF vector. Finally, we use a Deep Learning model to extract insights from the TF/IDF scores and train that model to perform accurate classifications.\n",
    "\n",
    "At the prediction phase, we apply confidence bounding using [Monte Carlo dropout](https://arxiv.org/pdf/1506.02142.pdf). Using this approach, we can choose the threshold confidence score that best suits our needs - the model will automatically route bugs to the appropriate backlogs if it can generate a prediction above the confidence threshold (a high confidence prediction) and will route bugs that it cannot confidently place into a queue for human review. Stakeholders at Pega can choose a proper trade off between accuracy and efficiency by tweaking the confidence threshold to the desired value."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Install Packages.\n",
    "The first script, [`install_conda.sh`](../nlp_engine/install_conda.sh), installs [miniconda](https://docs.conda.io/en/latest/miniconda.html) and sets up an environment called `nlp-workspace`. You can learn more about conda environments [here](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html). The environment essentially acts as a sand-boxed virtual machine where we keep all of the code dependencies. As you can see by inspecting the install script, we use python 3.7 as our base. Further, we install the `nb_conda` package. This package allows us to select an environment on a Jupyter Notebook (nb). Once the `install_conda` script has been run, you should be able to open a jupyter notebook on your host machine and select `Python (nlp-workspace)` as the kernel.\n",
    "\n",
    "\n",
    "The second script, [`install_packages.sh`](../nlp_engine/install_packages.sh), installs all of the requisite python packages necessary to train the machine learning model and deploy it. After successfully running this script, the environment should have all of the necessary packages and all imports should work correctly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uncomment this entire selection to perform the requisite installs\n",
    "#%%capture # The capture tag suppresses output. Comment it out if you want to see installation script output.\n",
    "#!bash ../nlp_engine/install_conda.sh\n",
    "#!bash ../nlp_engine/install_packages.sh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Version:  2.0.1\n",
      "Eager mode:  True\n",
      "GPU is available\n"
     ]
    }
   ],
   "source": [
    "# Import all needed libraries from the `nlp-workspace` environment\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.dummy import DummyClassifier\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer, HashingVectorizer\n",
    "from sklearn import metrics\n",
    "\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "from tensorflow.keras import layers\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Embedding, LSTM, GRU, Dropout, Flatten, Input, Activation, PReLU, LeakyReLU, ThresholdedReLU\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns; sns.set()\n",
    "\n",
    "import pickle\n",
    "from tqdm import tqdm\n",
    "\n",
    "print(\"Version: \", tf.__version__)\n",
    "print(\"Eager mode: \", tf.executing_eagerly())\n",
    "print(\"GPU is\", \"available\" if tf.config.experimental.list_physical_devices(\"GPU\") else \"NOT AVAILABLE\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Pull in additional libraries.\n",
    "Here, we pull in additional libraries that we created to perform standard NLP test preprocessing and convenience functions for machine learning.\n",
    "You can inspect the functions we pull in here by taking a look at our [`MLFunctions`](../nlp_engine/MLFunctions.py) file or our [`Preprocessing`](../nlp_engine/Preprocessing.py) file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Spacy model is using GPU\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "sys.path.insert(1, '../nlp_engine')\n",
    "from MLFunctions import PrintDot, plot_history, clear_memory, test_with_uncertainty, \\\n",
    "                        predict_with_uncertainty, get_monte_carlo_accuracy, graph_confidence\n",
    "from Preprocessing import preprocess_training_text, preprocess_training_text_with_stops"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Read-in and Clean Data\n",
    "Here, we read in the email bug data and eliminate training examples that will not be useful to out model. Starting with the original dataset, we\n",
    "1. **Eliminate** \n",
    "    - bugs that were not assigned to a backlog \n",
    "    - bugs that were not assigned to a team (i.e. bugs that were never resolved)\n",
    "    - bugs that lack a description or label\n",
    "    \n",
    "    \n",
    "2. **Remove duplicate entries, if they exist.**\n",
    "\n",
    "3. **Remove backlogs that have been assigned less than 40 training examples.** Backlogs that have been assigned less than this number will be prone to model over-fitting, since they are so underrepresented in a dataset of the size we are working with. As such, we remove them to avoid inaccurate classifications. We settled on the number 40 after experimenting with a variety of different thresholds, but you may want to change this threshold as you collect additional training data to be fed into the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>work_id</th>\n",
       "      <th>label</th>\n",
       "      <th>description</th>\n",
       "      <th>status</th>\n",
       "      <th>team_id</th>\n",
       "      <th>team_name</th>\n",
       "      <th>backlog_id</th>\n",
       "      <th>backlog_name</th>\n",
       "      <th>feature_id</th>\n",
       "      <th>feature_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>BUG-530021</td>\n",
       "      <td>Issues with Reports and Notifications</td>\n",
       "      <td>Hi Team,\\n\\n\\n\\nThis is to report 2 issues.\\n\\...</td>\n",
       "      <td>Resolved-Enhancement</td>\n",
       "      <td>PROJ-10002</td>\n",
       "      <td>Cloud - Beyonders</td>\n",
       "      <td>BL-3339</td>\n",
       "      <td>Cloud</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>BUG-528121</td>\n",
       "      <td>OPD-PROD: Pre schedule validation stuck in pro...</td>\n",
       "      <td>Pre-scheduled validation is stuck at in-progre...</td>\n",
       "      <td>Resolved-Completed</td>\n",
       "      <td>PROJ-10002</td>\n",
       "      <td>Cloud - Beyonders</td>\n",
       "      <td>BL-3339</td>\n",
       "      <td>Cloud</td>\n",
       "      <td>FR-9155</td>\n",
       "      <td>Scheduling Patch Installs</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>BUG-529279</td>\n",
       "      <td>For inflight DBupgrde tasks, Validation tasks ...</td>\n",
       "      <td>For inflight DBupgrde tasks, Validation tasks ...</td>\n",
       "      <td>Resolved-Completed</td>\n",
       "      <td>PROJ-10002</td>\n",
       "      <td>Cloud - Beyonders</td>\n",
       "      <td>BL-3339</td>\n",
       "      <td>Cloud</td>\n",
       "      <td>FR-8238</td>\n",
       "      <td>Global Operations Console (GOC)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>BUG-528748</td>\n",
       "      <td>Issues in Q_7.22</td>\n",
       "      <td>1. Create a VCR dispute\\n\\r\\n2. Select Quality...</td>\n",
       "      <td>Resolved-Completed</td>\n",
       "      <td>PROJ-10005</td>\n",
       "      <td>20.1 Compliance for SD Issuers and Acquirers</td>\n",
       "      <td>GRP-15763</td>\n",
       "      <td>FS - Smart Dispute - Issuer Compliance Backlog</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>BUG-537524</td>\n",
       "      <td>Incorrect Deadline value in overview tab</td>\n",
       "      <td>Steps to reproduce:\\n\\n1. Login to QA server w...</td>\n",
       "      <td>New</td>\n",
       "      <td>PROJ-10005</td>\n",
       "      <td>20.1 Compliance for SD Issuers and Acquirers</td>\n",
       "      <td>GRP-15763</td>\n",
       "      <td>FS - Smart Dispute - Issuer Compliance Backlog</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      work_id                                              label  \\\n",
       "0  BUG-530021              Issues with Reports and Notifications   \n",
       "1  BUG-528121  OPD-PROD: Pre schedule validation stuck in pro...   \n",
       "2  BUG-529279  For inflight DBupgrde tasks, Validation tasks ...   \n",
       "3  BUG-528748                                   Issues in Q_7.22   \n",
       "4  BUG-537524           Incorrect Deadline value in overview tab   \n",
       "\n",
       "                                         description                status  \\\n",
       "0  Hi Team,\\n\\n\\n\\nThis is to report 2 issues.\\n\\...  Resolved-Enhancement   \n",
       "1  Pre-scheduled validation is stuck at in-progre...    Resolved-Completed   \n",
       "2  For inflight DBupgrde tasks, Validation tasks ...    Resolved-Completed   \n",
       "3  1. Create a VCR dispute\\n\\r\\n2. Select Quality...    Resolved-Completed   \n",
       "4  Steps to reproduce:\\n\\n1. Login to QA server w...                   New   \n",
       "\n",
       "      team_id                                     team_name backlog_id  \\\n",
       "0  PROJ-10002                             Cloud - Beyonders    BL-3339   \n",
       "1  PROJ-10002                             Cloud - Beyonders    BL-3339   \n",
       "2  PROJ-10002                             Cloud - Beyonders    BL-3339   \n",
       "3  PROJ-10005  20.1 Compliance for SD Issuers and Acquirers  GRP-15763   \n",
       "4  PROJ-10005  20.1 Compliance for SD Issuers and Acquirers  GRP-15763   \n",
       "\n",
       "                                     backlog_name feature_id  \\\n",
       "0                                           Cloud        NaN   \n",
       "1                                           Cloud    FR-9155   \n",
       "2                                           Cloud    FR-8238   \n",
       "3  FS - Smart Dispute - Issuer Compliance Backlog        NaN   \n",
       "4  FS - Smart Dispute - Issuer Compliance Backlog        NaN   \n",
       "\n",
       "                      feature_name  \n",
       "0                              NaN  \n",
       "1        Scheduling Patch Installs  \n",
       "2  Global Operations Console (GOC)  \n",
       "3                              NaN  \n",
       "4                              NaN  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read in data & rename columns\n",
    "email_bugs = pd.read_csv(\"../data/csvs/bug_emails.csv\")\n",
    "email_bugs = email_bugs.rename(columns={\"Work ID\" : \"work_id\", \"Label\" : \"label\", \"Description\" : \"description\", \n",
    "                             \"Status\" : \"status\", \"Team ID\" : \"team_id\", \"Team Name\" : \"team_name\", \n",
    "                             \"Backlog ID\" : \"backlog_id\", \"Backlog Name\" : \"backlog_name\", \"Feature ID\" : \"feature_id\", \n",
    "                             \"Feature Name\" : \"feature_name\"})\n",
    "\n",
    "# 1. Eliminate:\n",
    "assigned_bugs = email_bugs[email_bugs['backlog_id'].notnull()]   # bugs that were not assigned to a backlog\n",
    "assigned_bugs = assigned_bugs[assigned_bugs['team_id'].notnull()]  # bugs that lack a team\n",
    "assigned_bugs = assigned_bugs[assigned_bugs['description'].notnull()]  # bugs that lack a description\n",
    "assigned_bugs = assigned_bugs[assigned_bugs['label'].notnull()]        # or label\n",
    "\n",
    "# 2. Drop Duplicates:\n",
    "assigned_bugs.drop_duplicates(subset = \"work_id\", keep = False, inplace = True) # duplicate bugs\n",
    "\n",
    "# Visualize the dataset\n",
    "assigned_bugs.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of backlog IDs:  159\n",
      "Number of training samples:  7438\n"
     ]
    }
   ],
   "source": [
    "# Check current number of backlogs and samples\n",
    "print(\"Number of backlog IDs: \" , assigned_bugs['backlog_id'].nunique())\n",
    "print(\"Number of training samples: \" , len(assigned_bugs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Backlog ID's remaining: 47\n",
      "Number of training samples remaining:  6245\n",
      "% of remaining training examples:  0.8396074213498252\n"
     ]
    }
   ],
   "source": [
    "# 3. Remove backlogs that have been assigned 40 or less training examples:\n",
    "former_length = len(assigned_bugs)\n",
    "bugs_per_id = assigned_bugs['backlog_id'].value_counts()\n",
    "valid_id_list = bugs_per_id[bugs_per_id > 40].index.tolist()\n",
    "assigned_bugs = assigned_bugs[assigned_bugs['backlog_id'].isin(valid_id_list)]\n",
    "\n",
    "print(\"Backlog ID's remaining:\", len(valid_id_list))\n",
    "print(\"Number of training samples remaining: \" , len(assigned_bugs))\n",
    "print(\"% of remaining training examples: \", len(assigned_bugs) / former_length)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Preprocessing\n",
    "Below, we apply the preprocessing functions we created in our [`Preprocessing`](../nlp_engine/Preprocessing.py) library to our bug descriptions and subject lines ('label'). \n",
    "**The standard preprocessing function applies the following procedures to our bug descriptions:**\n",
    "- Strip HTML\n",
    "- Remove non-ascii characters (i.e. Unicode)\n",
    "- Remove excess whitespace (tabs, newlines, etc.)\n",
    "- Convert accented chars (café -> *cafe*)\n",
    "- Lowercase all characters\n",
    "- Remove numbers (*see note below*)\n",
    "- Remove punctuation\n",
    "- Expand contractions\n",
    "          don't → do not \n",
    "          I'll → I will be\n",
    "- Lemmatization \n",
    "          [Playing, plays, played]→ play\n",
    "          [am, are, is] → be\n",
    "- Remove stop words (the most common words in the English language)\n",
    "          buy a banana at market → buy banana market\n",
    "\n",
    "\n",
    "**Notes**:\n",
    "- You can choose to convert numbers into text (i.e. 7 -> seven) instead of removing them by setting the `convert_num` setting to `True` and the `remove_num` setting to `False`.\n",
    "- Any other of the preprocessing options can be changed simply by toying with the parameters, as you can see we have done with the `preprocess_with_caps` function below.\n",
    "\n",
    "---\n",
    "### Rationale:\n",
    "\n",
    "Each of the above procedures are designed to clean the text and increase its value in terms of the predictive power it delivers. \n",
    "\n",
    "**Lemmatization and contraction expansion** are designed to ensure that phrases and words that mean the same thing are always represented the same way in the text. This way, a machine learning model doesn't need to learn the relationship between play, played, plays, etc. and the prediction it is trying to make. Instead, it only needs to learn the relationship between the single word - play- and the prediction it is trying to make. Lemmatization and contraction expansion therefore work to better capture the semantic meaning of a text and remove variants that might confuse an ML model. \n",
    "\n",
    "**Removing stop words** helps to improve the value of a text by removing all words that lack predictive power, thus allowing an ML model to better \"attend to\" the words that matter.\n",
    "\n",
    "---\n",
    "### Usage:\n",
    "To use our preprocessing function we use the pandas [.apply syntax](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.apply.html). When we call `dataframe.apply(function_name)`, we apply that function to all of the entries (bugs) in our DataFrame. When you run the below cells, note that the preprocessing function itself uses a pre-trained machine learning model ([from spacy](https://spacy.io/models)) to perform lemmatization and contraction expansion. As such, the process is computationally intensive. It takes a few minutes to run on a GPU machine, and likely longer on a standard CPU machine.\n",
    "\n",
    "---\n",
    "\n",
    "### Additional Steps:\n",
    "After we finish preprocessing, we **remove all bugs that contain 10 or leas words in the description field**, as these bugs are unlikely to contain enough inferencable information. We perform this step after preprocessing so that we do not include stop words, emails, URLs, numbers, or other non-word symbols in the word count.\n",
    "\n",
    "Its possible that bugs with this few words could have a descriptive subject line, but we found subject lines to be significantly less predictive than the description field, often including only generic descriptions or identification numbers that mean nothing to our model.\n",
    "\n",
    "In our dataset, there are only a few hundred bugs with descriptions of 10 or less words... because we remove them, however, it's worth noting that our model is likely to perform poorly if fed extremely short emails in production unless those emails contain keywords our model has been trained to utilize."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess_with_stops_caps(text):\n",
    "    return preprocess_training_text(text, accented_chars=True, contractions=True, \n",
    "                       convert_num=False, extra_whitespace=True, \n",
    "                       lemmatization=True, lowercase=False, punctuations=False,\n",
    "                       remove_html=True, remove_num=True, special_chars=True, \n",
    "                       stop_words=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/root/miniconda3/envs/nlp-workspace/lib/python3.7/site-packages/tqdm/std.py:668: FutureWarning: The Panel class is removed from pandas. Accessing it from the top-level namespace will also be removed in the next version\n",
      "  from pandas import Panel\n",
      "preproccessing descriptions: 100%|██████████| 6245/6245 [02:21<00:00, 44.27it/s]\n",
      "preproccessing labels: 100%|██████████| 6245/6245 [01:47<00:00, 58.16it/s]\n"
     ]
    }
   ],
   "source": [
    "# Run this cell to perform text preprocessing\n",
    "tqdm.pandas(desc=\"preproccessing descriptions\")\n",
    "assigned_bugs['description'] = assigned_bugs['description'].progress_apply(preprocess_training_text)\n",
    "tqdm.pandas(desc=\"preproccessing labels\")\n",
    "assigned_bugs['label'] = assigned_bugs['label'].progress_apply(preprocess_training_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Concatenate bug labels and descriptions, and create a `combined` data column in our `assigned_bugs` dataframe. \n",
    "# We will pass this data into our ML model, since the label + description combined contains more information than \n",
    "# either text field individually.\n",
    "assigned_bugs['combined'] = assigned_bugs['label'].map(str) + ' ' + assigned_bugs['description']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of training samples remaining:  5434\n"
     ]
    }
   ],
   "source": [
    "# Additional Step: Remove all bugs that contain 10 or less words in the description:\n",
    "assigned_bugs = assigned_bugs[assigned_bugs['description'].str.split().apply(len) > 10]\n",
    "print(\"Number of training samples remaining: \" , len(assigned_bugs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The below code snippet can be used to save pre-processed text to a pickle file.\n",
    "By saving the pre-processed text to a pickle, you avoid having to continuously\n",
    "re-run the preprocessing function each time you work with this code. \n",
    "\n",
    "**Once you have run preprocessing on your bug data once, you can skip running the above preprocessing cell and simply load the relevant pickle file to obtain processed text.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the preprocessed bugs\n",
    "pd.to_pickle(assigned_bugs, \"../data/pickles/preprocessed_bugs.pkl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Restore the preprocessed bugs (run this to reload\n",
    "# preprocessed data)\n",
    "assigned_bugs = pd.read_pickle(\"../data/pickles/preprocessed_bugs.pkl\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check Bug Text\n",
    "Confirm that the text has been cleaned and processed correctly by examining the DataFrame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>work_id</th>\n",
       "      <th>label</th>\n",
       "      <th>description</th>\n",
       "      <th>status</th>\n",
       "      <th>team_id</th>\n",
       "      <th>team_name</th>\n",
       "      <th>backlog_id</th>\n",
       "      <th>backlog_name</th>\n",
       "      <th>feature_id</th>\n",
       "      <th>feature_name</th>\n",
       "      <th>combined</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>BUG-530021</td>\n",
       "      <td>issue  report  notification</td>\n",
       "      <td>hi  team  report  issue  pre  schedule  valida...</td>\n",
       "      <td>Resolved-Enhancement</td>\n",
       "      <td>PROJ-10002</td>\n",
       "      <td>Cloud - Beyonders</td>\n",
       "      <td>BL-3339</td>\n",
       "      <td>Cloud</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>issue  report  notification hi  team  report  ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>BUG-528121</td>\n",
       "      <td>opd  prod  pre  schedule  validation  stick  p...</td>\n",
       "      <td>pre  schedule  validation  stick  progress  go...</td>\n",
       "      <td>Resolved-Completed</td>\n",
       "      <td>PROJ-10002</td>\n",
       "      <td>Cloud - Beyonders</td>\n",
       "      <td>BL-3339</td>\n",
       "      <td>Cloud</td>\n",
       "      <td>FR-9155</td>\n",
       "      <td>Scheduling Patch Installs</td>\n",
       "      <td>opd  prod  pre  schedule  validation  stick  p...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>BUG-529279</td>\n",
       "      <td>inflight  dbupgrde  task  validation  task  no...</td>\n",
       "      <td>inflight  dbupgrde  task  validation  task  in...</td>\n",
       "      <td>Resolved-Completed</td>\n",
       "      <td>PROJ-10002</td>\n",
       "      <td>Cloud - Beyonders</td>\n",
       "      <td>BL-3339</td>\n",
       "      <td>Cloud</td>\n",
       "      <td>FR-8238</td>\n",
       "      <td>Global Operations Console (GOC)</td>\n",
       "      <td>inflight  dbupgrde  task  validation  task  no...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>BUG-530336</td>\n",
       "      <td>new  app  wizard  error</td>\n",
       "      <td>laura  protano  run  error  run  new  applicat...</td>\n",
       "      <td>Resolved-Completed</td>\n",
       "      <td>PROJ-10010</td>\n",
       "      <td>Zenith for Customer Service 8.5</td>\n",
       "      <td>BL-752</td>\n",
       "      <td>Pega Customer Service Development Backlog</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>new  app  wizard  error laura  protano  run  e...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>BUG-530860</td>\n",
       "      <td>v  dx  api  automation  need  description  fix</td>\n",
       "      <td>swagger  generator  use  automation  short  de...</td>\n",
       "      <td>Resolved-Completed</td>\n",
       "      <td>PROJ-10015</td>\n",
       "      <td>DX APIs - CodeNinjas</td>\n",
       "      <td>GRP-13931</td>\n",
       "      <td>Platform UI Backlog</td>\n",
       "      <td>FR-8192</td>\n",
       "      <td>Web APIs</td>\n",
       "      <td>v  dx  api  automation  need  description  fix...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7561</th>\n",
       "      <td>BUG-534804</td>\n",
       "      <td>qa  incorrect  sale  support  manager  onboard...</td>\n",
       "      <td>url  prweb  step  reproduce  login  john  john...</td>\n",
       "      <td>Resolved-CannotRecreate</td>\n",
       "      <td>PROJ-9995</td>\n",
       "      <td>Symphony [PCLM] 8.4</td>\n",
       "      <td>BL-9394</td>\n",
       "      <td>FS - Client Lifecycle Management (8.4)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>qa  incorrect  sale  support  manager  onboard...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7562</th>\n",
       "      <td>BUG-535030</td>\n",
       "      <td>fm  manage  product  fund  complete  default</td>\n",
       "      <td>step  start  onboarde  fm  b  add  fund  produ...</td>\n",
       "      <td>Resolved-Enhancement</td>\n",
       "      <td>PROJ-9995</td>\n",
       "      <td>Symphony [PCLM] 8.4</td>\n",
       "      <td>BL-9394</td>\n",
       "      <td>FS - Client Lifecycle Management (8.4)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>fm  manage  product  fund  complete  default s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7563</th>\n",
       "      <td>BUG-532532</td>\n",
       "      <td>pdc  exception  null  pointer  excep  retail</td>\n",
       "      <td>initial  analysis  initial  analysis  step  ch...</td>\n",
       "      <td>Resolved-Completed</td>\n",
       "      <td>PROJ-9995</td>\n",
       "      <td>Symphony [PCLM] 8.4</td>\n",
       "      <td>BL-9394</td>\n",
       "      <td>FS - Client Lifecycle Management (8.4)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>pdc  exception  null  pointer  excep  retail i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7564</th>\n",
       "      <td>BUG-531668</td>\n",
       "      <td>cib  case  stuck  client  synch  relate  add  ...</td>\n",
       "      <td>create  flat  hierarchical  add  product  cib ...</td>\n",
       "      <td>Resolved-Completed</td>\n",
       "      <td>PROJ-9996</td>\n",
       "      <td>Synergy 8.4</td>\n",
       "      <td>BL-9394</td>\n",
       "      <td>FS - Client Lifecycle Management (8.4)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>cib  case  stuck  client  synch  relate  add  ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7567</th>\n",
       "      <td>BUG-532794</td>\n",
       "      <td>pdc  exception  null  pointer  excepception  g...</td>\n",
       "      <td>initial  analysis  potentially  kyc  issue  ba...</td>\n",
       "      <td>Resolved-Completed</td>\n",
       "      <td>PROJ-9996</td>\n",
       "      <td>Synergy 8.4</td>\n",
       "      <td>BL-9394</td>\n",
       "      <td>FS - Client Lifecycle Management (8.4)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>pdc  exception  null  pointer  excepception  g...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5434 rows × 11 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         work_id                                              label  \\\n",
       "0     BUG-530021                        issue  report  notification   \n",
       "1     BUG-528121  opd  prod  pre  schedule  validation  stick  p...   \n",
       "2     BUG-529279  inflight  dbupgrde  task  validation  task  no...   \n",
       "39    BUG-530336                            new  app  wizard  error   \n",
       "48    BUG-530860     v  dx  api  automation  need  description  fix   \n",
       "...          ...                                                ...   \n",
       "7561  BUG-534804  qa  incorrect  sale  support  manager  onboard...   \n",
       "7562  BUG-535030       fm  manage  product  fund  complete  default   \n",
       "7563  BUG-532532       pdc  exception  null  pointer  excep  retail   \n",
       "7564  BUG-531668  cib  case  stuck  client  synch  relate  add  ...   \n",
       "7567  BUG-532794  pdc  exception  null  pointer  excepception  g...   \n",
       "\n",
       "                                            description  \\\n",
       "0     hi  team  report  issue  pre  schedule  valida...   \n",
       "1     pre  schedule  validation  stick  progress  go...   \n",
       "2     inflight  dbupgrde  task  validation  task  in...   \n",
       "39    laura  protano  run  error  run  new  applicat...   \n",
       "48    swagger  generator  use  automation  short  de...   \n",
       "...                                                 ...   \n",
       "7561  url  prweb  step  reproduce  login  john  john...   \n",
       "7562  step  start  onboarde  fm  b  add  fund  produ...   \n",
       "7563  initial  analysis  initial  analysis  step  ch...   \n",
       "7564  create  flat  hierarchical  add  product  cib ...   \n",
       "7567  initial  analysis  potentially  kyc  issue  ba...   \n",
       "\n",
       "                       status     team_id                        team_name  \\\n",
       "0        Resolved-Enhancement  PROJ-10002                Cloud - Beyonders   \n",
       "1          Resolved-Completed  PROJ-10002                Cloud - Beyonders   \n",
       "2          Resolved-Completed  PROJ-10002                Cloud - Beyonders   \n",
       "39         Resolved-Completed  PROJ-10010  Zenith for Customer Service 8.5   \n",
       "48         Resolved-Completed  PROJ-10015             DX APIs - CodeNinjas   \n",
       "...                       ...         ...                              ...   \n",
       "7561  Resolved-CannotRecreate   PROJ-9995              Symphony [PCLM] 8.4   \n",
       "7562     Resolved-Enhancement   PROJ-9995              Symphony [PCLM] 8.4   \n",
       "7563       Resolved-Completed   PROJ-9995              Symphony [PCLM] 8.4   \n",
       "7564       Resolved-Completed   PROJ-9996                      Synergy 8.4   \n",
       "7567       Resolved-Completed   PROJ-9996                      Synergy 8.4   \n",
       "\n",
       "     backlog_id                               backlog_name feature_id  \\\n",
       "0       BL-3339                                      Cloud        NaN   \n",
       "1       BL-3339                                      Cloud    FR-9155   \n",
       "2       BL-3339                                      Cloud    FR-8238   \n",
       "39       BL-752  Pega Customer Service Development Backlog        NaN   \n",
       "48    GRP-13931                        Platform UI Backlog    FR-8192   \n",
       "...         ...                                        ...        ...   \n",
       "7561    BL-9394     FS - Client Lifecycle Management (8.4)        NaN   \n",
       "7562    BL-9394     FS - Client Lifecycle Management (8.4)        NaN   \n",
       "7563    BL-9394     FS - Client Lifecycle Management (8.4)        NaN   \n",
       "7564    BL-9394     FS - Client Lifecycle Management (8.4)        NaN   \n",
       "7567    BL-9394     FS - Client Lifecycle Management (8.4)        NaN   \n",
       "\n",
       "                         feature_name  \\\n",
       "0                                 NaN   \n",
       "1           Scheduling Patch Installs   \n",
       "2     Global Operations Console (GOC)   \n",
       "39                                NaN   \n",
       "48                           Web APIs   \n",
       "...                               ...   \n",
       "7561                              NaN   \n",
       "7562                              NaN   \n",
       "7563                              NaN   \n",
       "7564                              NaN   \n",
       "7567                              NaN   \n",
       "\n",
       "                                               combined  \n",
       "0     issue  report  notification hi  team  report  ...  \n",
       "1     opd  prod  pre  schedule  validation  stick  p...  \n",
       "2     inflight  dbupgrde  task  validation  task  no...  \n",
       "39    new  app  wizard  error laura  protano  run  e...  \n",
       "48    v  dx  api  automation  need  description  fix...  \n",
       "...                                                 ...  \n",
       "7561  qa  incorrect  sale  support  manager  onboard...  \n",
       "7562  fm  manage  product  fund  complete  default s...  \n",
       "7563  pdc  exception  null  pointer  excep  retail i...  \n",
       "7564  cib  case  stuck  client  synch  relate  add  ...  \n",
       "7567  pdc  exception  null  pointer  excepception  g...  \n",
       "\n",
       "[5434 rows x 11 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "assigned_bugs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: Setting up the Data for Training\n",
    "Now that our data is preprocessed and ready to go, we need to get the data into a suitable format for training a machine learning model. To do so we\n",
    "1. Get the one-hot-encoded representation of the backlog ids to which each bug belongs.\n",
    "2. Perform a train test split on our text and labels to separate out train data and test data. Here, we perform an 80/20 train-test split\n",
    "3. Use a TF/IDF Vectorizer to convert plain text descriptions into TF/IDF vectors. See the [sklearn documentation](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html) for details.\n",
    "    * Note that you can use a hashing vectorizer instead if you so choose. We found this method to be slightly less effective that TF/IDF, but it does have the advantage of being stateless. To use or test the hashing vectorizer in place of TF/IDF, comment out the TF/IDF code and uncomment the hashing vectorizer code.\n",
    "4. Convert all datatypes into `float32`s. `float32`s can be processed much faster than `float64`s on both GPU and CPU instances. Generally, we convert everything to `float32`s before training a model because the decreased precision almost never decreases model accuracy, but almost always decreases training speed. If you have an NVIDIA GPU available to you that also has a large number of \"tensor cores\" or \"half-precision cores,\" you might experiment with changing the data type instead to `float16` to improve training speed even further."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1. Get the one-hot-encoded representation of the backlog ids to which each bug belongs.\n",
    "backlog_labels = pd.get_dummies(assigned_bugs['backlog_id'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 2. Perform a train test split on our text and labels to seperate out train data and test data.\n",
    "category='combined'\n",
    "random_state = 101 # random state is set to ensure our work is reproducable\n",
    "train_features, test_features, train_labels, test_labels = train_test_split(assigned_bugs[category], backlog_labels, test_size=0.2, random_state=random_state)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4347, 16442)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 3. Use a TF/IDF Vectorizer to convert plain text descprtions into TF/IDF vectors.\n",
    "tfidf_vectorizer = TfidfVectorizer(sublinear_tf=True,\n",
    "                        binary=False,\n",
    "                        min_df=3,\n",
    "                        max_df=0.5, \n",
    "                        norm='l2', \n",
    "                        ngram_range=(1, 2),\n",
    "                        lowercase=True)\n",
    "\n",
    "train_features = pd.DataFrame(tfidf_vectorizer.fit_transform(train_features).toarray()) # Fit the Vectorizer to the train data\n",
    "test_features = pd.DataFrame(tfidf_vectorizer.transform(test_features).toarray()) # Only transform (don't fit) the test data to emulate real-world predictions\n",
    "\n",
    "#hashing_vectorizer = HashingVectorizer(n_features=2**14)\n",
    "#train_features = pd.DataFrame(hashing_vectorizer.transform(train_features).toarray())\n",
    "#test_features = pd.DataFrame(hashing_vectorizer.transform(test_features).toarray())\n",
    "\n",
    "train_features.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train features type:  <class 'pandas.core.frame.DataFrame'>\n",
      "train labels type:  <class 'pandas.core.frame.DataFrame'>\n",
      "train features shape:  (4347, 16442)\n",
      "train labels shape:  (4347, 47)\n",
      "train features value type:  float32\n",
      "train labels value type:  float32\n"
     ]
    }
   ],
   "source": [
    "# 4. Convert all datatypes to float32\n",
    "train_features = train_features.astype('float32')\n",
    "test_features = test_features.astype('float32')\n",
    "train_labels = train_labels.astype('float32')\n",
    "test_labels = test_labels.astype('float32')\n",
    "\n",
    "# Double Check That all data looks correct\n",
    "print(\"train features type: \" , type(train_features))\n",
    "print(\"train labels type: \", type(train_labels))\n",
    "print(\"train features shape: \", train_features.shape)\n",
    "print(\"train labels shape: \", train_labels.shape)\n",
    "print(\"train features value type: \", train_features.dtypes[0])\n",
    "print(\"train labels value type: \", train_labels.dtypes[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 6: Building The Model\n",
    "Now that our data is ready to go, we build the model we'll use to crunch the data. The function `build_tfidf_model` below defines our keras model. We won't go into too much depth here, but here are some general features that define the model we've built and a visual of the model's layers and output:\n",
    "\n",
    "**Parameters**:\n",
    "\n",
    "- Number of Layers: 2\n",
    "- Layer 1 size (# of neurons): 2048\n",
    "- Layer 2 size: 512\n",
    "- Activation Function: parameterized relu (\"prelu\")\n",
    "- Dropout Percentage: 30%\n",
    "- Optimizer: Adam\n",
    "- Learning Rate: 0.0001\n",
    "- Loss: kullback_leibler_divergence\n",
    "\n",
    "**Visual**:\n",
    "\n",
    "![TFIDF Deep Learning Model](../figures/model.png)\n",
    "\n",
    "\n",
    "**Hyperparameter Optimization**:\n",
    "\n",
    "In order to choose the above parameters, we performed hyperparameter optimization using a grid search. You can read more about hyperparameter optimization and grid search [here](https://en.wikipedia.org/wiki/Hyperparameter_optimization). Essentially, grid search attempts to find optimal model parameters simply by generating several hundreds or thousands of different model permutations, each with different parameters, and testing each model's accuracy/ loss. Grid search is considered the must rigorous method to perform hyperparameter optimization. After our grid search was finished, we simply copied the parameters from highest-performing model out of hundreds of models that we tested. The code for our hyperparameter-optimization grid search can be found in [hyperopt_tfidf.py](./hyperopt_tfidf.py)\n",
    "\n",
    "### Additional Notes:\n",
    "- The confidence bounding approach we use, termed ['Monte-Carlo Dropout'](https://arxiv.org/pdf/1506.02142.pdf), does not work when TensorFlow is in eager execution mode. As such, we always must disable eager execution before we build the model.\n",
    "- This confidence bounding approach uses dropout to test model confidence. In order to enable our model to use the approach, we insert a dropout layer in-between each of the Dense (Feature Detecting) layers, and set the parameter `trainable=True`. This trainable parameter must be enabled for the model to leverage Monte Carlo Dropout."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function to build our TF/IDF model:\n",
    "def build_tfidf_model(features, labels, optimizer, activations, drop_rate, lr, layer1_size, layer2_size=None, layer3_size=None, layer4_size=None):\n",
    "    \n",
    "    model = Sequential()\n",
    "    model.add(Input(shape=[len(features.keys())], name=\"TFIDF_Features\"))\n",
    "    model.add(Dense(layer1_size, input_shape=[len(features.keys())]))\n",
    "    \n",
    "    if activations[0] == \"leaky\":\n",
    "        model.add(LeakyReLU())\n",
    "    elif activations[0] == \"prelu\":\n",
    "        model.add(PReLU())\n",
    "    else:\n",
    "        model.add(Activation(activations[0]))     \n",
    "        \n",
    "    if layer2_size:\n",
    "        model.add(Dropout(drop_rate, trainable=True))\n",
    "        model.add(Dense(layer2_size))\n",
    "        \n",
    "        if activations[1] == \"leaky\":\n",
    "            model.add(LeakyReLU())\n",
    "        elif activations[0] == \"prelu\":\n",
    "            model.add(PReLU())\n",
    "        else:\n",
    "            model.add(Activation(activations[1]))      \n",
    "            \n",
    "            \n",
    "    if layer3_size:\n",
    "        model.add(Dropout(drop_rate, trainable=True))\n",
    "        model.add(Dense(layer3_size))\n",
    "        \n",
    "        if activations[2] == \"leaky\":\n",
    "            model.add(LeakyReLU())\n",
    "        elif activations[0] == \"prelu\":\n",
    "            model.add(PReLU())\n",
    "        else:\n",
    "            model.add(Activation(activations[2]))   \n",
    "            \n",
    "            \n",
    "    if layer4_size:\n",
    "        model.add(Dropout(drop_rate, trainable=True))\n",
    "        model.add(Dense(layer4_size))\n",
    "        \n",
    "        if activations[3] == \"leaky\":\n",
    "            model.add(LeakyReLU())\n",
    "        elif activations[0] == \"prelu\":\n",
    "            model.add(PReLU())\n",
    "        else:\n",
    "            model.add(Activation(activations[3]))\n",
    "            \n",
    "            \n",
    "    model.add(Dropout(drop_rate, trainable=True))\n",
    "    model.add(Dense(len(labels.keys()), activation='softmax', name=\"softmax_output\"))\n",
    "    \n",
    "    # Parameters\n",
    "    if optimizer == 'adam':\n",
    "        optimizer = tf.keras.optimizers.Adam(lr)\n",
    "    elif optimizer == 'adamax':\n",
    "        optimizer = tf.keras.optimizers.Adamax() # Use default learning rate for adamax\n",
    "    elif optimizer == 'nadam':\n",
    "        optimizer = tf.keras.optimizers.Nadam(lr)\n",
    "    elif optimizer == 'rmsprop':\n",
    "        optimizer = tf.keras.optimizers.RMSprop(lr )\n",
    "    else:\n",
    "        print(\"ERROR: No valid optimizer passed\")\n",
    "        return None\n",
    "    \n",
    "    \n",
    "    model.compile(loss='kullback_leibler_divergence',\n",
    "                optimizer=optimizer,\n",
    "                metrics=['accuracy'])\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Disable Eager Execution\n",
    "tf.compat.v1.disable_eager_execution()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /root/miniconda3/envs/nlp-workspace/lib/python3.7/site-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "If using Keras pass *_constraint arguments to layers.\n"
     ]
    }
   ],
   "source": [
    "# Build the Model Itself\n",
    "\n",
    "# If the model already exists, make sure to deallocate it & clear system memory before allocating a new model:\n",
    "try:\n",
    "    assignment_model\n",
    "except NameError:\n",
    "    assignment_model = None\n",
    "    \n",
    "clear_memory() # Clear VRAM or RAM\n",
    "\n",
    "\n",
    "# Construct the model itself:\n",
    "assignment_model = build_tfidf_model(features = train_features, \n",
    "                                     labels = train_labels, \n",
    "                                     optimizer = 'adam',\n",
    "                                     activations = ['prelu', 'prelu'],\n",
    "                                     drop_rate=0.3,\n",
    "                                     lr=0.0001,\n",
    "                                     layer1_size = 2048,\n",
    "                                     layer2_size= 512, \n",
    "                                     layer3_size= None,\n",
    "                                     layer4_size= None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense (Dense)                (None, 2048)              33675264  \n",
      "_________________________________________________________________\n",
      "p_re_lu (PReLU)              (None, 2048)              2048      \n",
      "_________________________________________________________________\n",
      "dropout (Dropout)            (None, 2048)              0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 512)               1049088   \n",
      "_________________________________________________________________\n",
      "p_re_lu_1 (PReLU)            (None, 512)               512       \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 512)               0         \n",
      "_________________________________________________________________\n",
      "softmax_output (Dense)       (None, 47)                24111     \n",
      "=================================================================\n",
      "Total params: 34,751,023\n",
      "Trainable params: 34,751,023\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# Visualize the model's layers and output:\n",
    "assignment_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 7: Training the Model\n",
    "We've now got our model and data ready to go. It's time for training. To train the model, we use a portion of the training data for for validation. Overall, we use about 70% of our data for training, about 10% for validation, and 20% for testing. This is a standard split for an ML Experiment. If you'd like more background on the difference between validation/ test sets, consult [this resource](https://en.wikipedia.org/wiki/Training,_validation,_and_test_sets).\n",
    "\n",
    "In the training function, we've added a couple of parameters we should define:\n",
    "- `early_stop`: [Early stopping](https://en.wikipedia.org/wiki/Early_stopping) is a method used to prevent a machine learning model from overfitting the training data. Essentially, early stopping forces a model to stop learning once its validation accuracy fails to improve for more than a set number of epochs. In our case, since we've set `patience` to 30, our model will stop training when it fails to improve its validation accuracy for 30 consecutive epochs. By setting the `restore_best_weights` option to `True`, we restore the 'version' of the model that produced the highest validation accuracy over the course of the training cycle. Restoring the best version of the model is important because model accuracy often starts to degrade towards the end of the training cycle as a model begins to overfit the training data\n",
    "\n",
    "- `workers`: workers defines the number of processes to spin up in order to train the machine learning model. The more processes, the faster the model will train (even when on a GPU instance), but you'll need at least 1 CPU core per process. We set the number of workers to the number of available CPUs on the machine training the model, but you can feel free to use less cores. Note that if you set `use_multiprocessing` to False, the training function will only spin up a single process.\n",
    "\n",
    "- `epochs`: epochs defines the number of times the model should 'loop through' the training dataset in its entirety while it is learning. In practice, we'll never train for even close to 1000 epochs. Due to early stopping, we found our models typically only trained for about 70-80 epochs max.\n",
    "\n",
    "- `batch_size`: The batch size is the number of training examples the model will 'take in' at once. Our model could handle taking in 128 TF/IDF Vectors and corresponding backlog label vectors at once. If your machine has less RAM or VRAM, however, you may need to decrease the batch size to avoid overloading the system memory. Attempt to choose a batch size that is a power of 4.\n",
    "\n",
    "For a more in-depth and less dumbed-down description of epochs and batch size, check out [this resource](https://docs.paperspace.com/machine-learning/wiki/epoch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 3912 samples, validate on 435 samples\n",
      "Epoch 1/1000\n",
      "3912/3912 [==============================] - 1s 375us/sample - loss: 3.7879 - accuracy: 0.1518 - val_loss: 3.6831 - val_accuracy: 0.1977\n",
      "Epoch 2/1000\n",
      "3912/3912 [==============================] - 1s 339us/sample - loss: 3.4906 - accuracy: 0.1710 - val_loss: 3.3326 - val_accuracy: 0.1977\n",
      "Epoch 3/1000\n",
      "3912/3912 [==============================] - 1s 318us/sample - loss: 3.1358 - accuracy: 0.2037 - val_loss: 3.0327 - val_accuracy: 0.2529\n",
      "Epoch 4/1000\n",
      "3912/3912 [==============================] - 1s 317us/sample - loss: 2.8106 - accuracy: 0.2830 - val_loss: 2.7637 - val_accuracy: 0.3356\n",
      "Epoch 5/1000\n",
      "3912/3912 [==============================] - 1s 320us/sample - loss: 2.4814 - accuracy: 0.3783 - val_loss: 2.4944 - val_accuracy: 0.3977\n",
      "Epoch 6/1000\n",
      "3912/3912 [==============================] - 1s 313us/sample - loss: 2.1530 - accuracy: 0.5072 - val_loss: 2.2449 - val_accuracy: 0.5195\n",
      "Epoch 7/1000\n",
      "3912/3912 [==============================] - 1s 297us/sample - loss: 1.8510 - accuracy: 0.6150 - val_loss: 2.0303 - val_accuracy: 0.5655\n",
      "Epoch 8/1000\n",
      "3912/3912 [==============================] - 1s 319us/sample - loss: 1.5864 - accuracy: 0.6912 - val_loss: 1.8437 - val_accuracy: 0.6161\n",
      "Epoch 9/1000\n",
      "3912/3912 [==============================] - 1s 315us/sample - loss: 1.3550 - accuracy: 0.7497 - val_loss: 1.6873 - val_accuracy: 0.6667\n",
      "Epoch 10/1000\n",
      "3912/3912 [==============================] - 1s 318us/sample - loss: 1.1491 - accuracy: 0.8047 - val_loss: 1.5529 - val_accuracy: 0.6874\n",
      "Epoch 11/1000\n",
      "3912/3912 [==============================] - 1s 319us/sample - loss: 0.9778 - accuracy: 0.8517 - val_loss: 1.4433 - val_accuracy: 0.7034\n",
      "Epoch 12/1000\n",
      "3912/3912 [==============================] - 1s 320us/sample - loss: 0.8354 - accuracy: 0.8799 - val_loss: 1.3544 - val_accuracy: 0.7241\n",
      "Epoch 13/1000\n",
      "3912/3912 [==============================] - 1s 319us/sample - loss: 0.7090 - accuracy: 0.9059 - val_loss: 1.2789 - val_accuracy: 0.7425\n",
      "Epoch 14/1000\n",
      "3912/3912 [==============================] - 1s 317us/sample - loss: 0.6038 - accuracy: 0.9284 - val_loss: 1.2139 - val_accuracy: 0.7586\n",
      "Epoch 15/1000\n",
      "3912/3912 [==============================] - 1s 299us/sample - loss: 0.5148 - accuracy: 0.9422 - val_loss: 1.1600 - val_accuracy: 0.7586\n",
      "Epoch 16/1000\n",
      "3912/3912 [==============================] - 1s 315us/sample - loss: 0.4453 - accuracy: 0.9540 - val_loss: 1.1250 - val_accuracy: 0.7655\n",
      "Epoch 17/1000\n",
      "3912/3912 [==============================] - 1s 316us/sample - loss: 0.3747 - accuracy: 0.9632 - val_loss: 1.0864 - val_accuracy: 0.7632\n",
      "Epoch 18/1000\n",
      "3912/3912 [==============================] - 1s 318us/sample - loss: 0.3249 - accuracy: 0.9709 - val_loss: 1.0607 - val_accuracy: 0.7701\n",
      "Epoch 19/1000\n",
      "3912/3912 [==============================] - 1s 316us/sample - loss: 0.2774 - accuracy: 0.9726 - val_loss: 1.0367 - val_accuracy: 0.7678\n",
      "Epoch 20/1000\n",
      "3912/3912 [==============================] - 1s 319us/sample - loss: 0.2400 - accuracy: 0.9770 - val_loss: 1.0165 - val_accuracy: 0.7655\n",
      "Epoch 21/1000\n",
      "3912/3912 [==============================] - 1s 317us/sample - loss: 0.2065 - accuracy: 0.9796 - val_loss: 1.0104 - val_accuracy: 0.7724\n",
      "Epoch 22/1000\n",
      "3912/3912 [==============================] - 1s 317us/sample - loss: 0.1827 - accuracy: 0.9824 - val_loss: 0.9973 - val_accuracy: 0.7701\n",
      "Epoch 23/1000\n",
      "3912/3912 [==============================] - 1s 315us/sample - loss: 0.1603 - accuracy: 0.9836 - val_loss: 0.9892 - val_accuracy: 0.7793\n",
      "Epoch 24/1000\n",
      "3912/3912 [==============================] - 1s 304us/sample - loss: 0.1410 - accuracy: 0.9824 - val_loss: 0.9799 - val_accuracy: 0.7816\n",
      "Epoch 25/1000\n",
      "3912/3912 [==============================] - 1s 316us/sample - loss: 0.1288 - accuracy: 0.9847 - val_loss: 0.9758 - val_accuracy: 0.7839\n",
      "Epoch 26/1000\n",
      "3912/3912 [==============================] - 1s 317us/sample - loss: 0.1129 - accuracy: 0.9872 - val_loss: 0.9714 - val_accuracy: 0.7839\n",
      "Epoch 27/1000\n",
      "3912/3912 [==============================] - 1s 319us/sample - loss: 0.1055 - accuracy: 0.9847 - val_loss: 0.9686 - val_accuracy: 0.7816\n",
      "Epoch 28/1000\n",
      "3912/3912 [==============================] - 1s 316us/sample - loss: 0.0958 - accuracy: 0.9854 - val_loss: 0.9659 - val_accuracy: 0.7793\n",
      "Epoch 29/1000\n",
      "3912/3912 [==============================] - 1s 318us/sample - loss: 0.0893 - accuracy: 0.9852 - val_loss: 0.9684 - val_accuracy: 0.7816\n",
      "Epoch 30/1000\n",
      "3912/3912 [==============================] - 1s 314us/sample - loss: 0.0795 - accuracy: 0.9872 - val_loss: 0.9708 - val_accuracy: 0.7816\n",
      "Epoch 31/1000\n",
      "3912/3912 [==============================] - 1s 314us/sample - loss: 0.0767 - accuracy: 0.9852 - val_loss: 0.9709 - val_accuracy: 0.7793\n",
      "Epoch 32/1000\n",
      "3912/3912 [==============================] - 1s 301us/sample - loss: 0.0708 - accuracy: 0.9859 - val_loss: 0.9690 - val_accuracy: 0.7793\n",
      "Epoch 33/1000\n",
      "3912/3912 [==============================] - 1s 309us/sample - loss: 0.0657 - accuracy: 0.9875 - val_loss: 0.9693 - val_accuracy: 0.7816\n",
      "Epoch 34/1000\n",
      "3912/3912 [==============================] - 1s 315us/sample - loss: 0.0632 - accuracy: 0.9875 - val_loss: 0.9694 - val_accuracy: 0.7839\n",
      "Epoch 35/1000\n",
      "3912/3912 [==============================] - 1s 313us/sample - loss: 0.0591 - accuracy: 0.9877 - val_loss: 0.9691 - val_accuracy: 0.7747\n",
      "Epoch 36/1000\n",
      "3912/3912 [==============================] - 1s 315us/sample - loss: 0.0584 - accuracy: 0.9888 - val_loss: 0.9777 - val_accuracy: 0.7816\n",
      "Epoch 37/1000\n",
      "3912/3912 [==============================] - 1s 312us/sample - loss: 0.0571 - accuracy: 0.9875 - val_loss: 0.9737 - val_accuracy: 0.7816\n",
      "Epoch 38/1000\n",
      "3912/3912 [==============================] - 1s 318us/sample - loss: 0.0520 - accuracy: 0.9888 - val_loss: 0.9771 - val_accuracy: 0.7793\n",
      "Epoch 39/1000\n",
      "3912/3912 [==============================] - 1s 315us/sample - loss: 0.0495 - accuracy: 0.9877 - val_loss: 0.9761 - val_accuracy: 0.7839\n",
      "Epoch 40/1000\n",
      "3912/3912 [==============================] - 1s 314us/sample - loss: 0.0499 - accuracy: 0.9872 - val_loss: 0.9757 - val_accuracy: 0.7839\n",
      "Epoch 41/1000\n",
      "3912/3912 [==============================] - 1s 300us/sample - loss: 0.0489 - accuracy: 0.9865 - val_loss: 0.9785 - val_accuracy: 0.7793\n",
      "Epoch 42/1000\n",
      "3912/3912 [==============================] - 1s 313us/sample - loss: 0.0466 - accuracy: 0.9872 - val_loss: 0.9830 - val_accuracy: 0.7816\n",
      "Epoch 43/1000\n",
      "3912/3912 [==============================] - 1s 320us/sample - loss: 0.0432 - accuracy: 0.9885 - val_loss: 0.9797 - val_accuracy: 0.7816\n",
      "Epoch 44/1000\n",
      "3912/3912 [==============================] - 1s 316us/sample - loss: 0.0416 - accuracy: 0.9885 - val_loss: 0.9889 - val_accuracy: 0.7793\n",
      "Epoch 45/1000\n",
      "3912/3912 [==============================] - 1s 344us/sample - loss: 0.0408 - accuracy: 0.9882 - val_loss: 0.9920 - val_accuracy: 0.7793\n"
     ]
    }
   ],
   "source": [
    "# Define Params for this model:\n",
    "import multiprocessing\n",
    "workers = multiprocessing.cpu_count()\n",
    "use_multiprocessing = workers > 1\n",
    "\n",
    "early_stop = keras.callbacks.EarlyStopping(monitor='val_accuracy', restore_best_weights=True, patience=20)\n",
    "\n",
    "\n",
    "history_tfidf = assignment_model.fit(\n",
    "  train_features, train_labels,\n",
    "  batch_size=128,\n",
    "  epochs=1000, validation_split=0.1, verbose=1,\n",
    "  callbacks=[early_stop],\n",
    "  workers=workers, use_multiprocessing=use_multiprocessing)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Figure size 432x288 with 0 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEcCAYAAADUX4MJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nOzdd3gUVdvA4d/WbHognSSIBEhCh4TQRSnyUkNTEBWxvIgY7ApioSgK+CIfKiooIFYUlAABRRGkg4QWkCYhQEhCKgmp22a+PxZWYtqG9HDu68q1m52dmWdPNvPMKTNHIcuyjCAIgiCUQ1nbAQiCIAj1g0gYgiAIgk1EwhAEQRBsIhKGIAiCYBORMARBEASbiIQhCIIg2EQkjHokKCiIixcv1nYYglBjPv30U1577bXaDsNmBw4c4K677rLpvR9++CEvvfRSNUdUtUTCqEGPP/44ixcvLvb61q1b6dmzJyaTqdL7mD59Oq1btyY1NbXS26rLkpKS6NSpk/UnKCiIjh07Wn+PiYlh+vTptG3btsj7Nm/eDEDfvn3Zu3cvAD/99BMhISHW9/Tt25dXX32V+Ph46/4uX75MUFBQkW0NHz68Sj/Tv+MdOnQoCxcuJCcnp0r3Y4sbn7cqvpOVMXnyZObOnVst2w4KCqJ79+5FPqPRaKR79+4EBQVVyz7rO5EwatDIkSPZsGED/75WcsOGDQwbNgy1Wl2p7efn57NlyxacnZ3ZsGFDpbZVUTV9YGnSpAlHjhyx/gCsX7/e+ntYWBhgSdI3v2/w4MElbq9jx44cOXKEmJgYvvjiC+zs7Bg1ahRnz54t8r6DBw9at2VLGV++fJm+ffva/LluxLt//37eeecdjh49ygMPPEB+fr7N26gvajsZAbi4uLBz507r7zt37sTFxaUWI6rbRMKoQf379ycrK4uYmBjra9nZ2Wzfvp0RI0YQGxvL2LFjCQsLo1evXsyZMweDwWDz9n/99VdcXFyYMmUKUVFRRZZlZWXx6quv0qtXL7p06cKUKVOsy7Zu3UpERASdO3emf//+1n+gm8/CoWgV+sYZ6Jo1a7j77rt55JFHAHjmmWfo2bMnoaGhPPjgg/z999/W9QsLC5k3bx733HMPoaGhPPDAAxQWFjJp0iS++uqrIvEOGzaM3377zebPXlVUKhVNmzZl1qxZhIeH89FHH9V4DAB2dna0b9+eTz75hKysLH766SfrsrVr1zJo0CC6dOnC448/TmJionVZXFwcjz76KOHh4QwcONBaowJLDebNN9/k0UcfpVOnTjz00ENF1rVVTk4OM2bMoFevXvTu3ZtFixZhNpsBuHTpEhMmTKBr16507dqVF198kWvXrlnX7du3L8uWLWPYsGF07NiRixcvEhQUxLp167j77rvp2rUrn3zyifX9JX3nSntvYWEh06ZNo0uXLgwaNIjPPvus3OahiIiIIv8r69evZ8SIEUXek5KSwuTJkwkPD2fAgAH88MMPRfY5ffp0unTpwuDBgzl+/HixdadOnUq3bt3o27cvX375pa3FXCeJhFGDdDodgwYNKvIF/fnnn2nevDnBwcEolUpeffVV9u/fz+rVq9m3bx/ffvutzdtft24dQ4YMYciQIZw/f54TJ05Yl73yyisUFBSwadMm9u7dy8SJEwGIjY1l2rRpvPLKK8TExPDNN9/g5+dn8z4PHjzI5s2bWb58OQB33XUXW7ZsYd++fbRu3bpIG+38+fP566+/WL16NX/++Scvv/wySqWSESNGFDlbP336NKmpqfTp04dZs2Yxa9Ysm+OpSgMGDCiS3GuDk5MTPXr0sMaxdetWli5dykcffcS+ffsIDQ3lxRdfBCw1zMcee4yhQ4eyd+9eFi1axOzZszl37px1exs3bmTKlCkcOHCA4ODgW2pDnz59Omq1ml9//ZWoqCj27NnDmjVrAJBlmSeffJJdu3bx888/c+XKFT788MMi62/atIlly5YRExODSqUC4NChQ/zyyy+sWrWKJUuWEBcXV+r+S3vvRx99RGJiIlu3bmXlypU21QD79+9PTEwM165dIzs7m5iYGPr161fkPS+88AI+Pj7s2rWLDz74gPfff599+/ZZ93np0iV+++03li9fXuR/W5IknnrqKYKCgti5cyerVq1i1apV7Nq1y4ZSrptEwqhhI0aMYMuWLej1egCioqIYOXIkAG3btqVjx46o1Wr8/f0ZO3YsBw8etGm7SUlJHDhwgGHDhuHh4UH37t2tX97U1FR27tzJ7NmzcXV1RaPREB4eDljOVkePHk3Pnj1RKpV4e3sTGBho8+eZOnUqDg4O6HQ6AMaMGYOTkxNarZapU6dy+vRpcnJykCSJH3/8kddeew1vb29UKhWdO3dGq9XSr18/Lly4wIULFwDLWd6gQYPQarWVThgrVqwgLCyMsLAwunbtWqF1vby8yM7OLvJat27drNu7kSSr281xrF69mkmTJhEYGIharWby5MmcOnWKxMRE/vjjD/z8/Bg9ejRqtZrWrVszcOBAfvnlF+u27r77brp06YJWq+X555/n6NGjJCcn2xxLeno6O3bsYMaMGTg4OODu7s7EiRPZtGkTAHfccQc9e/ZEq9XSuHFjHn300WLf4YcffhhfX1/rdwYgMjISnU5HcHAwwcHBnD59utQYSnvvzz//zJNPPomrqys+Pj5MmDCh3M9jZ2fHPffcw+bNm9m8eTN9+/bFzs7Oujw5OZnDhw/z0ksvYWdnR0hICPfddx/r16+37nPy5Mm4ubnh6+vLww8/bF33+PHjZGZmEhkZiVarJSAggPvvv79Ira++qVyjuVBhYWFhNGrUiK1bt9KuXTuOHz9ubfaIj49n3rx5nDhxgoKCAsxmM23atLFpu+vXrycwMJCQkBDA0qQzb948pk2bxpUrV3B1dcXV1bXYesnJyfTp0+eWP4+Pj4/1udlsZtGiRfzyyy9kZmaiVFrOR65evYrBYECv1xMQEFBsG3Z2dgwaNIgNGzYQGRlJdHQ0H3zwwS3HdLPHHnuM559//pbWTUlJKVZm+/fvL7evaePGjcyePRuwnGXm5+db+1TA0mfVpEmTW4ojKSmJd955h/nz51uXy7JMSkoKiYmJxMbGFtmX2Wwu0jl/89/L0dERV1dXUlNT8fX1tSmWpKQkTCYTvXr1sr4mSZJ1/fT0dObOnUtMTAx5eXnIslysT6CkfXl4eFif29vbl9lnU9p7//05bv6sZRkxYgQLFy4EKFbjSk1NxdXVFScnJ+trTZo0sdbe/73Pm/+uiYmJpKamFvt73Px7fSMSRi240W4aHx9Pr169rP8As2bNonXr1ixcuBAnJye++OILtmzZYtM2o6KiSE5OpmfPnoClQzErK4sdO3bQvn17srOzuXbtWon/vJcuXSpxm/b29hQUFFh/T0tLK/YehUJhfb5x40Z+//13Vq5cib+/Pzk5OXTp0gVZlmnUqBF2dnYkJCQQHBxcbDsjR47klVdeITQ0FHt7ezp16mTT565OW7duvaV/7mHDhjFs2DDA0u4+YcIEtm3bdksx5OXlsW/fPiZPngxY/l6TJ08ucYRWUlISXbp0YeXKlaVu78qVK0W2nZ2djZeXl83x+Pj4oNVqS02c77//PgqFgo0bN+Lm5sbWrVuZM2dOkffc/J2pSp6enly5coUWLVoART9rWcLCwkhLS0OhUBAaGlrk/+FG7S43N9eaNJKTk/H29rbuMzk5mZYtW1qX3eDr64u/vz+//vprlXy+ukA0SdWCESNGsG/fPn744YciHWx5eXk4Ojri6OhIXFwc3333nU3bO3LkCAkJCaxZs4aoqCiioqKIjo5m6NChrF+/Hi8vL+666y5mz55NdnY2RqPR2kwwZswYfvrpJ/bt24ckSaSkpFjbhIODg9m8eTNGo5Hjx4+Xm7zy8vLQarU0atSIgoIC3n//fesypVLJ6NGjeffdd0lJScFsNnPkyBFrp36nTp1QKpXMmzevyoerVoTZbCYhIYG33nqLP//8k6effrpW4jAYDJw4cYKnn34aFxcXRo0aBcC4ceNYtmyZdTBBTk4OP//8M2Bpbrpw4QJRUVEYjUaMRiOxsbFF+gN27NhBTEwMBoOBxYsX06FDhzJrFzdqhjd+PDw86NmzJ/PmzSM3NxdJkrh06RJ//vknYPkOODg44OzsTEpKCp9//nl1FVExgwYNYunSpWRnZ5OSksLXX39t03oKhYJPP/2UTz75pFgy8/X1pVOnTrz//vvo9XpOnz7N2rVrrd/RQYMGsWzZMrKzs7ly5UqRwRvt27fH0dGRZcuWUVhYiNls5uzZs8TGxlbdh65hImHUAn9/fzp16kRBQUGRDrZp06YRHR1N586deeONN0odAvpv69ato1+/fgQFBeHp6Wn9eeSRR9i+fTtZWVksWLAAtVrNoEGD6NGjB6tWrQIsX+p3332Xd955h9DQUB566CGSkpIAeO6557h06RLh4eF8+OGH1rPm0owYMYImTZrQu3dvhgwZQseOHYssnzZtGq1atWLMmDGEh4fzv//9D0mSrMsjIiI4e/YsERER1tfefPNN3nzzTZvKoTKOHj1Kp06dCA0NZcKECeTm5rJ27doaH4+/fPlyOnXqRNeuXZk2bRpt2rRh9erVODg4AJaO+CeeeIIXXniBzp07M3ToUOuoNicnJ5YvX87mzZvp3bs3vXr14n//+1+RkXZDhw5lyZIldO3alb/++ov33nuvzHg6depE+/btrT/79+9nwYIFGI1GBg8eTJcuXXjmmWestc/IyEhOnjxJWFgYkyZN4t57762mkiru6aefxsfHh379+jFx4kQGDhyIVqu1ad2WLVtaawn/9v7775OYmEjv3r2JjIxk6tSp9OjRA7B83iZNmtCvXz8ee+yxIt9dlUrFp59+yunTp+nXrx/dunXj9ddfJzc3t/IftpYoxARKQl0RFRXF999/b3PNSqiY6dOn4+3tfct9OvXNt99+y+bNm22uaQjlEzUMoU4oKCjg22+/ZezYsbUdilBPpaamcujQISRJ4vz586xcuZL+/fvXdlgNSo0kjPnz59O3b1+CgoKKXDkbHx/P2LFjGThwIGPHjrUOqyxvmdCw7Nq1i+7du+Pu7s7QoUNrOxyhnjIajcycOZPOnTvzyCOP0K9fP8aPH1/bYTUoNdIkFRMTg5+fHw8++CCffvoprVq1AmDChAmMHj2aiIgI1q9fz48//mi9ErKsZYIgCELNq5EaRlhYWLGRGBkZGZw8edJ6Rjl06FBOnjxJZmZmmcsEQRCE2lFr12HcGMt849YAKpUKLy8vkpOTkWW51GWNGzeurZAFQRBua6LTWxAEQbBJrdUwfH19rRdwqVQqzGaz9TL7G7c6KGlZRV29mockVbybxt3diYyM+jteujqIMimuLpSJJEnk682olAp0WlW1XUkNltuQ6I0SBpMZrVqFnUZZbH+2loksyxjNMgajGb3RjMkklf5eQJYt68gySNcfQcYsyZhMEkazjNFkxmS2PDeZLdvTqpVo1Eq0ahVatRKtRoVGrUSWwWAyYzRJGEwSRpMZg1HCaJJQKECtUqJWK9CoVKjVSjQqBWqVEpVSgUqpRKlUWH4UoFQqrr9uWaZSKVCrFCgVCmv5uLs7kZ6egyzLmCQwmyVL7Obr+79erkaThMEoYTCXXSYACiwXHioU/zwqFQo8XO3x83Qs929QEqVSQaNGJa9rc8LIy8sjJycHZ2dnHB1vLZCbubu7ExISQnR0NBEREURHRxMSEmJtciprWUVIknxLCePGukJRokyKK6lMJEkmX2+yHgwN1w8GBqOEwWhGksFOYzl4aTVK7DQqywFNo0SSZHILjOQVmsgtMF5/bnnMzbc85uQbySkwkptvIK/wn3klVEoFTvYanBw0ONtrcHbQ4uSgQYkCkyRhMkmYrh+kzDcdVG8+2Nw4+AAUGkzkF17/0VsepZvGySgU4GCnxlGnwV6nxlGnxtVZR0GBEZNZsvxIMmazhMksYzRJ6I1mCg0mCg1mboerwNQqSxKRZEtiq4mP7Oak5f3IXuW/sYLKHCV19uxZvv/+e/744w+SkpKQZRmFQoGfnx933XUXY8eOtelK2Lfffptff/2V9PR0GjVqhJubG5s2bSIuLo7p06db73E0f/58mjdvDlDmsorIyMi9pYOcp6czaWk1P9NZXXY7l4ksWxJATr7loJ2TbyCnwIikUJCcmsu1fAPX8gzWx9x8Y5UfGFRKRdFEYK/B2UFz/VGL2SyRcyOZ5BusiSW3wIgsy5Yz5utnyWqV5SxYpVSiUFg+nyRdf5RBxnIGb69VXU8EGhzs1DjoLD86rRq9wUy+3pLYCgpN5BWayC80YpRkkGRU/9rfjed2WhU6648aO43luUZdvLZyM6Wi6Nn0jd+VSoW1FqFRKdFoVJZHtaXF/eakbXm0/K5QKCyJWmOpfdhp/6mBABhv1DpM0vXnlh+zZKkZSJJc5PHGjyUpWxLkjURpMks4O9lh0Jus5XBz+WjV108abj6B0FjiKcuNWpcEyJJsrXm5OFq+H7dCqVTg7u5U4rJSE8bzzz/PuXPnGDJkCOHh4QQGBuLo6EheXh5xcXEcPHiQ6OhoWrRowaJFi24psJogEkbVachlIssyWbkG0rIKSL1aQFqW5Sc1q4CM7EJyC4yYS/ke2WlVuDhocHHU4uKgtT462muw05RwIFCrUCjAYJKK1kCuP1cpFTjaa3C01+Ck0+Bor8bJXoOdpnqbm6pKQ/6eVEZ9KZdbShjbt2/nnnvuKXfjtr6vtoiEUXXqa5lk5+o5ezmb+ORr5BYYKdSbKNCbKDCYLY96y9mx8ab2YoUC3F10eLrZ4+Gqw8VRi/ONph4HrfXM/s6mjcnJLihj71VLlmWuXk3DYCiEGmncqDilUlnkHmGCRd0qFwVarY5GjTyLnYTcUsJoKETCqDr1oUxkWSYju5AzCVmcvf6TctVyQFerlDg7aNBpVTjYqdHZqS1NLtebWjxc7fFqZI+Xmz3urjrUqvIHEdZ0meTkZGEyGXFzc0ehqJuDHNVqZbmdtbejulQusiyRlZWOWq3F2dmtyLKyEkaFRkmlpKQwf/58zp49i7+/Py+++GKpd3gUhJpSoDdx6uJVYuMyOBGfQeY1y2yGjjo1Lf3d6NPRj1YBbjT1drIpCdRlBQW5NG7sXWeThVA/KBRKnJ0bkZmZUixhlKVCCWPOnDkMHDiQqVOnsn//fp5//nmio6MrHKwgVIYsy1zJzCc2LoPYuAzOJmRhlmR0WhVtmjVmUNdGBAW40cTTEWU9aPOvCEkyo1KJec+EylOp1EiSuULrlPnNi4yM5LXXXrNe/3Dt2jX69++Pg4MDLi4uLF68+NajFYQKSr2az94TV9j/VwqpWZZmJj8PRwZ0CaB9c3da+LvW+xqELepDx7dQ993K96jMhPHoo4/y3HPPWScHeeCBBxg8eDC+vr7Ex8czadKkWw5WEGyRX2ji4OkU9p64wt+Xs1EAwXc0YmDXprRr3hgPV/vaDvG29d//PoLRaMRkMpKQcIk77wwEICgoiFdfnWnTNqKi1qLX6xk79sEK799sNjN69FCCg0OYN+/98lcQKq3cTm+z2cwXX3zBzz//zEsvvURISAgXL17E19cXT0/PmorzlolO76pTU2UiyTIn4zPZfTyZI3+nYzRJ+Lo70KOtD93b+NDYRVftMdiqpr8nV65cxMfnjhrbny2Sk5N44omH2bTpd6Bo567JZCpx7u+qsHfvblasWEZSUiJff/0DjRu7V8t+bpAk6fp1ILdWw6tLnd43lPR9qlSnt0ql4vHHH2fIkCHMnTsXrVbL9OnT60WyEOqXQoOJPcevsPXQZVIy83HUqend3pee7Xxp5uMsmmLqiREjhtCv370cPnyQ5s1bMGnSFGbNeo28vDwMBgM9evRkypRnAVi+fCkFBQVERj7H5s0b+e23X3B2duH8+TicnZ14++0FuLt7lLifTZvWM2LEKE6cOM4vv2xi/PgJAOTm5vLBBws5ffokCoWSDh068sIL0zAajSxduoQDB/aiVKpo0sSPd9/9X5EY/h3T8uVLiY8/T15eLikpV/j005V8+eUKjh49jNFoxM3NjVdffRMfH0uz/Z49u1ixYhkmkwmlUsFrr83mzz/3kZyczLRprwKQmZnBI488wJo1G9Dp6s7Jjy3KTBh///03Cxcu5PLly7Ro0YIZM2Zw5swZnnjiCUaNGsWECRPEP7FQaWlZBfx+6DK7YpMp0Ju409eFScNaExbsdVv0SdyqPceT2R2bXC3b7nU9Ud+qvLw8PvvMMn+NXq9n/vxFODg4YDKZeOGFSPbv30u3bj2KrXfq1ElWrfoOb28f5s9/m7Vrv+fJJ58u9r6srCwOHYrhtddm0bRpMxYsmGtNGB98sBB7e3u++OI7lEolWVlZAHz11UqSkhJZseIbNBqN9fXynDx5ghUrvsHNzTKa6KGHJlqTy8aNUXzyyQfMnv0uly5dZP78t1my5DMCAppiMBgwmYwMHRrBQw/dz9Spz6LV6tiwYR0DBgysd8kCykkYL7zwAuPGjaN79+7s3buXmTNnsmzZMrp3787HH3/M/fffz5o1a2oqVqGBOZeYzc/7L3L0XDpKhYLQIE8GhAUQ6Oda26EJlfSf/wyxPpckiY8/Xszx47GATEZGBn//fbbEhNG+fQe8vX0AaNOmLQcPHihx+1u2bKJnz944ODjSvn1HTCYzJ07E0rZte/bu3cXnn3+NUmk52bhxoN+7dzeRkc+h0WiKvF6e7t17Fnnv/v17+OmnNRQU5GM2/zPK6ODBA3Tr1oOAgKYAaLVatFotAD173sXPP29iyJAINmxYx+LFn9i077qmzISRlpbGyJEjcXBwwNPTk59++gkAOzs7nn/+eSIiImokSKFhMZokftoZx5Y/E3Cy1zC42x3c08mvTvVN1Ac921WuFlCdHBz+GYzw/fffkJNzjWXLvsDOzo758+diMOhLXO/GARZAqVQVOSDfbPPmjVy9epUxY4YBlmaoTZs20LZt+wrHqlKpkOV/+hYMBkOR5fb2DtbnV64k8+GH7/PZZ1/SpIkfx48fY/bs18vdx5gxY5kz53VcXd1o1uxOa1Kpb8qs7z/xxBOMGTOGF198kfHjx/PEE08UWX4rNwMUbm+Jabm8/WUMW/5M4J5Ofrz3VA9G9wkUyaIBy8nJwd3dAzs7O9LSUtm9e0eltnfq1F/k5OSwfv0vrF27kbVrN/LVV9+zfftWCgsL6dGjN9999yU3xvPcaHrq0aMXP/zwHUajscjr/v4BnDlz2nKb+Pw89u7dVeq+8/LyUKs1uLu7I0kSUVE/WpeFh3dj//69JCRcAiyJJz8/D4DAwBa4urrxwQfvM2rUfZX6/LWpzBrGE088QUREBMnJyTRt2tTmKpwg/Jssy/x+6DJr/ohDp1XxzJj2dGxRcmem0LDcd9843nhjGg8/fD+ent6Ehnap1PY2bdpA//4Di/Sfenp60apVMNu3b2Xq1Bf44IOFPPzwWFQqFZ06dea5517moYcmsnTpRzz66HjUag3+/v68/fYC+vTpy++//8aDD47B29uHoKCQUvcdGNiCe+7pz0MP3Y+rqxvdu/fk2LEjAAQENOWVV15j5sxXMZslVColr702m8DAFgAMHz6CTz75iB49elfq89emUofVGgyGItXD0tj6vtoihtVWnVstk+xcPcs3n+LE+UzaB7rz6OAQXB3r7nemIsSw2uLq4vDRumD+/LcJCGhq7ZyvCyo6rLbUJqmIiAg+++wzUlJSSlyemprKZ599xogRIyoRrtDQHT2XzhvL/+TspSwevrcVz45p32CShSDYIj09jQceGMXly5cYNer+2g6nUkptkvrmm2/47LPPiIiIwNXVlTvvvNM6H0Z8fDw5OTmMHDmSr7/+uibjFeoJWZbZuPcCUbviaertxKRhbWjiUfmZGgWhvvHw8OS7735qEDWvcq/0NhgMxMbGcubMGXJycnBxcSEoKIj27dtbh6fVZaJJqurYWiZ6g5nlm08RczqV7m18mDgoCI1aVQMR1jzRJFVcQzgwVoe6WC5VfqW3VqslLCyMsLCwqolQaNAysgv58MdYEtJyuf+eFgwMDxAXdwpCAyHukyxUmbMJWSxZdxyTWebZMR1oH1i99/YRBKFmiYQhVImdx5L4assZPNzseWZ0O3zdRX+FIDQ0ImEIlSJJMqu3/c3WmMu0vbMxT0a0wVFX9/u2BEGoOHFnN+GWGYxmPok6wdaYy/QP8+fZ+9qLZHEbefHFZ4iKWlvkNVmWGTVqGEeOHCp1vblzZ/Hjj98Dlvkwvv/+mxLft3nzRl5//ZVy49i58w9Onjxh/f306ZM23a6josxmMyNGDGL69BeqfNv1Rak1jFdeKf8PBZZO8bfffrvKAhLqh9wCIx+sjSUuMZtx/Vpyb5eA2g5JqGFDhgxn9eqvGTFijPW1I0cOoVQq6Nixs03buHndW7Vr1x8EB4fQunVbAIKDWzNzZtUfkw4c2IeHhyexscfIzMyo8/NvVIdSE8bmzZuZPHlyuRtYsWKFSBi3mbSsAt7/4RgZ2YU8NaItYcFetR3Sbcl4dg/GMzurZduaoLvQtOpZ5nt69+7DwoXvcuFCPM2a3QlYbtsxZMhwzp+PY+HCeRQWFmAwGBg+fCT33z++2DZunnvCaDSyaNECDh+OwdXVjZYtg6zvi4s7V+L2DhzYx+7dO4mJ+ZONG9czdux4vL19WLJkMcuXfwXAzz9H8913X6FQKGjSxJ9XXplBo0aNa3z+DX9/f+bOfa/G5t948cVpQNXOv1FqwvD19SUyMrLcDURHR1cqAKF+iU++xuI1xzBLMi+N60irAHF/sduVRqNhwIBBbN68gSlTniU/P49du3bw9NNT0ekc+L//+xitVkt+fj6TJj1CeHh3a2Ipyfr1P5KcnMTXX6/BZDLx9NP/xdfXcmD09fUtcXtdu3anV6+7CA4OYfTosQAcPhxj3eb58+f49NOPWL78azw8PPjss09YtOg95sx5F6jZ+Tdyc7NtKteqmn/jqaem4uDgUKXzb5SaMH777TebNvDLL79UOgihfog5lcKCb7NK790AACAASURBVI/gZK/hlfs7iCu3a5mmVc9yawHVbciQ4bz00lSefDKS33//jXbtOuDl5U1qahoffTSPc+fOolAoSU9P49y5s2UmjMOHDzFo0FDUajVqtZqBAwcRG3sUgMLCwgpvz7LNGLp374mHh6XWEBExiokT/6np1Oz8G41sunCvqubf2LJlM8OGjajS+Tds6vQurclp7ty5VRKEUPftP3mFt1YcwLuxPa9NCBXJQgCgZctWuLt7sn//XjZvtjRHASxduoTGjd1ZseIbVq36jpCQNsXmmaiIqt7eDRWZf+PgwQOMGTOMMWOGkZV1lU2bNtzSPm9l/o1Zs+by1Vc/8Oqrb9r0uceMGUtU1Fp2795RpfNv2JQwbkyc9G8bNtxagQn1S+rVfL74+TQhzRozbXxn3JzsajskoQ4ZMmQ4K1YsIyHhEr179wEgNzcHLy9v1Go158+f49ixo+VuJzQ0jF9+2YzJZEKvL+S33/5pvShre46OjuTm5pa4zc6dw9i3bw8ZGemApUmnS5fwCn2+qpt/4ypQc/NvuLi4Vvn8G2Veh7F2rWXInNlstj6/ISEhQcyPcRuQZJkVm06hUip4cXwomEy1HZJQxwwY8B+WLFnM8OEjrc0vjzzyOG+99SabNq0nIKApHTt2Knc7w4eP4ty5czz00H24uroRHNyGq1czyt3ewIGDmTt3Ntu3/27t9L6hefMWTJ4cyfPPP32909uPl1+eUaHPV1XzbwQEBPDWW/NrbP6NYcNGsHTpkiqdf6PMmw8+/PDDABw6dIjQ0NB/VlIo8PDwYMKECXTs2LHKgqkO4uaDlfPrn5dYve0cjw8JYUTfVqJM/kXcfLC4uniTvbqgpstl3ry3aNr0jjLn36jSmw9+9ZVlWNqiRYt4/vnnKxqvUM8lZ+Tx487zdGzhQY+2PuWvIAhCrUtPT2Pq1Cdxd/fguedertJt23RrkBvJIiMjg/z8/CLLAgLEBVsNkVmS+Dz6FFq1kkf+E1SnLh4SBKF0N+bfqA42JYxdu3YxY8YM0tLSiryuUCg4depUtQQm1K5fDlwiPvkaTw5vg6vo5K5TZFkWCVyotHKmQiqRTQlj9uzZTJkyhZEjR1bJxR9C3XY5NZeoXfGEBXsRHiKu4q5LLEM/TajV4p5dQuWYzSaUyopNbGbTsNpr164xbtw4kSxuAyazxOebTuKoU/Pwva3EmWwdY2/vRE5OVpFx/IJQUbIskZNzFXv7kju3S2NTDWP06NH8+OOPjBlT+RuFlWT79u0sXrwYWZaRZZnIyEjuvfde4uPjmT59OllZWbi5uTF//nyaNWtWLTEIFtF7L3ApJZfIUe1wdtCWv4JQo5ycXLl6NY2UlMtAxZsUaoJSqUSSREL7t7pVLgq0Wh1OTq4VW6u8Ob0Bxo8fT2xsLH5+ftZL7G/45puSb01sK1mWCQ8P55tvvqFVq1acPn2aBx54gEOHDjFx4kRGjx5NREQE69ev58cff+TLL7+s0PbFsFrbXbySw9tfxhAe4sV/h7Uptvx2LJPyiDIpTpRJyepLuVRqTm+A++67j/vuq7qrBf9NqVSSk2MpyJycHLy8vLh69SonT55k5cqVAAwdOpS33nqLzMxMGjduXG2x3K4kWebLLadxctAwfkCr2g5HEIQ6yKaEMXLkyGoLQKFQ8H//939MmTIFBwcH8vLyWLZsGcnJyXh7e6NSWTplVCoVXl5eJCcnVyhhlJYpbeHp6XzL69Y3W/+8RHxyDi+O70yzgNLL93YqE1uJMilOlEnJ6nu52JQwZFlmzZo1REdHc/XqVTZu3MjBgwdJS0tj8ODBlQrAZDKxdOlSPv74Y0JDQzl06BDPPfccCxYsqNR2bxBNUuUr0JtYGf0XgX4utA5wLfVz305lYitRJsWJMilZfSmXspqkbBoltXjxYtauXcvYsWNJTk4GwMfHh88//7zSwZ06dYrU1FTrrUdCQ0Oxt7fHzs6OlJQU690jzWYzqamp1vvjC1Unet8FruUZGN9fjIoSBKF0NtUw1q1bx7p162jcuDGzZs0CwN/fn4SEhEoH4OPjw5UrVzh//jzNmzcnLi6OjIwM7rjjDkJCQoiOjiYiIoLo6GhCQkJE/0UVS7maz28HE+jZzoc7fV1qO5zbiiyZkfV5yPpcKLQ8yvo85MLrjzc/L8y1vM9cxs0fNXYo7BxR6Jwsj3aOKOycbvr9puc6J9Dao1AoLRdwmQ0l7pfSxsQoFCi0DkX3p3MCdfkXecqyBIaCIp+r+Ge9Xh7Xy0WhUqPya4u6aXtUvkEoVGVfhyLlZ2FOPY+cX/qkRQq1FoXuernYOYHOEYXWEYVSiWw2IeekIWWnWH6u3XhMBVMptxdXKFA4uKF09bb8uFx/dPVBoXOyjAI16q9/rps+oz6/7HK2c7ge442ydgK1tlZO7mxKGGazGUdHy/wHN4LMy8vDwcGhrNVs4unpyaxZs3j22Wet237nnXdwc3Nj1qxZTJ8+nY8//hgXFxfmz59f6f0JRX3/+zlUKiWj+wTWdigNmizLSJkJmC4dw3wpFvPVy2AoKH0FhcJ6kEDnhMLRDWVjPxSq0oY6y8gmveVAVJiLlJ2CXJgLhvxS3m/ZB1oHMOnLTkQVoVSTr3NAlks5mElmZEM+lHUdida+SHJTungh63MxntqG8cSvoLZD7dcaVdMOqAPaodA5I6VfxJx6HnNqHObUOOTcjFv/DFp7MOqLxqh1QOnqjcorEIWmlOvRZAkpLxNz6nlM5/8smgQ09uRIxqorZ5UahdYBKLmcle4BOAx+qWr2dRObEkafPn149913mTHDcltgWZZZvHgx99xzT5UEMXz4cIYPH17s9cDAQNasWVMl+xCKOxGfwdFz6Yy5O1DMcVFBsqEAc1o81xJzMRqU1w9wjv+cCaq1yIYCTIknMSccw3QpFjnfMk+C0uMONC16oLB3/leN4J+D5I2z/0rHKUnIhrybajD/nLVbznLzLWerOqfisdg5QGlXAksSsqHgpm3mwvXagZ3SRGFBKWfhSuVNn9WxyJkzN2pFpexTNukxJ53CdCkW06VjmC4eQQ+gUIFsabpWOLmj8gpE1XaA5eDu7GFJjMU2JoPJcFNNpmjtRqG1R+nqg9LVG4WrtyXOCpzRy2YjUk4a8o0aSk46Di5OFJi1xT63QusAylL+1tf/fvK/aqDo864n3lKKuVH1NN3bdB1Gbm4u06ZNY+fOnZhMJuzs7OjZsyfz58/HyenWRyHVBNHpXTKTWWLWyoOYTBJvPdEVjbr8g1N9KxNZn4c5Lf76Wed5pMzLKNR2RQ9QRQ7WjiVW/ZFlpKxEyzZSrm/raiJlXjin1oLZbDmQaexR+7dB3bQDqoB2KB0a9jwyNfE9kWUZKSsZc8Ix5MI8lF7NUXk1r9NlW1/+fyp9HYaTkxNLliwhPT2dpKQkfH198fT0rNIghZr1x5FEktLzmDq6nU3JojbJJj3mtAtIN5oc0i9eP1P911mxzgmF2g5z5mWk1PNIWUnXt6BA2cgXlU9LMJuQ9XlIuenI6Rct/QKltUkDqNSAEszX32PniMorEG3zMFRegXg0DyQjOa1Ym7usz0OhVKMKaIfKOxCF0qZ/NcFGCoUCVaMmqBo1qe1QbisV+hbrdDq8vb2RJImUlBQAvL29qyUwofrk5BuI2hVPm2aN6NjCo/wVqoEsS5gTjiOV1tYsS0hXkzCnxCFlJljbkxXOnqg8mwEKy0G6IBvpaqKlg9ZYaHmPzhmlV3O0LbpZmie87rze3lvKroo0TZRQ9ZfMqNybWg78Lt5FmiY0bs6ojJXvyxOE+sCmhLF3717eeOMNkpKSitwSV9zevH6K2hVPocHMuH4ta3ykhSzLmC4cxnAoypIIyqLRofJqjrbDYFTegSi9AlHalz6SS5ZMYCi0NDVV4HMp1FoUai04NrJ5HUG4HdmUMF577TWmTJnC4MGDxR1r67mE1Fz+OJpI387++HnWXP+TLMuYLx5FfygKKeMiCldvdPdMQuXXutR1FDoXFKV1Bpb0fqUadHW7T00Q6jObEoZer2fUqFHW23QI9dcP28/hYKcmotedNbI/WZYxJxxDf2g9Ulo8ChcvdHf/F3WLbqWOhhEEoW6yKWFMnDiRzz//nEmTJokrgeuxvy5k8ld8JmP7tsDJvvon4DGnnqdw37dIKedQOHug6/M46pbdRQewINRTNv3n3nvvvTz++OMsXbqURo2KtvP+/vvv1RKYULUkWWbN9nO4u+jo29m/eveVn4X+z7WYzu5GYe+CXe+JaIJ6iUQhCPWcTf/BzzzzDGFhYfznP/8RfRj11J8nU7iUkst/h7autmG0ssmA4cSvGI5Eg9mEtsNgtJ2GodDaV8v+BEGoWTYljMuXLxMVFYWyAh2QQt1hNEn8tPM8Tb2c6Nqm6odBy7KM6eJh9PtWI+ekob6jE3bdxqF0FUOuBaEhsSlh9OvXj/3799OjR4/qjkeoBtsPXyY9u5AXx3ZEWcV9UFJOGoW7VmG+fAJloyboBr+E2r9tle5DEIS6waaEYTAYeOqppwgLC8Pd3b3Isqqat0KoHvmFRjbuvUCbZo1oc2fV3elXliWMf/2O/s+1ANh1H4+mTV/RTyEIDZhN/90tW7akZcuW1R2LUA02779EXqGJMXe3qLJtSlnJFO5YgTnlb1T+bdH1nojSuXauGBcEoebYlDAiIyOrOw6hGmReK+S3mAS6tfHmDp/KTw0pm03oj0RjOBwFajt0dz+BumVPMdRaEG4TNrcfnD9/ntOnT5OfX/T++mPGjKnyoISqEbU7HlmWGdW7eaW3ZU49T+L6rzCkxKO+Mwy7ng/V6TuDCoJQ9WxKGJ9++ilLliwhODi4yLBahUIhEkYddTktlz3HkxkQFoCH260PazWnX0AfE4X50lFUjm7oBkSiuTOsCiMVBKG+sClhrFq1ijVr1hAcHFzd8QhVZO0fcei0aob2aHZL65szLmE4FIXpwmGwc0QbNoomd48k45q5agMVBKHesClh6HQ6mjevfLOGUDPOJmQRG5fBmLsDK3wLEHNmAoZD6zHFx4DWHm3oSLTtBqDQOqC0cwDq/gQwgiBUD5sSxrPPPsvbb79NZGQkHh5FR8OIi/nqni1/XsLJXkP/UNtvASLr8yjctxrT2V2g0aHtHIG23b2W6UIFQRCwMWFMnz4doMj82rIsi/kw6qCM7EKOnktnUNc70GpsuxusMT4G/e6vkAtz0LQfhF3HIZYpTAVBEG5iU8IQNxisP/44mgjA3Z3Kn7pSys9Cv+drTPExKN2bYj/oeVQezao5QkEQ6iubEoafn191xyFUAaNJYuexJDoEeuDhWvrIKFmWMf29l8J934JJj7bLGLQd/iOu0hYEoUw2HSFefvnlEi/O0mq1+Pj40L9/fzGCqg6IOZNKTr6RvqGlJ3gpP4vCHcsxJxxH6d0CXZ/HULmVXxsRBEGwqcfa2dmZ33//HVmW8fHxQZZltm3bhlKpJC4ujrFjxxIVFVXdsQrl2Hb4Mt6N7GndrOR7RsmyTOH2ZZiTzmDX40Echs0QyUIQBJvZVMO4cOECy5YtIzQ01PrakSNH+OCDD1i5ciU7d+7knXfeYcSIEdUWqFC2i1dyiEu8xrh+LUu9I63p772YE09i1/NhtG361XCEgiDUdzbVMI4dO0aHDh2KvNa2bVtiY2MB6N27NykpKVUfnWCz7Ucuo9Uo6dXOp8TlUmEO+v2rUXoFoml9Tw1HJwhCQ2BTwggJCWHRokXo9XoA9Ho9ixcvtvZbXL58GVdX1+qLUihTXqGR/X+l0K21Dw66ki/U0+//Hlmfj673RBQKce2MIAgVZ1OT1Lx583jppZcIDQ3F1dWV7Oxs2rZty3vvvQdAVlYWM2fOrNZAhdLtiU3GYJLo27nkzm5T0ilMZ3ej7TgElXtADUcnCEJDYVPC8Pf3Z/Xq1SQlJZGWloanpydNmvzTWdquXbtqC1AomyTLbDuSSAs/V5p6F7+FuWwyULjrCxTOnmg7R9RChIIgNBSlJowbV3IDSJIEgI+PDz4+PkVeE7cGqV0n4zNJvVrAiF53lrjccDQaOTsF+8EvoVBrazg6QRAaklITRmhoKIcPHwagdevWxa7DELcGqRu2HU7ExUFDaJBXsWXmq4kYjm5C3aK7mGdbEIRKKzVhbNq0yfpc3BqkbkrPKuDYuXSG9LgDjbpoTU+WJfS7VoFGh133B2opQkEQGpJSE4avr6/1+b9vDVJYWIhSqUSrFU0ctWn70URQwN0di3d2G0/vxHzlLLq7HkNp71IL0QmC0NDY1AExf/586zUXf/zxB+Hh4XTp0oVt27ZVa3BC6YwmM7uOJdOppSeNXXRFlkn5WegP/IDKNwh1UO9ailAQhIbGpoSxceNGWrZsCcCSJUt47733+OSTT1i0aFGVBKHX65k5cyb33nsvw4YN44033gAgPj6esWPHMnDgQMaOHcuFCxeqZH8NwZG/08ktMHJPCUNp9fu+A5Ph+jUXJV/1LQiCUFE2DastKCjA3t6eq1evkpCQwMCBAwFITEyskiDee+897Ozs2LJlCwqFgvT0dABmzpzJ+PHjiYiIYP369bz55pt8+eWXVbLP+u7YuXSc7DWENG1U5HVTQiymuANoQ0eidPMtZW1BEISKs6mG0axZMzZs2MA333xDz549AcjMzESn05WzZvny8vKIiori2WeftZ4Ne3h4kJGRwcmTJxk6dCgAQ4cO5eTJk2RmZlZ6n/WdJMkcP59Ju+buKJX/1CBko57C3V+idPNF23FwLUYoCEJDZFMNY+bMmbzzzjtoNBrmzp0LwO7du63JozISEhJwc3Pjo48+4sCBAzg6OvLss8+i0+nw9vZGpbLMGqdSqfDy8iI5OZnGjUu+G+vt4nzSNXILjHRo4V7kdf2hKOScdHTDXkWhqthc3oIgCOWxKWG0b9+e1atXF3lt+PDhDB8+vNIBmM1mEhISaN26NdOmTePYsWNMnjyZxYsXV3rbAO7utz7VqKdn8Sun64KfDyagVCro0+UOnOwtiUF/JZ6c41tw7tgfz/Zh1bbvulomtUmUSXGiTEpW38vFpoSxf/9+/Pz8CAgIIDU1lYULF6JUKnnhhRfw9PSsVAC+vr6o1Wpr01OHDh1o1KgROp2OlJQUzGYzKpUKs9lMampqkeG+tsjIyEWS5ArH5enpTFpaToXXqwn7YpNp6edKQW4hBbmFyJJE/oYlKHROyB1GVlvcdblMaosok+JEmZSsvpSLUqko9UTbpj6M2bNnW5uG5s+fj8lkQqFQWEczVUbjxo3p2rUre/bsASwjozIyMmjWrBkhISFER0cDEB0dTUhIyG3fHJWRXcjltFw6tPCwvmY8+TtSWjx23cejsHOsxegEQWjIbKphpKSk0KRJE0wmE7t372bbtm1oNBp6966aMf6zZ89mxowZzJ8/H7VazYIFC3BxcWHWrFlMnz6djz/+GBcXF+bPn18l+6vPYs9nANA+0NJ/IeVmoj/4Iyr/tqgDu9ZmaIIgNHA2JQwnJyfS09P5+++/CQwMxNHREYPBgMlkqpIgAgIC+Oqrr4q9HhgYyJo1a6pkHw3FsXPpeLrp8HV3AEC/92uQJHS9HhHXXAiCUK1sShgPPfQQY8aMwWg0MmPGDAAOHz5M8+bNqzU4oSi90cypi1e5q0MTFAoFxguHMF04jDb8fpQuletLEgRBKI9NCWPSpEkMGDAAlUpF06ZNAfD29rYOsRVqxumLVzGaJDq0cEc2FKDf8zXKxgFo299b26EJgnAbsHkyizvvvNOaLCRJ4uLFiyxZsqTaAhOKi43LwE6jIiigEfr93yHnZ6G7ayIKpU15XxAEoVIqdKQ5ffo069atIzo6msLCQiIixAxuNUWWZY7FpdO6WSMUl49hPL0TbcehqLwCazs0QRBuE+UmjIyMDDZs2MC6deuIi4sjLCyM/Px8Nm7ciL+/f03EKACJaXlkXtMzKtyTwl0foXRvijZ0RG2HJQjCbaTMhDFp0iT27t1Lq1atGDlyJIMHD8bb25tevXphb29fUzEKwLG4dECmTepmZH0+9kNeQaESTVGCINScMvswDh48iKOjI3fddRd33XUX3t7eNRWX8C/H4jIY5JmI8vJR7MJHo2osaneCINSsMk9R9+zZw6+//sq6detYunQpwcHBDBs2DKPRKMb816DcAiNXkxPp32g3Kt8gNO0G1nZIgiDchsqsYTg4ODBixAhWrVrF1q1bGTBgAN9//z3Z2dm8/PLL7Nixo6bivK0dj0tjvMMelEoFurv/i0Jh8+A2QRCEKmPzkcfPz48pU6awZcsWvvvuO/z8/HjllVeqMzbhOkPsFlpoUtD1fAils0f5KwiCIFSDW+o17dSpE506deL111+v6niEfzGmX6J9zi4u27ckuFWv2g5HEITbWKWG2Wi12qqKQyiBbDaR89unFMhaCjs+IPqNBEGoVaIxvA4zXTqGJieJnwq6ERzUtLbDEQThNicSRh1mTvwLA2oM3m2xtxPXXAiCULtEwqjD9Al/cc7gTbsWXrUdiiAIgm19GOPHjy+x/Vyr1eLj48OAAQPo27dvlQd3O5PyrqLMSeGsKZS+zd1rOxxBEATbahjh4eEkJibSpUsXhg8fTpcuXUhKSqJt27a4u7szY8YMPvvss+qO9bZiTjwJQKIqgCbXJ0sSBEGoTTbVMPbs2cPy5csJDPznzqjDhg1j+vTprFmzhnvvvZcXXniB//73v9UW6O3GlHiSfNkOF7/mYnSUIAh1gk01jPPnzxMQEFDkNT8/P+Lj4wFo3749GRkZVR/dbUqWZYyXT3LG6EPQHY1rOxxBEATAxoTRpUsXXn31VS5evIher+fixYu8/vrrhIaGAnDmzBk8PcUUoVVFzk5BUXCVv40+BDV1q+1wBEEQABsTxrx585AkiSFDhtChQweGDBmCJEm8++67AGg0GhYuXFitgd5OTEmW/ovLqgCaeDjWcjSCIAgWNvVhuLm5sWjRIiRJIjMzk8aNG6NU/pNrmjdvXm0B3o7MiSfJlh1x92uKUvRfCIJQR9h8NdiFCxeIjo4mNTUVLy8vhg4dSrNmzaoxtNuTLEsYE09x2uBDUNNGtR2OIAiClU1NUtu2bWPUqFHEx8fj6upKfHw8o0eP5vfff6/u+G47UkYCCkMeZ0X/hSAIdYxNNYxFixbx8ccf061bN+trBw4c4K233qJfv37VFtzt6Mb1F5dVAfh7OdVyNIIgCP+wqYZx5coVwsLCirwWGhrKlStXqiWo25kp6STpshs+fk1E/4UgCHWKTQkjODiYFStWFHlt5cqVhISEVEtQtyvZbMKUfJaTem/RHCUIQp1jU5PUrFmzeOqpp/jyyy/x9fUlOTkZe3t7Pv300+qO77ZiTjuPwqTnb6MvI0WHtyAIdYxNCSMwMJDNmzdz9OhR6yipDh06oNFoqju+24o58SQycFnpR4DovxAEoY6xeVitWq0u1o8hVC1z4kmu4Im/vxdKpei/EAShbik1YfTp08emm9798ccfVRnPbUs26jGlxvFXQTBBHUT/hSAIdU+pCeO9996ryThue+YrZ1BIZs4afbkvQPRfCIJQ95SaMMLDw63PU1JS8Pb2LvaeU6dOVU9UtyFT4inMqEhU+nKHj+i/EASh7rFpWO3jjz9OVlZWkddiY2PF/BdVyJx0kkS8aebvjkopZs4VBKHusenIdP/99/PYY4+Rl5cHwOHDh5kyZQpz586t1uBuF3JhLub0SxzP9yIoQPRfCIJQN9mUMCZMmEDfvn2ZNGkSO3bsIDIykgULFtCnT58qDeajjz4iKCiIs2fPAnD06FGGDx/OwIEDeeyxxxrsJE2mpFMokPnbJG44KAhC3WVz20dkZCTt2rXj+eefZ/HixfTo0aNKA/nrr784evQofn5+AEiSxMsvv8ybb77Jli1bCAsL43//+1+V7rOuMCedwqTQkqzwopmPc22HIwiCUKIKDauVJAlZlnn55Zetr1XFsFqDwcCcOXNYuHAhEyZMAODEiRPY2dlZr/0YN24c/fr1s07a1JCYE09yUfYh0K8RapXovxAEoW6qE8NqFy9ezPDhw/H397e+lpycTJMmTay/N27cGEmSyMrKws3N9nZ+d/dbH3Hk6Vn9Z/vGq1fIyb5CbH4Ynbp618g+K6Oux1cbRJkUJ8qkZPW9XGwaVludjhw5wokTJ3jppZeqZfsZGblIklzh9Tw9nUlLy6mGiIoq3Pk9kkLFEf0ddGvsUCP7vFU1VSb1iSiT4kSZlKy+lItSqSj1RNum9o/IyEhiYmKKvBYTE8MzzzxT6eAOHjxIXFwc/fr1o2/fvly5coXHH3+cixcvkpSUZH1fZmYmSqWyQrWLuk66lorxzG4uOHciX+XMnb4utR2SIAhCqWxKGAcPHqRTp05FXuvYsSMHDhyodACTJk1i9+7dbNu2jW3btuHj48Py5ct54oknKCwstCaq1atX85///KfS+6tL9IfWg1LNr3ltaOHnikYt+i8EQai7bLr5oFarpaCgACenf6op+fn5qNU237uwwpRKJQsWLGDmzJno9Xr8/Pwa1O1KzFlJmM7thZABnN4tE9Gr4dScBEFomGw64vfq1Ys333yTOXPm4OTkRG5uLnPmzKF3795VHtC2bduszzt37szGjRurfB91gSEmClRaTjmGI3OJ1nc2ru2QBEEQymRTG8j06dPJzc0lPDyc7t27Ex4eTm5uLjNmzKju+Bokc2YCpvN/om07gP1x+TRytqN5E9F/IQhC3WZTDcPV1ZVly5aRmprKlStX8PX1xdPTs7pja7AMMVGgsccU1J8TO47Qt7O/mL9bEIQ6r0KdEF5eXnh6eiLLMpIkAZa+BsF25rQLmC4cQhs6ksMJhZjMMmHBXrUdliAIQrlsShgpKSnMmTOHmJgYrl27VmSZuMV5xehjfgI7/2zvfgAAFVJJREFUR7Tt7uXghr9p7CKaowRBqB9sqh7MnDkTjUbDF198gYODA+vWraNv377Mnj27uuNrUMwp5zAnxKLtMIgCScOJ+AzCgrxEc5QgCPWCTTWMI0eOsH37dhwcHFAoFAQHBzN37lzGjRvH/fffX90xNhj6mHUodM5o2/Rn35k00RwlCEK9YlMNQ6lUWq+5cHFxITMzEwcHB1JSUqo1uIbElHwGc+JfaDsOQaHREXM6TTRHCYJQr9hUw+jQoQM7duxgwIAB9OrVi+eeew6dTkfbtm2rO74GQZZlDDE/oXBwQ9O6L/mFJk7EZ4jRUYIg1Cs2JYwFCxZYR0XNmDGD5cuXk5+fzyOPPFKtwTUUUlo85uQz2PV4EIVay9HTyZjMMl1Ec5QgCPWITQnDxeWfZhOdTsfTTz9dbQE1RMYzu0ClRdOqJ4BojhIEoV4qtw8jMzOTo0ePkpubC8APP/zA5MmTef/999Hr9dUeYH0nm/QYz+1H3TwMhdbB2hwVFuRVbIIqQRCEuqzMGsavv/7Kyy+/jKOjIyaTiSeffJKoqCh69+7N9u3bycnJYebMmTUVa71kij8ExgI0QZb7bh09lyaaowRBqJfKTBiLFy9myZIl9OrViz/++IOnn36a3377jSZNmjBhwgTuu+8+kTDKYTyzC4WLFyrfYAAOnkoVzVGCINRLZTZJJScn06tXLwDuvvtutFqtddpUHx8f8vLyqj/Ceky6loo56RSaVr1QKBTkFxr560KmaI4SBKFeqtCNoDQaTXXF0SAZz+wChQJNK0vSPfJ3umiOEgSh3iqzScpgMPDKK69Yf8/Pz7f+LssyBoOheqOrx2RJwnh2Dyr/diidLHNdxJxOxV00RwmCUE+VmTAmT55cod+Ff5gTTyDnZaLp/gCAtTmqb2d/0RwlCEK9VGbCiIyMrKk4GhzjmV0o7JxQ39EREM1RgiDUf6X2YZw/f96mDdj6vtuJVJiD6cJh1C17oFBZ+n1Ec5QgCPVdqQljzJgxNm1g3LhxVRZMQ2H6ex9IZjTBlmsvcguMnIjPJFSMjhIEoR4rtUmqsLCQBx98sNwNGI3/3969BzVxL3oA/yYgAURAECSoVGgtcuVa0RywiHpELFCRqueiDNXju+Uiatsp9TUDpyhTQcuIlin1+Jo+Dp7qKI4wVbEUrVa9UJ9RQY9VtCa8EQSUR7L3D9tUio8VEhLh+5lxhuwK+81vVr/sbva3LXoN9KITBAEtxccgdXKHmcMgAMBJZSk0WgH+3i5GTkdE1HFPLIykpCRRP4CPaG1LW3kT2upfIQv4O4CHBZJ/7g7c5bZw69/HyOmIiDruiYUxbdq0rszRbTycaLAXer3sBwC49mst1FWNmBc61MjJiIg6h4cHeiS0NqPlPydh7q6ARNYbAJB/7g6sZGbw9epv5HRERJ3DwtCj1huFQPN99Bo6DsDDi92FRRV4fZgLZBZmRk5HRNQ5LAw9ain+EZI+TjCTewIAfrqoRqtGi/EjBhg5GRFR54kqjKKiIkPneOFpSq89nGhw6HhIJNLfLnar8LKrLQY52xg7HhFRp4kqjLlz5yI8PBzbtm1DeXm5oTO9cARBiwcn/wVJ776w8J4EALh6+y5Kqxt5dEFE3Yaowjh+/DiWLl2K8+fPIzg4GPPnz8f+/ftx//59Q+d7IbRePQFtxQ3IfCMg6SUDABw9p4KVzBx/8eJUIETUPYgqDHNzcwQFBWHTpk04duwYQkNDsXXrVvj7++Ojjz7Czz//bOicJktovo+m/9sDqfPLMH9lNADgXmMzCovL4e/tAlkvXuwmou7huS56NzQ04MiRI8jJyUFZWRkmT56Ml156CXFxcfj4448NldGkNZ/LhnC/Fpb+b0MieTicJy6WolUjYPwIVyOnIyLSn6fOVvu7/Px87N+/H8eOHcPIkSMRERGBoKAgyGQPT7+8/fbbmDBhQo97XKu2rhzNFw7BfMgYmDl7AHh4Z/fR8yq8MsAOA514sZuIug9RhfHpp5/irbfewsqVK+Hs3P6cvL29PVatWqX3cKau6dS/AakZZL5/TNRYfOsuyqobETbZy4jJiIj0T1RhHDhw4Jl/JyIiotNhXiStdy6j9ebPsPjL3yDt3Ve3PP/cHVjLzPncCyLqdkRdw4iNjUVhYWGbZYWFhVi6dGmnA9TU1GDRokUIDg7GlClTEBsbi+rqagDAuXPnEB4ervtkVlVVVae3pw+CVoOmn/4FSZ9+sPjvYN3yusZm/FxcAX9vF1jwYjcRdTOiCqOgoAA+Pj5tlo0YMQKnT5/udACJRIKFCxfi0KFDOHDgAAYNGoQNGzZAq9UiLi4O8fHxOHToEBQKBTZs2NDp7elDS9FRaGt+hWx0JCTmFrrlJy6qodEKGO/Dey+IqPsRVRgWFhbt7rlobGyEubmoM1pPZW9vDz8/P93rESNGQKVSQalUQiaTQaFQAHj4oKaDBw92enudJTQ1oLlgL8zkQ2E+eJRu+YPmVuSfvYMhA+0woF9vIyYkIjIMUf/jBwQEID4+HomJibCxsUF9fT0SExMxduxYvYbRarXIzMxEYGAg1Go1XF3/+Fiqg4MDtFot7t69C3t7e71u9880FTdRe6MEzfUP2q9TF0NoboDMP0r39LymFg027bmAqtomzH7D06DZiIiMRVRhrFixAnFxcfD19YWdnR1qa2sxbtw4pKSk6DXMmjVrYG1tjVmzZiE3N1cvP9PR8fk/2lp24giqLh1/4nq70W/BcegwAEBziwZrtp9G8e27+CBqFP46cmCHs74InJz4EKg/45i0xzF5vBd9XEQVhp2dHbZs2YLy8nKUlpZCLpfDyclJr0GSk5NRUlKCjIwMSKVSyOVyqFQq3frq6mpIpdLnPrqoqqqHVis81/dIXp+Pl0LeQVXlvceslEBrYY2Kinto1Wjx2d6LuHC9CvPf9MKwQXaoqHjM93QTTk59uvX76wiOSXsck8d7UcZFKpU88Rft57oI4ezsDCcnJwiCAK1W+9sP7/wM6ampqVAqldiyZQssLB5eRPb29saDBw9QWFgIhUKBXbt2ISQkpNPbEkMilcLMsjckMu0T/06rRovPs5S4cL0Kfw/xRMBweZdkIyIyFlGFUVZWhsTERBQWFqKurq7NuitXrnQqwLVr1/DFF19g8ODBiIyMBAAMHDgQ6enpSElJQUJCApqamjBgwACsX7++U9vSF41Wi38euIyz1yoRFTQEf+WMtETUA4gqjISEBFhaWmLnzp2YNWsWvvnmG2zevBnjx4/vdIAhQ4aguLj4setGjhwp6qbBrqTVCtieU4SConLMmPAKghSDjB2JiKhLiCqMs2fP4ocffoC1tTUkEgmGDh2KpKQkREZGYsaMGYbOaDIaH7Tim9xinLxUhmnjPBDi52bsSEREXUZUYUilUt09F7a2tqiuroaNjQ3KysoMGs5UCIKAU5fL8G3ef1DX0IypY90xxX+wsWMREXUpUYXx2muv4ejRo5g0aRICAgLw3nvvwdLSEt7e3obOZ3S/VtTj68NXcfX2XbjL+2Dp/wyHu9zW2LGIiLqcqMJISUnRfSpq1apV2L59OxoaGjBnzhyDhjOmxgct+HfeNeQW/AormRnmhHhi7GuukP52sx4RUU/zzMLQaDRISkrCmjVrAACWlpaIiYkxeDBjUt6ows7vilFT9wBjX3PF38Z7oI+1xbO/kYioG3tmYZiZmeHEiRO6aTB6gpPKMjjZW+F/pw7Dy652xo5DRGQSRN11N2fOHGzevBktLS2GzmMSFk35L2xYNo5lQUT0CFHXML7++mtUVlZix44dcHBwaHO0kZ+fb6hsRERkQkQVhqncYU1ERMYjqjB8fX0NnYOIiEycqMJIS0t74rply5bpLQwREZkuUYVRWlra5nVFRQUKCgoQFBRkkFBERGR6RBXGJ5980m7ZsWPHkJOTo/dARERkmjr8MIuAgAAcOXJEn1mIiMiEiTrCuH37dpvX9+/fR3Z2NuRyPjSIiKinEFUYkyZNgkQigSA8fNSplZUVvLy8sG7dOoOGIyIi0yGqMIqKigydg4iITJyoaxhXrlyBWq1us0ytVrNIiIh6EFGFERcXh9bW1jbLWlpaEBcXZ5BQRERkekQVhkqlwqBBbZ9d7ebmhjt37hgkFBERmR5RheHi4oJLly61WXbp0iU4OzsbJBQREZkeURe9586di5iYGCxcuBBubm64desWtm/fjujoaEPnIyIiEyGqMGbMmIE+ffpgz549KC0thYuLC5YvX46QkBBD5yMiIhMhqjAAIDQ0FKGhoYbMQkREJkzUNYy1a9fizJkzbZadOXMGSUlJBglFRESmR1RhZGdnw9vbu80yb29vZGdnGyQUERGZHlGF8ei0IL/TaDTQarUGCUVERKZHVGEoFAps3LhRVxBarRabN2+GQqEwaDgiIjIdoi56r169Gu+++y4CAgLg6uoKtVoNJycnfP7554bOR0REJkJUYbi4uGDfvn04f/48SktLIZfLMXz4cENnIyIiEyL6AUpSqRQ+Pj4IDQ2FlZUV1q9fj3HjxhkyGxERmRDR92FUV1fjwIEDyMrKQlFREUaNGoXVq1cbMhsREZmQpxZGS0sL8vLysG/fPhw/fhxubm6YPHkyVCoV0tLS4Ojo2FU5iYjIyJ5aGGPGjIFEIsH06dOxZMkSDBs2DACQmZnZJeGIiMh0PPUahqenJ+7du4fz58/j4sWLqK2t7apcRERkYp5aGF999RVyc3MxZswYbN++HWPGjEF0dDQaGxvbPVDJUG7cuIGZM2ciODgYM2fOxM2bN7tku0RE1NYzPyU1YMAALF68GIcPH8bOnTvh5OQEqVSK8PBwpKSkGDxgQkICoqKicOjQIURFRSE+Pt7g2yQiovYkwp/n/BChqakJubm5yMrKwtatWw2RCwBQVVWF4OBgnD59GmZmZtBoNPDz88Phw4fh4OAg6mfU1DRAq33utwhHRxtUVdU/9/d1ZxyT9jgm7XFMHu9FGRepVIK+fXs/dp3oj9U+SiaTISwsDGFhYZ0K9ixqtRr9+/eHmZkZAMDMzAzOzs5Qq9WiC+NJb1wMR0ebDn9vd8UxaY9j0h7H5PFe9HERfeMeERH1bCZdGHK5HGVlZdBoNAAezpBbXl4OuVxu5GRERD2PSReGo6MjvLy8dM/dyM7OhpeXl+jTUUREpD8duujdla5fv44VK1agrq4Otra2SE5OhoeHh7FjERH1OCZfGEREZBpM+pQUERGZDhYGERGJwsIgIiJRWBhERCQKC+NPONkhkJycjMDAQHh6euLq1au65T15bGpqarBo0SIEBwdjypQpiI2NRXV1NQDg3LlzCA8PR3BwMObPn4+qqiojp+06MTExCA8Px9SpUxEVFYUrV64A6Nn7yu8+++yzNv+GusV+IlAbs2fPFrKysgRBEISsrCxh9uzZRk7U9QoKCgSVSiVMmDBBKC4u1i3vyWNTU1MjnDp1Svd63bp1wsqVKwWNRiMEBQUJBQUFgiAIQnp6urBixQpjxexydXV1uq9zc3OFqVOnCoLQs/cVQRAEpVIpLFiwQPdvqLvsJzzCeERVVRUuX76smyMrLCwMly9f1v0m2VMoFIp2d9P39LGxt7eHn5+f7vWIESOgUqmgVCohk8mgUCgAAJGRkTh48KCxYna5Pn366L6ur6+HRCLp8ftKc3MzEhMT8Y9//EO3rLvsJx2afLC70sdkh90Vx+YPWq0WmZmZCAwMhFqthqurq26dg4MDtFot7t69C3t7eyOm7DqrV6/GiRMnIAgCtm7d2uP3lbS0NISHh2PgwIG6Zd1lP+ERBtFzWrNmDaytrTFr1ixjRzEJSUlJyM/Px/vvv98lz8gxZWfPnoVSqURUVJSxoxgEC+MRnOzwyTg2DyUnJ6OkpAQbN26EVCqFXC6HSqXSra+uroZUKn2hfmvUl6lTp+L06dNwcXHpsftKQUEBrl+/jokTJyIwMBClpaVYsGABSkpKusV+wsJ4BCc7fDKODZCamgqlUon09HRYWFgAALy9vfHgwQMUFhYCAHbt2oWQkBBjxuwyDQ0NUKvVutd5eXmws7Pr0fvKO++8g+PHjyMvLw95eXlwcXHBtm3bsHDhwm6xn3AuqT/hZIfA2rVrcfjwYVRWVqJv376wt7dHTk5Ojx6ba9euISwsDIMHD4alpSUAYODAgUhPT8eZM2eQkJCApqYmDBgwAOvXr0e/fv2MnNjwKisrERMTg/v370MqlcLOzg7Lly/HsGHDevS+8qjAwEBkZGTg1Vdf7Rb7CQuDiIhE4SkpIiIShYVBRESisDCIiEgUFgYREYnCwiAiIlFYGEQmztPTEyUlJcaOQcS5pIieV2BgICorK3VzJQHAtGnTEB8fb8RURIbHwiDqgIyMDPj7+xs7BlGX4ikpIj3Zu3cvIiMjkZiYiFGjRiEkJAQnT57UrS8rK0N0dDR8fX0xadIkfPvtt7p1Go0GGRkZCAoKgo+PD6ZPn95m2o2ffvoJb7zxBhQKBT7++GPwflsyBh5hEOnRhQsXEBISglOnTiE3NxexsbH4/vvvYW9vjw8++ABDhgzBjz/+iF9++QXz5s3DoEGD8Prrr2PHjh3IycnBli1b4O7ujuLiYt0UJACQn5+PPXv2oL6+HtOnT8eECRMwbtw4I75T6ol4hEHUAYsXL4ZCodD9+f1owcHBAXPmzEGvXr3w5ptvwt3dHfn5+VCr1Thz5gw+/PBDyGQyeHl5ISIiAvv37wcA7N69G8uWLYOHhwckEgmGDh2Kvn376ra3aNEi2NrawtXVFX5+figqKjLK+6aejUcYRB2Qnp7e7hrG3r170b9/f0gkEt0yV1dXlJeXo7y8HHZ2drCxsWmzTqlUAgBKS0vh5ub2xO05OTnpvrayskJDQ4O+3gqRaDzCINKjsrKyNtcX1Go1nJ2d4ezsjNraWtTX17dZ179/fwCAi4sLbt261eV5iZ4HC4NIj6qrq/Hll1+ipaUF3333Ha5fv47x48dDLpfDx8cHqampaGpqQlFREfbs2YPw8HAAQEREBNLS0nDz5k0IgoCioiLU1NQY+d0QtcVTUkQdEB0d3eY+DH9/f0ycOBHDhw9HSUkJRo8ejX79+mHTpk26axGpqalISEjA2LFjYWtriyVLluhOa82bNw/Nzc2YP38+ampq4OHhgfT0dKO8N6In4fMwiPRk79692L17NzIzM40dhcggeEqKiIhEYWEQEZEoPCVFRESi8AiDiIhEYWEQEZEoLAwiIhKFhUFERKKwMIiISBQWBhERifL/YSS6fjw/J9UAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Check Training History\n",
    "plot_history(history_tfidf, \"Val Accuracy: TFIDF + Deep Learning Model\", save=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 8: Saving the Model\n",
    "In order to use the model we've just trained in production, we'll need to save the model and all of its dependencies. This way, we'll be able to easily reload the model in production to perform predictions. Below we save...\n",
    "\n",
    "- `tfidf_vectorizer`: The TF/IDF Vectorizer was 'fit' to the training dataset, and its vocabulary encompasses all of the words that show up in our training emails. In order to prepare plain-text/ HTML files to be fed into our model during deployment, we'll need to use this vectorizer once again to transform that text/HTML into the vectors that our deep learning model expects as input.\n",
    "- `label_to_id`: This variable stores a simple dictionary that maps prediction indices (indices of the labels in the one-hot-encoded label representation) to actual backlog-id names. Without this dictionary, we wouldn't be able to understanding the meaning of our model predictions.\n",
    "- `assignment_model`: lastly, of course, we save the classifier itself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'BL-3346'"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Generate a dictionary that maps prediction indicies \n",
    "# (indicies of one-hot-encoded labels) to actual backlog ID names \n",
    "label_to_id = {}\n",
    "for i, col in enumerate(test_labels.columns):\n",
    "    label_to_id[i] = col\n",
    "\n",
    "# Test:\n",
    "label_to_id[10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create Directory to save into:\n",
    "! mkdir -p ../saved_models/tfidf_model/\n",
    "# Save tfidf_vectorizer\n",
    "pickle.dump(tfidf_vectorizer, open(\"../saved_models/tfidf_model/tfidf_vectorizer.pkl\", \"wb\"))\n",
    "# Save assignment_model\n",
    "assignment_model.save('../saved_models/tfidf_model/tfidf_classifier.h5')\n",
    "# Save label_to_id dictionary\n",
    "pickle.dump(label_to_id, open('../saved_models/tfidf_model/label_to_id.pkl', 'wb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 9: Testing Model + Generating Predictions:\n",
    "To generate model predictions, we apply the Monte Carlo Dropout confidence bounding approach described in ['Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning\n",
    "'](https://arxiv.org/pdf/1506.02142.pdf). We've developed two prediction functions that use Monte Carlo Dropout under the hood\n",
    "- `test_with_uncertainty` - designed to be used to test model accuracy on a test set, and\n",
    "- `predict_with_uncertainty`, designed to be used in production to generate a prediction for a single email.\n",
    "You can find the code for these functions in [MLFunctions.py](../nlp_engine/MLFunctions.py). We've listed the function headers below to describe how each of these functions work.\n",
    "\n",
    "When we test our model, we'll use the function `test_with_uncertainty` to obtain our model's predictions for each sample in the test set, along with the associated confidence value for  each of those predictions. This way, we can evaluate the accuracy of the model at different confidence intervals.\n",
    "```python\n",
    "def test_with_uncertainty(model, test_features: pd.DataFrame, test_labels: pd.DataFrame, label_to_id: dict, n_iter=10) -> pd.DataFrame\n",
    "    \n",
    "    \"\"\"\n",
    "    USE FOR TESTING\n",
    "    ---------------\n",
    "\n",
    "    Parameters:\n",
    "\n",
    "        - model: a keras model trained with Monte Carlo Dropout\n",
    "\n",
    "        - test_features: one or multiple samples (1 sample = 1 vector of \n",
    "                         test_features).\n",
    "\n",
    "        - test_labels: the ground-truth labels corresponding to the samples. \n",
    "                       (one-hot-encoded)\n",
    "\n",
    "        - label_to_id: A dictionary that maps prediction indicies (indicies of\n",
    "                       the softmax outputs of the model) to label names. \n",
    "\n",
    "        - n_iter: the number of stochastic forward passes you'd like your model \n",
    "                  to perform for each given test sample. i.e. for n_iter=10, the\n",
    "                  function applies 10 distinct dropout schemas and generates\n",
    "                  10 different softmax outputs from the model. It then takes the final or 'master'\n",
    "                  prediction to be the maximum softmax index with the highest overall probability\n",
    "                  (i.e. final prediction = average prediction over all 10 trials) and the uncertainty \n",
    "                  to be the Standard Deviation of the set of 10 softmax outputs (more variable softmax output \n",
    "                  = more uncertainty)\n",
    "    \n",
    "    Returns: preds_df, a dataframe of labels predicted by your model, ground truth lables, and prediction\n",
    "             uncertainty values.\n",
    "    \"\"\"\n",
    "    \n",
    "def predict_with_uncertainty(model, test_features: pd.DataFrame, label_to_id: dict, n_iter=10) -> tuple\n",
    "    \n",
    "    \"\"\"\n",
    "    USE FOR DEPLOYMENT\n",
    "    ------------------\n",
    "\n",
    "    Wheras the above function is used during the testing phase (testing monte \n",
    "    carlo accuracies), this function should be used for deployment. Note that\n",
    "    there are no ground-truth labels passed into the function.\n",
    "\n",
    "    Parameters:\n",
    "\n",
    "        - model: Save as above\n",
    "\n",
    "        - test_features: a SINGLE sample (1 sample = 1 vector of \n",
    "                         test_features).\n",
    "\n",
    "        - label_to_id: Save as above\n",
    "\n",
    "        - n_iter: Same as above\n",
    "   \n",
    "    Returns: a tuple, including the predicted label value and an uncertainty value\n",
    "             ... i.e. return  (prediction id, uncertainty)\n",
    "    \"\"\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Test the Model\n",
    "preds_df = test_with_uncertainty(assignment_model, test_features, test_labels, label_to_id, n_iter=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Prediction</th>\n",
       "      <th>Label</th>\n",
       "      <th>Uncertainty</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>BL-3289</td>\n",
       "      <td>BL-3289</td>\n",
       "      <td>0.028436</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>BL-3289</td>\n",
       "      <td>BL-3289</td>\n",
       "      <td>0.001387</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>GRP-3974</td>\n",
       "      <td>GRP-3974</td>\n",
       "      <td>0.091664</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>GRP-28665</td>\n",
       "      <td>GRP-28665</td>\n",
       "      <td>0.060137</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>GRP-20686</td>\n",
       "      <td>GRP-20686</td>\n",
       "      <td>0.039525</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1082</th>\n",
       "      <td>GRP-13931</td>\n",
       "      <td>GRP-13931</td>\n",
       "      <td>0.014101</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1083</th>\n",
       "      <td>GRP-13931</td>\n",
       "      <td>BL-2287</td>\n",
       "      <td>0.109725</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1084</th>\n",
       "      <td>BL-2285</td>\n",
       "      <td>BL-2285</td>\n",
       "      <td>0.008118</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1085</th>\n",
       "      <td>BL-752</td>\n",
       "      <td>BL-752</td>\n",
       "      <td>0.004115</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1086</th>\n",
       "      <td>BL-6113</td>\n",
       "      <td>BL-6113</td>\n",
       "      <td>0.001472</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1087 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     Prediction      Label  Uncertainty\n",
       "0       BL-3289    BL-3289     0.028436\n",
       "1       BL-3289    BL-3289     0.001387\n",
       "2      GRP-3974   GRP-3974     0.091664\n",
       "3     GRP-28665  GRP-28665     0.060137\n",
       "4     GRP-20686  GRP-20686     0.039525\n",
       "...         ...        ...          ...\n",
       "1082  GRP-13931  GRP-13931     0.014101\n",
       "1083  GRP-13931    BL-2287     0.109725\n",
       "1084    BL-2285    BL-2285     0.008118\n",
       "1085     BL-752     BL-752     0.004115\n",
       "1086    BL-6113    BL-6113     0.001472\n",
       "\n",
       "[1087 rows x 3 columns]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Visualize and evaluate the prediction/ uncertainty values\n",
    "preds_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Testing Accuracy\n",
    "---\n",
    "\n",
    "Now that we've gotten our model predictions, we can pass the resulting `preds_df` DataFrame into another function `get_monte_carlo accuracy`. `get_monte_carlo accuracy` takes as input a predictions DataFrame, as well as a threshold uncertainty score. The function will then return the model's accuracy on all test samples that were predicted with an uncertainty value *below* the threshold uncertainty score. Further, it will let us know how much of the test set was discarded when the confidence threshold was applied.\n",
    "\n",
    "Ideally, you want to find a balance between accuracy and efficiency. It may be the case that that your model can achieve 80% accuracy on the entire test set (i.e. NO uncertainty threshold), however, your model can achieve a much higher 95% accuracy on 60% of the test set if you allow the model to throw emails that it is unable to confidently place into a queue for human review. In this case, you must decide whether you prefer\n",
    "- An 80% accurate model that automates 100% of the work or...\n",
    "- a 95% accurate model that automates 60% of the work.\n",
    "\n",
    "To determine the right balance for you, feel free to play around with the threshold uncertainty value and determine which threshold is best for your use case.\n",
    "\n",
    "**NOTE**: Do not change the `n_iter` value unless you truly understand what this parameter means. If you do decide to change the `n_iter` value, make sure that the model uses the exact same `n_iter` value in production (i.e. make sure that both test_with_uncertainty and predict_with_uncertainty take the same `n_iter` value. Changing the `n_iter` value alters the scale of the `uncertainty` metric, so we need to make sure `n_iter` is consistant across testing and production to ensure that the threshold uncertainty value we determine during testing works as we expect it to work in production.\n",
    "\n",
    "--- \n",
    "### Steps:\n",
    "- Test the models accuracy across the **entire** test set by setting the uncertainty threshold to `None`, and compare its accuracy to a dummy classifier (i.e. a model that classifies every email into the most frequently occuring backlog. \n",
    "- Show an example for how we can test the models accuracy at a particular uncertainty threshold. \n",
    "- Show a graph of the relationship between the percentage of testing data retained and the model accuracy\n",
    "- Show how we can select an uncertainty thrshold based on a target accuracy, or a target ammount of data we want to retain in production (the percentage of the bug classification that will be automated by the model)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test set reduced to % 100\n",
      "Accuracy : % 78.84084636614536\n"
     ]
    }
   ],
   "source": [
    "retained, accuracy = get_monte_carlo_accuracy(preds_df=preds_df, threshold=None)\n",
    "print('Test set reduced to %', retained)\n",
    "print('Accuracy : %', accuracy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Zero-Rule Baseline Accuracy: % 18.39926402943882\n",
      "Random (Uniform) Baseline Accuracy: % 2.391904323827047\n"
     ]
    }
   ],
   "source": [
    "# For reference, here is the zero-rule and random baseline accuracies\n",
    "# on the test set:\n",
    "\n",
    "# DummyClassifier only accepts numeric categorical labels, so we convert\n",
    "# our one-hot-encoded labels back to standard levels before passing in:\n",
    "int_train_labels = train_labels.idxmax(axis=1)\n",
    "int_test_labels = test_labels.idxmax(axis=1)\n",
    "\n",
    "clf = DummyClassifier(strategy='most_frequent')\n",
    "clf.fit(train_features, int_train_labels)\n",
    "print(\"Zero-Rule Baseline Accuracy: %\", clf.score(test_features, int_test_labels) * 100)\n",
    "\n",
    "clf = DummyClassifier(strategy='uniform')\n",
    "clf.fit(train_features, int_train_labels)\n",
    "print(\"Random (Uniform) Baseline Accuracy: %\", clf.score(test_features, int_test_labels) * 100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test set reduced to % 40.110395584176636\n",
      "Accuracy : % 96.78899082568807\n"
     ]
    }
   ],
   "source": [
    "# Accuracy with an uncertainty threshold applied:\n",
    "retained, accuracy = get_monte_carlo_accuracy(preds_df=preds_df, threshold=0.04)\n",
    "print('Test set reduced to %', retained)\n",
    "print('Accuracy : %', accuracy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('BL-3289', 0.026612099019358963)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Use this function to generate a prediction on a single email. \n",
    "# This function is not relevant in testing the model, but will be used to classify incoming emails later on in production\n",
    "predict_with_uncertainty(assignment_model, test_features.iloc[0], label_to_id, n_iter=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "92.05175600739372"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEcCAYAAADUX4MJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nOzdd3iUVdr48e9k0jvphSIEEumEBFBASuAFQarKggjyIi4gIqigIAi4CGiUBdQFFXXR3Z+72OhVVEBB5Q0KIj0EEkJ6g/RMMnN+f7DMEtImIZNJuT/XxXUx5zzlPpNJ7nme85xzNEophRBCCFEFK0sHIIQQomGQhCGEEMIkkjCEEEKYRBKGEEIIk0jCEEIIYRJJGEIIIUwiCUMIM4iIiOCnn36ydBh17qmnnmLr1q2WDkOYiSQMYVYRERF06tSJzMzMUuVjxowhJCSEa9eu3fU5Jk+ezJdffnlXx/jxxx95/PHHCQ0N5b777mPSpEl89913dx1bTR07doyQkBA2btxosRhq4qOPPmLs2LGWDkOYiSQMYXaBgYHs3r3b+PrChQsUFBRYMKLS9u3bx9y5cxkzZgw//PADP/30E3PmzOHgwYPVPlZJSUmtxLRt2zbc3d3Zvn17rRzPVEopDAZDnZ5TNBySMITZjR49mm3bthlfb9u2jTFjxpTaJicnh5deeon77ruPgQMHsmHDBuMfri1btvDYY48RGRlJjx49iIiI4PDhwwCsXbuW48ePs3z5ckJDQ1m+fDkAMTExTJ06lZ49ezJ06FD27NlTbmxKKd544w1mzZrFuHHjcHFxwcrKip49e7JixQoArl69yhNPPEGvXr3o1asX8+bNIzs723iMiIgINm7cyMiRI+nWrVuZpKHT6Vi5ciV9+/alb9++rFy5Ep1OV+H7lZ+fz759+1i6dClxcXH88ccfpeq/+OILhg0bRmhoKMOHD+fMmTMAJCUlMXv2bO677z569eplfC/effdd5s+fb9z/2rVrhISEGOOcPHkya9euZcKECXTt2pX4+Hi+/vpr4zkGDRrE5s2bS8Xw7bffMnr0aLp3787gwYP54YcfjMe6/Wrvq6++YtiwYfTo0YNp06aRkJBgfN9XrVrF/fffT/fu3Rk5ciQXL16s8D0R9YQSwowGDhyojh49qoYMGaIuXbqkSkpK1AMPPKCuXbumgoODVXx8vFJKqRdffFHNnDlT5eTkqPj4eDVkyBD1xRdfKKWU+vrrr1WHDh3U559/rkpKStRnn32m+vTpowwGg1JKqUmTJhm3VUqpvLw81a9fP/XVV1+p4uJidebMGdWzZ08VHR1dJr5Lly6p4OBgdfXq1QrbEBsbq44cOaKKiopURkaGmjhxolqxYkWpNo4aNUolJiaqgoKCUu1WSql169apcePGqfT0dJWRkaHGjx+v1q5dW+H5tm7dqvr06aNKSkrUjBkz1PLly411e/bsUX379lW///67MhgMKjY2Vl27dk2VlJSokSNHqpUrV6q8vDxVWFiooqKilFJKvfPOO2revHnGY8THx6vg4GBVXFxsfP/69++vLl68qIqLi5VOp1MHDx5UcXFxymAwqGPHjqkuXbqo06dPK6WU+v3331X37t3VkSNHlF6vV8nJyerSpUtlfhYHDhxQgwcPVpcuXVLFxcVq/fr1avz48UoppX744Qc1duxYdePGDWUwGNSlS5dUSkpKhe+JqB/kCkPUiVtXGUePHiUoKAhfX19jnV6vZ8+ePcybNw9nZ2eaN2/O1KlT2bFjh3GbgIAA/vSnP6HVahk7dixpaWmkp6eXe65Dhw4RGBjII488grW1NR06dGDo0KHs27evzLbXr18HwMfHp8LYW7VqRZ8+fbC1tcXDw4OpU6cSFRVVapvJkyfj7++Pvb19mf137tzJM888g6enJx4eHjzzzDOl2nanbdu2MWzYMLRaLSNGjGD37t0UFxcDN7+xP/XUU3Tp0gWNRkOrVq0IDAzk1KlTpKam8tJLL+Ho6IidnR3h4eEVnuNOY8eOpV27dlhbW2NjY8OAAQNo2bIlGo2Gnj170qdPH44fP26M4ZFHHqFPnz5YWVnh6+tLUFBQmWNu3ryZ6dOnExQUhLW1NTNnzuTcuXMkJCRgbW1NXl4ely9fRilFUFBQpT8DUT9YWzoA0TSMHj2aSZMmce3aNUaPHl2qLisri+LiYgICAoxlAQEBpKSkGF97eXkZ/+/g4ADcvHVTnoSEBE6dOlXqD6Zer2fUqFFltnV3dwcgNTWVFi1alHu89PR0Vq5cyfHjx8nLy0Mphaura6lt/P39y9331rHvbFtqamq52yYlJXHs2DFeeOEFAAYNGsSSJUs4fPgwgwcPJikpiZYtW5a7X0BAANbWNfuVvjP+w4cPs379emJjYzEYDBQWFhIcHGw8V//+/as8ZmJiIqtWrSIyMtJYppQiJSWF+++/n8cff5zly5eTkJDAkCFDWLBgAc7OzjWKX9QNSRiiTgQGBtK8eXMOHz7MypUrS9U1a9YMGxsbEhMTadu2LXDzj9LtVyHV4e/vT48ePdi0aVOV27Zp0wZ/f3+++eYbpk2bVu42a9asQaPRsHPnTtzd3fn222+N/QO3aDSaCs/h4+NDYmIi7dq1A262raJv09u3b8dgMPD0008by3Q6HVu3bmXw4MH4+/tz9erVctuclJRESUlJmaTh4OBAYWGh8XV5V2a3x6/T6ZgzZw6RkZEMGjQIGxsbZs2ahfrPxNYVxVBeTDNnziw3UQM88cQTPPHEE2RkZPDcc8/x0Ucf8dxzz1V5XGE5cktK1JmVK1fy6aef4ujoWKpcq9Xy4IMPsnbtWnJzc0lISGDTpk0V/qG5k5eXF/Hx8cbXAwYMIDY2lm3btlFcXExxcTGnTp0iJiamzL4ajYaFCxeyYcMGvv76a3JzczEYDBw/fpwlS5YAkJeXh6OjIy4uLqSkpPDRRx9Vq90PPfQQ7733HpmZmWRmZrJ+/XpGjhxZ7rZbt25l9uzZbNu2zfjvnXfe4fDhw2RlZfHoo4/y97//ndOnT6OUIi4ujoSEBLp06YK3tzd//etfyc/Pp6ioiF9//RWA9u3bExUVRWJiIjk5OXzwwQeVxqvT6dDpdHh4eGBtbc3hw4c5evSosf7RRx9ly5Yt/PzzzxgMBlJSUsp9bydMmMDGjRuJjo4Gbj7YsHfvXgBOnTrF77//TnFxMQ4ODtja2mJlJX+O6jv5CYk607JlSzp37lxu3ZIlS3BwcGDw4MFMnDiRESNG8Mgjj5h03CeeeIL9+/fTo0cPVqxYgbOzMx9//DF79uzhgQceoG/fvqxevbrCJ5NuJauvv/6aBx54gN69e/P2228zaNAgAGbPns3Zs2cJDw9n+vTpDBkypFrtnjVrFp06dWLUqFGMGjWKjh07MmvWrDLbnTx5ksTERB5//HG8vb2N/wYNGkSrVq3YvXs3w4YNY+bMmcybN4/u3bvzzDPPcOPGDbRaLe+//z5xcXEMHDiQfv36Gf849+nTh+HDhzNq1CgefvhhBg4cWGm8zs7OvPLKKzz33HP06NGDXbt2ERERYazv0qULr7/+OqtWrSIsLIxJkyaRmJhY5jj/8z//w1NPPcULL7xA9+7dGTFihPFpqry8PF555RV69uzJwIEDcXd3r/AKT9QfGqVkASUhhBBVkysMIYQQJpGEIYQQwiSSMIQQQphEEoYQQgiTSMIQQghhEkkYQgghTNLoR3pnZeVhMDS9J4c9PZ3JyMi1dBgWI+2X9kv7a9Z+KysNzZo5lVvX6BOGwaCaZMIAmmy7b5H2S/ubMnO0X25JCSGEMIkkDCGEECaRhCGEEMIkdZIwIiMjiYiIICQkpNQyjFeuXGH8+PEMHTqU8ePHExsba1KdEEKIulcnCWPQoEF89tlnBAYGlipftmwZEydOZP/+/UycOJGlS5eaVCeEEKLu1UnCCA8PL7OiV0ZGBmfPnmXEiBEAjBgxgrNnz5KZmVlpXV34MSqaRbf9q2h1NCGEaEos1odxa0U1rVYL3FxEx8fHh6SkpErrzO3HqGj23lG2Lu4GiZI0hBBNXKMfh+HpWb01gu9MFrf8MzGb1R3LLnRfn3l7u1g6BIuS9kv7mzJztN9iCcPf35+UlBT0ej1arRa9Xk9qair+/v7GheLLq6uujIzcag1gGQnsLKd8aoAraWk51T6/pXh7uzSoeGubtF/aL+2vWfutrDQVftG22C0pT09P2rdvz65duwDYtWsX7du3x8PDo9I6c7u/RzuG3VH2XCs3fHx8zH5uIYSoz+pkidYVK1bwzTffkJ6eTrNmzXB3d2f37t3ExMSwcOFCsrOzcXV1JTIykjZt2gBUWlcd1b3CsLRLsbH8Pa24TPlMXztatmxp8nHkG5a0X9ov7a+Jyq4wGv2a3vUxYWTn5/PZmQTiq7nfU752tDExacgvjLRf2i/tr4nKEkaj7/S2tIwbN/jrxdp5wmpTShGvmX6RIYQQtUoSRi3Lyslh7flkSsxw7D/72pnhqEIIYRpJGLVgZ1Q0P5v5HNXtwxBCiNomCaOGVkVFY87lWf7sZU3r1q355Xg0OxS8n1IEKdHMD/Ks9afF8oqK2BN9lRMF8D/20K9DG+OgSSGEuEUSRjUtioqutWONAHr3aFdh/a1kcbvVMRk8B/jUIGnsjYrmxyq2OVAIB367zKvd22ArSUMIcRtJGCb6KCqay3ex/5PNNLRt27Za+9yZLG75f/HXeaGShKHT6/khLokdZ5K4Ua0z/te1vCLauDqWKjt5OpovCkpvN8PHllatWtXwLEKIhkQSRhWy8/N540xCtfeb5AQdOlR89WCKhzWwpZyk8b8t3Eu9Tk5P5+9Xsmr1FtnnFxIw5aG8D1J1TCOOIEkaQjR6kjAqUd1ksaqS20s1ER7ejpI7bks917oZu2MyOBeTUavnulN1nuDelKrjhWbZ/DM6hRTDzTJnYHaIL66uruYITwhhATJwrxKm9Fc87ggdO9ZuoihPbmEhe2NTOJGjM/u5atP8EF887kgaeUVFHEpI50RGPvnl7NMeGN+1Fba2tjU+rwzckvZL+2XgXr3xINCvlq8oKpJbWMiqP6o7LrysUGBMt3uwsbEpU3c6M4d/xSTf9Tnu9O6FFIpIqdY+54BXf4/jla6tcLyLpCGEqF2SMKppiiuEhNRNorjlh5TsGu03zcOKoCDTpmS/19XBpO26Axc1kGviRVuRaZuV69eMXB7wN/+Ek0II00jCqMSqHu1K3Zaq7T4KU0X4unIktfLnne58Cqu6l6TW1tYsDW3NjhNXOEnlj/ympaWxNva6yceuqVB305KYEKJuSB9GA5FbWMi+uFROZRdRAjyqhe7dK05g5r6Hm5SWxsex1419EFbA8208OJarrzK5mWqslz09Wreo0b5yD1vaL+2XPowmy9nenkdDWvKopQP5D39vb17x9i5THuFUWGXC8AScrGFSO3+cnf/7wdTr9Sz97TK30vvW9EL87JNoUYOFs4QQtU8ShqhV9vb2LOrcgr1xqVzILqJfM1t63xOItXXVHzWtVos1cPtqIJuu5bJU8oUQ9YIkDFHrnO3tGRdSs4kSZzZ35t1r/x2C+JDF1oQUQtxJEoaoV1w9POC2hPG1AX6OisbKGia39cPFpfYXthdCmEYShqhXfi6noy4RoAReP5/Mi/dCs/8kjYuXL/NJht643YJgH9zc3OooUiGaHkkYol55wNuF75OyKqx/63wyUP4Aw8iLqcwLBk9JGkKYhdwhFvWKnZ0dfTwcq96wAp/W0nK4QoiyJGGIemdwc68a75sOXIiJqb1ghBBGkjBEvWNnZ8fiLi3p4VqzeaQ+zTSwaP9vXL9u/tHoQjQlkjBEveRkZ8fYkFYsDPGtdLuyQwdvStPDm9FpnIlPxGAw1H6AQjRB0ukt6jVXV1fmh8CmCylkUP508pXN5vtZch7WyTEsDQvC2kq+HwlxNyRhiHrPw9WVeT0qXojJ2d6el9p582Z0Wrn1JUB6YTF+jnZmilCIpkG+colGwd3dnRfaVXSDCly0dRiMEI2UJAzRaHi5u/Ncq/LHYGyNSaa4uLjcOiGEaSRhiEbFx8eH2a3ccNCULj+bp2PZyVgKJGkIUWOSMESjE+Djg7Nd+d1zp7Py6jgaIRoPSRiiUZrXNbDcct+S/HLLhRBVk4QhGiVPT0/mtim7Hvj/S5QrDCFqShKGaLR8PT15/h73UmW56ua4DSFE9UnCEI2at7c3IXd0gP+Qkm2ZYIRo4CRhiEZvbPuAUq/7ejlZKBIhGjZJGKLRS9eXvsRIKiyxUCRCNGySMESj18rJjttTxqeXU8nX6SwWjxANlSQM0ehptVr+x7905/evGbkVbC2EqIgkDNEkdHYpvbbG3msZXLt2zULRCNEw1YvZag8dOsTbb79NSUkJbm5uvP7667Ro0YKIiAhsbW2xs7s5y+j8+fN54IEHLBytaIh+zCwoU7YhqYDpXOOe5s0tEJEQDY/FE8aNGzdYsGABmzdvpnXr1mzfvp1XX32Vjz/+GIB33nmH4OBgC0cpGrqhfm78X3pOmfJNSQX8RfKFECax+C2puLg4vLy8aN26NQD9+/fnyJEjZGZmWjgy0Zg4ODjwkFvZj/skL4t/ZxKiwbD4b0vr1q1JT0/n1KlTdOnShZ07dwKQlJQE3LwNpZQiLCyMF154AVfXihfSKY+np3Otx9xQeHu7WDoEi7qz/SM9OrN7/++lyvLdmzXa96mxtstU0v7ab79GKaUqqszMzGT79u0cOnSI8+fPk5ubi7OzM/feey/9+vVj7NixeHiUna+nun766SfeffddioqK6NevH5999hn//Oc/cXNzw9/fH51Ox8qVK8nLy2P16tXVOnZGRi4GQ4VNbLS8vV1ISyt7C6apqKj9BYWFvHbHcq5P+9nTokWLugqtTsjPX9pf0/ZbWWkq/KJd4RXG6tWr2blzJ/379+fRRx8lKCgIJycn8vLyiImJISoqirFjxzJy5Ejmz59fo8Bu6d27N7179wYgPT2djz/+mJYtW+Lo6AiAra0tEydO5Omnn76r8wjhYG9PKzstcUV6Y9l7yYW84HIdL3f3SvYUQlSYMPz8/Dhw4AC2trZl6jp06MDIkSMpKiriyy+/vOsg0tLS8Pb2xmAwsGbNGiZMmABATk4OLi4uKKXYs2cP7du3v+tzCfGnVp68dTG1VNna6DRe6WSHg4ODhaISov6rMGFMmjSpyp3t7OxM2q4q69at47fffqO4uJg+ffowf/58UlNTefbZZ9Hr9RgMBoKCgli2bNldn0uIIhv7MmUKeO30NV5s60WzZs3qPighGoBK+zDulJubywcffMDFixdp0aIFf/7zn/H19TVnfHdN+jCapsrabzAYeO3EZYoq+Fws6twCZ/uySaUhkZ+/tN8cfRjVeqz2L3/5C46OjkyePBkHBwfmzp1bo4CEsCQrKysWh7ZhasvyryQOXE2r44iEaBgqTRi3nky6JSkpienTp9O3b1+efvppLl++bPYAhTAHaysr2vl6saBDAC3u+C2IulHIoqhoYmNjLRKbEPVVpeMwunTpwuTJk3nqqacYPnw4Q4YMYcyYMYSEhPDHH38wZsyYuopTCLNwc3Lihqb8uo1pxZAWTQgwoUtL4xQ1QjRVVfZh5OTksG7dOmJjY1m8eDEGg4GLFy/SvHlzunTpUldx1pj0YTRN1Wl/Wloaa2OvV7nd4i4tcWogSUN+/tL+Oh2HcYuLiwtLlizh9OnTLFq0iJ49e/LMM8/Ity3RaHh7e+Mbe52UKrb7OS2Hwc3lcy+arkr7MFJSUlixYgUzZsxg7969bNiwAV9fX8aPH893331XVzEKYXZPdQiocpvOdk3vSlWI21WaMObOnYutrS2TJk1CKcXKlSt5/PHH+eijj9i7dy8zZ86sqziFMCsnJycWdAigo5sD01p54GdTtmPj7djr6PT6cvYWommo9JbU5cuX+ec//4mNjQ09e/bkT3/6EwBeXl6sXr2aX375pU6CFKIuuDk58XiwEwCzvT3Y8NslEg2lt7mYmc3V/GIifF2xb+BjNYSorkoTxujRo5k6dSphYWEcP36csWPHlqq/7777zBqcEJZipdHwvyH+rDqXVKr8X7HpABxJvdEoBvgJUR2VJozFixdz6tQprl27xogRI2jXrl1dxSWExTk7O9PcCq4Zyq8/cDWVscEt6zYoISyoypHeXbp0Yfjw4ZIsRJP0yD0VT98fdaOIDVHRFBcX12FEQlhOhQnjkUceYe/eveh0unLrdTode/bsYdy4cWYLTghL+6Og8iejrgHLTsZSIElDNAEV3pKKjIzknXfe4dVXX6Vjx460bt3auB5GbGwsZ86c4b777uONN96oy3iFqFMPeLvwfVKW8XUH4Gw5253OyqOHj6ynIRq3Kkd6p6WlcfToUS5evEhOTg6urq6EhITQp08fPD096yrOGpOR3k1TbbY/r6iIn9NyeMDbBTs7O6Kiotl6xzbL6tnUIfLzl/abY6R3taY3b4gkYTRN5m7/8ZPRbLntLpQGeLGdN+71ZNU++flL+y0+vbkQ4qbwbu2Y0/G/T0gp4M3oNDKzsy0XlBBmJglDiBpSeWWTw1fxVU9iKERDJQlDiBr6LKFswvDOL6KkpMQC0QhhfiYlDL3MnyNEGVNblO2viAKWnrhCoSQN0QiZlDD69u3LihUr+OOPP8wdjxANhqenJ7MDncqtu5RdUMfRCGF+JiWMjz76CK1Wy9NPP82wYcN4//33SUpKqnpHIRq5gIAA5rZ0LVPeysYCwQhhZtV6rNZgMHDkyBF27NjBwYMH6dChA6NHj2b48OE4OjqaM84ak8dqm6a6bv/l5BQ+ii/dp7GoU3OcHRzqLIbbyc9f2m/xx2qtrKwICgqiTZs2eHh4kJqays6dOxkwYADbtm2rUXBCNAbKoewv2JrT19DJlCGiEalyiVaAGzdusHfvXrZv387ly5d58MEHiYyMpHv37gCcOnWKadOmMWbMGLMGK0R9dY9z2WnOC4FXT8ayNLQ19tYm/aoJUa+Z9Cnu378/vXr1YvLkyQwePBhbW9tS9V26dGHQoEFmCVCIhkCr1fJq9zZczMzmy9h0br+uOJeeTahfxbPeCtFQmJQwvv32W7y8vCrdRiYhFE2drVZLJ+9mBDo58NaZeGP5l/EZhHg443jHFy0hGhqT+jC2bNnCqVOnSpWdOnWKDz/80CxBCdGQuTvY4W6jLVV2PL3pdsCKxsOkhPGPf/yDtm3blioLCgri008/NUtQQjRkGo2G8a1KX5Fn5xfIQkuiwTPpllRxcTHWd3Ta2djYVLi4khBNXQs3ZzSkcOuB7p+yCvgpK5ZezRwZ2twTe1kLXDRAJl1hdOzYkX/961+lyjZv3kyHDh3MEpQQDZ2VlRVPtvUtU34sK5/lf8STW1hogaiEuDsmXWG8/PLLTJ06lR07dtCiRQvi4+NJS0tj06ZN5o5PiAbrWn7FV+Dfxqczpl3zOoxGiLtnUsJo164d+/fv59ChQyQlJTFkyBAGDBiAk1P58+gIIeB+b1f2J2aVW6dKilFKodFo6jgqIWrO5NFETk5OPPTQQ+aMRYhGxdbWlle6tmL/+TiiikrXReWWEH38EvPCgtBaySoDomEwKWGUlJTwr3/9i6ioKLKysrh9+qnPPvvMbMEJ0dA52toytks7Ol++zN8zSi8TcB149/cY5nRri5VcaYgGwKSvNq+//jqff/454eHhnDlzhiFDhpCRkcF9991n7viEaBTatmnDgmCfMuWpJZB6/YYFIhKi+kxKGN988w0ffvghU6ZMQavVMmXKFNavX8+xY8fMHZ8QjYabmxvzgzzLlL9zKY20tDQLRCRE9ZiUMAoLC/H39wfA3t6egoICgoKCOHv2rFmDE6Kx8fDw4MV7/cqUr429zpKoaEkcol4zKWEEBQUZV9vr1KkT7777Lhs2bMDXt+xz5jVx6NAhxo4dy8iRI5k0aRLx8Tfn4bly5Qrjx49n6NChjB8/ntjY2Fo5nxCWlFDBgG89NxNHgixOJuopkxLGokWLjCO9Fy5cyNmzZzl48CCvvfbaXQdw48YNFixYwJo1a9i5cyfjxo3j1VdfBWDZsmVMnDiR/fv3M3HiRJYuXXrX5xPC0u51rXxRpfXXcsnLy6ujaIQwXZUJQ6/Xc/HiReNcUvfccw+ffPIJX375JeHh4XcdQFxcHF5eXrRu3Rq4OZX6kSNHyMjI4OzZs4wYMQKAESNGcPbsWTIzM+/6nEJYkrW1NUu6tqp0m5VnE8mRpCHqmSoThlar5Y033iizBkZtad26Nenp6cbZcHfu3AlAUlISvr6+aLVaYxw+Pj6ylrhoFBxsbVkeFsSMkIAKt3n9bCKFJSV1GJUQlTNpHMbAgQP5/vvviYiIqPUAXFxcWLt2La+//jpFRUX069cPV1dX8vPza+X4Fa1N2xR4e7tYOgSLagjt9/d1o4Wrllei4sutjy0uoa+fe41GhDeE9puTtL/2229SwigqKmLOnDmEhobi5+dX6sP75ptv3nUQvXv3pnfv3gCkp6fz8ccfExgYSEpKCnq9Hq1Wi16vJzU11fi0lqkyMnIxGFTVGzYyd7MIfGPQkNpvhT0zfGz5ILXs3FP/OB3P95dTmdG+ebVGhDek9puDtL/m7bey0lT4RdukhBEcHExwcHCNTm6KtLQ0vL29MRgMrFmzhgkTJhAYGEj79u3ZtWsXo0ePZteuXbRv3x4PD1nqUjQ+rVq14iXPXP7flXQSC0s/RnUtv4j1Z67yTMeWMo2IsCiNun2eDwtZvHgxv/32G8XFxfTp04dFixZhZ2dHTEwMCxcuJDs7G1dXVyIjI2nTpk21ji1XGE1TQ21/SUkJS09cKbfO01bLnI4tsbGu+nteQ21/bZH2m+cKw6SE8fPPP1dYd//999coqLoiCaNpasjtv3HjBpEXUyusXxjii6ura6XHaMjtrw3Sfgveklq8eHGp11lZWRQXF+Pr68t3331Xo6CEEOVzc3NjXpsS/nq5/EfI37iQwlOBebT295Pp0UWdMilhfP/996Ve6/V63nvvPVkPQwgz8fT05GUHB14/k1Bu/UcJuXgkXLWKYCgAACAASURBVOKFcJnpVtSdGvWgabVaZs6cyUcffVTb8Qgh/sPF0ZHFHSoep5EJxKam111Aosmr8SMXR48elcthIczMycmJF9t6VVj/96vXa23MkhBVMemWVP/+/Uslh4KCAnQ6HcuWLTNbYEKIm5o1a8a8NoZy+zQMwIozCUz3d+Cs3pYIX1fs7e3rPkjRJJiUMN56661Srx0cHGjdujXOzk13FLUQdcnT05MF9vbsupbFmeyCMvUbkwqAAo6k3mBuS9cmP8pZmIdJCaNbt25oNBpsbGyMZcXFxeh0OrPNMSWEKM3NyYnHQ5zY8Ptlrun0FW739tVsXvJIJDpLT7dmTqV+b4W4Gyb1YUydOpUzZ86UKjtz5gzTpk0zS1BCiIpNal1xn8Ytb55MZmtcGstOxpKT03THI4jaZVLCuHjxIl27di1V1qVLF86fP2+WoIQQFXN1dWV+sA+mXje8fj6ZbEkaohaYlDBcXFxITy/9+F56ejoODpUvBCOEMA8PNzeWhbdlYdd7eLK5Cy5W8Hglw6LeOJ9MamrFo8eFMIVJCWPIkCHMmzePixcvUlBQwIULF1iwYAHDhg0zd3xCiApYaTS42trQ1t+Pl8Pa0bFDOxYE+1S4/bq4GySnpNRhhKKxMSlhPP/88wQFBTFu3Di6d+/O+PHjad26NS+88IK54xNCVIObmxsvd2pO3+Ye5d6yeudqNvm6stOoC2GKas1Wq5QiKyuLZs2aNZhBezL5YNMk7Xfh1IWrvBtTdiT4//i7M7C5twWiqjvy8zfP5IMmXWFs27aN8+fPo9Fo8PDwQKPRcP78ebZt21ajgIQQ5ufXzJ3mDmWvMw4kXUcnS7+KGjApYbz99ttlVrrz8/Pj7bffNktQQoi7p9FomNmxFQs6tyxT9+qJK/yWkMLXlxIoKCg7EFCI8piUMHJzc8uM6nZxcSE7O9ssQQkhaoeVRoObvR1T2pTtDP8qMZtfs/J57fQ1ziUkYzAYLBChaEhMShhBQUHs37+/VNmBAwcICgoyS1BCiNrV1r3yaXz+mZjDK7/G8FpUNFlZWXUUlWhoTJoaZP78+UyfPp29e/fSokULrl69ys8//8zGjRvNHZ8QohZotVqWdG7Ba3/EV7pdAfDWpXSebwvezZrVTXCiwTDpCiM8PJxdu3bRuXNnCgoK6NKlC7t27SIsLMzc8QkhaomDvT0v3+tn0rb/76pcZYiyTLrCAAgICGD69OkAGAwGfvjhB9544w3p+BaiAXFxcWFRZxt+SL5Bdzd73rlU/ujvNJ0enV6PrVZbxxGK+szkhAFw/vx5tm7dyq5duygsLGT06NHmiksIYSbO9vYMv+fmmhlLQ524lF2An6aENZdKj9mIy86nXTOZJl38V5UJIyMjgx07drB161ZiYmIIDw8nPz+fnTt30rx587qIUQhhJvbW1nTyuJkUFnV2YNVtfRybLiWzNNQBe+tqfa8UjViln4Tp06fz008/ERwczNixYxk+fDi+vr707dtXJh4UopGJzS8uU3Y2M4fuPs0oMRhILyzGx94GK6sar+wsGrhKE0ZUVBROTk7069ePfv364evrW1dxCSHq2L2uZb8EfhWXztnMHM7mFAE3n5LxAlo5W9MvwINj14vo5W7HiZxiHvB2wc7Orm6DFnWq0oRx9OhRvvnmG7Zu3coHH3zAvffey8iRIykuLm4wc0kJIUxjbW3Nki4tee3U1VLlt5IF3FxDPBVIzS0h6uLNDvMj/+k3/z4pi+fbenGhwEBeURG2VhrsrK3p4eMuq/41EiZPPpiQkMD27dvZvn07cXFx9OnThyeeeIL+/fubO8a7IpMPNk3S/pq3/6eENHYlXq/VeJZ0uweHOkwa8vO34OSDAIGBgcyaNYv9+/fz73//m8DAQF566aUaBSSEqL96+LjX+jGPp2RhUIqc4hKqMUG2qGeqNb35nXQ6Hba2trUZT62TK4ymSdp/d+1PTEzkbwl5tRaPHTe/nRZw8z64FzCurTd+7m5mub0tP3/zXGHc1fNy9T1ZCCFqJiAggGdt0vg0/jrTWrjj5uFBTHY+N/ILiM8ropObA9nKivau9pzPLmRbfIZxXzetBqVX3D41adFt/y8BkoF3L6XhRRrPhbfFSvpEGwR5wFoIUS5/b28Wev93oaX2zVygmQv33bFdT0dHunm7EZ9biKedNW52thQWFvLa6WtVniMdyCvR42Ijf4oaAnmgWghx12y1WoLcnHC3t0Oj0eDg4MCs1h5V7ucMOMjFRYNhUsJYsWJFueUrV66s1WCEEI1HoKcHfndcONxnB15WcOu6JRdYduIKhbLOeINgUsLYsmVLueU7duyo1WCEEI2HRqNhdre2zLnXnx6ezizp1JxRXdrxQlg7RocEGrdTwPLf4ygsLjvSXNQvld44/OqrrwDQ6/XG/98SHx+Pu3vtP34nhGg8rDQa/FycGetS+qmbVk52aLiZLG5ZeTKWZWFBWMvUI/VWpQlj+/btABQXFxv/Dze/OXh5eREZGWne6IQQjZJWq+WFDoH89WyCsUwPpOQVEujiaLnARKUqTRj//Oc/AVi7di3PP/98nQQkhGgaPBwd8LPTklykN5atP5/AhNY+dHB3wlpmya13TLr2mzJlCnl5Nwfx6PV6vv76a7Zt2yaLxgshakyj0dCrnFHlm6+ksvTEFQpLSiwQlaiMSQljxowZxMXFAbBmzRr+/ve/s2nTJt544w2zBieEaNy6e1a8QNP5zKY7Uru+MumaLzY2lvbt2wOwc+dONm/ejKOjIyNGjGDRokV3HcTBgwd5++23UUqhlGL27NkMGTKEiIgIbG1tjVMmz58/nwceeOCuzyeEqB9sbGxY1N6fVeeSytRtjUvH306Lj6uLzI5dT5iUMKysrCguLubKlSu4uLgQEBCAwWAw3qa6G0opXnrpJT777DOCg4M5f/48jz32GIMHDwbgnXfeITg4+K7PI4Son5ydnVkQ7EPkxdLrixcDb19MwVOTwnPdg9DK01MWZ1LC6NevH3PnzuX69esMHz4cgEuXLtXagkpWVlbk5Ny8/MzJycHHx0dW9RKiCXFzc2NRZzsOJl3nl/ScUo/bZihY/8cVZndpI3NOWZhJCWPlypVs3boVa2trRo8eDUBWVhbPPvvsXQeg0WhYt24ds2bNwtHRkby8PDZu3Gisnz9/PkopwsLCeOGFF3B1db3rcwoh6h9ne3tGtvZjkK8rK84klKpL1hm4lJpOOx8vuT1lQdWa3txgMJCeno6Pj0+tBVBSUsJTTz3Fs88+S1hYGL/++ivz5s1j9+7dZGdn4+/vj06nY+XKleTl5bF69epaO7cQon5Ky81l0Q8Xy5T72mtZ1q8jNvLIrUWY9K5nZ2fzl7/8hf3792Ntbc3Jkyf57rvvOHXq1F2Pzzh37hypqamEhYUBEBYWhoODAzExMXTp0gW4OY36xIkTefrpp6t9fFkPo2mS9jf89r98rx+vn08uVZZSqGfWN6d4pWMgjo4VD/BrDO2/GxZdcW/ZsmU4Ozvz/fffG9fmDQ0NZe/evTUK6HZ+fn4kJydz+fJlAGJiYsjIyMDX19fYr6GUYs+ePcYntYQQjZ+LiwsvBZd/N2PFmQSSr9+Q1fvqmElXGD///DM//vgjNjY2xvuHHh4eZGRkVLFn1by9vXn11VeZO3eu8dirVq1Cp9MxY8YM9Ho9BoOBoKAgli1bdtfnE0I0HO5ubizp5siH566SXFR6oPA70akE2KYzSzrD64xJCcPFxYWsrKxSfReJiYl437a4yt0YNWoUo0aNKlO+bdu2Wjm+EKLhcrCxYVqQHyvPJpapS9QZSEjLoIWPlwUia3oqvSW1a9cuAMaNG8ecOXP45ZdfMBgMnDhxggULFjBhwoQ6CVII0bQ5OTmxoEMA9zqVXRb6vbgssnNzLRBV01PpU1Ldu3fnt99+QynFP/7xDz7//HMSExPx9/dn/PjxTJkypd4/4iad3k2TtL/xtj8nJ6dMZzjAq93bUFxSwg/J1wl0s6e9k6Oxz7WpMVend6W3pG7lEo1Gw5QpU5gyZUqNAhBCiNri4uLCw9432JJWUKp886nLnL81X2FqNgATAt3o6OuJVqut4ygbp0oThsFg4Jdffqn0SYT777+/1oMSQojKhLbwZ0va5VJl58uZ3HZzwg1IuMHL9/rh4lLxRIfCNJUmDJ1Ox+LFiytMGBqNhu+++84sgQkhREW0Wi1LOrfgtT/iTdr+9fPJLOyoxbWSsRuiapUmDAcHB0kIQoh6ycHevtzBfRXZn3SdcUGSMO6GzPAnhGiwXFxcWNSpOY7/+UvmZAVzg31ZHdGBvh6lk0OIgzV6vb6cowhTmdTpLYQQ9ZWzgwOLurclr0SPs7UWjUaDm709w4MC6e2by5v/WWvjVn/GstDW2MlcVDVS6RXGiRMn6ioOIYSoMSuNBhcb6zKP+RdoyiaGdaeusDMmkbScXPlSXE2SZoUQjZajddnvxDf08HNmHj9n5uEIPN0hEA9Hh3o/pqw+kD4MIUSj5WprQ0vHsqPDb8kH/no2gchfL5GRly9XHFWQhCGEaLQ0Gg3TO7Tk6RD/SrfLVjcTx4Y/rmCQpFEhSRhCiEbNSqOhuYsTLRyqniYkoUhPVqGuDqJqmKQPQwjR6Gk0GmZ0bMX1Ih3pBTo0JTo2xWaWu+0/ohOY27m1TJleDrnCEEI0CVYaDR72dgQ3c6GdtyevdG3FkIBmuN6xXVqRnq0X4ykqKrJInPWZJAwhRJPkaGvLgEAvXgpvSz/v0vNM/ZpdxF9OXSWvsNBC0dVPkjCEEE2alUZDRIBHuXVr/oinREaHG0nCEEI0eba2tizu1LxMeQGw9LfLbIiKJiYtE4PBUHbnJkQShhBCAE4ODizt2qrcumvAx7EZvPJrDCVNOGlIwhBCiP+wt7Vlabd7Kv3DeCXzRp3FU99IwhBCiNvY29gwv3PLCuv/fSW9yQ7uk4QhhBB3cLOzpZWTXbl1hUBmQdN85FYShhBC3EGj0fDn9i2Y37klj7fxoZNj6THO68/EU1BQUMHejZckDCGEKMetgX4dPd34U0iLUnVFwGunr/F/11IoKSlnMfFGShKGEEJUwdramqXd7sH+jvJtSdksPXGFL89eobAJDPKThCGEECawt7HhPs/y1wQ/kVfC8j/iyW3kSUMShhBCmGhAc+9K67+9lk5BcTFRqdcpLi6uo6jqjiQMIYQwka2tLYs7t6iw/v+yCnjtZCxb49JYdjKWgkaWNCRhCCFENTjZ27M8LIin2vrQ1aXi1fwAtpyKJTG3oNFMKSLrYQghRDVZW1nRppkbbZq5MSIvj5VnE8vd7owBzpy7hhUwwMuJUB/3Br1+uFxhCCHEXXBycmJBsE+l2xiA79Pz+OvZBNafutxgR4pLwhBCiLvk5ubGos4tuN/LhcF3Pnt7h0SdgZScXE5n5jS4MRxyS0oIIWqBs709I1v7AX6EpKSw6Wo2+RVs++6FZOP/H2/pwb1e7mi12jqJ825IwhBCiFoW6OvLK76+FBUX85eTsZVu+9nVTLiayfwgTzw8yl/Iqb6QW1JCCGEmdjY2vNq9DZOCfOnpXvm9qtUxGaSmpdVRZDUjCUMIIczIVqulg4crI9sEVLntutjrpNXjpCEJQwgh6oBWq+XV7m0Y6+eMSyVP1a6Nvc6FuKuoevgklfRhCCFEHbHVaunRwp8eLcCgFNm6Yq6lpfOvpLxS232aWoR32iXmhrXFqh6N2agXVxgHDx5kzJgxjB49mlGjRvHNN98AcOXKFcaPH8/QoUMZP348sbGxlg1UCCFqiZVGg7udLZ2aB/D8Pe5l6tMUZObXr8kMLZ4wlFK89NJLvPnmm2zfvp0333yTBQsWYDAYWLZsGRMnTmT//v1MnDiRpUuXWjpcIYSodd7e3gwv5z7VmrPXKCiqP6v7WTxhAFhZWZGTkwNATk4OPj4+ZGVlcfbsWUaMGAHAiBEjOHv2LJmZmZYMVQghzOL+dq3LLX/t1FUWRUWz5ddoi8+Aa/E+DI1Gw7p165g1axaOjo7k5eWxceNGkpKS8PX1NQ5m0Wq1+Pj4kJSUVK1nlT09nc0Ver3n7e1i6RAsStov7W9o/to/hHmHL5Rbd9wAx0/G8lqvVvh5elZ5LHO03+IJo6SkhA8++IANGzYQFhbGr7/+ynPPPcebb75ZK8fPyMjFYKh/TxuYm7e3C2lpOZYOw2Kk/dL+htr+F9t68dal9ArrlxyL4/l7buDtXfHaHHfTfisrTYVftC1+S+rcuXOkpqYSFhYGQFhYGA4ODtjZ2ZGSkoJerwdAr9eTmpqKv7+/JcMVQgizatasGS93ak4Xt4oH+q2NvU6eBfo2LJ4w/Pz8SE5O5vLlywDExMSQkZFBq1ataN++Pbt27QJg165dtG/fvt4PnRdCiLvl4uDA+HbNaelkV+E2K09dJTs7uw6jAo2qB6NDduzYwYcffmicI37OnDkMHjyYmJgYFi5cSHZ2Nq6urkRGRtKmTZtqHVtuSTVN0n5pf2Nov0Ep8kr0OFtrWXz8UrnbzA/xxcPVtVSZuW5J1YuEYU6SMJomab+0v7G1/98novmjnNnQ73G0Y3rHlqXKGm0fhhBCiKo93KFlueWx+UVkZGRQYjCQnF9k1uVgLf6UlBBCiKrZ2dmxuHMLVv4RX6bur5czgZtj1LTAusGdzBKDXGEIIUQD4WRvX+40IrfTA89+e5pCM6zmJwlDCCEaEG9vb8aZcG9o+YkrtX5uSRhCCNHAhIa2Y4pr1dstioqu1fNKwhBCiAYoJKRdlbenapskDCGEaKC8vb1Z0CGATu6OzGtj/kHNkjCEEKIBc3NyYmK7QDw9PVnUuUWpulU92tXqueSxWiGEaCSc7e1Z1aOd2QYuyhWGEEIIk0jCEEIIYRJJGEIIIUwiCUMIIYRJJGEIIYQwSaN/SsrKSmPpECymKbcdpP3Sfml/be/X6NfDEEIIUTvklpQQQgiTSMIQQghhEkkYQgghTCIJQwghhEkkYQghhDCJJAwhhBAmkYQhhBDCJJIwhBBCmEQShhBCCJNIwmjAsrKy+POf/8zQoUMZOXIks2fPJjMzE4CTJ08yatQohg4dypNPPklGRoaFozWvv/3tb4SEhHDx4kWgabW/qKiIZcuWMWTIEEaOHMmSJUsAuHLlCuPHj2fo0KGMHz+e2NhYywZqJgcPHmTMmDGMHj2aUaNG8c033wCNt/2RkZFERESU+rxD5e2ttfdCiQYrKytL/fLLL8bXb7zxhnr55ZeVXq9XgwcPVlFRUUoppdavX68WLlxoqTDN7vTp02ratGlq4MCB6sKFC02u/a+99ppauXKlMhgMSiml0tLSlFJKTZ48WW3btk0ppdS2bdvU5MmTLRajuRgMBhUeHq4uXLiglFLq3Llzqlu3bkqv1zfa9kdFRanExETj5/2WytpbW++FJIxGZN++fWrKlCnq999/Vw899JCxPCMjQ3Xr1s2CkZlPUVGR+tOf/qTi4+ONv0BNqf25ubkqLCxM5ebmlipPT09XYWFhqqSkRCmlVElJiQoLC1MZGRmWCNNsDAaD6tmzpzp+/LhSSqn/+7//U0OGDGkS7b89YVTW3tp8Lxr9bLVNhcFg4N///jcREREkJSUREBBgrPPw8MBgMHD9+nXc3d0tGGXte/vttxk1ahTNmzc3ljWl9sfHx+Pu7s7f/vY3jh07hpOTE3PnzsXe3h5fX1+0Wi0AWq0WHx8fkpKS8PDwsHDUtUej0bBu3TpmzZqFo6MjeXl5bNy4kaSkpCbR/lsqa69SqtbeC+nDaCRee+01HB0dmTRpkqVDqTMnTpzg9OnTTJw40dKhWIxeryc+Pp4OHTqwZcsW5s+fz7PPPkt+fr6lQ6sTJSUlfPDBB2zYsIGDBw/y3nvv8dxzzzWZ9tc1ucJoBCIjI4mLi+P999/HysoKf39/EhMTjfWZmZlYWVk1um/XUVFRxMTEMGjQIACSk5OZNm0akydPbhLtB/D398fa2poRI0YA0LVrV5o1a4a9vT0pKSno9Xq0Wi16vZ7U1FT8/f0tHHHtOnfuHKmpqYSFhQEQFhaGg4MDdnZ2TaL9t/j7+1fYXqVUrb0XcoXRwK1Zs4bTp0+zfv16bG1tAejUqROFhYUcP34cgM2bN/Pggw9aMkyzmD59OkeOHOH777/n+++/x8/Pj48//pinnnqqSbQfbt5u69WrF0ePHgVuPg2TkZHBPffcQ/v27dm1axcAu3bton379o3udoyfnx/JyclcvnwZgJiYGDIyMmjVqlWTaP8tnp6eFba3srrqkgWUGrDo6GhGjBjBPffcg729PQDNmzdn/fr1/PbbbyxbtoyioiICAwN566238PLysnDE5hUREcH7779PcHBwk2p/fHw8ixYt4vr161hbW/Pcc8/Rv39/YmJiWLhwIdnZ2bi6uhIZGUmbNm0sHW6t27FjBx9++CEazc2V4ubMmcPgwYMbbftXrFjBN998Q3p6Os2aNcPd3Z3du3dX2t7aei8kYQghhDCJ3JISQghhEkkYQgghTCIJQwghhEkkYQghhDCJJAwhhBAmkYQhmrQDBw7Qv39/QkNDOXv2bJ2eOzExkdDQUPR6fZ2etzHasWMHTz75pFmOPXnyZL788kuzHLuhkcdqG4GIiAjS09PRarU4ODjQr18/lixZgpOTk6VDA+Ddd98lLi6O1atXWzqUMgYPHszChQsZPHhwqfLExEQeeugh4+v8/HwcHByMz/p/+OGHhIeHV+tcERERrFixgt69e9994FVYuHAhu3btwsbGBoDAwEAGDhzI9OnTcXFxMekYdxvv5MmTOXnyJNbW1tja2tKjRw+WLl2Kj4+PSfH7+vry/PPP1+jctWny5MmMGjWKcePGWToUi5MrjEbi/fff58SJE2zdupXTp0/z3nvvVWt/pRQGg8FM0dVfiYmJtGvXrkx5QEAAJ06cMP4D2L59u/F1dZOFJUybNo0TJ07wyy+/sGrVKk6ePMljjz1Wp/MsLV26lBMnTnDgwAHy8/OJjIyss3OL2icJo5Hx9fXlgQceIDo6Gri5kNCECRMIDw9n1KhRHDt2zLjt5MmTWbt2LRMmTKBr167Ex8cTHR3N1KlT6dmzJ7179+b9998Hbs6Gu3HjRgYPHkyvXr2YO3cu169fB+DatWuEhISwdetWBgwYQK9evYwJ64cffuCDDz5g7969hIaGMmrUKAC+/vprhg0bRmhoKIMGDWLz5s2l2vHhhx/St29f+vbty5dffklISAhxcXEA6HQ6IiMjGTBgAL1792bp0qUUFhaW+34YDAY2bNjAwIEDuf/++3nppZfIyclBp9MZbweNHj26zBVGZSo7f2ZmJjNmzCA8PJyePXsyceJEDAYDL774IomJicycOZPQ0FA+/PBD4/tWUlJi/HmsW7eOCRMmEBoaypNPPmlcEAtg27ZtDBw4kF69erF+/XoiIiL46aefqozXzs6OLl268N5773H9+nW2bNkCwNWrV3niiSfo1asXvXr1Yt68eWRnZwOUGy/cHEXdp08fwsLCePzxx42fs6q4uroyaNAgzp8/byyLiYkxftaGDh3Knj17APj888/ZuXMnH3/8MaGhocycORPA+PkLDQ1l+PDhHDhwwHisLVu28Nhjjxlfh4SE8O9//5shQ4YQHh7OX/7yF26/mfLVV18xbNgwevTowbRp00hISDDWHT16lAcffJCwsDCWL1+O3IS5zV1OyS7qgYEDB6qjR48qpZRKTExUw4cPV2vXrlXJycmqZ8+e6tChQ0qv16sjR46onj17GufBnzRpkurfv7+6ePGiKi4uVjk5OapPnz7q448/VoWFhSonJ0edPHlSKaXUJ598osaNG6eSkpJUUVGRWrJkiXr++eeVUkrFx8er4OBgtXjxYlVQUKDOnTunOnbsqC5duqSUUuqdd95R8+bNKxXzwYMHVVxcnDIYDOrYsWOqS5cu6vTp00oppQ4fPqx69+6tLl68qPLz89W8efNUcHCwio2NVUoptXLlSjVjxgyVlZWlcnJy1IwZM9Tq1avLfW++/PJLNXjwYHX16lWVm5urnnnmGTV//nxj/e3HrYyp51+9erVasmSJ0ul0SqfTqaioKOPCRrf/nG5/34qLi40/j0GDBqnLly+rgoICNWnSJPXWW28ppZSKjo5W3bp1U1FRUaqoqEi98cYbqkOHDqWOd7sFCxaoNWvWlCl/8cUX1dy5c5VSSsXGxqojR46ooqIilZGRoSZOnKhWrFhh3PbOeG+9nzk5OaqoqEitWLFCjRo1qsL3bNKkSeqLL75QSimVmZmppkyZombOnKmUUiovL0/169dPffXVV6q4uFidOXNG9ezZU0VHR1cY/549e1RycrLS6/Vq9+7dqmvXriolJUUppdTXX3+tJkyYYNw2ODhYTZ8+Xd24cUMlJCSoXr16qcOHDyullDpw4IAaPHiwunTpkiouLlbr169X48ePV0r9d+2UvXv3Kp1OpzZt2qTat29vbEdTJ1cYjcQzzzxDeHg4EydOpEePHsycOZPt27fTr18/+vfvj5WVFX369KFTp04cPnzYuN/YsWNp164d1tbWHDp0CC8vL5588kns7Oxwdnama9euwM0J/J5//nn8/PywtbVl9uzZ7N+/3/jtGGD27NnY29tz7733cu+995b6NnmnAQMG0LJlSzQaDT179qRPnz7GyQL37t3Lww8/TLt27XBwcODZZ5817qeU4osvvmDRokW4u7vj7OzMjBkz2L17d7nn2blzJ//7v/9LixYtcHJy4oUXXmDPnj2l4q6Oqs5vbW1NWloaiYmJ2NjYEB4ebuz3MMXDDz9M69atsbe358EHH+TcuXMA7Nu3j4EDBxIeHo6trS1z5syp1nFv8fHx4caNGwC0atWKPn36YGtri4eHB1OnTiUqKqrS/R999FGcnZ2xtbXl7lkwxAAABttJREFU2Wef5fz58+Tk5FS4/YoVKwgLC+O+++4jKyvLuHzsoUOHCAwM5JFHHsHa2poOHTowdOhQ9u3bV+Gxhg0bhq+vL1ZWVgwfPpxWrVpx6tSpCrf/85//jKurKwEBAfTq1cv4edy8eTPTp08nKCgIa2trZs6cyblz50hISOCHH36gXbt2PPjgg9jY2DBlypRGOwdZTcj05o3E+vXry3ROJiYmsm/fPg4ePGgsKykpoVevXsbXt09xnJSURMuWLcs9fmJiIs888wxWVv/9jmFlZVVqrezbf7EcHBwqvVd++PBh1q9fT2xsLAaDgcLCQoKDgwFITU2lU6dO5caYmZlJQUEBDz/8sLFMVdL/kpqaSmBgoPF1YGAgJSUlZGRk4OvrW2F8Fanq/NOmTeNvf/ub8Ymd8ePHM336dJOP7+3tbfz/7e9hamoqfn5+pepqMl17SkoKbm5uAKSnp7Ny5UqOHz9OXl4eSilcXV0r3Fev17N27Vr27dtnnDIebq4tX1FH+iuvvMK4ceO4cOECM2fOJDk5mYCAABISEjh16lSpviC9Xm+8ZVmebdu2sWnTJuPto/z8fLKysirc/s73Mi8vD7j5WV61alWp/hT1nynA73yfNRpNo50SvSYkYTRi/v7+jB49mhUrVlS4ze3fUv39/Y33ke/k5+fHqlWrjOsO3O7atWuVxnHnN2GdTsecOXOIjIxk0KBB2NjYMGvWLOO9Yh8fH1JSUozbJyUlGf9/a62H3bt3m/QH38fHp9T96cTERKytrfH09Kxy3/JUdX5nZ2cWLlzIwoULuXjxIlOmTKFz587cf//9NTrfLT4+Ply5csX4urCw0NiHZKq8vDx+/vlnY5/AmjVr0Gg07Ny5E3d3d7799luWL19e4f47d+7ku+++Y9OmTTRv3pycnBx69Ohh0j3+kJAQnn76aZYvX87WrVvx9/enR48ebNq0qdzt7/zMJCQk8Morr/DJJ58QGhqKVqtl9OjR1Wj9f/n7+zNz5sxyk1NcXBzJycnG10qpUp+/pk5uSTVio0aN4uDBg/z444/o9XqKioo4duxYqV+I2w0YMIC0tDQ++eQTdDodubm5/P777wA89thjrFu3zvjHNzMzk2+//dakODw9PUlISDB+C9fpdOh0Ojw8PLC2tubw4cPG9RwAHnzwQbZs2UJMTAwFBQVs2LDBWGdlZcW4ceNYtWqV8eomJSWFH3/8sdxzjxgxgk8//ZT4+Hjy8vJYu3Ytw4YNw9q6Zt+Vqjr/wYMHiYuLQymFi4sLWq3W+MfPy8uL+Pj4Gp136NChfP/99/z222/odDreffddkztjdTodp0+f5plnnsHV1dV4dZSXl4ejoyMuLi6kpKTw0Ucfldrvznjz8vKwtbWlWbNmFBQUsGbNmmq1YcyYMaSnp/Pdd9/9//bu4BWeMAzg+FdqD9uc3IhccHGx02xjMQ2r4WATqbWKiU0OTtqSC+WOg8uKGgenvSgHEUL5EygHbJm7i6StXYffYdsJ/X4ae/it1vM5z/S8M0097/s+7/TQ29vL4+MjBwcHFAoFCoUC19fXZLNZoPjNvJ+I5HI5ampqvB4O+/v7vgvunyUSCXZ2drz7X15eOD4+BsA0Te7v7zk9PeXt7Y29vT2enp7KilONJGFUsfr6etLpNNvb20QiEUzTxHGcf27fKIrC7u4ul5eXdHd3Mzg46J2qsm2baDRKMpkkFAoRj8e/3D9+r9S8SNd1RkdHURSF5eVlFhYWCIfDHB4eEo1GvetN02RqagrbtrEsy6ujlBpELS4u0tzcTDweR1VVpqenP8y+3xsbG2N4eJjJyUn6+/sJBALePnq5vorvui4zMzOEQiHGx8eZmJigs7MTKDZ82traQtM0HMf5VszW1lZWVlZIpVIYhkEwGKSurs57J39TOmWk6zpLS0u0t7eTyWQIBoNAseZ0e3uLpmnMzc0xMDDw4f7P4x0ZGaGhoQHDMBgaGqKjo+NbzxAIBLBtm3Q6jaIoOI7D0dERhmHQ09PD+vo6+XweKNZKHh4e0DSN+fl5WlpaSCaTJBIJurq6uLu7Q1XVb8UvsSyL2dlZUqkUqqoSi8W4uroCig2pNjc32djYQNd1XNctO041kh/3xI+XzWaJxWLc3NyUvTKoNq+vr4TDYU5OTmhqaqr0cMQvISsM8SOdnZ2Rz+d5fn5mbW2Nvr6+X58sLi4uyOVy3g9wbW1tNDY2VnpY4heRhCF+pEwmQyQSwbIsamtrWV1drfSQKu78/BzDMDAMA9d1vaK1EP+LbEkJIYTwRVYYQgghfJGEIYQQwhdJGEIIIXyRhCGEEMIXSRhCCCF8kYQhhBDClz+UihxCbKg5agAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Graph the trade-off between effeciency and accuracy:\n",
    "graph_confidence(preds_df, save=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "from MLFunctions import acc_to_uncertainty, proportion_to_uncertainty"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Select Threshold \n",
    "to use in production based on target ammount of data retained or based on target accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "% of test data retained 69.91720331186752\n",
      "Exact accuracy 86.71052631578947\n",
      "Threshold: 0.09997199118838754\n"
     ]
    }
   ],
   "source": [
    "# Retain 70% of data\n",
    "threshold = proportion_to_uncertainty(preds_df, target_prop=70)\n",
    "print('Threshold:', threshold)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "% of test data retained 54.7378104875805\n",
      "Exact accuracy 90.08403361344538\n",
      "Threshold: 0.07020009347002944\n"
     ]
    }
   ],
   "source": [
    "# Reach 90% of accuracy\n",
    "threshold = acc_to_uncertainty(preds_df, target_acc=90)\n",
    "print('Threshold:', threshold)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Deallocate the model:\n",
    "clear_memory(assignment_model)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "nlp-workspace",
   "language": "python",
   "name": "nlp-workspace"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "toc-autonumbering": false,
  "toc-showcode": false,
  "toc-showmarkdowntxt": false
 },
 "nbformat": 4,
 "nbformat_minor": 4
}