transformers/ex3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Analytics: 5th Assignment (Part 2: Exercise 3)\n",
    "## MSc in Data Science (2023/2024)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-03-11 18:48:58.938503: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
      "2024-03-11 18:48:58.938541: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
      "2024-03-11 18:48:58.940077: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n",
      "2024-03-11 18:48:58.949531: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n",
      "To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "2024-03-11 18:49:00.155716: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n"
     ]
    }
   ],
   "source": [
    "import tasks.preprocessing, tasks.models, tasks.tuning, tasks.util\n",
    "\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "import sklearn\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from tqdm.auto import tqdm\n",
    "\n",
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'\n",
    "\n",
    "INPUT_DIR = \"input\"\n",
    "INPUT_MODEL_PATH = os.path.join(INPUT_DIR, \"models\")\n",
    "OUTPUT_DIR = \"output\"\n",
    "INTERMEDIATE_DIR = \"intermediate\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing with  /physical_device:GPU:0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-03-11 18:49:01.857252: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n",
      "2024-03-11 18:49:01.924397: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n",
      "2024-03-11 18:49:01.924723: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:901] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355\n"
     ]
    }
   ],
   "source": [
    "gpus = tf.config.list_physical_devices(\"GPU\")\n",
    "print(\"Executing with \", gpus[0].name if len(gpus) != 0 else \"CPU\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset\n",
    "\n",
    "Acquiring and preprocessing our data with the goal of eventually acquiring a sufficient representation of our text is the most difficult and time-consuming task. We thus split it in distinct phases:\n",
    "\n",
    "* Original dataset acquisition and parsing\n",
    "* Qualitative analysis and preprocessing\n",
    "* Transformation for the NLP task\n",
    "\n",
    "Note that due to the relative custom code complexity, most of the code used in this section was developed and imported from python source files located in the `tasks` module. In-depth documentation and implementation details can be found in these files. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading training dataset...\n",
      "\tReading data...\n",
      "\tParsing data...\n",
      "\tGetting words...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ef05b92e97514502af7f139fa74e95e0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/12544 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tGetting POS tags...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "78c4f0cee8bb41c8b36625ad5ae192c9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/12544 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tGetting Sentence ids...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "def84e3cca78486ca78f17a6093a7b50",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/12544 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading validation dataset...\n",
      "\tReading data...\n",
      "\tParsing data...\n",
      "\tGetting words...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ad41a7e28e2c4a539151ec20ccbe5139",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/2001 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tGetting POS tags...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "af1baab437ea4be4bf5682a06ae3a55e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/2001 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tGetting Sentence ids...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3ccb2f4a387b42fa8c654543b3f14da2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/2001 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading test dataset...\n",
      "\tReading data...\n",
      "\tParsing data...\n",
      "\tGetting words...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a4d8a53332c949eda82ae12ea9b46ff0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/2077 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tGetting POS tags...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "72239010d4684063accb412ed60bd438",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/2077 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\tGetting Sentence ids...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ad72ff22850f47e3bcacc7fa59114037",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/2077 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training data shape: (207227, 3)\n",
      "Validation data shape: (25511, 3)\n",
      "Test data shape: {test_df.shape}\n"
     ]
    }
   ],
   "source": [
    "print(\"Loading training dataset...\")\n",
    "train_df = tasks.preprocessing.conllu_to_pd(\n",
    "    \"input/UD_English-EWT/en_ewt-ud-train.conllu\"\n",
    ")\n",
    "print(\"Loading validation dataset...\")\n",
    "val_df = tasks.preprocessing.conllu_to_pd(\n",
    "    \"input/UD_English-EWT/en_ewt-ud-dev.conllu\"\n",
    ")\n",
    "print(\"Loading test dataset...\")\n",
    "test_df = tasks.preprocessing.conllu_to_pd(\n",
    "    \"input/UD_English-EWT/en_ewt-ud-test.conllu\"\n",
    ")\n",
    "\n",
    "print(\n",
    "    f\"Training data shape: {train_df.shape}\\nValidation data shape: {val_df.shape}\"\n",
    "    \"\\nTest data shape: {test_df.shape}\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below we can see a preview of our parsed training dataset. Our preprocessing exploits pandas's ordering scheme in order to make sure the words are inserted in the order they appear in the sentence. This ordering will prove important later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>words</th>\n",
       "      <th>pos</th>\n",
       "      <th>sent_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>al</td>\n",
       "      <td>PROPN</td>\n",
       "      <td>weblog-juancole.com_juancole_20051126063000_EN...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-</td>\n",
       "      <td>PUNCT</td>\n",
       "      <td>weblog-juancole.com_juancole_20051126063000_EN...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>zaman</td>\n",
       "      <td>PROPN</td>\n",
       "      <td>weblog-juancole.com_juancole_20051126063000_EN...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>:</td>\n",
       "      <td>PUNCT</td>\n",
       "      <td>weblog-juancole.com_juancole_20051126063000_EN...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>american</td>\n",
       "      <td>ADJ</td>\n",
       "      <td>weblog-juancole.com_juancole_20051126063000_EN...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>207222</th>\n",
       "      <td>on</td>\n",
       "      <td>ADP</td>\n",
       "      <td>reviews-319816-0029</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>207223</th>\n",
       "      <td>my</td>\n",
       "      <td>PRON</td>\n",
       "      <td>reviews-319816-0029</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>207224</th>\n",
       "      <td>car</td>\n",
       "      <td>NOUN</td>\n",
       "      <td>reviews-319816-0029</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>207225</th>\n",
       "      <td>)</td>\n",
       "      <td>PUNCT</td>\n",
       "      <td>reviews-319816-0029</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>207226</th>\n",
       "      <td>.</td>\n",
       "      <td>PUNCT</td>\n",
       "      <td>reviews-319816-0029</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>207227 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           words    pos                                            sent_id\n",
       "0             al  PROPN  weblog-juancole.com_juancole_20051126063000_EN...\n",
       "1              -  PUNCT  weblog-juancole.com_juancole_20051126063000_EN...\n",
       "2          zaman  PROPN  weblog-juancole.com_juancole_20051126063000_EN...\n",
       "3              :  PUNCT  weblog-juancole.com_juancole_20051126063000_EN...\n",
       "4       american    ADJ  weblog-juancole.com_juancole_20051126063000_EN...\n",
       "...          ...    ...                                                ...\n",
       "207222        on    ADP                                reviews-319816-0029\n",
       "207223        my   PRON                                reviews-319816-0029\n",
       "207224       car   NOUN                                reviews-319816-0029\n",
       "207225         )  PUNCT                                reviews-319816-0029\n",
       "207226         .  PUNCT                                reviews-319816-0029\n",
       "\n",
       "[207227 rows x 3 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As mentioned above, our dataset features words connected with punctuation such as \"don't\". These are normally treated as two words, with the first being their intuitive POS tag (\"do\" - AUX) and the second as part of the first (\"n't\" - PART).\n",
    "\n",
    "This dataset contains both the full words and their split versions, with only the latter featuring valid POS tags. The former are instead marked by a pseudo-tag (here \"_\")."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>words</th>\n",
       "      <th>pos</th>\n",
       "      <th>sent_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>176</th>\n",
       "      <td>don't</td>\n",
       "      <td>_</td>\n",
       "      <td>weblog-juancole.com_juancole_20051126063000_EN...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>704</th>\n",
       "      <td>won't</td>\n",
       "      <td>_</td>\n",
       "      <td>weblog-juancole.com_juancole_20051126063000_EN...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>868</th>\n",
       "      <td>don't</td>\n",
       "      <td>_</td>\n",
       "      <td>weblog-blogspot.com_healingiraq_20040409053012...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1058</th>\n",
       "      <td>don't</td>\n",
       "      <td>_</td>\n",
       "      <td>weblog-blogspot.com_healingiraq_20040409053012...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1078</th>\n",
       "      <td>doesn't</td>\n",
       "      <td>_</td>\n",
       "      <td>weblog-blogspot.com_healingiraq_20040409053012...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>207078</th>\n",
       "      <td>couldn't</td>\n",
       "      <td>_</td>\n",
       "      <td>reviews-319816-0025</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>207091</th>\n",
       "      <td>don't</td>\n",
       "      <td>_</td>\n",
       "      <td>reviews-319816-0025</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>207096</th>\n",
       "      <td>employees'</td>\n",
       "      <td>_</td>\n",
       "      <td>reviews-319816-0025</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>207164</th>\n",
       "      <td>i'm</td>\n",
       "      <td>_</td>\n",
       "      <td>reviews-319816-0027</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>207186</th>\n",
       "      <td>didn't</td>\n",
       "      <td>_</td>\n",
       "      <td>reviews-319816-0028</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2613 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             words pos                                            sent_id\n",
       "176          don't   _  weblog-juancole.com_juancole_20051126063000_EN...\n",
       "704          won't   _  weblog-juancole.com_juancole_20051126063000_EN...\n",
       "868          don't   _  weblog-blogspot.com_healingiraq_20040409053012...\n",
       "1058         don't   _  weblog-blogspot.com_healingiraq_20040409053012...\n",
       "1078       doesn't   _  weblog-blogspot.com_healingiraq_20040409053012...\n",
       "...            ...  ..                                                ...\n",
       "207078    couldn't   _                                reviews-319816-0025\n",
       "207091       don't   _                                reviews-319816-0025\n",
       "207096  employees'   _                                reviews-319816-0025\n",
       "207164         i'm   _                                reviews-319816-0027\n",
       "207186      didn't   _                                reviews-319816-0028\n",
       "\n",
       "[2613 rows x 3 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "invalid_idx = train_df.pos == \"_\"\n",
    "train_df[invalid_idx]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"don't won't doesn't haven't didn't others it's elena's women's children's i'm people's musharraf's sharon's hamas's right's cannot isn't one's let's reporter's he's that's pakistan's world's bush's military's sharif's can't couldn't\""
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\" \".join(train_df[invalid_idx].words.unique()[:30])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below we can see an example of a word being contained both times in the dataset, one in full with the pseudo-tag, and the other as split words with valid POS tags."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>words</th>\n",
       "      <th>pos</th>\n",
       "      <th>sent_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>176</th>\n",
       "      <td>don't</td>\n",
       "      <td>_</td>\n",
       "      <td>weblog-juancole.com_juancole_20051126063000_EN...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>177</th>\n",
       "      <td>do</td>\n",
       "      <td>AUX</td>\n",
       "      <td>weblog-juancole.com_juancole_20051126063000_EN...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>178</th>\n",
       "      <td>n't</td>\n",
       "      <td>PART</td>\n",
       "      <td>weblog-juancole.com_juancole_20051126063000_EN...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     words   pos                                            sent_id\n",
       "176  don't     _  weblog-juancole.com_juancole_20051126063000_EN...\n",
       "177     do   AUX  weblog-juancole.com_juancole_20051126063000_EN...\n",
       "178    n't  PART  weblog-juancole.com_juancole_20051126063000_EN..."
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.iloc[176:179]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We thus remove the full words including the pseudo-tag from our datasets, ensuring that all target POS tags will be compliant with the UPOS scheme."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df = train_df[~invalid_idx]\n",
    "val_df = val_df[val_df.pos != \"_\"]\n",
    "test_df = test_df[test_df.pos != \"_\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Qualitative Analysis\n",
    "\n",
    "We analyze our dataset in two granualities: sentences and individual words. We begin by analyzing how many words are in each sentence, which will give us an idea on the size of context available for each word."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def length_sentences(df: pd.DataFrame) -> float:\n",
    "    lengths = df.groupby([\"sent_id\"]).agg(lambda x: len(x))\n",
    "    return lengths.words\n",
    "\n",
    "\n",
    "train_length = length_sentences(train_df)\n",
    "val_length = length_sentences(val_df)\n",
    "test_length = length_sentences(test_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Figure saved to output/ex_2_dataset_stats.png\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkQAAAHFCAYAAAAT5Oa6AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAABle0lEQVR4nO3deVxU5f4H8M+w7yOLMKIomIgguKRmagmmoihqeosUI7maZW6h4lapaKVXK5eLlWXuy9W6qZkaCqmoF1cUzQ3DUFxAXGAARdbn9wc/ThwWF0Rmhvm8X695vTzPec45zwPDzNdnVQghBIiIiIj0mIGmC0BERESkaQyIiIiISO8xICIiIiK9x4CIiIiI9B4DIiIiItJ7DIiIiIhI7zEgIiIiIr3HgIiIiIj0HgMiIiIi0nsMiEjnrF69GgqFAmZmZrh69WqF835+fvD29tZAyYD9+/dDoVDgv//9r0ae/7SuXLmCvn37ws7ODgqFAmFhYZoukkxcXBwiIiKQmZmp6aLUGk2+fzUpNDQUrq6umi7GM9u1axciIiI0XQyqBgZEpLPy8vLwySefaLoYOm3ChAk4evQoVq5cicOHD2PChAmaLpJMXFwcZs+erVcBEem2Xbt2Yfbs2ZouBlUDAyLSWb1798bGjRtx+vRpTRel1uXm5qImtiE8e/YsXnrpJbz++ut4+eWX0aRJkxooHVFFBQUFKCws1HQxiKrEgIh01pQpU2Bvb4+pU6c+Mt+VK1egUCiwevXqCucUCoWseTsiIgIKhQJnzpzBm2++CaVSCTs7O0ycOBGFhYVITExE7969YW1tDVdXVyxYsKDSZz58+BATJ06ESqWCubk5fH19cerUqQr5Tpw4gf79+8POzg5mZmZo27YtfvzxR1me0i7CPXv2YPjw4ahfvz4sLCyQl5dXZZ1TUlLw9ttvw9HREaampvD09MRXX32F4uJiAH937SUlJeG3336DQqGAQqHAlStXqrznTz/9hI4dO0KpVMLCwgJNmzbF8OHDZXmysrIQHh4ONzc3mJiYoGHDhggLC8P9+/cr/NzHjh2LdevWwdPTExYWFmjdujV27Ngh+11MnjwZAODm5iaVcf/+/VKezZs3o1OnTrC0tISVlRV69epV4eccGhoKKysrJCUloU+fPrCysoKLiwsmTZpU4WeYl5eHOXPmwNPTE2ZmZrC3t0e3bt0QFxcn5RFC4JtvvkGbNm1gbm4OW1tbvPHGG/jrr79k9zp16hQCAwOl34GzszP69u2L69evV/kzLuvgwYN4+eWXYW5ujoYNG2LGjBkoKiqSyuDu7o5evXpVuC4nJwdKpRJjxoyp8t5vvvkmWrZsKUvr168fFAoFfvrpJynt5MmTUCgU+PXXX6W0s2fPYsCAAbC1tYWZmRnatGmDNWvWyO5V+v5at24dJk2ahIYNG8LU1BRJSUkASt7THh4e0ntz7dq1T/QzKbVx40Z06tQJVlZWsLKyQps2bbBixQpZnpUrV6J169YwMzODnZ0dBg4ciAsXLsjy+Pn5wc/Pr8L9y3fflX6GfPnll1i4cCHc3NxgZWWFTp064ciRI7Lrvv76awCQ3q+P+7siLSKIdMyqVasEAHH8+HGxZMkSAUD8/vvv0nlfX1/RsmVL6Tg5OVkAEKtWrapwLwBi1qxZ0vGsWbMEAOHh4SE+/fRTER0dLaZMmSIAiLFjx4oWLVqIf//73yI6Olr885//FADEzz//LF2/b98+AUC4uLiIAQMGiF9//VWsX79eNGvWTNjY2IjLly9Leffu3StMTEzEq6++KjZv3iyioqJEaGhohbKW1rdhw4bivffeE7/99pv473//KwoLCyv9+aSnp4uGDRuK+vXri2XLlomoqCgxduxYAUB88MEHQggh1Gq1OHz4sFCpVKJLly7i8OHD4vDhw+Lhw4eV3jMuLk4oFAoxePBgsWvXLrF3716xatUqERISIuW5f/++aNOmjXBwcBALFy4UMTExYsmSJUKpVIrXXntNFBcXy37urq6u4qWXXhI//vij2LVrl/Dz8xNGRkbSz+jatWti3LhxAoDYsmWLVEa1Wi2EEOLzzz8XCoVCDB8+XOzYsUNs2bJFdOrUSVhaWopz585Jzxo2bJgwMTERnp6e4ssvvxQxMTFi5syZQqFQiNmzZ0v5CgoKRLdu3YSRkZEIDw8Xu3btEtu3bxcfffSR+M9//iPlGzlypDA2NhaTJk0SUVFRYuPGjaJFixbCyclJpKWlCSGEyMnJEfb29qJ9+/bixx9/FLGxsWLz5s1i1KhR4vz585X+jEv5+voKe3t74ezsLP7973+L3bt3i/HjxwsAYsyYMVK+JUuWCIVCIS5duiS7/uuvvxYAZD+D8pYtWyYAiJs3b0p1t7a2Fubm5mLkyJFSvvnz5wsjIyORlZUlhBDi4sWLwtraWrzwwgti7dq1YufOnWLIkCECgJg/f750XenfQcOGDcUbb7whtm/fLnbs2CHu3r0rvZ/L/324uLiIJk2aPPJnI4QQM2bMEADEoEGDxE8//ST27NkjFi5cKGbMmCHlmTt3rgAghgwZInbu3CnWrl0rmjZtKpRKpezn5evrK3x9fSs8Y9iwYbKylH6GuLq6it69e4tt27aJbdu2CR8fH2FraysyMzOFEEIkJSWJN954QwCQ3q+P+rsi7cKAiHRO2YAoLy9PNG3aVLRv3176wq2JgOirr76S5WvTpo30xVyqoKBA1K9fXwwaNEhKK/0iePHFF2UBwJUrV4SxsbF49913pbQWLVqItm3bioKCAtmzAgMDRYMGDURRUZGsvu+8884T/XymTZsmAIijR4/K0j/44AOhUChEYmKilNakSRPRt2/fx97zyy+/FACkD/7KzJs3TxgYGIjjx4/L0v/73/8KAGLXrl1SGgDh5OQkfdEKIURaWpowMDAQ8+bNk9K++OILAUAkJyfL7pmSkiKMjIzEuHHjZOnZ2dlCpVKJoKAgKW3YsGECgPjxxx9lefv06SM8PDyk47Vr1woAYvny5VXW8fDhw5W+P65duybMzc3FlClThBBCnDhxQgAQ27Ztq/JeVfH19RUAxC+//CJLHzlypDAwMBBXr14VQgiRlZUlrK2txYcffijL5+XlJbp16/bIZyQlJQkAYu3atUIIIQ4dOiQAiClTpgg3NzcpX8+ePUXnzp2l48GDBwtTU1ORkpIiu19AQICwsLCQ3h+lfwddu3aV5SsqKhLOzs5V/n08LiD666+/hKGhoRg6dGiVeTIyMoS5ubno06ePLD0lJUWYmpqK4OBgKe1pAyIfHx/Zf0SOHTsmAMgC5jFjxgi2NegmdpmRTjMxMcFnn32GEydOVOhqehaBgYGyY09PTygUCgQEBEhpRkZGaNasWaUz3YKDg6FQKKTjJk2aoHPnzti3bx8AICkpCRcvXsTQoUMBAIWFhdKrT58+SE1NRWJiouye//jHP56o7Hv37oWXlxdeeuklWXpoaCiEENi7d+8T3aesDh06AACCgoLw448/4saNGxXy7NixA97e3mjTpo2sPr169arQ1QUA3bp1g7W1tXTs5OQER0fHSn+e5e3evRuFhYV45513ZM8yMzODr69vhWcpFAr069dPltaqVSvZs3777TeYmZlV6AYsX0eFQoG3335b9lyVSoXWrVtLz23WrBlsbW0xdepULFu2DOfPn39sncqytrZG//79ZWnBwcEoLi7GgQMHpDz//Oc/sXr1aqlLcu/evTh//jzGjh37yPu/8MILcHV1RUxMDAAgOjoaPj4+ePvtt5GcnIzLly8jLy8Phw4dQo8ePaTr9u7di+7du8PFxUV2v9DQUDx48ACHDx+WpZd/zyYmJuLmzZtV/n08TnR0NIqKih7ZHXj48GHk5uYiNDRUlu7i4oLXXnsNv//++2OfU5W+ffvC0NBQOm7VqhUAPNF7lrQfAyLSeYMHD8aLL76Ijz/+GAUFBTVyTzs7O9mxiYkJLCwsYGZmViH94cOHFa5XqVSVpt29excAcOvWLQBAeHg4jI2NZa/Ro0cDAO7cuSO7vkGDBk9U9rt371aa19nZWTr/tLp27Ypt27ZJQUijRo3g7e2N//znP1KeW7du4cyZMxXqY21tDSFEhfrY29tXeI6pqSlyc3MfW57Sn1+HDh0qPG/z5s0VnlXZ787U1FT2u7t9+zacnZ1hYFD1x+KtW7cghICTk1OF5x45ckR6rlKpRGxsLNq0aYOPPvoILVu2hLOzM2bNmvVE71EnJ6cKaaXvqbK/v3HjxiE7OxsbNmwAACxduhSNGjXCgAEDHvuM7t27S8FBTEwMevbsCR8fHzg5OSEmJgb/+9//kJubKwuInva9VT5v6fmq/j4e5/bt2wCARo0aVZmn9BlVlbM67/9S5d+zpqamAPBE71nSfkaaLgDRs1IoFJg/fz569uyJ77//vsL50i/C8gNon+WD8XHS0tIqTSv9QHVwcAAATJ8+HYMGDar0Hh4eHrLjsv+jfhR7e3ukpqZWSL9586bs2U9rwIABGDBgAPLy8nDkyBHMmzcPwcHBcHV1RadOneDg4ABzc3OsXLmy0uur+9xH3eu///1vjc2Mq1+/Pg4dOoTi4uIqgyIHBwcoFAocPHhQ+jIsq2yaj48PNm3aBCEEzpw5g9WrV2POnDkwNzfHtGnTHlmW0oCvrNL3VNkv5WbNmiEgIABff/01AgICsH37dsyePVvWilGV7t27Y8WKFTh27BiOHj0qLWHx2muvITo6GlevXoWVlRVefvll6ZqnfW+Vf8+Wlr2qv4/HqV+/PgDg+vXrFVqpyj+jqnKWLaOZmRnUanWFfOUDatIPbCGiOqFHjx7o2bMn5syZg5ycHNk5JycnmJmZ4cyZM7L0X3755bmV5z//+Y9sWvzVq1cRFxcnzWjx8PCAu7s7Tp8+jfbt21f6Ktud9DS6d++O8+fP4+TJk7L0tWvXQqFQoFu3btWuF1Dype/r64v58+cDgDSrKzAwEJcvX4a9vX2l9anOontV/Q+8V69eMDIywuXLl6v8+T2tgIAAPHz4sNLZiKUCAwMhhMCNGzcqfaaPj0+FaxQKBVq3bo1FixahXr16FX4vlcnOzsb27dtlaRs3boSBgQG6du0qS//www9x5swZDBs2DIaGhhg5cuQT1bd79+5QKBSYMWOG7L49evTAvn37EB0dja5du8LY2Fh2zd69e6UAqNTatWthYWEhC54q4+HhgQYNGlT59/E4/v7+MDQ0xLfffltlnk6dOsHc3Bzr16+XpV+/fl3q8ivl6uqKS5cuyf6zdPfu3ScqS1XYaqS72EJEdcb8+fPRrl07pKeny6YUl475WLlyJV544QW0bt0ax44dw8aNG59bWdLT0zFw4ECMHDkSarUas2bNgpmZGaZPny7l+e677xAQEIBevXohNDQUDRs2xL1793DhwgWcPHlSNv35aUyYMAFr165F3759MWfOHDRp0gQ7d+7EN998gw8++ADNmzd/6nvOnDkT169fR/fu3dGoUSNkZmZiyZIlMDY2hq+vLwAgLCwMP//8M7p27YoJEyagVatWKC4uRkpKCvbs2YNJkyahY8eOT/Xc0gBjyZIlGDZsGIyNjeHh4QFXV1fMmTMHH3/8Mf766y/07t0btra2uHXrFo4dOwZLS8unXhxvyJAhWLVqFUaNGoXExER069YNxcXFOHr0KDw9PTF48GB06dIF7733Hv75z3/ixIkT6Nq1KywtLZGamopDhw7Bx8cHH3zwAXbs2IFvvvkGr7/+Opo2bQohBLZs2YLMzEz07NnzsWWxt7fHBx98gJSUFDRv3hy7du3C8uXL8cEHH6Bx48ayvD179oSXlxf27dsnLbXwJBwdHeHt7Y09e/agW7dusLCwAFASEN27dw/37t3DwoULZdfMmjULO3bsQLdu3TBz5kzY2dlhw4YN2LlzJxYsWAClUvnIZxoYGODTTz/Fu+++K/19ZGZmIiIi4om6zFxdXfHRRx/h008/RW5uLoYMGQKlUonz58/jzp07mD17NurVq4cZM2bgo48+wjvvvIMhQ4bg7t27mD17NszMzDBr1izpfiEhIfjuu+/w9ttvY+TIkbh79y4WLFgAGxubJ/oZVqb0PTt//nwEBATA0NAQrVq1gomJSbXvSbVEc+O5iaqn7Cyz8oKDgwUA2SwzIUqmmb/77rvCyclJWFpain79+okrV65UOcvs9u3bsuuHDRsmLC0tKzyv/Iy20tk169atE+PHjxf169cXpqam4tVXXxUnTpyocP3p06dFUFCQcHR0FMbGxkKlUonXXntNLFu27InqW5WrV6+K4OBgYW9vL4yNjYWHh4f44osvpJlrpZ50ltmOHTtEQECAaNiwoTAxMRGOjo6iT58+4uDBg7J8OTk54pNPPhEeHh7CxMREKJVK4ePjIyZMmCBNSRdCVJhCXrY8w4YNk6VNnz5dODs7CwMDAwFA7Nu3Tzq3bds20a1bN2FjYyNMTU1FkyZNxBtvvCFiYmKkPFX97kp/12Xl5uaKmTNnCnd3d2FiYiLs7e3Fa6+9JuLi4mT5Vq5cKTp27CgsLS2Fubm5eOGFF8Q777wj/Y4vXrwohgwZIl544QVhbm4ulEqleOmll8Tq1asf/YMWf7+n9u/fL9q3by9MTU1FgwYNxEcffVRhRmKpiIgIAUAcOXLksfcva8KECQKA+Pzzz2Xp7u7uAoA4c+ZMhWv++OMP0a9fP6FUKoWJiYlo3bp1hRmcpX8HP/30U6XP/eGHH6SfcfPmzcXKlSsrzOx6lLVr14oOHToIMzMzYWVlJdq2bVuhDD/88INo1aqV9D4cMGBApUsRrFmzRnh6egozMzPh5eUlNm/eXOUssy+++KLC9eU/Q/Ly8sS7774r6tevLxQKRaWzJEk7KYSogeVuiYhIY9q3bw+FQoHjx49ruihEOotdZkREOigrKwtnz57Fjh07EB8fj61bt2q6SEQ6jQEREZEOOnnyJLp16wZ7e3vMmjULr7/+uqaLRKTT2GVGREREeo/T7omIiEjvMSAiIiIivceAiIiIiPQeB1U/oeLiYty8eRPW1tZPvIUCERERaZYQAtnZ2Y/dq5AB0RO6efNmlXvnEBERkXa7du3aIzcGZkD0hEr3lbp27dozLetOREREtScrKwsuLi6P3R+SAdETKu0ms7GxYUBERESkYx433IWDqomIiEjvMSAiIiIivceAiIiIiPQexxAREZHeKSoqQkFBgaaLQTXA2NgYhoaGz3wfBkRERKQ3hBBIS0tDZmampotCNahevXpQqVTPtE4gAyIiItIbpcGQo6MjLCwsuNCujhNC4MGDB0hPTwcANGjQoNr3YkBERER6oaioSAqG7O3tNV0cqiHm5uYAgPT0dDg6Ola7+4yDqomISC+UjhmysLDQcEmoppX+Tp9lXBgDIiIi0ivsJqt7auJ3yoCIiIiI9B4DIiIiItJ7DIiIiIi0mJ+fH8LCwjRdjDqPARERERHpPQZEREREWio0NBSxsbFYsmQJFAoFFAoFjIyM8OWXX8rynT17FgYGBrh8+TKAkkHG3377LQICAmBubg43Nzf89NNPsmtu3LiBt956C7a2trC3t8eAAQNw5cqV2qqa1mFApGGFhYU4d+6c7FVYWKjpYhERkRZYsmQJOnXqhJEjRyI1NRWpqamYPXs2Vq1aJcu3cuVKvPrqq3jhhRektBkzZuAf//gHTp8+jbfffhtDhgzBhQsXAAAPHjxAt27dYGVlhQMHDuDQoUOwsrJC7969kZ+fX6t11BYMiDQsMTERo77egfCfEhD+UwJGfb0DiYmJmi4WERFpAaVSCRMTE1hYWEClUkGlUmH48OFITEzEsWPHAJSsvbN+/XoMHz5cdu2bb76Jd999F82bN8enn36K9u3bIzIyEgCwadMmGBgY4IcffoCPjw88PT2xatUqpKSkYP/+/bVdTa3Alaq1gJVjIyidm2q6GEREpAMaNGiAvn37YuXKlXjppZewY8cOPHz4EG+++aYsX6dOnSocJyQkAADi4+ORlJQEa2trWZ6HDx9K3W76hgERERGRjnn33XcREhKCRYsWYdWqVXjrrbeeaAXu0gUMi4uL0a5dO2zYsKFCnvr169d4eXUBAyIiIiItZmJigqKiIllanz59YGlpiW+//Ra//fYbDhw4UOG6I0eO4J133pEdt23bFgDw4osvYvPmzXB0dISNjc3zrYCO4BgiIiIiLebq6oqjR4/iypUruHPnDoqLi2FoaIjQ0FBMnz4dzZo1q9A9BgA//fQTVq5ciUuXLmHWrFk4duwYxo4dCwAYOnQoHBwcMGDAABw8eBDJycmIjY3Fhx9+iOvXr9d2FbUCAyIiIiItFh4eDkNDQ3h5eaF+/fpISUkBAIwYMQL5+fkVBlOXmj17NjZt2oRWrVphzZo12LBhA7y8vACUbIZ64MABNG7cGIMGDYKnpyeGDx+O3NxcvW0xYpcZERGRFmvevDkOHz5cIT01NRVGRkaybrGynJ2dsWfPnirvq1KpsGbNmhorp65jQERERKRD8vLycO3aNcyYMQNBQUFwcnLSdJHqBHaZERER6ZD//Oc/8PDwgFqtxoIFCzRdnDqDLUREREQ6JDQ0FKGhoY/MI4SoncLUIRptITpw4AD69esHZ2dnKBQKbNu2rUKeCxcuoH///lAqlbC2tsbLL78sDSgDSpoOx40bBwcHB1haWqJ///4VRshnZGQgJCQESqUSSqUSISEhyMzMfM61IyIiIl2h0YDo/v37aN26NZYuXVrp+cuXL+OVV15BixYtsH//fpw+fRozZsyAmZmZlCcsLAxbt27Fpk2bcOjQIeTk5CAwMFC2ZkNwcDASEhIQFRWFqKgoJCQkICQk5LnXj4iIiHSDRrvMAgICEBAQUOX5jz/+GH369JH1kTZt+vcWF2q1GitWrMC6devQo0cPAMD69evh4uKCmJgY9OrVCxcuXEBUVBSOHDmCjh07AgCWL1+OTp06ITExER4eHs+pdkRERKQrtHZQdXFxMXbu3InmzZujV69ecHR0RMeOHWXdavHx8SgoKIC/v7+U5uzsDG9vb8TFxQEADh8+DKVSKQVDAPDyyy9DqVRKeSqTl5eHrKws2YuIiIjqJq0NiNLT05GTk4N//etf6N27N/bs2YOBAwdi0KBBiI2NBQCkpaXBxMQEtra2smudnJyQlpYm5XF0dKxwf0dHRylPZebNmyeNOVIqlXBxcanB2hEREZE20dqAqLi4GAAwYMAATJgwAW3atMG0adMQGBiIZcuWPfJaIYS0gR0A2b+rylPe9OnToVarpde1a9eqWRMiIiLSdlo77d7BwQFGRkbSMuOlPD09cejQIQAlq2zm5+cjIyND1kqUnp6Ozp07S3lu3bpV4f63b99+5GJWpqamMDU1rYmqEBGRlktJScGdO3dq5VkODg5o3LhxrTyrMq6urggLC0NYWJjGyqCNtDYgMjExQYcOHZCYmChLv3TpEpo0aQIAaNeuHYyNjREdHY2goCAAJUuZnz17VhqI3alTJ6jVahw7dgwvvfQSAODo0aNQq9VS0ERERPorJSUFLVp4Ijf3Qa08z9zcAhcvXniqoMjPzw9t2rTB4sWLn/n5x48fh6Wl5TPfp67RaECUk5ODpKQk6Tg5ORkJCQmws7ND48aNMXnyZLz11lvo2rUrunXrhqioKPz666/Yv38/AECpVGLEiBGYNGkS7O3tYWdnh/DwcPj4+Eizzjw9PdG7d2+MHDkS3333HQDgvffeQ2BgIGeYERER7ty5g9zcB+g4fBZsGrg+12dlpV7B0ZWzcefOnRptJRJCoKioCEZGj/9ar1+/fo09ty7R6BiiEydOoG3btmjbti0AYOLEiWjbti1mzpwJABg4cCCWLVuGBQsWwMfHBz/88AN+/vlnvPLKK9I9Fi1ahNdffx1BQUHo0qULLCws8Ouvv8LQ0FDKs2HDBvj4+MDf3x/+/v5o1aoV1q1bV7uVJSIirWbTwBV2jT2e66s6AVdoaChiY2OxZMkSKBQKKBQKrF69GgqFArt370b79u1hamqKgwcP4vLlyxgwYACcnJxgZWWFDh06ICYmRnY/V1dXWUuTQqHADz/8gIEDB8LCwgLu7u7Yvn37M/40dY9GW4j8/Pweu7z48OHDMXz48CrPm5mZITIyEpGRkVXmsbOzw/r166tdTiIiIk1ZsmQJLl26BG9vb8yZMwcAcO7cOQDAlClT8OWXX6Jp06aoV68erl+/jj59+uCzzz6DmZkZ1qxZg379+iExMfGRLVKzZ8/GggUL8MUXXyAyMhJDhw7F1atXYWdnVyt11AZaO8uMiIiISoaHmJiYwMLCAiqVCiqVSuoFmTNnDnr27IkXXngB9vb2aN26Nd5//334+PjA3d0dn332GZo2bfrYFp/Q0FAMGTIEzZo1w9y5c3H//n0cO3asNqqnNRgQERER6aj27dvLju/fv48pU6bAy8sL9erVg5WVFS5evCjbA7QyrVq1kv5taWkJa2trpKenP5cyayutnWVGREREj1Z+ttjkyZOxe/dufPnll2jWrBnMzc3xxhtvID8//5H3MTY2lh0rFAppPUB9wYCIiIhIy5mYmMg2La/KwYMHERoaioEDBwIomc195cqV51y6uoFdZkRERFrO1dUVR48exZUrV3Dnzp0qW2+aNWuGLVu2ICEhAadPn0ZwcLDetfRUF1uIiIiIULJGkLY+Izw8HMOGDYOXlxdyc3OxatWqSvMtWrQIw4cPR+fOneHg4ICpU6dyc/InxICIiIj0moODA8zNLXB05exaeZ65uQUcHBye6prmzZvj8OHDsrTQ0NAK+VxdXbF3715Z2pgxY2TH5bvQKlv+JjMz86nKVxcwICIiIr3WuHFjXLx4QW/2MqPKMSAiIiK917hxYwYpeo6DqomIiEjvMSAiIiIivceAiIiIiPQeAyIiIiLSewyIiIiISO8xICIiIiK9x4CIiIiI9B7XISIiIr2XkpJSpxdmdHV1RVhYGMLCwgCU7Ga/detWvP7665Xmv3LlCtzc3HDq1Cm0adOm2s+tqfvUBgZERESk11JSUuDZwgMPch/WyvMszM1w4WKiRheCTE1Nha2tbY3eMzQ0FJmZmdi2bZuU5uLigtTU1KfeqkQTGBAREZFeu3PnDh7kPsT699rAs4HVc33WhdQcvP19Au7cuaPRgEilUtXKcwwNDWvtWc+KY4iIiIgAeDawwouuyuf6qk7A9d1336Fhw4YoLi6Wpffv3x/Dhg3D5cuXMWDAADg5OcHKygodOnRATEzMI++pUChkLTnHjh1D27ZtYWZmhvbt2+PUqVOy/EVFRRgxYgTc3Nxgbm4ODw8PLFmyRDofERGBNWvW4JdffoFCoYBCocD+/ftx5coVKBQKJCQkSHljY2Px0ksvwdTUFA0aNMC0adNQWFgonffz88P48eMxZcoU2NnZQaVSISIi4ql/bk+LAREREZEWe/PNN3Hnzh3s27dPSsvIyMDu3bsxdOhQ5OTkoE+fPoiJicGpU6fQq1cv9OvXDykpKU90//v37yMwMBAeHh6Ij49HREQEwsPDZXmKi4vRqFEj/Pjjjzh//jxmzpyJjz76CD/++CMAIDw8HEFBQejduzdSU1ORmpqKzp07V3jWjRs30KdPH3To0AGnT5/Gt99+ixUrVuCzzz6T5VuzZg0sLS1x9OhRLFiwAHPmzEF0dPTT/uieCrvMiIiItJidnR169+6NjRs3onv37gCAn376CXZ2dujevTsMDQ3RunVrKf9nn32GrVu3Yvv27Rg7duxj779hwwYUFRVh5cqVsLCwQMuWLXH9+nV88MEHUh5jY2PMnj1bOnZzc0NcXBx+/PFHBAUFwcrKCubm5sjLy3tkF9k333wDFxcXLF26FAqFAi1atMDNmzcxdepUzJw5EwYGJe00rVq1wqxZswAA7u7uWLp0KX7//Xf07Nnz6X54T4EtRERERFpu6NCh+Pnnn5GXlwegJIgZPHgwDA0Ncf/+fUyZMgVeXl6oV68erKyscPHixSduIbpw4QJat24NCwsLKa1Tp04V8i1btgzt27dH/fr1YWVlheXLlz/xM8o+q1OnTlAoFFJaly5dkJOTg+vXr0tprVq1kl3XoEEDpKenP9WznhYDIiIiIi3Xr18/FBcXY+fOnbh27RoOHjyIt99+GwAwefJk/Pzzz/j8889x8OBBJCQkwMfHB/n5+U90byHEY/P8+OOPmDBhAoYPH449e/YgISEB//znP5/4GWWfVTYYKvv8sunGxsayPAqFosIYqprGLjMiIiItZ25ujkGDBmHDhg1ISkpC8+bN0a5dOwDAwYMHERoaioEDBwIAcnJycOXKlSe+t5eXF9atW4fc3FyYm5sDAI4cOSLLc/DgQXTu3BmjR4+W0i5fvizLY2JigqKiosc+6+eff5YFRnFxcbC2tkbDhg2fuMzPA1uIiIiIdMDQoUOxc+dOrFy5UmodAoBmzZphy5YtSEhIwOnTpxEcHPxUrSnBwcEwMDDAiBEjcP78eezatQtffvmlLE+zZs1w4sQJ7N69G5cuXcKMGTNw/PhxWR5XV1ecOXMGiYmJuHPnDgoKCio8a/To0bh27RrGjRuHixcv4pdffsGsWbMwceJEafyQprCFiIiICCVrBGnzM1577TXY2dkhMTERwcHBUvqiRYswfPhwdO7cGQ4ODpg6dSqysrKe+L5WVlb49ddfMWrUKLRt2xZeXl6YP38+/vGPf0h5Ro0ahYSEBLz11ltQKBQYMmQIRo8ejd9++03KM3LkSOzfvx/t27dHTk4O9u3bB1dXV9mzGjZsiF27dmHy5Mlo3bo17OzsMGLECHzyySfV/rnUFIV4ks5DQlZWFpRKJdRqNWxsbGrsvufOnUP4TwlQOjcFAKhv/oUv32yDli1b1tgziIgIePjwIZKTk+Hm5gYzMzMpXR9Xqq5rqvrdAk/+/c0WIiIi0muNGzfGhYuJdXovM3o8BkRERKT3GjduzCBFz3FQNREREek9jQZEBw4cQL9+/eDs7FxhX5Xy3n//fSgUCixevFiWnpeXh3HjxsHBwQGWlpbo37+/bHEnoGSJ85CQECiVSiiVSoSEhCAzM7PmK0REREQ6SaMB0f3799G6dWssXbr0kfm2bduGo0ePwtnZucK5sLAwbN26FZs2bcKhQ4eQk5ODwMBA2VoIwcHBSEhIQFRUFKKiopCQkICQkJAarw8RERHpJo2OIQoICEBAQMAj89y4cQNjx47F7t270bdvX9k5tVqNFStWYN26dejRowcAYP369XBxcUFMTAx69eqFCxcuICoqCkeOHEHHjh0BAMuXL0enTp2QmJgIDw+P51M5IiIi0hlaPYaouLgYISEhmDx5cqXT0OPj41FQUAB/f38pzdnZGd7e3oiLiwMAHD58GEqlUgqGAODll1+GUqmU8lQmLy8PWVlZshcRERHVTVodEM2fPx9GRkYYP358pefT0tJgYmICW1tbWbqTkxPS0tKkPI6OjhWudXR0lPJUZt68edKYI6VSCRcXl2eoCREREWkzrQ2I4uPjsWTJEqxevbrCRnCPU37zuMqur2yDubKmT58OtVotva5du/ZUZSAiIiLdobXrEB08eBDp6emydSGKioowadIkLF68GFeuXIFKpUJ+fj4yMjJkrUTp6eno3LkzAEClUuHWrVsV7n/79m04OTlV+XxTU1OYmprWYI2IiEhbpaSkcGFGPae1AVFISIg0ULpUr169EBISgn/+858AgHbt2sHY2BjR0dEICgoCAKSmpuLs2bNYsGABAKBTp05Qq9U4duwYXnrpJQDA0aNHoVarpaCJiIj0V0pKClp4tkDug9xaeZ65hTkuXrj4VEGRn58f2rRpU2HpmeoKDQ1FZmbmI5e70TcaDYhycnKQlJQkHScnJyMhIQF2dnZo3Lgx7O3tZfmNjY2hUqmkmWFKpRIjRozApEmTYG9vDzs7O4SHh8PHx0cKpjw9PdG7d2+MHDkS3333HQDgvffeQ2BgIGeYERER7ty5g9wHueg6pSuULsrn+iz1NTUOLDiAO3fusJVIy2g0IDpx4gS6desmHU+cOBEAMGzYMKxevfqJ7rFo0SIYGRkhKCgIubm56N69O1avXg1DQ0Mpz4YNGzB+/HhpNlr//v0fu/YRERHpF6WLEg7uDpouRgWhoaGIjY1FbGwslixZAqCkAeHBgwcIDw/HgQMHYGlpCX9/fyxatAgODiV1+O9//4vZs2cjKSkJFhYWaNu2LX755Rd88cUXWLNmDYC/x9ju27cPfn5+GqmfttBoQOTn5wchxBPnv3LlSoU0MzMzREZGIjIyssrr7OzssH79+uoUkYiISKOWLFmCS5cuwdvbG3PmzAFQMqbW19cXI0eOxMKFC5Gbm4upU6ciKCgIe/fuRWpqKoYMGYIFCxZg4MCByM7OxsGDByGEQHh4OC5cuICsrCysWrUKQMn3pL7T2jFEREREVDI8xMTEBBYWFlCpVACAmTNn4sUXX8TcuXOlfCtXroSLiwsuXbqEnJwcFBYWYtCgQWjSpAkAwMfHR8prbm6OvLw86X7EgIiIiEjnxMfHY9++fbCysqpw7vLly/D390f37t3h4+ODXr16wd/fH2+88UaFdfvob1q7DhERERFVrri4GP369UNCQoLs9eeff6Jr164wNDREdHQ0fvvtN3h5eSEyMhIeHh5ITk7WdNG1FgMiIiIiLWdiYiLbtPzFF1/EuXPn4OrqimbNmslelpaWAEoGTHfp0gWzZ8/GqVOnYGJigq1bt1Z6P2KXGREREYCSKfHa+gxXV1ccPXoUV65cgZWVFcaMGYPly5djyJAhmDx5MhwcHJCUlIRNmzZh+fLlOHHiBH7//Xf4+/vD0dERR48exe3bt+Hp6Sndb/fu3UhMTIS9vT2USiWMjY1rsqo6hwERERHpNQcHB5hbmOPAggO18jxzC3NpavyTCg8Px7Bhw+Dl5YXc3FwkJyfjf//7H6ZOnYpevXohLy8PTZo0Qe/evWFgYAAbGxscOHAAixcvRlZWFpo0aYKvvvoKAQEBAICRI0di//79aN++PXJycjjtHgyIiIhIzzVu3BgXL1zU6q07mjdvjsOHD1dI37JlS6X5PT09ERUVVeX96tevjz179jxVGeo6BkRERKT3GjduzJWj9RwHVRMREZHeY0BEREREeo8BEREREek9jiGqAwoLC5GYmCgde3h4wMiIv1oioso8zR6apBtq4nfKb806IDExEaO+3gErx0bISb+OZWOAli1barpYRERapXSdnQcPHsDc3FzDpaGa9ODBAwB4prWUGBDVEVaOjaB0bqrpYhARaS1DQ0PUq1cP6enpAAALCwsoFAoNl4qehRACDx48QHp6OurVqwdDQ8Nq34sBERER6Y3S3d1LgyKqG+rVqyf9bquLAREREekNhUKBBg0awNHREQUFBZouDtUAY2PjZ2oZKsWAiIiI9I6hoWGNfIlS3cFp90RERKT3GBARERGR3mNARERERHqPARERERHpPQZEREREpPcYEBEREZHeY0BEREREeo8BEREREek9BkRERESk9xgQERERkd5jQERERER6jwERERER6T0GRERERKT3GBARERGR3tNoQHTgwAH069cPzs7OUCgU2LZtm3SuoKAAU6dOhY+PDywtLeHs7Ix33nkHN2/elN0jLy8P48aNg4ODAywtLdG/f39cv35dlicjIwMhISFQKpVQKpUICQlBZmZmLdSQiIiIdIFGA6L79++jdevWWLp0aYVzDx48wMmTJzFjxgycPHkSW7ZswaVLl9C/f39ZvrCwMGzduhWbNm3CoUOHkJOTg8DAQBQVFUl5goODkZCQgKioKERFRSEhIQEhISHPvX5ERESkG4w0+fCAgAAEBARUek6pVCI6OlqWFhkZiZdeegkpKSlo3Lgx1Go1VqxYgXXr1qFHjx4AgPXr18PFxQUxMTHo1asXLly4gKioKBw5cgQdO3YEACxfvhydOnVCYmIiPDw8nm8liYiISOvp1BgitVoNhUKBevXqAQDi4+NRUFAAf39/KY+zszO8vb0RFxcHADh8+DCUSqUUDAHAyy+/DKVSKeUhIiIi/abRFqKn8fDhQ0ybNg3BwcGwsbEBAKSlpcHExAS2trayvE5OTkhLS5PyODo6Vrifo6OjlKcyeXl5yMvLk46zsrJqohpERESkhXSihaigoACDBw9GcXExvvnmm8fmF0JAoVBIx2X/XVWe8ubNmycNwlYqlXBxcale4YmIiEjraX1AVFBQgKCgICQnJyM6OlpqHQIAlUqF/Px8ZGRkyK5JT0+Hk5OTlOfWrVsV7nv79m0pT2WmT58OtVotva5du1ZDNSIiIiJto9UBUWkw9OeffyImJgb29vay8+3atYOxsbFs8HVqairOnj2Lzp07AwA6deoEtVqNY8eOSXmOHj0KtVot5amMqakpbGxsZC8iIiKqmzQ6hignJwdJSUnScXJyMhISEmBnZwdnZ2e88cYbOHnyJHbs2IGioiJpzI+dnR1MTEygVCoxYsQITJo0Cfb29rCzs0N4eDh8fHykWWeenp7o3bs3Ro4cie+++w4A8N577yEwMJAzzIiIiAiAhgOiEydOoFu3btLxxIkTAQDDhg1DREQEtm/fDgBo06aN7Lp9+/bBz88PALBo0SIYGRkhKCgIubm56N69O1avXg1DQ0Mp/4YNGzB+/HhpNlr//v0rXfuIiIiI9JNGAyI/Pz8IIao8/6hzpczMzBAZGYnIyMgq89jZ2WH9+vXVKiMRERHVfVo9hoiIiIioNjAgIiIiIr3HgIiIiIj0HgMiIiIi0nsMiIiIiEjvMSAiIiIivceAiIiIiPQeAyIiIiLSewyIiIiISO9pdKVq0k6FhYVITEyUjj08PGBkxLcKERHVXfyWowoSExMx6usdsHJshJz061g2BmjZsqWmi0VERPTcMCCiSlk5NoLSuammi0FERFQrOIaIiIiI9B4DIiIiItJ7DIiIiIhI7zEgIiIiIr3HgIiIiIj0HgMiIiIi0nsMiIiIiEjvMSAiIiIivceAiIiIiPQeAyIiIiLSewyIiIiISO8xICIiIiK9x4CIiIiI9B4DIiIiItJ7DIiIiIhI7zEgIiIiIr3HgIiIiIj0HgMiIiIi0nsMiIiIiEjvMSAiIiIivafRgOjAgQPo168fnJ2doVAosG3bNtl5IQQiIiLg7OwMc3Nz+Pn54dy5c7I8eXl5GDduHBwcHGBpaYn+/fvj+vXrsjwZGRkICQmBUqmEUqlESEgIMjMzn3PtiIiISFdoNCC6f/8+WrdujaVLl1Z6fsGCBVi4cCGWLl2K48ePQ6VSoWfPnsjOzpbyhIWFYevWrdi0aRMOHTqEnJwcBAYGoqioSMoTHByMhIQEREVFISoqCgkJCQgJCXnu9SMiIiLdYKTJhwcEBCAgIKDSc0IILF68GB9//DEGDRoEAFizZg2cnJywceNGvP/++1Cr1VixYgXWrVuHHj16AADWr18PFxcXxMTEoFevXrhw4QKioqJw5MgRdOzYEQCwfPlydOrUCYmJifDw8KidyhIREZHW0toxRMnJyUhLS4O/v7+UZmpqCl9fX8TFxQEA4uPjUVBQIMvj7OwMb29vKc/hw4ehVCqlYAgAXn75ZSiVSilPZfLy8pCVlSV7ERERUd2ktQFRWloaAMDJyUmW7uTkJJ1LS0uDiYkJbG1tH5nH0dGxwv0dHR2lPJWZN2+eNOZIqVTCxcXlmepDRERE2ktrA6JSCoVCdiyEqJBWXvk8leV/3H2mT58OtVotva5du/aUJSciIiJdobUBkUqlAoAKrTjp6elSq5FKpUJ+fj4yMjIemefWrVsV7n/79u0KrU9lmZqawsbGRvYiIiKiuklrAyI3NzeoVCpER0dLafn5+YiNjUXnzp0BAO3atYOxsbEsT2pqKs6ePSvl6dSpE9RqNY4dOyblOXr0KNRqtZSHiIiI9JtGZ5nl5OQgKSlJOk5OTkZCQgLs7OzQuHFjhIWFYe7cuXB3d4e7uzvmzp0LCwsLBAcHAwCUSiVGjBiBSZMmwd7eHnZ2dggPD4ePj48068zT0xO9e/fGyJEj8d133wEA3nvvPQQGBnKGGREREQHQcEB04sQJdOvWTTqeOHEiAGDYsGFYvXo1pkyZgtzcXIwePRoZGRno2LEj9uzZA2tra+maRYsWwcjICEFBQcjNzUX37t2xevVqGBoaSnk2bNiA8ePHS7PR+vfvX+XaR0RERKR/NBoQ+fn5QQhR5XmFQoGIiAhERERUmcfMzAyRkZGIjIysMo+dnR3Wr1//LEUlIiKiOkxrxxARERER1ZZqBURNmzbF3bt3K6RnZmaiadOmz1woIiIiotpUrYDoypUrsr3CSuXl5eHGjRvPXCgiIiKi2vRUY4i2b98u/Xv37t1QKpXScVFREX7//Xe4urrWWOGIiIiIasNTBUSvv/46gJLBzsOGDZOdMzY2hqurK7766qsaKxwRERFRbXiqgKi4uBhAyaKJx48fh4ODw3MpFBEREVFtqta0++Tk5JouBxEREZHGVHsdot9//x2///470tPTpZajUitXrnzmghERERHVlmoFRLNnz8acOXPQvn17NGjQ4LG7zxMRERFps2oFRMuWLcPq1asREhJS0+UhIiIiqnXVWocoPz+fO8UTERFRnVGtgOjdd9/Fxo0ba7osRERERBpRrS6zhw8f4vvvv0dMTAxatWoFY2Nj2fmFCxfWSOGIiIiIakO1AqIzZ86gTZs2AICzZ8/KznGANREREemaagVE+/btq+lyEBEREWlMtcYQEREREdUl1Woh6tat2yO7xvbu3VvtAhERERHVtmoFRKXjh0oVFBQgISEBZ8+erbDpKxEREZG2q1ZAtGjRokrTIyIikJOT80wFIiIiIqptNTqG6O233+Y+ZkRERKRzajQgOnz4MMzMzGrylkRERETPXbW6zAYNGiQ7FkIgNTUVJ06cwIwZM2qkYERERES1pVoBkVKplB0bGBjAw8MDc+bMgb+/f40UjIiIiKi2VCsgWrVqVU2Xg4iIiEhjqhUQlYqPj8eFCxegUCjg5eWFtm3b1lS5iIiIiGpNtQKi9PR0DB48GPv370e9evUghIBarUa3bt2wadMm1K9fv6bLSURERPTcVGuW2bhx45CVlYVz587h3r17yMjIwNmzZ5GVlYXx48fXdBmJiIiInqtqtRBFRUUhJiYGnp6eUpqXlxe+/vprDqomIiIinVOtFqLi4mIYGxtXSDc2NkZxcfEzF4qIiIioNlUrIHrttdfw4Ycf4ubNm1LajRs3MGHCBHTv3r3GCkdERERUG6oVEC1duhTZ2dlwdXXFCy+8gGbNmsHNzQ3Z2dmIjIys6TISERERPVfVCohcXFxw8uRJ7Ny5E2FhYRg/fjx27dqF+Ph4NGrUqMYKV1hYiE8++QRubm4wNzdH06ZNMWfOHFm3nBACERERcHZ2hrm5Ofz8/HDu3DnZffLy8jBu3Dg4ODjA0tIS/fv3x/Xr12usnERERKTbniog2rt3L7y8vJCVlQUA6NmzJ8aNG4fx48ejQ4cOaNmyJQ4ePFhjhZs/fz6WLVuGpUuX4sKFC1iwYAG++OILWSvUggULsHDhQixduhTHjx+HSqVCz549kZ2dLeUJCwvD1q1bsWnTJhw6dAg5OTkIDAxEUVFRjZWViIiIdNdTBUSLFy/GyJEjYWNjU+GcUqnE+++/j4ULF9ZY4Q4fPowBAwagb9++cHV1xRtvvAF/f3+cOHECQEnr0OLFi/Hxxx9j0KBB8Pb2xpo1a/DgwQNs3LgRAKBWq7FixQp89dVX6NGjB9q2bYv169fjjz/+QExMTI2VlYiIiHTXUwVEp0+fRu/evas87+/vj/j4+GcuVKlXXnkFv//+Oy5duiQ9/9ChQ+jTpw8AIDk5GWlpabKp/qampvD19UVcXByAktW0CwoKZHmcnZ3h7e0t5alMXl4esrKyZC8iIiKqm55qHaJbt25VOt1eupmREW7fvv3MhSo1depUqNVqtGjRAoaGhigqKsLnn3+OIUOGAADS0tIAAE5OTrLrnJyccPXqVSmPiYkJbG1tK+Qpvb4y8+bNw+zZs2usLkRERKS9nqqFqGHDhvjjjz+qPH/mzBk0aNDgmQtVavPmzVi/fj02btyIkydPYs2aNfjyyy+xZs0aWT6FQiE7FkJUSCvvcXmmT58OtVotva5du1b9ihAREZFWe6qAqE+fPpg5cyYePnxY4Vxubi5mzZqFwMDAGivc5MmTMW3aNAwePBg+Pj4ICQnBhAkTMG/ePACASqUCgAotPenp6VKrkUqlQn5+PjIyMqrMUxlTU1PY2NjIXkRERFQ3PVVA9Mknn+DevXto3rw5FixYgF9++QXbt2/H/Pnz4eHhgXv37uHjjz+uscI9ePAABgbyIhoaGkrT7t3c3KBSqRAdHS2dz8/PR2xsLDp37gwAaNeuHYyNjWV5UlNTcfbsWSmPNikuLkJSUhLOnTuHc+fOobCwUNNFIiIiqvOeagyRk5MT4uLi8MEHH2D69OkQQgAo6bLq1asXvvnmm0e2ujytfv364fPPP0fjxo3RsmVLnDp1CgsXLsTw4cOl54aFhWHu3Llwd3eHu7s75s6dCwsLCwQHBwMomf02YsQITJo0Cfb29rCzs0N4eDh8fHzQo0ePGitrTbl/JxWfbb8CB5cc5KRfx7IxQMuWLTVdLCIiojrtqTd3bdKkCXbt2oWMjAwkJSVBCAF3d/cKg5ZrQmRkJGbMmIHRo0cjPT0dzs7OeP/99zFz5kwpz5QpU5Cbm4vRo0cjIyMDHTt2xJ49e2BtbS3lWbRoEYyMjBAUFITc3Fx0794dq1evhqGhYY2XuSZYOjSE0rmppotBRESkN6q12z0A2NraokOHDjVZlgqsra2xePFiLF68uMo8CoUCERERiIiIqDKPmZkZIiMjua0IERERVapaW3cQERER1SUMiIiIiEjvMSAiIiIivceAiIiIiPQeAyIiIiLSewyIiIiISO9Ve9o90ZMqLCxEYmKidOzh4QEjI771iIhIe/BbiZ67xMREjPp6B6wcG3H1bSIi0koMiKhWWDk24urbRESktTiGiIiIiPQeAyIiIiLSewyIiIiISO8xICIiIiK9x4CIiIiI9B4DIiIiItJ7DIiIiIhI7zEgIiIiIr3HgIiIiIj0HgMiIiIi0nvcukNHld0wNSkpCUIIDZeIiIhIdzEg0lFlN0y9dTEeNk24WSoREVF1sctMh5VumGph56TpohAREek0BkRERESk9xgQERERkd7jGCI9UnYgNgB4eHjAyIhvASIiIn4b6pGyA7Fz0q9j2RigZUsOxiYiImJAVMcUFxchKSlJlla2Jah0IDYRERH9jQFRHXP/Tio+234FDi45AIDstKuYHNASzZo143pFREREVWBAVAdZOjSUWoGy06/js+2n4eCSw/WKiIiIqsBZZnqgNEDiekVERESVYwuRjig/Q4zdX0RERDVH61uIbty4gbfffhv29vawsLBAmzZtEB8fL50XQiAiIgLOzs4wNzeHn58fzp07J7tHXl4exo0bBwcHB1haWqJ///64fv16bVflmZTOEAv/KQHhPyVg9n9i8fBhXrXvVzr4+ty5c9KrsLCwBktMRESkO7Q6IMrIyECXLl1gbGyM3377DefPn8dXX32FevXqSXkWLFiAhQsXYunSpTh+/DhUKhV69uyJ7OxsKU9YWBi2bt2KTZs24dChQ8jJyUFgYCCKioo0UKvqK50hVhPdXyWDr09LAdaor3fIWqCIiIj0iVZ3mc2fPx8uLi5YtWqVlObq6ir9WwiBxYsX4+OPP8agQYMAAGvWrIGTkxM2btyI999/H2q1GitWrMC6devQo0cPAMD69evh4uKCmJgY9OrVq1brpE3KDr4mIiLSZ1rdQrR9+3a0b98eb775JhwdHdG2bVssX75cOp+cnIy0tDT4+/tLaaampvD19UVcXBwAID4+HgUFBbI8zs7O8Pb2lvJUJi8vD1lZWbIXERER1U1aHRD99ddf+Pbbb+Hu7o7du3dj1KhRGD9+PNauXQsASEtLAwA4Ocm7j5ycnKRzaWlpMDExga2tbZV5KjNv3jwolUrp5eLiUpNVIyIiIi2i1QFRcXExXnzxRcydOxdt27bF+++/j5EjR+Lbb7+V5VMoFLJjIUSFtPIel2f69OlQq9XS69q1a9WvCBEREWk1rQ6IGjRoAC8vL1map6cnUlJSAAAqlQoAKrT0pKenS61GKpUK+fn5yMjIqDJPZUxNTWFjYyN7ERERUd2k1QFRly5dKsx8unTpEpo0aQIAcHNzg0qlQnR0tHQ+Pz8fsbGx6Ny5MwCgXbt2MDY2luVJTU3F2bNnpTxERESk37R6ltmECRPQuXNnzJ07F0FBQTh27Bi+//57fP/99wBKusrCwsIwd+5cuLu7w93dHXPnzoWFhQWCg4MBAEqlEiNGjMCkSZNgb28POzs7hIeHw8fHR5p1RkRERPpNqwOiDh06YOvWrZg+fTrmzJkDNzc3LF68GEOHDpXyTJkyBbm5uRg9ejQyMjLQsWNH7NmzB9bW1lKeRYsWwcjICEFBQcjNzUX37t2xevVqGBoaaqJaREREpGW0OiACgMDAQAQGBlZ5XqFQICIiAhEREVXmMTMzQ2RkJCIjI59DCeuG0pWrAW4LQkRE+kfrAyJ9VptBSsnK1Vfg4JKDWxfjYdOk5XN7FhERkbZhQKTFajtIKV25Ojtdt/Z5IyIielZaPcuM/g5SnnXvMiIiIqoaAyIiIiLSewyIiIiISO8xICIiIiK9x4CIiIiI9B4DIiIiItJ7nHavYYWFhbhz7TJycx8CADLTrsHeyl7DpSIiItIvDIi0wDsPN6NxrjkA4GheBv6HNpotEBERkZ5hQKRhRkZGeM3LAS0b/r33WpxCocESERER6R8GRPRIZbcPKeXh4QEjI751iIio7uC3Gj1S2e1DACAn/TqWjQFatuReZ0REVHcwIKLHKt0+hIiIqK7itHsiIiLSewyIiIiISO8xICIiIiK9x4CIiIiI9B4DIiIiItJ7DIiIiIhI73HaPT2V8gs1cpFGIiKqC/hNRk+l7EKNXKSRiIjqCgZE9NS4UCMREdU1DIio2th9RkREdQW/veoCIZBx4y8AQGbaNdhb2dfKY9l9RkREdQUDojrA0MoOfVOWo6OdLY7mZeB/aFNrz2b3GRER1QUMiOoAhYEhOr5giwEvqgAAcQqFhktERESkWxgQUY0oP54I4JgiIiLSHfy2ohpRdjwRAGSnXcXkgJZo1qwZkpKSIISosWcVFhYiMTFROmbgRUREz4rfIlRjyo4nyk6/js+2n4aDSw5uXYyHTZOaG2ydmJiIUV/vgJVjIw7mJiKiGsGAiJ6b0gApO/16jd/byrERB3MTEVGN0am9zObNmweFQoGwsDApTQiBiIgIODs7w9zcHH5+fjh37pzsury8PIwbNw4ODg6wtLRE//79cf16zX9JExERkW7SmYDo+PHj+P7779GqVStZ+oIFC7Bw4UIsXboUx48fh0qlQs+ePZGdnS3lCQsLw9atW7Fp0yYcOnQIOTk5CAwMRFFRUW1Xg4iIiLSQTgREOTk5GDp0KJYvXw5bW1spXQiBxYsX4+OPP8agQYPg7e2NNWvW4MGDB9i4cSMAQK1WY8WKFfjqq6/Qo0cPtG3bFuvXr8cff/yBmJgYTVWJiIiItIhOBERjxoxB37590aNHD1l6cnIy0tLS4O/vL6WZmprC19cXcXFxAID4+HgUFBTI8jg7O8Pb21vKU5m8vDxkZWXJXkRERFQ3af2g6k2bNuHkyZM4fvx4hXNpaWkAACcnJ1m6k5MTrl69KuUxMTGRtSyV5im9vjLz5s3D7Nmzn7X4REREpAO0uoXo2rVr+PDDD7F+/XqYmZlVmU9RbmVmIUSFtPIel2f69OlQq9XS69q1a09XeCIiItIZWh0QxcfHIz09He3atYORkRGMjIwQGxuLf//73zAyMpJahsq39KSnp0vnVCoV8vPzkZGRUWWeypiamsLGxkb2IiIiorpJqwOi7t27448//kBCQoL0at++PYYOHYqEhAQ0bdoUKpUK0dHR0jX5+fmIjY1F586dAQDt2rWDsbGxLE9qairOnj0r5SEiIiL9ptVjiKytreHt7S1Ls7S0hL29vZQeFhaGuXPnwt3dHe7u7pg7dy4sLCwQHBwMAFAqlRgxYgQmTZoEe3t72NnZITw8HD4+PhUGaRMREZF+0uqA6ElMmTIFubm5GD16NDIyMtCxY0fs2bMH1tbWUp5FixbByMgIQUFByM3NRffu3bF69WoYGhpqsORERESkLXQuINq/f7/sWKFQICIiAhEREVVeY2ZmhsjISERGRj7fwtFjFRcXISkpSZbGzVmJiEjT+C1Eter+nVR8tv0KHFxyAICbsxIRkVZgQES1rnTTVyIiIm2h1bPMiIiIiGoDAyIiIiLSewyIiIiISO8xICIiIiK9x4CIiIiI9B4DIiIiItJ7DIiIiIhI7zEgIiIiIr3HhRlJo8pv5cFtPIiISBP4zUMaVXYrD27jQUREmsKAiDSOW3kQEZGmcQwRERER6T0GRERERKT3GBARERGR3uMYIg0rLCzExZR7eJCbCwBIvJmJe2Z/AQAy067B3spek8UjIiLSCwyItMBCQwtYGVoBAG4ZPMCwh5vRPLcejuZl4H9oo9nCERER6QEGRBpmZGQE59bOsHW1ldL8irLRoZkjACBOodBU0YiIiPQGxxARERGR3mNARERERHqPXWaklQoLC5GYmChL47YeRET0vPDbhbRG2X3NkpKS8EXURVg7NQIAbutBRETPFQMi0hpl9zW7dTEeNk1acksPIiKqFRxDRFqldF8zCzsnTReFiIj0CAMiIiIi0nvsMqOKhEDGDa6WTURE+oMBUR0ghMCfqWocTzLAn6lqCDfxTPcztLJD35Tl6Ghny9WyiYhILzAgqiPWGJrhN0NrpBvmwu4Z76UwMETHF2wx4EUVAK6WTUREdR8DIl1VtlvrZjIcmjmgSecmAICihwxgiIiIngYDIh1Vtlvrzzw1dioaabpIREREOkurZ5nNmzcPHTp0gLW1NRwdHfH6669XWL1YCIGIiAg4OzvD3Nwcfn5+OHfunCxPXl4exo0bBwcHB1haWqJ///64fv16bValxpXt1vL1sIWBgVb/KomIiLSaVn+LxsbGYsyYMThy5Aiio6NRWFgIf39/3L9/X8qzYMECLFy4EEuXLsXx48ehUqnQs2dPZGdnS3nCwsKwdetWbNq0CYcOHUJOTg4CAwNRVFSkiWoRERGRltHqLrOoqCjZ8apVq+Do6Ij4+Hh07doVQggsXrwYH3/8MQYNGgQAWLNmDZycnLBx40a8//77UKvVWLFiBdatW4cePXoAANavXw8XFxfExMSgV69etV4venrlt/UQ4tlm0hEREZWl1S1E5anVagCAnV3JPKrk5GSkpaXB399fymNqagpfX1/ExcUBAOLj41FQUCDL4+zsDG9vbymPTvj/QdRpl88j7fJ5ZNz4C8V6FBSUbOtxGuE/JWD2f2Lx8GGepotERER1iFa3EJUlhMDEiRPxyiuvwNvbGwCQlpYGAHBykm/z4OTkhKtXr0p5TExMYGtrWyFP6fWVycvLQ17e31+6WVlZNVKP6io7iBoAjmZkQNX4WSfY65bSbT2y03V7/BcREWkfnQmIxo4dizNnzuDQoUMVzinKrZMjhKiQVt7j8sybNw+zZ8+uXmFrSGFxman1qVfRoWk9aW0gADAy4PR6IiKimqATAdG4ceOwfft2HDhwAI0a/T29XKUqCQ7S0tLQoEEDKT09PV1qNVKpVMjPz0dGRoaslSg9PR2dO3eu8pnTp0/HxIkTpeOsrCy4uLjUWJ2ehAJASN6PcM9V6mWLUG0pLCysMHvRw8MDRkY68edBREQ1QKvHEAkhMHbsWGzZsgV79+6Fm5ub7LybmxtUKhWio6OltPz8fMTGxkrBTrt27WBsbCzLk5qairNnzz4yIDI1NYWNjY3sVdsMDRTw9SiZWt/xBVu2CD0niYmJGPX1DoT/lIDwnxIw6usdFQIkIiKq27T6v8BjxozBxo0b8csvv8Da2loa86NUKmFubg6FQoGwsDDMnTsX7u7ucHd3x9y5c2FhYYHg4GAp74gRIzBp0iTY29vDzs4O4eHh8PHxkWad1SXFRcVSNxvAzVmflJVjIyidm2q6GEREpCFaHRB9++23AAA/Pz9Z+qpVqxAaGgoAmDJlCnJzczF69GhkZGSgY8eO2LNnD6ytraX8ixYtgpGREYKCgpCbm4vu3btj9erVMDQ0rK2q1B7F391sALg5KxER0RPQ6oDoSdaaUSgUiIiIQERERJV5zMzMEBkZicjIyBosnXYyMDCAr4ctOjRzlNL0aXPW8uOBOBaIiIieBL8pSKeVXbARKFm08Yuoi7B2aoSc9OtYNgZo2bKlBktIRES6gAFRHVd26n5G6jUY/v96SnVlbFHJgo1X4OCSAwC4dTEeNk1acjwQERE9FQZEWqa4qBgXrmcCAP5MVcPZut4z3a/s1P3DD+/BztAAHrn16tTYotIFGwFw0UYiIqoWBkTaRgEsNrSCraE10g1z4fuMtyudul86psjZxkD6tz6NLSIiInoUBkRaxsDAAI4+jnD2dC5JKMqunQeLMl1rN/5Csa3+7JNGRETEgEhPlR1bBABZajX6qH9EJzvbOrsqNmegERFRVfhtoKfKji0CgKPqDPRrbYdOzUu2PKkLq2JzBhoRET0pBkR6qvzYIqBuBEFlcQYaERE9KQZEVKdVNQOtbOtRUlLSEy0CSkREdRcDIqqgWAjZ1H/h9ozBQpkB29qy/lHZ1qPSliMiItJfDIh0VNmg5cL1TIgmVjV2b5W1AWbeNoXj/0/9f9bh1YZWduibshwd7Wy1av2j0tajml67qPzgbYADuImItB0/oXWULGh5mIsGNjX3qzQyUMDRwxFNOjcBABQ9fLaxRQoDQ3R8wRYDXlQBqPvrHyUmJmLU1ztg5dgIADiAm4hIBzAg0lHlgxYDQwMNl4jKsnJsxMHbREQ6hAGRjijbRQbUfDdZlc8tKpatV6QtY4CIiIhqEgMiLVZ2X7NMdRYW5pnB0dAaAGq8m6xKinLrFWnRGKDnpfz6RRz/Q0RU9/FTXpuV3dcsPxcNXmqARt6NpNO10U1mYGBQYb2iuj4GqOwMNI7/ISLSDwyItFj5fc2qEwCVn0LvbF3vmcpUdssPWfeZEHWqa63s+kVERFT3MSCq48pPofd9xvuV3fKjbPdZ2an1QN3sWuN0eiKiuouf5HVM2XFHAPDnTTXqezRCk85N/v/cDUChqHZrUdktPwqLBXbeTIZCoUBm6lV0aFpPmloP1I2utfIrWpfuhQZwOj0RUV3CgKiuKTPuCCg3+LrsmKRyrUXVWehR1lqUkQFV42ddwlH7VLaiNbvSiIjqHgZEdUz5cUfA32OPKpwrypbyVGehx/IbxJbdHLbsWKOMG3+h2LYG9wqr5a1AqlrRmrPRiIjqDn56E4CaX+jxebYeactWIJyNRkRUdzAg0lPlxxrV9EKPZVuPioXAnzfVMDJQ1Mhmsdq0FUhp61H5sUZC1GCLGBERPXcMiPTVo8Ya1TBZd5wiF4r/H4gNABmp12CYlQVAt6fqVzbWqKZwdhsR0fPHT1Q99aixRmVVmLVWjdlpZbvjiouL0ffi3ytfH354D3aGBvDIrYe4hxn47aayZNaaDgZHVY01elbcLJaI6PljQESPVr4lSZELxxvqak/dr2zla2cbA3Ro5oiiYoH6tyqucUTcLJaI6HljQESPVL4lqbi4GItvqkum7pcJji7dzERm1t8tTNUJlqpa4wiQd6cJIfBnqhrHkwxqZEzS81Z2fFFhYSEASN1d7PoiItIO/CSmp1I2QCobHN0yeABjA2PYlbYkPeOq2GVnqQEVV75eY2iG3/5/PaVnnr/2nLcdKT++yNCyHhxcXnhk11fZcUMcpE1E9PwxIKJqK996ZGxr/HdLUplVsaszg638GkeyVbFvJsOhmYO0+va9K882SLs2th0pO77IyNqhwsy0UqUtRmXHDdX0IG0iIqqIARE9H2VXxX7EDLayK2SX7XYr3wWXlKrG24rNaJ5bD3/mqbFT0Uh6TtmWpKoGaQNVb0arqW1HyrYcAUB22lVMDmiJZs2aISkpCZb1Hz9Iu/wMNHbBERFVDz856bko33pU1UKPZafkl+12q9AFZ5yLlS+YoYO7E44nGeA3g79X336SQdoAqtyMtsa3HXmKLrjSliMAyE6/js+2n36qqftlW5KetAsOYOBERFSeXn0ifvPNN/jiiy+QmpqKli1bYvHixXj11Vc1XSy98qjNZwF5t9uzdsE9stutXKtQ2W1HntWzdME9ydT98otAlrYklVd+HFLpxrSctk9EVJHeBESbN29GWFgYvvnmG3Tp0gXfffcdAgICcP78eTRu3FjTxdMf1V0QsoouuPIBVtmutvIz3araTqRstx0AJN7MxD2zZ9grTWEAOwsDONuUlMPO4tm2QSnf4nTnz1P49Loj6jeWtySVH5NUNggquzHto8YulfWoBSGftMWJLVNEpCv05pNp4cKFGDFiBN59910AwOLFi7F79258++23mDdvnoZLpz+edEHIx10nXVMuwCrb1VZ2WQAAuJymhqNVSaBia67Apf/fTiRTnYWFeWZwLL0HHsD32ko0fGCBszfu4w/zknFIZQdsA/IB3LJ/30zGGkXJLDgAstW5y19TfPcuHj58WHKcchkqr4rBV6UtTg7TK7QklR+TVDYIelS+smOXyi4LUDagAuQLQpbtqit7PYAq71G+ZapssFTTyxEwECOip6UXnxD5+fmIj4/HtGnTZOn+/v6Ii4vTUKmoJlQWYJV2tZVdFgBAlWOU0vNz0eClBmjkXfLFX1xcjMNKE9g2ssUtg1vwvb4SDXMtcPZGDmxMFWj8wBIAZMdl/33j3gOce9VD6gYsLCiEd1xpgPV3vmv37iPa1ArCriQIKi64h4yb9aXAqTRYUqddha25ovIWJyGQfuUSHj58iHspl2FWrz5yc0sCrLy8fNy6kljmXJlAzMQChhmZJcfXr+DTXwqlFqfSZQHKBlQlP5e/W5YuXryI3Id5MMx9iIxbNxG+LAk2jg0BAJkpF2FgZg0bx4bITElEow7+lbZMlW/BKn3uowKsqv5dmi8pKQlGRkZISUnB5vMPnqqL8HlvkfKoAPBJn1XTgR4DR6K/6cU7/86dOygqKoKTk5Ms3cnJCWlpaZVek5eXh7y8POlYrVYDALLKtBDUhJycHNy7cg8FuQUlz7mhhlGWEYxgVOW/me/J8mWnZsPExgTmtuYAADMbMxhZG8Hc1lz2b1MrU2Rdz0K6YToAyK4ztTHFngdWsLGxQYaFgIG5AZQ2JYO0yx6X/XdmJuCQlYv0CyX3y0nLwR6LivfIzAQcPO1R360+AKAwrxAvXvoBDdQWuHHvPnbnmUDx0AbZN7LwnYMZNl0vAgBk3i2GYdEfuJ+Viay0m3hwcwvw0Ab3b6nxyr1CNFJbAIDsHmXPlU0HgPvpWRAm/wBMLJCVcQ+G+cUQxiX/vp//B3LUGSXPTT6L8QfzYGnvhJzbqeh0fz88GljgYup9WJko0Mio5LkXs+/DKq/kODH7AU4lNcD9rEzZ9QCQdSMJlg2a4aEwlD03M/kSxi/+Q5bP0MwalvZOVf4bAO7fvYWXs/fCo4EFElMf4HbjIXgoDPEw4zZWrlxZ4e+/vFu3bmHbmVswVToAAPLUd/B6K6fHXvekyt6/fNmf9Fll71ET5avp+xE9i1GjRj2X+5Z+bz92PTehB27cuCEAiLi4OFn6Z599Jjw8PCq9ZtasWQIAX3zxxRdffPFVB17Xrl17ZKygFy1EDg4OMDQ0rNAalJ6eXuX/hqZPn46JEydKx8XFxbh37x7s7e2ldW2qKysrCy4uLrh27RpsbGye6V7aTB/qyTrWDaxj3cA61g01XUchBLKzs+Hs7PzIfHoREJmYmKBdu3aIjo7GwIEDpfTo6GgMGDCg0mtMTU1hamoqS6tXr16NlsvGxqbOvqHL0od6so51A+tYN7COdUNN1lGpVD42j14ERAAwceJEhISEoH379ujUqRO+//57pKSkPLc+SyIiItIdehMQvfXWW7h79y7mzJmD1NRUeHt7Y9euXWjSpImmi0ZEREQapjcBEQCMHj0ao0eP1nQxYGpqilmzZlXokqtr9KGerGPdwDrWDaxj3aCpOiqEeNw8NCIiIqK67Rn3FCAiIiLSfQyIiIiISO8xICIiIiK9x4CIiIiI9B4DIg345ptv4ObmBjMzM7Rr1w4HDx7UdJGqbd68eejQoQOsra3h6OiI119/vcIGmUIIREREwNnZGebm5vDz88O5c+c0VOJnM2/ePCgUCoSFhUlpdaV+N27cwNtvvw17e3tYWFigTZs2iI+Pl87rej0LCwvxySefwM3NDebm5mjatCnmzJmD4uJiKY+u1fHAgQPo168fnJ2doVAosG3bNtn5J6lPXl4exo0bBwcHB1haWqJ///64fv16Ldbi0R5Vx4KCAkydOhU+Pj6wtLSEs7Mz3nnnHdy8eVN2D12uY3nvv/8+FAoFFi9eLEuvC3W8cOEC+vfvD6VSCWtra7z88stISUmRzj/vOjIgqmWbN29GWFgYPv74Y5w6dQqvvvoqAgICZL90XRIbG4sxY8bgyJEjiI6ORmFhIfz9/XH//n0pz4IFC7Bw4UIsXboUx48fh0qlQs+ePZGdna3Bkj+948eP4/vvv0erVq1k6XWhfhkZGejSpQuMjY3x22+/4fz58/jqq69kq7Prej3nz5+PZcuWYenSpbhw4QIWLFiAL774ApGRkVIeXavj/fv30bp1ayxdurTS809Sn7CwMGzduhWbNm3CoUOHkJOTg8DAQBQVFdVWNR7pUXV88OABTp48iRkzZuDkyZPYsmULLl26hP79+8vy6XIdy9q2bRuOHj1a6RYUul7Hy5cv45VXXkGLFi2wf/9+nD59GjNmzICZmZmU57nX8Zl3TqWn8tJLL4lRo0bJ0lq0aCGmTZumoRLVrPT0dAFAxMbGCiGEKC4uFiqVSvzrX/+S8jx8+FAolUqxbNkyTRXzqWVnZwt3d3cRHR0tfH19xYcffiiEqDv1mzp1qnjllVeqPF8X6tm3b18xfPhwWdqgQYPE22+/LYTQ/ToCEFu3bpWOn6Q+mZmZwtjYWGzatEnKc+PGDWFgYCCioqJqrexPqnwdK3Ps2DEBQFy9elUIUXfqeP36ddGwYUNx9uxZ0aRJE7Fo0SLpXF2o41tvvSX9LVamNurIFqJalJ+fj/j4ePj7+8vS/f39ERcXp6FS1Sy1Wg0AsLOzAwAkJycjLS1NVmdTU1P4+vrqVJ3HjBmDvn37okePHrL0ulK/7du3o3379njzzTfh6OiItm3bYvny5dL5ulDPV155Bb///jsuXboEADh9+jQOHTqEPn36AKgbdSzrSeoTHx+PgoICWR5nZ2d4e3vrZJ2Bks8ghUIhtW7WhToWFxcjJCQEkydPRsuWLSuc1/U6FhcXY+fOnWjevDl69eoFR0dHdOzYUdatVht1ZEBUi+7cuYOioiI4OTnJ0p2cnJCWlqahUtUcIQQmTpyIV155Bd7e3gAg1UuX67xp0yacPHkS8+bNq3CuLtQPAP766y98++23cHd3x+7duzFq1CiMHz8ea9euBVA36jl16lQMGTIELVq0gLGxMdq2bYuwsDAMGTIEQN2oY1lPUp+0tDSYmJjA1ta2yjy65OHDh5g2bRqCg4OlTUHrQh3nz58PIyMjjB8/vtLzul7H9PR05OTk4F//+hd69+6NPXv2YODAgRg0aBBiY2MB1E4d9WrrDm2hUChkx0KICmm6aOzYsThz5gwOHTpU4Zyu1vnatWv48MMPsWfPHllfdnm6Wr9SxcXFaN++PebOnQsAaNu2Lc6dO4dvv/0W77zzjpRPl+u5efNmrF+/Hhs3bkTLli2RkJCAsLAwODs7Y9iwYVI+Xa5jZapTH12sc0FBAQYPHozi4mJ88803j82vK3WMj4/HkiVLcPLkyacur67UsXRiw4ABAzBhwgQAQJs2bRAXF4dly5bB19e3ymtrso5sIapFDg4OMDQ0rBDNpqenV/hfnK4ZN24ctm/fjn379qFRo0ZSukqlAgCdrXN8fDzS09PRrl07GBkZwcjICLGxsfj3v/8NIyMjqQ66Wr9SDRo0gJeXlyzN09NTGuyv679HAJg8eTKmTZuGwYMHw8fHByEhIZgwYYLU8lcX6ljWk9RHpVIhPz8fGRkZVebRBQUFBQgKCkJycjKio6Ol1iFA9+t48OBBpKeno3HjxtJn0NWrVzFp0iS4uroC0P06Ojg4wMjI6LGfQc+7jgyIapGJiQnatWuH6OhoWXp0dDQ6d+6soVI9GyEExo4diy1btmDv3r1wc3OTnXdzc4NKpZLVOT8/H7GxsTpR5+7du+OPP/5AQkKC9Grfvj2GDh2KhIQENG3aVKfrV6pLly4Vlku4dOkSmjRpAkD3f49AyYwkAwP5R56hoaH0v9O6UMeynqQ+7dq1g7GxsSxPamoqzp49qzN1Lg2G/vzzT8TExMDe3l52XtfrGBISgjNnzsg+g5ydnTF58mTs3r0bgO7X0cTEBB06dHjkZ1Ct1LFGhmbTE9u0aZMwNjYWK1asEOfPnxdhYWHC0tJSXLlyRdNFq5YPPvhAKJVKsX//fpGamiq9Hjx4IOX517/+JZRKpdiyZYv4448/xJAhQ0SDBg1EVlaWBktefWVnmQlRN+p37NgxYWRkJD7//HPx559/ig0bNggLCwuxfv16KY+u13PYsGGiYcOGYseOHSI5OVls2bJFODg4iClTpkh5dK2O2dnZ4tSpU+LUqVMCgFi4cKE4deqUNMPqSeozatQo0ahRIxETEyNOnjwpXnvtNdG6dWtRWFioqWrJPKqOBQUFon///qJRo0YiISFB9hmUl5cn3UOX61iZ8rPMhND9Om7ZskUYGxuL77//Xvz5558iMjJSGBoaioMHD0r3eN51ZECkAV9//bVo0qSJMDExES+++KI0RV0XAaj0tWrVKilPcXGxmDVrllCpVMLU1FR07dpV/PHHH5or9DMqHxDVlfr9+uuvwtvbW5iamooWLVqI77//XnZe1+uZlZUlPvzwQ9G4cWNhZmYmmjZtKj7++GPZF6eu1XHfvn2V/v0NGzZMCPFk9cnNzRVjx44VdnZ2wtzcXAQGBoqUlBQN1KZyj6pjcnJylZ9B+/btk+6hy3WsTGUBUV2o44oVK0SzZs2EmZmZaN26tdi2bZvsHs+7jgohhKiZtiYiIiIi3cQxRERERKT3GBARERGR3mNARERERHqPARERERHpPQZEREREpPcYEBEREZHeY0BEREREeo8BERHRU1i9ejXq1aun6WIQUQ1jQERERER6jwEREVEl8vPzNV0EIqpFDIiISCf9+uuvqFevnrRbfUJCAhQKBSZPnizlef/99zFkyBAAwM8//4yWLVvC1NQUrq6u+Oqrr2T3c3V1xWeffYbQ0FAolUqMHDkSQEkXWePGjWFhYYGBAwfi7t27sutOnz6Nbt26wdraGjY2NmjXrh1OnDjxPKtORM8BAyIi0kldu3ZFdnY2Tp06BQCIjY2Fg4MDYmNjpTz79++Hr68v4uPjERQUhMGDB+OPP/5AREQEZsyYgdWrV8vu+cUXX8Db2xvx8fGYMWMGjh49iuHDh2P06NFISEhAt27d8Nlnn8muGTp0KBo1aoTjx48jPj4e06ZNg7Gx8XOvPxHVLG7uSkQ6q127dggODsakSZMwcOBAdOjQAbNnz8adO3dw//59NGjQABcuXMCnn36K27dvY8+ePdK1U6ZMwc6dO3Hu3DkAJS1Ebdu2xdatW6U8wcHByMjIwG+//SalDR48GFFRUcjMzAQA2NjYIDIyEsOGDaudShPRc8EWIiLSWX5+fti/fz+EEDh48CAGDBgAb29vHDp0CPv27YOTkxNatGiBCxcuoEuXLrJru3Tpgj///BNFRUVSWvv27WV5Lly4gE6dOsnSyh9PnDgR7777Lnr06IF//etfuHz5cg3XkohqAwMiItJZfn5+OHjwIE6fPg0DAwN4eXnB19cXsbGxUncZAAghoFAoZNdW1jhuaWn52DzlRURE4Ny5c+jbty/27t0LLy8vWSsTEekGBkREpLNKxxEtXrwYvr6+UCgU8PX1xf79+2UBkZeXFw4dOiS7Ni4uDs2bN4ehoWGV9/fy8sKRI0dkaeWPAaB58+aYMGEC9uzZg0GDBmHVqlU1UDsiqk0MiIhIZymVSrRp0wbr16+Hn58fgJIg6eTJk7h06ZKUNmnSJPz+++/49NNPcenSJaxZswZLly5FeHj4I+8/fvx4REVFYcGCBbh06RKWLl2KqKgo6Xxubi7Gjh2L/fv34+rVq/jf//6H48ePw9PT83lVmYieEwZERKTTunXrhqKiIin4sbW1hZeXF+rXry8FJi+++CJ+/PFHbNq0Cd7e3pg5cybmzJmD0NDQR9775Zdfxg8//IDIyEi0adMGe/bswSeffCKdNzQ0xN27d/HOO++gefPmCAoKQkBAAGbPnv28qktEzwlnmREREZHeYwsRERER6T0GRERERKT3GBARERGR3mNARERERHqPARERERHpPQZEREREpPcYEBEREZHeY0BEREREeo8BEREREek9BkRERESk9xgQERERkd5jQERERER67/8Avu3qcHns3yIAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "\n",
    "stats_df = pd.DataFrame({\"words\": pd.concat([train_length, val_length, test_length], ignore_index=True),\n",
    "                        \"type\": len(train_length)*[\"train\"] +\n",
    "                         len(val_length)*[\"validation\"] + \n",
    "                         len(test_length)*[\"test\"]})\n",
    "\n",
    "sns.histplot(x=\"words\", \n",
    "             hue=\"type\", \n",
    "             data=stats_df, \n",
    "             multiple=\"stack\")\n",
    "\n",
    "plt.title(\"Number of sentences by word count\")\n",
    "tasks.util.save_plot(\"ex_2_dataset_stats.png\", OUTPUT_DIR)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see from the graph above, there is a sizable portion of our sentences that feature very few words. In order to make the RNN training more efficient, we choose to discard sentences with very few words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def exclude_small_sentences(conllu_df: pd.DataFrame, min_len: int) -> pd.DataFrame:\n",
    "    assert 1 <= min_len\n",
    "\n",
    "    length_df = length_sentences(conllu_df)\n",
    "    valid_length_df = length_df[length_df >= min_len]\n",
    "    valid_ids = set(valid_length_df.index)\n",
    "    return conllu_df[conllu_df.sent_id.isin(valid_ids)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "MIN_SENTENCE_LENGTH = 5\n",
    "\n",
    "train_df = exclude_small_sentences(train_df, MIN_SENTENCE_LENGTH)\n",
    "val_df = exclude_small_sentences(val_df, MIN_SENTENCE_LENGTH)\n",
    "test_df = exclude_small_sentences(test_df, MIN_SENTENCE_LENGTH)\n",
    "\n",
    "train_length = length_sentences(train_df)\n",
    "val_length = length_sentences(val_df)\n",
    "test_length = length_sentences(test_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    10539.000000\n",
       "mean        18.967170\n",
       "std         11.782365\n",
       "min          5.000000\n",
       "25%         10.000000\n",
       "50%         16.000000\n",
       "75%         24.000000\n",
       "max        159.000000\n",
       "Name: words, dtype: float64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_length.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    1538.000000\n",
       "mean       15.607932\n",
       "std        10.050704\n",
       "min         5.000000\n",
       "25%         8.000000\n",
       "50%        13.000000\n",
       "75%        20.000000\n",
       "max        75.000000\n",
       "Name: words, dtype: float64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "val_length.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    1535.000000\n",
       "mean       15.512052\n",
       "std        10.332400\n",
       "min         5.000000\n",
       "25%         8.000000\n",
       "50%        13.000000\n",
       "75%        20.000000\n",
       "max        81.000000\n",
       "Name: words, dtype: float64"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_length.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vocabulary size: 15967\n"
     ]
    }
   ],
   "source": [
    "vocab_size = len(set(train_df.words))\n",
    "print(f\"Vocabulary size: {vocab_size}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total word count:\n",
      "Training: 199895\n",
      "Validation: 24005\n",
      "Testing: 23811\n"
     ]
    }
   ],
   "source": [
    "print(f\"Total word count:\\nTraining: {train_df.shape[0]}\"\n",
    "      f\"\\nValidation: {val_df.shape[0]}\"\n",
    "      f\"\\nTesting: {test_df.shape[0]}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total sentence count:\n",
      "Training: 10539\n",
      "Validation: 1538\n",
      "Testing: 1535\n"
     ]
    }
   ],
   "source": [
    "print(f\"Total sentence count:\\nTraining: {len(set(train_df.sent_id))}\"\n",
    "      f\"\\nValidation: {len(set(val_df.sent_id))}\"\n",
    "      f\"\\nTesting: {len(set(test_df.sent_id))}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Vectorization\n",
    "\n",
    "In order to make the RNN training more efficient, we choose to discard sentences with very few words. We also set a window size equal to the 90\\% percentile of sentence word count, meaning tht 90\\% of our windows will fully fit the training sentences. The rest will be automatically split into more sentences, and as such don't need to be excluded from the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "34"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "MAX_SEQUENCE_LENGTH = int(np.quantile(train_length, 0.9))\n",
    "MAX_SEQUENCE_LENGTH"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will be using a combination of the `keras.preprocessing.Tokenizer` and `keras.utils.pad_sequences` utilities to create custom windows of words to be fed to our model, since it uses Time Distributed outputs. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def encode(tokenizer, values):\n",
    "    return tokenizer.texts_to_sequences(values)\n",
    "\n",
    "\n",
    "def encode_with_padding(tokenizer, max_seq_len, values):\n",
    "    tokens = encode(tokenizer, values)\n",
    "    padded_tokens = keras.utils.pad_sequences(\n",
    "        tokens, maxlen=max_seq_len, padding=\"pre\", truncating=\"post\"\n",
    "    )\n",
    "    return padded_tokens\n",
    "\n",
    "\n",
    "def decode(tokenizer, encoded_sequence, axis=2):\n",
    "    return np.array(\n",
    "        [\n",
    "            tokenizer.index_word[str(x[-1])]\n",
    "            for x in np.argmax(encoded_sequence, axis=axis)\n",
    "        ]\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# encode X\n",
    "word_tokenizer = keras.preprocessing.text.Tokenizer(filters=\"\")\n",
    "word_tokenizer.fit_on_texts(train_df.words.values)\n",
    "\n",
    "train_data = encode_with_padding(\n",
    "    word_tokenizer, MAX_SEQUENCE_LENGTH, train_df.words.values\n",
    ")\n",
    "val_data = encode_with_padding(\n",
    "    word_tokenizer, MAX_SEQUENCE_LENGTH, val_df.words.values\n",
    ")\n",
    "test_data = encode_with_padding(\n",
    "    word_tokenizer, MAX_SEQUENCE_LENGTH, test_df.words.values\n",
    ")\n",
    "\n",
    "tag_tokenizer = keras.preprocessing.text.Tokenizer()\n",
    "tag_tokenizer.fit_on_texts(train_df.pos.values)\n",
    "\n",
    "# start label counting from 0, since to_categorical assumes argmax = number_of_categories\n",
    "tag_tokenizer.word_index = {\n",
    "    key: value - 1 for key, value in tag_tokenizer.word_index.items()\n",
    "}\n",
    "\n",
    "# start label counting from 0, since to_categorical assumes argmax = number_of_categories\n",
    "tag_tokenizer.index_word = {\n",
    "    str(int(key) - 1): value for key, value in tag_tokenizer.index_word.items()\n",
    "}\n",
    "\n",
    "y_train = keras.utils.to_categorical(\n",
    "    encode_with_padding(tag_tokenizer, MAX_SEQUENCE_LENGTH, train_df.pos.values)\n",
    ")\n",
    "y_valid = keras.utils.to_categorical(\n",
    "    encode_with_padding(tag_tokenizer, MAX_SEQUENCE_LENGTH, val_df.pos.values)\n",
    ")\n",
    "y_test = keras.utils.to_categorical(\n",
    "    encode_with_padding(tag_tokenizer, MAX_SEQUENCE_LENGTH, test_df.pos.values)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'noun': 0,\n",
       " 'punct': 1,\n",
       " 'verb': 2,\n",
       " 'pron': 3,\n",
       " 'adp': 4,\n",
       " 'det': 5,\n",
       " 'adj': 6,\n",
       " 'aux': 7,\n",
       " 'propn': 8,\n",
       " 'adv': 9,\n",
       " 'cconj': 10,\n",
       " 'part': 11,\n",
       " 'sconj': 12,\n",
       " 'num': 13,\n",
       " 'sym': 14,\n",
       " 'intj': 15,\n",
       " 'x': 16}"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check that counting is continuous starting from 0\n",
    "tag_tokenizer.word_index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(199895, 34, 17)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# input shape\n",
    "y_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transformers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will be using the Transformers library in order to leverage already pretrained BERT models for the POS tagging task."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Representation\n",
    "\n",
    "We start by splitting our text into individual sentences and loading them in a dataset dict (from the datasets library) in order to properly feed them to our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Converting dataframe to individual sentences...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "04ddcfd3bf1d4608ba810662e8cb6b08",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/10539 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e1095cbb32454dfa873bdb4a3c6dcec3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/10539 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a48dca1218334722ac10abdb1f754d7c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1538 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2aa81f46114e45589ed7a81c21b9bd90",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1538 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5a3db4cb62e047548c82c0a3994d88e4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1535 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fb2e393c7f4c4477bdb41505e662210d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1535 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    train: Dataset({\n",
       "        features: ['labels', 'words'],\n",
       "        num_rows: 10539\n",
       "    })\n",
       "    val: Dataset({\n",
       "        features: ['labels', 'words'],\n",
       "        num_rows: 1538\n",
       "    })\n",
       "    test: Dataset({\n",
       "        features: ['labels', 'words'],\n",
       "        num_rows: 1535\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import datasets\n",
    "import transformers\n",
    "\n",
    "\n",
    "def encode_bert(df, col_name):\n",
    "    sentences = []\n",
    "\n",
    "    for sentence_id in tqdm(set(df.sent_id)):\n",
    "        words_df = df[df.sent_id == sentence_id]\n",
    "        sentence = list(words_df[col_name])\n",
    "        sentences.append(sentence)\n",
    "\n",
    "    return sentences\n",
    "\n",
    "\n",
    "def decode_bert(encoded_sequence, num_classes):\n",
    "    # create dummy class equal to num_classes for -100 tokens\n",
    "    return keras.utils.to_categorical(\n",
    "        [x if x != -100 else num_classes for x in encoded_sequence.flatten()],\n",
    "        num_classes=num_classes + 1,\n",
    "    )\n",
    "\n",
    "\n",
    "print(\"Converting dataframe to individual sentences...\")\n",
    "data_dict = datasets.DatasetDict(\n",
    "    {\n",
    "        \"train\": datasets.Dataset.from_dict(\n",
    "            {\n",
    "                \"labels\": tag_tokenizer.texts_to_sequences(\n",
    "                    encode_bert(train_df, \"pos\")\n",
    "                ),\n",
    "                \"words\": encode_bert(train_df, \"words\"),\n",
    "            }\n",
    "        ),\n",
    "        \"val\": datasets.Dataset.from_dict(\n",
    "            {\n",
    "                \"labels\": tag_tokenizer.texts_to_sequences(\n",
    "                    encode_bert(val_df, \"pos\")\n",
    "                ),\n",
    "                \"words\": encode_bert(val_df, \"words\"),\n",
    "            }\n",
    "        ),\n",
    "        \"test\": datasets.Dataset.from_dict(\n",
    "            {\n",
    "                \"labels\": tag_tokenizer.texts_to_sequences(\n",
    "                    encode_bert(test_df, \"pos\")\n",
    "                ),\n",
    "                \"words\": encode_bert(test_df, \"words\"),\n",
    "            }\n",
    "        ),\n",
    "    }\n",
    ")\n",
    "data_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model we will be using is the [Tweebank NLP Bert Model](https://huggingface.co/TweebankNLP/bertweet-tb2_ewt-pos-tagging) which is pretrained on the POS tagging task on a similar dataset to ours.\n",
    "\n",
    "Unfortunately, while the authors do provide their own tokenizer, it is a \"slow\" tokenizer, meaning that its API is restricted. Since we need parts of the API to properly align labels and tokens, we will be using another \"fast\" tokenizer. \n",
    "\n",
    "We pick the vanilla bert-base-uncased tokenizer since its unlikely to deviate significantly from the tweebank tokenizer, and we choose the uncased version since, for the reasons mentioned above, we have converted all words to lowercase."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RobertaForTokenClassification(\n",
      "  (roberta): RobertaModel(\n",
      "    (embeddings): RobertaEmbeddings(\n",
      "      (word_embeddings): Embedding(64001, 768, padding_idx=1)\n",
      "      (position_embeddings): Embedding(130, 768, padding_idx=1)\n",
      "      (token_type_embeddings): Embedding(1, 768)\n",
      "      (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
      "      (dropout): Dropout(p=0.1, inplace=False)\n",
      "    )\n",
      "    (encoder): RobertaEncoder(\n",
      "      (layer): ModuleList(\n",
      "        (0-11): 12 x RobertaLayer(\n",
      "          (attention): RobertaAttention(\n",
      "            (self): RobertaSelfAttention(\n",
      "              (query): Linear(in_features=768, out_features=768, bias=True)\n",
      "              (key): Linear(in_features=768, out_features=768, bias=True)\n",
      "              (value): Linear(in_features=768, out_features=768, bias=True)\n",
      "              (dropout): Dropout(p=0.1, inplace=False)\n",
      "            )\n",
      "            (output): RobertaSelfOutput(\n",
      "              (dense): Linear(in_features=768, out_features=768, bias=True)\n",
      "              (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
      "              (dropout): Dropout(p=0.1, inplace=False)\n",
      "            )\n",
      "          )\n",
      "          (intermediate): RobertaIntermediate(\n",
      "            (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
      "            (intermediate_act_fn): GELUActivation()\n",
      "          )\n",
      "          (output): RobertaOutput(\n",
      "            (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
      "            (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
      "            (dropout): Dropout(p=0.1, inplace=False)\n",
      "          )\n",
      "        )\n",
      "      )\n",
      "    )\n",
      "  )\n",
      "  (dropout): Dropout(p=0.1, inplace=False)\n",
      "  (classifier): Linear(in_features=768, out_features=17, bias=True)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "def model_init():\n",
    "    return transformers.AutoModelForTokenClassification.from_pretrained(\n",
    "        \"TweebankNLP/bertweet-tb2_ewt-pos-tagging\",\n",
    "        num_labels=len(train_df.pos.unique()),\n",
    "    )\n",
    "\n",
    "\n",
    "bert_tokenizer = transformers.AutoTokenizer.from_pretrained(\n",
    "    \"bert-base-uncased\", use_fast=True\n",
    ")\n",
    "\n",
    "print(model_init())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An important issue in token classification for BERT models, is that the underlying model does not operate on words as tokens, but to sub-word tokens. This effectively means that for each word we must predict pseudo-tokens followed by the real POS token. \n",
    "\n",
    "The modified function below will map each token to multiple pseudo-labels (with the value -100), while the last token will be the actual POS tag from our dataset. We apply this function to all data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5e15fd06f0374a8b942999479b0100bd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/10539 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6243978c279b4630ac56d7b2bd6e76c1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/1538 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2549291dc5a54f25bb009412f34f2cf1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/1535 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    train: Dataset({\n",
       "        features: ['labels', 'words', 'input_ids', 'token_type_ids', 'attention_mask'],\n",
       "        num_rows: 10539\n",
       "    })\n",
       "    val: Dataset({\n",
       "        features: ['labels', 'words', 'input_ids', 'token_type_ids', 'attention_mask'],\n",
       "        num_rows: 1538\n",
       "    })\n",
       "    test: Dataset({\n",
       "        features: ['labels', 'words', 'input_ids', 'token_type_ids', 'attention_mask'],\n",
       "        num_rows: 1535\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def tokenize_and_align_labels(examples):\n",
    "    tokenized_inputs = bert_tokenizer(\n",
    "        examples[\"words\"],\n",
    "        truncation=True,\n",
    "        is_split_into_words=True,\n",
    "        padding=\"max_length\",\n",
    "        max_length=MAX_SEQUENCE_LENGTH,\n",
    "    )\n",
    "\n",
    "    labels = []\n",
    "    for i, label in enumerate(examples[\"labels\"]):\n",
    "        word_ids = tokenized_inputs.word_ids(\n",
    "            batch_index=i\n",
    "        )  # Map tokens to their respective word.\n",
    "        previous_word_idx = None\n",
    "        label_ids = []\n",
    "        for word_idx in word_ids:  # Set the special tokens to -100.\n",
    "            if word_idx is None:\n",
    "                label_ids.append(-100)\n",
    "            elif (\n",
    "                word_idx != previous_word_idx\n",
    "            ):  # Only label the first token of a given word.\n",
    "                label_ids.append(label[word_idx])\n",
    "            else:\n",
    "                label_ids.append(-100)\n",
    "            previous_word_idx = word_idx\n",
    "        labels.append(label_ids)\n",
    "\n",
    "    tokenized_inputs[\"labels\"] = labels\n",
    "    return tokenized_inputs\n",
    "\n",
    "\n",
    "tokenized_data_dict = data_dict.map(tokenize_and_align_labels, batched=True)\n",
    "tokenized_data_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also add a data collator, which will handle padding for our sentences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_collator = transformers.DataCollatorWithPadding(tokenizer=bert_tokenizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to extract different metrics for each evaluation period, we construct our own compute_metrics function. This is necessary since a lot of custom processing is need for token classification tasks, unlike with binary classification tasks where the code is trivial.\n",
    "\n",
    "We specifically turn the predictions into a continuous array of one-hot vectors, which can be used by sklearn to compute the metrics. We represent the dummy class (-100) with an extra integer, which is used to create a mask. This mask is applied both to the true labels and predictions, leading to two matrices composed of one-hot-vectors of only valid POS tags. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_metrics(pred):\n",
    "    labels = decode_bert(pred.label_ids, len(np.unique(pred.label_ids)))\n",
    "    preds = decode_bert(pred.predictions.argmax(-1), len(np.unique(pred.label_ids)))\n",
    "\n",
    "    # ignore dummy label\n",
    "    mask = labels[:, -1] != 1\n",
    "    labels = labels[mask]\n",
    "    preds = preds[mask]\n",
    "\n",
    "    # calculate accuracy using sklearn's function\n",
    "    acc = sklearn.metrics.accuracy_score(labels, preds)\n",
    "    precision = sklearn.metrics.precision_score(\n",
    "        labels, preds, average=\"macro\", zero_division=0\n",
    "    )\n",
    "    recall = sklearn.metrics.recall_score(\n",
    "        labels, preds, average=\"macro\", zero_division=0\n",
    "    )\n",
    "    f1 = sklearn.metrics.f1_score(labels, preds, average=\"macro\", zero_division=0)\n",
    "\n",
    "    return {\n",
    "        \"val_accuracy\": acc,\n",
    "        \"val_precision\": precision,\n",
    "        \"val_recall\": recall,\n",
    "        \"val_f1\": f1\n",
    "    }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hyper parameter tuning and training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will be using the Trainer class from the transformers library to train models of various hyper-parameters.\n",
    "\n",
    "* We exploit the gradient_accumulation_steps parameter to simulate a larger batch size, without incurring further GPU VRAM cost\n",
    "* We set up early stopping with respect to validation accuracy like with the models in previous assignments\n",
    "* We set the evaluation and save procedures to be executed each 100 steps, in order to strike a balance between the evaluation overhead and accurate statistics\n",
    "* We provide our compute_metrics function in order to get detailed statistics in every evaluation period\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_args = transformers.TrainingArguments(\n",
    "    output_dir=OUTPUT_DIR,  # output directory\n",
    "    num_train_epochs=8,  # total number of training epochs\n",
    "    per_device_train_batch_size=16,  # batch size per device during training\n",
    "    per_device_eval_batch_size=16,  # batch size for evaluation\n",
    "    gradient_accumulation_steps=2,  # Number of update steps (forward passes) to accumulate the gradients for, before performing a backward/update pass\n",
    "    logging_dir=\"./logs\",  # directory for storing logs\n",
    "    eval_steps=100,\n",
    "    save_steps=100,\n",
    "    evaluation_strategy=transformers.IntervalStrategy.STEPS,\n",
    "    load_best_model_at_end=True,\n",
    "    metric_for_best_model = \"val_accuracy\",\n",
    "    save_total_limit=3,\n",
    "    # hyperparameters to be tuned\n",
    "    learning_rate=1e-5,\n",
    "    weight_decay=0.01,  # strength of weight decay\n",
    "    warmup_steps=500,  # number of warmup steps for learning rate scheduler\n",
    ")\n",
    "\n",
    "trainer = transformers.Trainer(\n",
    "    model_init=model_init,  # the instantiated Transformers model to be trained\n",
    "    args=training_args,  # training arguments, as defined above\n",
    "    train_dataset=tokenized_data_dict[\"train\"],  # training dataset\n",
    "    eval_dataset=tokenized_data_dict[\"val\"],  # evaluation dataset\n",
    "    tokenizer=bert_tokenizer,  # tokenizer\n",
    "    data_collator=data_collator, # batch creator (padding)\n",
    "    callbacks=[\n",
    "        transformers.EarlyStoppingCallback(early_stopping_patience=5)\n",
    "    ], \n",
    "   compute_metrics=compute_metrics    \n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model features hundreds of millions of parameters making training very expensive. Given that we run our model locally, this places a very heavy restriction on the number of tuning runs we can afford and restraining us from choices we otherwise would like to explore. Namely:\n",
    "\n",
    "* We only consider one pretrained model. However, this model is a large BERT model pretrained on the same task and very similar dataset. It is therefore improbable that another model could outperform it.\n",
    "* We do not unfreeze any BERT layers since that would sharply increase the training time. However, as mentioned above, the specific model has been pretrained on the same family of datasets as ours. Thus it is unlikely that the underlying BERT model would \"learn\" anything new from our fine-tuning.\n",
    "* We restrict the hyperparameter runs to only 2. However, given the relative lack of hyperparameters to tune, this should not significantly restrict the final performance of our model.\n",
    "\n",
    "Given the above limitations we define our search space as:\n",
    "* The learning rate\n",
    "* The weight decay\n",
    "* The warmup steps\n",
    "\n",
    "Unlike keras_tuner, unfortunately we [cannot access automatically the best run](https://discuss.huggingface.co/t/oserror-unable-to-load-weights-from-pytorch-checkpoint-file/3406). Thus we choose the best model by hand from our local files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[I 2024-03-11 18:54:59,082] A new study created in memory with name: no-name-5615f3ca-5e0d-434c-913f-a6e90814bf80\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1114e8302d694f45bb5f633e0ab69f53",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/2632 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e11a7c71e5604ef3a59ecf82ebb6cb47",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 2.2895572185516357, 'eval_val_accuracy': 0.281227789623312, 'eval_val_precision': 0.20770931162422182, 'eval_val_recall': 0.14807040287477546, 'eval_val_f1': 0.141506418142216, 'eval_runtime': 14.1204, 'eval_samples_per_second': 108.921, 'eval_steps_per_second': 6.87, 'epoch': 0.3}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e677880d83b44166bdbd19b6b41d4c77",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 1.0403493642807007, 'eval_val_accuracy': 0.6592484008528785, 'eval_val_precision': 0.5308896706467112, 'eval_val_recall': 0.44873890156380436, 'eval_val_f1': 0.4475089794972891, 'eval_runtime': 14.0297, 'eval_samples_per_second': 109.625, 'eval_steps_per_second': 6.914, 'epoch': 0.61}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b7a9b2006ac848ae822089538ace7576",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.7816298604011536, 'eval_val_accuracy': 0.747867803837953, 'eval_val_precision': 0.656236313125712, 'eval_val_recall': 0.5568110557995077, 'eval_val_f1': 0.5792111074382582, 'eval_runtime': 14.058, 'eval_samples_per_second': 109.404, 'eval_steps_per_second': 6.9, 'epoch': 0.91}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "625622fd45f24aec99c5be7bd64fac95",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.6788504719734192, 'eval_val_accuracy': 0.7778962331201137, 'eval_val_precision': 0.6618899768537495, 'eval_val_recall': 0.6108393753472136, 'eval_val_f1': 0.6236876864192361, 'eval_runtime': 14.1625, 'eval_samples_per_second': 108.597, 'eval_steps_per_second': 6.849, 'epoch': 1.21}\n",
      "{'loss': 1.6491, 'grad_norm': 2.235081911087036, 'learning_rate': 1.1525954770195341e-05, 'epoch': 1.52}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a2ece1c9df1f4f9aaf49e73a3ad31b62",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.6100931167602539, 'eval_val_accuracy': 0.8039712153518124, 'eval_val_precision': 0.6942061131697919, 'eval_val_recall': 0.6289722118969084, 'eval_val_f1': 0.6479404008534901, 'eval_runtime': 13.9919, 'eval_samples_per_second': 109.921, 'eval_steps_per_second': 6.933, 'epoch': 1.52}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2f3ab048f35c43858640ad9febe9d366",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5571183562278748, 'eval_val_accuracy': 0.8213841506751954, 'eval_val_precision': 0.7078465844665469, 'eval_val_recall': 0.6470971435475196, 'eval_val_f1': 0.6674480433616473, 'eval_runtime': 14.1088, 'eval_samples_per_second': 109.01, 'eval_steps_per_second': 6.875, 'epoch': 1.82}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8858d27ac72842cf8caeaa6d46d78471",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5246677398681641, 'eval_val_accuracy': 0.8326226012793176, 'eval_val_precision': 0.7116732272522, 'eval_val_recall': 0.6539314690931907, 'eval_val_f1': 0.6731896673573482, 'eval_runtime': 14.1831, 'eval_samples_per_second': 108.439, 'eval_steps_per_second': 6.839, 'epoch': 2.12}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "57da3742e0c440a08b833c8ec03719d3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5061463117599487, 'eval_val_accuracy': 0.8404406538734897, 'eval_val_precision': 0.717730207571798, 'eval_val_recall': 0.6601349590081078, 'eval_val_f1': 0.6784436520977993, 'eval_runtime': 14.3975, 'eval_samples_per_second': 106.824, 'eval_steps_per_second': 6.737, 'epoch': 2.43}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c98775226f764da4bd3b0e13ca42107a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.48714911937713623, 'eval_val_accuracy': 0.8459932480454868, 'eval_val_precision': 0.7162251610890968, 'eval_val_recall': 0.6698892551376994, 'eval_val_f1': 0.6844613929182299, 'eval_runtime': 14.2934, 'eval_samples_per_second': 107.602, 'eval_steps_per_second': 6.786, 'epoch': 2.73}\n",
      "{'loss': 0.5134, 'grad_norm': 2.4881861209869385, 'learning_rate': 8.822869692757409e-06, 'epoch': 3.03}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0bcb431b51914a76b5939d9328f935c9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.45911985635757446, 'eval_val_accuracy': 0.8549218194740583, 'eval_val_precision': 0.724381200812568, 'eval_val_recall': 0.6780709808013179, 'eval_val_f1': 0.6944893403215192, 'eval_runtime': 14.0326, 'eval_samples_per_second': 109.602, 'eval_steps_per_second': 6.912, 'epoch': 3.03}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f20c2709f59e4f44bdb05cb520678261",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.45033904910087585, 'eval_val_accuracy': 0.8594083155650319, 'eval_val_precision': 0.721792554639697, 'eval_val_recall': 0.6784434446119366, 'eval_val_f1': 0.6940331020887488, 'eval_runtime': 14.0479, 'eval_samples_per_second': 109.483, 'eval_steps_per_second': 6.905, 'epoch': 3.34}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "46bbf9e979c6472ea8891b0ae78dbdf0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.43498703837394714, 'eval_val_accuracy': 0.8638503909026297, 'eval_val_precision': 0.7248721196185203, 'eval_val_recall': 0.6947291627026653, 'eval_val_f1': 0.706272220049348, 'eval_runtime': 14.1154, 'eval_samples_per_second': 108.959, 'eval_steps_per_second': 6.872, 'epoch': 3.64}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "85fb959f018544a8bb0d3946aff0f39e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.4230602979660034, 'eval_val_accuracy': 0.8669598436389481, 'eval_val_precision': 0.729166449314794, 'eval_val_recall': 0.6916947449673134, 'eval_val_f1': 0.7061393450103848, 'eval_runtime': 14.0887, 'eval_samples_per_second': 109.165, 'eval_steps_per_second': 6.885, 'epoch': 3.95}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b513adec0c044384a99cc74f302c8c60",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.41802892088890076, 'eval_val_accuracy': 0.871090973702914, 'eval_val_precision': 0.7280796746065606, 'eval_val_recall': 0.6950290490512061, 'eval_val_f1': 0.7077968590488897, 'eval_runtime': 14.185, 'eval_samples_per_second': 108.425, 'eval_steps_per_second': 6.838, 'epoch': 4.25}\n",
      "{'loss': 0.3947, 'grad_norm': 2.7936789989471436, 'learning_rate': 6.119784615319478e-06, 'epoch': 4.55}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "70b8601b93ab45c094d1c9f3637d9b0d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.4073221981525421, 'eval_val_accuracy': 0.8735341151385928, 'eval_val_precision': 0.728022690841692, 'eval_val_recall': 0.7039639377683824, 'eval_val_f1': 0.7134383560739861, 'eval_runtime': 13.9546, 'eval_samples_per_second': 110.215, 'eval_steps_per_second': 6.951, 'epoch': 4.55}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "57a202d7381441bc8cfa825c25a41ae2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.40537986159324646, 'eval_val_accuracy': 0.8746002132196162, 'eval_val_precision': 0.7285198979686421, 'eval_val_recall': 0.7032276511395744, 'eval_val_f1': 0.7131545469763402, 'eval_runtime': 14.1746, 'eval_samples_per_second': 108.504, 'eval_steps_per_second': 6.843, 'epoch': 4.86}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0dff504750b647bc948fb28406ef7b83",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.40025070309638977, 'eval_val_accuracy': 0.8765547263681592, 'eval_val_precision': 0.7329005182075231, 'eval_val_recall': 0.7053766224707697, 'eval_val_f1': 0.7166986366976763, 'eval_runtime': 14.2137, 'eval_samples_per_second': 108.206, 'eval_steps_per_second': 6.824, 'epoch': 5.16}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f698c8d8b3f24e29925f8747e1448c1b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.3983023762702942, 'eval_val_accuracy': 0.8773098791755508, 'eval_val_precision': 0.7367526692486267, 'eval_val_recall': 0.6996098639105034, 'eval_val_f1': 0.7142789942204363, 'eval_runtime': 14.13, 'eval_samples_per_second': 108.846, 'eval_steps_per_second': 6.865, 'epoch': 5.46}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b1a7b4603bbb4f279e5cc5d274b6edf0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.38873234391212463, 'eval_val_accuracy': 0.8801083866382374, 'eval_val_precision': 0.736026741082204, 'eval_val_recall': 0.711340101250299, 'eval_val_f1': 0.7214404400623572, 'eval_runtime': 14.104, 'eval_samples_per_second': 109.047, 'eval_steps_per_second': 6.877, 'epoch': 5.77}\n",
      "{'loss': 0.3373, 'grad_norm': 2.606633424758911, 'learning_rate': 3.416699537881546e-06, 'epoch': 6.07}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "aac32c37976a490ab90d6dca15933b6a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.3901931941509247, 'eval_val_accuracy': 0.8805525941719972, 'eval_val_precision': 0.7359626060701457, 'eval_val_recall': 0.7067979705030215, 'eval_val_f1': 0.7182534060805864, 'eval_runtime': 14.087, 'eval_samples_per_second': 109.179, 'eval_steps_per_second': 6.886, 'epoch': 6.07}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a32c4572bc2749b08cd7ecc82f0be642",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.388123482465744, 'eval_val_accuracy': 0.8817075337597725, 'eval_val_precision': 0.7397224796828031, 'eval_val_recall': 0.7106473354747063, 'eval_val_f1': 0.7223802654341922, 'eval_runtime': 14.1798, 'eval_samples_per_second': 108.464, 'eval_steps_per_second': 6.841, 'epoch': 6.37}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "077073c816ee465b87fd2b0efe712b10",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.3839591443538666, 'eval_val_accuracy': 0.8821073205401564, 'eval_val_precision': 0.73548699007709, 'eval_val_recall': 0.7101414585681551, 'eval_val_f1': 0.7202725921141495, 'eval_runtime': 14.289, 'eval_samples_per_second': 107.636, 'eval_steps_per_second': 6.788, 'epoch': 6.68}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e365557809c14608837c1709762074d2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.3833221197128296, 'eval_val_accuracy': 0.8832178393745558, 'eval_val_precision': 0.7394886336133901, 'eval_val_recall': 0.7139557825505021, 'eval_val_f1': 0.7242106889475065, 'eval_runtime': 14.1897, 'eval_samples_per_second': 108.389, 'eval_steps_per_second': 6.836, 'epoch': 6.98}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "12a5b57946da40749dc1c24b103dc6e7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.3806934058666229, 'eval_val_accuracy': 0.8831289978678039, 'eval_val_precision': 0.736551004362954, 'eval_val_recall': 0.7144951673437706, 'eval_val_f1': 0.7235628155914561, 'eval_runtime': 14.2231, 'eval_samples_per_second': 108.134, 'eval_steps_per_second': 6.82, 'epoch': 7.28}\n",
      "{'loss': 0.3057, 'grad_norm': 2.547905206680298, 'learning_rate': 7.13614460443614e-07, 'epoch': 7.59}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "10607589399f4a22a6bb548b8ab44d4b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.3773425221443176, 'eval_val_accuracy': 0.884728144989339, 'eval_val_precision': 0.7401694958670075, 'eval_val_recall': 0.7144758967580753, 'eval_val_f1': 0.7250738775604496, 'eval_runtime': 13.9035, 'eval_samples_per_second': 110.619, 'eval_steps_per_second': 6.977, 'epoch': 7.59}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fd74fdc8043b44d398ab953f208b3441",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.37734630703926086, 'eval_val_accuracy': 0.8855277185501066, 'eval_val_precision': 0.7394849891160207, 'eval_val_recall': 0.7160341035939369, 'eval_val_f1': 0.725633110857248, 'eval_runtime': 14.0936, 'eval_samples_per_second': 109.128, 'eval_steps_per_second': 6.883, 'epoch': 7.89}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[I 2024-03-11 19:43:53,435] Trial 0 finished with value: 3.0666799221173124 and parameters: {'learning_rate': 1.326674156006537e-05, 'weight_decay': 0.004665512327548703, 'warmup_steps': 178}. Best is trial 0 with value: 3.0666799221173124.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'train_runtime': 2933.404, 'train_samples_per_second': 28.742, 'train_steps_per_second': 0.897, 'train_loss': 0.6233263305617683, 'epoch': 7.99}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "da8987775f724f3fa59c4c8fb19f77ee",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/2632 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "14a8f76c922549cbbbfa7c4483263426",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 2.3950819969177246, 'eval_val_accuracy': 0.18821073205401564, 'eval_val_precision': 0.1356099358616886, 'eval_val_recall': 0.09484712789610579, 'eval_val_f1': 0.07483496650125018, 'eval_runtime': 13.9836, 'eval_samples_per_second': 109.986, 'eval_steps_per_second': 6.937, 'epoch': 0.3}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dc3597f117b24eff848541f77d889767",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 1.3147863149642944, 'eval_val_accuracy': 0.5759594882729211, 'eval_val_precision': 0.3851492018095372, 'eval_val_recall': 0.36296766576682127, 'eval_val_f1': 0.3592079069524317, 'eval_runtime': 14.0447, 'eval_samples_per_second': 109.507, 'eval_steps_per_second': 6.907, 'epoch': 0.61}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "52aa028f93cd4305a194a12ed4dc8021",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.8517316579818726, 'eval_val_accuracy': 0.7256130063965884, 'eval_val_precision': 0.6534268158722861, 'eval_val_recall': 0.5293681387883854, 'eval_val_f1': 0.5500741764280876, 'eval_runtime': 14.0354, 'eval_samples_per_second': 109.58, 'eval_steps_per_second': 6.911, 'epoch': 0.91}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5928d0fdd12b41b5815d71df397c78a2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.7356997728347778, 'eval_val_accuracy': 0.7595504619758351, 'eval_val_precision': 0.6491710095778443, 'eval_val_recall': 0.5763857765450645, 'eval_val_f1': 0.5914979385525125, 'eval_runtime': 14.0558, 'eval_samples_per_second': 109.421, 'eval_steps_per_second': 6.901, 'epoch': 1.21}\n",
      "{'loss': 1.8261, 'grad_norm': 2.509305715560913, 'learning_rate': 1.0262482216512226e-05, 'epoch': 1.52}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9a773a0eda174fb491a715f56b02cd8e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.6524848937988281, 'eval_val_accuracy': 0.7884683724235964, 'eval_val_precision': 0.6829737051965935, 'eval_val_recall': 0.6146525669155468, 'eval_val_f1': 0.6352491209526633, 'eval_runtime': 13.9003, 'eval_samples_per_second': 110.645, 'eval_steps_per_second': 6.978, 'epoch': 1.52}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "264baed9e6b3447ab75d2bd40f7f8a51",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5940662622451782, 'eval_val_accuracy': 0.8059257285003554, 'eval_val_precision': 0.6981170370328511, 'eval_val_recall': 0.6311565696117111, 'eval_val_f1': 0.6529923715208772, 'eval_runtime': 14.1257, 'eval_samples_per_second': 108.88, 'eval_steps_per_second': 6.867, 'epoch': 1.82}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "35b2d813d9b147dda69de4e047883f1c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5532407760620117, 'eval_val_accuracy': 0.8224946695095949, 'eval_val_precision': 0.707690430606708, 'eval_val_recall': 0.6459400432294906, 'eval_val_f1': 0.6668709132095422, 'eval_runtime': 14.024, 'eval_samples_per_second': 109.669, 'eval_steps_per_second': 6.917, 'epoch': 2.12}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9bda065f53b240bd9055aec5d7323475",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5391104221343994, 'eval_val_accuracy': 0.8302238805970149, 'eval_val_precision': 0.7116310584997733, 'eval_val_recall': 0.6496536029095196, 'eval_val_f1': 0.6690684445718375, 'eval_runtime': 14.0463, 'eval_samples_per_second': 109.495, 'eval_steps_per_second': 6.906, 'epoch': 2.43}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "424efe3b057b4e73b51b5140010ff193",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.5143762230873108, 'eval_val_accuracy': 0.8372867803837953, 'eval_val_precision': 0.7178423603785965, 'eval_val_recall': 0.65897797966175, 'eval_val_f1': 0.6771576236427079, 'eval_runtime': 14.1239, 'eval_samples_per_second': 108.893, 'eval_steps_per_second': 6.868, 'epoch': 2.73}\n",
      "{'loss': 0.5537, 'grad_norm': 2.444669485092163, 'learning_rate': 7.855708713577838e-06, 'epoch': 3.03}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2d2f8c65ccb84227b4ee39809bd595f1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.4837438464164734, 'eval_val_accuracy': 0.8466595593461265, 'eval_val_precision': 0.7208762150020039, 'eval_val_recall': 0.669979925591904, 'eval_val_f1': 0.6879526239751579, 'eval_runtime': 13.9154, 'eval_samples_per_second': 110.525, 'eval_steps_per_second': 6.971, 'epoch': 3.03}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "97bce593ed684b0bb9072cb2e6bc0d72",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.47861653566360474, 'eval_val_accuracy': 0.8509683724235964, 'eval_val_precision': 0.7218039750558128, 'eval_val_recall': 0.6689517866516295, 'eval_val_f1': 0.6866750947467928, 'eval_runtime': 14.0875, 'eval_samples_per_second': 109.175, 'eval_steps_per_second': 6.886, 'epoch': 3.34}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1290a5a29e1b454a9ed192a2b4a264a0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.4585321247577667, 'eval_val_accuracy': 0.8542999289267946, 'eval_val_precision': 0.7180672439097245, 'eval_val_recall': 0.6798747241043808, 'eval_val_f1': 0.6936482932186963, 'eval_runtime': 14.0135, 'eval_samples_per_second': 109.751, 'eval_steps_per_second': 6.922, 'epoch': 3.64}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f746b350c0ec4b0daf8db2d98fa9a98e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.4455214738845825, 'eval_val_accuracy': 0.8605632551528074, 'eval_val_precision': 0.7286963557627002, 'eval_val_recall': 0.6857907352925633, 'eval_val_f1': 0.7016863589465753, 'eval_runtime': 14.0662, 'eval_samples_per_second': 109.34, 'eval_steps_per_second': 6.896, 'epoch': 3.95}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5594aff0b4c04465a709fd09a5908ea9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.44131192564964294, 'eval_val_accuracy': 0.8627398720682303, 'eval_val_precision': 0.7271117822881257, 'eval_val_recall': 0.684818367608053, 'eval_val_f1': 0.7005253112577672, 'eval_runtime': 13.9794, 'eval_samples_per_second': 110.019, 'eval_steps_per_second': 6.939, 'epoch': 4.25}\n",
      "{'loss': 0.4278, 'grad_norm': 2.9219138622283936, 'learning_rate': 5.448935210643452e-06, 'epoch': 4.55}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "998ff1b504ec4cc881b4c970db370519",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.4287553131580353, 'eval_val_accuracy': 0.8653162757640369, 'eval_val_precision': 0.7229939082898612, 'eval_val_recall': 0.6955720790420472, 'eval_val_f1': 0.7061925122715633, 'eval_runtime': 13.8949, 'eval_samples_per_second': 110.688, 'eval_steps_per_second': 6.981, 'epoch': 4.55}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d4f551894c1443a3986198f53d84b1c9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.42695847153663635, 'eval_val_accuracy': 0.8670931058990761, 'eval_val_precision': 0.7277297634925718, 'eval_val_recall': 0.6949310254359338, 'eval_val_f1': 0.7074799785625216, 'eval_runtime': 14.067, 'eval_samples_per_second': 109.334, 'eval_steps_per_second': 6.896, 'epoch': 4.86}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "cb13392762ba446fb9a6dd469f7151e8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.42116397619247437, 'eval_val_accuracy': 0.8697139303482587, 'eval_val_precision': 0.7294879433028028, 'eval_val_recall': 0.6933493488852324, 'eval_val_f1': 0.7078786687498849, 'eval_runtime': 14.1876, 'eval_samples_per_second': 108.405, 'eval_steps_per_second': 6.837, 'epoch': 5.16}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e28adea984254546a50a5ae5dae1fa58",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.41709104180336, 'eval_val_accuracy': 0.8722903340440654, 'eval_val_precision': 0.7335901399882897, 'eval_val_recall': 0.693471896304754, 'eval_val_f1': 0.7090808249401142, 'eval_runtime': 14.1847, 'eval_samples_per_second': 108.427, 'eval_steps_per_second': 6.838, 'epoch': 5.46}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d0f77ff6b4a34c66822c3e0e448af9ea",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.4085210859775543, 'eval_val_accuracy': 0.8727345415778252, 'eval_val_precision': 0.7312261620673794, 'eval_val_recall': 0.6999662616433541, 'eval_val_f1': 0.7126223686683726, 'eval_runtime': 14.0534, 'eval_samples_per_second': 109.44, 'eval_steps_per_second': 6.902, 'epoch': 5.77}\n",
      "{'loss': 0.3678, 'grad_norm': 2.6123743057250977, 'learning_rate': 3.042161707709065e-06, 'epoch': 6.07}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "48803ec5397443d0be931617e71fddc5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.4102620482444763, 'eval_val_accuracy': 0.8743336886993603, 'eval_val_precision': 0.733041546349626, 'eval_val_recall': 0.6987644243182074, 'eval_val_f1': 0.7119976592694294, 'eval_runtime': 13.9299, 'eval_samples_per_second': 110.41, 'eval_steps_per_second': 6.963, 'epoch': 6.07}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "29968e7d9e8b43d7bfa6ad41d939578e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.40693002939224243, 'eval_val_accuracy': 0.8754886282871357, 'eval_val_precision': 0.7341164426567997, 'eval_val_recall': 0.6993743818676211, 'eval_val_f1': 0.7132729378066435, 'eval_runtime': 14.0775, 'eval_samples_per_second': 109.252, 'eval_steps_per_second': 6.89, 'epoch': 6.37}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2f2fdd6e1b8c4881982b7882123f2caa",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.40312904119491577, 'eval_val_accuracy': 0.8757995735607675, 'eval_val_precision': 0.7319248957432298, 'eval_val_recall': 0.7001523738902371, 'eval_val_f1': 0.7126511632002726, 'eval_runtime': 14.1743, 'eval_samples_per_second': 108.506, 'eval_steps_per_second': 6.843, 'epoch': 6.68}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7a7fc8c92def4947a8a2ba7cad21fe38",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.40199118852615356, 'eval_val_accuracy': 0.8765547263681592, 'eval_val_precision': 0.7341805891005609, 'eval_val_recall': 0.7007268639820944, 'eval_val_f1': 0.7140716416090435, 'eval_runtime': 14.1622, 'eval_samples_per_second': 108.599, 'eval_steps_per_second': 6.849, 'epoch': 6.98}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d5cb137f06214f1090182e5c3087359c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.39898672699928284, 'eval_val_accuracy': 0.8768212508884151, 'eval_val_precision': 0.7315535001211422, 'eval_val_recall': 0.7034095715509543, 'eval_val_f1': 0.714848939494529, 'eval_runtime': 14.1481, 'eval_samples_per_second': 108.707, 'eval_steps_per_second': 6.856, 'epoch': 7.28}\n",
      "{'loss': 0.3351, 'grad_norm': 2.4046945571899414, 'learning_rate': 6.353882047746781e-07, 'epoch': 7.59}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dab93763da7541ae8346220c7e8cc86f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.39517247676849365, 'eval_val_accuracy': 0.8773987206823027, 'eval_val_precision': 0.7343106248130895, 'eval_val_recall': 0.7022618297004076, 'eval_val_f1': 0.7152764324949239, 'eval_runtime': 13.9521, 'eval_samples_per_second': 110.234, 'eval_steps_per_second': 6.952, 'epoch': 7.59}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "426ee74cc41a4aafb4de02c9e8241e34",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/97 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.3955319821834564, 'eval_val_accuracy': 0.8781094527363185, 'eval_val_precision': 0.7330961088044301, 'eval_val_recall': 0.7031560382114157, 'eval_val_f1': 0.7152192730995107, 'eval_runtime': 14.0168, 'eval_samples_per_second': 109.726, 'eval_steps_per_second': 6.92, 'epoch': 7.89}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[I 2024-03-11 20:32:35,747] Trial 1 finished with value: 3.029580872851675 and parameters: {'learning_rate': 1.1571767002108532e-05, 'weight_decay': 0.03446941200999968, 'warmup_steps': 228}. Best is trial 0 with value: 3.0666799221173124.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'train_runtime': 2920.7783, 'train_samples_per_second': 28.866, 'train_steps_per_second': 0.901, 'train_loss': 0.683700506448021, 'epoch': 7.99}\n"
     ]
    }
   ],
   "source": [
    "def get_search_space(trial):\n",
    "    return {\n",
    "        \"learning_rate\": trial.suggest_float(\"learning_rate\", 1e-5, 1e-3, log=True),\n",
    "        \"weight_decay\": trial.suggest_float(\"weight_decay\", 1e-5, 0.1, log=True),\n",
    "        \"warmup_steps\": trial.suggest_int(\"warmup_steps\", 100, 500),\n",
    "    }\n",
    "\n",
    "\n",
    "# remember to change this to best run directory if get_search_space changes\n",
    "# I need to do this manually since the transformer devs apparently gave up on\n",
    "# doing this automatically  \n",
    "# https://discuss.huggingface.co/t/oserror-unable-to-load-weights-from-pytorch-checkpoint-file/3406\n",
    "model_dir = \"output/run-0/checkpoint-2500\" \n",
    "\n",
    "if not os.path.exists(model_dir):\n",
    "    best_run = trainer.hyperparameter_search(\n",
    "        direction=\"maximize\", backend=\"optuna\", n_trials=2, hp_space=get_search_space\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "finetuned_model = transformers.AutoModelForTokenClassification.from_pretrained(\n",
    "    model_dir,\n",
    "    local_files_only=True,\n",
    ").to(\"cuda\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Aggregated results\n",
    "\n",
    "The fine-tuned BERT model seems to *out-perform all baselines* except from the MLP baseline by a very narrow margin, which aligns with our expectations. We hypothesize that with more resources, enabling further hyper-parameter tuning and more datasets, this model could outperform the MLP classifier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Label-wise results\n",
    "\n",
    "We can adapt the provided get_prediction function in order to compare the true and predicted POS tags for each sentence in our dataset. \n",
    "\n",
    "However, a significant issue becomes evident once we attempt to apply this to our data; our data are tokenized twice by the tokenizer (as explained more in-depth in the class forum). While post-processing allows some sentences to successfully get parsed into aligned POS tags, any sentences including tokens such as apostrophes force the algorithm to fail. This seems to be a restriction of the tokenizer API which we can not solve at the current moment in time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_prediction(model, text):\n",
    "    # prepare our text into tokenized sequence\n",
    "    inputs = bert_tokenizer(\n",
    "        text,\n",
    "        is_split_into_words=True,\n",
    "        truncation=True,\n",
    "        padding=\"max_length\",\n",
    "        max_length=MAX_SEQUENCE_LENGTH,\n",
    "        return_tensors=\"pt\",\n",
    "    ).to(\"cuda\")\n",
    "    # perform inference to our model\n",
    "    outputs = model(**inputs)\n",
    "    # get output probabilities by doing softmax\n",
    "    probs = outputs[0][0].softmax(1)\n",
    "    # executing argmax function to get the candidate tags\n",
    "    tokens_n_tags = [\n",
    "        (\n",
    "            bert_tokenizer.decode(inputs[\"input_ids\"][0][i].item()),\n",
    "            tag_tokenizer.index_word[str(tag_id.item())],\n",
    "        )\n",
    "        for i, tag_id in enumerate(probs.argmax(axis=1))\n",
    "    ]\n",
    "    # remove CLS and SEP tokens because skip_special_tokens doesnt work\n",
    "    tokens_n_tags = [\n",
    "        x for x in tokens_n_tags if x[0] not in [\"[CLS]\", \"[SEP]\", \"[PAD]\"]\n",
    "    ]\n",
    "    print(tokens_n_tags)\n",
    "    return pd.DataFrame(tokens_n_tags, columns=[\"token\", \"y_pred\"])\n",
    "\n",
    "\n",
    "def get_pred_true(model, text, labels):\n",
    "    try:\n",
    "        df = get_prediction(model, text)\n",
    "        df[\"y_true\"] = [\n",
    "            tag_tokenizer.index_word[str(x)] for x in labels if x != -100\n",
    "        ]\n",
    "    except ValueError:\n",
    "        #print(\n",
    "        #    \"**************\\n\",\n",
    "        #    df,\n",
    "        #    [tag_tokenizer.index_word[str(x)] for x in labels if x != -100],\n",
    "        #    sep=\"\\n\",\n",
    "        #)\n",
    "        return pd.DataFrame()\n",
    "    return df\n",
    "\n",
    "\n",
    "#get_pred_true(\n",
    "#    finetuned_model,\n",
    "#    tokenized_data_dict[\"train\"][\"words\"][:500],\n",
    "#    tokenized_data_dict[\"train\"][\"labels\"][:500],\n",
    "#)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.concat(\n",
    "    [\n",
    "        get_pred_true(\n",
    "            finetuned_model,\n",
    "            data_dict[\"train\"][\"words\"][i],\n",
    "            data_dict[\"train\"][\"labels\"][i],\n",
    "        )\n",
    "        for i in tqdm(range(40))\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the method worked correctly, we could feed the aligned true and prediction POS tags as seen above to the tasks.util.stats_by_label function which would automatically generate label-wise accuracy, precision, recall, f1 and PR-AUC scores."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "\n",
    "tasks.util.stats_by_label(\n",
    "    y_true_train=tag_tokenizer.sequences_to_text(\n",
    "        tokenized_data_dict[\"train\"][\"labels\"]\n",
    "    ),\n",
    "    y_true_valid=tokenized_data_dict[\"val\"][\"labels\"],\n",
    "    y_true_test=tokenized_data_dict[\"test\"][\"labels\"],\n",
    "    y_true_train=tokenized_data_dict[\"train\"][\"labels\"],\n",
    "    y_true_train=tokenized_data_dict[\"train\"][\"labels\"],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ChatGPT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the final POS tagging task, we will be using ChatGPT to automatically produce POS tags from our dataset. The input and output is inserted/extracted manually using the ChatGPT web API.\n",
    "\n",
    "We utilize a few-shot (demonstrator) prompting approach with a role prompt and detailed instructions and definitions. The model is prompted to answer via standard Input: Output: indicators. The prompt used and a breakdown of its strucuture can be found in the image below:\n",
    "\n",
    "![the prompt used for the LLM POS tagging task](images/prompt.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We thus produce the model input:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input:  as far as george bush knew , he and his unit could have been transferred to vietnam .\n",
      "Output: adv adv sconj propn propn verb punct pron cconj pron noun aux aux aux verb adp propn punct\n",
      "Input:  are you unsure of agel ?\n",
      "Output: aux pron adj adp propn punct\n",
      "Input:  jan smutney - jones , who was the iso board chairman at the time and executive director of a group that represents power generators , said winter did not consult him about eliminating the price cap .\n",
      "Output: propn propn punct propn punct pron aux det propn noun noun adp det noun cconj adj noun adp det noun pron verb noun noun punct verb propn aux part verb\n",
      "Input:  transwestern will own and operate the interconnect .\n",
      "Output: propn aux verb cconj verb det noun punct\n",
      "Input:  now that michelle is in your fold , do you want to reconsider her being the contact person with aaa on this ?\n",
      "Output: adv sconj propn aux adp pron noun punct aux pron verb part verb pron aux det noun noun adp propn adp pron punct\n",
      "Input:  do this repeatedly and frequently and he will learn to walk on a leash .\n",
      "Output: verb pron adv cconj adv cconj pron aux verb part verb adp det noun punct\n"
     ]
    }
   ],
   "source": [
    "llm_train_sentences = [\n",
    "    \" \".join(tokenized_data_dict[\"train\"][\"words\"][i]) for i in range(5, 11)\n",
    "]\n",
    "llm_train_labels = [\n",
    "    [x for x in tokenized_data_dict[\"train\"][\"labels\"][i] if x!= -100]\n",
    "    for i in range(5, 11)\n",
    "]\n",
    "\n",
    "for sentence, pos in zip(llm_train_sentences, llm_train_labels):\n",
    "    print(\"Input: \", sentence)\n",
    "    print(\"Output:\", \" \".join([tag_tokenizer.index_word[str(x)] for x in pos]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input:  as far as george bush knew , he and his unit could have been transferred to vietnam .\n",
      "Input:  are you unsure of agel ?\n",
      "Input:  jan smutney - jones , who was the iso board chairman at the time and executive director of a group that represents power generators , said winter did not consult him about eliminating the price cap .\n",
      "Input:  transwestern will own and operate the interconnect .\n",
      "Input:  now that michelle is in your fold , do you want to reconsider her being the contact person with aaa on this ?\n",
      "Input:  do this repeatedly and frequently and he will learn to walk on a leash .\n"
     ]
    }
   ],
   "source": [
    "llm_test_sentences = [\n",
    "    \" \".join(tokenized_data_dict[\"test\"][\"words\"][i]) for i in range(5, 11)\n",
    "]\n",
    "llm_test_labels = [\n",
    "    [x for x in tokenized_data_dict[\"test\"][\"labels\"][i] if x!= -100]\n",
    "    for i in range(5, 11)\n",
    "]\n",
    "\n",
    "expected_outputs = []\n",
    "for sentence, pos in zip(llm_train_sentences, llm_train_labels):\n",
    "    print(\"Input: \", sentence)\n",
    "    expected_outputs.append([tag_tokenizer.index_word[str(x)] for x in pos])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And parse its output:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "gpt_responses = \"\"\"propn verb verb adp pron aux det verb adp pron noun punct\n",
    "pron verb pron aux adp det adj noun punct adv punct pron verb det noun punct conj adv punct pron verb det noun adp det noun punct\n",
    "adv verb adj num punct\n",
    "pron verb det noun adj punct num punct punct propn sym num noun punct punct punct propn propn num adj punct punct punct punct adv verb pron verb pron part verb pron adp verb punct\n",
    "verb verb verb det propn propn num punct punct punct propn propn num num adj punct punct adv verb pron verb det propn propn num num num punct part verb pron adp propn punct\n",
    "det noun verb adv adj punct\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_true_llm = expected_outputs\n",
    "y_pred_llm = [sentence.split(\" \") for sentence in gpt_responses.split(\"\\n\")]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "import itertools\n",
    "\n",
    "\n",
    "lb = sklearn.preprocessing.LabelBinarizer()\n",
    "y_true_llm_cat = lb.fit_transform(np.array(list(itertools.chain.from_iterable(y_true_llm))))\n",
    "y_pred_llm_cat = lb.transform(np.array(list(itertools.chain.from_iterable(y_pred_llm))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unfortunately, our approach suffers from the same issue encountered in the Transformers Result section. Since we can not reliably tokenize the words into sub-token strings, the LLM produces misaligned POS labels."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "sklearn.metrics.accuracy_score(y_true_llm_cat, y_pred_llm_cat)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "tf",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}