Token Classification/Monolingual/WikiNeural - Transformer Comparison/POS Project with Wikineural Dataset - SqueezeBERT Transformer.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Monolingual Named Entity Recognition Token Classification (WikiNeural Dataset)\n",
    "\n",
    "### SqueezeBERT\n",
    "\n",
    "Dataset Source: https://huggingface.co/datasets/Babelscape/wikineural"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Import Necessary Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Git LFS initialized.\n"
     ]
    }
   ],
   "source": [
    "import os, sys\n",
    "os.environ['TOKENIZERS_PARALLELISM']='false'\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import sklearn\n",
    "from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay\n",
    "\n",
    "import datasets\n",
    "from datasets import load_dataset, DatasetDict\n",
    "\n",
    "import torch\n",
    "from torch import nn\n",
    "from torch.nn.functional import cross_entropy\n",
    "\n",
    "import transformers\n",
    "from transformers import AutoTokenizer, AutoModelForTokenClassification\n",
    "from transformers import DataCollatorForTokenClassification\n",
    "from transformers import Trainer, TrainingArguments\n",
    "\n",
    "import evaluate\n",
    "\n",
    "!git lfs install"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Display Library Versions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "          Python : 3.9.12\n",
      "           NumPy : 1.24.3\n",
      "           Torch : 2.0.0\n",
      "          Pandas : 2.0.1\n",
      "    Transformers : 4.28.1\n",
      "        datasets : 2.11.0\n",
      "         Sklearn : 1.2.2\n"
     ]
    }
   ],
   "source": [
    "print(\"Python :\".rjust(18), sys.version[0:6])\n",
    "print(\"NumPy :\".rjust(18), np.__version__)\n",
    "print(\"Torch :\".rjust(18), torch.__version__)\n",
    "print(\"Pandas :\".rjust(18), pd.__version__)\n",
    "print(\"Transformers :\".rjust(18), transformers.__version__)\n",
    "print(\"datasets :\".rjust(18), datasets.__version__)\n",
    "print(\"Sklearn :\".rjust(18), sklearn.__version__)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Ingest English Subset of Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Found cached dataset parquet (/Users/briandunn/.cache/huggingface/datasets/Babelscape___parquet/Babelscape--wikineural-579d1dc98d2a6b93/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c802d1a8060946659de6e0a15a0c972f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/27 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    train_es: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 76320\n",
       "    })\n",
       "    test_es: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 9618\n",
       "    })\n",
       "    test_de: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 12372\n",
       "    })\n",
       "    val_it: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 11050\n",
       "    })\n",
       "    train_en: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 92720\n",
       "    })\n",
       "    train_it: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 88400\n",
       "    })\n",
       "    train_nl: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 83680\n",
       "    })\n",
       "    train_pl: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 108160\n",
       "    })\n",
       "    test_pl: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 13585\n",
       "    })\n",
       "    val_ru: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 11540\n",
       "    })\n",
       "    test_nl: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 10547\n",
       "    })\n",
       "    train_fr: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 100800\n",
       "    })\n",
       "    val_de: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 12330\n",
       "    })\n",
       "    test_it: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 11069\n",
       "    })\n",
       "    test_fr: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 12678\n",
       "    })\n",
       "    val_en: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 11590\n",
       "    })\n",
       "    val_nl: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 10460\n",
       "    })\n",
       "    test_pt: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 10160\n",
       "    })\n",
       "    val_fr: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 12600\n",
       "    })\n",
       "    test_en: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 11597\n",
       "    })\n",
       "    train_pt: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 80560\n",
       "    })\n",
       "    test_ru: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 11580\n",
       "    })\n",
       "    val_pl: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 13520\n",
       "    })\n",
       "    val_pt: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 10070\n",
       "    })\n",
       "    train_de: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 98640\n",
       "    })\n",
       "    val_es: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 9540\n",
       "    })\n",
       "    train_ru: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 92320\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = load_dataset(\"Babelscape/wikineural\")\n",
    "\n",
    "data = data.remove_columns([\"lang\"])\n",
    "\n",
    "data"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Split Dataset into DatasetDict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training data shape: (92720, 2)\n",
      "Testing data shape: (11597, 2)\n",
      "Validation data shape: (11590, 2)\n"
     ]
    }
   ],
   "source": [
    "ds = DatasetDict({\n",
    "    'train': data['train_en'], \n",
    "    'test': data['test_en'], \n",
    "    'eval': data['val_en']})\n",
    "\n",
    "print('Training data shape:', ds['train'].shape)\n",
    "print('Testing data shape:', ds['test'].shape)\n",
    "print('Validation data shape:', ds['eval'].shape)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'tokens': ['It',\n",
       "  'has',\n",
       "  'a',\n",
       "  'leaf',\n",
       "  'pattern',\n",
       "  'similar',\n",
       "  'to',\n",
       "  'the',\n",
       "  'members',\n",
       "  'of',\n",
       "  'the',\n",
       "  'genera',\n",
       "  '\"',\n",
       "  'Kedrostis',\n",
       "  '\"',\n",
       "  ',',\n",
       "  '\"',\n",
       "  'Melothria',\n",
       "  '\"',\n",
       "  'and',\n",
       "  '\"',\n",
       "  'Zehneria',\n",
       "  '\"',\n",
       "  '.'],\n",
       " 'ner_tags': [0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  7,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  7,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  7,\n",
       "  0,\n",
       "  0]}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example = ds['train'][12]\n",
    "\n",
    "example"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Display Feature Information About Each Feature"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tokens: \n",
      "Sequence(feature=Value(dtype='string', id=None), length=-1, id=None)\n",
      "\n",
      "ner_tags: \n",
      "Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for k, v in ds[\"train\"].features.items():\n",
    "    print(f\"{k}: \\n{v}\\n\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define Tag Values & Conversions Between String & Integer Values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "List of tag values: \n",
      "['O', 'B-PER', 'I-PER', 'B-ORG', 'I-ORG', 'B-LOC', 'I-LOC', 'B-MISC', 'I-MISC']\n",
      "Number of NER Tags: \n",
      "9\n",
      "id2label: \n",
      "{0: 'O', 1: 'B-PER', 2: 'I-PER', 3: 'B-ORG', 4: 'I-ORG', 5: 'B-LOC', 6: 'I-LOC', 7: 'B-MISC', 8: 'I-MISC'}\n",
      "label2id: \n",
      "{'O': 0, 'B-PER': 1, 'I-PER': 2, 'B-ORG': 3, 'I-ORG': 4, 'B-LOC': 5, 'I-LOC': 6, 'B-MISC': 7, 'I-MISC': 8}\n"
     ]
    }
   ],
   "source": [
    "label2id = {\n",
    "    'O': 0, \n",
    "    'B-PER': 1, \n",
    "    'I-PER': 2, \n",
    "    'B-ORG': 3, \n",
    "    'I-ORG': 4, \n",
    "    'B-LOC': 5, \n",
    "    'I-LOC': 6, \n",
    "    'B-MISC': 7, \n",
    "    'I-MISC': 8\n",
    "    }\n",
    "\n",
    "id2label = {tag: idx for idx, tag in label2id.items()}\n",
    "\n",
    "pos_tag_values = list(label2id.keys())\n",
    "NUM_OF_LABELS = len(pos_tag_values)\n",
    "\n",
    "print(f\"List of tag values: \\n{pos_tag_values}\")\n",
    "print(f\"Number of NER Tags: \\n{NUM_OF_LABELS}\")\n",
    "print(f\"id2label: \\n{id2label}\")\n",
    "print(f\"label2id: \\n{label2id}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Basic Values/Constants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "MODEL_CKPT = \"squeezebert/squeezebert-uncased\"\n",
    "MODEL_NAME = f\"{MODEL_CKPT.split(f'/')[-1]}-finetuned-WikiCorpus-PoS\"\n",
    "\n",
    "NUM_OF_EPOCHS = 2\n",
    "BATCH_SIZE = 16\n",
    "\n",
    "STRATEGY = \"epoch\"\n",
    "REPORTS_TO = \"tensorboard\"\n",
    "\n",
    "WEIGHT_DECAY = 0.01\n",
    "LR = 2e-5\n",
    "\n",
    "DEVICE = torch.device(\"cpu\")\n",
    "STEPS = 1250"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Function to Tokenize & Align Inputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = AutoTokenizer.from_pretrained(MODEL_CKPT)\n",
    "\n",
    "def tokenize_and_align_labels(samples):\n",
    "    tokenized_inputs = tokenizer(samples[\"tokens\"], \n",
    "                                      truncation=True, \n",
    "                                      is_split_into_words=True)\n",
    "    \n",
    "    labels = []\n",
    "    \n",
    "    for idx, label in enumerate(samples[f\"ner_tags\"]):\n",
    "        word_ids = tokenized_inputs.word_ids(batch_index=idx)\n",
    "        prev_word_idx = None\n",
    "        label_ids = []\n",
    "        for word_idx in word_ids: # set special tokens to -100\n",
    "            if word_idx is None or word_idx == prev_word_idx:\n",
    "                label_ids.append(-100)\n",
    "            else:\n",
    "                label_ids.append(label[word_idx])\n",
    "            prev_word_idx = word_idx\n",
    "        labels.append(label_ids)\n",
    "    \n",
    "    tokenized_inputs[\"labels\"] = labels\n",
    "    return tokenized_inputs"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Apply Above Function to Tokenize Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "01c73bafb63f4fff972f635299bd6f00",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/92720 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "509f3b83f5e54997a6cd09dacec79bb1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/11597 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "36a49a58221b43918b42261552649026",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/11590 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "encoded_ds = ds.map(tokenize_and_align_labels, \n",
    "                    batched=True, \n",
    "                    remove_columns=\n",
    "                        [\n",
    "                            'ner_tags', \n",
    "                            'tokens'\n",
    "                        ]\n",
    "                    )"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define Data Collator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_collator = DataCollatorForTokenClassification(tokenizer=tokenizer)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load Custom Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at squeezebert/squeezebert-uncased were not used when initializing SqueezeBertForTokenClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.seq_relationship.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight']\n",
      "- This IS expected if you are initializing SqueezeBertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing SqueezeBertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of SqueezeBertForTokenClassification were not initialized from the model checkpoint at squeezebert/squeezebert-uncased and are newly initialized: ['classifier.bias', 'classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "model = (AutoModelForTokenClassification.from_pretrained(\n",
    "    MODEL_CKPT,\n",
    "    num_labels=NUM_OF_LABELS,\n",
    "    id2label=id2label,\n",
    "    label2id=label2id\n",
    "    ).to(DEVICE))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create Compute Metrics Function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "label_list = pos_tag_values\n",
    "\n",
    "seqeval = evaluate.load(\"seqeval\")\n",
    "\n",
    "labels = [label_list[i] for i in example[f'ner_tags']]\n",
    "\n",
    "def compute_metrics(eval_preds):\n",
    "    predictions, labels = eval_preds\n",
    "    predictions = np.argmax(predictions, \n",
    "                            axis=2)\n",
    "    \n",
    "    true_predictions = [\n",
    "        [label_list[p] for (p, l) in zip(prediction, label) if l != -100]\n",
    "        for prediction, label in zip(predictions, labels)\n",
    "    ]\n",
    "    \n",
    "    true_labels = [\n",
    "        [label_list[l] for (p, l) in zip(prediction, label) if l != -100]\n",
    "        for prediction, label in zip(predictions, labels)\n",
    "    ]\n",
    "    \n",
    "    results = seqeval.compute(predictions=true_predictions, \n",
    "                              references=true_labels)\n",
    "    \n",
    "    return results"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define TrainingArguments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "args = TrainingArguments(\n",
    "    MODEL_NAME,\n",
    "    log_level=\"error\",\n",
    "    logging_first_step=True,\n",
    "    learning_rate=LR,\n",
    "    num_train_epochs=NUM_OF_EPOCHS,\n",
    "    per_device_train_batch_size=BATCH_SIZE,\n",
    "    per_device_eval_batch_size=BATCH_SIZE,\n",
    "    evaluation_strategy=STRATEGY,\n",
    "    report_to=REPORTS_TO,\n",
    "    disable_tqdm=False,\n",
    "    logging_steps=STEPS,\n",
    "    weight_decay=WEIGHT_DECAY,\n",
    "    save_strategy=STRATEGY,\n",
    "    hub_private_repo=True,\n",
    "    push_to_hub=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Subclass Trainer to Handle Class Imbalance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CustomTrainer(Trainer):\n",
    "    def compute_loss(self, \n",
    "                     model, \n",
    "                     inputs, \n",
    "                     return_outputs=False):\n",
    "        \n",
    "        labels = inputs.get(\"labels\")\n",
    "        # forward pass\n",
    "        outputs = model(**inputs)\n",
    "        logits = outputs.get(\"logits\")\n",
    "        # compute custom loss \n",
    "        loss_fct = nn.CrossEntropyLoss(weight=torch.tensor(\n",
    "            [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0], \n",
    "            device=model.device)\n",
    "        )\n",
    "        loss = loss_fct(logits.view(-1, \n",
    "                                    self.model.config.num_labels), \n",
    "                        labels.view(-1)\n",
    "                        )\n",
    "        return (loss, outputs) if return_outputs else loss"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define Trainer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Cloning https://huggingface.co/DunnBC22/squeezebert-uncased-finetuned-WikiCorpus-PoS into local empty directory.\n"
     ]
    }
   ],
   "source": [
    "trainer = CustomTrainer(model, \n",
    "                  args=args,\n",
    "                  data_collator=data_collator,\n",
    "                  compute_metrics=compute_metrics,\n",
    "                  tokenizer=tokenizer,\n",
    "                  train_dataset=encoded_ds[\"train\"],\n",
    "                  eval_dataset=encoded_ds[\"test\"]\n",
    "                  )"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Train Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/briandunn/Documents/deep_learning/dl/lib/python3.9/site-packages/transformers/optimization.py:391: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c44f476afa764685b7df0a669c1d6150",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/11590 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 2.182, 'learning_rate': 1.9998274374460744e-05, 'epoch': 0.0}\n",
      "{'loss': 0.6442, 'learning_rate': 1.7842968075927526e-05, 'epoch': 0.22}\n",
      "{'loss': 0.2905, 'learning_rate': 1.5685936151855047e-05, 'epoch': 0.43}\n",
      "{'loss': 0.2253, 'learning_rate': 1.3528904227782573e-05, 'epoch': 0.65}\n",
      "{'loss': 0.1983, 'learning_rate': 1.1371872303710096e-05, 'epoch': 0.86}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e5a3c92115644702b76846d671027c8e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/725 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.16816049814224243, 'eval_LOC': {'precision': 0.8308694961055476, 'recall': 0.8777497900923593, 'f1': 0.8536665033480321, 'number': 5955}, 'eval_MISC': {'precision': 0.6943174395819726, 'recall': 0.8401501679509978, 'f1': 0.760303978542691, 'number': 5061}, 'eval_ORG': {'precision': 0.8382269904009034, 'recall': 0.8608292258625688, 'f1': 0.8493777714203976, 'number': 3449}, 'eval_PER': {'precision': 0.9495403872481909, 'recall': 0.9318618042226487, 'f1': 0.9406180373922309, 'number': 5210}, 'eval_overall_precision': 0.8212149976269577, 'eval_overall_recall': 0.8794409148665819, 'eval_overall_f1': 0.8493312062829794, 'eval_overall_accuracy': 0.9795287379985401, 'eval_runtime': 502.7178, 'eval_samples_per_second': 23.069, 'eval_steps_per_second': 1.442, 'epoch': 1.0}\n",
      "{'loss': 0.18, 'learning_rate': 9.214840379637619e-06, 'epoch': 1.08}\n",
      "{'loss': 0.1569, 'learning_rate': 7.057808455565142e-06, 'epoch': 1.29}\n",
      "{'loss': 0.1517, 'learning_rate': 4.900776531492667e-06, 'epoch': 1.51}\n",
      "{'loss': 0.149, 'learning_rate': 2.74374460742019e-06, 'epoch': 1.73}\n",
      "{'loss': 0.1467, 'learning_rate': 5.867126833477136e-07, 'epoch': 1.94}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "372d686d99f749a0a7132632bd4a6667",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/725 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.15034307539463043, 'eval_LOC': {'precision': 0.8425851863701808, 'recall': 0.8844668345927792, 'f1': 0.8630181877765034, 'number': 5955}, 'eval_MISC': {'precision': 0.6964743589743589, 'recall': 0.8587235724165184, 'f1': 0.7691354747367489, 'number': 5061}, 'eval_ORG': {'precision': 0.8521814006888634, 'recall': 0.8608292258625688, 'f1': 0.8564834847829222, 'number': 3449}, 'eval_PER': {'precision': 0.9536982538748283, 'recall': 0.9330134357005758, 'f1': 0.9432424565829048, 'number': 5210}, 'eval_overall_precision': 0.8277809415337889, 'eval_overall_recall': 0.886556543837357, 'eval_overall_f1': 0.8561611897808428, 'eval_overall_accuracy': 0.9802811102169152, 'eval_runtime': 943.3302, 'eval_samples_per_second': 12.294, 'eval_steps_per_second': 0.769, 'epoch': 2.0}\n",
      "{'train_runtime': 30369.8159, 'train_samples_per_second': 6.106, 'train_steps_per_second': 0.382, 'train_loss': 0.23565051170954734, 'epoch': 2.0}\n"
     ]
    }
   ],
   "source": [
    "train_results = trainer.train()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Push Model To Hub"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Several commits (2) will be pushed upstream.\n",
      "The progress bars may be unreliable.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f93b44b5a7e0400abd357df260aaa6a6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Upload file pytorch_model.bin:   0%|          | 1.00/195M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7c79a4fd5c4545dab5063cd0a69344df",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Upload file runs/Jul10_17-53-18_Brians-Mac-mini/events.out.tfevents.1689029603.Brians-Mac-mini.4584.0:   0%|  …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "To https://huggingface.co/DunnBC22/squeezebert-uncased-finetuned-WikiCorpus-PoS\n",
      "   4dbf63e..60c0a84  main -> main\n",
      "\n",
      "To https://huggingface.co/DunnBC22/squeezebert-uncased-finetuned-WikiCorpus-PoS\n",
      "   60c0a84..76914cf  main -> main\n",
      "\n"
     ]
    }
   ],
   "source": [
    "kwargs = {\n",
    "    \"tasks\": \"token-classification\",\n",
    "    \"tags\": ['token-classification'],\n",
    "}\n",
    "\n",
    "if args.push_to_hub:\n",
    "    trainer.push_to_hub('All DUNN!!!', **kwargs)\n",
    "else:\n",
    "    trainer.create_model_card(**kwargs)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Save & Log Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "To https://huggingface.co/DunnBC22/squeezebert-uncased-finetuned-WikiCorpus-PoS\n",
      "   76914cf..703404d  main -> main\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "***** train metrics *****\n",
      "  epoch                    =        2.0\n",
      "  train_loss               =     0.2357\n",
      "  train_runtime            = 8:26:09.81\n",
      "  train_samples_per_second =      6.106\n",
      "  train_steps_per_second   =      0.382\n"
     ]
    }
   ],
   "source": [
    "trainer.save_model()\n",
    "trainer.log_metrics(\"train\", train_results.metrics)\n",
    "trainer.save_metrics(\"train\", train_results.metrics)\n",
    "trainer.save_state()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Error Analysis"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define Method to Apply to Validation Dataset (& Then Apply it)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def forward_pass_with_label(batch):\n",
    "    # Convert dict of lists to list of dicts suitable for data collator\n",
    "    features = [dict(zip(batch, t)) for t in zip(*batch.values())]\n",
    "    # Pad inputs and labels and put all tensors on device\n",
    "    batch = data_collator(features)\n",
    "    input_ids = batch[\"input_ids\"].to(DEVICE)\n",
    "    attention_mask = batch[\"attention_mask\"].to(DEVICE)\n",
    "    labels = batch[\"labels\"].to(DEVICE)\n",
    "    with torch.no_grad():\n",
    "        # Pass data through model  \n",
    "        output = trainer.model(input_ids, \n",
    "                               attention_mask\n",
    "                               )\n",
    "        # Logit.size: [batch_size, sequence_length, classes]\n",
    "        predicted_label = torch.argmax(output.logits, \n",
    "                                       axis=-1\n",
    "                                       ).cpu().numpy()\n",
    "        \n",
    "    # Calculate loss per token after flattening batch dimension with view\n",
    "    loss = cross_entropy(output.logits.view(-1, 9), \n",
    "                         labels.view(-1), \n",
    "                         reduction=\"none\")\n",
    "    # Unflatten batch dimension and convert to numpy array\n",
    "    loss = loss.view(len(input_ids), -1).cpu().numpy()\n",
    "\n",
    "    return {\"loss\":loss, \"predicted_label\": predicted_label}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Apply Above Function to Entire Evaluation Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Parameter 'function'=<function forward_pass_with_label at 0x7fc5c01aa0d0> of the transform datasets.arrow_dataset.Dataset._map_single couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.\n",
      "Loading cached processed dataset at /Users/briandunn/.cache/huggingface/datasets/Babelscape___parquet/Babelscape--wikineural-579d1dc98d2a6b93/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec/cache-1c80317fa3b1799d.arrow\n"
     ]
    }
   ],
   "source": [
    "eval_set = encoded_ds['eval']\n",
    "\n",
    "eval_set = eval_set.map(forward_pass_with_label,\n",
    "                        batched=True,\n",
    "                        batch_size=32)\n",
    "\n",
    "eval_df = eval_set.to_pandas()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Clean Up Padding Tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>input_ids</th>\n",
       "      <th>token_type_ids</th>\n",
       "      <th>attention_mask</th>\n",
       "      <th>labels</th>\n",
       "      <th>loss</th>\n",
       "      <th>predicted_label</th>\n",
       "      <th>input_tokens</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[101, 2014, 3942, 2000, 10722, 10175, 2226, 20...</td>\n",
       "      <td>[2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...</td>\n",
       "      <td>[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...</td>\n",
       "      <td>[IGN, O, O, O, B-LOC, IGN, IGN, O, O, O, O, O,...</td>\n",
       "      <td>[0.0, 5.5073175e-05, 5.4953973e-05, 0.00020072...</td>\n",
       "      <td>[O, O, O, O, B-LOC, I-LOC, I-LOC, O, O, O, O, ...</td>\n",
       "      <td>[[CLS], her, visit, to, tu, ##val, ##u, was, c...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           input_ids   \n",
       "0  [101, 2014, 3942, 2000, 10722, 10175, 2226, 20...  \\\n",
       "\n",
       "                                      token_type_ids   \n",
       "0  [2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...  \\\n",
       "\n",
       "                                      attention_mask   \n",
       "0  [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...  \\\n",
       "\n",
       "                                              labels   \n",
       "0  [IGN, O, O, O, B-LOC, IGN, IGN, O, O, O, O, O,...  \\\n",
       "\n",
       "                                                loss   \n",
       "0  [0.0, 5.5073175e-05, 5.4953973e-05, 0.00020072...  \\\n",
       "\n",
       "                                     predicted_label   \n",
       "0  [O, O, O, O, B-LOC, I-LOC, I-LOC, O, O, O, O, ...  \\\n",
       "\n",
       "                                        input_tokens  \n",
       "0  [[CLS], her, visit, to, tu, ##val, ##u, was, c...  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "id2label[-100] = \"IGN\"\n",
    "eval_df[\"input_tokens\"] = eval_df[\"input_ids\"].apply(\n",
    "    lambda x: tokenizer.convert_ids_to_tokens(x))\n",
    "eval_df[\"predicted_label\"] = eval_df[\"predicted_label\"].apply(\n",
    "    lambda x: [id2label[i] for i in x])\n",
    "eval_df[\"labels\"] = eval_df[\"labels\"].apply(\n",
    "    lambda x: [id2label[i] for i in x])\n",
    "eval_df['loss'] = eval_df.apply(\n",
    "    lambda x: x['loss'][:len(x['input_ids'])], axis=1)\n",
    "eval_df['predicted_label'] = eval_df.apply(\n",
    "    lambda x: x['predicted_label'][:len(x['input_ids'])], axis=1)\n",
    "eval_df.head(1)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Unwrap Each Token Within Sample Separately"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>input_ids</th>\n",
       "      <th>token_type_ids</th>\n",
       "      <th>attention_mask</th>\n",
       "      <th>labels</th>\n",
       "      <th>loss</th>\n",
       "      <th>predicted_label</th>\n",
       "      <th>input_tokens</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2014</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>O</td>\n",
       "      <td>0.0</td>\n",
       "      <td>O</td>\n",
       "      <td>her</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3942</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>O</td>\n",
       "      <td>0.0</td>\n",
       "      <td>O</td>\n",
       "      <td>visit</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2000</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>O</td>\n",
       "      <td>0.0</td>\n",
       "      <td>O</td>\n",
       "      <td>to</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>10722</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>0.0</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>tu</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2001</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>O</td>\n",
       "      <td>0.0</td>\n",
       "      <td>O</td>\n",
       "      <td>was</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3013</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>O</td>\n",
       "      <td>0.0</td>\n",
       "      <td>O</td>\n",
       "      <td>cut</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2460</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>O</td>\n",
       "      <td>0.0</td>\n",
       "      <td>O</td>\n",
       "      <td>short</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  input_ids token_type_ids attention_mask labels  loss predicted_label   \n",
       "0      2014              0              1      O   0.0               O  \\\n",
       "0      3942              0              1      O   0.0               O   \n",
       "0      2000              0              1      O   0.0               O   \n",
       "0     10722              0              1  B-LOC   0.0           B-LOC   \n",
       "0      2001              0              1      O   0.0               O   \n",
       "0      3013              0              1      O   0.0               O   \n",
       "0      2460              0              1      O   0.0               O   \n",
       "\n",
       "  input_tokens  \n",
       "0          her  \n",
       "0        visit  \n",
       "0           to  \n",
       "0           tu  \n",
       "0          was  \n",
       "0          cut  \n",
       "0        short  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "eval_df_tokens = eval_df.apply(pd.Series.explode)\n",
    "eval_df_tokens = eval_df_tokens.query(\"labels != 'IGN'\")\n",
    "eval_df_tokens[\"loss\"] = eval_df_tokens[\"loss\"].astype(float).round(2)\n",
    "eval_df_tokens.head(7)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### See Which Tokens Have Accumulated Most Loss in Evaluation Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>input_tokens</th>\n",
       "      <td>the</td>\n",
       "      <td>of</td>\n",
       "      <td>batman</td>\n",
       "      <td>ass</td>\n",
       "      <td>–</td>\n",
       "      <td>american</td>\n",
       "      <td>'</td>\n",
       "      <td>french</td>\n",
       "      <td>english</td>\n",
       "      <td>test</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>17506</td>\n",
       "      <td>7366</td>\n",
       "      <td>104</td>\n",
       "      <td>48</td>\n",
       "      <td>486</td>\n",
       "      <td>114</td>\n",
       "      <td>1730</td>\n",
       "      <td>62</td>\n",
       "      <td>57</td>\n",
       "      <td>36</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.018</td>\n",
       "      <td>0.038</td>\n",
       "      <td>2.275</td>\n",
       "      <td>4.14</td>\n",
       "      <td>0.238</td>\n",
       "      <td>0.695</td>\n",
       "      <td>0.045</td>\n",
       "      <td>0.85</td>\n",
       "      <td>0.916</td>\n",
       "      <td>1.432</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sum</th>\n",
       "      <td>323.04</td>\n",
       "      <td>280.12</td>\n",
       "      <td>236.62</td>\n",
       "      <td>198.7</td>\n",
       "      <td>115.74</td>\n",
       "      <td>79.19</td>\n",
       "      <td>78.37</td>\n",
       "      <td>52.68</td>\n",
       "      <td>52.2</td>\n",
       "      <td>51.57</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   0       1       2      3       4         5      6       7   \n",
       "input_tokens     the      of  batman    ass       –  american      '  french  \\\n",
       "count          17506    7366     104     48     486       114   1730      62   \n",
       "mean           0.018   0.038   2.275   4.14   0.238     0.695  0.045    0.85   \n",
       "sum           323.04  280.12  236.62  198.7  115.74     79.19  78.37   52.68   \n",
       "\n",
       "                    8      9  \n",
       "input_tokens  english   test  \n",
       "count              57     36  \n",
       "mean            0.916  1.432  \n",
       "sum              52.2  51.57  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(\n",
    "    eval_df_tokens.groupby(\"input_tokens\")[[\"loss\"]]\n",
    "    .agg([\"count\", \"mean\", \"sum\"])\n",
    "    .droplevel(level=0, axis=1)  # Get rid of multi-level columns\n",
    "    .sort_values(by=\"sum\", ascending=False)\n",
    "    .reset_index()\n",
    "    .round(3)\n",
    "    .head(10)\n",
    "    .T\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### See Which Label IDs Have Most Loss in Evaluation Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>labels</th>\n",
       "      <td>B-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>B-ORG</td>\n",
       "      <td>I-LOC</td>\n",
       "      <td>B-PER</td>\n",
       "      <td>I-ORG</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>O</td>\n",
       "      <td>I-PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>4173</td>\n",
       "      <td>4555</td>\n",
       "      <td>3512</td>\n",
       "      <td>2370</td>\n",
       "      <td>5593</td>\n",
       "      <td>2963</td>\n",
       "      <td>5904</td>\n",
       "      <td>234662</td>\n",
       "      <td>4107</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.393</td>\n",
       "      <td>0.337</td>\n",
       "      <td>0.31</td>\n",
       "      <td>0.233</td>\n",
       "      <td>0.223</td>\n",
       "      <td>0.211</td>\n",
       "      <td>0.193</td>\n",
       "      <td>0.026</td>\n",
       "      <td>0.026</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sum</th>\n",
       "      <td>1639.1</td>\n",
       "      <td>1536.22</td>\n",
       "      <td>1089.68</td>\n",
       "      <td>552.99</td>\n",
       "      <td>1248.04</td>\n",
       "      <td>625.31</td>\n",
       "      <td>1138.46</td>\n",
       "      <td>6186.3</td>\n",
       "      <td>107.08</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             0        1        2       3        4       5        6       7   \n",
       "labels  B-MISC   I-MISC    B-ORG   I-LOC    B-PER   I-ORG    B-LOC       O  \\\n",
       "count     4173     4555     3512    2370     5593    2963     5904  234662   \n",
       "mean     0.393    0.337     0.31   0.233    0.223   0.211    0.193   0.026   \n",
       "sum     1639.1  1536.22  1089.68  552.99  1248.04  625.31  1138.46  6186.3   \n",
       "\n",
       "             8  \n",
       "labels   I-PER  \n",
       "count     4107  \n",
       "mean     0.026  \n",
       "sum     107.08  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(\n",
    "    eval_df_tokens.groupby(\"labels\")[[\"loss\"]] \n",
    "    .agg([\"count\", \"mean\", \"sum\"])\n",
    "    .droplevel(level=0, axis=1)\n",
    "    .sort_values(by=\"mean\", ascending=False)\n",
    "    .reset_index()\n",
    "    .round(3)\n",
    "    .fillna(0)\n",
    "    .T\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create Function to Display Confusion Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay\n",
    "\n",
    "def plot_confusion_matrix(y_preds, y_true, labels):\n",
    "    cm = confusion_matrix(y_true, y_preds, normalize=\"true\")\n",
    "    fig, ax = plt.subplots(figsize=(10, 10))\n",
    "    disp = ConfusionMatrixDisplay(confusion_matrix=cm, display_labels=labels)\n",
    "    disp.plot(cmap=\"Blues\", values_format=\".2f\", ax=ax, colorbar=False)\n",
    "    plt.title(\"Normalized Confusion Matrix\")\n",
    "    plt.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Display Confusion Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA2QAAANXCAYAAABaBpzLAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAADNvUlEQVR4nOzdd1yV5f/H8fcBBAdDRQhFEhEFxfXLXSnO3Cv3TE3LkZq501LT1Py2y225M1dus8xdlitxz9ymoiAgijg4vz/Qo0cOpsXhRng9H4/zKO5z3TfX9eHynPt97nFMZrPZLAAAAABAqnMwugMAAAAAkFERyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAB2VblyZVWuXNny86lTp2QymTRjxoxU7UeHDh3k7++fqr/z35o9e7aCg4OVKVMmZc+ePcW3P3z4cJlMphTf7rPKqDkJABKBDAAMN2PGDJlMJmXOnFnnz59P8nzlypVVtGhRA3qWsS1ZskS1a9dWrly55OzsrDx58qh58+Zav369XX/v4cOH1aFDBxUoUEBTp07VlClT7Pr7UpvJZJLJZFLnzp1tPj9kyBBLmytXrjz19levXq3hw4f/x14CQOohkAFAGhEfH6+xY8ca3Q27y5cvn+Li4tSuXTuju2KT2WxWx44d9eqrr+rSpUt65513NGnSJPXo0UMnTpxQtWrVtHXrVrv9/o0bNyohIUFffPGFOnTooObNm6f47xg6dKji4uJSfLtPKnPmzFq8eLFu3bqV5Ll58+Ypc+bM/3rbq1ev1ogRI55qnbQ+JwGkbwQyAEgjSpYsqalTp+rvv/+22+8wm82G7ohLshwNdHR0NLQfyfnkk080Y8YMvf3229q1a5feffddderUSUOGDNHOnTs1a9YsOTk52e33h4eHS5JdTlW8z8nJ6T+Fnv+qVq1aiomJ0Y8//mi1fOvWrTp58qTq1q2bKv24c+eObt26lebnJID0jUAGAGnEu+++q7t37z7RUbI7d+5o5MiRKlCggFxcXOTv7693331X8fHxVu38/f1Vr149/fTTTypdurSyZMmiyZMna+PGjTKZTFqwYIFGjBghX19fubm5qWnTpoqOjlZ8fLzefvtteXt7y9XVVR07dkyy7enTp6tq1ary9vaWi4uLihQpookTJ/5j3x+9Xud+X2w9Hr3m68cff1TFihWVLVs2ubm5qW7dujpw4ECS37F06VIVLVpUmTNnVtGiRbVkyZJ/7JckxcXFacyYMQoODtbHH39s8zqrdu3aqWzZspafT5w4oWbNmilnzpzKmjWrypcvr1WrVlmt83C9P/zwQ+XNm1eZM2dWtWrVdPz4cUs7f39/DRs2TJLk5eUlk8lkOf3u4f9/mL+/vzp06GD5+fbt2xoxYoQKFiyozJkzy9PTUy+//LLWrl1raWPrGrKnnVO//vqrypYtq8yZMysgIECzZs16fHEf4uvrq0qVKum7776zWj537lwVK1bM5im6W7ZsUbNmzfT888/LxcVFfn5+6tOnj9UHDB06dND48eMt9br/kB7Mu48//liff/65ZZwHDx5MMifDw8Pl5eWlypUry2w2W7Z//PhxZcuWTS1atHjisQLAP7HfR3wAgKeSP39+tW/fXlOnTtWgQYOUJ0+eZNt27txZM2fOVNOmTdW3b19t27ZNY8aM0aFDh5KEjyNHjqhVq1Z688031aVLFwUFBVmeGzNmjLJkyaJBgwbp+PHj+uqrr5QpUyY5ODjo6tWrGj58uP744w/NmDFD+fPn1/vvv29Zd+LEiQoJCVGDBg3k5OSkFStWqHv37kpISFCPHj2eeNyFCxfW7NmzrZZFRUXpnXfekbe3t2XZ7Nmz9dprr6lmzZr66KOPdOPGDU2cOFEvv/yydu/ebQlvP//8s5o0aaIiRYpozJgxioiIUMeOHZU3b95/7Muvv/6qyMhIvf322090tOTSpUt68cUXdePGDfXq1Uuenp6aOXOmGjRooEWLFqlx48ZW7ceOHSsHBwf169dP0dHRGjdunNq0aaNt27ZJkj7//HPNmjVLS5Ys0cSJE+Xq6qrixYv/Yz8eNnz4cI0ZM0adO3dW2bJlFRMTo507d+rPP/9UjRo1kl3vaebU8ePH1bRpU73++ut67bXX9O2336pDhw4qVaqUQkJCnqifrVu3Vu/evRUbGytXV1fduXNHCxcu1DvvvKObN28mab9w4ULduHFD3bp1k6enp7Zv366vvvpK586d08KFCyVJb775pv7++2+tXbs2yZy6b/r06bp586beeOMNubi4KGfOnEpISLBq4+3trYkTJ6pZs2b66quv1KtXLyUkJKhDhw5yc3PThAkTnmiMAPBEzAAAQ02fPt0sybxjxw7zX3/9ZXZycjL36tXL8nxoaKg5JCTE8nNYWJhZkrlz585W2+nXr59Zknn9+vWWZfny5TNLMq9Zs8aq7YYNG8ySzEWLFjXfunXLsrxVq1Zmk8lkrl27tlX7ChUqmPPly2e17MaNG0nGUrNmTXNAQIDVstDQUHNoaKjl55MnT5olmadPn26zHgkJCeZ69eqZXV1dzQcOHDCbzWbztWvXzNmzZzd36dLFqu3FixfNHh4eVstLlixpzp07tzkqKsqy7OeffzZLSjKGR33xxRdmSeYlS5Y8tt19b7/9tlmSecuWLZZl165dM+fPn9/s7+9vvnv3rtlsflDvwoULm+Pj45P8vn379lmWDRs2zCzJfPnyZavfJck8bNiwJH3Ily+f+bXXXrP8XKJECXPdunUf2+/7v+O+fzOnNm/ebFkWHh5udnFxMfft2/exv/f+OHr06GGOjIw0Ozs7m2fPnm02m83mVatWmU0mk/nUqVM2a2Brvo0ZM8ZsMpnMp0+ftizr0aOH2dbuzf155+7ubg4PD7f53KNzslWrVuasWbOajx49av7f//5nlmReunTpP44RAJ4GpywCQBoSEBCgdu3aacqUKbpw4YLNNqtXr5YkvfPOO1bL+/btK0lJTpfLnz+/atasaXNb7du3V6ZMmSw/lytXTmazWZ06dbJqV65cOZ09e1Z37tyxLMuSJYvl/6Ojo3XlyhWFhobqxIkTio6O/qehJmvkyJFauXKlZsyYoSJFikiS1q5dq6ioKLVq1UpXrlyxPBwdHVWuXDlt2LBBknThwgWFhYXptddek4eHh2WbNWrUsGzrcWJiYiRJbm5uT9TX1atXq2zZsnr55Zcty1xdXfXGG2/o1KlTOnjwoFX7jh07ytnZ2fJzxYoVJSWe9phSsmfPrgMHDujYsWNPvM7TzqkiRYpY+i4lnl4ZFBT0VOPIkSOHatWqpXnz5kmSvvvuO7344ovKly+fzfYPz7fr16/rypUrevHFF2U2m7V79+4n/r1NmjSRl5fXE7X9+uuv5eHhoaZNm+q9995Tu3bt1LBhwyf+XQDwJAhkAJDGDB06VHfu3En2WrLTp0/LwcFBgYGBVst9fHyUPXt2nT592mp5/vz5k/1dzz//vNXP90OMn59fkuUJCQlWQeu3335T9erVlS1bNmXPnl1eXl569913JelfB7I1a9ZoxIgRGjx4sJo0aWJZfj9cVK1aVV5eXlaPn3/+2XIjjPtjL1iwYJJtP3yqZnLc3d0lSdeuXXui/p4+fdrmdgsXLmzVn/serXeOHDkkSVevXn2i3/ckPvjgA0VFRalQoUIqVqyY+vfvr7179z52naedU4+OQ0ocy9OOo3Xr1lq7dq3OnDmjpUuXqnXr1sm2PXPmjDp06KCcOXPK1dVVXl5eCg0NlfR08+1x/x4elTNnTn355Zfau3evPDw89OWXXz7xugDwpLiGDADSmICAALVt21ZTpkzRoEGDkm33pF/s+/CRhUcld51UcsvN925w8Ndff6latWoKDg7Wp59+Kj8/Pzk7O2v16tX67LPPklyT8yROnjypNm3aqEaNGho1apTVc/e3N3v2bPn4+CRZN6XuehgcHCxJ2rdvnxo1apQi23zYP9X137h7967Vz5UqVdJff/2lZcuW6eeff9a0adP02WefadKkScl+99d9TzqnUmocDRo0kIuLi1577TXFx8cne4v/u3fvqkaNGoqMjNTAgQMVHBysbNmy6fz58+rQocNTzbfH/Xuw5aeffpKUGJrPnTtn17tfAsiYCGQAkAYNHTpUc+bM0UcffZTkuXz58ikhIUHHjh2zHImREm8wERUVlewpXylpxYoVio+P1/Lly62Oltw/dfBpxcXF6dVXX1X27Nk1b948OThYn8BRoEABSYk3W6hevXqy27k/dlun6x05cuQf+/Hyyy8rR44cmjdvnt59991/vLFHvnz5bG738OHDVv1JCTly5FBUVJTVslu3btk8tTVnzpzq2LGjOnbsqNjYWFWqVEnDhw9PNpAZNaeyZMmiRo0aac6cOZYv4bZl3759Onr0qGbOnKn27dtblj9858j7njRUPok1a9Zo2rRpGjBggObOnavXXntN27Zts+vXHgDIeDhlEQDSoAIFCqht27aaPHmyLl68aPVcnTp1JCXeke9hn376qSSlync43Q8qDx8RiY6O1vTp0//V9rp27aqjR49qyZIlltP4HlazZk25u7tr9OjRun37dpLnL1++LEnKnTu3SpYsqZkzZ1qdxrZ27dok13PZkjVrVg0cOFCHDh3SwIEDbR7xmTNnjrZv3y4p8W+xfft2/f7775bnr1+/rilTpsjf3/+Jrlt7UgUKFNDmzZutlk2ZMiXJEbKIiAirn11dXRUYGJjk9vUPM3JO9evXT8OGDdN7772XbBtb881sNuuLL75I0jZbtmySlCS8Pq2oqCjLnSpHjx6tadOm6c8//9To0aP/03YB4FF8xAMAadSQIUM0e/ZsHTlyxOpW4iVKlNBrr72mKVOmKCoqSqGhodq+fbtmzpypRo0aqUqVKnbv2yuvvCJnZ2fVr19fb775pmJjYzV16lR5e3snezOS5KxatUqzZs1SkyZNtHfvXqvrnVxdXdWoUSO5u7tr4sSJateunV544QW1bNlSXl5eOnPmjFatWqWXXnpJX3/9taTEW/nXrVtXL7/8sjp16qTIyEh99dVXCgkJUWxs7D/2p3///jpw4IA++eQTbdiwQU2bNpWPj48uXryopUuXavv27dq6daskadCgQZo3b55q166tXr16KWfOnJo5c6ZOnjypxYsXJznS91907txZXbt2VZMmTVSjRg3t2bNHP/30U5KjSkWKFFHlypVVqlQp5cyZUzt37tSiRYv01ltvJbttI+dUiRIlVKJEice2CQ4OVoECBdSvXz+dP39e7u7uWrx4sc1r1kqVKiVJ6tWrl2rWrClHR0e1bNnyqfvVu3dvRURE6JdffpGjo6Nq1aqlzp07a9SoUWrYsOE/9hkAnhSBDADSqMDAQLVt21YzZ85M8ty0adMUEBCgGTNmaMmSJfLx8dHgwYMtXypsb0FBQVq0aJGGDh2qfv36ycfHR926dZOXl1eSOzT+k/tHtxYvXqzFixdbPZcvXz7LtVytW7dWnjx5NHbsWP3vf/9TfHy8fH19VbFiRXXs2NGyTq1atbRw4UINHTpUgwcPVoECBTR9+nQtW7ZMGzdu/Mf+ODg4aNasWWrYsKGmTJmijz/+WDExMfLy8lKlSpU0btw4VahQQZL03HPPaevWrRo4cKC++uor3bx5U8WLF9eKFStS/KhSly5ddPLkSX3zzTdas2aNKlasqLVr16patWpW7Xr16qXly5fr559/Vnx8vPLly6dRo0apf//+j92+0XPqcTJlyqQVK1aoV69eGjNmjDJnzqzGjRvrrbfeShKMXn31VfXs2VPff/+95syZI7PZ/NSBbPny5Zo1a5Y++eQTy3WFUuIRw7Vr1+q1117Tjh07rO5QCgD/lsn8X64kBgAAAAD8a1xDBgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBB+B4yO0pISNDff/8tNzc3mUwmo7sDAAAAIBWYzWZdu3ZNefLkkYPD44+BEcjs6O+//5afn5/R3QAAAABggLNnzypv3ryPbUMgsyM3NzdJknPFITI5ZTa4N+nHyQU9je5CusQxXDwrOOHAPjiTA8i4bt66a3QX0p1r12JUtJC/JQ88DoHMju6/uZmcMhPIUpC7u7vRXUiX2BXDs4LcYB8EMiDjciaQ2c2TvLZyUw8AAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIEvHOtcrqT3Tu+jC0re19rM2eqGQT7JtnRwd1L9VBf35TWddWPq2tnzdXtVK+Vu16dO8rNZ93lZnFvXS0e+6a857DRXom8POo0hbvlm4Wf/XaJh8K/bRK50+1p8HTj22/bJ1u1W++Uj5Vuyjiq1Ha+1vB6yeX7khTE17jlfBGgOVq1xP7Tt6zo69T7umLdysko2GKU/FPqrR6WPteoK6lms+Unkq9tHLNuq6YkOYmvQcr8AaA+WZQetKTe1j2sLNKtFwmHK/3EfVO/5zXZf+slvlmo1U7pf76KVWtuv6as/xKlB9oHKWzZh1nbpgk4o3eF8+L72t6h3+9wQ1/VNlm46Uz0tv68WWH+rnR2pqNps1etJKBdd6V7lf7qNG3b/SX2fC7TiCtIm62gd1TXnTF29RmSYj5F+lr+p0+VS7D55+bPsV63fr5VYfyr9KX1VpN1brth5Itu2AcfOV+6XemjJ/Ywr3OmURyNKpxpWCNKpLZX303e+q3HO29p8I1+KRTZXLI6vN9kPbv6wOtYtr4MR1Kt91uqav3qPZQxuqWIC3pc2LRf00beVuvfLOXL06ZKEyOTrqhw+bKatLptQalqGWrN2l975Yov6v19b6mQMUEuirZr0n6HLkNZvtt+89oTfem6E29Stow6yBqlOpuNoPmKpDf/1taXMj7pbKlQjQ+281TK1hpDmP1rXoE9S1y3sz1PahurazUdfyJQI0LIPWlZraxw9rd2no50s0oHNtbZg1QEUL+qppr+Truu1eXds0qKCNsweqTmhxte0/VQepq8UPPyfWdGDn2to4e6CKFvRVk57jk6/pnhPqPHSG2jasoE1zBqluaAm17TdFB48/qOkXs37R5Pmb9Ongllo7vZ+yZnFWk57jdTP+dmoNy3DU1T6oa8pb9sufGv7VEvXtVFM/fdtfRQLzqNU7E3Xlqu2a7th3Ut2Gz1LreuX18/T+qlWxmDoO/kaHT/ydpO3qTXv054HT8snlYe9h/GdpNpB16NBBJpPJ8vD09FStWrW0d+/eZNc5deqUTCaTwsLCkm2zdetW1alTRzly5FDmzJlVrFgxffrpp7p7926Sths2bFCdOnXk6emprFmzqkiRIurbt6/Onz+fEkO0q+6NS2vWmn36bu1+HTkboXe+Xqsb8bfV9pWiNts3r1pEny3YprU7T+r0xWh9u3qP1u48qbdeLW1p0+z9xZr3ywEdPhOh/Scvq/unP8rP210lCz6XWsMy1MR5G9SuYQW1rl9eQQG59cmgFsqS2VnfrfjdZvvJ8zeqavnC6tmuugrl99HgrvVUPMhP0xZutrRpXqes+neurdAyQak1jDRnwr26tqlfXsEP1XXuY+pa7V5dg/L76F0bdW2RwetKTe1jwncb1L7Rg7p+OqiFsj6urt8n1rXXvboO6VpPxYP9NG2BdV0HdK6tymUzZl0nfLde7Ru9qDYNKiTWdHBLZc3srDnLH1PTCg/VtFs9lQj209SFmyQlHm2YNG+D+nWqqTqhxVW0oK8mjmivi1eitWrTntQcmqGoq31Q15Q3ef5Gtan/olrWLa+g/D4a17+5srg4a97KP2y2n7Zgk6qUC1b3NtVUyN9HA9+oq2KF8urbRVus2l24HKWhny3W+GHt5OTkmBpD+U/SbCCTpFq1aunChQu6cOGC1q1bJycnJ9WrV+9fb2/JkiUKDQ1V3rx5tWHDBh0+fFi9e/fWqFGj1LJlS5nNZkvbyZMnq3r16vLx8dHixYt18OBBTZo0SdHR0frkk09SYnh2k8nJQSUDn9PGsAeHfM1maVPYGZUJzmNzHZdMjrp5647Vspvxd1Q+xDfZ3+OezUWSdPXazRToddp26/Yd7Tl8VqEP7TQ5ODgotEyQduw7ZXOdnftOJdl5rVI+WDv3nbRnV58p/6auO2zUtWr5YO2grpKoqb1Y6lrmKetalrom59btOwo7fNYqjDo4OCi0bFCyNdq+76Qqlwm2Wla1fGHL3+D0+QhdiohR5bIP2ni4ZlGpEH/t2HsqxceQFlFX+6CuKe/W7Tvae+SsKpYpZFnm4OCgiqULadf+UzbX2XngpCqWtn5drVwu2OrU0YSEBPX8YI66ta6qoIDc9uh6inMyugOP4+LiIh+fxOuefHx8NGjQIFWsWFGXL1+Wl5fXU23r+vXr6tKlixo0aKApU6ZYlnfu3FnPPfecGjRooAULFqhFixY6d+6cevXqpV69eumzzz6ztPX391elSpUUFRWVIuOzF0/3LHJydNDlq9etll+Ouq6CfjltrrP+z1Pq3ri0tu4/p5MXohRaMp/qvVhQjo4mm+1NJmnMm1X0x4FzOnT6SoqPIa2JiLquu3cT5JXT3Wq5V043HTt9yeY64REx8srpZrXMO6ebwiNsH4bPiO7X1fuRuno/ZV29qKsFNbWPx70GHH1MXb1tvQYkc3pTRhMRFXuvpo/OPXcdO/WYueppa67GSJIu3fvvo228PR+0Se+oq31Q15QXaXldTVqj48lcR3c54loy71cP6vX1nHVydHRQ52ahKd9pO0nTR8geFhsbqzlz5igwMFCenp5Pvf7PP/+siIgI9evXL8lz9evXV6FChTRv3jxJ0sKFC3Xr1i0NGDDA5rayZ89uc3l8fLxiYmKsHs+KQZPW68TfV7V9cieFL39H47pV03e/7FdCgtlm+4+7V1fhfLn0+tiVqdxTAAAAIKk9h89q2sJN+mJIG5lMtg8qpEVp+gjZypUr5erqKinxCFfu3Lm1cuVKOTg8fY48evSoJKlw4cI2nw8ODra0OXbsmNzd3ZU799Md5hwzZoxGjBjx1H1LaRExcbpzN0FeObJZLffKnk3hkdeTXaftyGVyyeSonO5ZdCEiVsM7VtKpi9FJ2o7rVk01ywaozoD5+jsi1i5jSGs8s2eTo6ODLkdah+zLkdeSHIm4z9vTPcmFvuGR1+T9yCdhGdn9uoY/Utfwp6zrZepqQU3t43GvAc95Jl/XR4+GJf4dqKskeWZ3vVfTR+dejLwfU9PLEbbmamL7+3+LyxHXrC7kD4+4pmKF8qZk99Ms6mof1DXl5bS8rtqoUTKvk16ebsm8XyXWctuev3TlaqxKNxluef7u3QSN+Hqppi7YpB2Lh6XsIFJImj5CVqVKFYWFhSksLEzbt29XzZo1Vbt2bZ0+fVq1a9eWq6urXF1dFRIS8sTbfPg6sce1+TepevDgwYqOjrY8zp49+9TbSAm37yQo7PglhZZ43rLMZJIqlXxeOw4nvQvNw+Jv39WFiFg5OTqo/ksF9eMfx62eH9etmupWCFSDwQt05lLSsJZeOWdyUolgP23ecdSyLCEhQZt3HFWZYv421yldzF+bdx61WrZp+xGVLpbfnl19pvybupaxUdeN24+oDHWVRE3tJbm6btr5D3Xd8Uhdt1HX+5wzOalksJ827ThiWfZgrtquUdli+a3aS9KGbYctf4N8vp56ztPdqk1MbJx2HTilMsX9U3wMaRF1tQ/qmvKcMzmpeJCfft1p/br6666jKlXU3+Y6pUPy69dd1q+rm3ccUamQxPZNa5XR+lkD9MuM/paHTy4PdW9dVfM+7WqvofxnafoIWbZs2RQYGGj5edq0afLw8NDUqVM1bdo0xcXFSZIyZfrn264XKpR4weChQ4f04osvJnn+0KFDKlKkiKVtdHS0Lly48FRHyVxcXOTi4vLE7e1pwpKdmvBObe0+dkl/Hr2gbg1LKZtLJs1du1+SNLFvbV2IiNUHMxLvSlMqyEe5Pd2070S48ni6amCbF+VgMumLRTss2/y4e3U1rRys1h8sVWzcLXnnSLyFfsz1W0luCJIedWtVRW99MEclCz+vF4rk06TvN+rGzXi1qldektR9+Czl9squ93o0kCS92aKyGnT9QuPnrtMrL4Xoh7V/KuzQGX06uKVlm1ejr+vcpau6eDkx3B6/dy2Kt6d7sp+6pzfdW1VRj4fqOvleXVvfq2u3e3V9/6G61r9X1xovhWjJvbp+Rl0tqKl9dG9dRT1G3KtryL3XgLiH6jpslnJ7P1TXlpVV/80v9PX914Cf79X13eTrev86P++c7nouV/qva/fWVdV9xGz9X+Hn9UKIvybO26DrcfFqUz+xpl2HzVJuLw/L1wK82bKy6r35ub6es06vvByiH37epbBDZ/T5u60kSSaTSV1bVdHH365RgJ+X8vl6avSkVfLJ5aG6oSUMG2dqo672QV1T3pstKqv3h3NVIvh5lSzyvKYu2KQbN2+pZd1ykqSeI+fIJ5eHhnSrL0nq3DxUr/b4UpPmrVe1F0O07Jc/tefwWf1vYAtJUk6PbMrpYX2GmJOTo7xyuiswX9q9K3iaDmSPMplMcnBwUFxcnHx9k7/7ny2vvPKKcubMqU8++SRJIFu+fLmOHTumkSNHSpKaNm2qQYMGady4cVY39bgvKioq2evI0oolm48ol3tWvdvuJXnnyKp9Jy6r6fuLdDnqhiQpr5e71fVhLpmcNKT9y/L38dD1uFtau/Okun68WjHX4y1tXq9XUpK0alxLq9/V/dMfNe+X5L+UL71oXKOUIqJiNXbKKoVHXFPRQr5a8Hl3y2Hyc5euysHhwZHVssUDNHlkB42etFIfTlypAD8vzRrXRYULPLjT5Zot+9Rz5FzLz12GzpAk9e9cWwO71EmdgRmscY1SuvKYup63UdcpIzvow0krNepeXWc/UtcfH6lr53t1HZBB6kpN7ePVGqUUcTVWYx6q68Ivkn8NKHevrqMnrdSoCYl1nfO/LirySF3f+uChug6ZISmxroPeSP91ffWVxLk6evKqe6dp+WrRlz0e1PRipBweOmOlXIkATR3VQR9OXKmRE1Yk1vTjN1Qk8EFNe7evrhtx8eozep6iY+NUvkQBLfqyuzJnkO/MlKirvVDXlNew+guKiIrVuGmrdTkyRiEF8+q7T7pabqB0/tJVq5qWKZZfE4a310dTVmvM5JXKn9dL08e8ruAA23cRf1aYzE9yDp8BOnTooEuXLmn69OmSpKtXr+rrr7/WxIkTtX79elWuXDnJOqdOnVL+/Pn1/fffKyjI+paYISEhWrZsmVq2bKlOnTrprbfekru7u9atW6f+/furWrVqWrBggeVUxQkTJuitt95Sx44d1b59e/n7++vcuXOaNWuWXF1dn+jW9zExMfLw8JBLlZEyOWX+70WBJOnKyr5GdyFdenYufUVG9wxdp/1MeZYugAeQsm7eSvp9vPhvYmJilC93TkVHR8vd/fFnPKTpI2Rr1qyxnDLo5uam4OBgLVy40GYYe1jLli2TLDt79qyaNm2qDRs26MMPP1TFihV18+ZNFSxYUEOGDNHbb79t9WbUvXt3FSpUSB9//LEaN26suLg4+fv7q169enrnnXdSdJwAAAAAMqY0e4QsPeAImX1whMw++GwczwoO5NgHR8iAjIsjZCnvaY6Qpem7LAIAAABAekYgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIE5GdyAjOP79W3J3dze6G+lGiXfXGN2FdCnsw5pGdyFdun3XbHQX0p0szo5GdyFduh5/x+gupDvZXNjNwrMhOu620V1Id67dfPKacoQMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAziZHQHYD/fLtqsCXPXKzwyRkUCfTX6naZ6ISRfsu2Xr9utj6as0tmLkcqf10vv9Wig6i+GWJ7/37TVWrr2T50Pj5JzJkcVD/LT4K71VCrEPxVGkza0LP+8OobmVy5XFx25cE2jlx/U/nPRNttOf6OsygR4Jlm++XC4us/YlWT5+41C1Lz88xq74pDm/HYqpbuepn2zaLPGz0mcqyGBvhrT9/Fzddm63Ro7ZZXOXohUgF/iXK1xb67evnNXYyat1C+/H9Tp8xFyc82s0DJBeq97A/l4eaTWkAw3ffEWTfxuvS7f+/c/qk8T/V+R5Gu6Yv1ujZu6Wufu/fsf0q2+qj307/9hA8fN1+xlWzWiV2N1aVHZTiNIm6Yu2KSv5qxTeESMihb01Uf9mz32NXDpL39q9KRVOnMhQgF+Xhres5FeeelBXc1ms8ZMXqVZS7cqOjZO5YoH6JNBLVTgee9UGE3aMGPxFk2at16XI6+pcIE8GvkPc3Xl+jD9b1riXPXP66V3u9VXtQpFLM/3+XCuFv64w2qd0LLBmvtpV7uNIS1irtoHdU15c5f9pm8WbNSVyGsKLpBbQ99qrOLBz9tse+zURX054ycdOHZOf1+6qsHdGui1JpWs2uzY+5e+WbBRB46d1+WIGH09ooOqv1Q0NYbyr6WZI2QdOnSQyWSyPDw9PVWrVi3t3bv3H9c9cOCAmjdvLi8vL7m4uKhQoUJ6//33dePGDat2/v7+lu1nzZpVxYoV07Rp05Jsz2w2a+rUqapQoYLc3d3l6uqqkJAQ9e7dW8ePH0+xMdvT0l/+1LAvl6jv67W0dkZ/hRT0Vcs+E3Q58prN9jv2nlDXYTPVun4F/TJzgGpXKq4OA6fp0F9/W9oE+HlrdN9m2jhnkJZPelt+uXOqRe8JunLV9jbTm1rFfTSgXmFN/OW4mn21VUcuxGjy62WUM5uzzfa9Z+9W6Kh1lkfDT7fozt0E/bTvYpK21UKeU/Hns+tS9E17DyPNWbL2T73/xRL161xL62YmztXmbyc/V7fvPaE335+pNvUraP29ufragAdzNe7mLe09ck7vdKypdTP7a8bY13X8dLja9p+SmsMy1LJf/tSIr5bonU419dO3/VUkMI9avzMx2X+rO/adVPfhs9SqXnn9PL2/alUspk6Dv9HhE38nafvjpj3adeC0fHJlnHB73w8/79LQz5doYOfa2jh7oIoW9FWTnuOTnavb9pxQ56Ez1LZhBW2aM0h1Q0uobb8pOnj8QV2/mPWLJs/fpE8Ht9Ta6f2UNYuzmvQcr5vxt1NrWIZavu5PffD1UvXpWEs/ftNPRQJ91fadScnO1Z37TqrHiFlqWa+81nzbT7UqFlPnwd/o8IkLVu0qlwvWn8s+sDzGD2+fGsNJM5ir9kFdU97qDWEaO2m5erSroR8mva2ggDzqPGiqIpJ5Dbh585b8cudU38515JXTzWabuJu3FByQR+/3bGzPrqeoNBPIJKlWrVq6cOGCLly4oHXr1snJyUn16tV77Dp//PGHypUrp1u3bmnVqlU6evSoPvzwQ82YMUM1atTQrVu3rNp/8MEHunDhgvbv36+2bduqS5cu+vHHHy3Pm81mtW7dWr169VKdOnX0888/6+DBg/rmm2+UOXNmjRo1yi5jT2mT5m1Q2wYvqlW98grKn1v/G9BcWVycNW/lHzbbT1mwSVXKFVaPttVUyN9Hg96sq2JBefXtoi2WNk1qllZo2SD5++ZScEBufdC7sa5dv2n1wpKetX85vxZtP6ulu87rRHisPlh6QDdv3VXj0nltto+Ju62I2FuWR4WCuXTzdoJ+3msdyLzdXTS4QREN/H6P7iQkpMZQ0pRJ8zaobcMX1freXP14YHNlyeys75Kbq/M3qWr5wnqrbTUVyu+jwW/WVfGgvPrm3lx1d82iRV/1UKPqLygw33MqXTS/xvZrqj2Hz+rcxcjUHJphpszfqNb1X1TLuuVVKL+PPur/+H//0xZsUpVywereppoK+vtowBt1VaxQXk1/6N+/JF24HKWhny3W+GHt5OTkmBpDSVMmfLde7Ru9qDYNKig4ILc+HdxSWTM7a87y3222n/z9RlWrUFi92lVXUH4fDelWTyWC/TR14SZJie83k+ZtUL9ONVUntLiKFvTVxBHtdfFKtFZt2pOaQzPMlO83qlX9CmpRt5wK5ffR2P7NlDmzs75fuc1m+28WblLlcsHq1rqqCvr7qH+XOipaKK9mLLaeqy7OTvL2dLc8srtnTY3hpBnMVfugrilvxuJNalannJrUKqvAfD4a8XYTZXbJpMVrdthsXyz4eQ14s77qVvk/Zcpk+0S/SmUL6+1OtVXj5WL27HqKSlOBzMXFRT4+PvLx8VHJkiU1aNAgnT17VpcvX7bZ3mw26/XXX1fhwoX1ww8/qGzZssqXL5+aNWumFStW6Pfff9dnn31mtY6bm5t8fHwUEBCggQMHKmfOnFq7dq3l+fnz5+v777/X/Pnz9d5776l8+fJ6/vnnVb58eX300UeaPn26XWuQEm7dvqO9R86qYpkgyzIHBwdVKhOknftP2lxn1/5TqlSmkNWyKuUKJ9v+1u07mr10q9xdsyikoG/KdT6NcnI0qYivu/44fsWyzGyW/jh+RSXyZX+ibbxaJq9+3PO34m7ftSwzmaQxLUpoxuYT+is8NqW7nebdun1He46cVaitubrP9tzbaWuuli+cbHtJiom9KZPJJA+3LCnT8TTswb//BzVycHBQxdKFtGv/KZvr7DpwUhVLB1ktCy0XrF0HHrRPSEhQrw/mqFvrqgoKyG2Prqdpt27fUdjhs6pc1nquhpYN0o5k5t72fSdVuUyw1bKq5Qtrx75TkqTT5yN0KSJGlcs+aOPhmkWlQvy1Y++pFB9DWnPr9h3tO3pOFUsnnat/PjT3HrZr/ymr9tK9ufrI3P5993GVqDdUlVp9qMEfL9DV6Osp3f00i7lqH9Q15d26fUcHjp7Xiy9YvwZUeKGgwg6eNrBnqS9NBbKHxcbGas6cOQoMDJSnZ9LrcCQpLCxMBw8e1DvvvCMHB+uhlChRQtWrV9e8efNsrpuQkKDFixfr6tWrcnZ+cMrZvHnzFBQUpAYNGthcz2QyJdvn+Ph4xcTEWD2MEBl1XXfvJiQ5lOuV003hEbYPAYdHxMgrp/s/tv/51/3KX7Wfng/tq8nfb9SCL7rLM7tryg4gDcqR1VlOjg6KiLU+4hoRe0u5XF3+cf2ieT1UyMdNi3ecs1r+emiA7t41a85vGeuF577k5qp3jsfPVe9H5+pj2t+Mv60Pxi/TqzVekFu29B/IkqtprpxuyZ5WcznimnLZfL148Bo2fs46OTo66PVmoSnf6WdARFRsMq+r7lZ1elh4RIy8PJOv66V7/320jbenW7LbTE8io5Ofq8mN/3LkNeXK8UhNc7jpcuSD9pXLFdbnQ9vq+y+6691u9fVH2F9q22+y7t7NGGcgMFftg7qmvKvR13U3IUGeOaz3I3PlcNOVq+l//A9LU4Fs5cqVcnV1laurq9zc3LR8+XLNnz8/Sdi67+jRo5KkwoUL23y+cOHCljb3DRw4UK6urnJxcVHTpk2VI0cOde7c2WqbQUHWnxS//fbbln7lzWv79DRJGjNmjDw8PCwPPz+/Jxr3s+SlUgW1fuZArZzytqqUL6wuQ6cnu5OHB14tk1dHL8RY3QCkiK+72r7kryEL//k6Sfw7t+/cVech02U2S/8b2Nzo7jyz9h4+q2kLN+nzIW0e+6EUkBY0rP6CXnm5qAoXyKNalYprxkddtOfQGf2++9m4BhxAxpOmAlmVKlUUFhamsLAwbd++XTVr1lTt2rV1+vRp1a5d2xKKQkKs7/xlNpuf+Hf0799fYWFhWr9+vcqVK6fPPvtMgYGBj11nyJAhCgsL0/vvv6/Y2ORPKxs8eLCio6Mtj7Nnzz5xv1JSzuzZ5OjokCQoXY68Jm9P2xdAenu6W33CmFz7bFlclN/PS6WL5tfnQ1rLydFR362wfe50enL1xi3duZsgT1frG3h4ujrrSmz8Y9fNkslRtUvk1g87rY+OveCfUzmzOWvtoMoK+7Cmwj6sKd8cWdW/brB+GpgxjkIkN1fDrz5+roY/OldttL8fxs5djNSir3pkiKNjUvI1vRJ5LdkLoL083XTF5utF4pHIbXv+0pWrsSrTZLj8KvWRX6U+OncxUiO+XqqyTUbYZyBpjGd212ReV2MsdXqUt6e7LkckX9fn7v330TbhEdeS3WZ6ktMj+bma3Pi9crolueHH5avXkpzh8bB8vrmUM3s2nTpn+/KH9Ia5ah/UNeXl8MgmRwcHRVy13re+cvWacuVI/+N/WJoKZNmyZVNgYKACAwNVpkwZTZs2TdevX9fUqVM1bdo0S1hbvXq1JKlQocRzTg8dOmRze4cOHbK0uS9XrlwKDAxUxYoVtXDhQvXq1UsHDx60PF+wYEEdOXLEah0vLy8FBgbK2/vxtyB1cXGRu7u71cMIzpmcVDzIT1t2Pjg6mJCQoC07j6h00fw21ylV1N+qvSRt2n442faW7ZoTdOv2nf/e6TTuzl2zDp6PUbnAB6fPmkxSucBc2nM66rHrvlLcR86ODlqx2/rmJyt2n9erX/yqpl/+Znlcir6p6ZtP6M1vdtpjGGmOcyYnlQjy0+Ydj8zVHUdUupjtuVe6qL+27LAxVx9qfz+MnTh7WYu+6qGcHtnsM4A06P6//18f+ff/666jKlXU3+Y6pULya8su65pu3nHEcivnJrXKaN2sAVo7o7/l4ZPLQ91aV9V3GeRW4s6ZnFQy2E+bdjx4f0hISNDmHUdVJpm5WrZYfqv2krRh22GVKeYvScrn66nnPN2t2sTExmnXgVMqU9w/xceQ1jhnclKxQnn1665jlmX35+oLydxGvFRRf/2685jVsi07jiQ7tyXp7/AoXY2+Ie8McmdQ5qp9UNeU55zJSSGFfPX7n9avAX/sPq6Sj/nqi/QoTQWyR5lMJjk4OCguLk6+vr6WsJYvX+IfqWTJkgoODtZnn32mhEfuTrdnzx798ssvatWqVbLb9/PzU4sWLTR48GDLslatWunIkSNatmyZfQaVSrq2qqK5y7dq/qptOnrqogaMW6AbN2+pZb1ykqS3RszWqAnLLe3faB6qDX8c0sTv1uvYqUv637TV2nP4rDo1rShJuh4Xrw8nrtDO/Sd19kKk9hw+o96j5uri5WjVr/p/howxtc369aSalvFTgxd8FeCVTe81ClEWZ0ct3ZV45Gt08+J6u2ahJOu9Wjqv1h+8pOgb1rewjb5xW8cvxVo97iQk6Mq1Wzp1JeNcgN61VRXNWb5V36/apqMnL6r/vbnaqm7iXO0xYrZGPjxXW4Rq/R+HNGFu4lwdN3W1wg6d1ev35urtO3fVafA3Cjt0RhNHtNfdBLMuRcToUkRMhvjwQJLeaFFZ3634XQtWb9exUxc16OOFif/+79W018g5Gj1xhaV95+ah2vjHIU2at17HTl/Sx9/8qL2Hz6rjvZrm9Mim4IA8Vg8nJ0d553RXYL7nDBmjEbq3rqpZS7dq3so/dOTkRb0zdr6ux8WrTf3ykqSuw2ZpxNcP3jvebFlZ634/qK/nrNPRUxc1dsoqhR06oy73rsMzmUzq2qqKPv52jVZv2qsDx8+r2/DZ8snlobqhJQwZY2p7o2VlzVvxuxb+mDhXB3+8UHFxt9Ti3lztPXKOxkx6MFdfbxaqjdsOafK8DTp++pI+uTdXOzS59151I14jxy/Trv2ndPZChH7deVSvD5omf99cCi0bbLMP6RFz1T6oa8rr0CRUC1dv05Kfd+iv05c0/IsfFHfzll6tVUaSNHDsPH0ybbWl/a3bd3To+HkdOn5et+/c1aUr0Tp0/LxOn39w07XrcfGWNpJ07kKkDh0/r78vXU3dwT2FNPXF0PHx8bp4MfGW4FevXtXXX3+t2NhY1a9f32Z7k8mkb775RjVq1FCTJk00ePBg+fj4aNu2berbt68qVKigt99++7G/s3fv3ipatKh27typ0qVLq2XLlvrhhx/UsmVLDR48WDVr1tRzzz2n06dPa/78+XJ0fDZu9dyo+guKuBqrcdNWKzwiRiEF82reZ90sN0M4f+mqHBweXAtSpniAJo54TWOnrNLoSSuU389bMz7qrMIF8kiSHB0cdPz0JS1YvV2R0bHK4ZFNJQs/r2UTeys4g9xxbc3ei8qRzVlv1SioXG4uOvx3jLp+u8Nyo4/c2TMr4ZHTZ/1zZVOp/DnVZdp2I7r8TGhc4wVFRMXqo6mr733RZl7N/6yb5XSNcxevWl23VLZ4gCZ98JrGTF6lDyetUICft2aOezBXL4RHac2W/ZKkKu0+svpdS8f31EulCqbSyIzTsHpiTf83bbUuRyb++5/7SVfLaV3nL12Vw0M1LVMsv8YPb6+PpqzW2MkrlT+vl74d87qCA/IYNYQ06dVXSulKVKxGT16l8IhrKlbIV4u+7PHQXI20qmu5EgGaOqqDPpy4UiMnrFCAn5fmfPyGigQ+qGvv9tV1Iy5efUbPU3RsnMqXKKBFX3ZXZpdMqT4+IzSo9oIioq7r42k/Wr7EfPYnb1pOr330vap0sfz6elh7jZu6Sh9NSZyr08a8bnkfcnA06fBff2vRjzsUExun53K5q1KZYPXvUkcuzmlql8eumKv2QV1TXp0qJRUZHauvZvyky1cTvxx+6pjOlpv3/B1+VaaHXgPCI2LUuOuDO6h/u3CTvl24SWWKB2j2p90lSfuPnNVr/SZZ2oydlPihbqNXSmvsgJapMaynZjI/zQVYdtShQwfNnDnT8rObm5uCg4M1cOBANWnS5LHr7tu3TyNGjNCGDRt07do1Pf/882rVqpUGDx6srFkffPeIv7+/3n777SQhrVatWnJwcLCcCpmQkKCpU6dq+vTp2r9/v27fvq28efOqWrVq6tOnT7I3EXlUTEyMPDw8dPbSVcNOX0yPXhj6k9FdSJfCPqxpdBfSpdt308RLbLqSxfnZ+GDsWXM9PmMcQU5N2VwyTgjEs+1S9E2ju5DuXLsWo2L5n1N0dPQ/5oA0E8jSIwKZfRDI7INAZh8EspRHILMPAlnKI5DhWUEgS3lPE8jS9DVkAAAAAJCeEcgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADOJkdAcyAgdT4gMpI+zDmkZ3IV3yenW80V1Il64secvoLgBPJKuzo9FdSHfMZrPRXUiXTCZ2qlKaR5ZMRnch3THdfvKacoQMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwiJPRHYD9fLNos8bPWa/wyBiFBPpqTN+meiEkX7Ltl63brbFTVunshUgF+HnpvR4NVOPFEEnS7Tt3NWbSSv3y+0GdPh8hN9fMCi0TpPe6N5CPl0dqDclw1NQ+Otcppp6NXpB3jqzaf+qKBk7ZrD+PXUq2fdf6JdSpdjHlzeWmyGtxWrb1uD6Y9bvib9+VJLlmyaR3W5dXvfIByuWRVftOXtagqZu1+3h4ag3JcN8s3Kyv565TeESMQgr6amzfpnohxD/Z9svW7daYySstc/X9Hg1V46UQy/MrN4Rpxg+/ac/hM7oac0MbZg9UsUJ5U2EkacvUBZv01ZzEuhYt6KuP+jdTqcfUdekvf2r0pFU6cyFCAX5eGt6zkV55qK5ms1ljJq/SrKVbFR0bp3LFA/TJoBYq8Lx3KowmbZi2cLOlpiEFffVRv6b/UNPEuXrm3lwd/pb1XDWbzRozZbVmW2qaXx8PzFg1lVK+ris2hGn6D79pz6HE14BNc3gN4DUgZUxfvEUTvluvy5ExKhLoqw/7NNH/FUl+32rF+t36aOpqnbsYqfx5vTS0W31VezHEZtsB4+Zr9rKtGtGrsd5oUdlOI/jvOEKWTi1Z+6fe/2KJ+nWupXUz+yukoK+avz1BlyOv2Wy/fe8Jvfn+TLWpX0HrZw5Q7UrF9dqAaTr019+SpLibt7T3yDm907Gm1s3srxljX9fx0+Fq239Kag7LUNTUPhq/XFCjOlXUR/O3q/I732v/yStaPLyBcnlksdm+aaVCGtb+RY37frvKvTVHPb9ap8YvF9R77SpY2nzxVjVVLumnrp+t1Uu9vtP63We09INGyp0zW2oNy1BL1u7Se18sUf/Xa2v9zAEKCfRVs96Pn6tvvDdDbepX0IZZA1WnUnG1HzDVMlcl6UbcLZUrEaD332qYWsNIc374eZeGfr5EAzvX1sbZA1W0oK+a9ByfbF237TmhzkNnqG3DCto0Z5DqhpZQ235TdPD4g7p+MesXTZ6/SZ8Obqm10/spaxZnNek5Xjfjb6fWsAz1w9rEmg7oXFsbZg1Q0YK+ator+bm6be8JdXlvhto0qKCNsweqTmhxte0/VQcfmqtfzvpFU+Zv0ieDWmjtt32VNYuLmvaakGFqKtmnrjfibql8iQAN4zWA14AUtOyXPzX8qyXq26mmfvq2v4oE5lGrdybqylXbNd2x76S6DZ+l1vXK6+fp/VWrYjF1HPyNDp/4O0nb1Zv26M8Dp+WTK+1/yJ2mA1mHDh3UqFGjx7aJi4vTsGHDVKhQIbm4uChXrlxq1qyZDhw4YNVu+PDhMplMMplMcnR0lJ+fn9544w1FRkYm2ebu3bvVokUL5c6dWy4uLsqXL5/q1aunFStWyGw2p+QQ7WbSvA1q2/BFta5XXkH5c+vjgc2VJbOzvlv5h832U+ZvUtXyhfVW22oqlN9Hg9+sq+JBefXNoi2SJHfXLFr0VQ81qv6CAvM9p9JF82tsv6bac/iszl1MWsP0iJraR/eGJTXr5wP6bt0hHTl7Ve9M3KAb8XfUtnoRm+3LBufWtkMXtGjzUZ0Nv6YNYWe1ePMxlSr4nCQps7OjGlQooOEztmrrwb918mK0Pvp+u05ciFan2sVSc2iGmThvg9o1rKDW9csrKCC3PhnUInGurvjdZvvJ8zeqavnC6tmueuJc7VpPxYP8NG3hZkub5nXKqn/n2gotE5Raw0hzJny3Xu0bvag2DSooOCC3Ph3cUlkzO2vO8mTq+v1GVatQWL3aVVdQfh8N6VZPJYL9NHXhJkmJn4xPmrdB/TrVVJ3Q4ipa0FcTR7TXxSvRWrVpT2oOzTATvtug9o0qqE398ok1HdRCWTM7a25yc/X7japW/qGadq2n4sF+mrYgca6azWZN+n6j+t6raUhBX00c3u5eTfem5tAMldJ1laQWdcpqQOfaqlyW1wBeA1LO5Pkb1ab+i2pZt7yC8vtoXP/myuLirHnJ7FtNW7BJVcoFq3ubairk76OBb9RVsUJ59e29fav7LlyO0tDPFmv8sHZycnJMjaH8J2k6kP2T+Ph4Va9eXd9++61GjRqlo0ePavXq1bpz547KlSunP/6w/mOGhITowoULOnPmjKZPn641a9aoW7duVm2WLVum8uXLKzY2VjNnztShQ4e0Zs0aNW7cWEOHDlV0dHRqDvFfuXX7jvYcOWu14+Tg4KBKZYK0c99Jm+vs3H9KlcoUslpWpXzhZNtLUkzsTZlMJnm42T6SkZ5QU/vI5OSgkgW8tXHPWcsys1natOesygT52Fxn++ELKlnAWy/cC2D5nnNXjVL5tHbXaUmSk6ODnBwddPP2Hav1bt66o/KFc9tpJGnHrdt3tOfwWYWWtZ6roWWCtGPfKZvr7Nx3KknQqlI++LFzNaO5dfuOwg6ftdoZdXBwUGjZIO1Ipk7b951U5TLBVsuqli9s+TucPh+hSxExqlz2QRsP1ywqFeKvHXtPpfgY0hrLXC3z5HN1x75TVnNbkqqWD7b8DU7/fb+mD9q4369pBpnP9qgreA2wh1u372jvkbOq+NC+koODgyqWLqRd+0/ZXGfngZOqWNp6rlYuF6xdBx60T0hIUM8P5qhb66oKCng23vef6WvIPv/8c/3+++/avXu3SpQoIUnKly+fFi9erHLlyun111/X/v37ZTKZJElOTk7y8UncyfP19VWzZs00ffp0y/auX7+u119/XXXr1tUPP/xg9bsKFy6s119//Zk4QhYZdV137ybIK6eb1XLvHG46fsr2dTnhETHyzulutcwrh5vCI2wfMr4Zf1sfjF+mV2u8ILds6T88UFP78HTPIidHB12OumG1/HLUDRXMm8PmOos2H1VO98z6cUwTmUxSJidHffvjPn26aKckKTbutrYfvqD+zcvo6LmrCo+6oaYVC6lMkI9OXEz7H6j8VxGWufrI3MvppmOnk5+rSeZ2zuTnakYUERVr8zXAK6e7jj3mNcDL89H2bgqPiJEkXbr330fbeHs+aJOePW6uHn3MXPW2NVfvnTJmqWmSv1PGqKlkn7qC1wB7SG7fyiunm46fsX3N9+WIa//47/vrOevk6Oigzs1CU77TdvJMHyH77rvvVKNGDUsYu8/BwUF9+vTRwYMHtWeP7UO+p06d0k8//SRnZ2fLsp9//lkREREaMGBAsr/zfrizJT4+XjExMVaP9Oj2nbvqPGS6zGbpfwObG92ddIGaPrmXivrqnaal1W/yRlV+Z77ajlmlV0r7q1/zMpY2b372s0wmkw5N76RLi7rrjXoltHjLUSUkpP0PVAAAwL+z5/BZTVu4SV8MafPYffa05pkOZEePHlXhwoVtPnd/+dGjRy3L9u3bJ1dXV2XJkkX58+fXgQMHNHDgQKvtSVJQ0INDoTt27JCrq6vlsXLlymT7M2bMGHl4eFgefn5+/2l8/1bO7Nnk6OiQ5CLT8KvX5P3IpzD3eXu6KzzSOkBettH+fnA4dzFSi77qkWGO5FBT+4iIidOduwnyyp7VarlX9qwKv3rD5jpDWpfXgo1HNHvtQR08HaFVf5zQyNm/q0/TUrr/2nvqYozqDflBvs0nqujr01W9/wI5OTnq9KX0+SHJwzwtc/WRuRd5LckR2/u8Pd2Tzu3I5Od2RuSZ3dXma8DlyBh5ez6mrhGPtr9maf/cvf8+2iY84lqy20xPHjdXn3tMTR89ahMeec1ydMdS0yR/p4xRU8k+dQWvAfaQ3L7V5cfMPS9Pt8f++9625y9duRqr0k2GK2+lPspbqY/OXYzUiK+XqkyTEfYZSAp4JgLZ3LlzrULRli0PLtx7mlMIg4KCFBYWph07dmjgwIGqWbOmevbs+dh1ihcvrrCwMIWFhen69eu6c+dOsm0HDx6s6Ohoy+Ps2bPJtrUn50xOKhHkp807HoTRhIQEbdlxRKWL5be5Tumi/tryUHtJ2rT9sFX7+8HhxNnLWvRVD+X0yBh3rJOoqb3cvpOgsL/CFVr8wa2TTSapUnE/7Thy0eY6WVyckhzpupuQcG9d60/DbsTf0aWrN+SRzUXVSj6v1dtOpPAI0h7nTE4qEZx0rm7ecVRlivnbXKd0MX9t3vnoXE1+bmdEzpmcVDLYT5t2HLEse1BX23UqWyy/VXtJ2rDtsOXvkM/XU895ulu1iYmN064Dp1SmuH+KjyGtSW6ubtqZ/FwtU8zfqr0kbdx2xPI3yJfnMTXNIPPZHnUFrwH24JzJScWD/PTrTuu5+uuuoypV1N/mOqVD8uvXXdZzdfOOI5avHmhaq4zWzxqgX2b0tzx8cnmoe+uqmvdpV3sN5T97JgJZgwYNLKEoLCxMpUuXliQVKlRIhw4dsrnO/eWFCj24UNDZ2VmBgYEqWrSoxo4dK0dHR40Y8SAtFyxYUJJ05MiDfxguLi4KDAxUYGDgP/bTxcVF7u7uVg+jdG1VRXOWb9X3q7bp6MmL6j9ugW7cvKVWdctJknqMmK2RE5Zb2r/RIlTr/zikCXPX69ipSxo3dbXCDp3V600rSkoMDp0Gf6OwQ2c0cUR73U0w61JEjC5FxOjW7eRDanpCTe1jwrIwtX8lRC2rBKtQ3hz6tGsVZcvspLm/HJQkTXy7ht5/6Jb2a3acVMfaxfRqxYJ63ttdlUv46d025bVmxylLUKv6f8+r2v89b3l+xajGOnr+quaus/16kd50a1VFs5c9mKv9PlqgGzfj1apeeUlS9+GzNHL8g7n6ZovKWv/7QY2fu07HTl3UR1NXK+zQGXVuVsnS5mr0de07ek5HTiYG5eOnL2nf0XOWayAygu6tq2rW0q2at/IPHTl5Ue+Mna/rcfFqUz+xrl2HzdKIr5dZ2r/ZsrLW/X5QX89Zp6OnLmrslFUKO3RGXe5d12AymdS1VRV9/O0ard60VweOn1e34bPlk8tDdUNL2OxDetO9dRXNWrZV81Zu05GTF9X3owW6ERev1vfmardhs/TBw3P1fk3n3q/pvbnaPHGumkwmdW1ZWZ98+5N+3LxPB4//re6WmhY3ZIxGSOm6SklfA47dfw24wmsArwH/3pstKmvuit+1YPV2HT11UQM/XqgbN2+p5b19q54j5+jDiSss7Ts3D9WGPw5p0rz1Onb6kj7+5kftOXxWne7tW+X0yKbggDxWDycnR3nldFdgvucMGeOTeCZu6uHm5iY3t6SHLlu2bKkhQ4Zoz549VteRJSQk6LPPPlORIkWSXF/2sKFDh6pq1arq1q2b8uTJo1deeUU5c+bURx99pCVLlthlLKmlcY0XFBEVq4+mrr735YV5Nf+zbpZDuucuXrU6mlC2eIAmffCaxkxepQ8nrVCAn7dmjuuswgXySJIuhEdpzZb9kqQq7T6y+l1Lx/fUS6UKptLIjENN7WPJr8eUyz2L3m1dTt45smnfyctqOmK5LkfHSZLy5nK1OiL28YIdMpulIW3KK3dOV0XExGnNjpMaOefBbYfdszrr/XYvKk8uV129dlMrfv9Lo+b8rjt3E1J9fEZoXKOUIqJiNXbKKoVHXFPRQr5a8Hn3B3P10lU5OFjP1ckjO2j0pJX6cOJKBfh5ada4Lpa5KklrtuxTz5FzLT93GTpDktS/c20N7FIndQZmsFdfKaUrUbEaPTmxrsUK+WrRlz0eeg2IlMNDrwHlSgRo6qgO+nDiSo2csEIBfl6a8/EbKhL4oK6921fXjbh49Rk9T9GxcSpfooAWfdldmV0ypfr4jPBqjVKKuBqrMQ/N1YVfJD9XyxUP0JR7c3XUhMS5Oud/XVTkobnaq311Xb9566GaBmjhFxmnppJ96vrjln1664MHrwGdh8yQJA3oXFuD3uA1QOI14N9oWD1x32rctNW6HBmjkIJ59d0nXS03pTl/6apVTcsUy68Jw9vroymrNWbySuXP66XpY15XcECe5H7FM8FkTsO3DezQoYOioqK0dOlSm8/fvHlTlStX1t9//61PPvlE5cqV06VLlzR69GitXbtWv/zyi8qXT/zUYvjw4Vq6dKnCwsKstlGuXDmVKVNGX3/9tSRpyZIlatGihWrUqKFevXqpYMGCio2N1Zo1azRw4EAtX75c9evXf6L+x8TEyMPDQ+fDrxp6tAx4El6vjje6C+nSlSVvGd2FdMfR4dm5UPtZkoZ3BwArz9LNGp4VN2/dNboL6U5MTIzy5c6p6Ojof8wBz8Qpi8nJnDmz1q9fr/bt2+vdd99VYGCgatWqJUdHR/3xxx+WMPY4ffr00bRp0yzXezVu3Fhbt25V1qxZ1b59ewUFBalq1apav369vv/+e9WrV8/ewwIAAACQQaTpI2TPOo6Q4VnCETL74AhZyuMImX2wO4BnBUfIUh5HyFJehjlCBgAAAADPMgIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGcTK6AxmBo4NJjg4mo7uRbpjNRvcgfbq6rKfRXUiXcrw8wOgupDtXfx1ndBfSJZOJ9ykgo8rkxDGalPY0NaX6AAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQJ6M7APuZtnCzvpqzTuERMQop6KuP+jVVqRD/ZNsv/WW3xkxeqTMXIhXg56XhbzVUjZdCLM+v2BCm6T/8pj2HzuhqzA1tmjNQxQrlTYWRpB3TFm7W13Mf1HRs38fXdNm63Ro9eaXO3qvpsB5Jazrjh9+053BiTTfOzng1laSpCzZZ5mrRgr76qH+zf5irf2r0pFU6cyEica72bKRXHqqr2WzWmMmrNGvpVkXHxqlc8QB9MqiFCjzvnQqjSRs6v1pBPVuFyjunm/b/dUEDP1umPw+dtdnWydFBfdpVVavapZQ7l7uOn72s4RNXa922o5Y2exYO0vO5cyZZd9oPW9X/06X2Gkaaw1xNedTUPqirfVDXlPeNjX2rF/5h32rMQ/tW7z+yb7XykX2rDc/AvtUzd4SsQ4cOatSoUbLPV65cWSaTSSaTSZkzZ1aRIkU0YcIEy/MzZsywPP/wI3PmzFa/4/7yTJkyKX/+/BowYIBu3rxpz6GlqB/W7tLQz5doQOfa2jBrgIoW9FXTXhN0OfKazfbb9p5Ql/dmqE2DCto4e6DqhBZX2/5TdfCvvy1tbsTdUvkSARr2VsPUGkaasmTtLr33xRL1f7221s8coKKBvmrWO/mabr9X07b1K2jDrIGqU6m42g2YqkPU1MoPPyfO1YGda2vj7IEqWtBXTXqOT36u7jmhzkNnqG3DCto0Z5DqhpZQ235TdPD4g7p+MesXTZ6/SZ8Obqm10/spaxZnNek5Xjfjb6fWsAzVuGoJjXqrvj6a/osqv/6F9h+/oMWfvq5c2bPZbD/0jZrq0LCcBn62TOXbfaLpS//Q7NGvqVjBPJY2Vbt8paAGH1gejd6eIklaumFvqowpLWCupjxqah/U1T6oa8p7dN8q5An2rd54b4baPLRv1d7GvlW5EgF6/xnat3rmAtmT6NKliy5cuKCDBw+qefPm6tGjh+bNm2d53t3dXRcuXLB6nD592mobtWrV0oULF3TixAl99tlnmjx5soYNG5baQ/nXJny3Qe0bVVCb+uUVHJBbnw5qoayZnTV3xe8220/+fqOqlS+sXu2qKyi/j4Z0rafiwX6atmCzpU2LOmU1oHNtVS4blFrDSFMmzNugdg0f1PSTQS2U5XE1nZ9Y0573avpu13oqHuSnaQuta9q/c22FlsmYNZWkCd+tV/tGL6pNgwqJc3VwS2XN7Kw5yx8zVys8NFe71VOJYD9NXbhJUuKnjZPmbVC/TjVVJ7S4ihb01cQR7XXxSrRWbdqTmkMzTPeWFTVrxTZ9t3qnjpwK1zv/+0E3bt5W23plbLZvXrOUPpu9Xmv/OKzTf0fq26V/aO3vh/VWy0qWNhFR1xUeGWt51HyxsE6cu6Lfdp9IrWEZjrma8qipfVBX+6CuKW/ivX2r1vXLK+ihfavvHrNvVfXevlWh/D4abGPfqvkzuG+VLgNZ1qxZ5ePjo4CAAA0fPlwFCxbU8uXLLc+bTCb5+PhYPZ577jmrbbi4uMjHx0d+fn5q1KiRqlevrrVr16b2UP6VW7fvaM/hs1YT0cHBQaFlgrRj3ymb6+zYd0qhjwStquWDtWPfSXt29ZlhqWnZp6xpGWr6OLdu31HY4bNWId/BwUGhZYOSrdP2fSdVuUyw1bKq5Qtb/g6nz0foUkSMKpd90MbDNYtKhfhrx95TKT6GtCaTk6NKFvLVxp3HLcvMZrM27TymMiH5bK7jkslRN+PvWC27GX9b5Yv7J/s7mr/yguau2pFi/U7rmKspj5raB3W1D+qa8v7NvtVOG/tWVcoHa+czvm+VLgPZo7JkyaJbt2796/X379+vrVu3ytnZ+bHt4uPjFRMTY/UwQkTUdd29myCvnO5Wy71yuulShO0+hUfEyDunm9Uy75xuCk/mkHFGc7+m3o/UNLFGydfU65GaeuV0U3gENb0vIir23lx9tE7uCn/MXPXytFXXxPb35/ijbbw93ZLdZnri6ZFNTk6OSU73uBwZK+9HanLf+u1H1b1lRQXkzSWTyaTKpQuqXmhRPefpbrN93Uoh8nDNrO9W70rx/qdVzNWUR03tg7raB3VNeY/bX32afSvvdLBvla4D2d27dzVnzhzt3btXVatWtSyPjo6Wq6ur1aN27dpW665cuVKurq7KnDmzihUrpvDwcPXv3/+xv2/MmDHy8PCwPPz8/OwyLgBISYO+WK4TZ69o+9x+Ct8wWuPeaaTvVu9Ugtlss33bumX0y7YjupgBdhgAALC3ZzaQzZ071ypQbdmyxfLchAkT5OrqqixZsqhLly7q06ePunXrZnnezc1NYWFhVo9p06ZZbb9KlSoKCwvTtm3b9Nprr6ljx45q0qTJY/s0ePBgRUdHWx5nz9q+o5m9eWbPJkdHB11+5NOFy5HXkv3E29vTPcnRsPDIa0mOmmVU92v66Cc2iTVKvqZJj1JcS/YoRUbkmd313lx9tE4x8n7MXL0cYauuie3vz/FH24RHXEt2m+lJRPR13blz18anuK7JfoIYEXVdbd+dJd8aQ1W86RiVbf0/XY+L16m/I5K09XsuuyqXLqhZK7bbpf9pFXM15VFT+6Cu9kFdU97j9lefZt8qPB3sWz2zgaxBgwZWgap06dKW59q0aaOwsDCdPHlS169f16effioHhwdDdXBwUGBgoNXD19fXavvZsmVTYGCgSpQooW+//Vbbtm3TN99889g+ubi4yN3d3ephBOdMTioR7KfNOx7csjohIUGbdh5VmWL+NtcpU8zfqr0kbdx2RGWK5bdnV58ZydV0845/qOnOR2q6nZo+zDmTk0oG+2nTjiOWZQ/qartOZYvlt2ovSRu2Hbb8HfL5euo5T3erNjGxcdp14JTKJHNNVHpy+85dhR09r9BSgZZlJpNJlUoFaseB049ZU4q/dUcXrsTIydFB9UOL6cctB5O0aV23jC5fjdXPvx9O8b6nZczVlEdN7YO62gd1TXn/Zt+qtI19q03bj6j0M75v9cwGMjc3N6tAlSVLFstzHh4elpD1cBD7txwcHPTuu+9q6NChiouL+8/bSw3dW1fRrGVbNW/lNh05eVF9P1qgG3Hxal2vvCSp27BZ+mD8gxudvNmystb9flBfz12no6cuauyU1Qo7dEadmz+4y9rV6Ovad/Scjpy8KEk6dvqS9h09p0tXMsZpS91bVdHsZVs1b1ViTft9tEA3bj5U0+GP1LRFYk3H36vpR1Pv1bRZ8jU9fr+mGehUsO6tq2rW0q2at/IPHTl5Ue+Mna/rcfFqUz+xrl2HzdKIr5dZ2lvm6pz7c3WVwg6dUZdmoZISw0fXVlX08bdrtHrTXh04fl7dhs+WTy4P1Q0tYcgYU9uE77eoff2yalmrlArl89an/RorWxZnzV21U5I0cWgLvf9mLUv7UkX8VK9SUeXLk1MVivtr0Sevy8HBpC++22i1XZPJpDZ1Suv7Nbt0925Cag4pTWCupjxqah/U1T6oa8rrdm/f6vtV23T0oX2rVvf2rboPn6WRj+xbrb+3b3UsHe1bZcgvhjabzbp48WKS5d7e3skGuGbNmql///4aP368+vXrZ+8u/mev1iiliKuxGjNllcIjrqloIV8t/KK75RD4uUtX5eBgsrQvVzxAU0Z20OhJKzVqwkoF+Hlpzv+6qEiBB99D9OOWfXrrg7mWnzsPmSFJGtC5tga9USd1BmagxjVK6UpUrMY+VNMFnz+o6flHalr2Xk0/nLRSoyYm1nT2uC4q/EhNe458qKZDZ0hKrOnALum/ppL06iuJdR09ObGuxQr5atGXPR7M1YuRcjA9NFdLBGjqqA76cOJKjZywInGufvyGigQ+qGvv9tV1Iy5efUbPU3RsnMqXKKBFX3ZXZpdMqT4+IyxZv0e5smfTu51fkXdON+07/rea9v1Gl6/GSpLyPpddCQkPrg9zcc6kIV1qyj9PTl2Pu6W1fxxW15HzFRNr/d2LlUsHys8nh+ZkoLsrPoy5mvKoqX1QV/ugrimvcY1SinjMvtWj+6tliwdo8r391Q/v7VvNemTfas0j+1Zd7u1b9U/D+1YmszmZq7bTqA4dOigqKkpLly61+XzlypVVsmRJff755zafnzFjhjp27GjzuQsXLsjHxyfZ3zF27Fh9+umnOnnypLJls/0Fqw+LiYmRh4eHLl6JMuz0xfTo2Zqxz46HX/CQcnK8PMDoLqQ7V38dZ3QXACBduZvAzlVKi4mJUR6v7IqOjv7HHPDMBbJnCYHMPpix9kEgsw8CWcojkAFAyiKQpbynCWTP7DVkAAAAAPCsI5ABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGMTJ6A5kBAnmxAdShqODyeguAE/s6q/jjO5CupOz5bdGdyFdivy+k9FdSHcSePO3Cwf2A1JcROwto7uQ7lx7ippyhAwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIM4PUmj5cuXP/EGGzRo8K87AwAAAAAZyRMFskaNGj3Rxkwmk+7evftf+gMAAAAAGcYTBbKEhAR79wMAAAAAMpz/dA3ZzZs3U6ofAAAAAJDhPHUgu3v3rkaOHClfX1+5urrqxIkTkqT33ntP33zzTYp3EAAAAADSq6cOZB9++KFmzJihcePGydnZ2bK8aNGimjZtWop2DgAAAADSs6cOZLNmzdKUKVPUpk0bOTo6WpaXKFFChw8fTtHOAQAAAEB69tSB7Pz58woMDEyyPCEhQbdv306RTgEAAABARvDUgaxIkSLasmVLkuWLFi3S//3f/6VIpwAAAAAgI3ii294/7P3339drr72m8+fPKyEhQT/88IOOHDmiWbNmaeXKlfboIwAAAACkS099hKxhw4ZasWKFfvnlF2XLlk3vv/++Dh06pBUrVqhGjRr26CMAAAAApEtPfYRMkipWrKi1a9emdF8AAAAAIEP5V4FMknbu3KlDhw5JSryurFSpUinWKQAAAADICJ46kJ07d06tWrXSb7/9puzZs0uSoqKi9OKLL+r7779X3rx5U7qPAAAAAJAuPfU1ZJ07d9bt27d16NAhRUZGKjIyUocOHVJCQoI6d+5sjz4CAAAAQLr01EfINm3apK1btyooKMiyLCgoSF999ZUqVqyYop0DAAAAgPTsqY+Q+fn52fwC6Lt37ypPnjwp0ikAAAAAyAieOpD973//U8+ePbVz507Lsp07d6p37976+OOPU7RzAAAAAJCePdEpizly5JDJZLL8fP36dZUrV05OTomr37lzR05OTurUqZMaNWpkl44CAAAAQHrzRIHs888/t3M3AAAAACDjeaJA9tprr9m7HwAAAACQ4fzrL4aWpJs3b+rWrVtWy9zd3f9ThwAAAAAgo3jqm3pcv35db731lry9vZUtWzblyJHD6gEAAAAAeDJPHcgGDBig9evXa+LEiXJxcdG0adM0YsQI5cmTR7NmzbJHHwEAAAAgXXrqUxZXrFihWbNmqXLlyurYsaMqVqyowMBA5cuXT3PnzlWbNm3s0U8AAAAASHee+ghZZGSkAgICJCVeLxYZGSlJevnll7V58+aU7R0AAAAApGNPfYQsICBAJ0+e1PPPP6/g4GAtWLBAZcuW1YoVK5Q9e3Y7dBH/1jcLN+vruesUHhGjkIK+Gtu3qV4I8U+2/bJ1uzVm8kqdvRCpAD8vvd+joWq8FGJ5fuWGMM344TftOXxGV2NuaMPsgSpWKG8qjCTtmLpgk76ak1jTogV99VH/Zir1mJou/eVPjZ60SmcuRCjAz0vDezbSKw/V1Gw2a8zkVZq1dKuiY+NUrniAPhnUQgWe906F0aQd1DXlUVP7eP2VwupZv6i8PbLowJmrGjj9d/3515Vk23etXUQdaxRW3lzZFHntppZvO6UP5u1S/O27kqSwr5rpeS+3JOtN++mQBkz/3W7jSEuYq/YxzcY+wOPqumzdbo1+aB9g2CP7ACse2QfYmAH3ASTmqz3MXfqrvlmwUZcjrym4QB6917Oxigc/b7PtsVMX9eWMNTpw9JzOX7qqwd0bqkOTSlZtduz9S9/M36j9x87pckSMxo/ooOovF0uNofxrT32ErGPHjtqzZ48kadCgQRo/frwyZ86sPn36qH///ineQfw7S9bu0ntfLFH/12tr/cwBCgn0VbPeE3Q58prN9tv3ntAb781Qm/oVtGHWQNWpVFztB0zVob/+trS5EXdL5UoE6P23GqbWMNKUH37epaGfL9HAzrW1cfZAFS3oqyY9xydb0217Tqjz0Blq27CCNs0ZpLqhJdS23xQdPP6gpl/M+kWT52/Sp4Nbau30fsqaxVlNeo7XzfjbqTUsw1HXlEdN7aNxhfwa1a6sxi0KU5XBy7X/dKQWDa6pXO6ZbbZv8lKA3m9VWuMW71b5vj+o1+Rf1ah8gN5rWcrSptq7KxT85jzLo/GoNZKkZdtOpsqYjMZctY9H9wGKPsE+QJf3ZqjtQ/sA7WzsA5QvEaBhGXQfQGK+2sPqDbs1ZtJy9Wj/ipZM6qPgAnn0+sApirhqu6ZxN28pb25P9e1cV145k36YJSXO1aACeTSs16v27HqKeupA1qdPH/Xq1UuSVL16dR0+fFjfffeddu/erd69e6dIpzp06KBGjRol+3zlypVlMpk0duzYJM/VrVtXJpNJw4cPt2r/9ttvW34+efKkWrdurTx58ihz5szKmzevGjZsqMOHD1tta8OGDapTp448PT2VNWtWFSlSRH379tX58+f/6xDtbuK8DWrXsIJa1y+voIDc+mRQC2XJ7KzvVtj+xHXy/I2qWr6werarrkL5fTS4az0VD/LTtIUPTkNtXqes+neurdAyQak1jDRlwnfr1b7Ri2rToIKCA3Lr08EtlTWzs+YsT6am329UtQqF1atddQXl99GQbvVUIthPUxdukpT4qdikeRvUr1NN1QktrqIFfTVxRHtdvBKtVZv2pObQDEVdUx41tY/udYtq1voj+m7TMR05H6V3pv2mG7fuqE3lQjbbly3krW1Hw7X4txM6ezlWG/b+rR+2ntALBbwsbSKu3VR4dJzlUfMFP524GKPfDl5MrWEZirlqHxPu7QO0qV9ewQ/tA8x9zD5AtXv7AEH5ffSujX2AFhl8H0BivtrD9EWb1bxOeTWpVVaB/j4a8XYTZXbJpMVrtttsXzz4eQ18s77qVv0/OWeyfaJfaLnC6tOptmqk8aNiD3vqQPaofPny6dVXX1Xx4sVToj9PzM/PTzNmzLBadv78ea1bt065c+dOdr3bt2+rRo0aio6O1g8//KAjR45o/vz5KlasmKKioiztJk+erOrVq8vHx0eLFy/WwYMHNWnSJEVHR+uTTz6x06hSxq3bd7Tn8FmFln3woung4KDQMkHase+UzXV27juV5EW2Svlg7dyXMT6l/Se3bt9R2OGzqvxoTcsGaUcyNdq+76Qqlwm2Wla1fGHL3+D0+QhdiohR5bIP2ni4ZlGpEH/t2HsqxceQFlHXlEdN7SOTo4NK5PfUpn0PPtk2m6VN+/5WmUJeNtfZfjRcJfN76oUCuSRJ+bzdVOP/8mpt2Nlkf0ezlwto7sajKT+ANIi5ah//Zh9gh419gKrlg5P9O2REzNeUd+v2HR04ek4vvlDQsszBwUEvvlBIuw+eNrBnqe+JriH78ssvn3iD94+e2Vu9evW0YMEC/fbbb3rppZckSTNnztQrr7yiM2fOJLvegQMH9Ndff2ndunXKly+fpMRQeX8bknTu3Dn16tVLvXr10meffWZZ7u/vr0qVKlkFt7QoIuq67t5NkFdO6y/p9srppmOnL9lcJzwiJsmhX++cbgqPsH3IOKOJiIq9V1PrGnnldNexU4+pqeej7d0UHhEjSbp077+PtvH2fNAmvaOuKY+a2oenu4ucHB10OTrOavnl6DgV8s1uc53Fv52Qp1tmrR5RVyaZlMnJQd+uPaTPlu612b5umXzyyOaseZuOpXT30yTmqn3c3wfwfmQfwPsp9wG82AewwnxNeVejr+tuQoI8c1iP3zOHq06cDTeoV8Z4okD2cCh5HJPJlGqBzNnZWW3atNH06dMtYWrGjBkaN26c1emKj/Ly8pKDg4MWLVqkt99+W46OjknaLFy4ULdu3dKAAQNsbiO5m5fEx8crPj7e8nNMTPr/xwQAsO2lIj7q06i4+n/zu3Yev6wAH3eNea2c+r16Qx//kPR0pLZVCuqXsHO6eDXOxtYAAOnVE52yePLkySd6nDhxwt79tdKpUyctWLBA169f1+bNmxUdHa169eo9dh1fX199+eWXev/995UjRw5VrVpVI0eOtOr7sWPH5O7u/thTH20ZM2aMPDw8LA8/P79/Na7/yjN7Njk6OuhypHUgvBx5LcknZvd5e7onuSg1PPKavD1tXzCZ0Xhmd71XU+saXY6MkbfnY2oa8Wj7a5b2z93776NtwiOuJbvN9Ia6pjxqah8RMfG6czdBXh5ZrJZ7eWTRpagbNtd5t/kLWrDlL83ecFSHzl7Vqh2nNfL7XXq7YQmZTNZt8+bKptBieTR7fcY4XVFirtrL/X2A8Ef2AcKfch/gMvsAVpivKS+HRzY5OjgkuYFHxNVY5Urmhh3p1X++hsye5s6dK1dXV8tjy5YtVs+XKFFCBQsW1KJFi/Ttt9+qXbt2cnL654N+PXr00MWLFzV37lxVqFBBCxcuVEhIiNauXSsp8SJL06Pvlk9g8ODBio6OtjzOnrV9nYC9OWdyUolgP23e8eCNPSEhQZt3HFWZYv421yldzF+bd1rvCGzafkSli+W3Z1efGc6ZnFQy2E+bdhyxLHtQU9s1Klssv1V7Sdqw7bDlb5DP11PPebpbtYmJjdOuA6dUprh/io8hLaKuKY+a2sftuwnaczJClYrmsSwzmaTQonm04+hlm+tkcXZSgtlstexuQuLPJlm/x7SpXEiXo2/q593GvG8YgblqH/9mH6CMjX2AjduPJPt3yIiYrynPOZOTQgrl1e+7H5ymnZCQoN93H9P/FclnYM9S31N/D1lqatCggcqVK2f52dfXN0mbTp06afz48Tp48KC2b7d9RxZb3NzcVL9+fdWvX1+jRo1SzZo1NWrUKNWoUUOFChVSdHS0Lly48FRHyVxcXOTi4vLE7e2pW6sqeuuDOSpZ+Hm9UCSfJn2/UTduxqtVvfKSpO7DZym3V3a916OBJOnNFpXVoOsXGj93nV55KUQ/rP1TYYfO6NPBLS3bvBp9XecuXdXFy9GSpOP3zkX39nS3fMqTnnVvXVXdR8zW/xV+Xi+E+GvivA26HhevNvUTa9p12Czl9vKw3BL4zZaVVe/Nz/X1nHV65eUQ/fDzLoUdOqPP320lKfEU366tqujjb9cowM9L+Xw9NXrSKvnk8lDd0BKGjTO1UdeUR03tY8Kq/RrfraLCTlzRn8cvq2udEGV1cdJ3mxJ3ZCd0r6QLkdc18vtdkqSf/jyr7nVCtO9khOWUxXebv6Cf/jxjFdRMJql1aEF9v/m4JbBlFMxV++jeqop6PLQPMPnePkDre/sA3e7tA7z/0D5A/Xv7ADVeCtGSe/sAn7EPYIX5mvI6Nq2kgR99r6KF/FQ8+HnNXLxZcTdv6dWaZSVJA8Z+p+dyeahv57qSEm8E8te9uXfrzl1duhKtQ8fPK2sWF+XzTbyB0vW4eJ05/+D7Ic9djNSh4+fl4ZZVeZ7LkcojfDJpOpC5ubnJze3xhyxbt26tfv36qUSJEipSpMi/+j0mk0nBwcHaunWrJKlp06YaNGiQxo0bZ/P6uaioqDT/JdiNa5RSRFSsxk5ZpfCIaypayFcLPu9uOQR+7tJVOTg8+IS2bPEATR7ZQaMnrdSHE1cqwM9Ls8Z1UeECDz4NXrNln3qOnGv5ucvQGZKk/p1ra2CXOqkzMAO9+kopXYmK1ejJiTUtVshXi77s8aCmFyPl8NCR1XIlAjR1VAd9OHGlRk5YoQA/L835+A0VCXxQ097tq+tGXLz6jJ6n6Ng4lS9RQIu+7K7MLplSfXxGoa4pj5rax5LfT8rTPbMGN3tB3tmzaP/pSDUb+7MuR9+UlHja4cNB6+MfwmQ2m/Vui1LKnTOrImJuas2usxo1f5fVdisXyyM/L9cMc3fFhzFX7aNxjcS6JrcPcN7GPsCUkR304aSVGnVvH2D2I/sAPz6yD9D53j7AgAyyDyAxX+2hTpX/U2T0dX054yddvhqjwgV8NW1sF8spixfCo6xqGh4Ro0Zvfmr5+dsFG/Xtgo0qW6KAZn/aXZK0/8hZte870dJmzMTlkqTGr5TW2IGtUmNYT81kNpvT3MdxHTp0UFRUlJYuXWrz+cqVK6tkyZL6/PPPJSUGpEyZMilbtmySpJIlS6pRo0aWm3s83D4sLEzDhg1Tu3btVKRIETk7O2vTpk3q3bu3Bg4cqPfee0+SNGHCBL311lvq2LGj2rdvL39/f507d06zZs2Sq6vrE936PiYmRh4eHvr7cpTc3TPGp0epwdHh6U8nBZB+5Gz5rdFdSJciv+9kdBfSnYQMdsQztTiwH5DiwmPi/7kRnsq1mBgVze+t6Ojof8wBafoI2ZN6mqNVefPmlb+/v0aMGKFTp07JZDJZfu7Tp4+lXffu3VWoUCF9/PHHaty4seLi4uTv76969erpnXfescMoAAAAAGQ0/+oI2ZYtWzR58mT99ddfWrRokXx9fTV79mzlz59fL7/8sj36+UziCJl9cIQMyNg4QmYfHCFLeRwhsw+OkKU8jpClvKc5QvbUd1lcvHixatasqSxZsmj37t2W792Kjo7W6NGj/12PAQAAACADeupANmrUKE2aNElTp05VpkwPLjh86aWX9Oeff6Zo5wAAAAAgPXvqQHbkyBFVqlQpyXIPDw9FRUWlRJ8AAAAAIEN46kDm4+Oj48ePJ1n+66+/KiAgIEU6BQAAAAAZwVMHsi5duqh3797atm2bTCaT/v77b82dO1f9+vVTt27d7NFHAAAAAEiXnvq294MGDVJCQoKqVaumGzduqFKlSnJxcVG/fv3Us2dPe/QRAAAAANKlpw5kJpNJQ4YMUf/+/XX8+HHFxsaqSJEicnV1tUf/AAAAACDd+tdfDO3s7KwiRYqkZF8AAAAAIEN56kBWpUoVmUzJfyHf+vXr/1OHAAAAACCjeOpAVrJkSaufb9++rbCwMO3fv1+vvfZaSvULAAAAANK9pw5kn332mc3lw4cPV2xs7H/uEAAAAABkFE992/vktG3bVt9++21KbQ4AAAAA0r0UC2S///67MmfOnFKbAwAAAIB076lPWXz11Vetfjabzbpw4YJ27typ9957L8U6BgAAAADp3VMHMg8PD6ufHRwcFBQUpA8++ECvvPJKinUMAAAAANK7pwpkd+/eVceOHVWsWDHlyJHDXn0CAAAAgAzhqa4hc3R01CuvvKKoqCg7dQcAAAAAMo6nvqlH0aJFdeLECXv0BQAAAAAylKcOZKNGjVK/fv20cuVKXbhwQTExMVYPAAAAAMCTeeJryD744AP17dtXderUkSQ1aNBAJpPJ8rzZbJbJZNLdu3dTvpcAAAAAkA49cSAbMWKEunbtqg0bNtizPwAAAACQYTxxIDObzZKk0NBQu3UGAAAAADKSp7qG7OFTFAEAAAAA/81TfQ9ZoUKF/jGURUZG/qcOAQAAAEBG8VSBbMSIEfLw8LBXXwAAAAAgQ3mqQNayZUt5e3vbqy8AAAAAkKE88TVkXD8GAAAAACnriQPZ/bssAgAAAABSxhOfspiQkGDPfgAAAABAhvNUt70HAAAAAKQcAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABnni7yHDv+foYJKjg8nobqQbd+7ynXj24OTI5zN4NkR+38noLqRLORpPNLoL6U7E4q5GdwF4Ii5O7AOktFtPUVOqDwAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGMTJ6A7AfqYu2KSv5qxTeESMihb01Uf9m6lUiH+y7Zf+8qdGT1qlMxciFODnpeE9G+mVl0Isz5vNZo2ZvEqzlm5VdGycyhUP0CeDWqjA896pMJq04ZtFmzV+znqFR8YoJNBXY/o21Qsh+ZJtv2zdbo2dskpnL0QqwM9L7/VooBovJtb09p27GjNppX75/aBOn4+Qm2tmhZYJ0nvdG8jHyyO1hpQmMFdTHjW1D+qa8jrXDlHPRiXlnT2r9p+K0MBpv+rPY+HJtu9ar7g61QpR3lyuirx2U8u2/qUP5mxT/O27kiTXzJn0buuyqlcuv3J5ZNG+k1c06Jtftfv45dQaUpowbeFmfT03ca6GFPTV2L5NHztXl63brdGTV1rer4b1aKgaD83VFRvCNOOH37Tn8BldjbmhjbMHqlihvKkwkrSF14CUN2vJr5r8/XpdjrymwgXyaETvV1WycPL7Vqs2hOmTb3/UuYuRyu/rpUFd66lK+SKW5/1D+9hcb3DX+nqzVdUU739K4AhZMs6ePatOnTopT548cnZ2Vr58+dS7d29FREQY3bUn8sPPuzT08yUa2Lm2Ns4eqKIFfdWk53hdjrxms/22PSfUeegMtW1YQZvmDFLd0BJq22+KDh7/29Lmi1m/aPL8Tfp0cEutnd5PWbM4q0nP8boZfzu1hmWoJWv/1PtfLFG/zrW0bmZ/hRT0VfO3JyRb0+17T+jN92eqTf0KWj9zgGpXKq7XBkzTob8Saxp385b2HjmndzrW1LqZ/TVj7Os6fjpcbftPSc1hGY65mvKoqX1Q15TX+KUCGtXxJX00f6cq912k/acitPj9esrlkcVm+6YVC2pYu3IaN3+nyvX8Xj2/3qDGLwfqvbblLG2+6FFZlUvkVdcv1umlt+drfdhZLR1eX7lzZkutYRluydpdeu+LJer/em2tnzlARQN91az349+vurw3Q23rV9CGWQNVp1JxtRsw1fJ+JUk34m6pfIkADXurYWoNI83hNSDlrVi/W6PGL1Xv12pq1dS+KlIgj9r3m6wrV23XdNf+k+o1crZa1Cmn1VP76ZWKRfXGkG915MQFS5vtP4yweowb2FImk0m1Q4un1rCeGoHMhhMnTqh06dI6duyY5s2bp+PHj2vSpElat26dKlSooMjISKO7+I8mfLde7Ru9qDYNKig4ILc+HdxSWTM7a87y3222n/z9RlWrUFi92lVXUH4fDelWTyWC/TR14SZJiZ/gTJq3Qf061VSd0OIqWtBXE0e018Ur0Vq1aU9qDs0wk+ZtUNuGL6p1vfIKyp9bHw9sriyZnfXdyj9stp8yf5Oqli+st9pWU6H8Phr8Zl0VD8qrbxZtkSS5u2bRoq96qFH1FxSY7zmVLppfY/s11Z7DZ3XuYtqfYymFuZryqKl9UNeU171BCc1ae1DfrT+iI+eu6p1Jm3Qj/rbaVgu22b5s8HPadviiFm05prOXr2nDnnNavOWYShVMPJqQ2dlRDSoEaPis37X14AWdvBijj+bv1ImLMepUK8TmNtOjCfM2qF3DCmpTv7yCA3Lrk0EtlCWzs+auSGauzt+oauULq+e9ufpu13oqHuSnaQs3W9q0qFNW/TvXVmiZoNQaRprDa0DKm7Zgo1rWq6DmdcqpoL+PPuzbTFkyO2vB6m0223+7aLNCywbrzVZVFej/nPq+XkchhfJq5pItljbenu5Wj7W/7VeF/wvU83lypdawnhqBzIYePXrI2dlZP//8s0JDQ/X888+rdu3a+uWXX3T+/HkNGTLE6C4+1q3bdxR2+Kwql33woung4KDQskHase+kzXW27zupymWs3wCrli+sHftOSZJOn4/QpYgYVS77oI2HaxaVCvHXjr2nUnwMac2t23e058hZqzciBwcHVSoTpJ3J1HTn/lOqVKaQ1bIq5Qsn216SYmJvymQyycPN9qfD6Q1zNeVRU/ugrikvk5ODShbw0sY95yzLzGZp097zKhP0nM11th++pJIFvPTCvQCW7zk31SiVT2t3nZEkOTk4yMnRQTdv3bVa7+atOypf2MdOI0lbbt2+oz2Hzyr00blaJsgy9x61Y9+pJEGravngZOd2RsRrQMq7dfuO9h89p5dKPdhXcnBw0EulCurPA6dtrrP7wCmr9pJUqUxQsu0vR17Tht8PqkWdcjafTysIZI+IjIzUTz/9pO7duytLFuudYh8fH7Vp00bz58+X2WxOsm58fLxiYmKsHkaIiIrV3bsJ8srpZrXcK6e7wiNs9yk8IkZeno+2d7O0v3Tvv4+28fZ0S3ab6Ulk1HWbNfXO4abwCNuH1cMjYuSd091qmddj2t+Mv60Pxi/TqzVekFu2jBHImKspj5raB3VNeZ5umeXk6KDL0XFWyy9H3ZB39qw211m05ZhGz9uhHz9spPCFbyhsUlv9tv9vfbr4T0lS7M3b2n74ovo3LyWfHFnl4GBS89CCKlPoOT2XI2Ocshhx7/3q0fcf75xuCo98zFxNMreTf7/KiHgNSHlXoxPnaq4cj9Qoh5suJzNXL0des9n+SjLtF6/ZrmxZM6tmpbR7uqJEIEvi2LFjMpvNKly4sM3nCxcurKtXr+ry5aQXB48ZM0YeHh6Wh5+fn727i3Ti9p276jxkusxm6X8DmxvdHQBIk14KyaN3mrygflO2qHLfRWo7do1eKfW8+jUrZWnz5hfrZDKZdOjb13RpwRt6o24xLf71uBJsfJAKIH1b8ON2Nar+gjK7ZDK6K49FIEuGrSNg/2Tw4MGKjo62PM6ePWuHnv0zz+yucnR0SHKR6eXIGHl7uttcx9vTXZcjHm1/zdL+uXv/fbRNeMS1ZLeZnuTMns1mTcOvXpP3I59s3eft6Z7k08jLNtrfD2PnLkZq0Vc9MszRMYm5ag/U1D6oa8qLuHZTd+4myOuRG3h4Zc+q8KgbNtcZ0rqsFmw6qtm/HNLBM5Fate2kRs7dpj5N/k8mU2KbUxdjVG/oMvm2nKqiXWar+oAf5OTooNMX0/8RB0nyvPd+9ej7T3jktSRHze7z9nS3MbeTf3/LiHgNSHk5PBLn6qM38Lh89Zq8kpmrXjndbLbPZaP99j1/6cSZcLWoVz7lOm0nBLJHBAYGJn6yduiQzecPHTqkHDlyyMvLK8lzLi4ucnd3t3oYwTmTk0oG+2nTjiOWZQkJCdq846jKFMtvc52yxfJbtZekDdsOq0wxf0lSPl9PPefpbtUmJjZOuw6cUpni/ik+hrTGOZOTSgT5afOOo5ZlCQkJ2rLjiEonU9PSRf215aH2krRp+2Gr9vfD2Imzl7Xoqx7K6ZExTqm5j7ma8qipfVDXlHf7ToLC/rqs0OIPbp1uMkmVivlqx5FLNtfJ4uKkhATrD0zv3jXfW9dktfxG/B1dunpDHtmcVe3//LR6e8a4Hso5k5NKBCd9v0qcq/421ylTzF+bd1q/X23cfiTZuZ0R8RqQ8pwzOaloobzaust6rm7981iyXyn0fyH+Vu0l6dedR222n796m4oF5VWRQN+U7bgdEMge4enpqRo1amjChAmKi7M+r/3ixYuaO3euWrRokeSFP63p3rqqZi3dqnkr/9CRkxf1ztj5uh4Xrzb1Ez8l6DpslkZ8vczS/s2WlbXu94P6es46HT11UWOnrFLYoTPq0ixUUuIbXddWVfTxt2u0etNeHTh+Xt2Gz5ZPLg/VDS1hyBhTW9dWVTRn+VZ9v2qbjp68qP7jFujGzVtqVTfxQtEeI2Zr5ITllvZvtAjV+j8OacLc9Tp26pLGTV2tsENn9XrTipISw1inwd8o7NAZTRzRXncTzLoUEaNLETG6dfuOIWM0AnM15VFT+6CuKW/C8j1qX6OwWlYJUqG82fXpm5WULXMmzV13WJI0sVdVvf/QLe3X7DiljrVC9OrLgXre202VS+TVu63Las2O05agVrWkn6r9n5/l+RUjG+rouSjNXX/EZh/So+6tqmj2sq2at2qbjpy8qH4fLdCNm/Fqfe9IQbfhs/TB+AfvV2+2SJyr4+cmztWPpq5W2KEz6tyskqXN1ejr2nf0nI6cvChJOn76kvYdPWe5Dioj4DUg5XVuXlnzVv2hRWu26/ipSxry6SLdiLulZrUT/92/8+FcfTRlpaV9p6aVtGn7YU2dv0HHT1/SZ9PXaN+Rs3qtcUWr7V67flOrN+5Ri7pp/+iYxBdD2/T111//f3t3Hh/Tvf9x/D2RRcgiiRAitUXEErQI6t6KXhrUUqX2lp/Sq1TRanXR0lqri1utrdaqLdWWFkXttNpSFZRYUoJWkCaRiCVBzu8PNYwstpmcSF/Px2MeeeSczzlzvp+cOTPvOTMnevDBBxUZGamRI0eqfPny2rNnj1566SUFBgZq1KhRZm/iTT3+SG39dTpNo6cu16nEMwoLCdQXE/pZT4H/cSJJTteFyno1K2jayB4aNXmZRkxaqgpB/pr73jOqGlzaWjPgqSY6dz5dg0YvUEraedWvWVFfTOib7z+Xay9tmz6gxNNpemfat3//Q8gyihr/7HU9TbYJ6uE1KmjK2901ZupyjZqyVBWCSujTcb1UpeKVnsafOq2Vm3+TJDV+8h2b+1oysb8a1q6URyMzF/uq/dFTx6Cv9rf4h99V3Mtdr3WqqxI+RbT78F9q//Yy64U+yvh72Hz3671F22UYVz66WMq3qBJTz2vlL0c0Yu61S2R7FXHVm0/WU2k/DyWfuaClPx3SyHlbdelyZp6Pzyxtm17ZV8d+cmVfrR4SqM//19e6r/55MllOTrbPV5+M6KFRU5Zp5ORlqhDkr8/G9bY+X0nSis271X/EPOvvvYbOliS93Ku5hvRukTcDMxnHAPtr9fD9SjqdpvEzVyohKVVVggP16bv/tV485c9TybJct6/Wrl5eH77xpN6f8a3enbZc5cr465NRPVW5Qimb9S5d+6sMw1Dr/zyQp+O5UxbjTr4s9Q9w5MgRDRs2TCtXrlRSUpICAgL02GOPadiwYfLz87uldaSmpsrb21snE1NM+/hiQfRPelLNS86FOGEO/JP5tJ1s9iYUOIlf9jF7Ewqk68Mk7CPl3D/jH1HnpTOpqaoUVFwpKTfPAZwhy0HZsmU1e/ZsszcDAAAAQAHGW+IAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJnM3eAOB2ORfifQRHMAzD7E0okCwWi9mbUOBczmRfdYSkr/qYvQkFjm+Ld83ehAIpecXLZm9CgePsxHOVvd1OT3llCwAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmMTZ7A2A40z7fKM+mrtWpxJTVb1SoN556QnVrlYux/ola37V6CnLdTQ+URWC/DW8/2N6pGE163zDMDRm6nLNWbJFKWnnVa9GBb3/SkdVvK9EHowmf6CnjjF90SZrX6tVCtQ7g9vfpK87NGbqMh2NT7rS1+faqOmNff3kW31m7Wt5vTfkn9VX9lXHmLFokz6ed21fHftiez2QS1+/XntlXz329776Zj/bfXXZ+mjN/uoH7dx3VMmp57T+syEKCymTByPJP3j8O0avlverf/twlfApqt8OndKQyWv064ET2dY6F3LSoI711blJNZXy81TsH0kaPnOj1m4/bK0Z1KGeWjYMUaUyfrqQcVFb9x7X8JkbFftnUl4NKV/g2Gp/s7/arCkL1ikh6YyqVCytEQPb6f6qZXOsX7Y+Wu9O/1Z/nEhSuTL+eq1PK/2nQVWbmoNxJzR6ylL9FP27Ll3OVEi5kvpkZE8FlvRx9HDuCGfICqivvtuuof9brCG9mmvDZ0NUvVKg2vWfqISkM9nW/7zzkHoNna1ubRpo49xX9Gijmuo2+BPtjT1urflwzhpNjdqoD17tpNWzBquIu6va9Z+oC+kX82pYpqKnjvHV6it9fblXc62f87KqVwpU++cn5dzXXYfU+43Z6tq6gTZ8NkQtGtVQt5emae/v1/o6Yc4afRK1Ue+/0lGrZ76oIu5uav/8pH9MX9lXHWPx6u1648PFeunp5lr36cuqFhyoJwbkvK9u3XVIz7wxW11bNdD6OUPU4qEaeurlaYq5bl89dz5D9WpW0JvPtcmrYeQrPP4do+1DoRr5TGO9M+8HRfT/VL8dTtCXIzuouHeRbOuHdv+3ejSvqSGT16r+f2do1rfR+uyNxxRW8VooeDAsSNOX7tAjgz7T4699LhdnJ3016gkVcXPJq2GZjmOr/X2z9le9/fESDerRTCumD1bV4EB1e3GK/krOvqe/7D6sfm/NUadH62vljMFq9u8w9XpthvYdirfWxP35l9r2m6CK95XUognPafXslzWge6TcXPPveShTA1mPHj1ksVisNz8/PzVr1ky7du3KcZm4uDhZLBYVKlRIf/75p828+Ph4OTs7y2KxKC4uzqY+OjraWrd48WLVr19f3t7e8vT0VLVq1TRw4ECbdWVkZGjcuHGqWbOmihQpouLFi6thw4aaNWuWLl7M/w+SSfPX6anHHlTX1g0UWqGUPni1k4oUdtXcb37Mtn7qwg36T4Mqev7JJqpcPkCvP9tSNUODNG3RRklX3sGZsmC9BveMVItGNVS9UqAmv/WUTvyVouUbd+bl0ExDTx1j0vz1euqxBuraqv6Vvr7SUUUKu2re0lz6Wv+6vvZpqRqhQZr++SZJf/d14Qa9+Hdfq1UK1OThT/7d15yPLQUJ+6pjTF6wXk+2aaAureqrcoVSev+VjnIv7Kr5Oe2rURv0cP0q6v9kE4WUD9CrfVqqRuUgTV+0yVrToUW4XurVXI3qVs6rYeQrPP4do2/bOpqzYpfmr/5N+48m6oWPVulc+kV1eyQs2/oOD1fT+KiftHrbIR05kaKZy6O1etshPfd4XWvNE298oQVrftO+o4n67XCC+n7wrYJKeqtWpZJ5NSzTcWy1v0+iNqhzqwbq+Gg9hZQP0NjBT6hwYVctXP5ztvUzvtioiPBQPdvlYVUqF6CXerVQ9ZAymv3VZmvNuE+W6+H6VTW0b2tVDymjcoHF9ci/qqu4j2deDeu2mX6GrFmzZoqPj1d8fLzWrl0rZ2dntWzZ8qbLBQYGas6cOTbTPv30UwUGBua63Nq1a9WxY0e1a9dOW7du1fbt2zVq1CibkJWRkaHIyEiNHTtWzzzzjLZs2aKtW7eqX79++uijj7Rnz547G2weybh4SdH7jiki/NoTvJOTkxqFV9a23YezXWbr7sOKqBtqM+3h+lW0bXecJOnIn4k6mZiqiPBrNd4e7qpdrZy27Yqz+xjyG3rqGBkXL2nnvmM2L0adnJzUqG5la59utG13nBqF2754fbh+qPXvcOT41b5eq/G62tcc/lYFCfuqY1j31Rv7msu++svuuCxBq3H9UP3yD9gPbwWPf8dwcXZSrUoB2hAdZ51mGNLG6COqW6V0tsu4uRTShYxLNtMuZFxS/Wo5f3zWq4ibJCn5zIW73+h7AMdW+8u4eEm7D/yhf9cOsU5zcnLSv+uE6Nc9cdkus/23OP27TojNtEbhodr+25X6zMxMrf1xryoE+avrC5NVs9VQtXzmA63clL/fkDE9kLm5uSkgIEABAQGqVauWXnnlFR07dkwJCQm5Lte9e3fNmjXLZtqsWbPUvXv3XJdbunSpGjZsqJdeekmVK1dWSEiIHnvsMU2cONFa87///U+bNm3S2rVr1a9fP9WqVUsVKlRQly5d9PPPP6tSpUp3PuA8kHg6TZcvZ8rf1/adAH9fL51KTM12mVOJqfL3u7He01p/8u+fN9aU8PPMcZ0FCT11jMTTZ//uq5fNdH9fT2t/bnQqMVUlbvg7lPD11Km/PzJi7WuWv9U/o6/sq46R2756KimXvma3ryZm/1Gcfxoe/47h51VEzoWclJB8zmZ6QvJZlfApmu0y67YfVt/H66pCaR9ZLFLE/WXV8sEQlfTNvt5ikcb89z/6ac8fijnyl93HkB9xbLW/pJSz2fa0uE/O409IOqPi2Ty+E/4+Dv+VnKaz59M1cd5aRdSrovkf9FGzh2qo99BZ+nFHrGMGYgemB7LrpaWlae7cuQoODpafn1+uta1bt1ZycrK+//57SdL333+v5ORktWrVKtflAgICtGfPHv3222851sybN09NmjTR/fffn2Wei4uLihbN/gCVnp6u1NRUmxsAAEB+9srUtTr0Z7K2fvK0Ti0drHF9m2r+6t3KzDSyrX+vX1NVKVdcT4/9Jo+3FMhdpnFln33kX9XVu2OEqlUqo+e6NVGTB6tq7tc/mLx1OTM9kC1btkweHh7y8PCQp6envvnmG0VFRcnJKfdNc3FxUbdu3TRz5kxJ0syZM9WtWze5uOT+5dL+/furbt26CgsLU7ly5dSpUyfNnDlT6enp1pqDBw8qNDQ0l7Vkb8yYMfL29rbegoKCbnsd9uBXzEOFCjll+ZJpQlKqSvh5ZbtMCT8vJSTeWH/GWl/y75831pxKPJPjOgsSeuoYfsWK/t1X2zcvEpLOWPtzoxJ+XtZ3w686lXTG+q65ta9Z/lb/jL6yrzpGbvtqCd9c+prdvuqXf7/HkJd4/DtGYuo5XbqcKX8f2wt4+PsU1anks9kvk3Je3UYsVmDb8arRfYrCe0/X2QsXFXciJUvtuGebKDK8oloNWajjf6U5ZAz5EcdW+/P1LpptT/9Kznn8/r6e+iubx/fVM+2+3kXlXMhJIeUCbGqCy5bUnydP22/j7cz0QNa4cWNFR0crOjpaW7duVWRkpJo3b64jR46oefPm1rBWrVq1LMv27NlTixYt0okTJ7Ro0SL17NnzpvdXtGhRLV++XLGxsRo6dKg8PDz04osvKjw8XOfOXTm9bxjZvyN0M6+++qpSUlKst2PHjt3Reu6Wq4uzaoUGaeO2/dZpmZmZ2rTtgOqGlc92mfCw8jb1krT+532qG1ZOklQ20E8l/bxsalLTzmv7njjVrVHO7mPIb+ipY7i6OKtmaJA2bTtgnZaZmamNvxyw9ulGdcPK2dRL0oaf91v/DmVL59LXHP5WBQn7qmPktK9e6Wu5bJepE1ZOm36x3Vc3bt2vOv+A/fBW8Ph3jIuXMhV98IQa1bp22XCLRXqoVlltizmey5JS+sXLik9Mk3MhJ7VqGKIVPx60mT/u2SZ69MFKav1KlI6ezBrWCjKOrfbn6uKssJAy+n77tf0sMzNT328/kOO/E6ldvZxNvSRt/mW/alcvZ11nzSr36fejp2xqDh1LUGBA/rzkvZQPAlnRokUVHBys4OBg1a1bV9OnT9fZs2c1bdo0TZ8+3RrWvv322yzLhoWFKTQ0VJ07d1aVKlVUvXr1W77fihUrqlevXpo+fbp+/fVX7d27V1FRUZKkkJAQ7du377bH4ubmJi8vL5ubWfp2eVhzlmzRgmU/af/hE3phbJTOnk9X11b1JUl9hs3RWx9/ba3/b6cIrf1xrz6eu1YH4k5o7CfLFR1zVL2faCRJslgs6tO5sd6buVLfbtylPbF/6tnhnymguLcebVTTlDHmNXrqGH27NNacr7dowbKftf/wCb34zuc6dz5dXVpe6euzw+bo7YnXPhZj7eu8q339VtExR9Wrw0OS/u5rpwi9P3OVVmzarb2xx9XX2tcapowxr7GvOsaznRvrs6+3aOHyn3Xg8AkNfudznbuQrs5/76t9h8/RiOv31Y4RWvfjXk2ct1YH407onWl/76tPPGStSU45q90H/tD+w1f+P1TskZPafeCPHL9DVdDw+HeMSYt/0VPNaqpTk2oKCfLVB889oqJuLpq3erckafKLLfRmj2v7Ye3KpdTywUoqG+CtBtXK6IuRT8jJYtGHX2y11rzXr6k6PFxVvcctU9r5DJXwKaoSPkVVOB9fStzeOLba3zMdI7Rg2Y9atGKrDsad0KvvL9L58xnq2KKeJGnAyLkaM2Wptf7p9o204ecYTV24XrFHTur9mSu0a98x9Xj839aaPp0f1tJ1OzTvmx91+I8Ezfpys9Zs2aPubf+V5+O7VfnuUWSxWOTk5KTz58/f9IqJ0pWzZH379tXkyZPv+D7LlSunIkWK6OzZK6fyu3Tpotdee007duzI8j2yixcvKiMjI8fvkeUXjz9SW3+dTtPoqct1KvGMwkIC9cWEftZTwH+cSJKTxWKtr1ezgqaN7KFRk5dpxKSlqhDkr7nvPaOqwdeuyDTgqSY6dz5dg0YvUEraedWvWVFfTOirwv+Q/0FCTx3j8aa1lZicpjGfXOlr9ZBALfqw77W+nkyWk9N1fa1RQZ+M6KHRU5Zp5KRlV/r6bm9VrXitr88/1URnL2Rc19cKWvThP6ev7KuO0bZpbSWeTtPY6/bVz/+X874aXqOCpv69r46afGVfnTOut6pct6+u3Lxb/UfMs/7ee+hsSdJLvZprSO8WeTMwE/H4d4zFm/apuLe7Xuv2L5XwLardv59S+zcWKeH0lU8ClSnhZf2ujSS5uTrr9e7/VrmAYjp7PkOrtx1Sn3eXK/Xsta9zPN3yyuuh5eM629xX3/e/1YI1OX8vvyDh2Gp/rf/zgBJPn9V7M1YoISlVVYMD9dl7/7Ve6OPPk8k2Pa0TVl4fD3tK46Yt1zufLFP5Mv6aPvpphVYoZa1p/lANjRn8hD6eu0ZvfviVKt7nr09G/J/Ca1TI8/HdKotxp5/Ps4MePXro5MmT1qslJicn6+OPP9bkyZO1bt06RUREZFkmLi5O5cuX144dO1SrVi1dunRJp0+fVrFixeTs7Kzo6Gjdf//9Onz4sMqVK5elfvjw4Tp37pxatGihsmXL6vTp05owYYKioqK0Y8cOVa5cWenp6WratKl+++03jRgxQv/617/k6empX375Re+8845mzJihWrVq3XR8qamp8vb21snEFFPPlgG3wsRDQYFmue6JBPZxOYcLDeDuOLGr2p1vi3fN3oQCKXnFy2ZvQoFz9sKlmxfhtpxJTVX5QD+lpNw8B5h+hmzlypUqVepKqvX09FRoaKgWLVqUbRjLjrOzs4oXL37L99eoUSNNnDhRTz31lE6ePCkfHx/df//9+u6771S58pX/LeHm5qbVq1dr/Pjxmjp1qgYPHqwiRYqoSpUqev7552/ro5EAAAAAkBNTz5AVdJwhw72EQ4FjcIbM/jhD5hicIbM/zpA5BmfI7I8zZPZ3O2fITL+oBwAAAAD8UxHIAAAAAMAkBDIAAAAAMAmBDAAAAABMQiADAAAAAJMQyAAAAADAJAQyAAAAADAJgQwAAAAATEIgAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAExCIAMAAAAAkxDIAAAAAMAkBDIAAAAAMAmBDAAAAABMQiADAAAAAJMQyAAAAADAJAQyAAAAADAJgQwAAAAATEIgAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAExCIAMAAAAAkxDIAAAAAMAkBDIAAAAAMAmBDAAAAABMQiADAAAAAJMQyAAAAADAJAQyAAAAADAJgQwAAAAATEIgAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAEzibPYGAMgfLBaL2ZsA3JJCTuyruDckr3jZ7E0okHzC+5u9CQVO8taPzN6EAudyxq3HLM6QAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQFWDTPt+oGq3fVEDDgWrS411t3xOXa/2SNb8qvP0IBTQcqAc7jdJ3P+yxmW8YhkZPWabQZq+p1L8G6bG+H+n3o6ccOIL8h546Bn21P3rqGPTV/uipY9BX++vV/t/auWS44jd/oNUzX9QDVcvmWOtcyEkvPd1Mv371puI3f6DN817Rf+pXsanxKOKm0YMe166v39LxTe9r1fRBur/KfY4eRr7DvkogK7C++m67hv5vsYb0aq4Nnw1R9UqBatd/ohKSzmRb//POQ+o1dLa6tWmgjXNf0aONaqrb4E+0N/a4tebDOWs0NWqjPni1k1bPGqwi7q5q13+iLqRfzKthmYqeOgZ9tT966hj01f7oqWPQV/tr2+QBjRzYVu9MX6GIp8bpt4N/6ssJfVXcxyPb+qHPtlSPtg015L0vVL/jKM366nt9Nq6XwkLKWGs+fL2LIuqFqs/wOWrYZYzW/bxPSyY+p1L+3nk1LNOxr16RrwNZjx49ZLFYrDc/Pz81a9ZMu3btynGZuLi4LMs88sgj2rFjh7UmIiLCpubqrU+fPtaa66d7eXmpbt26+vrrrx06XnuaNH+dnnrsQXVt3UChFUrpg1c7qUhhV8395sds66cu3KD/NKii559sosrlA/T6sy1VMzRI0xZtlHTl3YYpC9ZrcM9ItWhUQ9UrBWryW0/pxF8pWr5xZ14OzTT01DHoq/3RU8egr/ZHTx2Dvtpf3y6NNWfJj5q/7GftP3xCL4yN0rkLGerWqkG29R2ah2v87O+0esteHTmeqJlffq/VW/bqua4PS5IKu7modeOaGv7R19qy43cd/uMvvTNthQ4dS1DPdv/Ky6GZin31inwdyCSpWbNmio+PV3x8vNauXStnZ2e1bNnypsutWbNG8fHxWrVqldLS0tS8eXOdPn3aOr93797W9V69jRs3zmYds2bNUnx8vH755Rc1bNhQ7du31+7du+09RLvLuHhJ0fuOKSK8snWak5OTGoVX1rbdh7NdZuvuw4qoG2oz7eH6VbRtd5wk6cifiTqZmKqI8Gs13h7uql2tnLbtirP7GPIbeuoY9NX+6Klj0Ff7o6eOQV/tz8W5kGqFBmnDtv3WaYZhaOO2/aobVi7bZdxcnXUhw/aMzIX0i6pfs4KkKx9pdHYulENNRfsOIJ9iX70m3wcyNzc3BQQEKCAgQLVq1dIrr7yiY8eOKSEhIdfl/Pz8FBAQoDp16ui9997TyZMn9fPPP1vnFylSxLreqzcvLy+bdRQrVkwBAQEKCQnRiBEjdOnSJa1fvz7H+0xPT1dqaqrNzQyJp9N0+XKm/H09bab7+3rpVGL223QqMVX+fjfWe1rrT/7988aaEn6eOa6zIKGnjkFf7Y+eOgZ9tT966hj01f78ihWVs3MhJSTZjjUh6YxK+Hllu8y6n2LUt8vDqhDkL4vFoojwymrZuKZKFr9Sn3YuXVt3HdJLPZspoLiXnJws6tCsjuqGlbfWFHTsq9fk+0B2vbS0NM2dO1fBwcHy8/O75eXc3d0lSRkZGXd0v5cuXdKMGTMkSa6urjnWjRkzRt7e3tZbUFDQHd0fAAAA7l2vvP+lDh1L0NbPh+rUD+M17qUnNH/pT8rMNKw1/x32mSwWKebbUTr5/Xg90zFCX3633aYG/wz5PpAtW7ZMHh4e8vDwkKenp7755htFRUXJyenWNv306dMaMWKEPDw8FB4ebp0+adIk63qv3ubNm2ezbOfOneXh4SE3NzcNGjRI5cqVU4cOHXK8r1dffVUpKSnW27Fjx+5s0HfJr5iHChVyyvKFyISk1BzfySnh56WExBvrr73zU/LvnzfWnErM+d2hgoSeOgZ9tT966hj01f7oqWPQV/tLPH1Wly5dlr+v7VivPzOTdZk0dXtpmgIbvagabYYp/ImROns+XXHHE601cX/+pZZ9JijwoRdVvdWbavJ/78nZuZCO/JmY7ToLGvbVa/J9IGvcuLGio6MVHR2trVu3KjIyUs2bN9eRI0fUvHlza5iqVq2azXIPPvigPDw85OPjo507dyoqKkolS5a0zu/atat1vVdvrVu3tlnH+PHjFR0drRUrVqhq1aqaPn26fH19c9xWNzc3eXl52dzM4OrirFqhQdp43WedMzMztWnbAdUNK5/tMuFh5W3qJWn9z/usn40uG+inkn5eNjWpaee1fU+c6tYoZ/cx5Df01DHoq/3RU8egr/ZHTx2DvtrfxUuXFb3vmBrVDbFOs1gseqhOiPW7SzlJz7ik+IQUORdyUqvGtbRiY9ZrEZy7kKGTiany9nTXf+qH6ttNOV+8riBhX73G2ewNuJmiRYsqODjY+vv06dPl7e2tadOmafr06Tp//rwkycXFxWa5qKgoVa1aVX5+fipWrFiW9Xp7e9usNzsBAQEKDg5WcHCwZs2apRYtWmjv3r0qUaLE3Q/Mwfp2eVh93/pM91e5Tw9UK6fJC9br7Pl0dW1VX5LUZ9gclfL31rDn2kiS/tspQi3/+z99PHetHvlXNX313XZFxxzV/17rLOnKgadP58Z6b+ZKVQjyV9lAP42eslwBxb31aKOapo0zL9FTx6Cv9kdPHYO+2h89dQz6an+T5q/XpGHdtCPmqH7dc0TPdopQUXc3zVv2kyRp8vAnFX/qtN6etFSSVLtaWZXyL6bdB/5Q6RLFNKR3czk5WfThZ2us63y4fqgssujg0VOqUKa43n7+MR2IO6l5S38yZYxmYF+9It8HshtZLBY5OTnp/PnzCgwMzLEuKChIFSva7yo14eHhql27tkaNGqUPP/zQbut1lMcfqa2/Tqdp9NTlOpV4RmEhgfpiQj/r6do/TiTJyWKx1terWUHTRvbQqMnLNGLSUlUI8tfc955R1eDS1poBTzXRufPpGjR6gVLSzqt+zYr6YkJfFXZzyXL/BRE9dQz6an/01DHoq/3RU8egr/a3eM2vKu7jodeeeVQl/Dy1+8Cfaj9gkvXjdmVK+th898vN1UWv93lU5QKL6+z5dK3esld9hs1Ratp5a42Xh7ve7NtKpUsUU3LqOS1dt1MjJy/VpcuZeT4+s7CvXmExDCPffnOwR48eOnnypGbNmiVJSk5O1scff6zJkydr3bp1ioiIyLJMXFycypcvrx07dqhWrVrZrjciIkIhISF6++23baa7ubnJx8dH0pXgt3jxYj322GPW+StWrFDbtm31+++/5xoGr0pNTZW3t7dOJqaY9vFFAACAgsQnvL/Zm1DgJG/9yOxNKHBSU1NV0s9bKSk3zwH5/jtkK1euVKlSpVSqVCnVq1dP27Zt06JFi7INY7dj2rRp1vVevXXu3DnXZZo1a6by5ctr1KhRd3XfAAAAACDl8zNk9zrOkAEAANgXZ8jsjzNk9legzpABAAAAQEFFIAMAAAAAkxDIAAAAAMAkBDIAAAAAMAmBDAAAAABMQiADAAAAAJMQyAAAAADAJAQyAAAAADAJgQwAAAAATEIgAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAExCIAMAAAAAkxDIAAAAAMAkBDIAAAAAMAmBDAAAAABMQiADAAAAAJMQyAAAAADAJAQyAAAAADAJgQwAAAAATEIgAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAExCIAMAAAAAkxDIAAAAAMAkBDIAAAAAMAmBDAAAAABMQiADAAAAAJMQyAAAAADAJAQyAAAAADAJgQwAAAAATEIgAwAAAACTEMgAAAAAwCTOZm8AAAAAcKuSt35k9iYUOD51nzN7Ewoc43LGLddyhgwAAAAATEIgAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAExCIAMAAAAAkxDIAAAAAMAkBDIAAAAAMAmBDAAAAABMQiADAAAAAJMQyAAAAADAJAQyAAAAADAJgQwAAAAATEIgAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAExCIAMAAAAAkxDIAAAAAMAkBDIAAAAAMAmBDAAAAABMQiADAAAAAJMQyAAAAADAJAQyAAAAADAJgQwAAAAATEIgAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAExCIAMAAAAAkxDIAAAAAMAkBDIAAAAAMAmBDAAAAABMQiADAAAAAJMQyAAAAADAJAQyAAAAADAJgQwAAAAATEIgK8Cmfb5RNVq/qYCGA9Wkx7vavicu1/ola35VePsRCmg4UA92GqXvfthjM98wDI2eskyhzV5TqX8N0mN9P9LvR085cAT5Dz11DPpqf/TUMeir/dFTx6CvjkFf7evB+ytqwQf/1d5vRyl528dq0ajGTZdp+EAlbfhsiE78MF7bvxqmzi3rZanp9cRD2vn1W4r/frxWzxqsB6qWdcTm2809Fch69Oihxx57LMf5ERERGjhwYI7zk5KSNHDgQJUtW1aurq4qXbq0evbsqaNHj2apPXHihPr3768KFSrIzc1NQUFBatWqldauXWuHkTjeV99t19D/LdaQXs214bMhql4pUO36T1RC0pls63/eeUi9hs5WtzYNtHHuK3q0UU11G/yJ9sYet9Z8OGeNpkZt1AevdtLqWYNVxN1V7fpP1IX0i3k1LFPRU8egr/ZHTx2DvtofPXUM+uoY9NX+iri76bcDf+qlcVG3VH9faT9F/a+PNm8/oIe6jtWUBes14fUuerh+FWtN26YPaOTAtnpn+gpFPPmOfjv4p778qJ+K+3g4ahh37Z4KZHcjKSlJ9evX15o1azRlyhTFxsZq4cKFio2NVd26dXXo0CFrbVxcnGrXrq1169bp3Xff1e7du7Vy5Uo1btxY/fr1M3EUt27S/HV66rEH1bV1A4VWKKUPXu2kIoVdNfebH7Otn7pwg/7ToIqef7KJKpcP0OvPtlTN0CBNW7RR0pV3cKYsWK/BPSPVolENVa8UqMlvPaUTf6Vo+cadeTk009BTx6Cv9kdPHYO+2h89dQz66hj01f7WbNmrUVOWafmGXbdU3/Pxf+no8US98b/FOhB3UtMWbdI366L1bJfG1pq+XR7WnCVbNH/pT9p/+IReGLNQ5y5kqFvrBo4axl37xwSy119/XcePH9eaNWvUvHlz3XfffXrooYe0atUqubi42AStvn37ymKxaOvWrWrXrp1CQkJUrVo1vfDCC/rpp59MHMWtybh4SdH7jikivLJ1mpOTkxqFV9a23YezXWbr7sOKqBtqM+3h+lW0bXecJOnIn4k6mZiqiPBrNd4e7qpdrZy27Yqz+xjyG3rqGPTV/uipY9BX+6OnjkFfHYO+5g91w8prw9b9NtPW/hSj8LDykiQX50KqFRpkU2MYhjZu3a+6f9fkR/+IQJaZmamFCxeqa9euCggIsJnn7u6uvn37atWqVUpKSlJSUpJWrlypfv36qWjRolnWVaxYsRzvJz09XampqTY3MySeTtPly5ny9/W0me7v66VTidlv06nEVPn73Vjvaa0/+ffPG2tK+HnmuM6ChJ46Bn21P3rqGPTV/uipY9BXx6Cv+UMJP68sHxFNSEyVl4e7Cru5yK+Yh5ydC2WtSUpVCT+vvNzU2/KPCGQJCQk6ffq0qlSpku38KlWqyDAMxcbGKjY2VoZhKDQ0NNva3IwZM0be3t7WW1BQ0N1uOgAAAIAC7J4MZPPmzZOHh4f1tnnz5ltazjAMu9Tk5NVXX1VKSor1duzYsTte193wK+ahQoWcbuvdgRJ+XkpIvLH+jLW+5N8/b6w5lXgmX7/jYC/01DHoq/3RU8egr/ZHTx2DvjoGfc0fTiWmZj1L6eel1LTzupB+UYmn03Tp0uXbOpOZH9yTgax169aKjo623urUqZNrvb+/v4oVK6aYmJhs58fExMhisSg4OFiVKlWSxWLRvn37bnu73Nzc5OXlZXMzg6uLs2qFBmnjtmufn83MzNSmbQdy/PxseFh5m3pJWv/zPtUNKydJKhvop5J+XjY1qWnntX1PnOrWKGf3MeQ39NQx6Kv90VPHoK/2R08dg746Bn3NH7btPqxGdSvbTGscHqqtf3+P7+Kly4red8ymxmKx6KG6ITl+1y8/uCcDmaenp4KDg603d3f3XOudnJzUoUMHzZ8/XydOnLCZd/78eU2aNEmRkZHy9fWVr6+vIiMjNXHiRJ09ezbLuk6fPm3PoTjM1SvMLFj29xVmxkbp7Pl0dW1VX5LUZ9gcvfXx19b6/3aK0Nof9+rjuWt1IO6Exn6yXNExR9X7iUaSruzMfTo31nszV+rbjbu0J/ZPPTv8MwUU99ajjWqaMsa8Rk8dg77aHz11DPpqf/TUMeirY9BX+yvq7qrqIYGqHhIoSSpb2k/VQwJVpqSPJOnNfq01efiT1vqZX32vsoF+eqt/G1UqW1JPt/+3HmtyvybPX2+tuXo1zE6P1lNIuZL64JWOKurupnlL8++F+ZzN3gB7S0hIUHR0tM20UqVKafTo0Vq7dq2aNm2qcePGqXr16jp8+LCGDh2qixcvauLEidb6iRMnqmHDhgoPD9fbb7+tGjVq6NKlS1q9erUmT56c45m2/OTxR2rrr9NpGj11uU4lnlFYSKC+mNDPegr8jxNJcrJYrPX1albQtJE9NGryMo2YtFQVgvw1971nVDW4tLVmwFNNdO58ugaNXqCUtPOqX7OivpjQV4XdXPJ8fGagp45BX+2PnjoGfbU/euoY9NUx6Kv91apSVsumDrD+PvqFdpKk+ct+Ur+35qpkcS+VCfC1zj96PFEdB07R6Bce1387Rej4qdN6ftR8rfvp2mvzxat/VfFiHnrtv4+qhJ+ndh/4U+2fz/n/xeUHFuNuvjSVx3r06KHTp09ryZIl2c6PiIjQxo0bs0wfMWKEhg4dqr/++ktvv/22lixZohMnTsjX11fNmzfXW2+9pfvuu89mmfj4eI0aNUrLli1TfHy8/P39Vbt2bQ0aNEgRERG3tL2pqany9vbWycQU0z6+CAAAAOTGp+5zZm9CgWNczlD67mlKSbl5DrinAtm9hkAGAACA/I5AZn+3E8juye+QAQAAAEBBQCADAAAAAJMQyAAAAADAJAQyAAAAADAJgQwAAAAATEIgAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAExCIAMAAAAAkxDIAAAAAMAkBDIAAAAAMAmBDAAAAABMQiADAAAAAJMQyAAAAADAJAQyAAAAADAJgQwAAAAATEIgAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAExCIAMAAAAAkxDIAAAAAMAkBDIAAAAAMAmBDAAAAABMQiADAAAAAJMQyAAAAADAJAQyAAAAADAJgQwAAAAATEIgAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAExCIAMAAAAAkxDIAAAAAMAkzmZvQEFmGIYk6UxqqslbAgAAAGTPuJxh9iYUOFd7ejUP5IZA5kBnzpyRJAWXDzJ5SwAAAADktTNnzsjb2zvXGotxK7ENdyQzM1PHjx+Xp6enLBaL2ZuTq9TUVAUFBenYsWPy8vIye3MKDPpqf/TUMeir/dFTx6Cv9kdPHYO+Osa90lfDMHTmzBmVLl1aTk65f0uMM2QO5OTkpDJlypi9GbfFy8srX+/c9yr6an/01DHoq/3RU8egr/ZHTx2DvjrGvdDXm50Zu4qLegAAAACASQhkAAAAAGASAhkkSW5ubho2bJjc3NzM3pQChb7aHz11DPpqf/TUMeir/dFTx6CvjlEQ+8pFPQAAAADAJJwhAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAExCILuH9ejRQxaLxXrz8/NTs2bNtGvXrhyXiYuLk8ViUXR0dI41W7ZsUYsWLeTj46PChQsrLCxMH3zwgS5fvpyldv369WrRooX8/PxUpEgRVa1aVS+++KL+/PNPewwxT91JP6/as2ePOnToIH9/f7m5uSkkJERvvvmmzp07Z1NXrlw56/qLFCmisLAwTZ8+Pcv6DMPQtGnT1KBBA3l5ecnDw0PVqlXTgAEDFBsba7cx56UePXrosccey7Xm/PnzGjZsmEJCQuTm5qbixYvriSee0J49e2zqhg8fbu1joUKFFBQUpGeeeUZJSUlZ1rljxw517NhRpUqVkpubm8qWLauWLVtq6dKlKkjXNLpZfyMiIqw9K1y4sKpWrapJkyZZ58+ePdtm/7++9vr7uDrdxcVF5cuX18svv6wLFy44cmh56lb7OHbs2CzzHn30UVksFg0fPtymfuDAgdbfDx8+rC5duqh06dIqXLiwypQpozZt2mjfvn026ypIx1Z7OXbsmHr27KnSpUvL1dVVZcuW1YABA5SYmGj2puWZu3neL1SoUJb9Jz4+Xs7OzrJYLIqLi7Opv/51wuLFi1W/fn15e3vL09NT1apVs9mvJSkjI0Pjxo1TzZo1VaRIERUvXlwNGzbUrFmzdPHiRXu1IE/cTZ+vX+aRRx7Rjh07rDXXH4evv/Xp08dac/10Ly8v1a1bV19//bVDx5uXbuUYe+O+db2kpCQNHDhQZcuWlaurq0qXLq2ePXvq6NGjWWpPnDih/v37q0KFCnJzc1NQUJBatWqltWvX2mEk9kUgu8c1a9ZM8fHxio+P19q1a+Xs7KyWLVve8foWL16sRo0aqUyZMlq/fr327dunAQMGaOTIkerUqZPNC9ipU6eqSZMmCggI0Jdffqm9e/dqypQpSklJ0fvvv2+P4eW5O+nnTz/9pHr16ikjI0PLly/XgQMHNGrUKM2ePVtNmzZVRkaGTf3bb7+t+Ph4/fbbb+rWrZt69+6tFStWWOcbhqEuXbro+eefV4sWLfTdd99p7969mjFjhgoXLqyRI0c6ZOxmS09PV5MmTTRz5kyNHDlSBw4c0LfffqtLly6pXr16+umnn2zqq1Wrpvj4eB09elSzZs3SypUr9eyzz9rUfP3116pfv77S0tL06aefKiYmRitXrlTbtm01dOhQpaSk5OUQTde7d2/Fx8dr79696tChg/r166cFCxZY53t5eVn3/6u3I0eO2Kzj6mPk0KFDGj9+vKZOnaphw4bl9VBMFRQUpNmzZ9tM+/PPP7V27VqVKlUqx+UuXryopk2bKiUlRV999ZX279+vqKgohYWF6fTp09a6gnhsvVuHDh1SnTp1dPDgQS1YsECxsbGaMmWK1q5dqwYNGmT7ZkxBdafP+4GBgZozZ47NtE8//VSBgYG5Lrd27Vp17NhR7dq109atW7V9+3aNGjXKJmRlZGQoMjJSY8eO1TPPPKMtW7Zo69at6tevnz766KMsb6rdC+60z2vWrFF8fLxWrVqltLQ0NW/e3ObxffU4fP1t3LhxNuuYNWuW4uPj9csvv6hhw4Zq3769du/ebe8h3nOSkpJUv359rVmzRlOmTFFsbKwWLlyo2NhY1a1bV4cOHbLWxsXFqXbt2lq3bp3effdd7d69WytXrlTjxo3Vr18/E0eRAwP3rO7duxtt2rSxmbZ582ZDknHq1Klslzl8+LAhydixY0eWeWlpaYafn5/x+OOPZ5n3zTffGJKMhQsXGoZhGMeOHTNcXV2NgQMHZns/ycnJtzWW/OBO+pmZmWlUrVrVqFOnjnH58mWbedHR0YbFYjHGjh1rnVa2bFlj/PjxNnW+vr7GoEGDrL8vWLDAkGR8/fXXOd7nvSi7/l5v7NixhsViMaKjo22mX7582ahTp45RtWpV69iHDRtm1KxZ06buhRdeMHx8fKy/X92f27Ztm+N93qu9zM7N+tuoUSNjwIABNtMqVapkdOrUyTAMw5g1a5bh7e192/fx+OOPG/fff/8dbHH+dCt9fPbZZw0/Pz/j+++/t04fNWqU0apVK6NmzZrGsGHDbOqv9n3Hjh2GJCMuLi7H9RfEY6s9NGvWzChTpoxx7tw5m+nx8fFGkSJFjD59+pi0ZXnrbp73hw4dalSqVMlmXkhIiPHGG28YkozDhw/b1F99nTBgwAAjIiIi1+165513DCcnJ+PXX3/NMi8jI8NIS0u7tQHmE/Z6ffXDDz8YkoyVK1cahpH9cfhGkozFixdbf09NTTUkGR9++OGdDCXfuZPnqqv69OljFC1a1IiPj7eZfu7cOSMwMNBo1qyZdVrz5s2NwMDAbPe9/Hgc5QxZAZKWlqa5c+cqODhYfn5+t738d999p8TERA0ePDjLvFatWikkJMT6bvqiRYuUkZGhl19+Odt1FStW7LbvP7+5lX5GR0dr7969euGFF+TkZPtwqlmzppo0aWJzBuJ6mZmZ+vLLL5WcnCxXV1fr9AULFqhy5cpq3bp1tstZLJY7HFH+Nn/+fDVt2lQ1a9a0me7k5KRBgwZp79692rlzZ7bLxsXFadWqVTZ9vLo/57SPSgW3l7fK3d09yxnc2/Hbb79py5YtNn3/J3B1dVXXrl01a9Ys67TZs2erZ8+euS7n7+8vJycnffHFF9l+BFz6Zxxbb1dSUpJWrVqlvn37yt3d3WZeQECAunbtqqioqAL1EeRbdTvP+61bt1ZycrK+//57SdL333+v5ORktWrVKtflAgICtGfPHv3222851sybN09NmjTR/fffn2Wei4uLihYtegujyb/u9PXV1f31To+zly5d0owZMyTpH3ecvVFmZqYWLlyorl27KiAgwGaeu7u7+vbtq1WrVikpKUlJSUlauXKl+vXrl+2+lx+PowSye9yyZcvk4eEhDw8PeXp66ptvvlFUVFSWcHArDhw4IEmqUqVKtvNDQ0OtNQcPHpSXl1euH8+5F91uP2/WsypVqlhrrhoyZIg8PDzk5uam9u3by8fHR7169bJZZ+XKlW2WGThwoHW7ypQpczdDzLcOHDiQax+v1ly1e/dueXh4yN3dXeXLl9eePXs0ZMgQm/VJsunltm3brH308PDQsmXLHDGUfO/y5cuaO3eudu3apYcfftg6PSUlxaY/Hh4eat68uc2yVx8jV79feurUKb300kt5PQTT9ezZU59//rnOnj2rTZs2KSUl5aYfZwoMDNSECRP05ptvysfHRw8//LBGjBhh8zGbgnpsvRsHDx6UYRi5Hh+Sk5OVkJCQx1tmjjt93ndxcVG3bt00c+ZMSdLMmTPVrVs3ubi45Lpc//79VbduXYWFhalcuXLq1KmTZs6cqfT0dGvNwYMHFRoaeveDy0fu9vXV6dOnNWLECHl4eCg8PNw6fdKkSVmOs/PmzbNZtnPnztbXCYMGDVK5cuXUoUMHu47vXpOQkKDTp0/nehwwDEOxsbGKjY2VYRj31D5JILvHNW7cWNHR0YqOjtbWrVsVGRmp5s2b68iRI2revLn1wV6tWrVbXuetvMtoGEaBPLtwp/28nXdmX3rpJUVHR2vdunWqV6+exo8fr+Dg4FyXef311xUdHa0333xTaWlpdzS2/GLevHk2T0SbN2+2zrudPlauXFnR0dHatm2bhgwZosjISPXv3z/XZWrUqGH9+549e1aXLl2643HkV7n19+oLAXd3d/Xu3VuDBg2y+d6dp6entT9XbzdedObqY+Tnn39W9+7d9X//939q165dno0vr+TWR+nKGfBKlSrpiy++0MyZM/Xkk0/K2dn5puvt16+fTpw4oXnz5qlBgwZatGiRqlWrptWrV0squMdWe/gnngHLzt087/fs2VOLFi3SiRMntGjRopue1ZWkokWLavny5YqNjdXQoUPl4eGhF198UeHh4dYLVxXEv82d9vnBBx+Uh4eHfHx8tHPnTkVFRalkyZLW+V27ds1ynL3xEzHjx49XdHS0VqxYoapVq2r69Ony9fXNk3HnlZsdY3Nyq69R7zU3f/ZAvla0aFGbF/PTp0+Xt7e3pk2bpunTp+v8+fOSdNN3wCQpJCREkhQTE6MHH3wwy/yYmBhVrVrVWpuSkqL4+PgC9U7u7fbz+p5l91GNmJgYa81VxYsXV3BwsIKDg7Vo0SKFhYWpTp061t5WqlRJ+/fvt1nG399f/v7+KlGihP0Ga5LWrVurXr161t+vfqE8JCREMTEx2S5zdfr1vXR1dbX+rcaOHatHH31Ub731lkaMGCHpSh8laf/+/apfv74kyc3N7abh916XU3+lKy8EXn/9dbm7u6tUqVJZ3ul1cnK6aX+uf4zMnDlTNWvW1IwZM/T000/bcRTmy62PV/Xs2VMTJ07U3r17tXXr1ltet6enp1q1aqVWrVpp5MiRioyM1MiRI9W0adMCe2y9G8HBwbJYLIqJiVHbtm2zzI+JiZGPj4/8/f1N2Lq8dzfP+2FhYQoNDVXnzp1VpUoVVa9ePderLl+vYsWKqlixonr16qXXX39dISEhioqK0v/93/8pJCQky5VC73V32ueoqChVrVpVfn5+2X40ztvb+6bH2YCAAOvrhFmzZqlFixbau3dvgXgNcNWtHGOv5+/vr2LFiuX6OsFisVh7a7FY7ql9kjNkBYzFYpGTk5POnz+vwMBA6wO6bNmyN132kUceka+vb7ZX8frmm2908OBBde7cWZLUvn17ubq6Zrky0FXXX1HoXnazftaqVUuhoaEaP368MjMzbZbduXOn1qxZY+1ZdoKCgtSxY0e9+uqr1mmdO3fW/v37C9Rlbq/n6elp7WNwcLD1M/adOnXSmjVrsnxPLDMzU+PHj1fVqlWzfL/sekOHDtV7772n48ePS7q2P7/zzjuOG0w+lFN/pWsvBAIDA+/oY803cnJy0muvvaahQ4daX5wUFLn18aouXbpo9+7dql69uvUNldtlsVgUGhqqs2fPSvrnHFtvh5+fn5o2bapJkyZl2c+unm3s2LHjP/bM4u0+7/fs2VMbNmy4pbNjOSlXrpyKFCli3W+7dOmiNWvW2Fzi/aqLFy9a6+5lt9rnoKAgVaxY0W7fUwoPD1ft2rU1atQou6wvv7iVY+z1nJyc1KFDB82fP18nTpywmXf+/HlNmjRJkZGR8vX1la+vryIjIzVx4sRs9738eBwlkN3j0tPTdeLECZ04cUIxMTHq37+/0tLSbvol3f3792c5Ze7q6qqpU6fq66+/1jPPPKNdu3YpLi5OM2bMUI8ePdS+fXvrZ5iDgoI0fvx4ffjhh3r66ae1ceNGHTlyRD/88IP++9//Ws9S3Gtut58Wi0UzZszQ3r17rZcEPnr0qBYtWqRWrVqpQYMGuf4/DUkaMGCAli5dql9++UXSlWDSvn17derUSW+//bZ+/vlnxcXFaePGjYqKilKhQoXsPex8YdCgQQoPD1erVq20aNEiHT16VNu2bVO7du0UExOjGTNm5PqCq0GDBqpRo4ZGjx4tSfLw8ND06dO1fPlyPfroo1q1apUOHTqkXbt2WV/sFtRe3inDMKz7//W3G99suN4TTzyhQoUKaeLEiXm4pfmDj4+P9ZLYtyI6Olpt2rTRF198ob179yo2NlYzZszQzJkz1aZNG0kF99h6tz7++GOlp6crMjJSmzZt0rFjx7Ry5Uo1bdpUgYGBBe7Fam7u9Hn/qt69eyshIcHmu8u5GT58uF5++WVt2LBBhw8f1o4dO9SzZ0/rv3GQrnzPuWHDhvrPf/6jiRMnaufOnTp06JA+//xz1a9fXwcPHrzj8Zrlbvuck3PnzmU5xiYnJ+e6zMCBAzV16tR/zP8hTEhIyPIa9eTJkxo9erQCAgLUtGlTrVixQseOHdOmTZsUGRmpixcv2jwPTZw4UZcvX1Z4eLi+/PJLHTx4UDExMZowYYIaNGhg4uhyYMalHWEf3bt3NyRZb56enkbdunWNL774Isdlrl6WNbvbsWPHDMMwjE2bNhmRkZGGl5eX4erqalSrVs147733jEuXLmVZ3+rVq43IyEjDx8fHKFy4sBEaGmoMHjzYOH78uMPG7Sh30s+rdu3aZbRr187w9fU1XFxcjIoVKxpDhw41zp49a1OX3WXvDcMwIiMjjebNm1t/v3z5sjFlyhSjXr16RtGiRQ1XV1ejQoUKRu/evY29e/fe9VjNcLNL3RqGYZw9e9Z4/fXXjeDgYMPFxcXw9fU12rVrZ+zevdumLrvL3hvGlX8Z4ObmZhw9etQ6bdu2bUb79u2NEiVKGM7Ozoafn58RGRlpLFy48B9/2fvrzZo1K8djw9VLDOd0H2PGjDH8/f3vuUtbZ+du+5jbZe8TEhKM559/3qhevbrh4eFheHp6GmFhYcZ7772X5d9mFKRjq73ExcUZ3bt3N0qWLGm4uLgYQUFBRv/+/Y2//vrL7E3LM3fzvJ/dv7sxjGv/jiGny96vW7fOaNeunREUFGS4uroaJUuWNJo1a2Zs3rzZZj0XLlwwxowZY4SFhRmFCxc2fH19jYYNGxqzZ882Ll68aI/h5xlH9NkwrhwPsjvGRkZGWmt0w2XvDePKv2gJDQ01nn322bsdmulu5RibXY9GjBhhGMaV42j//v2NoKAgw8XFxShZsqTRo0cP48iRI1nWdfz4caNfv35G2bJlDVdXVyMwMNBo3bq1sX79egeN7s5ZDOMe/OYbAAAAABQAfGQRAAAAAExCIAMAAAAAkxDIAAAAAMAkBDIAAAAAMAmBDAAAAABMQiADAAAAAJMQyAAAAADAJAQyAAAAADAJgQwA8I/Wo0cPPfbYY9bfIyIiNHDgwDzfjg0bNshisej06dM51lgsFi1ZsuSW1zl8+HDVqlXrrrYrLi5OFotF0dHRd7UeAED2CGQAgHynR48eslgsslgscnV1VXBwsN5++21dunTJ4ff91VdfacSIEbdUeyshCgCA3DibvQEAAGSnWbNmmjVrltLT0/Xtt9+qX79+cnFx0auvvpqlNiMjQ66urna5X19fX7usBwCAW8EZMgBAvuTm5qaAgACVLVtWzz77rJo0aaJvvvlG0rWPGY4aNUqlS5dW5cqVJUnHjh1Thw4dVKxYMfn6+qpNmzaKi4uzrvPy5ct64YUXVKxYMfn5+enll1+WYRg293vjRxbT09M1ZMgQBQUFyc3NTcHBwZoxY4bi4uLUuHFjSZKPj48sFot69OghScrMzNSYMWNUvnx5ubu7q2bNmvriiy9s7ufbb79VSEiI3N3d1bhxY5vtvFVDhgxRSEiIihQpogoVKuiNN97QxYsXs9RNnTpVQUFBKlKkiDp06KCUlBSb+dOnT1eVKlVUuHBhhYaGatKkSbe9LQCAO0MgAwDcE9zd3ZWRkWH9fe3atdq/f79Wr16tZcuW6eLFi4qMjJSnp6c2b96sH374QR4eHmrWrJl1uffff1+zZ8/WzJkz9f333yspKUmLFy/O9X6feuopLViwQBMmTFBMTIymTp0qDw8PBQUF6csvv5Qk7d+/X/Hx8frwww8lSWPGjNGcOXM0ZcoU7dmzR4MGDVK3bt20ceNGSVeC4+OPP65WrVopOjpavXr10iuvvHLbPfH09NTs2bO1d+9effjhh5o2bZrGjx9vUxMbG6vPP/9cS5cu1cqVK7Vjxw717dvXOn/evHl68803NWrUKMXExGj06NF644039Omnn9729gAA7oABAEA+0717d6NNmzaGYRhGZmamsXr1asPNzc0YPHiwdX7JkiWN9PR06zKfffaZUblyZSMzM9M6LT093XB3dzdWrVplGIZhlCpVyhg3bpx1/sWLF40yZcpY78swDKNRo0bGgAEDDMMwjP379xuSjNWrV2e7nevXrzckGcnJydZpFy5cMIoUKWJs2bLFpvbpp582OnfubBiGYbz66qtG1apVbeYPGTIky7puJMlYvHhxjvPfffddo3bt2tbfhw0bZhQqVMj4448/rNNWrFhhODk5GfHx8YZhGEbFihWN+fPn26xnxIgRRoMGDQzDMIzDhw8bkowdO3bkeL8AgDvHd8gAAPnSsmXL5OHhoYsXLyozM1NdunTR8OHDrfPDwsJsvje2c+dOxcbGytPT02Y9Fy5c0O+//66UlBTFx8erXr161nnOzs6qU6dOlo8tXhUdHa1ChQqpUaNGt7zdsbGxOnfunJo2bWozPSMjQ/fff78kKSYmxmY7JKlBgwa3fB9XRUVFacKECfr999+VlpamS5cuycvLy6bmvvvuU2BgoM39ZGZmav/+/fL09NTvv/+up59+Wr1797bWXLp0Sd7e3re9PQCA20cgAwDkS40bN9bkyZPl6uqq0qVLy9nZ9imraNGiNr+npaWpdu3amjdvXpZ1+fv739E2uLu73/YyaWlpkqTly5fbBCHpyvfi7OXHH39U165d9dZbbykyMlLe3t5auHCh3n///dve1mnTpmUJiIUKFbLbtgIAckYgAwDkS0WLFlVwcPAt1z/wwAOKiopSiRIlspwluqpUqVL6+eef9dBDD0m6ciZo+/bteuCBB7KtDwsLU2ZmpjZu3KgmTZpkmX/1DN3ly5et06pWrSo3NzcdPXo0xzNrVapUsV6g5Kqffvrp5oO8zpYtW1S2bFm9/vrr1mlHjhzJUnf06FEdP35cpUuXtt6Pk5OTKleurJIlS6p06dI6dOiQunbtelv3DwCwDy7qAQAoELp27arixYurTZs22rx5sw4fPqwNGzbo+eef1x9//CFJGjBggMaOHaslS5Zo37596tu3b67/Q6xcuXLq3r27evbsqSVLlljX+fnnn0uSypYtK4vFomXLlikhIUFpaWny9PTU4MGDNWjQIH366af6/fff9euvv+qjjz6yXiijT58+OnjwoF566SXt379f8+fP1+zZs29rvJUqVdLRo0e1cOFC/f7775owYUK2FygpXLiwunfvrp07d2rz5s16/vnn1aFDBwUEBEiS3nrrLY0ZM0YTJkzQgQMHtHv3bs2aNUsffPDBbW0PAODOEMgAAAVCkSJFtGnTJt133316/PHHVaVKFT399NO6cOGC9YzZiy++qCeffFLdu3dXgwYN5OnpqbZt2+a63smTJ6t9+/bq27evQkND1bt3b509e1aSFBgYqLfeekuvvPKKSpYsqeeee06SNGLECL3xxhsaM2aMqlSpombNmmn58uUqX768pCvf6/ryyy+1ZMkS1axZU1OmTNHo0aNva7ytW7fWoEGD9Nxzz6lWrVrasmWL3njjjSx1wcHBevzxx9WiRQs98sgjqlGjhs1l7Xv16qXp06dr1qxZCgsLU6NGjTR79mzrtgIAHMti5PRNZgAAAACAQ3GGDAAAAABMQiADAAAAAJMQyAAAAADAJAQyAAAAADAJgQwAAAAATEIgAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAExCIAMAAAAAkxDIAAAAAMAk/w++kn+hMM/UgwAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 1000x1000 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "eval_token_list = list(set(eval_df_tokens['labels']))\n",
    "\n",
    "plot_confusion_matrix(eval_df_tokens[\"labels\"], eval_df_tokens[\"predicted_label\"],\n",
    "                      eval_token_list)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define & Call Function to Display Example Token Sequences Along With Labels & Losses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "      <th>14</th>\n",
       "      <th>15</th>\n",
       "      <th>16</th>\n",
       "      <th>17</th>\n",
       "      <th>18</th>\n",
       "      <th>19</th>\n",
       "      <th>20</th>\n",
       "      <th>21</th>\n",
       "      <th>22</th>\n",
       "      <th>23</th>\n",
       "      <th>24</th>\n",
       "      <th>25</th>\n",
       "      <th>26</th>\n",
       "      <th>27</th>\n",
       "      <th>28</th>\n",
       "      <th>29</th>\n",
       "      <th>30</th>\n",
       "      <th>31</th>\n",
       "      <th>32</th>\n",
       "      <th>33</th>\n",
       "      <th>34</th>\n",
       "      <th>35</th>\n",
       "      <th>36</th>\n",
       "      <th>37</th>\n",
       "      <th>38</th>\n",
       "      <th>39</th>\n",
       "      <th>40</th>\n",
       "      <th>41</th>\n",
       "      <th>42</th>\n",
       "      <th>43</th>\n",
       "      <th>44</th>\n",
       "      <th>45</th>\n",
       "      <th>46</th>\n",
       "      <th>47</th>\n",
       "      <th>48</th>\n",
       "      <th>49</th>\n",
       "      <th>50</th>\n",
       "      <th>51</th>\n",
       "      <th>52</th>\n",
       "      <th>53</th>\n",
       "      <th>54</th>\n",
       "      <th>55</th>\n",
       "      <th>56</th>\n",
       "      <th>57</th>\n",
       "      <th>58</th>\n",
       "      <th>59</th>\n",
       "      <th>60</th>\n",
       "      <th>61</th>\n",
       "      <th>62</th>\n",
       "      <th>63</th>\n",
       "      <th>64</th>\n",
       "      <th>65</th>\n",
       "      <th>66</th>\n",
       "      <th>67</th>\n",
       "      <th>68</th>\n",
       "      <th>69</th>\n",
       "      <th>70</th>\n",
       "      <th>71</th>\n",
       "      <th>72</th>\n",
       "      <th>73</th>\n",
       "      <th>74</th>\n",
       "      <th>75</th>\n",
       "      <th>76</th>\n",
       "      <th>77</th>\n",
       "      <th>78</th>\n",
       "      <th>79</th>\n",
       "      <th>80</th>\n",
       "      <th>81</th>\n",
       "      <th>82</th>\n",
       "      <th>83</th>\n",
       "      <th>84</th>\n",
       "      <th>85</th>\n",
       "      <th>86</th>\n",
       "      <th>87</th>\n",
       "      <th>88</th>\n",
       "      <th>89</th>\n",
       "      <th>90</th>\n",
       "      <th>91</th>\n",
       "      <th>92</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>tokens</th>\n",
       "      <td>the</td>\n",
       "      <td>former</td>\n",
       "      <td>us</td>\n",
       "      <td>biological</td>\n",
       "      <td>warfare</td>\n",
       "      <td>program</td>\n",
       "      <td>categorized</td>\n",
       "      <td>its</td>\n",
       "      <td>weapon</td>\n",
       "      <td>##ized</td>\n",
       "      <td>anti</td>\n",
       "      <td>-</td>\n",
       "      <td>personnel</td>\n",
       "      <td>bio</td>\n",
       "      <td>-</td>\n",
       "      <td>agents</td>\n",
       "      <td>as</td>\n",
       "      <td>either</td>\n",
       "      <td>\"</td>\n",
       "      <td>lethal</td>\n",
       "      <td>agents</td>\n",
       "      <td>\"</td>\n",
       "      <td>(</td>\n",
       "      <td>\"</td>\n",
       "      <td>ba</td>\n",
       "      <td>##ci</td>\n",
       "      <td>##llus</td>\n",
       "      <td>ant</td>\n",
       "      <td>##hra</td>\n",
       "      <td>##cis</td>\n",
       "      <td>\"</td>\n",
       "      <td>,</td>\n",
       "      <td>\"</td>\n",
       "      <td>francis</td>\n",
       "      <td>##ella</td>\n",
       "      <td>tu</td>\n",
       "      <td>##lar</td>\n",
       "      <td>##ensis</td>\n",
       "      <td>\"</td>\n",
       "      <td>,</td>\n",
       "      <td>bot</td>\n",
       "      <td>##ulin</td>\n",
       "      <td>##um</td>\n",
       "      <td>toxin</td>\n",
       "      <td>)</td>\n",
       "      <td>or</td>\n",
       "      <td>\"</td>\n",
       "      <td>inca</td>\n",
       "      <td>##pac</td>\n",
       "      <td>##itating</td>\n",
       "      <td>agents</td>\n",
       "      <td>\"</td>\n",
       "      <td>(</td>\n",
       "      <td>\"</td>\n",
       "      <td>bruce</td>\n",
       "      <td>##lla</td>\n",
       "      <td>sui</td>\n",
       "      <td>##s</td>\n",
       "      <td>\"</td>\n",
       "      <td>,</td>\n",
       "      <td>\"</td>\n",
       "      <td>cox</td>\n",
       "      <td>##iel</td>\n",
       "      <td>##la</td>\n",
       "      <td>burn</td>\n",
       "      <td>##eti</td>\n",
       "      <td>##i</td>\n",
       "      <td>\"</td>\n",
       "      <td>,</td>\n",
       "      <td>venezuelan</td>\n",
       "      <td>e</td>\n",
       "      <td>##quin</td>\n",
       "      <td>##e</td>\n",
       "      <td>en</td>\n",
       "      <td>##ce</td>\n",
       "      <td>##pha</td>\n",
       "      <td>##lit</td>\n",
       "      <td>##is</td>\n",
       "      <td>virus</td>\n",
       "      <td>,</td>\n",
       "      <td>st</td>\n",
       "      <td>##ap</td>\n",
       "      <td>##hyl</td>\n",
       "      <td>##oco</td>\n",
       "      <td>##cca</td>\n",
       "      <td>##l</td>\n",
       "      <td>enter</td>\n",
       "      <td>##oto</td>\n",
       "      <td>##xin</td>\n",
       "      <td>b</td>\n",
       "      <td>)</td>\n",
       "      <td>.</td>\n",
       "      <td>[SEP]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>labels</th>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>preds</th>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-ORG</td>\n",
       "      <td>I-ORG</td>\n",
       "      <td>I-ORG</td>\n",
       "      <td>I-ORG</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>losses</th>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>3.85</td>\n",
       "      <td>1.82</td>\n",
       "      <td>3.26</td>\n",
       "      <td>3.26</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>7.01</td>\n",
       "      <td>8.16</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.09</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.59</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.74</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.36</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.55</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.17</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>6.40</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>6.41</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.93</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.61</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.25</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.23</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.18</td>\n",
       "      <td>0.90</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.65</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.44</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.77</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.99</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.14</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          0       1      2           3        4        5            6     7    \n",
       "tokens   the  former     us  biological  warfare  program  categorized   its  \\\n",
       "labels     O       O  B-LOC           O        O        O            O     O   \n",
       "preds      O       O  B-ORG       I-ORG    I-ORG    I-ORG            O     O   \n",
       "losses  0.00    0.00   3.85        1.82     3.26     3.26         0.00  0.00   \n",
       "\n",
       "            8       9     10    11         12    13    14      15    16   \n",
       "tokens  weapon  ##ized  anti     -  personnel   bio     -  agents    as  \\\n",
       "labels       O     IGN     O   IGN        IGN     O   IGN     IGN     O   \n",
       "preds        O       O     O     O          O     O     O       O     O   \n",
       "losses    0.00    0.00  0.00  0.00       0.00  0.00  0.00    0.00  0.00   \n",
       "\n",
       "            17    18      19      20    21    22    23      24    25      26   \n",
       "tokens  either     \"  lethal  agents     \"     (     \"      ba  ##ci  ##llus  \\\n",
       "labels       O     O  I-MISC  I-MISC     O     O     O       O   IGN     IGN   \n",
       "preds        O     O       O       O     O     O     O  I-MISC     O       O   \n",
       "losses    0.00  0.00    7.01    8.16  0.00  0.00  0.00    1.09  0.00    0.00   \n",
       "\n",
       "          27     28      29    30    31    32       33      34    35     36   \n",
       "tokens   ant  ##hra   ##cis     \"     ,     \"  francis  ##ella    tu  ##lar  \\\n",
       "labels     O    IGN     IGN     O     O     O   I-MISC     IGN     O    IGN   \n",
       "preds      O      O  I-MISC     O     O     O   I-MISC  I-MISC     O      O   \n",
       "losses  0.59   0.00    0.00  0.00  0.00  0.00     0.74    0.00  0.36   0.00   \n",
       "\n",
       "             37    38    39      40      41    42     43    44    45    46   \n",
       "tokens  ##ensis     \"     ,     bot  ##ulin  ##um  toxin     )    or     \"  \\\n",
       "labels      IGN     O     O  I-MISC     IGN   IGN      O     O     O     O   \n",
       "preds    I-MISC     O     O       O       O     O      O     O     O     O   \n",
       "losses     0.00  0.00  0.00    1.55    0.00  0.00   0.17  0.00  0.00  0.00   \n",
       "\n",
       "            47     48         49      50    51    52    53      54      55   \n",
       "tokens    inca  ##pac  ##itating  agents     \"     (     \"   bruce   ##lla  \\\n",
       "labels  I-MISC    IGN        IGN  I-MISC     O     O     O       O     IGN   \n",
       "preds        O      O          O       O     O     O     O  I-MISC  I-MISC   \n",
       "losses    6.40   0.00       0.00    6.41  0.00  0.00  0.00    1.93    0.00   \n",
       "\n",
       "          56    57    58    59    60      61     62    63    64     65    66   \n",
       "tokens   sui   ##s     \"     ,     \"     cox  ##iel  ##la  burn  ##eti   ##i  \\\n",
       "labels     O   IGN     O     O     O       O    IGN   IGN     O    IGN   IGN   \n",
       "preds      O     O     O     O     O  I-MISC      O     O     O      O     O   \n",
       "losses  0.61  0.00  0.00  0.00  0.00    1.25   0.00  0.00  0.23   0.00  0.00   \n",
       "\n",
       "          67    68          69      70      71    72    73    74     75   \n",
       "tokens     \"     ,  venezuelan       e  ##quin   ##e    en  ##ce  ##pha  \\\n",
       "labels     O     O      I-MISC       O     IGN   IGN     O   IGN    IGN   \n",
       "preds      O     O      I-MISC  I-MISC  I-MISC     O     O     O      O   \n",
       "losses  0.00  0.00        0.18    0.90    0.00  0.00  0.65  0.00   0.00   \n",
       "\n",
       "           76    77      78    79      80    81     82     83     84    85   \n",
       "tokens  ##lit  ##is   virus     ,      st  ##ap  ##hyl  ##oco  ##cca   ##l  \\\n",
       "labels    IGN   IGN       O     O  I-MISC   IGN    IGN    IGN    IGN   IGN   \n",
       "preds       O     O  I-MISC     O       O     O      O      O      O     O   \n",
       "losses   0.00  0.00    1.44  0.00    1.77  0.00   0.00   0.00   0.00  0.00   \n",
       "\n",
       "            86     87     88      89    90    91     92  \n",
       "tokens   enter  ##oto  ##xin       b     )     .  [SEP]  \n",
       "labels  I-MISC    IGN    IGN  I-MISC     O     O    IGN  \n",
       "preds        O      O      O  I-MISC     O     O      O  \n",
       "losses    1.99   0.00   0.00    0.14  0.00  0.00   0.00  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "      <th>14</th>\n",
       "      <th>15</th>\n",
       "      <th>16</th>\n",
       "      <th>17</th>\n",
       "      <th>18</th>\n",
       "      <th>19</th>\n",
       "      <th>20</th>\n",
       "      <th>21</th>\n",
       "      <th>22</th>\n",
       "      <th>23</th>\n",
       "      <th>24</th>\n",
       "      <th>25</th>\n",
       "      <th>26</th>\n",
       "      <th>27</th>\n",
       "      <th>28</th>\n",
       "      <th>29</th>\n",
       "      <th>30</th>\n",
       "      <th>31</th>\n",
       "      <th>32</th>\n",
       "      <th>33</th>\n",
       "      <th>34</th>\n",
       "      <th>35</th>\n",
       "      <th>36</th>\n",
       "      <th>37</th>\n",
       "      <th>38</th>\n",
       "      <th>39</th>\n",
       "      <th>40</th>\n",
       "      <th>41</th>\n",
       "      <th>42</th>\n",
       "      <th>43</th>\n",
       "      <th>44</th>\n",
       "      <th>45</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>tokens</th>\n",
       "      <td>the</td>\n",
       "      <td>zodiac</td>\n",
       "      <td>traditionally</td>\n",
       "      <td>begins</td>\n",
       "      <td>with</td>\n",
       "      <td>the</td>\n",
       "      <td>sign</td>\n",
       "      <td>of</td>\n",
       "      <td>the</td>\n",
       "      <td>rat</td>\n",
       "      <td>,</td>\n",
       "      <td>and</td>\n",
       "      <td>the</td>\n",
       "      <td>cycle</td>\n",
       "      <td>proceeds</td>\n",
       "      <td>through</td>\n",
       "      <td>11</td>\n",
       "      <td>other</td>\n",
       "      <td>animals</td>\n",
       "      <td>signs</td>\n",
       "      <td>:</td>\n",
       "      <td>the</td>\n",
       "      <td>ox</td>\n",
       "      <td>,</td>\n",
       "      <td>tiger</td>\n",
       "      <td>,</td>\n",
       "      <td>rabbit</td>\n",
       "      <td>,</td>\n",
       "      <td>dragon</td>\n",
       "      <td>,</td>\n",
       "      <td>snake</td>\n",
       "      <td>,</td>\n",
       "      <td>horse</td>\n",
       "      <td>,</td>\n",
       "      <td>goat</td>\n",
       "      <td>,</td>\n",
       "      <td>monkey</td>\n",
       "      <td>,</td>\n",
       "      <td>rooster</td>\n",
       "      <td>,</td>\n",
       "      <td>dog</td>\n",
       "      <td>,</td>\n",
       "      <td>and</td>\n",
       "      <td>pig</td>\n",
       "      <td>.</td>\n",
       "      <td>[SEP]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>labels</th>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-PER</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>B-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>B-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>B-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>B-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>B-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>B-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>B-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>B-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>B-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-PER</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>preds</th>\n",
       "      <td>O</td>\n",
       "      <td>B-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>losses</th>\n",
       "      <td>0.00</td>\n",
       "      <td>3.98</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.03</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.00</td>\n",
       "      <td>4.26</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.69</td>\n",
       "      <td>0.00</td>\n",
       "      <td>2.29</td>\n",
       "      <td>0.00</td>\n",
       "      <td>2.86</td>\n",
       "      <td>0.00</td>\n",
       "      <td>3.74</td>\n",
       "      <td>0.00</td>\n",
       "      <td>4.46</td>\n",
       "      <td>0.00</td>\n",
       "      <td>4.38</td>\n",
       "      <td>0.00</td>\n",
       "      <td>4.61</td>\n",
       "      <td>0.00</td>\n",
       "      <td>3.62</td>\n",
       "      <td>0.00</td>\n",
       "      <td>4.03</td>\n",
       "      <td>0.00</td>\n",
       "      <td>5.16</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>6.21</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          0       1              2       3     4     5     6     7     8    \n",
       "tokens   the  zodiac  traditionally  begins  with   the  sign    of   the  \\\n",
       "labels     O       O              O       O     O     O     O     O     O   \n",
       "preds      O  B-MISC              O       O     O     O     O     O     O   \n",
       "losses  0.00    3.98           0.00    0.00  0.00  0.00  0.03  0.01  0.00   \n",
       "\n",
       "           9     10    11    12     13        14       15    16     17   \n",
       "tokens    rat     ,   and   the  cycle  proceeds  through    11  other  \\\n",
       "labels  B-PER     O     O     O      O         O        O     O      O   \n",
       "preds       O     O     O     O      O         O        O     O      O   \n",
       "losses   4.26  0.00  0.00  0.00   0.00      0.00     0.00  0.00   0.00   \n",
       "\n",
       "             18     19    20    21      22    23      24    25      26    27   \n",
       "tokens  animals  signs     :   the      ox     ,   tiger     ,  rabbit     ,  \\\n",
       "labels        O      O     O     O  B-MISC     O  B-MISC     O  B-MISC     O   \n",
       "preds         O      O     O     O       O     O       O     O       O     O   \n",
       "losses     0.00   0.00  0.00  0.00    1.69  0.00    2.29  0.00    2.86  0.00   \n",
       "\n",
       "            28    29      30    31      32    33      34    35      36    37   \n",
       "tokens  dragon     ,   snake     ,   horse     ,    goat     ,  monkey     ,  \\\n",
       "labels  B-MISC     O  B-MISC     O  B-MISC     O  B-MISC     O  B-MISC     O   \n",
       "preds        O     O       O     O       O     O       O     O       O     O   \n",
       "losses    3.74  0.00    4.46  0.00    4.38  0.00    4.61  0.00    3.62  0.00   \n",
       "\n",
       "             38    39      40    41    42     43    44     45  \n",
       "tokens  rooster     ,     dog     ,   and    pig     .  [SEP]  \n",
       "labels   B-MISC     O  B-MISC     O     O  B-PER     O    IGN  \n",
       "preds         O     O       O     O     O      O     O      O  \n",
       "losses     4.03  0.00    5.16  0.00  0.00   6.21  0.00   0.00  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "      <th>14</th>\n",
       "      <th>15</th>\n",
       "      <th>16</th>\n",
       "      <th>17</th>\n",
       "      <th>18</th>\n",
       "      <th>19</th>\n",
       "      <th>20</th>\n",
       "      <th>21</th>\n",
       "      <th>22</th>\n",
       "      <th>23</th>\n",
       "      <th>24</th>\n",
       "      <th>25</th>\n",
       "      <th>26</th>\n",
       "      <th>27</th>\n",
       "      <th>28</th>\n",
       "      <th>29</th>\n",
       "      <th>30</th>\n",
       "      <th>31</th>\n",
       "      <th>32</th>\n",
       "      <th>33</th>\n",
       "      <th>34</th>\n",
       "      <th>35</th>\n",
       "      <th>36</th>\n",
       "      <th>37</th>\n",
       "      <th>38</th>\n",
       "      <th>39</th>\n",
       "      <th>40</th>\n",
       "      <th>41</th>\n",
       "      <th>42</th>\n",
       "      <th>43</th>\n",
       "      <th>44</th>\n",
       "      <th>45</th>\n",
       "      <th>46</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>tokens</th>\n",
       "      <td>soon</td>\n",
       "      <td>the</td>\n",
       "      <td>scenes</td>\n",
       "      <td>were</td>\n",
       "      <td>split</td>\n",
       "      <td>into</td>\n",
       "      <td>two</td>\n",
       "      <td>registers</td>\n",
       "      <td>,</td>\n",
       "      <td>as</td>\n",
       "      <td>in</td>\n",
       "      <td>the</td>\n",
       "      <td>dog</td>\n",
       "      <td>##matic</td>\n",
       "      <td>sar</td>\n",
       "      <td>##co</td>\n",
       "      <td>##pha</td>\n",
       "      <td>##gus</td>\n",
       "      <td>or</td>\n",
       "      <td>the</td>\n",
       "      <td>sar</td>\n",
       "      <td>##co</td>\n",
       "      <td>##pha</td>\n",
       "      <td>##gus</td>\n",
       "      <td>of</td>\n",
       "      <td>jun</td>\n",
       "      <td>##ius</td>\n",
       "      <td>bass</td>\n",
       "      <td>##us</td>\n",
       "      <td>(</td>\n",
       "      <td>the</td>\n",
       "      <td>last</td>\n",
       "      <td>of</td>\n",
       "      <td>these</td>\n",
       "      <td>ex</td>\n",
       "      <td>##em</td>\n",
       "      <td>##plify</td>\n",
       "      <td>##ing</td>\n",
       "      <td>a</td>\n",
       "      <td>partial</td>\n",
       "      <td>revival</td>\n",
       "      <td>of</td>\n",
       "      <td>classic</td>\n",
       "      <td>##ism</td>\n",
       "      <td>)</td>\n",
       "      <td>.</td>\n",
       "      <td>[SEP]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>labels</th>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-MISC</td>\n",
       "      <td>IGN</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-MISC</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>IGN</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>preds</th>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-PER</td>\n",
       "      <td>I-PER</td>\n",
       "      <td>I-PER</td>\n",
       "      <td>I-PER</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>losses</th>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>3.09</td>\n",
       "      <td>0.00</td>\n",
       "      <td>6.24</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>6.50</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>8.36</td>\n",
       "      <td>9.87</td>\n",
       "      <td>0.00</td>\n",
       "      <td>10.45</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.29</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          0     1       2     3      4     5     6          7     8     9    \n",
       "tokens  soon   the  scenes  were  split  into   two  registers     ,    as  \\\n",
       "labels     O     O       O     O      O     O     O          O     O     O   \n",
       "preds      O     O       O     O      O     O     O          O     O     O   \n",
       "losses  0.00  0.00    0.00  0.00   0.00  0.00  0.00       0.00  0.00  0.00   \n",
       "\n",
       "          10    11      12       13      14    15     16     17    18    19   \n",
       "tokens    in   the     dog  ##matic     sar  ##co  ##pha  ##gus    or   the  \\\n",
       "labels     O     O  B-MISC      IGN  I-MISC   IGN    IGN    IGN     O     O   \n",
       "preds      O     O       O        O       O     O      O      O     O     O   \n",
       "losses  0.00  0.00    3.09     0.00    6.24  0.00   0.00   0.00  0.00  0.00   \n",
       "\n",
       "            20    21     22     23      24      25     26      27     28   \n",
       "tokens     sar  ##co  ##pha  ##gus      of     jun  ##ius    bass   ##us  \\\n",
       "labels  B-MISC   IGN    IGN    IGN  I-MISC  I-MISC    IGN  I-MISC    IGN   \n",
       "preds        O     O      O      O       O   B-PER  I-PER   I-PER  I-PER   \n",
       "losses    6.50  0.00   0.00   0.00    8.36    9.87   0.00   10.45   0.00   \n",
       "\n",
       "          29    30    31    32     33    34    35       36     37    38   \n",
       "tokens     (   the  last    of  these    ex  ##em  ##plify  ##ing     a  \\\n",
       "labels     O     O     O     O      O     O   IGN      IGN    IGN     O   \n",
       "preds      O     O     O     O      O     O     O        O      O     O   \n",
       "losses  0.00  0.00  0.00  0.00   0.00  0.00  0.00     0.00   0.00  0.00   \n",
       "\n",
       "             39       40    41       42     43    44    45     46  \n",
       "tokens  partial  revival    of  classic  ##ism     )     .  [SEP]  \n",
       "labels        O        O     O        O    IGN     O     O    IGN  \n",
       "preds         O        O     O   B-MISC      O     O     O      O  \n",
       "losses     0.00     0.00  0.00     1.29   0.00  0.00  0.00   0.00  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def get_samples(df):\n",
    "    for _, row in df.iterrows():\n",
    "        labels, preds, tokens, losses = [], [], [], []\n",
    "        for i, mask in enumerate(row[\"attention_mask\"]):\n",
    "            if i not in {0, len(row[\"attention_mask\"])}:\n",
    "                labels.append(row[\"labels\"][i])\n",
    "                preds.append(row[\"predicted_label\"][i])\n",
    "                tokens.append(row[\"input_tokens\"][i])\n",
    "                losses.append(f\"{row['loss'][i]:.2f}\")\n",
    "        eval_df_tmp = pd.DataFrame({\"tokens\": tokens, \"labels\": labels, \n",
    "                               \"preds\": preds, \"losses\": losses}).T\n",
    "        yield eval_df_tmp\n",
    "\n",
    "eval_df[\"total_loss\"] = eval_df[\"loss\"].apply(sum)\n",
    "eval_df_tmp = eval_df.sort_values(by=\"total_loss\", ascending=False).head(3)\n",
    "\n",
    "pd.set_option('display.max_columns', None)\n",
    "\n",
    "for sample in get_samples(eval_df_tmp):\n",
    "    display(sample)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Notes & Other Takeaways From This Project\n",
    "\n",
    "****\n",
    "- This model is part of a comparison of Token Classification Models using this particular dataset.\n",
    "\n",
    "****"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Citations\n",
    "\n",
    "- Model Checkpoint\n",
    "\n",
    "    > @article{2020_SqueezeBERT, author = {Forrest N. Iandola and Albert E. Shaw and Ravi Krishna and Kurt W. Keutzer}, title = {{SqueezeBERT}: What can computer vision teach NLP about efficient neural networks?}, journal = {arXiv:2006.11316}, year = {2020}}\n",
    "\n",
    "- Dataset\n",
    "\n",
    "    > @inproceedings{tedeschi-etal-2021-wikineural-combined, title = \"{W}iki{NE}u{R}al: {C}ombined Neural and Knowledge-based Silver Data Creation for Multilingual {NER}\", author = \"Tedeschi, Simone and Maiorca, Valentino and Campolungo, Niccol{\\`o} and Cecconi, Francesco and Navigli, Roberto\", booktitle = \"Findings of the Association for Computational Linguistics: EMNLP 2021\", month = nov, year = \"2021\", address = \"Punta Cana, Dominican Republic\", publisher = \"Association for Computational Linguistics\", url = \"https://aclanthology.org/2021.findings-emnlp.215\", pages = \"2521--2533\", abstract = \"Multilingual Named Entity Recognition (NER) is a key intermediate task which is needed in many areas of NLP. In this paper, we address the well-known issue of data scarcity in NER, especially relevant when moving to a multilingual scenario, and go beyond current approaches to the creation of multilingual silver data for the task. We exploit the texts of Wikipedia and introduce a new methodology based on the effective combination of knowledge-based approaches and neural models, together with a novel domain adaptation technique, to produce high-quality training corpora for NER. We evaluate our datasets extensively on standard benchmarks for NER, yielding substantial improvements up to 6 span-based F1-score points over previous state-of-the-art systems for data creation.\",}\n",
    "\n",
    "- Metric (PosEval)\n",
    "\n",
    "    > @article{scikit-learn, title={Scikit-learn: Machine Learning in {P}ython}, author={Pedregosa, F. and Varoquaux, G. and Gramfort, A. and Michel, V. and Thirion, B. and Grisel, O. and Blondel, M. and Prettenhofer, P. and Weiss, R. and Dubourg, V. and Vanderplas, J. and Passos, A. and Cournapeau, D. and Brucher, M. and Perrot, M. and Duchesnay, E.}, journal={Journal of Machine Learning Research}, volume={12}, pages={2825--2830}, year={2011}}"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "dl",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}