Token Classification/Monolingual/WikiNeural - Transformer Comparison/POS Project with Wikineural Dataset - MEGA Transformer.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Monolingual Named Entity Recognition Token Classification (WikiNeural Dataset)\n",
    "\n",
    "### MEGA Transformer\n",
    "\n",
    "Dataset Source: https://huggingface.co/datasets/Babelscape/wikineural"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Import Necessary Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Git LFS initialized.\n"
     ]
    }
   ],
   "source": [
    "import os, sys\n",
    "os.environ['TOKENIZERS_PARALLELISM']='false'\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import sklearn\n",
    "from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay\n",
    "\n",
    "import datasets\n",
    "from datasets import load_dataset, DatasetDict\n",
    "\n",
    "import torch\n",
    "from torch import nn\n",
    "from torch.nn.functional import cross_entropy\n",
    "\n",
    "import transformers\n",
    "from transformers import AutoTokenizer, AutoModelForTokenClassification\n",
    "from transformers import DataCollatorForTokenClassification\n",
    "from transformers import Trainer, TrainingArguments\n",
    "\n",
    "import evaluate\n",
    "\n",
    "!git lfs install"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Display Library Versions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "          Python : 3.9.12\n",
      "           NumPy : 1.24.3\n",
      "           Torch : 2.0.0\n",
      "          Pandas : 2.0.1\n",
      "    Transformers : 4.28.1\n",
      "        Datasets : 2.11.0\n",
      "         Sklearn : 1.2.2\n"
     ]
    }
   ],
   "source": [
    "print(\"Python :\".rjust(18), sys.version[0:6])\n",
    "print(\"NumPy :\".rjust(18), np.__version__)\n",
    "print(\"Torch :\".rjust(18), torch.__version__)\n",
    "print(\"Pandas :\".rjust(18), pd.__version__)\n",
    "print(\"Transformers :\".rjust(18), transformers.__version__)\n",
    "print(\"Datasets :\".rjust(18), datasets.__version__)\n",
    "print(\"Sklearn :\".rjust(18), sklearn.__version__)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Ingest English Subset of Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Found cached dataset parquet (/Users/briandunn/.cache/huggingface/datasets/Babelscape___parquet/Babelscape--wikineural-579d1dc98d2a6b93/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "265eca14fdab48ecaf3926d24f9b3118",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/27 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    train_es: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 76320\n",
       "    })\n",
       "    test_es: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 9618\n",
       "    })\n",
       "    test_de: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 12372\n",
       "    })\n",
       "    val_it: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 11050\n",
       "    })\n",
       "    train_en: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 92720\n",
       "    })\n",
       "    train_it: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 88400\n",
       "    })\n",
       "    train_nl: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 83680\n",
       "    })\n",
       "    train_pl: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 108160\n",
       "    })\n",
       "    test_pl: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 13585\n",
       "    })\n",
       "    val_ru: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 11540\n",
       "    })\n",
       "    test_nl: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 10547\n",
       "    })\n",
       "    train_fr: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 100800\n",
       "    })\n",
       "    val_de: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 12330\n",
       "    })\n",
       "    test_it: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 11069\n",
       "    })\n",
       "    test_fr: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 12678\n",
       "    })\n",
       "    val_en: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 11590\n",
       "    })\n",
       "    val_nl: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 10460\n",
       "    })\n",
       "    test_pt: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 10160\n",
       "    })\n",
       "    val_fr: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 12600\n",
       "    })\n",
       "    test_en: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 11597\n",
       "    })\n",
       "    train_pt: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 80560\n",
       "    })\n",
       "    test_ru: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 11580\n",
       "    })\n",
       "    val_pl: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 13520\n",
       "    })\n",
       "    val_pt: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 10070\n",
       "    })\n",
       "    train_de: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 98640\n",
       "    })\n",
       "    val_es: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 9540\n",
       "    })\n",
       "    train_ru: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 92320\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = load_dataset(\"Babelscape/wikineural\")\n",
    "\n",
    "data = data.remove_columns([\"lang\"])\n",
    "\n",
    "data"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Split Dataset into DatasetDict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training data shape: (92720, 2)\n",
      "Testing data shape: (11597, 2)\n",
      "Validation data shape: (11590, 2)\n"
     ]
    }
   ],
   "source": [
    "ds = DatasetDict({\n",
    "    'train': data['train_en'], \n",
    "    'test': data['test_en'], \n",
    "    'eval': data['val_en']})\n",
    "\n",
    "print('Training data shape:', ds['train'].shape)\n",
    "print('Testing data shape:', ds['test'].shape)\n",
    "print('Validation data shape:', ds['eval'].shape)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'tokens': ['It',\n",
       "  'has',\n",
       "  'a',\n",
       "  'leaf',\n",
       "  'pattern',\n",
       "  'similar',\n",
       "  'to',\n",
       "  'the',\n",
       "  'members',\n",
       "  'of',\n",
       "  'the',\n",
       "  'genera',\n",
       "  '\"',\n",
       "  'Kedrostis',\n",
       "  '\"',\n",
       "  ',',\n",
       "  '\"',\n",
       "  'Melothria',\n",
       "  '\"',\n",
       "  'and',\n",
       "  '\"',\n",
       "  'Zehneria',\n",
       "  '\"',\n",
       "  '.'],\n",
       " 'ner_tags': [0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  7,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  7,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  7,\n",
       "  0,\n",
       "  0]}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example = ds['train'][12]\n",
    "\n",
    "example"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Display Feature Information About Each Feature"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tokens: \n",
      "Sequence(feature=Value(dtype='string', id=None), length=-1, id=None)\n",
      "\n",
      "ner_tags: \n",
      "Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for k, v in ds[\"train\"].features.items():\n",
    "    print(f\"{k}: \\n{v}\\n\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define Tag Values & Conversions Between String & Integer Values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "List of tag values: \n",
      "['O', 'B-PER', 'I-PER', 'B-ORG', 'I-ORG', 'B-LOC', 'I-LOC', 'B-MISC', 'I-MISC']\n",
      "Number of NER Tags: \n",
      "9\n",
      "id2label: \n",
      "{0: 'O', 1: 'B-PER', 2: 'I-PER', 3: 'B-ORG', 4: 'I-ORG', 5: 'B-LOC', 6: 'I-LOC', 7: 'B-MISC', 8: 'I-MISC'}\n",
      "label2id: \n",
      "{'O': 0, 'B-PER': 1, 'I-PER': 2, 'B-ORG': 3, 'I-ORG': 4, 'B-LOC': 5, 'I-LOC': 6, 'B-MISC': 7, 'I-MISC': 8}\n"
     ]
    }
   ],
   "source": [
    "label2id = {\n",
    "    'O': 0, \n",
    "    'B-PER': 1, \n",
    "    'I-PER': 2, \n",
    "    'B-ORG': 3, \n",
    "    'I-ORG': 4, \n",
    "    'B-LOC': 5, \n",
    "    'I-LOC': 6, \n",
    "    'B-MISC': 7, \n",
    "    'I-MISC': 8\n",
    "    }\n",
    "\n",
    "id2label = {tag: idx for idx, tag in label2id.items()}\n",
    "\n",
    "pos_tag_values = list(label2id.keys())\n",
    "NUM_OF_LABELS = len(pos_tag_values)\n",
    "\n",
    "print(f\"List of tag values: \\n{pos_tag_values}\")\n",
    "print(f\"Number of NER Tags: \\n{NUM_OF_LABELS}\")\n",
    "print(f\"id2label: \\n{id2label}\")\n",
    "print(f\"label2id: \\n{label2id}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Basic Values/Constants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "MODEL_CKPT = \"mnaylor/mega-base-wikitext\"\n",
    "MODEL_NAME = f\"{MODEL_CKPT.split(f'/')[-1]}-finetuned-WikiNeural\"\n",
    "\n",
    "NUM_OF_EPOCHS = 2\n",
    "BATCH_SIZE = 16\n",
    "\n",
    "STRATEGY = \"epoch\"\n",
    "REPORTS_TO = \"tensorboard\"\n",
    "\n",
    "WEIGHT_DECAY = 0.01\n",
    "LR = 2e-5\n",
    "\n",
    "DEVICE = torch.device(\"cpu\")\n",
    "STEPS = 1250"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Function to Tokenize & Align Inputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = AutoTokenizer.from_pretrained(MODEL_CKPT,\n",
    "                                          add_prefix_space=True)\n",
    "\n",
    "def tokenize_and_align_labels(samples):\n",
    "    tokenized_inputs = tokenizer(samples[\"tokens\"], \n",
    "                                      truncation=True, \n",
    "                                      is_split_into_words=True)\n",
    "    \n",
    "    labels = []\n",
    "    \n",
    "    for idx, label in enumerate(samples[f\"ner_tags\"]):\n",
    "        word_ids = tokenized_inputs.word_ids(batch_index=idx)\n",
    "        prev_word_idx = None\n",
    "        label_ids = []\n",
    "        for word_idx in word_ids: # set special tokens to -100\n",
    "            if word_idx is None or word_idx == prev_word_idx:\n",
    "                label_ids.append(-100)\n",
    "            else:\n",
    "                label_ids.append(label[word_idx])\n",
    "            prev_word_idx = word_idx\n",
    "        labels.append(label_ids)\n",
    "    \n",
    "    tokenized_inputs[\"labels\"] = labels\n",
    "    return tokenized_inputs"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Apply Above Function to Tokenize Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading cached processed dataset at /Users/briandunn/.cache/huggingface/datasets/Babelscape___parquet/Babelscape--wikineural-579d1dc98d2a6b93/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec/cache-f7ca1fa2d8390229.arrow\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6be7eb946b4c478e9604724499d2a3ab",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/11597 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading cached processed dataset at /Users/briandunn/.cache/huggingface/datasets/Babelscape___parquet/Babelscape--wikineural-579d1dc98d2a6b93/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec/cache-1c8343d593cffa64.arrow\n"
     ]
    }
   ],
   "source": [
    "encoded_ds = ds.map(tokenize_and_align_labels, \n",
    "                    batched=True, \n",
    "                    remove_columns=\n",
    "                        [\n",
    "                            'ner_tags', \n",
    "                            'tokens'\n",
    "                        ]\n",
    "                    )"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define Data Collator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_collator = DataCollatorForTokenClassification(tokenizer=tokenizer)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load Custom Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of MegaForTokenClassification were not initialized from the model checkpoint at mnaylor/mega-base-wikitext and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "model = (AutoModelForTokenClassification.from_pretrained(\n",
    "    MODEL_CKPT,\n",
    "    num_labels=NUM_OF_LABELS,\n",
    "    id2label=id2label,\n",
    "    label2id=label2id\n",
    "    ).to(DEVICE))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create Compute Metrics Function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "label_list = pos_tag_values\n",
    "\n",
    "seqeval = evaluate.load(\"seqeval\")\n",
    "\n",
    "labels = [label_list[i] for i in example[f'ner_tags']]\n",
    "\n",
    "def compute_metrics(eval_preds):\n",
    "    predictions, labels = eval_preds\n",
    "    predictions = np.argmax(predictions, \n",
    "                            axis=2)\n",
    "    \n",
    "    true_predictions = [\n",
    "        [label_list[p] for (p, l) in zip(prediction, label) if l != -100]\n",
    "        for prediction, label in zip(predictions, labels)\n",
    "    ]\n",
    "    \n",
    "    true_labels = [\n",
    "        [label_list[l] for (p, l) in zip(prediction, label) if l != -100]\n",
    "        for prediction, label in zip(predictions, labels)\n",
    "    ]\n",
    "    \n",
    "    results = seqeval.compute(predictions=true_predictions, \n",
    "                              references=true_labels)\n",
    "    \n",
    "    return results"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define TrainingArguments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "args = TrainingArguments(\n",
    "    MODEL_NAME,\n",
    "    log_level=\"error\",\n",
    "    logging_first_step=True,\n",
    "    learning_rate=LR,\n",
    "    num_train_epochs=NUM_OF_EPOCHS,\n",
    "    per_device_train_batch_size=BATCH_SIZE,\n",
    "    per_device_eval_batch_size=BATCH_SIZE,\n",
    "    evaluation_strategy=STRATEGY,\n",
    "    report_to=REPORTS_TO,\n",
    "    disable_tqdm=False,\n",
    "    logging_steps=STEPS,\n",
    "    weight_decay=WEIGHT_DECAY,\n",
    "    save_strategy=STRATEGY,\n",
    "    hub_private_repo=True,\n",
    "    push_to_hub=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Subclass Trainer to Handle Class Imbalance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CustomTrainer(Trainer):\n",
    "    def compute_loss(self, \n",
    "                     model, \n",
    "                     inputs, \n",
    "                     return_outputs=False):\n",
    "        \n",
    "        labels = inputs.get(\"labels\")\n",
    "        # forward pass\n",
    "        outputs = model(**inputs)\n",
    "        logits = outputs.get(\"logits\")\n",
    "        # compute custom loss \n",
    "        loss_fct = nn.CrossEntropyLoss(weight=torch.tensor(\n",
    "            [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0], \n",
    "            device=model.device)\n",
    "        )\n",
    "        loss = loss_fct(logits.view(-1, \n",
    "                                    self.model.config.num_labels), \n",
    "                        labels.view(-1)\n",
    "                        )\n",
    "        return (loss, outputs) if return_outputs else loss"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define Trainer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Cloning https://huggingface.co/DunnBC22/mega-base-wikitext-finetuned-WikiNeural into local empty directory.\n"
     ]
    }
   ],
   "source": [
    "trainer = CustomTrainer(model, \n",
    "                  args=args,\n",
    "                  data_collator=data_collator,\n",
    "                  compute_metrics=compute_metrics,\n",
    "                  tokenizer=tokenizer,\n",
    "                  train_dataset=encoded_ds[\"train\"],\n",
    "                  eval_dataset=encoded_ds[\"test\"]\n",
    "                  )"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Train Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/briandunn/Documents/deep_learning/dl/lib/python3.9/site-packages/transformers/optimization.py:391: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "37432c07768e4484b314e5da3f6a7724",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/11590 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 2.0333, 'learning_rate': 1.9998274374460744e-05, 'epoch': 0.0}\n",
      "{'loss': 0.7329, 'learning_rate': 1.7842968075927526e-05, 'epoch': 0.22}\n",
      "{'loss': 0.4255, 'learning_rate': 1.5685936151855047e-05, 'epoch': 0.43}\n",
      "{'loss': 0.3791, 'learning_rate': 1.3528904227782573e-05, 'epoch': 0.65}\n",
      "{'loss': 0.3565, 'learning_rate': 1.1371872303710096e-05, 'epoch': 0.86}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9af51f17fdde4a428ea8e4fbac07dbcc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/725 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.30699771642684937, 'eval_LOC': {'precision': 0.6732659365346813, 'recall': 0.8052057094878253, 'f1': 0.7333486273610156, 'number': 5955}, 'eval_MISC': {'precision': 0.4832294842900703, 'recall': 0.7202133965619443, 'f1': 0.5783878133925738, 'number': 5061}, 'eval_ORG': {'precision': 0.5199884958297383, 'recall': 0.5242099159176573, 'f1': 0.5220906728270287, 'number': 3449}, 'eval_PER': {'precision': 0.7719087635054022, 'recall': 0.7404990403071017, 'f1': 0.7558777429467084, 'number': 5210}, 'eval_overall_precision': 0.6095937770095073, 'eval_overall_recall': 0.7169504447268107, 'eval_overall_f1': 0.6589279458133832, 'eval_overall_accuracy': 0.9595440848945369, 'eval_runtime': 158.6264, 'eval_samples_per_second': 73.109, 'eval_steps_per_second': 4.57, 'epoch': 1.0}\n",
      "{'loss': 0.3452, 'learning_rate': 9.214840379637619e-06, 'epoch': 1.08}\n",
      "{'loss': 0.3322, 'learning_rate': 7.057808455565142e-06, 'epoch': 1.29}\n",
      "{'loss': 0.3259, 'learning_rate': 4.900776531492667e-06, 'epoch': 1.51}\n",
      "{'loss': 0.3224, 'learning_rate': 2.74374460742019e-06, 'epoch': 1.73}\n",
      "{'loss': 0.3176, 'learning_rate': 5.867126833477136e-07, 'epoch': 1.94}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b89671ed155241c7aa2d813d12083e54",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/725 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.28621095418930054, 'eval_LOC': {'precision': 0.705460401112575, 'recall': 0.8092359361880772, 'f1': 0.7537932113248866, 'number': 5955}, 'eval_MISC': {'precision': 0.4989403973509934, 'recall': 0.7443193044852796, 'f1': 0.5974149551978432, 'number': 5061}, 'eval_ORG': {'precision': 0.5464716006884681, 'recall': 0.5523340098579298, 'f1': 0.5493871665465032, 'number': 3449}, 'eval_PER': {'precision': 0.7899617012699053, 'recall': 0.7522072936660269, 'f1': 0.7706223576836104, 'number': 5210}, 'eval_overall_precision': 0.6312423339758192, 'eval_overall_recall': 0.7324015247776366, 'eval_overall_f1': 0.6780697833094135, 'eval_overall_accuracy': 0.9618723213115982, 'eval_runtime': 56.506, 'eval_samples_per_second': 205.235, 'eval_steps_per_second': 12.83, 'epoch': 2.0}\n",
      "{'train_runtime': 3446.4695, 'train_samples_per_second': 53.806, 'train_steps_per_second': 3.363, 'train_loss': 0.3912277421218964, 'epoch': 2.0}\n"
     ]
    }
   ],
   "source": [
    "train_results = trainer.train()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Push Model To Hub"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Several commits (2) will be pushed upstream.\n",
      "The progress bars may be unreliable.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8108874fb43640139bc29354e499587c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Upload file pytorch_model.bin:   0%|          | 1.00/28.0M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e4fbcab613e4486ea07516e971f13c07",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Upload file runs/Jul09_14-06-18_Brians-Mac-mini/events.out.tfevents.1688929583.Brians-Mac-mini.2647.0:   0%|  …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "To https://huggingface.co/DunnBC22/mega-base-wikitext-finetuned-WikiNeural\n",
      "   b5729c5..7cd5e9c  main -> main\n",
      "\n",
      "To https://huggingface.co/DunnBC22/mega-base-wikitext-finetuned-WikiNeural\n",
      "   7cd5e9c..2771f58  main -> main\n",
      "\n"
     ]
    }
   ],
   "source": [
    "kwargs = {\n",
    "    \"tasks\": \"token-classification\",\n",
    "    \"tags\": ['token-classification'],\n",
    "}\n",
    "\n",
    "if args.push_to_hub:\n",
    "    trainer.push_to_hub('All DUNN!!!', **kwargs)\n",
    "else:\n",
    "    trainer.create_model_card(**kwargs)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Save & Log Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "To https://huggingface.co/DunnBC22/mega-base-wikitext-finetuned-WikiNeural\n",
      "   2771f58..2320bd5  main -> main\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "***** train metrics *****\n",
      "  epoch                    =        2.0\n",
      "  train_loss               =     0.3912\n",
      "  train_runtime            = 0:57:26.46\n",
      "  train_samples_per_second =     53.806\n",
      "  train_steps_per_second   =      3.363\n"
     ]
    }
   ],
   "source": [
    "trainer.save_model()\n",
    "trainer.log_metrics(\"train\", train_results.metrics)\n",
    "trainer.save_metrics(\"train\", train_results.metrics)\n",
    "trainer.save_state()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Error Analysis"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define Method to Apply to Validation Dataset (& Then Apply it)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def forward_pass_with_label(batch):\n",
    "    # Convert dict of lists to list of dicts suitable for data collator\n",
    "    features = [dict(zip(batch, t)) for t in zip(*batch.values())]\n",
    "    # Pad inputs and labels and put all tensors on device\n",
    "    batch = data_collator(features)\n",
    "    input_ids = batch[\"input_ids\"].to(DEVICE)\n",
    "    attention_mask = batch[\"attention_mask\"].to(DEVICE)\n",
    "    labels = batch[\"labels\"].to(DEVICE)\n",
    "    with torch.no_grad():\n",
    "        # Pass data through model  \n",
    "        output = trainer.model(input_ids, \n",
    "                               attention_mask\n",
    "                               )\n",
    "        # Logit.size: [batch_size, sequence_length, classes]\n",
    "        predicted_label = torch.argmax(output.logits, \n",
    "                                       axis=-1\n",
    "                                       ).cpu().numpy()\n",
    "        \n",
    "    # Calculate loss per token after flattening batch dimension with view\n",
    "    loss = cross_entropy(output.logits.view(-1, 9), \n",
    "                         labels.view(-1), \n",
    "                         reduction=\"none\")\n",
    "    # Unflatten batch dimension and convert to numpy array\n",
    "    loss = loss.view(len(input_ids), -1).cpu().numpy()\n",
    "\n",
    "    return {\"loss\":loss, \"predicted_label\": predicted_label}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Apply Above Function to Entire Evaluation Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Parameter 'function'=<function forward_pass_with_label at 0x7f9a18537f70> of the transform datasets.arrow_dataset.Dataset._map_single couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.\n",
      "Loading cached processed dataset at /Users/briandunn/.cache/huggingface/datasets/Babelscape___parquet/Babelscape--wikineural-579d1dc98d2a6b93/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec/cache-1c80317fa3b1799d.arrow\n"
     ]
    }
   ],
   "source": [
    "eval_set = encoded_ds['eval']\n",
    "\n",
    "eval_set = eval_set.map(forward_pass_with_label,\n",
    "                        batched=True,\n",
    "                        batch_size=32)\n",
    "\n",
    "eval_df = eval_set.to_pandas()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Clean Up Padding Tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>input_ids</th>\n",
       "      <th>token_type_ids</th>\n",
       "      <th>attention_mask</th>\n",
       "      <th>labels</th>\n",
       "      <th>loss</th>\n",
       "      <th>predicted_label</th>\n",
       "      <th>input_tokens</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[101, 2014, 3942, 2000, 10722, 10175, 2226, 20...</td>\n",
       "      <td>[2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...</td>\n",
       "      <td>[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...</td>\n",
       "      <td>[IGN, O, O, O, B-LOC, IGN, IGN, O, O, O, O, O,...</td>\n",
       "      <td>[0.0, 5.5073175e-05, 5.4953973e-05, 0.00020072...</td>\n",
       "      <td>[O, O, O, O, B-LOC, I-LOC, I-LOC, O, O, O, O, ...</td>\n",
       "      <td>[Ġlike, Ġstreet, Ġinvestor, ĠNigeria, Ġclouds,...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           input_ids   \n",
       "0  [101, 2014, 3942, 2000, 10722, 10175, 2226, 20...  \\\n",
       "\n",
       "                                      token_type_ids   \n",
       "0  [2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...  \\\n",
       "\n",
       "                                      attention_mask   \n",
       "0  [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...  \\\n",
       "\n",
       "                                              labels   \n",
       "0  [IGN, O, O, O, B-LOC, IGN, IGN, O, O, O, O, O,...  \\\n",
       "\n",
       "                                                loss   \n",
       "0  [0.0, 5.5073175e-05, 5.4953973e-05, 0.00020072...  \\\n",
       "\n",
       "                                     predicted_label   \n",
       "0  [O, O, O, O, B-LOC, I-LOC, I-LOC, O, O, O, O, ...  \\\n",
       "\n",
       "                                        input_tokens  \n",
       "0  [Ġlike, Ġstreet, Ġinvestor, ĠNigeria, Ġclouds,...  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "id2label[-100] = \"IGN\"\n",
    "eval_df[\"input_tokens\"] = eval_df[\"input_ids\"].apply(\n",
    "    lambda x: tokenizer.convert_ids_to_tokens(x))\n",
    "eval_df[\"predicted_label\"] = eval_df[\"predicted_label\"].apply(\n",
    "    lambda x: [id2label[i] for i in x])\n",
    "eval_df[\"labels\"] = eval_df[\"labels\"].apply(\n",
    "    lambda x: [id2label[i] for i in x])\n",
    "eval_df['loss'] = eval_df.apply(\n",
    "    lambda x: x['loss'][:len(x['input_ids'])], axis=1)\n",
    "eval_df['predicted_label'] = eval_df.apply(\n",
    "    lambda x: x['predicted_label'][:len(x['input_ids'])], axis=1)\n",
    "eval_df.head(1)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Unwrap Each Token Within Sample Separately"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>input_ids</th>\n",
       "      <th>token_type_ids</th>\n",
       "      <th>attention_mask</th>\n",
       "      <th>labels</th>\n",
       "      <th>loss</th>\n",
       "      <th>predicted_label</th>\n",
       "      <th>input_tokens</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2014</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>O</td>\n",
       "      <td>0.0</td>\n",
       "      <td>O</td>\n",
       "      <td>Ġstreet</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3942</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>O</td>\n",
       "      <td>0.0</td>\n",
       "      <td>O</td>\n",
       "      <td>Ġinvestor</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2000</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>O</td>\n",
       "      <td>0.0</td>\n",
       "      <td>O</td>\n",
       "      <td>ĠNigeria</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>10722</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>0.0</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>Ġclouds</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2001</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>O</td>\n",
       "      <td>0.0</td>\n",
       "      <td>O</td>\n",
       "      <td>af</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3013</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>O</td>\n",
       "      <td>0.0</td>\n",
       "      <td>O</td>\n",
       "      <td>Ġeasier</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2460</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>O</td>\n",
       "      <td>0.0</td>\n",
       "      <td>O</td>\n",
       "      <td>Ġprepared</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  input_ids token_type_ids attention_mask labels  loss predicted_label   \n",
       "0      2014              0              1      O   0.0               O  \\\n",
       "0      3942              0              1      O   0.0               O   \n",
       "0      2000              0              1      O   0.0               O   \n",
       "0     10722              0              1  B-LOC   0.0           B-LOC   \n",
       "0      2001              0              1      O   0.0               O   \n",
       "0      3013              0              1      O   0.0               O   \n",
       "0      2460              0              1      O   0.0               O   \n",
       "\n",
       "  input_tokens  \n",
       "0      Ġstreet  \n",
       "0    Ġinvestor  \n",
       "0     ĠNigeria  \n",
       "0      Ġclouds  \n",
       "0           af  \n",
       "0      Ġeasier  \n",
       "0    Ġprepared  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "eval_df_tokens = eval_df.apply(pd.Series.explode)\n",
    "eval_df_tokens = eval_df_tokens.query(\"labels != 'IGN'\")\n",
    "eval_df_tokens[\"loss\"] = eval_df_tokens[\"loss\"].astype(float).round(2)\n",
    "eval_df_tokens.head(7)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### See Which Tokens Have Accumulated Most Loss in Evaluation Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>input_tokens</th>\n",
       "      <td>Ġasking</td>\n",
       "      <td>Ġstatements</td>\n",
       "      <td>Ġvegetables</td>\n",
       "      <td>Ġprofits</td>\n",
       "      <td>Ġgets</td>\n",
       "      <td>over</td>\n",
       "      <td>ĠEurope</td>\n",
       "      <td>ob</td>\n",
       "      <td>Ġbalance</td>\n",
       "      <td>Ġcounter</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>17506</td>\n",
       "      <td>7366</td>\n",
       "      <td>104</td>\n",
       "      <td>48</td>\n",
       "      <td>486</td>\n",
       "      <td>114</td>\n",
       "      <td>1730</td>\n",
       "      <td>62</td>\n",
       "      <td>57</td>\n",
       "      <td>36</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.018</td>\n",
       "      <td>0.038</td>\n",
       "      <td>2.275</td>\n",
       "      <td>4.14</td>\n",
       "      <td>0.238</td>\n",
       "      <td>0.695</td>\n",
       "      <td>0.045</td>\n",
       "      <td>0.85</td>\n",
       "      <td>0.916</td>\n",
       "      <td>1.432</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sum</th>\n",
       "      <td>323.04</td>\n",
       "      <td>280.12</td>\n",
       "      <td>236.62</td>\n",
       "      <td>198.7</td>\n",
       "      <td>115.74</td>\n",
       "      <td>79.19</td>\n",
       "      <td>78.37</td>\n",
       "      <td>52.68</td>\n",
       "      <td>52.2</td>\n",
       "      <td>51.57</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    0            1            2         3       4      5   \n",
       "input_tokens  Ġasking  Ġstatements  Ġvegetables  Ġprofits   Ġgets   over  \\\n",
       "count           17506         7366          104        48     486    114   \n",
       "mean            0.018        0.038        2.275      4.14   0.238  0.695   \n",
       "sum            323.04       280.12       236.62     198.7  115.74  79.19   \n",
       "\n",
       "                    6      7         8         9  \n",
       "input_tokens  ĠEurope     ob  Ġbalance  Ġcounter  \n",
       "count            1730     62        57        36  \n",
       "mean            0.045   0.85     0.916     1.432  \n",
       "sum             78.37  52.68      52.2     51.57  "
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(\n",
    "    eval_df_tokens.groupby(\"input_tokens\")[[\"loss\"]]\n",
    "    .agg([\"count\", \"mean\", \"sum\"])\n",
    "    .droplevel(level=0, axis=1)  # Get rid of multi-level columns\n",
    "    .sort_values(by=\"sum\", ascending=False)\n",
    "    .reset_index()\n",
    "    .round(3)\n",
    "    .head(10)\n",
    "    .T\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### See Which Label IDs Have Most Loss in Evaluation Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>labels</th>\n",
       "      <td>B-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>B-ORG</td>\n",
       "      <td>I-LOC</td>\n",
       "      <td>B-PER</td>\n",
       "      <td>I-ORG</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>O</td>\n",
       "      <td>I-PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>4173</td>\n",
       "      <td>4555</td>\n",
       "      <td>3512</td>\n",
       "      <td>2370</td>\n",
       "      <td>5593</td>\n",
       "      <td>2963</td>\n",
       "      <td>5904</td>\n",
       "      <td>234662</td>\n",
       "      <td>4107</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.393</td>\n",
       "      <td>0.337</td>\n",
       "      <td>0.31</td>\n",
       "      <td>0.233</td>\n",
       "      <td>0.223</td>\n",
       "      <td>0.211</td>\n",
       "      <td>0.193</td>\n",
       "      <td>0.026</td>\n",
       "      <td>0.026</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sum</th>\n",
       "      <td>1639.1</td>\n",
       "      <td>1536.22</td>\n",
       "      <td>1089.68</td>\n",
       "      <td>552.99</td>\n",
       "      <td>1248.04</td>\n",
       "      <td>625.31</td>\n",
       "      <td>1138.46</td>\n",
       "      <td>6186.3</td>\n",
       "      <td>107.08</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             0        1        2       3        4       5        6       7   \n",
       "labels  B-MISC   I-MISC    B-ORG   I-LOC    B-PER   I-ORG    B-LOC       O  \\\n",
       "count     4173     4555     3512    2370     5593    2963     5904  234662   \n",
       "mean     0.393    0.337     0.31   0.233    0.223   0.211    0.193   0.026   \n",
       "sum     1639.1  1536.22  1089.68  552.99  1248.04  625.31  1138.46  6186.3   \n",
       "\n",
       "             8  \n",
       "labels   I-PER  \n",
       "count     4107  \n",
       "mean     0.026  \n",
       "sum     107.08  "
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(\n",
    "    eval_df_tokens.groupby(\"labels\")[[\"loss\"]] \n",
    "    .agg([\"count\", \"mean\", \"sum\"])\n",
    "    .droplevel(level=0, axis=1)\n",
    "    .sort_values(by=\"mean\", ascending=False)\n",
    "    .reset_index()\n",
    "    .round(3)\n",
    "    .fillna(0)\n",
    "    .T\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create Function to Display Confusion Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay\n",
    "\n",
    "def plot_confusion_matrix(y_preds, y_true, labels):\n",
    "    cm = confusion_matrix(y_true, y_preds, normalize=\"true\")\n",
    "    fig, ax = plt.subplots(figsize=(10, 10))\n",
    "    disp = ConfusionMatrixDisplay(confusion_matrix=cm, display_labels=labels)\n",
    "    disp.plot(cmap=\"Blues\", values_format=\".2f\", ax=ax, colorbar=False)\n",
    "    plt.title(\"Normalized Confusion Matrix\")\n",
    "    plt.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Display Confusion Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA2QAAANXCAYAAABaBpzLAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAADODElEQVR4nOzdd3xN9x/H8feNSIwMEokQIWKFWL/abUkUtUdL7SpKrVKtXdpSVKt72FqbWkWNatXusoPaaisSiSRCxMj9/REuV26U9t6cSF7Px+M+2nvu95x8vx/f3Jz3PeOazGazWQAAAACANOdkdAcAAAAAILMikAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAcKiwsTGFhYZbnJ06ckMlk0vTp09O0Hx07dlRgYGCa/sx/a9asWQoODlbWrFmVK1cuu29/+PDhMplMdt/u48qoOQkAEoEMAAw3ffp0mUwmZcuWTWfPnk3xelhYmEqXLm1AzzK3JUuWqH79+sqTJ49cXFyUP39+tWzZUuvWrXPozz148KA6duyoIkWKaMqUKZo8ebJDf15aM5lMMplM6tKli83Xhw4damlz8eLFR97+qlWrNHz48P/YSwBIOwQyAEgnEhMT9f777xvdDYcrVKiQEhIS9OKLLxrdFZvMZrM6deqk559/XhcuXNAbb7yhiRMnqlevXjp27Jhq1aql3377zWE/f8OGDUpKStLnn3+ujh07qmXLlnb/GcOGDVNCQoLdt/uwsmXLpsWLF+v69espXps3b56yZcv2r7e9atUqjRgx4pHWSe9zEkDGRiADgHSifPnymjJliv7++2+H/Qyz2Wzojrgky9HALFmyGNqP1Hz88ceaPn26+vbtqx07dujNN99U586dNXToUG3fvl0zZ86Us7Ozw35+RESEJDnkVMU7nJ2d/1Po+a/q1aunuLg4/fDDD1bLf/vtNx0/flwNGzZMk37cvHlT169fT/dzEkDGRiADgHTizTff1K1btx7qKNnNmzc1cuRIFSlSRK6urgoMDNSbb76pxMREq3aBgYFq1KiRfvzxR1WsWFHZs2fXpEmTtGHDBplMJi1YsEAjRoyQv7+/3N3d1aJFC8XGxioxMVF9+/aVr6+v3Nzc1KlTpxTbnjZtmp555hn5+vrK1dVVpUqV0oQJE/6x7/dfr3OnL7Ye91/z9cMPP6h69erKmTOn3N3d1bBhQ+3bty/Fz1i6dKlKly6tbNmyqXTp0lqyZMk/9kuSEhISNGbMGAUHB+ujjz6yeZ3Viy++qMqVK1ueHzt2TC+88IK8vLyUI0cOVa1aVStXrrRa5956jx49WgUKFFC2bNlUq1YtHT161NIuMDBQ77zzjiTJx8dHJpPJcvrdvf9/r8DAQHXs2NHy/MaNGxoxYoSKFSumbNmyydvbW08//bTWrFljaWPrGrJHnVO//PKLKleurGzZsikoKEgzZ858cHHv4e/vrxo1amju3LlWy+fMmaMyZcrYPEV38+bNeuGFF1SwYEG5uroqICBAr7/+utUHDB07dtS4ceMs9brzkO7Ou48++kifffaZZZz79+9PMScjIiLk4+OjsLAwmc1my/aPHj2qnDlzqlWrVg89VgD4J477iA8A8EgKFy6sDh06aMqUKRo8eLDy58+fatsuXbpoxowZatGihfr166ctW7ZozJgxOnDgQIrwcejQIbVp00bdunVT165dVaJECctrY8aMUfbs2TV48GAdPXpUX375pbJmzSonJyddunRJw4cP1x9//KHp06ercOHCevvtty3rTpgwQSEhIWrSpImcnZ21fPly9ezZU0lJSerVq9dDj7tkyZKaNWuW1bKYmBi98cYb8vX1tSybNWuWXnrpJdWtW1cffPCBrl69qgkTJujpp5/Wrl27LOHtp59+UvPmzVWqVCmNGTNGUVFR6tSpkwoUKPCPffnll18UHR2tvn37PtTRkgsXLujJJ5/U1atX1adPH3l7e2vGjBlq0qSJFi1apOeee86q/fvvvy8nJyf1799fsbGxGjt2rNq1a6ctW7ZIkj777DPNnDlTS5Ys0YQJE+Tm5qayZcv+Yz/uNXz4cI0ZM0ZdunRR5cqVFRcXp+3bt2vnzp2qU6dOqus9ypw6evSoWrRooZdfflkvvfSSvvnmG3Xs2FEVKlRQSEjIQ/Wzbdu2eu211xQfHy83NzfdvHlTCxcu1BtvvKFr166laL9w4UJdvXpVPXr0kLe3t7Zu3aovv/xSZ86c0cKFCyVJ3bp1099//601a9akmFN3TJs2TdeuXdMrr7wiV1dXeXl5KSkpyaqNr6+vJkyYoBdeeEFffvml+vTpo6SkJHXs2FHu7u4aP378Q40RAB6KGQBgqGnTppklmbdt22b+66+/zM7OzuY+ffpYXg8NDTWHhIRYnoeHh5slmbt06WK1nf79+5slmdetW2dZVqhQIbMk8+rVq63arl+/3izJXLp0afP169cty9u0aWM2mUzm+vXrW7WvVq2auVChQlbLrl69mmIsdevWNQcFBVktCw0NNYeGhlqeHz9+3CzJPG3aNJv1SEpKMjdq1Mjs5uZm3rdvn9lsNpsvX75szpUrl7lr165Wbc+fP2/29PS0Wl6+fHlzvnz5zDExMZZlP/30k1lSijHc7/PPPzdLMi9ZsuSB7e7o27evWZJ58+bNlmWXL182Fy5c2BwYGGi+deuW2Wy+W++SJUuaExMTU/y8vXv3Wpa98847ZknmyMhIq58lyfzOO++k6EOhQoXML730kuV5uXLlzA0bNnxgv+/8jDv+zZzatGmTZVlERITZ1dXV3K9fvwf+3Dvj6NWrlzk6Otrs4uJinjVrltlsNptXrlxpNplM5hMnTtisga35NmbMGLPJZDKfPHnSsqxXr15mW7s3d+adh4eHOSIiwuZr98/JNm3amHPkyGE+fPiw+cMPPzRLMi9duvQfxwgAj4JTFgEgHQkKCtKLL76oyZMn69y5czbbrFq1SpL0xhtvWC3v16+fJKU4Xa5w4cKqW7euzW116NBBWbNmtTyvUqWKzGazOnfubNWuSpUqOn36tG7evGlZlj17dsv/x8bG6uLFiwoNDdWxY8cUGxv7T0NN1ciRI7VixQpNnz5dpUqVkiStWbNGMTExatOmjS5evGh5ZMmSRVWqVNH69eslSefOnVN4eLheeukleXp6WrZZp04dy7YeJC4uTpLk7u7+UH1dtWqVKleurKefftqyzM3NTa+88opOnDih/fv3W7Xv1KmTXFxcLM+rV68uKfm0R3vJlSuX9u3bpyNHjjz0Oo86p0qVKmXpu5R8emWJEiUeaRy5c+dWvXr1NG/ePEnS3Llz9eSTT6pQoUI22987365cuaKLFy/qySeflNls1q5dux765zZv3lw+Pj4P1farr76Sp6enWrRoobfeeksvvviimjZt+tA/CwAeBoEMANKZYcOG6ebNm6leS3by5Ek5OTmpaNGiVsv9/PyUK1cunTx50mp54cKFU/1ZBQsWtHp+J8QEBASkWJ6UlGQVtH799VfVrl1bOXPmVK5cueTj46M333xTkv51IFu9erVGjBihIUOGqHnz5pbld8LFM888Ix8fH6vHTz/9ZLkRxp2xFytWLMW27z1VMzUeHh6SpMuXLz9Uf0+ePGlzuyVLlrTqzx331zt37tySpEuXLj3Uz3sY7777rmJiYlS8eHGVKVNGAwYM0J49ex64zqPOqfvHISWP5VHH0bZtW61Zs0anTp3S0qVL1bZt21Tbnjp1Sh07dpSXl5fc3Nzk4+Oj0NBQSY823x70+3A/Ly8vffHFF9qzZ488PT31xRdfPPS6APCwuIYMANKZoKAgtW/fXpMnT9bgwYNTbfewX+x775GF+6V2nVRqy823b3Dw119/qVatWgoODtYnn3yigIAAubi4aNWqVfr0009TXJPzMI4fP6527dqpTp06GjVqlNVrd7Y3a9Ys+fn5pVjXXnc9DA4OliTt3btXzZo1s8s27/VPdf03bt26ZfW8Ro0a+uuvv7Rs2TL99NNPmjp1qj799FNNnDgx1e/+uuNh55S9xtGkSRO5urrqpZdeUmJiYqq3+L9165bq1Kmj6OhoDRo0SMHBwcqZM6fOnj2rjh07PtJ8e9Dvgy0//vijpOTQfObMGYfe/RJA5kQgA4B0aNiwYZo9e7Y++OCDFK8VKlRISUlJOnLkiOVIjJR8g4mYmJhUT/myp+XLlysxMVHff/+91dGSO6cOPqqEhAQ9//zzypUrl+bNmycnJ+sTOIoUKSIp+WYLtWvXTnU7d8Zu63S9Q4cO/WM/nn76aeXOnVvz5s3Tm2+++Y839ihUqJDN7R48eNCqP/aQO3duxcTEWC27fv26zVNbvby81KlTJ3Xq1Enx8fGqUaOGhg8fnmogM2pOZc+eXc2aNdPs2bMtX8Jty969e3X48GHNmDFDHTp0sCy/986RdzxsqHwYq1ev1tSpUzVw4EDNmTNHL730krZs2eLQrz0AkPlwyiIApENFihRR+/btNWnSJJ0/f97qtQYNGkhKviPfvT755BNJSpPvcLoTVO49IhIbG6tp06b9q+11795dhw8f1pIlSyyn8d2rbt268vDw0HvvvacbN26keD0yMlKSlC9fPpUvX14zZsywOo1tzZo1Ka7nsiVHjhwaNGiQDhw4oEGDBtk84jN79mxt3bpVUvK/xdatW/X7779bXr9y5YomT56swMDAh7pu7WEVKVJEmzZtslo2efLkFEfIoqKirJ67ubmpaNGiKW5ffy8j51T//v31zjvv6K233kq1ja35Zjab9fnnn6domzNnTklKEV4fVUxMjOVOle+9956mTp2qnTt36r333vtP2wWA+/ERDwCkU0OHDtWsWbN06NAhq1uJlytXTi+99JImT56smJgYhYaGauvWrZoxY4aaNWummjVrOrxvzz77rFxcXNS4cWN169ZN8fHxmjJlinx9fVO9GUlqVq5cqZkzZ6p58+bas2eP1fVObm5uatasmTw8PDRhwgS9+OKLeuKJJ9S6dWv5+Pjo1KlTWrlypZ566il99dVXkpJv5d+wYUM9/fTT6ty5s6Kjo/Xll18qJCRE8fHx/9ifAQMGaN++ffr444+1fv16tWjRQn5+fjp//ryWLl2qrVu36rfffpMkDR48WPPmzVP9+vXVp08feXl5acaMGTp+/LgWL16c4kjff9GlSxd1795dzZs3V506dbR79279+OOPKY4qlSpVSmFhYapQoYK8vLy0fft2LVq0SK+++mqq2zZyTpUrV07lypV7YJvg4GAVKVJE/fv319mzZ+Xh4aHFixfbvGatQoUKkqQ+ffqobt26ypIli1q3bv3I/XrttdcUFRWln3/+WVmyZFG9evXUpUsXjRo1Sk2bNv3HPgPAwyKQAUA6VbRoUbVv314zZsxI8drUqVMVFBSk6dOna8mSJfLz89OQIUMsXyrsaCVKlNCiRYs0bNgw9e/fX35+furRo4d8fHxS3KHxn9w5urV48WItXrzY6rVChQpZruVq27at8ufPr/fff18ffvihEhMT5e/vr+rVq6tTp06WderVq6eFCxdq2LBhGjJkiIoUKaJp06Zp2bJl2rBhwz/2x8nJSTNnzlTTpk01efJkffTRR4qLi5OPj49q1KihsWPHqlq1apKkvHnz6rffftOgQYP05Zdf6tq1aypbtqyWL19u96NKXbt21fHjx/X1119r9erVql69utasWaNatWpZtevTp4++//57/fTTT0pMTFShQoU0atQoDRgw4IHbN3pOPUjWrFm1fPly9enTR2PGjFG2bNn03HPP6dVXX00RjJ5//nn17t1b3377rWbPni2z2fzIgez777/XzJkz9fHHH1uuK5SSjxiuWbNGL730krZt22Z1h1IA+LdM5v9yJTEAAAAA4F/jGjIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADML3kDlQUlKS/v77b7m7u8tkMhndHQAAAABpwGw26/Lly8qfP7+cnB58DIxA5kB///23AgICjO4GAAAAAAOcPn1aBQoUeGAbApkDubu7S5Jcqg+VyTmbwb3JOI4v6G10FzIkjuHiccEJB47BmRxA5nXt+i2ju5DhXL4cp9LFAy154EEIZA5054+byTkbgcyOPDw8jO5ChsSuGB4X5AbHIJABmZcLgcxhHua9lZt6AAAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAJZBtalUXntntZV55b21ZpP2+mJ4n6ptnXO4qQBbapp59dddG5pX23+qoNqVQi0avN6y8pa+1l7nVrUR4fn9tTst5qqqH9uB48iffl64Sb9r9k78q/+up7t/JF27jvxwPbL1u5S1ZYj5V/9dVVv+57W/LrP6vUV68PVovc4FaszSHmq9Nbew2cc2Pv0a+rCTSrf7B3lr/666nT+SDseoq5VWo5U/uqv62kbdV2+PlzNe49T0TqD5J1J60pNHWPqwk0q1/Qd5Xv6ddXu9M91XfrzLlV5YaTyPf26nmpju67P9x6nIrUHyaty5qzrlAUbVbbJ2/J7qq9qd/zwIWq6U5VbjJTfU331ZOvR+um+mprNZr03cYWC672pfE+/rmY9v9RfpyIcOIL0ibo6BnW1v2mLN6tS8xEKrNlPDbp+ol37Tz6w/fJ1u/R0m9EKrNlPNV98X2t/25dq24Fj5yvfU69p8vwNdu61fRHIMqjnapTQqK5h+mDu7wrrPUt/HovQ4pEtlMczh832wzo8rY71y2rQhLWq2n2apq3arVnDmqpMkK+lzZOlAzR1xS49+8YcPT90obJmyaLvRr+gHK5Z02pYhlqyZofe+nyJBrxcX+tmDFRIUX+98Np4RUZfttl+655jeuWt6WrXuJrWzxykBjXKqsPAKTrw19+WNlcTrqtKuSC9/WrTtBpGunN/XUs/RF27vjVd7e+p64s26lq1XJDeyaR1paaO8d2aHRr22RIN7FJf62cOVOli/mrRJ/W6brld13ZNqmnDrEFqEFpW7QdM0X7qavHdT8k1HdSlvjbMGqTSxfzVvPe41Gu6+5i6DJuu9k2raePswWoYWk7t+0/W/qN3a/r5zJ81af5GfTKktdZM668c2V3UvPc4XUu8kVbDMhx1dQzqan/Lft6p4V8uUb/OdfXjNwNUqmh+tXljgi5esl3TbXuPq8fwmWrbqKp+mjZA9aqXUachX+vgsb9TtF21cbd27jspvzyejh7Gf5ZuA1nHjh1lMpksD29vb9WrV0979uxJdZ0TJ07IZDIpPDw81Ta//fabGjRooNy5cytbtmwqU6aMPvnkE926dStF2/Xr16tBgwby9vZWjhw5VKpUKfXr109nz561xxAdqudzFTVz9V7NXfOnDp2O0htfrdHVxBtq/2xpm+1bPlNKny7YojXbj+vk+Vh9s2q31mw/rlefr2hp88LbizXv5306eCpKfx6PVM9PflCAr4fKF8ubVsMy1IR56/Vi02pq27iqSgTl08eDWyl7NhfNXf67zfaT5m/QM1VLqveLtVW8sJ+GdG+ksiUCNHXhJkublg0qa0CX+gqtVCKthpHujL9d13aNqyr4nrrOeUBda92ua4nCfnrTRl1bZfK6UlPHGD93vTo0u1vXTwa3Uo4H1fXb5Lr2uV3Xod0bqWxwgKYusK7rwC71FVY5c9Z1/Nx16tDsSbVrUi25pkNaK0c2F83+/gE1rXZPTXs0UrngAE1ZuFFS8tGGifPWq3/numoQWlali/lrwogOOn8xVis37k7LoRmKujoGdbW/SfM3qF3jJ9W6YVWVKOynsQNaKruri+at+MNm+6kLNqpmlWD1bFdLxQP9NOiVhipTvIC+WbTZqt25yBgN+3Sxxr3zopyds6TFUP6TdBvIJKlevXo6d+6czp07p7Vr18rZ2VmNGjX619tbsmSJQkNDVaBAAa1fv14HDx7Ua6+9plGjRql169Yym82WtpMmTVLt2rXl5+enxYsXa//+/Zo4caJiY2P18ccf22N4DpPV2Unli+bVhvC7h3zNZmlj+ClVCs5vcx3XrFl07fpNq2XXEm+qaoh/qj/HI6erJOnS5Wt26HX6dv3GTe0+eFqh9+w0OTk5KbRSCW3be8LmOtv3nkix81qzarC27z3uyK4+Vv5NXbfZqOszVYO1jbpKoqaOYqlrpUesa2XqmprrN24q/OBpqzDq5OSk0MolUq3R1r3HFVYp2GrZM1VLWv4NTp6N0oWoOIVVvtvG0y27KoQEatueE3YfQ3pEXR2Dutrf9Rs3tefQaVWvVNyyzMnJSdUrFteOP0/YXGf7vuOqXtH6fTWsSrDVqaNJSUnq/e5s9Wj7jEoE5XNE1+3O2egOPIirq6v8/JKve/Lz89PgwYNVvXp1RUZGysfH55G2deXKFXXt2lVNmjTR5MmTLcu7dOmivHnzqkmTJlqwYIFatWqlM2fOqE+fPurTp48+/fRTS9vAwEDVqFFDMTExdhmfo3h7ZJdzFidFXrpitTwy5oqKBXjZXGfdzhPq+VxF/fbnGR0/F6PQ8oXU6MliypLFZLO9ySSN6VZTf+w7owMnL9p9DOlNVMwV3bqVJB8vD6vlPl7uOnLygs11IqLi5OPlbrXM18tdEVG2D8NnRnfq6ntfXX0fsa4+1NWCmjrGg94DDj+grr623gNSOb0ps4mKib9d0/vnnoeOnHjAXPW2NVfjJEkXbv/3/ja+3nfbZHTU1TGoq/1FW95XU9boaCrX0UVGXU7l79Xden01e62yZHFSlxdC7d9pB0nXR8juFR8fr9mzZ6to0aLy9vZ+5PV/+uknRUVFqX///ilea9y4sYoXL6558+ZJkhYuXKjr169r4MCBNreVK1cum8sTExMVFxdn9XhcDJ64Tsf+vqStkzor4vs3NLZHLc39+U8lJZlttv+oZ22VLJRHL7+/Io17CgAAAKS0++BpTV24UZ8PbSeTyfZBhfQoXR8hW7Fihdzc3CQlH+HKly+fVqxYISenR8+Rhw8fliSVLFnS5uvBwcGWNkeOHJGHh4fy5Xu0w5xjxozRiBEjHrlv9hYVl6Cbt5Lkkzun1XKfXDkVEX0l1XXaj1wm16xZ5OWRXeei4jW8Uw2dOB+bou3YHrVUt3KQGgycr7+j4h0yhvTGO1dOZcnipMho65AdGX05xZGIO3y9PVJc6BsRfVm+930SlpndqWvEfXWNeMS6RlJXC2rqGA96D8jrnXpd7z8alvzvQF0lyTuX2+2a3j/34uT7gJpGRtmaq8nt7/xbREZdtrqQPyLqssoUL2DP7qdb1NUxqKv9eVneV23UKJX3SR9v91T+XiXXcsvuv3TxUrwqNh9uef3WrSSN+GqppizYqG2L37HvIOwkXR8hq1mzpsLDwxUeHq6tW7eqbt26ql+/vk6ePKn69evLzc1Nbm5uCgkJeeht3nud2IPa/JtUPWTIEMXGxloep0+ffuRt2MONm0kKP3pBoeUKWpaZTFKN8gW17WDKu9DcK/HGLZ2LipdzFic1fqqYfvjjqNXrY3vUUsNqRdVkyAKdupAyrGVULlmdVS44QJu2HbYsS0pK0qZth1WpTKDNdSqWCdSm7Yetlm3cekgVyxR2ZFcfK/+mrpVs1HXD1kOqRF0lUVNHSa2uG7f/Q1233VfXLdT1DpesziofHKCN2w5Zlt2dq7ZrVLlMYav2krR+y0HLv0Ehf2/l9fawahMXn6Ad+06oUtlAu48hPaKujkFd7c8lq7PKlgjQL9ut31d/2XFYFUoH2lynYkhh/bLD+n1107ZDqhCS3L5FvUpaN3Ogfp4+wPLwy+Opnm2f0bxPujtqKP9Zuj5CljNnThUtWtTyfOrUqfL09NSUKVM0depUJSQkSJKyZv3n264XL558weCBAwf05JNPpnj9wIEDKlWqlKVtbGyszp0790hHyVxdXeXq6vrQ7R1p/JLtGv9Gfe06ckE7D59Tj6YVlNM1q+as+VOSNKFffZ2Lite705PvSlOhhJ/yebtr77EI5fd206B2T8rJZNLni7ZZtvlRz9pqERastu8uVXzCdfnmTr6FftyV6yluCJIR9WhTU6++O1vlSxbUE6UKaeK3G3T1WqLaNKoqSeo5fKby+eTSW72aSJK6tQpTk+6fa9yctXr2qRB9t2anwg+c0idDWlu2eSn2is5cuKTzkcnh9ujta1F8vT1S/dQ9o+nZpqZ63VPXSbfr2vZ2XXvcruvb99S18e261nkqREtu1/VT6mpBTR2jZ9ua6jXidl1Dbr8HJNxT13dmKp/vPXVtHabG3T7XV3feA366Xdc3U6/rnev8fL08lDdPxq9rz7bPqOeIWfpfyYJ6IiRQE+at15WERLVrnFzT7u/MVD4fT8vXAnRrHaZG3T7TV7PX6tmnQ/TdTzsUfuCUPnuzjSTJZDKpe5ua+uib1QoK8FEhf2+9N3Gl/PJ4qmFoOcPGmdaoq2NQV/vr1ipMr42eo3LBBVW+VEFNWbBRV69dV+uGVSRJvUfOll8eTw3t0ViS1KVlqJ7v9YUmzlunWk+GaNnPO7X74Gl9OKiVJMnLM6e8PK3PEHN2ziIfLw8VLZR+7wqergPZ/Uwmk5ycnJSQkCB//9Tv/mfLs88+Ky8vL3388ccpAtn333+vI0eOaOTIkZKkFi1aaPDgwRo7dqzVTT3uiImJSfU6svRiyaZDyuORQ2+++JR8c+fQ3mORavH2IkXGXJUkFfDxsLo+zDWrs4Z2eFqBfp66knBda7YfV/ePVinuSqKlzcuNykuSVo5tbfWzen7yg+b9nPqX8mUUz9WpoKiYeL0/eaUioi6rdHF/Lfisp+Uw+ZkLl+TkdPfIauWyQZo0sqPem7hCoyesUFCAj2aO7aqSRe7e6XL15r3qPXKO5XnXYdMlSQO61Negrg3SZmAGe65OBV18QF3P2qjr5JEdNXriCo26XddZ99X1h/vq2uV2XQdmkrpSU8d4vk4FRV2K15h76rrw89TfA6rcrut7E1do1Pjkus7+sKtK3VfXV9+9p65Dp0tKruvgVzJ+XZ9/Nnmuvjdp5e3TtPy16Ited2t6PlpO95yxUqVckKaM6qjRE1Zo5PjlyTX96BWVKnq3pq91qK2rCYl6/b15io1PUNVyRbToi57Klkm+M1Oiro5CXe2vae0nFBUTr7FTVykyOk4hxQpo7sfdLTdQOnvhklVNK5UprPHDO+iDyas0ZtIKFS7go2ljXlZwkO27iD8uTOaHOYfPAB07dtSFCxc0bdo0SdKlS5f01VdfacKECVq3bp3CwsJSrHPixAkVLlxY3377rUqUsL4lZkhIiJYtW6bWrVurc+fOevXVV+Xh4aG1a9dqwIABqlWrlhYsWGA5VXH8+PF69dVX1alTJ3Xo0EGBgYE6c+aMZs6cKTc3t4e69X1cXJw8PT3lWnOkTM7Z/ntRIEm6uKKf0V3IkB6fS1+R2T1G12k/Vh6nC+AB2Ne16ym/jxf/TVxcnArl81JsbKw8PB58xkO6PkK2evVqyymD7u7uCg4O1sKFC22GsXu1bt06xbLTp0+rRYsWWr9+vUaPHq3q1avr2rVrKlasmIYOHaq+ffta/THq2bOnihcvro8++kjPPfecEhISFBgYqEaNGumNN96w6zgBAAAAZE7p9ghZRsARMsfgCJlj8Nk4HhccyHEMjpABmRdHyOzvUY6Qpeu7LAIAAABARkYgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIM5GdyAzOPrtq/Lw8DC6GxlGuTdXG92FDCl8dF2ju5Ah3bhlNroLGU52lyxGdyFDupJ40+guZDg5XdnNwuMhNuGG0V3IcC5fe/iacoQMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAzibHQH4DjfLNqk8XPWKSI6TqWK+uu9N1roiZBCqbb/fu0ufTB5pU6fj1bhAj56q1cT1X4yxPL6h1NXaemanTobESOXrFlUtkSAhnRvpAohgWkwmvShddWC6hRaWHncXHXo3GW99/1+/Xkm1mbbaa9UVqUg7xTLNx2MUM/pO1Isf7tZiFpWLaj3lx/Q7F9P2Lvr6drXizZp3OzkuRpS1F9j+j14ri5bu0vvT16p0+eiFRSQPFfr3J6rN27e0piJK/Tz7/t18myU3N2yKbRSCb3Vs4n8fDzTakiGm7Z4sybMXafI27//o15vrv+VSr2my9ft0tgpq3Tm9u//0B6NVeue3/97DRo7X7OW/aYRfZ5T11ZhDhpB+jRlwUZ9OXutIqLiVLqYvz4Y8MID3wOX/rxT701cqVPnohQU4KPhvZvp2afu1tVsNmvMpJWaufQ3xcYnqErZIH08uJWKFPRNg9GkD9MXb9bEeesUGX1ZJYvk18h/mKsr1oXrw6nJczWwgI/e7NFYtaqVsrz++ug5WvjDNqt1QisHa84n3R02hvSIueoY1NX+5iz7VV8v2KCL0ZcVXCSfhr36nMoGF7TZ9siJ8/pi+o/ad+SM/r5wSUN6NNFLzWtYtdm25y99vWCD9h05q8ioOH01oqNqP1U6LYbyr6XrI2QdO3aUyWSyPLy9vVWvXj3t2bMn1XVOnDiRYp1nn31Wu3btsrQJCwuzanPn0b373Tfre5d7eHioUqVKWrZsmUPHa09Lf96pd75Yon4v19Oa6QMUUsxfrV8fr8joyzbbb9tzTN3fmaG2javp5xkDVb9GWXUcNFUH/vrb0iYowFfv9XtBG2YP1vcT+yogn5davTZeFy/Z3mZGU6+snwY2KqkJPx/VC1/+pkPn4jTp5Uryyulis/1rs3YpdNRay6PpJ5t181aSftx7PkXbWiF5VbZgLl2IveboYaQ7S9bs1NufL1H/LvW0dkbyXG3ZN/W5unXPMXV7e4baNa6mdbfn6ksD787VhGvXtefQGb3Rqa7Wzhig6e+/rKMnI9R+wOS0HJahlv28UyO+XKI3OtfVj98MUKmi+dX2jQmp/q5u23tcPYfPVJtGVfXTtAGqV72MOg/5WgeP/Z2i7Q8bd2vHvpPyy5N5wu0d3/20Q8M+W6JBXeprw6xBKl3MX817j0t1rm7ZfUxdhk1X+6bVtHH2YDUMLaf2/Sdr/9G7df185s+aNH+jPhnSWmum9VeO7C5q3nucriXeSKthGer7tTv17ldL9Xqnevrh6/4qVdRf7d+YmOpc3b73uHqNmKnWjapq9Tf9Va96GXUZ8rUOHjtn1S6sSrB2LnvX8hg3vENaDCfdYK46BnW1v1Xrw/X+xO/V68U6+m5iX5UIyq8ug6coKpX3gGvXrisgn5f6dWkgHy93m20Srl1XcFB+vd37OUd23a7SdSCTpHr16uncuXM6d+6c1q5dK2dnZzVq1Ogf1/v555917tw5/fjjj4qPj1f9+vUVExNjeb1r166W7d55jB071mob06ZN07lz57R9+3Y99dRTatGihfbu3WvvITrExHnr1b7Jk2rTqKpKFM6nDwe2VHZXF81b8YfN9pMXbFTNKiXVq30tFQ/00+BuDVWmRAF9s2izpU3zuhUVWrmEAv3zKDgon9597TldvnLN6o0lI+vwdGEt2npaS3ec1bGIeL27dJ+uXb+l5yoWsNk+LuGGouKvWx7ViuXRtRtJ+mmPdSDz9XDVkCalNOjb3bqZlJQWQ0lXJs5br/ZNn1Tb23P1o0EtlT2bi+amNlfnb9QzVUvq1fa1VLywn4Z0a6iyJQro69tz1cMtuxZ92UvNaj+hooXyqmLpwnq/fwvtPnhaZ85Hp+XQDDN5/ga1bfykWjesquKF/fTBgAf//k9dsFE1qwSrZ7taKhbop4GvNFSZ4gU07Z7ff0k6FxmjYZ8u1rh3XpSzc5a0GEq6Mn7uOnVo9qTaNamm4KB8+mRIa+XI5qLZ3/9us/2kbzeoVrWS6vNibZUo7KehPRqpXHCApizcKCn5k/GJ89arf+e6ahBaVqWL+WvCiA46fzFWKzfuTsuhGWbytxvUpnE1tWpYRcUL++n9AS8oWzYXfbtii832Xy/cqLAqwerR9hkVC/TTgK4NVLp4AU1fbD1XXV2c5evtYXnk8siRFsNJN5irjkFd7W/64o16oUEVNa9XWUUL+WlE3+bK5ppVi1dvs9m+THBBDezWWA1r/k9Zs9o+0a9G5ZLq27m+6jxdxpFdt6t0H8hcXV3l5+cnPz8/lS9fXoMHD9bp06cVGRn5wPW8vb3l5+enihUr6qOPPtKFCxe0ZcvdN/gcOXJYtnvn4eHhYbWNXLlyyc/PT8WLF9fIkSN18+ZNrV+/3iHjtKfrN25qz6HTql6phGWZk5OTalQqoe1/Hre5zo4/T6hGpeJWy2pWKZlq++s3bmrW0t/k4ZZdIcX87df5dMo5i0ml/D30x9GLlmVms/TH0YsqVyjXQ23j+UoF9MPuv5Vw45ZlmckkjWlVTtM3HdNfEfH27na6d/3GTe0+dFqhtubqXttzb7utuVq1ZKrtJSku/ppMJpM83bPbp+Pp2N3f/7s1cnJyUvWKxbXjzxM219mx77iqVyxhtSy0SrB27LvbPikpSX3ena0ebZ9RiaB8juh6unb9xk2FHzytsMrWczW0cgltS2Xubd17XGGVgq2WPVO1pLbtPSFJOnk2Shei4hRW+W4bT7fsqhASqG17Tth9DOnN9Rs3tffwGVWvmHKu7rxn7t1rx58nrNpLt+fqfXP7911HVa7RMNVoM1pDPlqgS7FX7N39dIu56hjU1f6u37ipfYfP6sknrN8Dqj1RTOH7TxrYs7T3WF1DFh8fr9mzZ6to0aLy9k55bU5qsmdP3gm7fv36v/q5N2/e1Ndffy1JcnGxfXqaJCUmJioxMdHyPC4u7l/9vP8qOuaKbt1KSnEo18fLXUdOXrC5TkRUnHy8PFK0j4iyPmT80y9/qtvb05Vw7Ybyentowec95Z3Lzb4DSIdy53CRcxYnRcVbz6Go+Osq7PPP4y9dwFPF/dz19iLrI6wvhwbp1i2zZv+aud547khtrvrmdtfRE6nPVd/752rulHP1jmuJN/TuuGV6vs4Tcs+Z8QNZajXN4+Wuo6cibK4TGXVZeWy8X0RE3X0PGzd7rbJkcdLLL4Tav9OPgaiY+FTeVz105AFz1cc79bpeuP3f+9v4elvXPqOKjn3AXE3lb1Vk9GXlyX1fTXO7KzL6br3CqpRU/dByCsjnpZNnL+qDySvVvv8kfT+xr7JkSfefQ/9nzFXHoK72dyn2im4lJck7t/V+VJ7c7jp+2vbfq4wq3QeyFStWyM0t+R/qypUrypcvn1asWCEnp4d7U42JidHIkSPl5uamypUrW5aPHz9eU6dOtWo7adIktWvXzvK8TZs2ypIlixISEpSUlKTAwEC1bNky1Z81ZswYjRgx4lGG99h5qkIxrZsxSFGx8Zq97Hd1HTZNP0ztl+p5vEj2fKUCOnwuzuoGIKX8PdT+qUC98MWvBvYsY7tx85a6DJ0ms1n6cFDqv7t4sD0HT2vqwo368ZsBMplMRncHeKCmtZ+w/H/JIvlVskh+PdVqlH7fdVRP33d0DQDSg3T/UVHNmjUVHh6u8PBwbd26VXXr1lX9+vV18uRJ1a9fX25ubnJzc1NIiPXdwJ588km5ubkpd+7c2r17t+bPn6+8efNaXm/Xrp1lu3ceTZo0sdrGp59+qvDwcP3www8qVaqUpk6dKi8vr1T7OmTIEMXGxloep0+ftm8xHpJXrpzKksUpxUWmkdGX5ettOzj5entYfcKYWvuc2V1VOMBHFUsX1mdD28o5SxbNXW773OmM5NLV67p5K0nebtZHSL3dXHQxPjGVtZJlz5pF9cvl03fbz1gtfyLQS145XbRmcJjCR9dV+Oi68s+dQwMaBuvHQZnjKERqczXi0oPnasT9c9VG+zth7Mz5aC36slemODompV7Ti9GXU/3gxMfbXRdtvl8kH4ncsvsvXbwUr0rNhyugxusKqPG6zpyP1oivlqpy84z9IdQd3rncUnlfjbPU6X6+3h6KjEq9rnlv//f+NhFRl1PdZkbi5Zn6XE1t/D5e7ilu+BF56XKKMzzuVcg/j7xy5dSJMw++1CGjYK46BnW1v9yeOZXFyUlRl6wv2bh46bLy5M74479Xug9kOXPmVNGiRVW0aFFVqlRJU6dO1ZUrVzRlyhRNnTrVEqZWrVpltd78+fO1e/duXbp0SX/99ZcaNGhg9bqnp6dlu3ce7u7WOyt+fn4qWrSonn32WU2bNk2tWrVSRETqh1BdXV3l4eFh9TCCS1ZnlS0RoM3bD1uWJSUlafP2Q6pYurDNdSqUDrRqL0kbtx5Mtb1lu+YkXb9x8793Op27ecus/WfjVKXo3VNlTSapStE82n0y5oHrPlvWTy5ZnLR8l/XNT5bvOqvnP/9FLb741fK4EHtN0zYdU7evtztiGOmOS1ZnlSsRoE3b7pur2w6pYhnbc69i6UBt3mZjrt7T/k4YO3Y6Uou+7CUvz5yOGUA6dOf3/5f7fv9/2XFYFUoH2lynQkhhbd5hXdNN2w5ZbuXcvF4lrZ05UGumD7A8/PJ4qkfbZzQ3k9xK3CWrs8oHB2jjtkOWZUlJSdq07bAqpTJXK5cpbNVektZvOahKZQIlSYX8vZXX28OqTVx8gnbsO6FKZQPtPob0xiWrs8oUL6BfdhyxLLszV59I5TbiFUoH6pftR6yWbd52KNW5LUl/R8ToUuxV+WaSO4MyVx2DutqfS1ZnhRT31+87rd8D/th1VOUf8NUXGVG6D2T3M5lMcnJyUkJCgvz9/S1hqlAh63+4gIAAFSlSRLly5bLLz61cubIqVKig0aNH22V7jta9TU3N+f43zV+5RYdPnNfAsQt09dp1tW5URZL06ohZGjX+e0v7V1qGav0fBzRh7jodOXFBH05dpd0HT6tzi+qSpCsJiRo9Ybm2/3lcp89Fa/fBU3pt1Bydj4xV42f+Z8gY09rMX46rRaUANXnCX0E+OfVWsxBld8mipTuSj3y917Ks+tZNeTrM8xULaN3+C4q9an0L29irN3T0QrzV42ZSki5evq4TFzPPBejd29TU7O9/07crt+jw8fMacHuutmmYPFd7jZilkffO1VahWvfHAY2fkzxXx05ZpfADp/Xy7bl64+YtdR7ytcIPnNKEER10K8msC1FxuhAVlyk+PJCkV1qFae7y37Vg1VYdOXFegz9amPz7f7umfUbO1nsTllvad2kZqg1/HNDEeet05OQFffT1D9pz8LQ63a6pl2dOBQflt3o4O2eRr5eHihbKa7MPGVHPts9o5tLfNG/FHzp0/LzeeH++riQkql3jqpKk7u/M1Iiv7n49SrfWYVr7+359NXutDp84r/cnr1T4gVPqevs6PJPJpO5tauqjb1Zr1cY92nf0rHoMnyW/PJ5qGFrOkDGmtVdah2ne8t+18IfkuTrko4VKSLiuVrfn6msjZ2vMxLtz9eUXQrVhywFNmrdeR09e0Me352rH5rf/Vl1N1Mhxy7TjzxM6fS5Kv2w/rJcHT1Wgfx6FVg622YeMiLnqGNTV/jo2D9XCVVu05Kdt+uvkBQ3//DslXLuu5+tVkiQNen+ePp5696DL9Rs3deDoWR04elY3bt7ShYuxOnD0rE6evXvTtSsJiZY2knTmXLQOHD2rvy9cStvBPYJ0fw1ZYmKizp9Pvk34pUuX9NVXXyk+Pl6NGzf+T9u9evWqZbt3uLq6Knfu3Kmu07dvXz333HMaOHCg/P3T950Fm9V+QlGX4jV26ipFRMUppFgBzfu0h+VmCGcvXJKT091rQSqVDdKEES/p/ckr9d7E5Soc4KvpH3RRySL5JUlZnJx09OQFLVi1VdGx8crtmVPlSxbUsgmvKTiT3HFt9Z7zyp3TRa/WKaY87q46+Hecun+zzXKjj3y5sinJbLZaJzBPTlUo7KWuU7ca0eXHwnN1nlBUTLw+mLLq9hdtFtD8T3tYTtc4c/6S1XVLlcsGaeK7L2nMpJUaPXG5ggJ8NWPs3bl6LiJGqzf/KUmq+eIHVj9r6bjeeqpCsTQamXGa1k6u6YdTVykyOvn3f87H3S2ndZ29cElO99S0UpnCGje8gz6YvErvT1qhwgV89M2YlxUclN+oIaRLzz9bQRdj4vXepJWKiLqsMsX9teiLXvfM1WirulYpF6Qpozpq9IQVGjl+uYICfDT7o1dUqujdur7WobauJiTq9ffmKTY+QVXLFdGiL3oqm2vWNB+fEZrUekJRMVf00dQfLF9iPuvjbpbTa+//W1WxTGF99U4HjZ2yUh9MTp6rU8e8bPk75JTFpIN//a1FP2xTXHyC8ubxUI1KwRrQtYFcXdL9Lo/dMFcdg7raX4Oa5RUdG68vp/+oyEvJXw4/ZUwXy817/o64JNM97wERUXF6rvunluffLNyobxZuVKWyQZr1SU9J0p+HTuul/hMtbd6fmPyhbrNnK+r9ga3TYliPzGQ237cHmY507NhRM2bMsDx3d3dXcHCwBg0apObNm9tc58SJEypcuLB27dql8uXL22wTFhamjRs3plhet25drV69WlLypxZLlixRs2bNLK+bzWaVKlVKNWvW1Pjx4/+x/3FxcfL09NTpC5cMO30xI3pi2I9GdyFDCh9d1+guZEg3bqXbt9jHVnaXzPcdaGnhSmLmOIKclnK6Zp4QiMfbhdhrRnchw7l8OU5lCudVbGzsP+aAdP1OMX36dE2fPv2R1gkMDNQ/ZcwNGzb843ZsbcNkMunAgQOP1B8AAAAASM1jdw0ZAAAAAGQUBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgzgb3YHMwMmU/IB9hI+ua3QXMiSf58cZ3YUM6eKSV43uAvBQcrhkMboLGY7ZbDa6CxmSycROlb15Zs9qdBcyHNONh68pR8gAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDOBvdATjO14s2adzsdYqIjlNIUX+N6ddCT4QUSrX9srW79P7klTp9LlpBAT56q1cT1XkyRJJ04+YtjZm4Qj//vl8nz0bJ3S2bQiuV0Fs9m8jPxzOthmQ4auoYXRqUUe9mT8g3dw79eeKiBk3epJ1HLqTavnvjcupcv4wK5HFX9OUELfvtqN6d+bsSb9ySJLllz6o321ZVo6pByuOZQ3uPR2rwlE3adTQirYZkuK8XbtJXc9YqIipOIcX89X6/FnoiJDDV9svW7tKYSSssc/XtXk1V56kQy+sr1odr+ne/avfBU7oUd1XrZw1SmeIF0mAk6cuUBRv15ezkupYu5q8PBrygCg+o69Kfd+q9iSt16lyUggJ8NLx3Mz17T13NZrPGTFqpmUt/U2x8gqqUDdLHg1upSEHfNBhN+jB14SZLTUOK+euD/i3+oabJc/XU7bk6/FXruWo2mzVm8irNstS0sD4alLlqKtm/rsvXh2vad79q94Hk94CNs3kP4D3APqYt3qzxc9cpMjpOpYr6a/TrzfW/UqnvWy1ft0sfTFmlM+ejVbiAj4b1aKxaT4bYbDtw7HzNWvabRvR5Tq+0CnPQCP47jpBlUEvW7NTbny9R/y71tHbGAIUU81fLvuMVGX3ZZvute46p29sz1K5xNa2bMVD1a5TVSwOn6sBff0uSEq5d155DZ/RGp7paO2OApr//so6ejFD7AZPTcliGoqaO8dzTxTSqc3V9MH+rwt74Vn8ev6jFw5soj2d2m+1b1Ciudzo8qbHfblWVV2er95dr9dzTxfTWi9UsbT5/tZbCygeo+6dr9FSfuVq365SWvttM+bxyptWwDLVkzQ699fkSDXi5vtbNGKiQov564bUHz9VX3pqudo2raf3MQWpQo6w6DJximauSdDXhuqqUC9LbrzZNq2GkO9/9tEPDPluiQV3qa8OsQSpdzF/Ne49Lta5bdh9Tl2HT1b5pNW2cPVgNQ8upff/J2n/0bl0/n/mzJs3fqE+GtNaaaf2VI7uLmvcep2uJN9JqWIb6bk1yTQd2qa/1MweqdDF/teiT+lzdsueYur41Xe2aVNOGWYPUILSs2g+Yov33zNUvZv6syfM36uPBrbTmm37Kkd1VLfqMzzQ1lRxT16sJ11W1XJDe4T2A9wA7WvbzTg3/con6da6rH78ZoFJF86vNGxN08ZLtmm7be1w9hs9U20ZV9dO0AapXvYw6DflaB4/9naLtqo27tXPfSfnlSf8fcj92gaxjx45q1qxZqq+HhYXJZDLJZDIpW7ZsKlWqlMaPH295ffr06ZbX731ky5bN6mfcWZ41a1YVLlxYAwcO1LVr1xw5NLuaOG+92jd9Um0bVVWJwvn00aCWyp7NRXNX/GGz/eT5G/VM1ZJ6tX0tFS/spyHdGqpsiQL6etFmSZKHW3Yt+rKXmtV+QkUL5VXF0oX1fv8W2n3wtM6cj07LoRmGmjpGz6blNfOnfZq79oAOnb6kNyas19XEm2pfu5TN9pWD82nLgXNatOmwTkdc1vrw01q86YgqFMsrScrmkkVNqhXR8Om/6bf9f+v4+Vh98O1WHTsXq871y6Tl0AwzYd56vdi0mto2rqoSQfn08eBWyXN1+e8220+av0HPVC2p3i/WTp6r3RupbIkATV24ydKmZYPKGtClvkIrlUirYaQ74+euU4dmT6pdk2oKDsqnT4a0Vo5sLpr9fSp1/XaDalUrqT4v1laJwn4a2qORygUHaMrCjZKSPxmfOG+9+neuqwahZVW6mL8mjOig8xdjtXLj7rQcmmHGz12vDs2qqV3jqsk1HdxKObK5aE5qc/XbDapV9Z6adm+kssEBmrogea6azWZN/HaD+t2uaUgxf00Y/uLtmu5Jy6EZyt51laRWDSprYJf6CqvMewDvAfYzaf4GtWv8pFo3rKoShf00dkBLZXd10bxU9q2mLtiomlWC1bNdLRUP9NOgVxqqTPEC+ub2vtUd5yJjNOzTxRr3zotyds6SFkP5Tx67QPYwunbtqnPnzmn//v1q2bKlevXqpXnz5lle9/Dw0Llz56weJ0+etNpGvXr1dO7cOR07dkyffvqpJk2apHfeeSeth/KvXL9xU7sPnbbacXJyclKNSiW0fe9xm+ts//OEalQqbrWsZtWSqbaXpLj4azKZTPJ0t30kIyOhpo6R1dlJ5Yv4asPu05ZlZrO0cfdpVSrhZ3OdrQfPqXwRXz1xO4AVyuuhOhUKac2O5N9h5yxOcs7ipGs3blqtd+36TVUtmc9BI0k/rt+4qd0HTyu0svVcDa1UQtv2nrC5zva9J1IErZpVgx84VzOb6zduKvzgaaudUScnJ4VWLqFtqdRp697jCqsUbLXsmaolLf8OJ89G6UJUnMIq323j6ZZdFUICtW3PCbuPIb2xzNVKDz9Xt+09YTW3JemZqsGWf4OTf9+p6d02HndqmknmsyPqCt4DHOH6jZvac+i0qt+zr+Tk5KTqFYtrx58nbK6zfd9xVa9oPVfDqgRrx7677ZOSktT73dnq0fYZlQh6PP7uZ8hryHLkyCE/v+SdueHDh2vu3Ln6/vvv1aZNG0mSyWSyvJ4aV1dXS5uAgADVrl1ba9as0QcffODYzttBdMwV3bqVJB8vd6vlvrnddfSE7etyIqLi5OvlYbXMJ7e7IqJsHzK+lnhD745bpufrPCH3nBk/PFBTx/D2yC7nLE6KjLlqtTwy5qqKFchtc51Fmw7LyyObfhjTXCaTlNU5i775Ya8+WbRdkhSfcENbD57TgJaVdPjMJUXEXFWL6sVVqYSfjp2PdfiYjBZlmav3zT0vdx05mfpcTTG3vVKfq5lRVEy8zfcAHy8PHXnAe4CP9/3t3RURFSdJunD7v/e38fW+2yYje9BcPfyAuepra67ePmXMUtMU/06Zo6aSY+oK3gMcIbV9Kx8vdx09Zfua78ioy//4+/3V7LXKksVJXV4ItX+nHSRDHiG7X/bs2XX9+vV/vf6ff/6p3377TS4uLg9sl5iYqLi4OKtHRnTj5i11GTpNZrP04aCWRncnQ6CmD++p0v56o0VF9Z+0QWFvzFf7MSv1bMVA9W9ZydKm26c/yWQy6cC0zrqwqKdeaVROizcfVlKS2cCeAwAAR9p98LSmLtyoz4e2k8lkMro7Dy1DHiG749atW5o3b5727NmjV155xbI8NjZWbm5uVm2rV6+uH374wfJ8xYoVcnNz082bN5WYmCgnJyd99dVXD/x5Y8aM0YgRI+w7iH/BK1dOZcnilOIi04hLl+V736cwd/h6eygi2jpARtpofyc4nDkfre/G9c40R3KoqWNExSXo5q0k+eTKYbXcJ1cORVy6anOdoW2rasGGQ5q1Zr8kaf/JKOV0zapPe9XUxwu3yWyWTpyPU6Oh3ymHq7Pcc7jowqWr+npAPZ28kDE/JLmXt2Wu3jf3oi+nOGJ7h6+3R8q5HZ363M6MvHO52XwPiIyOk6/3A+oadX/7y5b2eW//NzLqstVF5xFRlzPF3eseNFfzPqCm9x+1iYi+bDm6Y6lptHVNI6Mvq3Rxf3t2P91yRF3Be4AjpLZvFfmAuefj7W67/e1abtn9ly5eilfF5sMtr9+6laQRXy3VlAUbtW1x+rz86LE9QjZnzhy5ublZHps3372Yb/z48XJzc1P27NnVtWtXvf766+rRo4fldXd3d4WHh1s9pk6darX9mjVrKjw8XFu2bNFLL72kTp06qXnz5g/s05AhQxQbG2t5nD59+oHtHcUlq7PKlQjQpm2HLcuSkpK0edshVSxT2OY6FUsHavM97SVp49aDVu3vBIdjpyO16Mte8vLMHHesk6ipo9y4maTwvyIUWvbuHx6TSapRNkDbDp23uU52V+cUR7puJSXdXtf607CriTd14dJVeeZ0Va3yBbVqyzE7jyD9ccnqrHLBKefqpm2HValMoM11KpYJ1Kbt98/V1Od2ZuSS1VnlgwO0cdshy7K7dbVdp8plClu1l6T1Ww5a/h0K+Xsrr7eHVZu4+ATt2HdClcoG2n0M6U1qc3Xj9tTnaqUygVbtJWnDlkOWf4NC+R9Q00wynx1RV/Ae4AguWZ1VtkSAftluPVd/2XFYFUoH2lynYkhh/bLDeq5u2nbI8tUDLepV0rqZA/Xz9AGWh18eT/Vs+4zmfdLdUUP5zx7bI2RNmjRRlSpVLM/9/e9+8tWuXTsNHTpU2bNnV758+eTkZJ07nZycVLRo0QduP2fOnJY233zzjcqVK6evv/5aL7/8cqrruLq6ytXV9d8Mx+66t6mp3iNnq3zJAD1RqpAmzd+gq9euq03D5Jr1GjFLfj6eeqtnE0nSK61C1bTHFxo/Z53qPBWiJWt2KPzAaX08uLWk5ODQecjX2nPojOZ83E23ksyWc59ze+SQS9bHdio9NGrqGOOXhWv8a7W162iEdh65oB6NyytnNmfN+Tn5CNiEvnV0Lipe785KvovV6m3H1bPp/7TneKS2H7qgoHyeerNdVa3edsIS1J75X0GZJB05G6OgfJ56t+NTOnz2kuasPWDUMNNUjzY19eq7s1W+ZEE9UaqQJn67QVevJapNo6qSpJ7DZyqfTy691St5rnZrFaYm3T/XuDlr9exTIfpuzU6FHzilT4a0tmzzUuwVnblwSecjk6/DO3r7WhRfb49UP3XPaHq2fUY9R8zS/0oW1BMhgZowb72uJCSqXePkunZ/Z6by+XhabgverXWYGnX7TF/NXqtnnw7Rdz/tUPiBU/rszbvXM3dvU1MffbNaQQE+KuTvrfcmrpRfHk81DC1n2DjTUs+2NdVrxO25GnJ7riYkqu3tudrjnZnK55tLb9+Zq63D1Ljb5/rqzlz9KXmufvpm8lw1mUzq3jpMH3/zo4oE+KpQfm+9N3HF7ZqWNWycac3edZVSvgfcuSbV18tDefPwHiDxHvBvdGsVptdGz1G54IIqX6qgpizYqKvXrqv17X2r3iNnyy+Pp4b2aCxJ6tIyVM/3+kIT561TrSdDtOznndp98LQ+HNRKkuTlmTPFh9vOzlnk4+WhooXypu3gHsFju8fn7u4ud3fbhzM9PT3/MXA9CicnJ7355pt644031LZtW2XPnv5PKXuuzhOKionXB1NW3f7ywgKa/2kPyyHdM+cvWR1NqFw2SBPffUljJq3U6InLFRTgqxlju6hkkfySpHMRMVq9+U9JUs0XrW9ssnRcbz1VoVgajcw41NQxlvxyRHk8suvNtlXkmzun9h6PVIsR3ysyNkGSVCCPm9URsY8WJJ+WOLRdVeXzclNUXIJWbzuukbPv3nbYI4eL3n7xSeXP46ZLl69p+e9/adTs33XzVlKaj88Iz9WpoKiYeL0/eaUiopJP1VrwWc+7c/XCJTk5Wc/VSSM76r2JKzR6wgoFBfho5tiulrkqSas371XvkXMsz7sOmy5JGtClvgZ1bZA2AzPY889W0MWYeL03aeXtU4r8teiLXve8B0TL6Z73gCrlgjRlVEeNnrBCI8cvV1CAj2Z/9IpKFb1b19c61NbVhES9/t48xcYnqGq5Ilr0RU9lc82a5uMzwvN1KijqUrzG3DNXF36e+lytUjZIk2/P1VHjk+fq7A+7qtQ9c7VPh9q6cu36PTUN0sLPM09NJcfU9YfNe/Xqu3ffA7oMnS5JGtilvga/wnuAxHvAv9G0dvK+1dipqxQZHaeQYgU09+PulpvSnL1wyaqmlcoU1vjhHfTB5FUaM2mFChfw0bQxLys4KH9qP+KxYDKbzY/VVe4dO3ZUTEyMli5davP1sLAwlS9fXp999pnN16dPn67XXntNhw4dSvGar6+vnJycbP6MmzdvKjAwUH379lX//v0fqq9xcXHy9PTU2YhL8vDIHJ8e4fHl8/w4o7uQIV1c8qrRXchwsjg9PhdqP04es90BZGKP080aHhfXrt8yugsZTlxcnArl81JsbOw/5oDH9hqy/yIuLk758uVL8YiIsH2LTUlydnbWq6++qrFjx+rKlStp2FsAAAAAGdVjd4TsccIRMjxOOELmGBwhsz+OkDkGuwN4XHCEzP44QmZ/HCEDAAAAgMcAgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIM4G92BzCCLk0lZnExGdyPDMJuN7kHGdGlZb6O7kCHlfnqg0V3IcC79MtboLmRIJhN/p4DMKqszx2js7VFqSvUBAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCDORncAjjN14SZ9OXutIqLiFFLMXx/0b6EKIYGptl/68y6NmbRCp85FKyjAR8Nfbao6T4VYXl++PlzTvvtVuw+c0qW4q9o4e5DKFC+QBiNJP6Yu3KSv5tyt6fv9HlzTZWt36b1JK3T6dk3f6ZWyptO/+1W7DybXdMOszFdTSZqyYKNlrpYu5q8PBrzwD3N1p96buFKnzkUlz9XezfTsPXU1m80aM2mlZi79TbHxCapSNkgfD26lIgV902A06UOX56upd5tQ+Xq568+/zmnQp8u088Bpm22dszjp9RefUZv6FZQvj4eOno7U8AmrtHbLYUub3QsHq2A+rxTrTv3uNw34ZKmjhpHuMFftj5o6BnV1DOpqf1/b2Ld64h/2rcbcs2/19n37Vivu27da/xjsW6WbI2QdO3aUyWSyPLy9vVWvXj3t2bPnH9fdt2+fWrZsKR8fH7m6uqp48eJ6++23dfXqVat2gYGBlu3nyJFDZcqU0dSpU1Nsz2w2a8qUKapWrZo8PDzk5uamkJAQvfbaazp69KjdxuxI363ZoWGfLdHALvW1fuZAlS7mrxZ9xisy+rLN9lv2HFPXt6arXZNq2jBrkBqEllX7AVO0/6+/LW2uJlxX1XJBeufVpmk1jHRlyZodeuvzJRrwcn2tmzFQpYv664XXUq/p1ts1bd+4mtbPHKQGNcrqxYFTdICaWvnup+S5OqhLfW2YNUili/mree9xqc/V3cfUZdh0tW9aTRtnD1bD0HJq33+y9h+9W9fPZ/6sSfM36pMhrbVmWn/lyO6i5r3H6VrijbQalqGee6acRr3aWB9M+1lhL3+uP4+e0+JPXlaeXDltth/2Sl11bFpFgz5dpqovfqxpS//QrPdeUpli+S1tnun6pUo0edfyaNZ3siRp6fp/fo/OKJir9kdNHYO6OgZ1tb/7961CHmLf6pW3pqvdPftWHWzsW1UpF6S3H6N9q3QTyCSpXr16OnfunM6dO6e1a9fK2dlZjRo1euA6f/zxh6pUqaLr169r5cqVOnz4sEaPHq3p06erTp06un79ulX7d999V+fOndOff/6p9u3bq2vXrvrhhx8sr5vNZrVt21Z9+vRRgwYN9NNPP2n//v36+uuvlS1bNo0aNcohY7e38XPXq0OzamrXuKqCg/Lpk8GtlCObi+Ys/91m+0nfblCtqiXV58XaKlHYT0O7N1LZ4ABNXbDJ0qZVg8oa2KW+wiqXSKthpCvj563Xi03v1vTjwa2U/UE1nZ9c0963a/pm90YqWyJAUxda13RAl/oKrZQ5aypJ4+euU4dmT6pdk2rJc3VIa+XI5qLZ3z9grla7Z672aKRywQGasnCjpOTf4Ynz1qt/57pqEFpWpYv5a8KIDjp/MVYrN+5Oy6EZpmfr6pq5fIvmrtquQyci9MaH3+nqtRtq36iSzfYt61bQp7PWac0fB3Xy72h9s/QPrfn9oF5tXcPSJirmiiKi4y2Puk+W1LEzF/XrrmNpNSzDMVftj5o6BnV1DOpqfxNu71u1bVxVJe7Zt5r7gH2rZ27vWxUv7KchNvatWj6G+1bpKpC5urrKz89Pfn5+Kl++vAYPHqzTp08rMjLSZnuz2ayXX35ZJUuW1HfffafKlSurUKFCeuGFF7R8+XL9/vvv+vTTT63WcXd3l5+fn4KCgjRo0CB5eXlpzZo1ltfnz5+vb7/9VvPnz9dbb72lqlWrqmDBgqpatao++OADTZs2zaE1sIfrN25q98HTVhPRyclJoZVKaNveEzbX2bb3hELvC1rPVA3Wtr3HHdnVx4alppUfsaaVqOmDXL9xU+EHT1uFfCcnJ4VWLpFqnbbuPa6wSsFWy56pWtLy73DybJQuRMUprPLdNp5u2VUhJFDb9pyw+xjSm6zOWVS+uL82bL97NN9sNmvj9iOqFFLI5jquWbPoWuJNq2XXEm+oatnAVH9Gy2ef0JyV2+zW7/SOuWp/1NQxqKtjUFf7+zf7Vttt7FvVrBqs7Y/5vlW6CmT3io+P1+zZs1W0aFF5e3vbbBMeHq79+/frjTfekJOT9VDKlSun2rVra968eTbXTUpK0uLFi3Xp0iW5uLhYls+bN08lSpRQkyZNbK5nMplS7XNiYqLi4uKsHkaIirmiW7eS5OPlYbXcx8tdF6Js9ykiKk6+Xu5Wy3y93BWRyiHjzOZOTX3vq2lyjVKvqc99NfXxcldEFDW9Iyom/vZcvb9OHop4wFz18bZV1+T2d+b4/W18vd1T3WZG4u2ZU87OWVKc7hEZHS/f+2pyx7qth9WzdXUFFcgjk8mksIrF1Ci0tPJ6e9hs37BGiDzdsmnuqh127396xVy1P2rqGNTVMair/T1of/VR9q18M8C+VboKZCtWrJCbm5vc3Nzk7u6u77//XvPnz08Rtu44fDj5gvOSJUvafL1kyZKWNncMGjRIbm5ucnV1VYsWLZQ7d2516dLFapslSlgn7759+1r6VaBA6hcFjhkzRp6enpZHQEDAQ40bAIw0+PPvdez0RW2d018R69/T2Deaae6q7Uoym222b9+wkn7eckjnM8EOAwAAjpauAlnNmjUVHh6u8PBwbd26VXXr1lX9+vV18uRJ1a9f3xKKQkJCrNYzp7LTYMuAAQMUHh6udevWqUqVKvr0009VtGjRB64zdOhQhYeH6+2331Z8fHyq7YYMGaLY2FjL4/Rp23c0czTvXDmVJYuTIu/7dCEy+nKqn3j7enukOBoWEX05xVGzzOpOTe//xCa5RqnXNOVRisupHqXIjLxzud2eq/fXKU6+D5irkVG26prc/s4cv79NRNTlVLeZkUTFXtHNm7dsfIrrluoniFExV9T+zZnyrzNMZVuMUeW2H+pKQqJO/B2Vom1A3lwKq1hMM5dvdUj/0yvmqv1RU8egro5BXe3vQfurj7JvFZEB9q3SVSDLmTOnihYtqqJFi6pSpUqaOnWqrly5oilTpmjq1KmWsLZq1SpJUvHixSVJBw4csLm9AwcOWNrckSdPHhUtWlTVq1fXwoUL1adPH+3fv9/yerFixXTo0CGrdXx8fFS0aFH5+j74FqSurq7y8PCwehjBJauzygUHaNO2u0cHk5KStHH7YVUqE2hznUplAq3aS9KGLYdUqUxhR3b1sZFaTTdt+4eabr+vplup6b1csjqrfHCANm67+zt3t66261S5TGGr9pK0fstBy79DIX9v5fX2sGoTF5+gHftOqFIq10RlJDdu3lL44bMKrXD3gyaTyaQaFYpq276TD1w38fpNnbsYJ+csTmocWkY/bN6fok3bhpUUeSleP/1+0O59T8+Yq/ZHTR2DujoGdbW/f7NvVdHGvtXGrYdU8THft0pXgex+JpNJTk5OSkhIkL+/vyWsFSqUfGF6+fLlFRwcrE8//VRJSUlW6+7evVs///yz2rRpk+r2AwIC1KpVKw0ZMsSyrE2bNjp06JCWLVvmmEGlkZ5ta2rmst80b8UWHTp+Xv0+WKCrCYlq26iqJKnHOzP17rjvLe27tQ7T2t/366s5a3X4xHm9P3mVwg+cUpeWd++ydin2ivYePqNDx89Lko6cvKC9h8/owsXMcdpSzzY1NWvZb5q3Mrmm/T9YoKvX7qnp8Ptq2iq5puNu1/SDKbdr+kLqNT16p6aZ6FSwnm2f0cylv2neij906Ph5vfH+fF1JSFS7xsl17f7OTI346u7vo2Wuzr4zV1cq/MApdX0hVFLy+0b3NjX10TertWrjHu07elY9hs+SXx5PNQwtZ8gY09r4bzerQ+PKal2vgooX8tUn/Z9TzuwumrNyuyRpwrBWertbPUv7CqUC1KhGaRXK76VqZQO16OOX5eRk0udzN1ht12QyqV2Divp29Q7dumX9npsZMFftj5o6BnV1DOpqfz1u71t9u3KLDt+zb9Xm9r5Vz+EzNfK+fat1t/etjmSgfat09cXQiYmJOn8+uXiXLl3SV199pfj4eDVu3Nhme5PJpK+//lp16tRR8+bNNWTIEPn5+WnLli3q16+fqlWrpr59+z7wZ7722msqXbq0tm/frooVK6p169b67rvv1Lp1aw0ZMkR169ZV3rx5dfLkSc2fP19ZsmSx97Ad4vk6FRR1KV5jJq9URNRllS7ur4Wf97QcAj9z4ZKcnO7eoKRK2SBNHtlR701coVHjVygowEezP+yqUkXufg/RD5v36tV351iedxk6XZI0sEt9DX6lQdoMzEDP1amgizHxev+emi747G5Nz95X08q3azp64gqNmpBc01lju6rkfTXtPfKemg6bLim5poO6ZvyaStLzzybX9b1JyXUtU9xfi77odXeuno+W0z0306lSLkhTRnXU6AkrNHL88uS5+tErKlX0bl1f61BbVxMS9fp78xQbn6Cq5Ypo0Rc9lc01a5qPzwhL1u1Wnlw59WaXZ+Xr5a69R/9Wi35fK/JS8inXBfLmUlLS3VO9XV2yamjXugrM76UrCde15o+D6j5yvuLir1ltN6xiUQX45dbsTHR3xXsxV+2PmjoGdXUM6mp/z9WpoKgH7Fvdv79auWyQJt3eXx19e99q5n37Vqvv27fqenvfakA63rcymR/lAiwH6tixo2bMmGF57u7uruDgYA0aNEjNmzd/4Lp79+7ViBEjtH79el2+fFkFCxZUmzZtNGTIEOXIkcPSLjAwUH379k0R0urVqycnJyfLqZBJSUmaMmWKpk2bpj///FM3btxQgQIFVKtWLb3++uup3kTkfnFxcfL09NT5izGGnb6YEaWPGZvx3PuGB/vJ/fRAo7uQ4Vz6ZazRXQCADOVWEjtX9hYXF6f8PrkUGxv7jzkg3QSyjIhA5hjMWMcgkDkGgcz+CGQAYF8EMvt7lECWrq8hAwAAAICMjEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBBnozuQGSSZkx+wjyxOJqO7ADy0S7+MNboLGY5X62+M7kKGFP1tZ6O7kOEk8cffIZzYD7C7qPjrRnchw7n8CDXlCBkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAZxfphG33///UNvsEmTJv+6MwAAAACQmTxUIGvWrNlDbcxkMunWrVv/pT8AAAAAkGk8VCBLSkpydD8AAAAAINP5T9eQXbt2zV79AAAAAIBM55ED2a1btzRy5Ej5+/vLzc1Nx44dkyS99dZb+vrrr+3eQQAAAADIqB45kI0ePVrTp0/X2LFj5eLiYlleunRpTZ061a6dAwAAAICM7JED2cyZMzV58mS1a9dOWbJksSwvV66cDh48aNfOAQAAAEBG9siB7OzZsypatGiK5UlJSbpx44ZdOgUAAAAAmcEjB7JSpUpp8+bNKZYvWrRI//vf/+zSKQAAAADIDB7qtvf3evvtt/XSSy/p7NmzSkpK0nfffadDhw5p5syZWrFihSP6CAAAAAAZ0iMfIWvatKmWL1+un3/+WTlz5tTbb7+tAwcOaPny5apTp44j+ggAAAAAGdIjHyGTpOrVq2vNmjX27gsAAAAAZCr/KpBJ0vbt23XgwAFJydeVVahQwW6dAgAAAIDM4JED2ZkzZ9SmTRv9+uuvypUrlyQpJiZGTz75pL799lsVKFDA3n0EAAAAgAzpka8h69Kli27cuKEDBw4oOjpa0dHROnDggJKSktSlSxdH9BEAAAAAMqRHPkK2ceNG/fbbbypRooRlWYkSJfTll1+qevXqdu0cAAAAAGRkj3yELCAgwOYXQN+6dUv58+e3S6cAAAAAIDN45ED24Ycfqnfv3tq+fbtl2fbt2/Xaa6/po48+smvnAAAAACAje6hTFnPnzi2TyWR5fuXKFVWpUkXOzsmr37x5U87OzurcubOaNWvmkI4CAAAAQEbzUIHss88+c3A3AAAAACDzeahA9tJLLzm6HwAAAACQ6fzrL4aWpGvXrun69etWyzw8PP5ThwAAAAAgs3jkm3pcuXJFr776qnx9fZUzZ07lzp3b6gEAAAAAeDiPHMgGDhyodevWacKECXJ1ddXUqVM1YsQI5c+fXzNnznREHwEAAAAgQ3rkUxaXL1+umTNnKiwsTJ06dVL16tVVtGhRFSpUSHPmzFG7du0c0U8AAAAAyHAe+QhZdHS0goKCJCVfLxYdHS1Jevrpp7Vp0yb79g4AAAAAMrBHPkIWFBSk48ePq2DBggoODtaCBQtUuXJlLV++XLly5XJAF/Fvfb1wk76as1YRUXEKKeav9/u10BMhgam2X7Z2l8ZMWqHT56IVFOCjt3s1VZ2nQiyvr1gfrunf/ardB0/pUtxVrZ81SGWKF0iDkaQfUxZs1Jezk2taupi/Phjwgio8oKZLf96p9yau1KlzUQoK8NHw3s307D01NZvNGjNppWYu/U2x8QmqUjZIHw9upSIFfdNgNOkHdbU/auoYLz9bUr0bl5avZ3btO3VJg6b9rp1/XUy1fff6pdSpTkkVyJNT0Zev6fstJ/TuvB1KvHFLkhT+5Qsq6OOeYr2pPx7QwGm/O2wc6Qlz1TGm2tgHeFBdl63dpffu2Qd45759gOX37QNsyIT7ABLz1RHmLP1FXy/YoMjoywoukl9v9X5OZYML2mx75MR5fTF9tfYdPqOzFy5pSM+m6ti8hlWbbXv+0tfzN+jPI2cUGRWncSM6qvbTZdJiKP/aIx8h69Spk3bv3i1JGjx4sMaNG6ds2bLp9ddf14ABA+zeQfw7S9bs0FufL9GAl+tr3YyBCinqrxdeG6/I6Ms222/dc0yvvDVd7RpX0/qZg9SgRll1GDhFB/7629LmasJ1VSkXpLdfbZpWw0hXvvtph4Z9tkSDutTXhlmDVLqYv5r3HpdqTbfsPqYuw6arfdNq2jh7sBqGllP7/pO1/+jdmn4+82dNmr9RnwxprTXT+itHdhc17z1O1xJvpNWwDEdd7Y+aOsZz1Qpr1IuVNXZRuGoO+V5/nozWoiF1lccjm832zZ8K0tttKmrs4l2q2u879Zn0i5pVDdJbrStY2tR6c7mCu82zPJ4btVqStGzL8TQZk9GYq45x/z5A6YfYB+j61nS1v2cf4EUb+wBVywXpnUy6DyAxXx1h1fpdGjPxe/Xq8KyWTHxdwUXy6+VBkxV1yXZNE65dV4F83urXpaF8vFJ+mCUlz9USRfLrnT7PO7LrdvXIgez1119Xnz59JEm1a9fWwYMHNXfuXO3atUuvvfaa3Tt4r44dO6pZs2apvh4WFqa+ffum+np0dLT69u2rQoUKycXFRfnz51fnzp116tSpFG3Pnz+v3r17KygoSK6urgoICFDjxo21du1aO4zE8SbMW68Xm1ZT28ZVVSIonz4e3ErZs7lo7nLbn7hOmr9Bz1Qtqd4v1lbxwn4a0r2RypYI0NSFd09DbdmgsgZ0qa/QSiXSahjpyvi569Sh2ZNq16SagoPy6ZMhrZUjm4tmf59KTb/doFrVSqrPi7VVorCfhvZopHLBAZqycKOk5E/FJs5br/6d66pBaFmVLuavCSM66PzFWK3cuDsth2Yo6mp/1NQxejYsrZnrDmnuxiM6dDZGb0z9VVev31S7sOI221cu7qsthyO0+NdjOh0Zr/V7/tZ3vx3TE0V8LG2iLl9TRGyC5VH3iQAdOx+nX/efT6thGYq56hjjb+8DtGtcVcH37APMecA+QK3b+wAlCvvpTRv7AK0y+T6AxHx1hGmLNqllg6pqXq+yigb6aUTf5srmmlWLV2+12b5scEEN6tZYDZ/5n1yy2j7RL7RKSb3eub7qpPOjYvd65EB2v0KFCun5559X2bJl7dEfh4mOjlbVqlX1888/a+LEiTp69Ki+/fZbHT16VJUqVdKxY8csbU+cOKEKFSpo3bp1+vDDD7V3716tXr1aNWvWVK9evQwcxcO5fuOmdh88rdDKd980nZycFFqphLbtPWFzne17T6R4k61ZNVjb92aOT2n/yfUbNxV+8LTC7q9p5RLalkqNtu49rrBKwVbLnqla0vJvcPJslC5ExSms8t02nm7ZVSEkUNv2nLD7GNIj6mp/1NQxsmZxUrnC3tq49+4n22aztHHv36pU3MfmOlsPR6h8YW89USSPJKmQr7vq/K+A1oSfTvVnvPB0Ec3ZcNj+A0iHmKuO8W/2AbbZ2Ad4pmpwqv8OmRHz1f6u37ipfYfP6MknilmWOTk56cknimvX/pMG9iztPdQ1ZF988cVDb/DO0bP0ZujQofr777919OhR+fn5SZIKFiyoH3/8UcWKFVOvXr30ww8/SJJ69uwpk8mkrVu3KmfOnJZthISEqHPnzob0/1FExVzRrVtJ8vGy/pJuHy93HTl5weY6EVFxKQ79+nq5KyLK9iHjzCYqJv52Ta1r5OPloSMnHlBT7/vbuysiKk6SdOH2f+9v4+t9t01GR13tj5o6hreHq5yzOCkyNsFqeWRsgor757K5zuJfj8nbPZtWjWgok0zK6uykb9Yc0KdL99hs37BSIXnmdNG8jUfs3f10ibnqGHf2AXzv2wfwfcR9AB/2AawwX+3vUuwV3UpKkndu6/F753bTsdMRBvXKGA8VyD799NOH2pjJZEqXgSwpKUnffvut2rVrZwljd2TPnl09e/bUsGHDLHeMXL16tUaPHm0Vxu540I1LEhMTlZiYaHkeF5fxf5kAALY9VcpPrzcrqwFf/67tRyMV5OehMS9VUf/nr+qj71KejtS+ZjH9HH5G5y8l2NgaACCjeqhAdvz4433IOjIyUjExMSpZsqTN10uWLCmz2ayjR49KSj6nNzg42GbbBxkzZoxGjBjxn/pqD965cipLFidFRlsHwsjoyyk+MbvD19sjxUWpEdGX5ett+4LJzMY7l9vtmlrXKDI6Tr7eD6hp1P3tL1va573938ioy/LL42lpExF1OdPcuYq62h81dYyouETdvJUkH8/sVst9PLPrQsxVm+u82fIJLdj8l2atTz4F8cDpS8rh6qxPuz6lj5fsltl8t22BPDkVWia/Ony8zmFjSG+Yq45xZx8g4r59gIhH3AeIZB/ACvPV/nJ75lQWJ6cUN/CIuhSvPKncsCOj+s/XkBlhzpw5cnNzszw2b978UOuZ7/3r9x/apGbIkCGKjY21PE6ftn2dgKO5ZHVWueAAbdp29zqEpKQkbdp2WJXKBNpcp2KZQG3abn3dwsath1SxTGFHdvWx4ZLVWeWDA7Rx2yHLsrs1tV2jymUKW7WXpPVbDlr+DQr5eyuvt4dVm7j4BO3Yd0KVygbafQzpEXW1P2rqGDduJWn38SjVKJ3fssxkkkJL59e2w5E218nu4qyk+/6m3EpKfm6SyWp5u7Diioy9pp92GfN3wwjMVcf4N/sAlWzsA2zYeijVf4fMiPlqfy5ZnRVSvIB+33X3NO2kpCT9vuuI/leqkIE9S3uP/D1k6UGTJk1UpUoVy3N/f/8Htvfx8VGuXLl04MABm68fOHBAJpNJRYsWlZR86uXBgwcfuV+urq5ydXV95PUcoUebmnr13dkqX7KgnihVSBO/3aCr1xLVplFVSVLP4TOVzyeX3urVRJLUrVWYmnT/XOPmrNWzT4XouzU7FX7glD4Z0tqyzUuxV3TmwiWdj4yVJB29fS66r7eH5VOejKxn22fUc8Qs/a9kQT0REqgJ89brSkKi2jVOrmn3d2Yqn4+n5ZbA3VqHqVG3z/TV7LV69ukQfffTDoUfOKXP3mwjKXmedW9TUx99s1pBAT4q5O+t9yaulF8eTzUMLWfYONMadbU/auoY41f+qXE9qiv82EXtPBqp7g1ClMPVWXM3Ju/Iju9ZQ+eir2jktzskST/uPK2eDUK093iU5ZTFN1s+oR93nrIKaiaT1Da0mL7ddNQS2DIL5qpj9GxTU73u2QeYdHsfoO3tfYAet/cB3r5nH6Dx7X2AOk+FaMntfYBP2Qewwny1v04tamjQB9+qdPEAlQ0uqBmLNynh2nU9X7eyJGng+3OVN4+n+nVpKCn5RiB/3Z5712/e0oWLsTpw9KxyZHdVIf/kGyhdSUjUqbN3vx/yzPloHTh6Vp7uOZQ/b+40HuHDeSwDmbu7u9zdH/5QppOTk1q2bKk5c+bo3XfftbqOLCEhQePHj1fdunXl5eUlSapbt67GjRunPn36pLiOLCYm5rH4Auzn6lRQVEy83p+8UhFRl1W6uL8WfNbTcpj8zIVLcnK6+wlt5bJBmjSyo96buEKjJ6xQUICPZo7tqpJF7n4avHrzXvUeOcfyvOuw6ZKkAV3qa1DXBmkzMAM9/2wFXYyJ13uTVt4+ncBfi77odbem56PlZLpb0yrlgjRlVEeNnrBCI8cvV1CAj2Z/9IpKFb1b09c61NbVhES9/t48xcYnqGq5Ilr0RU9lc82a5uMzCnW1P2rqGEt+Py5vj2wa8sIT8s2VXX+ejNYL7/+kyNhrkpJPO7w3aH30XbjMZrPebFVB+bxyKCrumlbvOK1R83dYbTesTH4F+Lhlmrsr3ou56hjP1Umua2r7AGdt7ANMHtlRoyeu0Kjb+wCz7tsH+OG+fYAut/cBBmaSfQCJ+eoIDWr+T9GxV/TF9B8VeSlOJYv4a+r7XS2nLJ6LiLGqaURUnJp1+8Ty/JsFG/TNgg2qXK6IZn3SU5L056HT6tBvgqXNmAnfS5Kee7ai3h/UJi2G9chM5v9yjl4a69ixo2JiYrR06VKbr4eFhcnf3z/FF1Tny5dPzs7OqlKlirJnz66xY8eqdOnSOn78uIYNG6ZDhw7p999/V1BQkCTp2LFjeuqpp+Tl5aV3331XZcuW1c2bN7VmzRpNmDAh1SNt94uLi5Onp6f+joyRh0fm+PQoLWRxMv1zIwAZllfrb4zuQoYU/W36v4vw4yYpkx3xTCtO7AfYXURc4j83wiO5HBen0oV9FRsb+4854LG8huxB5s6dq//9739WjylTpsjb21t//PGHatasqW7duqlIkSJq2bKlihQpom3btlnCmCQFBQVp586dqlmzpvr166fSpUurTp06Wrt2rSZMmPCAnw4AAAAAD+9fHSHbvHmzJk2apL/++kuLFi2Sv7+/Zs2apcKFC+vpp592RD8fSxwhcwyOkAGZG0fIHIMjZPbHETLH4AiZ/XGEzP4ceoRs8eLFqlu3rrJnz65du3ZZvncrNjZW77333r/rMQAAAABkQo8cyEaNGqWJEydqypQpypr17gWHTz31lHbu3GnXzgEAAABARvbIgezQoUOqUaNGiuWenp6KiYmxR58AAAAAIFN45EDm5+eno0ePplj+yy+/WN0YAwAAAADwYI8cyLp27arXXntNW7Zskclk0t9//605c+aof//+6tGjhyP6CAAAAAAZ0iN/MfTgwYOVlJSkWrVq6erVq6pRo4ZcXV3Vv39/9e7d2xF9BAAAAIAM6ZEDmclk0tChQzVgwAAdPXpU8fHxKlWqlNzc3BzRPwAAAADIsB45kN3h4uKiUqVK2bMvAAAAAJCpPHIgq1mzpkym1L+Qb926df+pQwAAAACQWTxyICtfvrzV8xs3big8PFx//vmnXnrpJXv1CwAAAAAyvEcOZJ9++qnN5cOHD1d8fPx/7hAAAAAAZBaPfNv71LRv317ffPONvTYHAAAAABme3QLZ77//rmzZstlrcwAAAACQ4T3yKYvPP/+81XOz2axz585p+/bteuutt+zWMQAAAADI6B45kHl6elo9d3JyUokSJfTuu+/q2WeftVvHAAAAACCje6RAduvWLXXq1EllypRR7ty5HdUnAAAAAMgUHukasixZsujZZ59VTEyMg7oDAAAAAJnHI9/Uo3Tp0jp27Jgj+gIAAAAAmcojB7JRo0apf//+WrFihc6dO6e4uDirBwAAAADg4Tz0NWTvvvuu+vXrpwYNGkiSmjRpIpPJZHndbDbLZDLp1q1b9u8lAAAAAGRADx3IRowYoe7du2v9+vWO7A8AAAAAZBoPHcjMZrMkKTQ01GGdAQAAAIDM5JGuIbv3FEUAAAAAwH/zSN9DVrx48X8MZdHR0f+pQwAAAACQWTxSIBsxYoQ8PT0d1RcAAAAAyFQeKZC1bt1avr6+juoLAAAAAGQqD30NGdePAQAAAIB9PXQgu3OXRQAAAACAfTz0KYtJSUmO7AcAAAAAZDqPdNt7AAAAAID9EMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMMhDfw8Z/r0sTiZlcTIZ3Y0M4+YtvhPPEZyz8PkMHg/R33Y2ugsZUu7nJhjdhQwnanF3o7sAPBRXZ/YB7O36I9SU6gMAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAZxNroDcJwpCzbqy9lrFREVp9LF/PXBgBdUISQw1fZLf96p9yau1KlzUQoK8NHw3s307FMhltfNZrPGTFqpmUt/U2x8gqqUDdLHg1upSEHfNBhN+vD1ok0aN3udIqLjFFLUX2P6tdATIYVSbb9s7S69P3mlTp+LVlCAj97q1UR1nkyu6Y2btzRm4gr9/Pt+nTwbJXe3bAqtVEJv9WwiPx/PtBpSusBctT9q6hjU1f661A9R72bl5Zsrh/48EaVBU3/RziMRqbbv3qisOtcLUYE8boq+fE3LfvtL787eosQbtyRJbtmy6s22ldWoSmHl8cyuvccvavDXv2jX0ci0GlK6MHXhJn01J3muhhTz1/v9Wjxwri5bu0vvTVph+Xv1Tq+mqnPPXF2+PlzTv/tVuw+e0qW4q9owa5DKFC+QBiNJX3gPsL+ZS37RpG/XKTL6skoWya8Rrz2v8iVT37dauT5cH3/zg86cj1Zhfx8N7t5INauWsrweGPq6zfWGdG+sbm2esXv/7SFdHiHr2LGjmjVrlurrYWFhMplMev/991O81rBhQ5lMJg0fPtyqfd++fS3Pjx8/rrZt2yp//vzKli2bChQooKZNm+rgwYNW21q/fr0aNGggb29v5ciRQ6VKlVK/fv109uzZ/zpEh/vupx0a9tkSDepSXxtmDVLpYv5q3nucIqMv22y/ZfcxdRk2Xe2bVtPG2YPVMLSc2vefrP1H/7a0+Xzmz5o0f6M+GdJaa6b1V47sLmree5yuJd5Iq2EZasmanXr78yXq36We1s4YoJBi/mrZd3yqNd2655i6vT1D7RpX07oZA1W/Rlm9NHCqDvyVXNOEa9e159AZvdGprtbOGKDp77+soycj1H7A5LQcluGYq/ZHTR2Dutrfc08V0ahOT+mD+dsV1m+R/jwRpcVvN1Iez+w227eoXkzvvFhFY+dvV5Xe36r3V+v13NNF9Vb7KpY2n/cKU1i5Aur++Vo91Xe+1oWf1tLhjZXPK2daDctwS9bs0FufL9GAl+tr3YyBKl3UXy+89uC/V13fmq72jatp/cxBalCjrF4cOMXy90qSriZcV9VyQXrn1aZpNYx0h/cA+1u+bpdGjVuq116qq5VT+qlUkfzq0H+SLl6yXdMdfx5Xn5Gz1KpBFa2a0l/PVi+tV4Z+o0PHzlnabP1uhNVj7KDWMplMqh9aNq2G9cjSZSB7GAEBAZo+fbrVsrNnz2rt2rXKly9fquvduHFDderUUWxsrL777jsdOnRI8+fPV5kyZRQTE2NpN2nSJNWuXVt+fn5avHix9u/fr4kTJyo2NlYff/yxg0ZlP+PnrlOHZk+qXZNqCg7Kp0+GtFaObC6a/f3vNttP+naDalUrqT4v1laJwn4a2qORygUHaMrCjZKSP8GZOG+9+neuqwahZVW6mL8mjOig8xdjtXLj7rQcmmEmzluv9k2fVNtGVVWicD59NKilsmdz0dwVf9hsP3n+Rj1TtaRebV9LxQv7aUi3hipbooC+XrRZkuThll2LvuylZrWfUNFCeVWxdGG937+Fdh88rTPno9NyaIZirtofNXUM6mp/PZuU08w1+zV33SEdOnNJb0zcqKuJN9S+VrDN9pWD82rLwfNatPmITkde1vrdZ7R48xFVKJZ8NCGbSxY1qRak4TN/12/7z+n4+Th9MH+7jp2PU+d6ITa3mRGNn7deLzatpnaNqyo4KJ8+HtxK2bO5aM7yVObq/A2qVbWket+eq292b6SyJQI0deEmS5tWDSprQJf6Cq1UIq2Gke7wHmB/UxdsUOtG1dSyQRUVC/TT6H4vKHs2Fy1YtcVm+28WbVJo5WB1a/OMigbmVb+XGyikeAHNWLLZ0sbX28PqsebXP1Xtf0VVMH+etBrWI3tsA1mjRo108eJF/frrr5ZlM2bM0LPPPitf39QP8+7bt09//fWXxo8fr6pVq6pQoUJ66qmnNGrUKFWtWlWSdObMGfXp00d9+vTRN998o7CwMAUGBqpGjRqaOnWq3n77bYeP77+4fuOmwg+eVljlu2+aTk5OCq1cQtv2Hre5zta9xxVWyfoP4DNVS2rb3hOSpJNno3QhKk5hle+28XTLrgohgdq254Tdx5DeXL9xU7sPnbb6Q+Tk5KQalUpoeyo13f7nCdWoVNxqWc2qJVNtL0lx8ddkMpnk6W770+GMhrlqf9TUMair/WV1dlL5Ij7asPuMZZnZLG3cc1aVSuS1uc7WgxdUvoiPnrgdwArldVedCoW0ZscpSZKzk5Ocszjp2vVbVutdu35TVUv6OWgk6cv1Gze1++Bphd4/VyuVsMy9+23beyJF0HqmanCqczsz4j3A/q7fuKk/D5/RUxXu7is5OTnpqQrFtHPfSZvr7Np3wqq9JNWoVCLV9pHRl7X+9/1q1aCKzdfTi8c2kLm4uKhdu3aaNm2aZdn06dPVuXPnB67n4+MjJycnLVq0SLdu3bLZZuHChbp+/boGDhxo8/VcuXLZXJ6YmKi4uDirhxGiYuJ161aSfLzcrZb7eHkoIsp2nyKi4uTjfX97d0v7C7f/e38bX2/3VLeZkUTHXLFZU9/c7oqIsn1YPSIqTr5eHlbLfB7Q/lriDb07bpmer/OE3HNmjkDGXLU/auoY1NX+vN2zyTmLkyJjE6yWR8ZclW+uHDbXWbT5iN6bt00/jG6miIWvKHxie/3659/6ZPFOSVL8tRvaevC8BrSsIL/cOeTkZFLL0GKqVDyv8ubOHKcsRt3+e3X/3x9fL3dFRD9grqaY26n/vcqMeA+wv0uxyXM1T+77apTbXZGpzNXI6Ms2219Mpf3i1VuVM0c21a2Rfk9XlB7jQCZJnTt31oIFC3TlyhVt2rRJsbGxatSo0QPX8ff31xdffKG3335buXPn1jPPPKORI0fq2LFjljZHjhyRh4fHA099tGXMmDHy9PS0PAICAv7VuJD53Lh5S12GTpPZLH04qKXR3QGAdOmpkPx6o/kT6j95s8L6LVL791fr2QoF1f+FCpY23T5fK5PJpAPfvKQLC17RKw3LaPEvR5VkNhvYcwBGWPDDVjWr/YSyuWY1uisPlK4D2Zw5c+Tm5mZ5bN682er1cuXKqVixYlq0aJG++eYbvfjii3J2/ucbR/bq1Uvnz5/XnDlzVK1aNS1cuFAhISFas2aNpORzek0m0yP3d8iQIYqNjbU8Tp8+/cjbsAfvXG7KksUpxUWmkdFx8vX2sLmOr7eHIqPub3/Z0j7v7f/e3yYi6nKq28xIvHLltFnTiEuX5XvfJ1t3+Hp7pPg0MtJG+zth7Mz5aC36slemOTomMVcdgZo6BnW1v6jL13TzVpJ87ruBh0+uHIqIuWpznaFtK2vBxsOa9fMB7T8VrZVbjmvknC16vfn/dOfP9onzcWo0bJn8W09R6a6zVHvgd3LO4qST5zP+EQdJ8r799+r+vz8R0ZdTHDW7w9fbw8bcTv3vW2bEe4D95fZMnqv338Aj8tJl+aQyV3283G22z2Oj/dbdf+nYqQi1alTVfp12kHQdyJo0aaLw8HDLo2LFiinadO7cWePGjdOiRYv+8XTFe7m7u6tx48YaPXq0du/ererVq2vUqFGSpOLFiys2Nlbnzp37h61Yc3V1lYeHh9XDCC5ZnVU+OEAbtx2yLEtKStKmbYdVqUxhm+tULlPYqr0krd9yUJXKBEqSCvl7K6+3h1WbuPgE7dh3QpXKBtp9DOmNS1ZnlSsRoE3bDluWJSUlafO2Q6qYSk0rlg7U5nvaS9LGrQet2t8JY8dOR2rRl73k5Zk5Tqm5g7lqf9TUMair/d24maTwvyIVWvburdNNJqlGGX9tO3TB5jrZXZ2VlGR9pOvWLfPtda0/SL2aeFMXLl2VZ04X1fpfgFZtzRzXQ7lkdVa54JR/r5LnaqDNdSqVCdSm7dZ/rzZsPZTq3M6MeA+wP5eszipdvIB+22E9V3/beSTVrxT6X0igVXtJ+mX7YZvt56/aojIlCqhUUX/7dtwB0nUgc3d3V9GiRS2P7NlTHjlo27at9u7dq9KlS6tUqVI2tvLPTCaTgoODdeXKFUlSixYt5OLiorFjx9psf+/dGNOrnm2f0cylv2neij906Ph5vfH+fF1JSFS7xsmfEnR/Z6ZGfLXM0r5b6zCt/X2/vpq9VodPnNf7k1cq/MApdX0hVFJyjbq3qamPvlmtVRv3aN/Rs+oxfJb88niqYWg5Q8aY1rq3qanZ3/+mb1du0eHj5zVg7AJdvXZdbRomXyjaa8QsjRz/vaX9K61Cte6PAxo/Z52OnLigsVNWKfzAab3corqk5DDWecjXCj9wShNGdNCtJLMuRMXpQlScrt+4acgYjcBctT9q6hjU1f7Gf79bHeqUVOuaJVS8QC590q2GcmbLqjlrk7+GZkKfZ/T2Pbe0X73thDrVC9HzTxdVwf+3d+dhUZX/G8fvQQRRlhAXFAlURFARyz375pKGmkvmvqT+3HLJ1LKstLTc2yxL01wz17S0XLLctcw0EzUllxQ1BTVAEBdcOL8/zNGRxW2Gg/R+XddcXJzznDPP8+GZM3PPmTkU8lCt8GJ6o10Vrdx2xBrU6lTw15OP+FvXLx3eVPv/PqM5a/el24ecqHfb2vry282at/xX7Tscq4Fjv9L5iylq9++Zgl7DZumdCTeer55vfW2uTphzba6OnbJCkVFH1a3lE9Y2CYnntHv/39p3OFaSdPDISe3e/7f1e1D/BRwD7K9bq1qat3yLFq3cqoPRJzX4w0U6f+GSWja49rh/aeQcjf18mbV9lxZPaMPWPzVlwTodPHJS42as1O59x9Sp2f9s9nv23EWtWL9TrZ/O/mfHpBzwj6G9vb0VExOj3Lnv7LOhkZGRGjp0qJ577jmVKVNGLi4u2rBhg6ZPn65BgwZJunZJ/XHjxumFF15QUlKSOnbsqMDAQP3999+aNWuW3N3ds/2l7599qqL+OZOsUZOX61TcWYUF+2nR+D7WU+B/x8bL6aZ3E6uGl9CUEZ018rNlGj5xqUr4F9Ts93uoTFBRa5t+Hevq/IUUDRg1T4nJF1QtvKQWje+d7T+Xay/N6j2quDPJGjtlxb//ELKYFozrdVNNE2zeoa1SvoQmvdNJoycv18hJS1XCv5C+eLebQkteq2nMqTNauekPSVLt58ba3NeSCX1Vo2KpLBqZuZir9kdNHYO62t/in/9SAU83vdGmsgp559Xuw/+oxTvLrBf6KFbQ3ea7X+8v3C7DuPbRxSL58yku6YJW/nZEw2ffuES2Z14XvfVcVRX1cVfC2YtauuWQRszZqitXU7N8fGZpVu/aXB3z+bW5Wi7YT1991Ns6V4+fTJCTk+3z1efDO2vkpGUa8dkylfAvqC/f7W59vpKk7zftVt/hc6y/dxsyU5L0arcGGtS9YdYMzGQcA+yvcZ1HFH8mWeOmr9Tp+CSFBvnpi/eet1485fipBFlumqsVyxXXx28+pw+mrdB7U5YrsFhBfT6yi0qXsL3uw9I1v8swDDV58tEsHc+9shhG9vuWa+fOnXXmzBktWbIk3fW1atVShQoV9NFHH6W7vkKFCnrmmWes/xz65vb//POPhg8frrVr1yo6OloWi0WBgYHq1KmTBgwYICenGycNV69erffff19bt27VhQsXFBgYqEaNGumll166owt+JCUlycvLSyfjEk37+GJO9F96Us1Kzrmy9QlzAA7m3ewzs7uQ48R93dPsLuRIN4dJ2Efi+f/GP6LOSmeTklTKv4ASE2+fA7JlIMspCGSOQSBzDAIZ8N9GILM/ApljEMjsj0Bmf3cTyHgFBgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmMTZ7A4Ad8s5F+8jOIJhGGZ3IUeyWCxmdyHHuZrKXHWE+G96mt2FHCd/w/fM7kKOlPD9q2Z3IcdxduK5yt7upqa8sgUAAAAAkxDIAAAAAMAkBDIAAAAAMAmBDAAAAABMQiADAAAAAJMQyAAAAADAJAQyAAAAADAJgQwAAAAATEIgAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAExCIAMAAAAAkxDIAAAAAMAkBDIAAAAAMAmBDAAAAABMQiADAAAAAJMQyAAAAADAJAQyAAAAADAJgQwAAAAATEIgAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAExCIAMAAAAAkxDIAAAAAMAkBDIAAAAAMAmBDAAAAABMQiADAAAAAJMQyAAAAADAJAQyAAAAADAJgQwAAAAATEIgAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAEzibHYH4DhTvtqgT2av0am4JJUr5aexr7RUxbKBGbZfsvp3jZq0XEdj4lTCv6CG9X1GT9Uoa11vGIZGT16uWUs2KzH5gqqWL6EPXmutkg8XyoLRZA/U1DGmLtxorWvZUn4aO7DFbeq6Q6MnL9PRmPhrdX2hqerdWtfPV+hLa12L6/1B/626MlcdY9rCjfp0zo25OublFno0k7p+u+baXD3271x9q4/tXF22LlIzv/lZO/88qoSk81r35SCFBRfLgpFkHzz+HaNbo0fUt0UVFfLOpz8OndKgz1br9/2x6bZ1zuWkAa2rqW3dsiri46GDf8dr2PQNWrP9sLXNgFZV1ahGsEoV89HFS5e1de8JDZu+QQePx2fVkLIFjq32N/ObTZo0b61Ox59VaMmiGt6/uR4pE5Bh+2XrIvXe1BX6OzZegcUK6o2ejfVk9TI2bQ5Ex2rUpKXaEvmXrlxNVXBgYX0+oov8Cns7ejj3hDNkOdQ3P27XkI8Wa1C3Blr/5SCVK+Wn5n0n6HT82XTb/7rzkLoNmakOTatrw+zX9HTNcHUY+Ln2HjxhbfPxrNWavGCDPny9jVbNGKi8bi5q3neCLqZczqphmYqaOsY3q67V9dVuDbRu1qsqV8pPLV6cmHFddx1S9zdnqn2T6lr/5SA1rFleHV6Zor1/3ajr+Fmr9fmCDfrgtdZaNf1l5XVzVYsXJ/5n6spcdYzFq7brzY8X65WuDbT2i1dVNshPLftlPFe37jqkHm/OVPvG1bVu1iA1fKK8Or46RVE3zdXzFy6pangJvfVC06waRrbC498xmj0RohE9amvsnJ9Vq+8X+uPwaX09opUKeOVNt/2QTv9T5wbhGvTZGlV7fppmrIjUl28+o7CSN0LBY2H+mrp0h54a8KWefeMr5XZ20jcjWyqva+6sGpbpOLba33drftc7ny7RgM719f3UgSoT5KcOL0/SPwnp1/S33YfV5+1ZavN0Na2cNlD1/xembm9M05+HYqxtoo//o2Z9xqvkw4W1cPwLWjXzVfXrFCFXl+x7HopAloFjx46pS5cuKlq0qFxcXBQQEKB+/fopLi7O7K7dkYlz16rjM4+pfZPqCilRRB++3kZ587ho9ne/pNt+8vz1erJ6qF58rq5KF/fV4F6NFB7irykLN0i69g7OpHnrNLBLhBrWLK9ypfz02dsdFftPopZv2JmVQzMNNXWMiXPXqeMz1dW+cbVrdX2ttfLmcdGcpZnUtdpNde3ZSOVD/DX1q42S/q3r/PV6+d+6li3lp8+GPfdvXXdl5dBMw1x1jM/mrdNzTaurXeNqKl2iiD54rbXc8rhobkZzdcF61akWqr7P1VVwcV+93rORypf219SFG61tWjWsole6NVDNyqWzahjZCo9/x+jdrJJmfb9Lc1f9oX1H4/TSJz/ofMpldXgqLN32reqU1bgFW7Rq2yEdiU3U9OWRWrXtkF54trK1Tcs3F2ne6j/059E4/XH4tHp/uEL+hb1UoVThrBqW6Ti22t/nC9arbePqav10VQUX99WYgS2VJ4+L5i//Nd320xZtUK0qIerVro5KBfrqlW4NVS64mGZ+s8na5t3Pl6tOtTIa0ruJygUXU6BfAT31eDkV8PbIqmHdNQJZOg4dOqRKlSrpwIEDmjdvng4ePKhJkyZpzZo1ql69uuLjs/fp+UuXryjyz2OqVeXGE7yTk5NqVimtbbsPp7vN1t2HVatyiM2yOtVCtW13tCTpyPE4nYxLUq0qN9p4ubupYtlAbdsVbfcxZDfU1DEuXb6inX8es3kx6uTkpJqVS1vrdKttu6NVs4rti9c61UKsf4cjJ67X9UYbz+t1zeBvlZMwVx3DOldvrWsmc/W33dFpglbtaiH67T8wD+8Ej3/HyO3spAqlfLU+Mtq6zDCkDZFHVDm0aLrbuObOpYuXrtgsu3jpiqqVzfjjs555XSVJCWcv3n+nHwAcW+3v0uUr2r3/b/2vYrB1mZOTk/5XKVi/74lOd5vtf0Trf5WCbZbVrBKi7X9ca5+amqo1v+xVCf+Cav/SZwpvPESNenyolRuz9xsyBLJ09OnTRy4uLvrxxx9Vs2ZNPfzww2rQoIFWr16t48ePa/DgwWZ3MVNxZ5J19WqqCua3fSegYH5PnYpLSnebU3FJKuhza3sPa/uT//68tU0hH48M95mTUFPHiDtz7t+6etosL5jfw1qfW52KS1KhW/4OhfJ76NS/Hxmx1jXN3+q/UVfmqmNkNldPxWdS1/Tmalz6H8X5r+Hx7xg+nnnlnMtJpxPO2yw/nXBOhbzzpbvN2u2H1fvZyipR1FsWi1TrkQA1eixYhfOn395ikUY//6S27PlbUUf+sfsYsiOOrfYXn3gu3ZoW8M54/Kfjz6pAOo/v0/8eh/9JSNa5CymaMGeNalUN1dwPe6r+E+XVfcgM/bLjoGMGYgcEslvEx8frhx9+UO/eveXm5mazztfXV+3bt9eCBQtkGEaabVNSUpSUlGRzAwAAyM5em7xGh44naOvnXXVq6UC927ue5q7ardTUtK91JOn9PvUUGlhAXcd8l8U9BTKX+u/r86ceL6furWupbKlieqFDXdV9rIxmf/uzyb3LGIHsFgcOHJBhGAoNDU13fWhoqBISEnT69Ok060aPHi0vLy/rzd/f39HdTZfPQ+7KlcspzZdMT8cnqZCPZ7rbFPLx1Om4W9uftbYv/O/PW9ucijub4T5zEmrqGD4P5fu3rrZvXpyOP2utz60K+Xha3w2/7lT8Weu75ta6pvlb/Tfqylx1jMzmaqH8mdQ1vbnqk32/x5CVePw7RlzSeV25mqqC3rYX8CjonU+nEs6lv03iBXUYvlh+zcapfKdJqtJ9qs5dvKzo2MQ0bd/tVVcRVUqq8aD5OvFPskPGkB1xbLW//F750q3pPwkZj79gfg/9k87j+/qZ9vxe+eScy0nBgb42bYICCuv4yTP267ydEcgykN4ZsNt5/fXXlZiYaL0dO3bMAT27PZfczqoQ4q8N2/ZZl6Wmpmrjtv2qHFY83W2qhBW3aS9J6379U5XDAiVJAX4+KuzjadMmKfmCtu+JVuXygXYfQ3ZDTR3DJbezwkP8tXHbfuuy1NRUbfhtv7VOt6ocFmjTXpLW/7rP+ncIKJpJXTP4W+UkzFXHyGiuXqtrYLrbVAoL1MbfbOfqhq37VOk/MA/vBI9/x7h8JVWRB2JVs8KNy4ZbLNITFQK0LepEJltKKZevKiYuWc65nNS4RrC+/+WAzfp3e9XV04+VUpPXFujoybRhLSfj2Gp/LrmdFRZcTD9tvzHPUlNT9dP2/Rn+O5GK5QJt2kvSpt/2qWK5QOs+w0Mf1l9HT9m0OXTstPx8s+cl7yUCWRpBQUGyWCyKiopKd31UVJS8vb1VsGDBNOtcXV3l6elpczNL73Z1NGvJZs1btkX7DsfqpTELdO5Cito3riZJ6jl0lt7+9Ftr++fb1NKaX/bq09lrtD86VmM+X67IqKPq3rKmJMlisahn29p6f/pKrdiwS3sOHlevYV/Kt4CXnq4ZbsoYsxo1dYze7Wpr1rebNW/Zr9p3OFYvj/1K5y+kqF2ja3XtNXSW3plw42Mx1rrOuV7XFYqMOqpurZ6Q9G9d29TSB9N/0Pcbd2vvwRPqba1reVPGmNWYq47Rq21tffntZs1f/qv2H47VwLFf6fzFFLX9d672HjZLw2+eq61rae0vezVhzhodiI7V2Cn/ztWWT1jbJCSe0+79f2vf4Wv/H+rgkZPavf/vDL9DldPw+HeMiYt/U8f64WpTt6yC/fPrwxeeUj7X3Jqzarck6bOXG+qtzjfmYcXSRdTosVIK8PVS9bLFtGhESzlZLPp40VZrm/f71FOrOmXU/d1lSr5wSYW886mQdz7lycaXErc3jq3216N1Lc1b9osWfr9VB6Jj9foHC3XhwiW1blhVktRvxGyNnrTU2r5ri5pa/2uUJs9fp4NHTuqD6d9r15/H1PnZ/1nb9GxbR0vX7tCc737R4b9Pa8bXm7R68x51avZ4lo/vTv13HkV3yMfHR/Xq1dPEiRM1YMAAm++RxcbGas6cOerYsaMsFouJvby9Z5+qqH/OJGvU5OU6FXdWYcF+WjS+j/UU8N+x8XK6aQxVw0toyojOGvnZMg2fuFQl/Atq9vs9VCboxhWZ+nWsq/MXUjRg1DwlJl9QtfCSWjS+t/L8R/4HCTV1jGfrVVRcQrJGf36truWC/bTw49436noyQU5ON9W1fAl9PryzRk1aphETl12r63vdVabkjbq+2LGuzl28dFNdS2jhx/+dujJXHaNZvYqKO5OsMTfN1a8+yniuVilfQpP/nasjP7s2V2e9212hN83VlZt2q+/wOdbfuw+ZKUl6pVsDDereMGsGZiIe/46xeOOfKuDlpjc6PK5C+fNp91+n1OLNhTp95tqFPooV8rR+10aSXF2cNbjT/xTo+5DOXbikVdsOqed7y5V0LsXapmujRyRJy99ta3NfvT9YoXmr/8iCUZmPY6v9NXnyUcWdOaf3p32v0/FJKhPkpy/ff956oY/jJxNsaloprLg+HdpR705ZrrGfL1PxYgU1dVRXhZQoYm3T4InyGj2wpT6dvVpvffyNSj5cUJ8P/z9VKV8iy8d3pyzGvXw2L4c7cOCAHnvsMYWGhmrEiBEqXry49uzZo1deeUUpKSnasmWL8ufPf9v9JCUlycvLSyfjEk09WwbcCQ4FjpHd37x5EF3N4EIDuD9OTFW7y9/wPbO7kCMlfP+q2V3Icc5dvHL7RrgrZ5OSVNzPR4mJt88BfGQxHaVKldJvv/2mEiVKqFWrVipZsqR69Oih2rVr65dffrmjMAYAAAAAt8NHFjMQEBCgmTNnmt0NAAAAADkYZ8gAAAAAwCQEMgAAAAAwCYEMAAAAAExCIAMAAAAAkxDIAAAAAMAkBDIAAAAAMAmBDAAAAABMQiADAAAAAJMQyAAAAADAJAQyAAAAADAJgQwAAAAATEIgAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAExCIAMAAAAAkxDIAAAAAMAkBDIAAAAAMAmBDAAAAABMQiADAAAAAJMQyAAAAADAJAQyAAAAADAJgQwAAAAATEIgAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAExCIAMAAAAAkxDIAAAAAMAkBDIAAAAAMAmBDAAAAABMQiADAAAAAJMQyAAAAADAJAQyAAAAADAJgQwAAAAATEIgAwAAAACTOJvdAQDZg8ViMbsLwB3J5cRcxYMh4ftXze5CjuRdpa/ZXchxErZ+YnYXcpyrl+48ZnGGDAAAAABMQiADAAAAAJMQyAAAAADAJAQyAAAAADAJgQwAAAAATEIgAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAExCIAMAAAAAkxDIAAAAAMAkBDIAAAAAMAmBDAAAAABMQiADAAAAAJMQyAAAAADAJAQyAAAAADAJgQwAAAAATEIgAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAExCIAMAAAAAkxDIAAAAAMAkBDIAAAAAMAmBDAAAAABMQiADAAAAAJMQyAAAAADAJAQyAAAAADAJgQwAAAAATEIgAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAExCIAMAAAAAkxDIAAAAAMAkBDIAAAAAMAmBLAeb8tUGlW/ylnxr9Ffdzu9p+57oTNsvWf27qrQYLt8a/fVYm5H68ec9NusNw9CoScsUUv8NFXl8gJ7p/Yn+OnrKgSPIfqipY1BX+6OmjkFd7Y+aOgZ1tb9uLf6nnUuGKWbTh1o1/WU9WiYgw7bOuZz0Stf6+v2btxSz6UNtmvOanqwWatPGPa+rRg14Vru+fVsnNn6gH6YO0COhDzt6GNkOc5VAlmN98+N2DflosQZ1a6D1Xw5SuVJ+at53gk7Hn023/a87D6nbkJnq0LS6Nsx+TU/XDFeHgZ9r78ET1jYfz1qtyQs26MPX22jVjIHK6+ai5n0n6GLK5awalqmoqWNQV/ujpo5BXe2PmjoGdbW/ZnUf1Yj+zTR26veq1fFd/XHguL4e31sFvN3TbT+kVyN1blZDg95fpGqtR2rGNz/py3e7KSy4mLXNx4PbqVbVEPUcNks12o3W2l//1JIJL6hIQa+sGpbpmKvXZOtA1rlzZz3zzDOZtrlw4YKGDh2q4OBgubq6qkCBAmrZsqX27LFNy8OGDZPFYpHFYlGuXLnk7++vHj16KD4+Ps0+d+zYodatW6tIkSJydXVVQECAGjVqpKVLl8owDHsO0WEmzl2rjs88pvZNqiukRBF9+Hob5c3jotnf/ZJu+8nz1+vJ6qF68bm6Kl3cV4N7NVJ4iL+mLNwg6dq7DZPmrdPALhFqWLO8ypXy02dvd1TsP4lavmFnVg7NNNTUMair/VFTx6Cu9kdNHYO62l/vdrU1a8kvmrvsV+07HKuXxizQ+YuX1KFx9XTbt2pQReNm/qhVm/fqyIk4Tf/6J63avFcvtK8jScrjmltNaodr2CffavOOv3T47380dsr3OnTstLo0fzwrh2Yq5uo12TqQ3U5KSorq1q2r6dOna8SIEdq/f79WrFihK1euqGrVqtqyZYtN+7JlyyomJkZHjx7VjBkztHLlSvXq1cumzbfffqtq1aopOTlZX3zxhaKiorRy5Uo1a9ZMQ4YMUWJiYlYO8Z5cunxFkX8eU60qpa3LnJycVLNKaW3bfTjdbbbuPqxalUNsltWpFqptu6MlSUeOx+lkXJJqVbnRxsvdTRXLBmrbrmi7jyG7oaaOQV3tj5o6BnW1P2rqGNTV/nI751KFEH+t37bPuswwDG3Ytk+VwwLT3cbVxVkXL9mekbmYclnVwktIuvaRRmfnXBm0KWnfAWRTzNUbnM3uwP346KOP9Msvv2jHjh0KDw+XJAUEBOjrr79W1apV1bVrV/3xxx+yWCySJGdnZ/n6+kqS/Pz81LJlS82YMcO6v3Pnzqlr1656+umn9c0339jcV2hoqLp27ZrpGbKUlBSlpKRYf09KSrLbWO9G3JlkXb2aqoL5PWyWF8zvqQPRJ9Pd5lRckgr63NreQ6firo3h5L8/b21TyOdGm5yMmjoGdbU/auoY1NX+qKljUFf783kon5ydc+l0vO1YT8efVamAwulus3ZLlHq3q2M9+1WzcrAa1Q5XLqdrr0mTz6do665DeqVLfe0/HKtT8WfV4qmKqhxWXIf+Pu3wMWUHzNUbHugzZHPnzlW9evWsYew6JycnDRgwQHv37tXOnemfnoyOjtYPP/wgFxcX67Iff/xRcXFxevXVVzO8z+vhLj2jR4+Wl5eX9ebv73+XIwIAAMCD7rUPvtahY6e19ashOvXzOL37SkvNXbpFqak33th/fuiXslikqBUjdfKncerRupa+/nG7TRv8NzzQgWz//v0KDQ1Nd9315fv377cu2717t9zd3eXm5qbixYtrz549GjRokM3+JKl06RunTrdt2yZ3d3frbdmyZRn25/XXX1diYqL1duzYsfsa373yechduXI5pflC5On4JBXy8Ux3m0I+njodd2v7s9b2hf/9eWubU3FnM9xnTkJNHYO62h81dQzqan/U1DGoq/3FnTmnK1euqmB+27HefGYm7TbJ6vDKFPnVfFnlmw5VlZYjdO5CiqJPxFnbRB//R416jpffEy+rXOO3VPf/3pezcy4dOR6X7j5zGubqDQ9EIJszZ45NKNq0aZN13d1cZKN06dKKjIzUtm3bNGjQIEVERKhv376ZblO+fHlFRkYqMjJS586d05UrVzJs6+rqKk9PT5ubGVxyO6tCiL823PRZ59TUVG3ctl+Vw4qnu02VsOI27SVp3a9/Wj8bHeDno8I+njZtkpIvaPueaFUuH2j3MWQ31NQxqKv9UVPHoK72R00dg7ra3+UrVxX55zHVrBxsXWaxWPREpWDrd5cyknLpimJOJ8o5l5Ma166g7zfsTtPm/MVLOhmXJC8PNz1ZLUQrNu6y9xCyJebqDQ9EIGvSpIk1FEVGRqpSpUqSpODgYEVFRaW7zfXlwcE3HjwuLi4KCgpSuXLlNGbMGOXKlUtvv/22dX2pUqUkSfv23fgjurq6KigoSEFBQXYflyP1bldHs5Zs1rxlW6xXAzp3IUXtG1eTJPUcOktvf/qttf3zbWppzS979ensNdofHasxny9XZNRRdW9ZU9K1A0/PtrX1/vSVWrFhl/YcPK5ew76UbwEvPV0zPN0+5DTU1DGoq/1RU8egrvZHTR2DutrfxLnr1LHpY2rzdBUFBxbWh4NaKZ+bq+Ysu3YBuc+GPae3eje2tq9YNkCNaoUroKiPqlcoqUXje8vJyaKPv1xtbVOnWoierBaqh4v6qFaV0lr62YvaH31Sc5ZuSXP/ORVz9ZoH4qIeHh4e8vDwSLO8TZs2Gjx4sHbu3GnzPbLU1FSNGzdOZcqUSfP9spsNGTJEderUUa9evVS0aFE99dRTyp8/v8aOHavFixc7ZCxZ5dmnKuqfM8kaNXm5TsWdVViwnxaN72M9Xft3bLycbvo+XNXwEpoyorNGfrZMwycuVQn/gpr9fg+VCSpqbdOvY12dv5CiAaPmKTH5gqqFXzvA5HHNneXjMwM1dQzqan/U1DGoq/1RU8egrva3ePXvKuDtrjd6PK1CPh7avf+4WvSbaP24XbHC3jbf/XJ1ya3BPZ9WoF8BnbuQolWb96rn0FlKSr5gbePp7qa3ejdW0UIPKSHpvJau3akRny3VlaupWT4+szBXr7EY2fgfa3Xu3FlnzpzRkiVL0l1/8eJF1apVSydOnNAHH3ygqlWr6uTJkxo1apRWrVql1atXq1q1awl72LBhWrJkiSIjI232UbVqVVWuXFmffvqpJGnx4sVq3bq16tWrpxdffFGlSpVScnKyVq5cqUGDBum7775T48aNb+1KupKSkuTl5aWTcYmmfXwRAAAgJ/GukvnXTXD3ErZ+YnYXcpykpCQV9vFSYuLtc8AD8ZHFjOTJk0dr165Vx44d9cYbbygoKEj169dXrly5tGXLFmsYy8yAAQM0depU6wU4mjVrps2bNytv3rzq2LGjSpcurTp16mjt2rWaP3++GjVq5OhhAQAAAPiPyNZnyB50nCEDAACwL86Q2R9nyOzvP3OGDAAAAAAeZAQyAAAAADAJgQwAAAAATEIgAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAExCIAMAAAAAkxDIAAAAAMAkBDIAAAAAMAmBDAAAAABMQiADAAAAAJMQyAAAAADAJAQyAAAAADAJgQwAAAAATEIgAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAExCIAMAAAAAkxDIAAAAAMAkBDIAAAAAMAmBDAAAAABMQiADAAAAAJMQyAAAAADAJAQyAAAAADAJgQwAAAAATEIgAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAExCIAMAAAAAkxDIAAAAAMAkBDIAAAAAMAmBDAAAAABM4mx2BwAAAIA7lbD1E7O7kON4V37B7C7kOMbVS3fcljNkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAlkONuWrDSrf5C351uivup3f0/Y90Zm2X7L6d1VpMVy+NfrrsTYj9ePPe2zWG4ahUZOWKaT+Gyry+AA90/sT/XX0lANHkP1QU8egrvZHTR2DutofNXUM6uoY1NW+HnukpOZ9+Lz2rhiphG2fqmHN8rfdpsajpbT+y0GK/Xmctn8zVG0bVU3TplvLJ7Tz27cV89M4rZoxUI+WCXBE9+3G1EDWuXNnWSwW683Hx0f169fXrl27MtwmOjpaFotFuXLl0vHjx23WxcTEyNnZWRaLRdHR0TbtIyMjre0WL16satWqycvLSx4eHipbtqz69+9vs69Lly7p3XffVXh4uPLmzasCBQqoRo0amjFjhi5fvmyvEjjMNz9u15CPFmtQtwZa/+UglSvlp+Z9J+h0/Nl02/+685C6DZmpDk2ra8Ps1/R0zXB1GPi59h48YW3z8azVmrxggz58vY1WzRiovG4uat53gi6mZP962AM1dQzqan/U1DGoq/1RU8egro5BXe0vr5ur/th/XK+8u+CO2j9c1EcLPuqpTdv364n2YzRp3jqNH9xOdaqFWts0q/eoRvRvprFTv1et58bqjwPH9fUnfVTA291Rw7hvpp8hq1+/vmJiYhQTE6M1a9bI2dlZjRo1uu12fn5+mjVrls2yL774Qn5+fplut2bNGrVu3VrNmzfX1q1btX37do0cOdImZF26dEkREREaM2aMevTooc2bN2vr1q3q06ePPvnkE+3ZsyeTe8geJs5dq47PPKb2TaorpEQRffh6G+XN46LZ3/2SbvvJ89fryeqhevG5uipd3FeDezVSeIi/pizcIOnaOziT5q3TwC4RalizvMqV8tNnb3dU7D+JWr5hZ1YOzTTU1DGoq/1RU8egrvZHTR2DujoGdbW/1Zv3auSkZVq+PuOTMTfr8uzjOnoiTm9+tFj7o09qysKN+m5tpHq1q21t07tdHc1asllzl27RvsOxemn0fJ2/eEkdmlR31DDum+mBzNXVVb6+vvL19VWFChX02muv6dixYzp9+nSm23Xq1EkzZsywWTZjxgx16tQp0+2WLl2qGjVq6JVXXlHp0qUVHBysZ555RhMmTLC2+eijj7Rx40atWbNGffr0UYUKFVSiRAm1a9dOv/76q0qVKnXvA84Cly5fUeSfx1SrSmnrMicnJ9WsUlrbdh9Od5utuw+rVuUQm2V1qoVq2+5oSdKR43E6GZekWlVutPFyd1PFsoHativa7mPIbqipY1BX+6OmjkFd7Y+aOgZ1dQzqmj1UDiuu9Vv32SxbsyVKVcKKS5JyO+dShRB/mzaGYWjD1n2q/G+b7Mj0QHaz5ORkzZ49W0FBQfLx8cm0bZMmTZSQkKCffvpJkvTTTz8pISFBjRs3znQ7X19f7dmzR3/88UeGbebMmaO6devqkUceSbMud+7cypcvX7rbpaSkKCkpyeZmhrgzybp6NVUF83vYLC+Y31On4tLv06m4JBX0ubW9h7X9yX9/3tqmkI9HhvvMSaipY1BX+6OmjkFd7Y+aOgZ1dQzqmj0U8vFM8xHR03FJ8nR3Ux7X3PJ5yF3OzrnStolPUiEfz6zs6l0xPZAtW7ZM7u7ucnd3l4eHh7777jstWLBATk6Zdy137tzq0KGDpk+fLkmaPn26OnTooNy5c2e6Xd++fVW5cmWFhYUpMDBQbdq00fTp05WSkmJtc+DAAYWEhGSyl/SNHj1aXl5e1pu/v/9d7wMAAADAf4fpgax27dqKjIxUZGSktm7dqoiICDVo0EBHjhxRgwYNrGGtbNmyabbt0qWLFi5cqNjYWC1cuFBdunS57f3ly5dPy5cv18GDBzVkyBC5u7vr5ZdfVpUqVXT+/HlJ105t3ovXX39diYmJ1tuxY8fuaT/3y+chd+XK5XRX7w4U8vHU6bhb25+1ti/8789b25yKO5ut33GwF2rqGNTV/qipY1BX+6OmjkFdHYO6Zg+n4pLSnqX08VRS8gVdTLmsuDPJunLl6l2dycwOTA9k+fLlU1BQkIKCglS5cmVNnTpV586d05QpUzR16lRrWFuxYkWabcPCwhQSEqK2bdsqNDRU5cqVu+P7LVmypLp166apU6fq999/1969e7VgwbUrvAQHB+vPP/+867G4urrK09PT5mYGl9zOqhDirw3bbnx+NjU1VRu37c/w87NVworbtJekdb/+qcphgZKkAD8fFfbxtGmTlHxB2/dEq3L5QLuPIbuhpo5BXe2PmjoGdbU/auoY1NUxqGv2sG33YdWsXNpmWe0qIdr67/f4Ll+5qsg/j9m0sVgseqJycIbf9csOTA9kt7JYLHJyctKFCxfk5+dnDWsBAen//4AuXbpo/fr1d3R2LCOBgYHKmzevzp07J0lq166dVq9erR07dqRpe/nyZWu77Oz6FWbmLfv3CjNjFujchRS1b1xNktRz6Cy9/em31vbPt6mlNb/s1aez12h/dKzGfL5ckVFH1b1lTUnX/i4929bW+9NXasWGXdpz8Lh6DftSvgW89HTNcFPGmNWoqWNQV/ujpo5BXe2PmjoGdXUM6mp/+dxcVC7YT+WCr10lPaCoj8oF+6lYYW9J0lt9muizYc9Z20//5icF+Pno7b5NVSqgsLq2+J+eqfuIPpu7ztrm+tUw2zxdVcGBhfXha62Vz81Vc5ZuydrB3QVnszuQkpKi2NhYSVJCQoI+/fRTJScn3/biHNd1795dLVu21EMPPXRH7YcNG6bz58+rYcOGCggI0JkzZzR+/HhdvnxZ9erVkyT1799fy5cv15NPPqnhw4fr8ccfl4eHh3777TeNHTtW06ZNU4UKFe5luFnm2acq6p8zyRo1eblOxZ1VWLCfFo3vYz0F/ndsvJwsFmv7quElNGVEZ438bJmGT1yqEv4FNfv9HioTVNTapl/Hujp/IUUDRs1TYvIFVQsvqUXjeyuPa+bf28spqKljUFf7o6aOQV3tj5o6BnV1DOpqfxVCA7Rscj/r76Neai5Jmrtsi/q8PVuFC3iqmG9+6/qjJ+LUuv8kjXrpWT3fppZOnDqjF0fO1dotUdY2i1f9rgIPueuN559WIR8P7d5/XC1ezPj/xWUHFuNevzBlB507d9YXX3xh/d3Dw0MhISEaNGiQmjdvnu420dHRKl68uHbs2JFuKIqMjNQjjzyiw4cPKzAwME37devWacKECdq6datOnjwpb29vPfLIIxo8eLAef/xx635SUlI0btw4zZ07VwcOHFDevHkVGhqq7t27q3379nJ2vn2WTUpKkpeXl07GJZr28UUAAAAgM96VXzC7CzmOcfWSUnZPUWLi7XOAqYEspyOQAQAAILsjkNnf3QSybPcdMgAAAAD4ryCQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEmezO5CTGYYhSTqblGRyTwAAAID0GVcvmd2FHOd6Ta/ngcwQyBzo7NmzkqSg4v4m9wQAAABAVjt79qy8vLwybWMx7iS24Z6kpqbqxIkT8vDwkMViMbs7mUpKSpK/v7+OHTsmT09Ps7uTY1BX+6OmjkFd7Y+aOgZ1tT9q6hjU1TEelLoahqGzZ8+qaNGicnLK/FtinCFzICcnJxUrVszsbtwVT0/PbD25H1TU1f6oqWNQV/ujpo5BXe2PmjoGdXWMB6Gutzszdh0X9QAAAAAAkxDIAAAAAMAkBDJIklxdXTV06FC5urqa3ZUchbraHzV1DOpqf9TUMair/VFTx6CujpET68pFPQAAAADAJJwhAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAExCIHuAde7cWRaLxXrz8fFR/fr1tWvXrgy3iY6OlsViUWRkZIZtNm/erIYNG8rb21t58uRRWFiYPvzwQ129ejVN23Xr1qlhw4by8fFR3rx5VaZMGb388ss6fvy4PYZoivup683bPPXUU9qxY4e1Ta1atWzaXL/17NnT2ubm5Z6enqpcubK+/fZbh47XDJ07d9YzzzyT4fqba5UnTx6VKVNGEydOtK6fOXNmurXMkyePzX1cX547d24VL15cr776qi5evOjIoWWJe5mj1+3Zs0etWrVSwYIF5erqquDgYL311ls6f/68TbvAwEDr/vPmzauwsDBNnTo1zf4Mw9CUKVNUvXp1eXp6yt3dXWXLllW/fv108OBBu43ZDHcyT/v375/h+vj4ePXv318BAQFycXFR0aJF1aVLFx09ejRN29jYWPXt21clSpSQq6ur/P391bhxY61Zs8YOIzHfnT7mx4wZk2bd008/LYvFomHDhtm0v7n2hw8fVrt27VS0aFHlyZNHxYoVU9OmTfXnn3/a7CsnPmfdr2PHjqlLly4qWrSoXFxcFBAQoH79+ikuLs7srpnmdvNVki5cuKChQ4cqODhYrq6uKlCggFq2bKk9e/bYtBs2bJj1WJorVy75+/urR48eio+PT7PPHTt2qHXr1ipSpIhcXV0VEBCgRo0aaenSpXrQrsF3P6+lcuXKleYxGRMTI2dnZ1ksFkVHR9u0v/k17eLFi1WtWjV5eXnJw8NDZcuWTXOcvnTpkt59912Fh4crb968KlCggGrUqKEZM2bo8uXL9irBXSGQPeDq16+vmJgYxcTEaM2aNXJ2dlajRo3ueX+LFy9WzZo1VaxYMa1bt05//vmn+vXrpxEjRqhNmzY2B4TJkyerbt268vX11ddff629e/dq0qRJSkxM1AcffGCP4ZnmXuu6evVqxcTE6IcfflBycrIaNGigM2fOWNd3797dut/rt3fffddmHzNmzFBMTIx+++031ahRQy1atNDu3bvtPcRs73qt9u7dq1atWqlPnz6aN2+edb2np2eaWh45csRmH9f/jocOHdK4ceM0efJkDR06NKuH4hD3Mke3bNmiqlWr6tKlS1q+fLn279+vkSNHaubMmapXr54uXbpk0/6dd95RTEyM/vjjD3Xo0EHdu3fX999/b11vGIbatWunF198UQ0bNtSPP/6ovXv3atq0acqTJ49GjBjhkLE/COLj41WtWjWtXr1akyZN0sGDBzV//nwdPHhQlStX1qFDh6xto6OjVbFiRa1du1bvvfeedu/erZUrV6p27drq06ePiaPIWv7+/po5c6bNsuPHj2vNmjUqUqRIhttdvnxZ9erVU2Jior755hvt27dPCxYsUFhYmM3xNyc/Z92rQ4cOqVKlSjpw4IDmzZungwcPatKkSVqzZo2qV6+ebmiAlJKSorp162r69OkaMWKE9u/frxUrVujKlSuqWrWqtmzZYtO+bNmyiomJ0dGjRzVjxgytXLlSvXr1smnz7bffqlq1akpOTtYXX3yhqKgorVy5Us2aNdOQIUOUmJiYlUO0i3t9LeXn56dZs2bZLPviiy/k5+eX6XZr1qxR69at1bx5c23dulXbt2/XyJEjbULWpUuXFBERoTFjxqhHjx7avHmztm7dqj59+uiTTz5JE6izjIEHVqdOnYymTZvaLNu0aZMhyTh16lS62xw+fNiQZOzYsSPNuuTkZMPHx8d49tln06z77rvvDEnG/PnzDcMwjGPHjhkuLi5G//79072fhISEuxpLdmKvuv7888+GJGPlypWGYRhGzZo1jX79+mV635KMxYsXW39PSkoyJBkff/zxvQwl20qvxjdLr1alSpUy2rRpYxiGYcyYMcPw8vK66/t49tlnjUceeeQeepy93MscTU1NNcqUKWNUqlTJuHr1qs26yMhIw2KxGGPGjLEuCwgIMMaNG2fTLn/+/MaAAQOsv8+bN8+QZHz77bcZ3ueD7F7m6XU9e/Y08uXLZ8TExNgsP3/+vOHn52fUr1/fuqxBgwaGn5+fkZycnGY/D/Kx9GZ3UstevXoZPj4+xk8//WRdPnLkSKNx48ZGeHi4MXToUJv212u/Y8cOQ5IRHR2d4f5z8nPW/ahfv75RrFgx4/z58zbLY2JijLx58xo9e/Y0qWfmut18HTNmjGGxWIzIyEib5VevXjUqVapklClTxnr8Gzp0qBEeHm7T7qWXXjK8vb2tv19//dWsWbMM7/NBO57ez2upIUOGGKVKlbJZFxwcbLz55puGJOPw4cM27a+/9urXr59Rq1atTPs1duxYw8nJyfj999/TrLt06VK6x+GswBmyHCQ5OVmzZ89WUFCQfHx87nr7H3/8UXFxcRo4cGCadY0bN1ZwcLD1DMXChQt16dIlvfrqq+nu66GHHrrr+8+u7rWubm5ukpTmrMOdunLliqZNmyZJcnFxuad95CRubm73XEtJ+uOPP7R58+YcWcs7maORkZHau3evXnrpJTk52R76w8PDVbduXZszkDdLTU3V119/rYSEBJv6zZs3T6VLl1aTJk3S3c5isdzjiB5sqampmj9/vtq3by9fX1+bdW5uburdu7d++OEHxcfHKz4+XitXrlSfPn2UL1++NPvKScfS23FxcVH79u01Y8YM67KZM2eqS5cumW5XsGBBOTk5adGiRel+tF76bz1n3an4+Hj98MMP6t27t/X56jpfX1+1b99eCxYseOA+KpcV5s6dq3r16ik8PNxmuZOTkwYMGKC9e/dq586d6W4bHR2tH374weZYev31V0bzU3rwj6d381qqSZMmSkhI0E8//SRJ+umnn5SQkKDGjRtnup2vr6/27NmjP/74I8M2c+bMUd26dfXII4+kWZc7d+50j8NZgUD2gFu2bJnc3d3l7u4uDw8Pfffdd1qwYEGaF1x3Yv/+/ZKk0NDQdNeHhIRY2xw4cECenp6ZfozkQXa/dT1z5oyGDx8ud3d3ValSxbp84sSJ1v1ev82ZM8dm27Zt28rd3V2urq4aMGCAAgMD1apVK7uO70Fy9epVzZ49W7t27VKdOnWsyxMTE9PUskGDBjbbXv87Xv8u5KlTp/TKK69k9RAc4m7n6O0e36GhodY21w0aNMg6F1u0aCFvb29169bNZp+lS5e22aZ///7WfhUrVux+hvjAOn36tM6cOZNprQ3D0MGDB3Xw4EEZhqGQkJAs7mX21KVLF3311Vc6d+6cNm7cqMTExNt+xMnPz0/jx4/XW2+9JW9vb9WpU0fDhw+3+VhoTn/OuhcHDhyQYRiZztOEhASdPn06i3uW/e3fvz/Tul1vc93u3bvl7u4uNzc3FS9eXHv27NGgQYNs9ifJ5ni6bds2m+e3ZcuWOWIoDnWvr6Vy586tDh06aPr06ZKk6dOnq0OHDsqdO3em2/Xt21eVK1dWWFiYAgMD1aZNG02fPl0pKSnWNgcOHMiWx1sC2QOudu3aioyMVGRkpLZu3aqIiAg1aNBAR44cUYMGDawPhLJly97xPu/k3TDDMB74d2syc691feyxx+Tu7i5vb2/t3LlTCxYsUOHCha3r27dvb93v9dutZxfGjRunyMhIff/99ypTpoymTp2q/PnzZ8m4s9qcOXNsnnA2bdpkXXc9vLq5ual79+4aMGCAzWfuPTw80tTy1otOXP87/vrrr+rUqZP+7//+T82bN8+y8TnSvc7Ru3m3+5VXXlFkZKTWrl2rqlWraty4cQoKCsp0m8GDBysyMlJvvfWWkpOT72ls2U1m8zQzd3os/S+5XS3Dw8NVqlQpLVq0SNOnT9dzzz0nZ2fn2+63T58+io2N1Zw5c1S9enUtXLhQZcuW1apVqyTl/Oes+/Ffm4N3I7P5ejd1K126tCIjI7Vt2zYNGjRIERER6tu3b6bblC9f3nqMP3funK5cuXLP4zDL/bxG7dKlixYuXKjY2FgtXLjwtmfKJSlfvnxavny5Dh48qCFDhsjd3V0vv/yyqlSpYr1wVXad77c/yiFby5cvn80LpKlTp8rLy0tTpkzR1KlTdeHCBUm67bsKkhQcHCxJioqK0mOPPZZmfVRUlMqUKWNtm5iYqJiYmBz5juO91nXBggUqU6aMfHx80v0IjJeX121f0Pr6+iooKEhBQUGaMWOGGjZsqL1796pQoUL3P7BspkmTJqpatar195u/sNu+fXsNHjxYbm5uKlKkSJp31JycnG5by5v/jtOnT1d4eLimTZumrl272nEU5rjbOXrz4zu9j2pERUVZ21xXoEAB61xcuHChwsLCVKlSJetxoFSpUtq3b5/NNgULFlTBggVz1HzNbJ6mp2DBgnrooYcUFRWV7vqoqChZLBbr389isaS5GmBOdSe17NKliyZMmKC9e/dq69atd7xvDw8PNW7cWI0bN9aIESMUERGhESNGqF69ejn+OeteBAUFyWKxKCoqSs2aNUuzPioqSt7e3ipYsKAJvcseMpqvwcHBmT6+r7e5zsXFxfp4HzNmjJ5++mm9/fbbGj58uKRrx1JJ2rdvn6pVqyZJcnV1ve1zXHZ3P69Rw8LCFBISorZt2yo0NFTlypXL9ArhNytZsqRKliypbt26afDgwQoODtaCBQv0f//3fwoODs6Wx1vOkOUwFotFTk5OunDhgvz8/KwvpgICAm677VNPPaX8+fOne7Wp7777TgcOHFDbtm0lSS1atJCLi0uaKwRed/OVrXKCO62rv7+/SpYsabfvI1SpUkUVK1bUyJEj7bK/7MbDw8Nay6CgIJvvMVwPr35+fvf0EdxbOTk56Y033tCQIUOsTwI5ye3maIUKFRQSEqJx48YpNTXVZtudO3dq9erV1sd3evz9/dW6dWu9/vrr1mVt27bVvn37cuS/ZrhZZvM0PU5OTmrVqpXmzp2r2NhYm3UXLlzQxIkTFRERofz58yt//vyKiIjQhAkTdO7cuTT7ymnH0jupZbt27bR7926VK1fOGv7vlsViUUhIiLWm/7XnrDvh4+OjevXqaeLEiWmOidfPNrZu3fo/fWYxo/napk0brV69Os33xFJTUzVu3DiVKVMmzffLbjZkyBC9//77OnHihKQbr7/Gjh3ruMFkA3f7GrVLly5av379HZ0dy0hgYKDy5s1rPRa0a9dOq1evtvm3RNddvnw53eNwViCQPeBSUlIUGxur2NhYRUVFqW/fvkpOTr7tFx/37duX5uNeLi4umjx5sr799lv16NFDu3btUnR0tKZNm6bOnTurRYsW1u8y+fv7a9y4cfr444/VtWtXbdiwQUeOHNHPP/+s559/3vquz4PqXut6O+fPn7fu9/otISEh02369++vyZMn/6f/T056DMNIU8vY2Ng0YeNmLVu2VK5cuTRhwoQs7Klj3O0ctVgsmjZtmvbu3Wu9JPDRo0e1cOFCNW7cWNWrV8/0f2pJUr9+/bR06VL99ttvkq69KGnRooXatGmjd955R7/++quio6O1YcMGLViwQLly5bL3sLOd06dPpzmWnjx5UqNGjZKvr6/q1aun77//XseOHdPGjRsVERGhy5cv28zBCRMm6OrVq6pSpYq+/vprHThwQFFRURo/fryqV69u4ujM4e3tbb1M9p2IjIxU06ZNtWjRIu3du1cHDx7UtGnTNH36dDVt2lRSzn/OuleffvqpUlJSFBERoY0bN+rYsWNauXKl6tWrJz8/vxz7ZuD9GjBggKpUqaLGjRtr4cKFOnr0qLZt26bmzZsrKipK06ZNyzTIVq9eXeXLl9eoUaMkSe7u7po6daqWL1+up59+Wj/88IMOHTqkXbt2Wd9EeBCPp/f7Wqp79+46ffq0zXeXMzNs2DC9+uqrWr9+vQ4fPqwdO3aoS5cu1n+NIV17TVWjRg09+eSTmjBhgnbu3KlDhw7pq6++UrVq1XTgwIF7Hu99MePSjrCPTp06GZKsNw8PD6Ny5crGokWLMtzm+iVC07sdO3bMMAzD2LhxoxEREWF4enoaLi4uRtmyZY3333/fuHLlSpr9rVq1yoiIiDC8vb2NPHnyGCEhIcbAgQONEydOOGzcjnY/dU3v3wlcV7NmzXTrHhERYW2jWy57bxjXLnUbEhJi9OrV636Hlm3cz+XEDePaZe8zmsfXLzOe0X2MHj3aKFiwoGmXtrWHe5mj1+3atcto3ry5kT9/fiN37txGyZIljSFDhhjnzp2zaZfeZe8NwzAiIiKMBg0aWH+/evWqMWnSJKNq1apGvnz5DBcXF6NEiRJG9+7djb179973WM10J/M0vTk4fPhwwzAM4/Tp00bfvn0Nf39/I3fu3EbhwoWNzp07G0eOHEmzrxMnThh9+vQxAgICDBcXF8PPz89o0qSJsW7dOgeNLmvd72M+s8venz592njxxReNcuXKGe7u7oaHh4cRFhZmvP/++2n+xUNOfM66X9HR0UanTp2MwoULG7lz5zb8/f2Nvn37Gv/884/ZXTPN7earYRjGuXPnjMGDBxtBQUFG7ty5jfz58xvNmzc3du/ebdMuvcveG8a1fxvi6upqHD161Lps27ZtRosWLYxChQoZzs7Oho+PjxEREWHMnz//gbzsvb1fS13/FxcZXfZ+7dq1RvPmzQ1/f3/DxcXFKFy4sFG/fn1j06ZNNvu5ePGiMXr0aCMsLMzIkyePkT9/fqNGjRrGzJkzjcuXL9tj+HfNYhjZ9NttAAAAAJDD8ZFFAAAAADAJgQwAAAAATEIgAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAExCIAMAAAAAkxDIAAAAAMAkBDIAwH9a586d9cwzz1h/r1Wrlvr375/l/Vi/fr0sFovOnDmTYRuLxaIlS5bc8T6HDRumChUq3Fe/oqOjZbFYFBkZeV/7AQCkj0AGAMh2OnfuLIvFIovFIhcXFwUFBemdd97RlStXHH7f33zzjYYPH35Hbe8kRAEAkBlnszsAAEB66tevrxkzZiglJUUrVqxQnz59lDt3br3++utp2l66dEkuLi52ud/8+fPbZT8AANwJzpABALIlV1dX+fr6KiAgQL169VLdunX13XffSbrxMcORI0eqaNGiKl26tCTp2LFjatWqlR566CHlz59fTZs2VXR0tHWfV69e1UsvvaSHHnpIPj4+evXVV2UYhs393vqRxZSUFA0aNEj+/v5ydXVVUFCQpk2bpujoaNWuXVuS5O3tLYvFos6dO0uSUlNTNXr0aBUvXlxubm4KDw/XokWLbO5nxYoVCg4Olpubm2rXrm3Tzzs1aNAgBQcHK2/evCpRooTefPNNXb58OU27yZMny9/fX3nz5lWrVq2UmJhos37q1KkKDQ1Vnjx5FBISookTJ951XwAA94ZABgB4ILi5uenSpUvW39esWaN9+/Zp1apVWrZsmS5fvqyIiAh5eHho06ZN+vnnn+Xu7q769etbt/vggw80c+ZMTZ8+XT/99JPi4+O1ePHiTO+3Y8eOmjdvnsaPH6+oqChNnjxZ7u7u8vf319dffy1J2rdvn2JiYvTxxx9LkkaPHq1Zs2Zp0qRJ2rNnjwYMGKAOHTpow4YNkq4Fx2effVaNGzdWZGSkunXrptdee+2ua+Lh4aGZM2dq7969+vjjjzVlyhSNGzfOps3Bgwf11VdfaenSpVq5cqV27Nih3r17W9fPmTNHb731lkaOHKmoqCiNGjVKb775pr744ou77g8A4B4YAABkM506dTKaNm1qGIZhpKamGqtWrTJcXV2NgQMHWtcXLlzYSElJsW7z5ZdfGqVLlzZSU1Oty1JSUgw3Nzfjhx9+MAzDMIoUKWK8++671vWXL182ihUrZr0vwzCMmjVrGv369TMMwzD27dtnSDJWrVqVbj/XrVtnSDISEhKsyy5evGjkzZvX2Lx5s03brl27Gm3btjUMwzBef/11o0yZMjbrBw0alGZft5JkLF68OMP17733nlGxYkXr70OHDjVy5cpl/P3339Zl33//veHk5GTExMQYhmEYJUuWNObOnWuzn+HDhxvVq1c3DMMwDh8+bEgyduzYkeH9AgDuHd8hAwBkS8uWLZO7u7suX76s1NRUtWvXTsOGDbOuDwsLs/ne2M6dO3Xw4EF5eHjY7OfixYv666+/lJiYqJiYGFWtWtW6ztnZWZUqVUrzscXrIiMjlStXLtWsWfOO+33w4EGdP39e9erVs1l+6dIlPfLII5KkqKgom35IUvXq1e/4Pq5bsGCBxo8fr7/++kvJycm6cuWKPD09bdo8/PDD8vPzs7mf1NRU7du3Tx4eHvrrr7/UtWtXde/e3drmypUr8vLyuuv+AADuHoEMAJAt1a5dW5999plcXFxUtGhROTvbPmXly5fP5vfk5GRVrFhRc+bMSbOvggUL3lMf3Nzc7nqb5ORkSdLy5cttgpB07Xtx9vLLL7+offv2evvttxURESEvLy/Nnz9fH3zwwV33dcqUKWkCYq5cuezWVwBAxghkAIBsKV++fAoKCrrj9o8++qgWLFigQoUKpTlLdF2RIkX066+/6oknnpB07UzQ9u3b9eijj6bbPiwsTKmpqdqwYYPq1q2bZv31M3RXr161LitTpoxcXV119OjRDM+shYaGWi9Qct2WLVtuP8ibbN68WQEBARo8eLB12ZEjR9K0O3r0qE6cOKGiRYta78fJyUmlS5dW4cKFVbRoUR06dEjt27e/q/sHANgHF/UAAOQI7du3V4ECBdS0aVNt2rRJhw8f1vr16/Xiiy/q77//liT169dPY8aM0ZIlS/Tnn3+qd+/emf4PscDAQHXq1EldunTRkiVLrPv86quvJEkBAQGyWCxatmyZTp8+reTkZHl4eGjgwIEaMGCAvvjiC/3111/6/fff9cknn1gvlNGzZ08dOHBAr7zyivbt26e5c+dq5syZdzXeUqVK6ejRo5o/f77++usvjR8/Pt0LlOTJk0edOnXSzp07tWnTJr344otq1aqVfH19JUlvv/22Ro8erfHjx2v//v3avXu3ZsyYoQ8//PCu+gMAuDcEMgBAjpA3b15t3LhRDz/8sJ599lmFhoaqa9euunjxovWM2csvv6znnntOnTp1UvXq1eXh4aFmzZplut/PPvtMLVq0UO/evRUSEqLu3bvr3LlzkiQ/Pz+9/fbbeu2111S4cGG98MILkqThw4frzTff1OjRoxUaGqr69etr+fLlKl68uKRr3+v6+uuvtWTJEoWHh2vSpEkaNWrUXY23SZMmGjBggF544QVVqFBBmzdv1ptvvpmmXVBQkJ599lk1bNhQTz31lMqXL29zWftu3bpp6tSpmjFjhsLCwlSzZk3NnDnT2lcAgGNZjIy+yQwAAAAAcCjOkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACY5P8BcdRVkDQ04tkAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 1000x1000 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "eval_token_list = list(set(eval_df_tokens['labels']))\n",
    "\n",
    "plot_confusion_matrix(eval_df_tokens[\"labels\"], eval_df_tokens[\"predicted_label\"],\n",
    "                      eval_token_list)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define & Call Function to Display Example Token Sequences Along With Labels & Losses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "      <th>14</th>\n",
       "      <th>15</th>\n",
       "      <th>16</th>\n",
       "      <th>17</th>\n",
       "      <th>18</th>\n",
       "      <th>19</th>\n",
       "      <th>20</th>\n",
       "      <th>21</th>\n",
       "      <th>22</th>\n",
       "      <th>23</th>\n",
       "      <th>24</th>\n",
       "      <th>25</th>\n",
       "      <th>26</th>\n",
       "      <th>27</th>\n",
       "      <th>28</th>\n",
       "      <th>29</th>\n",
       "      <th>30</th>\n",
       "      <th>31</th>\n",
       "      <th>32</th>\n",
       "      <th>33</th>\n",
       "      <th>34</th>\n",
       "      <th>35</th>\n",
       "      <th>36</th>\n",
       "      <th>37</th>\n",
       "      <th>38</th>\n",
       "      <th>39</th>\n",
       "      <th>40</th>\n",
       "      <th>41</th>\n",
       "      <th>42</th>\n",
       "      <th>43</th>\n",
       "      <th>44</th>\n",
       "      <th>45</th>\n",
       "      <th>46</th>\n",
       "      <th>47</th>\n",
       "      <th>48</th>\n",
       "      <th>49</th>\n",
       "      <th>50</th>\n",
       "      <th>51</th>\n",
       "      <th>52</th>\n",
       "      <th>53</th>\n",
       "      <th>54</th>\n",
       "      <th>55</th>\n",
       "      <th>56</th>\n",
       "      <th>57</th>\n",
       "      <th>58</th>\n",
       "      <th>59</th>\n",
       "      <th>60</th>\n",
       "      <th>61</th>\n",
       "      <th>62</th>\n",
       "      <th>63</th>\n",
       "      <th>64</th>\n",
       "      <th>65</th>\n",
       "      <th>66</th>\n",
       "      <th>67</th>\n",
       "      <th>68</th>\n",
       "      <th>69</th>\n",
       "      <th>70</th>\n",
       "      <th>71</th>\n",
       "      <th>72</th>\n",
       "      <th>73</th>\n",
       "      <th>74</th>\n",
       "      <th>75</th>\n",
       "      <th>76</th>\n",
       "      <th>77</th>\n",
       "      <th>78</th>\n",
       "      <th>79</th>\n",
       "      <th>80</th>\n",
       "      <th>81</th>\n",
       "      <th>82</th>\n",
       "      <th>83</th>\n",
       "      <th>84</th>\n",
       "      <th>85</th>\n",
       "      <th>86</th>\n",
       "      <th>87</th>\n",
       "      <th>88</th>\n",
       "      <th>89</th>\n",
       "      <th>90</th>\n",
       "      <th>91</th>\n",
       "      <th>92</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>tokens</th>\n",
       "      <td>Ġasking</td>\n",
       "      <td>Ġcamera</td>\n",
       "      <td>Ġresponsible</td>\n",
       "      <td>Ġstops</td>\n",
       "      <td>Ġpure</td>\n",
       "      <td>ĠCoast</td>\n",
       "      <td>ĠStyles</td>\n",
       "      <td>ĠAug</td>\n",
       "      <td>Ġairline</td>\n",
       "      <td>Ġ44</td>\n",
       "      <td>Ġvideos</td>\n",
       "      <td>Ġball</td>\n",
       "      <td>acy</td>\n",
       "      <td>ĠEpisode</td>\n",
       "      <td>Ġball</td>\n",
       "      <td>ining</td>\n",
       "      <td>Ġhousing</td>\n",
       "      <td>Ġcurrency</td>\n",
       "      <td>X</td>\n",
       "      <td>ĠTab</td>\n",
       "      <td>ining</td>\n",
       "      <td>X</td>\n",
       "      <td>Ġworked</td>\n",
       "      <td>X</td>\n",
       "      <td>Ġpartnerships</td>\n",
       "      <td>Ġgrab</td>\n",
       "      <td>Ġdeploying</td>\n",
       "      <td>ĠSor</td>\n",
       "      <td>hm</td>\n",
       "      <td>chemical</td>\n",
       "      <td>X</td>\n",
       "      <td>Ġsoon</td>\n",
       "      <td>X</td>\n",
       "      <td>ĠThanks</td>\n",
       "      <td>Ġfinger</td>\n",
       "      <td>Ġclouds</td>\n",
       "      <td>Ġ110</td>\n",
       "      <td>Ġnewspapers</td>\n",
       "      <td>X</td>\n",
       "      <td>Ġsoon</td>\n",
       "      <td>ĠAme</td>\n",
       "      <td>Ġboast</td>\n",
       "      <td>Ġportfolio</td>\n",
       "      <td>Ġcomforting</td>\n",
       "      <td>Ġenergy</td>\n",
       "      <td>ĠSaudi</td>\n",
       "      <td>X</td>\n",
       "      <td>ĠReef</td>\n",
       "      <td>ĠRaleigh</td>\n",
       "      <td>ĠAston</td>\n",
       "      <td>ining</td>\n",
       "      <td>X</td>\n",
       "      <td>Ġworked</td>\n",
       "      <td>X</td>\n",
       "      <td>ĠLane</td>\n",
       "      <td>OC</td>\n",
       "      <td>ĠHeisman</td>\n",
       "      <td>Ġreality</td>\n",
       "      <td>X</td>\n",
       "      <td>Ġsoon</td>\n",
       "      <td>X</td>\n",
       "      <td>Ġunfortunately</td>\n",
       "      <td>ĠEddie</td>\n",
       "      <td>Ġbeautiful</td>\n",
       "      <td>Ġav</td>\n",
       "      <td>Ġfacto</td>\n",
       "      <td>ize</td>\n",
       "      <td>X</td>\n",
       "      <td>Ġsoon</td>\n",
       "      <td>Ġsatisfy</td>\n",
       "      <td>Ġparents</td>\n",
       "      <td>zes</td>\n",
       "      <td>ĠFox</td>\n",
       "      <td>Ġcommunications</td>\n",
       "      <td>ĠPlease</td>\n",
       "      <td>Ġpharm</td>\n",
       "      <td>Ġbarn</td>\n",
       "      <td>ĠExecutive</td>\n",
       "      <td>Ġretire</td>\n",
       "      <td>Ġsoon</td>\n",
       "      <td>Ġfiscal</td>\n",
       "      <td>asp</td>\n",
       "      <td>Ġintimacy</td>\n",
       "      <td>ĠArtists</td>\n",
       "      <td>ifi</td>\n",
       "      <td>Ġeventually</td>\n",
       "      <td>Ġkid</td>\n",
       "      <td>Ġcompetitor</td>\n",
       "      <td>Ġmonarch</td>\n",
       "      <td>Ġproperty</td>\n",
       "      <td>Ġenergy</td>\n",
       "      <td>ĠTV</td>\n",
       "      <td>a</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>labels</th>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>preds</th>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-ORG</td>\n",
       "      <td>I-ORG</td>\n",
       "      <td>I-ORG</td>\n",
       "      <td>I-ORG</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>losses</th>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>3.85</td>\n",
       "      <td>1.82</td>\n",
       "      <td>3.26</td>\n",
       "      <td>3.26</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>7.01</td>\n",
       "      <td>8.16</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.09</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.59</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.74</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.36</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.55</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.17</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>6.40</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>6.41</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.93</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.61</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.25</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.23</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.18</td>\n",
       "      <td>0.90</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.65</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.44</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.77</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.99</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.14</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             0        1             2       3      4       5        6     7    \n",
       "tokens  Ġasking  Ġcamera  Ġresponsible  Ġstops  Ġpure  ĠCoast  ĠStyles  ĠAug  \\\n",
       "labels        O        O         B-LOC       O      O       O        O     O   \n",
       "preds         O        O         B-ORG   I-ORG  I-ORG   I-ORG        O     O   \n",
       "losses     0.00     0.00          3.85    1.82   3.26    3.26     0.00  0.00   \n",
       "\n",
       "              8     9        10     11    12        13     14     15   \n",
       "tokens  Ġairline   Ġ44  Ġvideos  Ġball   acy  ĠEpisode  Ġball  ining  \\\n",
       "labels         O   IGN        O    IGN   IGN         O    IGN    IGN   \n",
       "preds          O     O        O      O     O         O      O      O   \n",
       "losses      0.00  0.00     0.00   0.00  0.00      0.00   0.00   0.00   \n",
       "\n",
       "              16         17    18      19      20    21       22    23   \n",
       "tokens  Ġhousing  Ġcurrency     X    ĠTab   ining     X  Ġworked     X  \\\n",
       "labels         O          O     O  I-MISC  I-MISC     O        O     O   \n",
       "preds          O          O     O       O       O     O        O     O   \n",
       "losses      0.00       0.00  0.00    7.01    8.16  0.00     0.00  0.00   \n",
       "\n",
       "                   24     25          26    27    28        29    30     31   \n",
       "tokens  Ġpartnerships  Ġgrab  Ġdeploying  ĠSor    hm  chemical     X  Ġsoon  \\\n",
       "labels              O    IGN         IGN     O   IGN       IGN     O      O   \n",
       "preds          I-MISC      O           O     O     O    I-MISC     O      O   \n",
       "losses           1.09   0.00        0.00  0.59  0.00      0.00  0.00   0.00   \n",
       "\n",
       "          32       33       34       35    36           37    38     39   \n",
       "tokens     X  ĠThanks  Ġfinger  Ġclouds  Ġ110  Ġnewspapers     X  Ġsoon  \\\n",
       "labels     O   I-MISC      IGN        O   IGN          IGN     O      O   \n",
       "preds      O   I-MISC   I-MISC        O     O       I-MISC     O      O   \n",
       "losses  0.00     0.74     0.00     0.36  0.00         0.00  0.00   0.00   \n",
       "\n",
       "            40      41          42           43       44      45    46   \n",
       "tokens    ĠAme  Ġboast  Ġportfolio  Ġcomforting  Ġenergy  ĠSaudi     X  \\\n",
       "labels  I-MISC     IGN         IGN            O        O       O     O   \n",
       "preds        O       O           O            O        O       O     O   \n",
       "losses    1.55    0.00        0.00         0.17     0.00    0.00  0.00   \n",
       "\n",
       "            47        48      49      50    51       52    53      54      55   \n",
       "tokens   ĠReef  ĠRaleigh  ĠAston   ining     X  Ġworked     X   ĠLane      OC  \\\n",
       "labels  I-MISC       IGN     IGN  I-MISC     O        O     O       O     IGN   \n",
       "preds        O         O       O       O     O        O     O  I-MISC  I-MISC   \n",
       "losses    6.40      0.00    0.00    6.41  0.00     0.00  0.00    1.93    0.00   \n",
       "\n",
       "              56        57    58     59    60              61      62   \n",
       "tokens  ĠHeisman  Ġreality     X  Ġsoon     X  Ġunfortunately  ĠEddie  \\\n",
       "labels         O       IGN     O      O     O               O     IGN   \n",
       "preds          O         O     O      O     O          I-MISC       O   \n",
       "losses      0.61      0.00  0.00   0.00  0.00            1.25    0.00   \n",
       "\n",
       "                63    64      65    66    67     68        69        70   \n",
       "tokens  Ġbeautiful   Ġav  Ġfacto   ize     X  Ġsoon  Ġsatisfy  Ġparents  \\\n",
       "labels         IGN     O     IGN   IGN     O      O    I-MISC         O   \n",
       "preds            O     O       O     O     O      O    I-MISC    I-MISC   \n",
       "losses        0.00  0.23    0.00  0.00  0.00   0.00      0.18      0.90   \n",
       "\n",
       "            71    72               73       74      75     76          77   \n",
       "tokens     zes  ĠFox  Ġcommunications  ĠPlease  Ġpharm  Ġbarn  ĠExecutive  \\\n",
       "labels     IGN   IGN                O      IGN     IGN    IGN         IGN   \n",
       "preds   I-MISC     O                O        O       O      O           O   \n",
       "losses    0.00  0.00             0.65     0.00    0.00   0.00        0.00   \n",
       "\n",
       "             78     79       80    81         82        83    84           85   \n",
       "tokens  Ġretire  Ġsoon  Ġfiscal   asp  Ġintimacy  ĠArtists   ifi  Ġeventually  \\\n",
       "labels        O      O   I-MISC   IGN        IGN       IGN   IGN          IGN   \n",
       "preds    I-MISC      O        O     O          O         O     O            O   \n",
       "losses     1.44   0.00     1.77  0.00       0.00      0.00  0.00         0.00   \n",
       "\n",
       "            86           87        88         89       90    91    92  \n",
       "tokens    Ġkid  Ġcompetitor  Ġmonarch  Ġproperty  Ġenergy   ĠTV     a  \n",
       "labels  I-MISC          IGN       IGN     I-MISC        O     O   IGN  \n",
       "preds        O            O         O     I-MISC        O     O     O  \n",
       "losses    1.99         0.00      0.00       0.14     0.00  0.00  0.00  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "      <th>14</th>\n",
       "      <th>15</th>\n",
       "      <th>16</th>\n",
       "      <th>17</th>\n",
       "      <th>18</th>\n",
       "      <th>19</th>\n",
       "      <th>20</th>\n",
       "      <th>21</th>\n",
       "      <th>22</th>\n",
       "      <th>23</th>\n",
       "      <th>24</th>\n",
       "      <th>25</th>\n",
       "      <th>26</th>\n",
       "      <th>27</th>\n",
       "      <th>28</th>\n",
       "      <th>29</th>\n",
       "      <th>30</th>\n",
       "      <th>31</th>\n",
       "      <th>32</th>\n",
       "      <th>33</th>\n",
       "      <th>34</th>\n",
       "      <th>35</th>\n",
       "      <th>36</th>\n",
       "      <th>37</th>\n",
       "      <th>38</th>\n",
       "      <th>39</th>\n",
       "      <th>40</th>\n",
       "      <th>41</th>\n",
       "      <th>42</th>\n",
       "      <th>43</th>\n",
       "      <th>44</th>\n",
       "      <th>45</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>tokens</th>\n",
       "      <td>Ġasking</td>\n",
       "      <td>ĠElectricity</td>\n",
       "      <td>Ġgrass</td>\n",
       "      <td>ĠDeputy</td>\n",
       "      <td>Ġsimple</td>\n",
       "      <td>Ġasking</td>\n",
       "      <td>Ġprovider</td>\n",
       "      <td>Ġstatements</td>\n",
       "      <td>Ġasking</td>\n",
       "      <td>ĠRd</td>\n",
       "      <td>Ġsoon</td>\n",
       "      <td>Ġfifth</td>\n",
       "      <td>Ġasking</td>\n",
       "      <td>Ġprincipal</td>\n",
       "      <td>ĠNokia</td>\n",
       "      <td>ĠSun</td>\n",
       "      <td>Ġnormal</td>\n",
       "      <td>ĠAb</td>\n",
       "      <td>Ġdegrees</td>\n",
       "      <td>Ġtheft</td>\n",
       "      <td>Ġofficers</td>\n",
       "      <td>Ġasking</td>\n",
       "      <td>cycle</td>\n",
       "      <td>Ġsoon</td>\n",
       "      <td>ĠRod</td>\n",
       "      <td>Ġsoon</td>\n",
       "      <td>ĠNative</td>\n",
       "      <td>Ġsoon</td>\n",
       "      <td>ved</td>\n",
       "      <td>Ġsoon</td>\n",
       "      <td>ju</td>\n",
       "      <td>Ġsoon</td>\n",
       "      <td>EC</td>\n",
       "      <td>Ġsoon</td>\n",
       "      <td>Ġforeigners</td>\n",
       "      <td>Ġsoon</td>\n",
       "      <td>cz</td>\n",
       "      <td>Ġsoon</td>\n",
       "      <td>channel</td>\n",
       "      <td>Ġsoon</td>\n",
       "      <td>form</td>\n",
       "      <td>Ġsoon</td>\n",
       "      <td>Ġfifth</td>\n",
       "      <td>Ġraid</td>\n",
       "      <td>ĠTV</td>\n",
       "      <td>a</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>labels</th>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-PER</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>B-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>B-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>B-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>B-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>B-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>B-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>B-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>B-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>B-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-PER</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>preds</th>\n",
       "      <td>O</td>\n",
       "      <td>B-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>losses</th>\n",
       "      <td>0.00</td>\n",
       "      <td>3.98</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.03</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.00</td>\n",
       "      <td>4.26</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.69</td>\n",
       "      <td>0.00</td>\n",
       "      <td>2.29</td>\n",
       "      <td>0.00</td>\n",
       "      <td>2.86</td>\n",
       "      <td>0.00</td>\n",
       "      <td>3.74</td>\n",
       "      <td>0.00</td>\n",
       "      <td>4.46</td>\n",
       "      <td>0.00</td>\n",
       "      <td>4.38</td>\n",
       "      <td>0.00</td>\n",
       "      <td>4.61</td>\n",
       "      <td>0.00</td>\n",
       "      <td>3.62</td>\n",
       "      <td>0.00</td>\n",
       "      <td>4.03</td>\n",
       "      <td>0.00</td>\n",
       "      <td>5.16</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>6.21</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             0             1       2        3        4        5          6    \n",
       "tokens  Ġasking  ĠElectricity  Ġgrass  ĠDeputy  Ġsimple  Ġasking  Ġprovider  \\\n",
       "labels        O             O       O        O        O        O          O   \n",
       "preds         O        B-MISC       O        O        O        O          O   \n",
       "losses     0.00          3.98    0.00     0.00     0.00     0.00       0.03   \n",
       "\n",
       "                 7        8      9      10      11       12          13   \n",
       "tokens  Ġstatements  Ġasking    ĠRd  Ġsoon  Ġfifth  Ġasking  Ġprincipal  \\\n",
       "labels            O        O  B-PER      O       O        O           O   \n",
       "preds             O        O      O      O       O        O           O   \n",
       "losses         0.01     0.00   4.26   0.00    0.00     0.00        0.00   \n",
       "\n",
       "            14    15       16    17        18      19         20       21   \n",
       "tokens  ĠNokia  ĠSun  Ġnormal   ĠAb  Ġdegrees  Ġtheft  Ġofficers  Ġasking  \\\n",
       "labels       O     O        O     O         O       O          O        O   \n",
       "preds        O     O        O     O         O       O          O        O   \n",
       "losses    0.00  0.00     0.00  0.00      0.00    0.00       0.00     0.00   \n",
       "\n",
       "            22     23      24     25       26     27      28     29      30   \n",
       "tokens   cycle  Ġsoon    ĠRod  Ġsoon  ĠNative  Ġsoon     ved  Ġsoon      ju  \\\n",
       "labels  B-MISC      O  B-MISC      O   B-MISC      O  B-MISC      O  B-MISC   \n",
       "preds        O      O       O      O        O      O       O      O       O   \n",
       "losses    1.69   0.00    2.29   0.00     2.86   0.00    3.74   0.00    4.46   \n",
       "\n",
       "           31      32     33           34     35      36     37       38   \n",
       "tokens  Ġsoon      EC  Ġsoon  Ġforeigners  Ġsoon      cz  Ġsoon  channel  \\\n",
       "labels      O  B-MISC      O       B-MISC      O  B-MISC      O   B-MISC   \n",
       "preds       O       O      O            O      O       O      O        O   \n",
       "losses   0.00    4.38   0.00         4.61   0.00    3.62   0.00     4.03   \n",
       "\n",
       "           39      40     41      42     43    44    45  \n",
       "tokens  Ġsoon    form  Ġsoon  Ġfifth  Ġraid   ĠTV     a  \n",
       "labels      O  B-MISC      O       O  B-PER     O   IGN  \n",
       "preds       O       O      O       O      O     O     O  \n",
       "losses   0.00    5.16   0.00    0.00   6.21  0.00  0.00  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "      <th>14</th>\n",
       "      <th>15</th>\n",
       "      <th>16</th>\n",
       "      <th>17</th>\n",
       "      <th>18</th>\n",
       "      <th>19</th>\n",
       "      <th>20</th>\n",
       "      <th>21</th>\n",
       "      <th>22</th>\n",
       "      <th>23</th>\n",
       "      <th>24</th>\n",
       "      <th>25</th>\n",
       "      <th>26</th>\n",
       "      <th>27</th>\n",
       "      <th>28</th>\n",
       "      <th>29</th>\n",
       "      <th>30</th>\n",
       "      <th>31</th>\n",
       "      <th>32</th>\n",
       "      <th>33</th>\n",
       "      <th>34</th>\n",
       "      <th>35</th>\n",
       "      <th>36</th>\n",
       "      <th>37</th>\n",
       "      <th>38</th>\n",
       "      <th>39</th>\n",
       "      <th>40</th>\n",
       "      <th>41</th>\n",
       "      <th>42</th>\n",
       "      <th>43</th>\n",
       "      <th>44</th>\n",
       "      <th>45</th>\n",
       "      <th>46</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>tokens</th>\n",
       "      <td>Ġnature</td>\n",
       "      <td>Ġasking</td>\n",
       "      <td>sts</td>\n",
       "      <td>Ġban</td>\n",
       "      <td>Ġfaith</td>\n",
       "      <td>Ġbelieves</td>\n",
       "      <td>Ġcharacter</td>\n",
       "      <td>Ġconfined</td>\n",
       "      <td>Ġsoon</td>\n",
       "      <td>Ġhousing</td>\n",
       "      <td>ha</td>\n",
       "      <td>Ġasking</td>\n",
       "      <td>form</td>\n",
       "      <td>Ġcandy</td>\n",
       "      <td>Ġbarbecue</td>\n",
       "      <td>Ġparticipants</td>\n",
       "      <td>Ġpharm</td>\n",
       "      <td>ples</td>\n",
       "      <td>ĠSaudi</td>\n",
       "      <td>Ġasking</td>\n",
       "      <td>Ġbarbecue</td>\n",
       "      <td>Ġparticipants</td>\n",
       "      <td>Ġpharm</td>\n",
       "      <td>ples</td>\n",
       "      <td>Ġstatements</td>\n",
       "      <td>ĠCre</td>\n",
       "      <td>Ġranked</td>\n",
       "      <td>burg</td>\n",
       "      <td>ĠEn</td>\n",
       "      <td>Ġworked</td>\n",
       "      <td>Ġasking</td>\n",
       "      <td>ĠChurch</td>\n",
       "      <td>Ġstatements</td>\n",
       "      <td>Ġfacility</td>\n",
       "      <td>Å¡</td>\n",
       "      <td>ĠMall</td>\n",
       "      <td>pressure</td>\n",
       "      <td>Ġran</td>\n",
       "      <td>Ġfootball</td>\n",
       "      <td>Ġdiabetes</td>\n",
       "      <td>Ġcontains</td>\n",
       "      <td>Ġstatements</td>\n",
       "      <td>une</td>\n",
       "      <td>Ġconducted</td>\n",
       "      <td>Ġenergy</td>\n",
       "      <td>ĠTV</td>\n",
       "      <td>a</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>labels</th>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-MISC</td>\n",
       "      <td>IGN</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-MISC</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>IGN</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>preds</th>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-PER</td>\n",
       "      <td>I-PER</td>\n",
       "      <td>I-PER</td>\n",
       "      <td>I-PER</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>losses</th>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>3.09</td>\n",
       "      <td>0.00</td>\n",
       "      <td>6.24</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>6.50</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>8.36</td>\n",
       "      <td>9.87</td>\n",
       "      <td>0.00</td>\n",
       "      <td>10.45</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.29</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             0        1     2     3       4          5           6    \n",
       "tokens  Ġnature  Ġasking   sts  Ġban  Ġfaith  Ġbelieves  Ġcharacter  \\\n",
       "labels        O        O     O     O       O          O           O   \n",
       "preds         O        O     O     O       O          O           O   \n",
       "losses     0.00     0.00  0.00  0.00    0.00       0.00        0.00   \n",
       "\n",
       "               7      8         9     10       11      12      13         14   \n",
       "tokens  Ġconfined  Ġsoon  Ġhousing    ha  Ġasking    form  Ġcandy  Ġbarbecue  \\\n",
       "labels          O      O         O     O        O  B-MISC     IGN     I-MISC   \n",
       "preds           O      O         O     O        O       O       O          O   \n",
       "losses       0.00   0.00      0.00  0.00     0.00    3.09    0.00       6.24   \n",
       "\n",
       "                   15      16    17      18       19         20   \n",
       "tokens  Ġparticipants  Ġpharm  ples  ĠSaudi  Ġasking  Ġbarbecue  \\\n",
       "labels            IGN     IGN   IGN       O        O     B-MISC   \n",
       "preds               O       O     O       O        O          O   \n",
       "losses           0.00    0.00  0.00    0.00     0.00       6.50   \n",
       "\n",
       "                   21      22    23           24      25       26      27   \n",
       "tokens  Ġparticipants  Ġpharm  ples  Ġstatements    ĠCre  Ġranked    burg  \\\n",
       "labels            IGN     IGN   IGN       I-MISC  I-MISC      IGN  I-MISC   \n",
       "preds               O       O     O            O   B-PER    I-PER   I-PER   \n",
       "losses           0.00    0.00  0.00         8.36    9.87     0.00   10.45   \n",
       "\n",
       "           28       29       30       31           32         33    34     35   \n",
       "tokens    ĠEn  Ġworked  Ġasking  ĠChurch  Ġstatements  Ġfacility    Å¡  ĠMall  \\\n",
       "labels    IGN        O        O        O            O          O     O    IGN   \n",
       "preds   I-PER        O        O        O            O          O     O      O   \n",
       "losses   0.00     0.00     0.00     0.00         0.00       0.00  0.00   0.00   \n",
       "\n",
       "              36    37         38         39         40           41      42   \n",
       "tokens  pressure  Ġran  Ġfootball  Ġdiabetes  Ġcontains  Ġstatements     une  \\\n",
       "labels       IGN   IGN          O          O          O            O       O   \n",
       "preds          O     O          O          O          O            O  B-MISC   \n",
       "losses      0.00  0.00       0.00       0.00       0.00         0.00    1.29   \n",
       "\n",
       "                43       44    45    46  \n",
       "tokens  Ġconducted  Ġenergy   ĠTV     a  \n",
       "labels         IGN        O     O   IGN  \n",
       "preds            O        O     O     O  \n",
       "losses        0.00     0.00  0.00  0.00  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def get_samples(df):\n",
    "    for _, row in df.iterrows():\n",
    "        labels, preds, tokens, losses = [], [], [], []\n",
    "        for i, mask in enumerate(row[\"attention_mask\"]):\n",
    "            if i not in {0, len(row[\"attention_mask\"])}:\n",
    "                labels.append(row[\"labels\"][i])\n",
    "                preds.append(row[\"predicted_label\"][i])\n",
    "                tokens.append(row[\"input_tokens\"][i])\n",
    "                losses.append(f\"{row['loss'][i]:.2f}\")\n",
    "        eval_df_tmp = pd.DataFrame({\"tokens\": tokens, \"labels\": labels, \n",
    "                               \"preds\": preds, \"losses\": losses}).T\n",
    "        yield eval_df_tmp\n",
    "\n",
    "eval_df[\"total_loss\"] = eval_df[\"loss\"].apply(sum)\n",
    "eval_df_tmp = eval_df.sort_values(by=\"total_loss\", ascending=False).head(3)\n",
    "\n",
    "pd.set_option('display.max_columns', None)\n",
    "\n",
    "for sample in get_samples(eval_df_tmp):\n",
    "    display(sample)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Notes & Other Takeaways From This Project\n",
    "\n",
    "****\n",
    "- This model is part of a comparison of Token Classification Models using this particular dataset.\n",
    "\n",
    "****"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Citations\n",
    "\n",
    "- Model Checkpoint\n",
    "\n",
    "    > @article{ma2022mega, title={Mega: Moving Average Equipped Gated Attention}, author={Ma, Xuezhe and Zhou, Chunting and Kong, Xiang and He, Junxian and Gui, Liangke and Neubig, Graham and May, Jonathan and Zettlemoyer Luke}, journal={arXiv preprint arXiv:2209.10655}, year={2022}}\n",
    "\n",
    "- Dataset\n",
    "\n",
    "    > @inproceedings{tedeschi-etal-2021-wikineural-combined, title = \"{W}iki{NE}u{R}al: {C}ombined Neural and Knowledge-based Silver Data Creation for Multilingual {NER}\", author = \"Tedeschi, Simone and Maiorca, Valentino and Campolungo, Niccol{\\`o} and Cecconi, Francesco and Navigli, Roberto\", booktitle = \"Findings of the Association for Computational Linguistics: EMNLP 2021\", month = nov, year = \"2021\", address = \"Punta Cana, Dominican Republic\", publisher = \"Association for Computational Linguistics\", url = \"https://aclanthology.org/2021.findings-emnlp.215\", pages = \"2521--2533\", abstract = \"Multilingual Named Entity Recognition (NER) is a key intermediate task which is needed in many areas of NLP. In this paper, we address the well-known issue of data scarcity in NER, especially relevant when moving to a multilingual scenario, and go beyond current approaches to the creation of multilingual silver data for the task. We exploit the texts of Wikipedia and introduce a new methodology based on the effective combination of knowledge-based approaches and neural models, together with a novel domain adaptation technique, to produce high-quality training corpora for NER. We evaluate our datasets extensively on standard benchmarks for NER, yielding substantial improvements up to 6 span-based F1-score points over previous state-of-the-art systems for data creation.\",}\n",
    "\n",
    "- Metric (PosEval)\n",
    "\n",
    "    > @article{scikit-learn, title={Scikit-learn: Machine Learning in {P}ython}, author={Pedregosa, F. and Varoquaux, G. and Gramfort, A. and Michel, V. and Thirion, B. and Grisel, O. and Blondel, M. and Prettenhofer, P. and Weiss, R. and Dubourg, V. and Vanderplas, J. and Passos, A. and Cournapeau, D. and Brucher, M. and Perrot, M. and Duchesnay, E.}, journal={Journal of Machine Learning Research}, volume={12}, pages={2825--2830}, year={2011}}"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "dl",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}