Token Classification/Monolingual/WikiNeural - Transformer Comparison/POS Project with Wikineural Dataset - BERT-Base Transformer.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Monolingual Named Entity Recognition Token Classification (WikiNeural Dataset)\n",
    "\n",
    "### BERT Base Cased Transformer\n",
    "\n",
    "Dataset Source: https://huggingface.co/datasets/Babelscape/wikineural"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Import Necessary Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Git LFS initialized.\n"
     ]
    }
   ],
   "source": [
    "import os, sys\n",
    "os.environ['TOKENIZERS_PARALLELISM']='false'\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import sklearn\n",
    "from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay\n",
    "\n",
    "import datasets\n",
    "from datasets import load_dataset, DatasetDict\n",
    "\n",
    "import torch\n",
    "from torch import nn\n",
    "from torch.nn.functional import cross_entropy\n",
    "\n",
    "import transformers\n",
    "from transformers import AutoTokenizer, AutoModelForTokenClassification\n",
    "from transformers import DataCollatorForTokenClassification\n",
    "from transformers import Trainer, TrainingArguments\n",
    "\n",
    "import evaluate\n",
    "\n",
    "!git lfs install"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Display Library Versions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "          Python : 3.9.12\n",
      "           NumPy : 1.24.3\n",
      "           Torch : 2.0.0\n",
      "          Pandas : 2.0.1\n",
      "    Transformers : 4.28.1\n",
      "        datasets : 2.11.0\n",
      "         Sklearn : 1.2.2\n"
     ]
    }
   ],
   "source": [
    "print(\"Python :\".rjust(18), sys.version[0:6])\n",
    "print(\"NumPy :\".rjust(18), np.__version__)\n",
    "print(\"Torch :\".rjust(18), torch.__version__)\n",
    "print(\"Pandas :\".rjust(18), pd.__version__)\n",
    "print(\"Transformers :\".rjust(18), transformers.__version__)\n",
    "print(\"datasets :\".rjust(18), datasets.__version__)\n",
    "print(\"Sklearn :\".rjust(18), sklearn.__version__)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Ingest English Subset of Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Found cached dataset parquet (/Users/briandunn/.cache/huggingface/datasets/Babelscape___parquet/Babelscape--wikineural-579d1dc98d2a6b93/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "36ae69ce4bc140bab1fe4e07ce05ffb0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/27 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    train_es: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 76320\n",
       "    })\n",
       "    test_es: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 9618\n",
       "    })\n",
       "    test_de: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 12372\n",
       "    })\n",
       "    val_it: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 11050\n",
       "    })\n",
       "    train_en: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 92720\n",
       "    })\n",
       "    train_it: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 88400\n",
       "    })\n",
       "    train_nl: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 83680\n",
       "    })\n",
       "    train_pl: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 108160\n",
       "    })\n",
       "    test_pl: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 13585\n",
       "    })\n",
       "    val_ru: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 11540\n",
       "    })\n",
       "    test_nl: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 10547\n",
       "    })\n",
       "    train_fr: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 100800\n",
       "    })\n",
       "    val_de: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 12330\n",
       "    })\n",
       "    test_it: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 11069\n",
       "    })\n",
       "    test_fr: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 12678\n",
       "    })\n",
       "    val_en: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 11590\n",
       "    })\n",
       "    val_nl: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 10460\n",
       "    })\n",
       "    test_pt: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 10160\n",
       "    })\n",
       "    val_fr: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 12600\n",
       "    })\n",
       "    test_en: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 11597\n",
       "    })\n",
       "    train_pt: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 80560\n",
       "    })\n",
       "    test_ru: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 11580\n",
       "    })\n",
       "    val_pl: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 13520\n",
       "    })\n",
       "    val_pt: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 10070\n",
       "    })\n",
       "    train_de: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 98640\n",
       "    })\n",
       "    val_es: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 9540\n",
       "    })\n",
       "    train_ru: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 92320\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = load_dataset(\"Babelscape/wikineural\")\n",
    "\n",
    "data = data.remove_columns([\"lang\"])\n",
    "\n",
    "data"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Split Dataset into DatasetDict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training data shape: (92720, 2)\n",
      "Testing data shape: (11597, 2)\n",
      "Validation data shape: (11590, 2)\n"
     ]
    }
   ],
   "source": [
    "ds = DatasetDict({\n",
    "    'train': data['train_en'], \n",
    "    'test': data['test_en'], \n",
    "    'eval': data['val_en']})\n",
    "\n",
    "print('Training data shape:', ds['train'].shape)\n",
    "print('Testing data shape:', ds['test'].shape)\n",
    "print('Validation data shape:', ds['eval'].shape)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'tokens': ['It',\n",
       "  'has',\n",
       "  'a',\n",
       "  'leaf',\n",
       "  'pattern',\n",
       "  'similar',\n",
       "  'to',\n",
       "  'the',\n",
       "  'members',\n",
       "  'of',\n",
       "  'the',\n",
       "  'genera',\n",
       "  '\"',\n",
       "  'Kedrostis',\n",
       "  '\"',\n",
       "  ',',\n",
       "  '\"',\n",
       "  'Melothria',\n",
       "  '\"',\n",
       "  'and',\n",
       "  '\"',\n",
       "  'Zehneria',\n",
       "  '\"',\n",
       "  '.'],\n",
       " 'ner_tags': [0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  7,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  7,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  7,\n",
       "  0,\n",
       "  0]}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example = ds['train'][12]\n",
    "\n",
    "example"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Display Feature Information About Each Feature"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tokens: \n",
      "Sequence(feature=Value(dtype='string', id=None), length=-1, id=None)\n",
      "\n",
      "ner_tags: \n",
      "Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for k, v in ds[\"train\"].features.items():\n",
    "    print(f\"{k}: \\n{v}\\n\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define Tag Values & Conversions Between String & Integer Values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "List of tag values: \n",
      "['O', 'B-PER', 'I-PER', 'B-ORG', 'I-ORG', 'B-LOC', 'I-LOC', 'B-MISC', 'I-MISC']\n",
      "Number of NER Tags: \n",
      "9\n",
      "id2label: \n",
      "{0: 'O', 1: 'B-PER', 2: 'I-PER', 3: 'B-ORG', 4: 'I-ORG', 5: 'B-LOC', 6: 'I-LOC', 7: 'B-MISC', 8: 'I-MISC'}\n",
      "label2id: \n",
      "{'O': 0, 'B-PER': 1, 'I-PER': 2, 'B-ORG': 3, 'I-ORG': 4, 'B-LOC': 5, 'I-LOC': 6, 'B-MISC': 7, 'I-MISC': 8}\n"
     ]
    }
   ],
   "source": [
    "label2id = {\n",
    "    'O': 0, \n",
    "    'B-PER': 1, \n",
    "    'I-PER': 2, \n",
    "    'B-ORG': 3, \n",
    "    'I-ORG': 4, \n",
    "    'B-LOC': 5, \n",
    "    'I-LOC': 6, \n",
    "    'B-MISC': 7, \n",
    "    'I-MISC': 8\n",
    "    }\n",
    "\n",
    "id2label = {tag: idx for idx, tag in label2id.items()}\n",
    "\n",
    "pos_tag_values = list(label2id.keys())\n",
    "NUM_OF_LABELS = len(pos_tag_values)\n",
    "\n",
    "print(f\"List of tag values: \\n{pos_tag_values}\")\n",
    "print(f\"Number of NER Tags: \\n{NUM_OF_LABELS}\")\n",
    "print(f\"id2label: \\n{id2label}\")\n",
    "print(f\"label2id: \\n{label2id}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Basic Values/Constants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "MODEL_CKPT = \"bert-base-cased\"\n",
    "MODEL_NAME = f\"{MODEL_CKPT.split(f'/')[-1]}-finetuned-WikiNeural\"\n",
    "\n",
    "NUM_OF_EPOCHS = 2\n",
    "BATCH_SIZE = 16\n",
    "\n",
    "STRATEGY = \"epoch\"\n",
    "REPORTS_TO = \"tensorboard\"\n",
    "\n",
    "WEIGHT_DECAY = 0.01\n",
    "LR = 2e-5\n",
    "\n",
    "DEVICE = torch.device(\"cpu\")\n",
    "STEPS = 1250"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Function to Tokenize & Align Inputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = AutoTokenizer.from_pretrained(MODEL_CKPT)\n",
    "\n",
    "def tokenize_and_align_labels(samples):\n",
    "    tokenized_inputs = tokenizer(samples[\"tokens\"], \n",
    "                                      truncation=True, \n",
    "                                      is_split_into_words=True)\n",
    "    \n",
    "    labels = []\n",
    "    \n",
    "    for idx, label in enumerate(samples[f\"ner_tags\"]):\n",
    "        word_ids = tokenized_inputs.word_ids(batch_index=idx)\n",
    "        prev_word_idx = None\n",
    "        label_ids = []\n",
    "        for word_idx in word_ids: # set special tokens to -100\n",
    "            if word_idx is None or word_idx == prev_word_idx:\n",
    "                label_ids.append(-100)\n",
    "            else:\n",
    "                label_ids.append(label[word_idx])\n",
    "            prev_word_idx = word_idx\n",
    "        labels.append(label_ids)\n",
    "    \n",
    "    tokenized_inputs[\"labels\"] = labels\n",
    "    return tokenized_inputs"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Apply Above Function to Tokenize Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading cached processed dataset at /Users/briandunn/.cache/huggingface/datasets/Babelscape___parquet/Babelscape--wikineural-579d1dc98d2a6b93/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec/cache-5017e8265a954f19.arrow\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "74cb884732c84147ad2d06f2f282971b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/11597 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading cached processed dataset at /Users/briandunn/.cache/huggingface/datasets/Babelscape___parquet/Babelscape--wikineural-579d1dc98d2a6b93/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec/cache-2df3e82508a9f790.arrow\n"
     ]
    }
   ],
   "source": [
    "encoded_ds = ds.map(tokenize_and_align_labels, \n",
    "                    batched=True, \n",
    "                    remove_columns=\n",
    "                        [\n",
    "                            'ner_tags', \n",
    "                            'tokens'\n",
    "                        ]\n",
    "                    )"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define Data Collator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_collator = DataCollatorForTokenClassification(tokenizer=tokenizer)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load Custom Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at bert-base-cased were not used when initializing BertForTokenClassification: ['cls.seq_relationship.bias', 'cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.seq_relationship.weight']\n",
      "- This IS expected if you are initializing BertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of BertForTokenClassification were not initialized from the model checkpoint at bert-base-cased and are newly initialized: ['classifier.bias', 'classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "model = (AutoModelForTokenClassification.from_pretrained(\n",
    "    MODEL_CKPT,\n",
    "    num_labels=NUM_OF_LABELS,\n",
    "    id2label=id2label,\n",
    "    label2id=label2id\n",
    "    ).to(DEVICE))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create Compute Metrics Function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "label_list = pos_tag_values\n",
    "\n",
    "seqeval = evaluate.load(\"seqeval\")\n",
    "\n",
    "labels = [label_list[i] for i in example[f'ner_tags']]\n",
    "\n",
    "def compute_metrics(eval_preds):\n",
    "    predictions, labels = eval_preds\n",
    "    predictions = np.argmax(predictions, \n",
    "                            axis=2)\n",
    "    \n",
    "    true_predictions = [\n",
    "        [label_list[p] for (p, l) in zip(prediction, label) if l != -100]\n",
    "        for prediction, label in zip(predictions, labels)\n",
    "    ]\n",
    "    \n",
    "    true_labels = [\n",
    "        [label_list[l] for (p, l) in zip(prediction, label) if l != -100]\n",
    "        for prediction, label in zip(predictions, labels)\n",
    "    ]\n",
    "    \n",
    "    results = seqeval.compute(predictions=true_predictions, \n",
    "                              references=true_labels)\n",
    "    \n",
    "    return results"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define TrainingArguments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "args = TrainingArguments(\n",
    "    MODEL_NAME,\n",
    "    log_level=\"error\",\n",
    "    logging_first_step=True,\n",
    "    learning_rate=LR,\n",
    "    num_train_epochs=NUM_OF_EPOCHS,\n",
    "    per_device_train_batch_size=BATCH_SIZE,\n",
    "    per_device_eval_batch_size=BATCH_SIZE,\n",
    "    evaluation_strategy=STRATEGY,\n",
    "    report_to=REPORTS_TO,\n",
    "    disable_tqdm=False,\n",
    "    logging_steps=STEPS,\n",
    "    weight_decay=WEIGHT_DECAY,\n",
    "    save_strategy=STRATEGY,\n",
    "    hub_private_repo=True,\n",
    "    push_to_hub=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Subclass Trainer to Handle Class Imbalance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CustomTrainer(Trainer):\n",
    "    def compute_loss(self, \n",
    "                     model, \n",
    "                     inputs, \n",
    "                     return_outputs=False):\n",
    "        \n",
    "        labels = inputs.get(\"labels\")\n",
    "        # forward pass\n",
    "        outputs = model(**inputs)\n",
    "        logits = outputs.get(\"logits\")\n",
    "        # compute custom loss \n",
    "        loss_fct = nn.CrossEntropyLoss(weight=torch.tensor(\n",
    "            [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0], \n",
    "            device=model.device)\n",
    "        )\n",
    "        loss = loss_fct(logits.view(-1, \n",
    "                                    self.model.config.num_labels), \n",
    "                        labels.view(-1)\n",
    "                        )\n",
    "        return (loss, outputs) if return_outputs else loss"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define Trainer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Cloning https://huggingface.co/DunnBC22/bert-base-cased-finetuned-WikiNeural into local empty directory.\n"
     ]
    }
   ],
   "source": [
    "trainer = CustomTrainer(model, \n",
    "                  args=args,\n",
    "                  data_collator=data_collator,\n",
    "                  compute_metrics=compute_metrics,\n",
    "                  tokenizer=tokenizer,\n",
    "                  train_dataset=encoded_ds[\"train\"],\n",
    "                  eval_dataset=encoded_ds[\"test\"]\n",
    "                  )"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Train Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/briandunn/Documents/deep_learning/dl/lib/python3.9/site-packages/transformers/optimization.py:391: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "da21ba80eb574622a206557338cde7f6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/11590 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 2.1548, 'learning_rate': 1.9998274374460744e-05, 'epoch': 0.0}\n",
      "{'loss': 0.2062, 'learning_rate': 1.7842968075927526e-05, 'epoch': 0.22}\n",
      "{'loss': 0.1157, 'learning_rate': 1.5685936151855047e-05, 'epoch': 0.43}\n",
      "{'loss': 0.1036, 'learning_rate': 1.3528904227782573e-05, 'epoch': 0.65}\n",
      "{'loss': 0.1, 'learning_rate': 1.1371872303710096e-05, 'epoch': 0.86}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3e56b7fff05747749914e927b9d3c591",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/725 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.09429672360420227, 'eval_LOC': {'precision': 0.9075480846937126, 'recall': 0.9429051217464316, 'f1': 0.9248888156811068, 'number': 5955}, 'eval_MISC': {'precision': 0.8320190720704199, 'recall': 0.8964631495751828, 'f1': 0.8630397565151225, 'number': 5061}, 'eval_ORG': {'precision': 0.9151428571428571, 'recall': 0.9286749782545666, 'f1': 0.9218592603252267, 'number': 3449}, 'eval_PER': {'precision': 0.9683036587751908, 'recall': 0.9499040307101727, 'f1': 0.9590155992636372, 'number': 5210}, 'eval_overall_precision': 0.9038565996740902, 'eval_overall_recall': 0.9303176620076239, 'eval_overall_f1': 0.9168962580774432, 'eval_overall_accuracy': 0.9901143530909022, 'eval_runtime': 867.3781, 'eval_samples_per_second': 13.37, 'eval_steps_per_second': 0.836, 'epoch': 1.0}\n",
      "{'loss': 0.0827, 'learning_rate': 9.214840379637619e-06, 'epoch': 1.08}\n",
      "{'loss': 0.061, 'learning_rate': 7.057808455565142e-06, 'epoch': 1.29}\n",
      "{'loss': 0.0587, 'learning_rate': 4.900776531492667e-06, 'epoch': 1.51}\n",
      "{'loss': 0.0595, 'learning_rate': 2.74374460742019e-06, 'epoch': 1.73}\n",
      "{'loss': 0.0578, 'learning_rate': 5.867126833477136e-07, 'epoch': 1.94}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "66998632fd644f05acaea95cdf81d26d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/725 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.0880606546998024, 'eval_LOC': {'precision': 0.9282034236330398, 'recall': 0.9378673383711167, 'f1': 0.9330103575008353, 'number': 5955}, 'eval_MISC': {'precision': 0.8336608897623727, 'recall': 0.9219521833629718, 'f1': 0.8755864139613436, 'number': 5061}, 'eval_ORG': {'precision': 0.9351851851851852, 'recall': 0.9370832125253696, 'f1': 0.9361332367849385, 'number': 3449}, 'eval_PER': {'precision': 0.9728037566034045, 'recall': 0.9543186180422265, 'f1': 0.9634725317314214, 'number': 5210}, 'eval_overall_precision': 0.914474010207621, 'eval_overall_recall': 0.9379923761118171, 'eval_overall_f1': 0.9260839020473706, 'eval_overall_accuracy': 0.9911586906477513, 'eval_runtime': 865.1849, 'eval_samples_per_second': 13.404, 'eval_steps_per_second': 0.838, 'epoch': 2.0}\n",
      "{'train_runtime': 46397.543, 'train_samples_per_second': 3.997, 'train_steps_per_second': 0.25, 'train_loss': 0.09300354592649972, 'epoch': 2.0}\n"
     ]
    }
   ],
   "source": [
    "train_results = trainer.train()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Push Model To Hub"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Several commits (2) will be pushed upstream.\n",
      "The progress bars may be unreliable.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7ee58b705a49498e9535bd78619fce4f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Upload file pytorch_model.bin:   0%|          | 1.00/411M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "92186d3282884d32b8720efd89fa525e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Upload file runs/Jul09_20-29-49_Brians-Mac-mini/events.out.tfevents.1688952594.Brians-Mac-mini.7897.0:   0%|  …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "To https://huggingface.co/DunnBC22/bert-base-cased-finetuned-WikiNeural\n",
      "   321e00a..c81e524  main -> main\n",
      "\n",
      "To https://huggingface.co/DunnBC22/bert-base-cased-finetuned-WikiNeural\n",
      "   c81e524..3aaf5cc  main -> main\n",
      "\n"
     ]
    }
   ],
   "source": [
    "kwargs = {\n",
    "    \"tasks\": \"token-classification\",\n",
    "    \"tags\": ['token-classification'],\n",
    "}\n",
    "\n",
    "if args.push_to_hub:\n",
    "    trainer.push_to_hub('All DUNN!!!', **kwargs)\n",
    "else:\n",
    "    trainer.create_model_card(**kwargs)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Save & Log Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "To https://huggingface.co/DunnBC22/bert-base-cased-finetuned-WikiNeural\n",
      "   3aaf5cc..5252ebe  main -> main\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "***** train metrics *****\n",
      "  epoch                    =         2.0\n",
      "  train_loss               =       0.093\n",
      "  train_runtime            = 12:53:17.54\n",
      "  train_samples_per_second =       3.997\n",
      "  train_steps_per_second   =        0.25\n"
     ]
    }
   ],
   "source": [
    "trainer.save_model()\n",
    "trainer.log_metrics(\"train\", train_results.metrics)\n",
    "trainer.save_metrics(\"train\", train_results.metrics)\n",
    "trainer.save_state()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Error Analysis"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define Method to Apply to Validation Dataset (& Then Apply it)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "def forward_pass_with_label(batch):\n",
    "    # Convert dict of lists to list of dicts suitable for data collator\n",
    "    features = [dict(zip(batch, t)) for t in zip(*batch.values())]\n",
    "    # Pad inputs and labels and put all tensors on device\n",
    "    batch = data_collator(features)\n",
    "    input_ids = batch[\"input_ids\"].to(DEVICE)\n",
    "    attention_mask = batch[\"attention_mask\"].to(DEVICE)\n",
    "    labels = batch[\"labels\"].to(DEVICE)\n",
    "    with torch.no_grad():\n",
    "        # Pass data through model  \n",
    "        output = trainer.model(input_ids, \n",
    "                               attention_mask\n",
    "                               )\n",
    "        # Logit.size: [batch_size, sequence_length, classes]\n",
    "        predicted_label = torch.argmax(output.logits, \n",
    "                                       axis=-1\n",
    "                                       ).cpu().numpy()\n",
    "        \n",
    "    # Calculate loss per token after flattening batch dimension with view\n",
    "    loss = cross_entropy(output.logits.view(-1, 9), \n",
    "                         labels.view(-1), \n",
    "                         reduction=\"none\")\n",
    "    # Unflatten batch dimension and convert to numpy array\n",
    "    loss = loss.view(len(input_ids), -1).cpu().numpy()\n",
    "\n",
    "    return {\"loss\":loss, \"predicted_label\": predicted_label}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Apply Above Function to Entire Evaluation Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b3d21b60ff5d4c628fe018675fb1c009",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/11590 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "eval_set = encoded_ds['eval']\n",
    "\n",
    "eval_set = eval_set.map(forward_pass_with_label,\n",
    "                        batched=True,\n",
    "                        batch_size=32)\n",
    "\n",
    "eval_df = eval_set.to_pandas()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Clean Up Padding Tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>input_ids</th>\n",
       "      <th>token_type_ids</th>\n",
       "      <th>attention_mask</th>\n",
       "      <th>labels</th>\n",
       "      <th>loss</th>\n",
       "      <th>predicted_label</th>\n",
       "      <th>input_tokens</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[101, 1430, 3143, 1106, 17037, 7501, 1358, 110...</td>\n",
       "      <td>[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...</td>\n",
       "      <td>[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...</td>\n",
       "      <td>[IGN, O, O, O, B-LOC, IGN, IGN, O, O, O, O, O,...</td>\n",
       "      <td>[0.0, 1.9430925e-05, 2.3483954e-05, 3.7192607e...</td>\n",
       "      <td>[O, O, O, O, B-LOC, I-LOC, I-LOC, O, O, O, O, ...</td>\n",
       "      <td>[[CLS], Her, visit, to, Tu, ##val, ##u, was, c...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           input_ids   \n",
       "0  [101, 1430, 3143, 1106, 17037, 7501, 1358, 110...  \\\n",
       "\n",
       "                                      token_type_ids   \n",
       "0  [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...  \\\n",
       "\n",
       "                                      attention_mask   \n",
       "0  [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...  \\\n",
       "\n",
       "                                              labels   \n",
       "0  [IGN, O, O, O, B-LOC, IGN, IGN, O, O, O, O, O,...  \\\n",
       "\n",
       "                                                loss   \n",
       "0  [0.0, 1.9430925e-05, 2.3483954e-05, 3.7192607e...  \\\n",
       "\n",
       "                                     predicted_label   \n",
       "0  [O, O, O, O, B-LOC, I-LOC, I-LOC, O, O, O, O, ...  \\\n",
       "\n",
       "                                        input_tokens  \n",
       "0  [[CLS], Her, visit, to, Tu, ##val, ##u, was, c...  "
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "id2label[-100] = \"IGN\"\n",
    "eval_df[\"input_tokens\"] = eval_df[\"input_ids\"].apply(\n",
    "    lambda x: tokenizer.convert_ids_to_tokens(x))\n",
    "eval_df[\"predicted_label\"] = eval_df[\"predicted_label\"].apply(\n",
    "    lambda x: [id2label[i] for i in x])\n",
    "eval_df[\"labels\"] = eval_df[\"labels\"].apply(\n",
    "    lambda x: [id2label[i] for i in x])\n",
    "eval_df['loss'] = eval_df.apply(\n",
    "    lambda x: x['loss'][:len(x['input_ids'])], axis=1)\n",
    "eval_df['predicted_label'] = eval_df.apply(\n",
    "    lambda x: x['predicted_label'][:len(x['input_ids'])], axis=1)\n",
    "eval_df.head(1)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Unwrap Each Token Within Sample Separately"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>input_ids</th>\n",
       "      <th>token_type_ids</th>\n",
       "      <th>attention_mask</th>\n",
       "      <th>labels</th>\n",
       "      <th>loss</th>\n",
       "      <th>predicted_label</th>\n",
       "      <th>input_tokens</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1430</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>O</td>\n",
       "      <td>0.0</td>\n",
       "      <td>O</td>\n",
       "      <td>Her</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3143</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>O</td>\n",
       "      <td>0.0</td>\n",
       "      <td>O</td>\n",
       "      <td>visit</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1106</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>O</td>\n",
       "      <td>0.0</td>\n",
       "      <td>O</td>\n",
       "      <td>to</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>17037</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>0.0</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>Tu</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1108</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>O</td>\n",
       "      <td>0.0</td>\n",
       "      <td>O</td>\n",
       "      <td>was</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2195</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>O</td>\n",
       "      <td>0.0</td>\n",
       "      <td>O</td>\n",
       "      <td>cut</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1603</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>O</td>\n",
       "      <td>0.0</td>\n",
       "      <td>O</td>\n",
       "      <td>short</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  input_ids token_type_ids attention_mask labels  loss predicted_label   \n",
       "0      1430              0              1      O   0.0               O  \\\n",
       "0      3143              0              1      O   0.0               O   \n",
       "0      1106              0              1      O   0.0               O   \n",
       "0     17037              0              1  B-LOC   0.0           B-LOC   \n",
       "0      1108              0              1      O   0.0               O   \n",
       "0      2195              0              1      O   0.0               O   \n",
       "0      1603              0              1      O   0.0               O   \n",
       "\n",
       "  input_tokens  \n",
       "0          Her  \n",
       "0        visit  \n",
       "0           to  \n",
       "0           Tu  \n",
       "0          was  \n",
       "0          cut  \n",
       "0        short  "
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "eval_df_tokens = eval_df.apply(pd.Series.explode)\n",
    "eval_df_tokens = eval_df_tokens.query(\"labels != 'IGN'\")\n",
    "eval_df_tokens[\"loss\"] = eval_df_tokens[\"loss\"].astype(float).round(2)\n",
    "eval_df_tokens.head(7)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### See Which Tokens Have Accumulated Most Loss in Evaluation Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>input_tokens</th>\n",
       "      <td>Batman</td>\n",
       "      <td>The</td>\n",
       "      <td>–</td>\n",
       "      <td>B</td>\n",
       "      <td>E</td>\n",
       "      <td>of</td>\n",
       "      <td>P</td>\n",
       "      <td>A</td>\n",
       "      <td>W</td>\n",
       "      <td>Mi</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>104</td>\n",
       "      <td>2487</td>\n",
       "      <td>486</td>\n",
       "      <td>242</td>\n",
       "      <td>202</td>\n",
       "      <td>7358</td>\n",
       "      <td>200</td>\n",
       "      <td>544</td>\n",
       "      <td>144</td>\n",
       "      <td>70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>2.543</td>\n",
       "      <td>0.05</td>\n",
       "      <td>0.228</td>\n",
       "      <td>0.437</td>\n",
       "      <td>0.452</td>\n",
       "      <td>0.012</td>\n",
       "      <td>0.437</td>\n",
       "      <td>0.15</td>\n",
       "      <td>0.48</td>\n",
       "      <td>0.907</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sum</th>\n",
       "      <td>264.47</td>\n",
       "      <td>124.88</td>\n",
       "      <td>110.98</td>\n",
       "      <td>105.66</td>\n",
       "      <td>91.37</td>\n",
       "      <td>87.44</td>\n",
       "      <td>87.31</td>\n",
       "      <td>81.53</td>\n",
       "      <td>69.17</td>\n",
       "      <td>63.51</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   0       1       2       3      4      5      6      7   \n",
       "input_tokens  Batman     The       –       B      E     of      P      A  \\\n",
       "count            104    2487     486     242    202   7358    200    544   \n",
       "mean           2.543    0.05   0.228   0.437  0.452  0.012  0.437   0.15   \n",
       "sum           264.47  124.88  110.98  105.66  91.37  87.44  87.31  81.53   \n",
       "\n",
       "                  8      9  \n",
       "input_tokens      W     Mi  \n",
       "count           144     70  \n",
       "mean           0.48  0.907  \n",
       "sum           69.17  63.51  "
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(\n",
    "    eval_df_tokens.groupby(\"input_tokens\")[[\"loss\"]]\n",
    "    .agg([\"count\", \"mean\", \"sum\"])\n",
    "    .droplevel(level=0, axis=1)  # Get rid of multi-level columns\n",
    "    .sort_values(by=\"sum\", ascending=False)\n",
    "    .reset_index()\n",
    "    .round(3)\n",
    "    .head(10)\n",
    "    .T\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### See Which Label IDs Have Most Loss in Evaluation Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>labels</th>\n",
       "      <td>I-MISC</td>\n",
       "      <td>B-ORG</td>\n",
       "      <td>B-MISC</td>\n",
       "      <td>B-PER</td>\n",
       "      <td>I-LOC</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>I-ORG</td>\n",
       "      <td>I-PER</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>4555</td>\n",
       "      <td>3512</td>\n",
       "      <td>4173</td>\n",
       "      <td>5593</td>\n",
       "      <td>2370</td>\n",
       "      <td>5904</td>\n",
       "      <td>2963</td>\n",
       "      <td>4107</td>\n",
       "      <td>234662</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.281</td>\n",
       "      <td>0.256</td>\n",
       "      <td>0.248</td>\n",
       "      <td>0.189</td>\n",
       "      <td>0.176</td>\n",
       "      <td>0.153</td>\n",
       "      <td>0.136</td>\n",
       "      <td>0.026</td>\n",
       "      <td>0.011</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sum</th>\n",
       "      <td>1278.88</td>\n",
       "      <td>900.28</td>\n",
       "      <td>1035.12</td>\n",
       "      <td>1055.01</td>\n",
       "      <td>417.27</td>\n",
       "      <td>903.81</td>\n",
       "      <td>402.82</td>\n",
       "      <td>107.87</td>\n",
       "      <td>2508.58</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              0       1        2        3       4       5       6       7   \n",
       "labels   I-MISC   B-ORG   B-MISC    B-PER   I-LOC   B-LOC   I-ORG   I-PER  \\\n",
       "count      4555    3512     4173     5593    2370    5904    2963    4107   \n",
       "mean      0.281   0.256    0.248    0.189   0.176   0.153   0.136   0.026   \n",
       "sum     1278.88  900.28  1035.12  1055.01  417.27  903.81  402.82  107.87   \n",
       "\n",
       "              8  \n",
       "labels        O  \n",
       "count    234662  \n",
       "mean      0.011  \n",
       "sum     2508.58  "
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(\n",
    "    eval_df_tokens.groupby(\"labels\")[[\"loss\"]] \n",
    "    .agg([\"count\", \"mean\", \"sum\"])\n",
    "    .droplevel(level=0, axis=1)\n",
    "    .sort_values(by=\"mean\", ascending=False)\n",
    "    .reset_index()\n",
    "    .round(3)\n",
    "    .fillna(0)\n",
    "    .T\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create Function to Display Confusion Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay\n",
    "\n",
    "def plot_confusion_matrix(y_preds, y_true, labels):\n",
    "    cm = confusion_matrix(y_true, y_preds, normalize=\"true\")\n",
    "    fig, ax = plt.subplots(figsize=(10, 10))\n",
    "    disp = ConfusionMatrixDisplay(confusion_matrix=cm, display_labels=labels)\n",
    "    disp.plot(cmap=\"Blues\", values_format=\".2f\", ax=ax, colorbar=False)\n",
    "    plt.title(\"Normalized Confusion Matrix\")\n",
    "    plt.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Display Confusion Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA2QAAANXCAYAAABaBpzLAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAADF+UlEQVR4nOzdd1yV5f/H8fdBBAdDQRBFEhEFFccvd6WgaWrO0twr03KkWTnTUtNcDbNyW27NUW6z3PrNcqQ4cmXuiYKAOFDk/P5ADx4E0zqHG/H1fDzOo7jPdd/nuj5e3Oe8zz0wmc1mswAAAAAA6c7B6A4AAAAAwNOKQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgCwq7CwMIWFhVl+PnHihEwmk6ZPn56u/Wjfvr38/f3T9TX/rVmzZik4OFhZs2ZVrly5bL79wYMHy2Qy2Xy7Tyqj5iQASAQyADDc9OnTZTKZlC1bNp09e/aB58PCwhQSEmJAz55uixcvVp06dZQnTx45OTkpf/78atq0qdavX2/X1z106JDat2+vwoULa8qUKZo8ebJdXy+9mUwmmUwmdezYMdXnBwwYYGlz+fLlx97+qlWrNHjw4P/YSwBIPwQyAMgg4uPjNXLkSKO7YXcFCxbUjRs31KZNG6O7kiqz2azXX39dr776qi5evKj33ntPEydOVLdu3XTs2DG9+OKL2rp1q91ef+PGjUpMTNTYsWPVvn17NW3a1OavMXDgQN24ccPm231U2bJl0w8//KBbt2498Ny8efOULVu2f73tVatWaciQIY+1TkafkwAyNwIZAGQQZcqU0ZQpU3Tu3Dm7vYbZbDb0g7gky9HALFmyGNqPtHz++eeaPn26evbsqT/++EMffPCBOnTooAEDBmjnzp2aOXOmHB0d7fb6ERERkmSXUxXvcXR0/E+h57+qXbu2YmNj9dNPP1kt37p1q44fP666deumSz8SEhJ069atDD8nAWRuBDIAyCA++OAD3blz55GOkiUkJGjo0KEqXLiwnJ2d5e/vrw8++EDx8fFW7fz9/VWvXj39/PPPKleunLJnz65JkyZp48aNMplMWrBggYYMGSJfX1+5urqqSZMmiomJUXx8vHr27Clvb2+5uLjo9ddff2Db06ZNU/Xq1eXt7S1nZ2cVL15cEyZM+Me+p7xe515fUnukvObrp59+UpUqVZQzZ065urqqbt26+vPPPx94jSVLligkJETZsmVTSEiIFi9e/I/9kqQbN25oxIgRCg4O1meffZbqdVZt2rRRhQoVLD8fO3ZMr732mjw8PJQjRw5VqlRJK1eutFrn/np/8sknKlCggLJly6YXX3xRR48etbTz9/fXoEGDJEleXl4ymUyW0+/u///7+fv7q3379pafb9++rSFDhqhIkSLKli2bPD099cILL2jNmjWWNqldQ/a4c+p///ufKlSooGzZsikgIEAzZ858eHHv4+vrq6pVq2ru3LlWy+fMmaOSJUumeoruli1b9Nprr+mZZ56Rs7Oz/Pz89O6771p9wdC+fXuNGzfOUq97Dyl53n322Wf68ssvLeM8cODAA3MyIiJCXl5eCgsLk9lstmz/6NGjypkzp5o1a/bIYwWAf2K/r/gAAI+lUKFCatu2raZMmaJ+/fopf/78abbt2LGjZsyYoSZNmuj999/Xtm3bNGLECB08ePCB8HH48GG1aNFCb731ljp16qSgoCDLcyNGjFD27NnVr18/HT16VF9//bWyZs0qBwcHXblyRYMHD9bvv/+u6dOnq1ChQvroo48s606YMEElSpRQgwYN5OjoqOXLl6tr165KTExUt27dHnncxYoV06xZs6yWRUdH67333pO3t7dl2axZs9SuXTvVqlVLo0aN0vXr1zVhwgS98MIL2r17tyW8/fLLL2rcuLGKFy+uESNGKDIyUq+//roKFCjwj3353//+p6ioKPXs2fORjpZcvHhRzz33nK5fv64ePXrI09NTM2bMUIMGDbRo0SK98sorVu1HjhwpBwcH9erVSzExMRo9erRatWqlbdu2SZK+/PJLzZw5U4sXL9aECRPk4uKiUqVK/WM/7jd48GCNGDFCHTt2VIUKFRQbG6udO3dq165dqlmzZprrPc6cOnr0qJo0aaI33nhD7dq103fffaf27durbNmyKlGixCP1s2XLlnrnnXcUFxcnFxcXJSQkaOHChXrvvfd08+bNB9ovXLhQ169fV5cuXeTp6ant27fr66+/1pkzZ7Rw4UJJ0ltvvaVz585pzZo1D8ype6ZNm6abN2/qzTfflLOzszw8PJSYmGjVxtvbWxMmTNBrr72mr7/+Wj169FBiYqLat28vV1dXjR8//pHGCACPxAwAMNS0adPMksw7duww//3332ZHR0dzjx49LM+HhoaaS5QoYfk5PDzcLMncsWNHq+306tXLLMm8fv16y7KCBQuaJZlXr15t1XbDhg1mSeaQkBDzrVu3LMtbtGhhNplM5jp16li1r1y5srlgwYJWy65fv/7AWGrVqmUOCAiwWhYaGmoODQ21/Hz8+HGzJPO0adNSrUdiYqK5Xr16ZhcXF/Off/5pNpvN5qtXr5pz5cpl7tSpk1XbCxcumN3d3a2WlylTxpwvXz5zdHS0Zdkvv/xilvTAGFIaO3asWZJ58eLFD213T8+ePc2SzFu2bLEsu3r1qrlQoUJmf39/8507d8xmc3K9ixUrZo6Pj3/g9fbt22dZNmjQILMk86VLl6xeS5J50KBBD/ShYMGC5nbt2ll+Ll26tLlu3boP7fe917jn38ypzZs3W5ZFRESYnZ2dze+///5DX/feOLp162aOiooyOzk5mWfNmmU2m83mlStXmk0mk/nEiROp1iC1+TZixAizyWQynzx50rKsW7du5tQ+3tybd25ubuaIiIhUn0s5J1u0aGHOkSOH+ciRI+ZPP/3ULMm8ZMmSfxwjADwOTlkEgAwkICBAbdq00eTJk3X+/PlU26xatUqS9N5771ktf//99yXpgdPlChUqpFq1aqW6rbZt2ypr1qyWnytWrCiz2awOHTpYtatYsaJOnz6thIQEy7Ls2bNb/j8mJkaXL19WaGiojh07ppiYmH8aapqGDh2qFStWaPr06SpevLgkac2aNYqOjlaLFi10+fJlyyNLliyqWLGiNmzYIEk6f/68wsPD1a5dO7m7u1u2WbNmTcu2HiY2NlaS5Orq+kh9XbVqlSpUqKAXXnjBsszFxUVvvvmmTpw4oQMHDli1f/311+Xk5GT5uUqVKpKSTnu0lVy5cunPP//UX3/99cjrPO6cKl68uKXvUtLplUFBQY81jty5c6t27dqaN2+eJGnu3Ll67rnnVLBgwVTb3z/frl27psuXL+u5556T2WzW7t27H/l1GzduLC8vr0dq+80338jd3V1NmjTRhx9+qDZt2qhhw4aP/FoA8CgIZACQwQwcOFAJCQlpXkt28uRJOTg4KDAw0Gq5j4+PcuXKpZMnT1otL1SoUJqv9cwzz1j9fC/E+Pn5PbA8MTHRKmj9+uuvqlGjhnLmzKlcuXLJy8tLH3zwgST960C2evVqDRkyRP3791fjxo0ty++Fi+rVq8vLy8vq8csvv1huhHFv7EWKFHlg2/efqpkWNzc3SdLVq1cfqb8nT55MdbvFihWz6s89KeudO3duSdKVK1ce6fUexccff6zo6GgVLVpUJUuWVO/evbV3796HrvO4cyrlOKSksTzuOFq2bKk1a9bo1KlTWrJkiVq2bJlm21OnTql9+/by8PCQi4uLvLy8FBoaKunx5tvDfh9S8vDw0FdffaW9e/fK3d1dX3311SOvCwCPimvIACCDCQgIUOvWrTV58mT169cvzXaP+od97z+ykFJa10mltdx89wYHf//9t1588UUFBwfriy++kJ+fn5ycnLRq1SqNGTPmgWtyHsXx48fVqlUr1axZU8OGDbN67t72Zs2aJR8fnwfWtdVdD4ODgyVJ+/btU6NGjWyyzfv9U13/jTt37lj9XLVqVf39999aunSpfvnlF02dOlVjxozRxIkT0/zbX/c86pyy1TgaNGggZ2dntWvXTvHx8Wne4v/OnTuqWbOmoqKi1LdvXwUHBytnzpw6e/as2rdv/1jz7WG/D6n5+eefJSWF5jNnztj17pcAnk4EMgDIgAYOHKjZs2dr1KhRDzxXsGBBJSYm6q+//rIciZGSbjARHR2d5ilftrR8+XLFx8dr2bJlVkdL7p06+Lhu3LihV199Vbly5dK8efPk4GB9AkfhwoUlJd1soUaNGmlu597YUztd7/Dhw//YjxdeeEG5c+fWvHnz9MEHH/zjjT0KFiyY6nYPHTpk1R9byJ07t6Kjo62W3bp1K9VTWz08PPT666/r9ddfV1xcnKpWrarBgwenGciMmlPZs2dXo0aNNHv2bMsf4U7Nvn37dOTIEc2YMUNt27a1LL//zpH3PGqofBSrV6/W1KlT1adPH82ZM0ft2rXTtm3b7PpnDwA8fThlEQAyoMKFC6t169aaNGmSLly4YPXcyy+/LCnpjnz3++KLLyQpXf6G072gcv8RkZiYGE2bNu1fba9z5846cuSIFi9ebDmN7361atWSm5ubhg8frtu3bz/w/KVLlyRJ+fLlU5kyZTRjxgyr09jWrFnzwPVcqcmRI4f69u2rgwcPqm/fvqke8Zk9e7a2b98uKenfYvv27frtt98sz1+7dk2TJ0+Wv7//I1239qgKFy6szZs3Wy2bPHnyA0fIIiMjrX52cXFRYGDgA7evv5+Rc6pXr14aNGiQPvzwwzTbpDbfzGazxo4d+0DbnDlzStID4fVxRUdHW+5UOXz4cE2dOlW7du3S8OHD/9N2ASAlvuIBgAxqwIABmjVrlg4fPmx1K/HSpUurXbt2mjx5sqKjoxUaGqrt27drxowZatSokapVq2b3vr300ktycnJS/fr19dZbbykuLk5TpkyRt7d3mjcjScvKlSs1c+ZMNW7cWHv37rW63snFxUWNGjWSm5ubJkyYoDZt2ujZZ59V8+bN5eXlpVOnTmnlypV6/vnn9c0330hKupV/3bp19cILL6hDhw6KiorS119/rRIlSiguLu4f+9O7d2/9+eef+vzzz7VhwwY1adJEPj4+unDhgpYsWaLt27dr69atkqR+/fpp3rx5qlOnjnr06CEPDw/NmDFDx48f1w8//PDAkb7/omPHjurcubMaN26smjVras+ePfr5558fOKpUvHhxhYWFqWzZsvLw8NDOnTu1aNEivf3222lu28g5Vbp0aZUuXfqhbYKDg1W4cGH16tVLZ8+elZubm3744YdUr1krW7asJKlHjx6qVauWsmTJoubNmz92v9555x1FRkZq7dq1ypIli2rXrq2OHTtq2LBhatiw4T/2GQAeFYEMADKowMBAtW7dWjNmzHjgualTpyogIEDTp0/X4sWL5ePjo/79+1v+qLC9BQUFadGiRRo4cKB69eolHx8fdenSRV5eXg/cofGf3Du69cMPP+iHH36weq5gwYKWa7latmyp/Pnza+TIkfr0008VHx8vX19fValSRa+//rplndq1a2vhwoUaOHCg+vfvr8KFC2vatGlaunSpNm7c+I/9cXBw0MyZM9WwYUNNnjxZn332mWJjY+Xl5aWqVatq9OjRqly5siQpb9682rp1q/r27auvv/5aN2/eVKlSpbR8+XKbH1Xq1KmTjh8/rm+//VarV69WlSpVtGbNGr344otW7Xr06KFly5bpl19+UXx8vAoWLKhhw4apd+/eD92+0XPqYbJmzarly5erR48eGjFihLJly6ZXXnlFb7/99gPB6NVXX1X37t31/fffa/bs2TKbzY8dyJYtW6aZM2fq888/t1xXKCUdMVyzZo3atWunHTt2WN2hFAD+LZP5v1xJDAAAAAD417iGDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACD8HfI7CgxMVHnzp2Tq6urTCaT0d0BAAAAkA7MZrOuXr2q/Pnzy8Hh4cfACGR2dO7cOfn5+RndDQAAAAAGOH36tAoUKPDQNgQyO3J1dZUkOZV7RyZHZ4N7k3mcWP6B0V0AHhkHx22PMw4AwLYS7iQa3YVM5+rVWAUXLmjJAw9DILOjex8aTI7OBDIbcnNzM7oLwCMjO9gegQwAbItAZj+P8p7FTT0AAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADCIo9EdgP10bFRB3Zs9L28PF+3/+6L6frVSuw6dTbWtYxYHvduqqlq8VEb5vFx19HSkBk/6Ret2HLW06duumvq1r2a13pFTl1Sx3dd2HUdGMnXhZn0zZ50iImNVooivRr7fRGVL+KfZfum63Ro+aYVOn49SgJ+XBnVrqJrPl7A8v3xDuKb/+Kv2HDqlK7HXtXFWX5UsWiAdRpKxUFfbm7pws76enVzTUb0eXtMla3drxKQVOnW3poPffrCm0378VXsOJtV00+ynr6aSNGXBJktdQ4r4alTv1/6hrrs0fOJKnTofmVTX7o300n11NZvNGjFppWYu2aqYuBuqWCpAn/drpsLPeKfDaDIGamof1NU+qKvtfbtos8bNXq+IqFiVCPTViPeb6NkSBdNsv3Tdbo2cvNLyGeDDbg1U87mkmt5OuKMRE1do7W8HdPJspFxdsim0fJA+7NpAPl7u6TWkx8YRskzqlWohGtaltkbN2KiwNydq/98X9MPotsqTK2eq7Qe+8aLa1yunvl+vVKX232jash2aNbSFSgb6WLU7ePyigl4dbXnU6f5tegwnQ1i85g99OHaxer9RR+tn9FFIoK9ee2e8LkVdTbX99r3H1OnD6Wpdv7I2zOyrl6uWUps+U3Tw73OWNtdv3FKl0gEa9HbD9BpGhkNdbe/HNX9o4JeL1adjHW2Y2UchRXzVpEfaNd12t6atGlTWxll99XJoKbXuPUUHqKmVH39JqmvfjnW0cVZfhRTxVePu49Ku655j6jhwulo3rKxNs/upbmhpte41WQeOJtd17My1mjR/k77o31xrpvVSjuxOatx9nG7G306vYRmKmtoHdbUP6mp7i9fs0kdjF6tXx9paN6O3ShTxVdOeD/8M8NZHM9SqfmWtn9FHdaqWUrs+Uy2fAW7cvKW9h8/ovddrad2M3po+8g0dPRmh1r0np+ewHtsTF8jat2+vRo0apfl8WFiYTCaTTCaTsmXLpuLFi2v8+PGW56dPn255/v5HtmzZrF7j3vKsWbOqUKFC6tOnj27evGnPodlU19ee08yVf2ju6t06fPKS3vtiua7fvK3WdZ5NtX3TmqU1Zu5mrdn2l06ev6Lvlu3Qmm1H9HbT563aJdxJVMSVOMsjKvZ6egwnQxg/b4PaNKysVvUrKTggnz7v10zZszlpzvLfUm0/af5GvVipmLq3qaGgQj76oHM9lQry09SFmy1tmr1cQb071lFo+aD0GkaGQ11tb/zcDWrbKLmmX/RrphwPq+n3STXtcbemAzrXU6lgP01dYF3TPh3rKKzC01lTSRo/d73aNnpOrRpUTqpr/+bKkc1Js5c9pK6V76trl3oqHeynKQs3SUr6ZnzivA3q1aGWXg4tpZAivpowpK0uXI7Ryk170nNohqGm9kFd7YO62t7EeRvUuuFzalmvkoIK5dNnfZsqezYnzV3xe6rtJ8/fpOqViunt1i+qaCEf9X+rrkoFFdC3i7ZIktxcsmvR193UqMazCiyYV+VCCmlkrybac+i0zlyISs+hPZYnLpA9ik6dOun8+fM6cOCAmjZtqm7dumnevHmW593c3HT+/Hmrx8mTJ622Ubt2bZ0/f17Hjh3TmDFjNGnSJA0aNCi9h/KvZHXMojJF82njH39blpnNZm3a9bfKl0j9FCPnrI66eSvBatnN+ARVKvmM1bIAX08dWNhLu+f01OQBjVXAO+Me/rWlW7cTtOfQaYXe92HUwcFBoeWDtGPfiVTX2bHvxAOBoHqlYO3Yd9yeXX2iUFfbs9S0/GPWtAI1fZhbtxMUfui0VSB1cHBQaIWgNOu0fd9xhZUPtlpWvVIxy7/DybORuhgZq7AKyW3cXbKrbAl/7dh7wuZjyGioqX1QV/ugrrZ363aC9hx+8P2qavkg7Uyjpjv3n1DV8kWtllWrVCzN9pIUG3dTJpNJ7q7ZbdNxO8iUgSxHjhzy8fFRQECABg8erCJFimjZsmWW500mk3x8fKweefPmtdqGs7OzfHx85Ofnp0aNGqlGjRpas2ZNeg/lX/F0zyHHLFl06co1q+WXrlyTt4drquus33lUXV97TgG+HjKZTAorW1j1qhRT3vva/3HwjLqNWqzX+s7S+1+uUEGf3Fo19g25ZHey63gygsjoa7pzJ1HeHm5Wy709XBURFZvqOhGRsfJKUW8vD1dFRKZ+GP5pRF1t715NvVLU1MvDVRcj065pyn1D0r8BNb0nMjrubl1Tzj03RTykrl6eqc3VpPb3/j1StvH2dE1zm5kJNbUP6mof1NX2oizvVynGnzvt9/Sk96sU728PaX8z/rY+HrdUr9Z8Vq45M24geypu6pE9e3bdunXrX6+/f/9+bd26VQULpn2BoSTFx8crPj7e8nNs7JPzy9Tv61Ua26uhts/oIbPMOn72iuau3q1W953iuHb7X5b///PYRe08cEb7vn9PjaqFaPaqXUZ0GwAAAHjA7YQ76jhgmsxm6dO+TY3uzkNlyiNk99y5c0ezZ8/W3r17Vb16dcvymJgYubi4WD3q1Kljte6KFSvk4uKibNmyqWTJkoqIiFDv3r0f+nojRoyQu7u75eHn52eXcf2TyJjrSrhzR165rW/g4ZU7Z5rfeEfGXFfrD+fJt84wlWr+hSq0+0rXbtzSifNX0nyd2Gs3dfRMpALye9i0/xmRZ66cypLF4YGjNhFRVx/4puYeb0+3By5KvRR1Vd6eqR+lfBpRV9u7V9NLKWp6Keqq8nqmXdOU+4akfwNqeo9nLpe7dU0592Ll/ZC6XopMba4mtb/375GyTUTk1TS3mZlQU/ugrvZBXW3Pw/J+lWL8V9J+T096v0rx/pZK+3th7MyFKC36uluGPjomPcGBbM6cOVaBasuWLZbnxo8fLxcXF2XPnl2dOnXSu+++qy5dulied3V1VXh4uNVj6tSpVtuvVq2awsPDtW3bNrVr106vv/66Gjdu/NA+9e/fXzExMZbH6dOnbTvoR3Q74Y7Cj5xX6LMBlmUmk0lVnw3Qjj/PPHTd+NsJOn/5qhyzOKh+1eL66ddDabbNmc1JhfLn1oWn4LQmp6yOKh3sp807jliWJSYmavOOIypf0j/VdcqX9NfmnUeslm3cfljlSxayZ1efKNTV9tKq6aad/1DTHSlquo2a3s8pq6PKBPtp047DlmXJczX1OlUoWciqvSRt2HbI8u9Q0NdTeT3drNrExt3QH3+eUPlS/jYfQ0ZDTe2DutoHdbU9p6yOKh304PvVlh2HVS6NmpYL8deWFO9Xm7Yfsmp/L4wdO31Ji77uJg/31O8wnpE8sacsNmjQQBUrVrT87Ovra/n/Vq1aacCAAcqePbvy5csnBwfr3Ong4KDAwMCHbj9nzpyWNt99951Kly6tb7/9Vm+88Uaa6zg7O8vZ2fnfDMfmxi/cqvH9XtHuI+e06+AZdWlSWTmzOWnO6qRTCyf0f1XnL8Xq46lrJUllixVQvjyu2nf0gvLncVPf9tXkYDJp7Lz/Wbb5cedaWv3bYZ2+EK18eVzVr3113Uk064d1+wwZY3rr2qKaun08W2WKPaNnixfUpO836vrNeLWsV0mS1GXwTOXzyqWPujWQJL3VLEz1O4/VuDnrVPP5Elq8ZpfCD57SmP7NLdu8EnNNZy5e0YVLMZKkoycvSkr6BiitoxmZDXW1va4tq6nbkLs1LVFQE7/fqOs37qvpoJnK531fTZuHqf5bY/XNnHV66fkS+vGXuzX9IO2a/nWvph5uypsn89dUkrq2rK6uQ2bp/4o9o2dL+GvCvA26diNereon1bXzoJnK5+Vu+dMAbzUPU723vtQ3s9fppRdK6Mdf/lD4wVP68oMWkpK+KOvcopo++261Avy8VNDXU8MnrpRPHnfVDS1t2DjTEzW1D+pqH9TV9jq3qKbuQ2erTDG/pM8A8zfq+s1balE36TN+tyGz5OPlrg+7Jr1fvdksVA27fKXxc9bf/Qzwh8IPntbn/ZLer24n3FGH/t9q7+EzmvP5W7qTaLZcq5fbLYecsmbM6JMxe/UIXF1d5eqa+uFMd3f3fwxcj8PBwUEffPCB3nvvPbVs2VLZs2fsw56StHjDfuVxz6EP2leXt4eL9v19QU36zrLc6KOAt7sSE82W9s5OjhrQ4UX558+tazduac22v9R5+A+KvZZ8q39fLzdNHdhEHm45dDnmmrbtO6Wa3SYrMubpuPX9KzXL6nJ0nEZOXqmIyKsKKeqrBV92tZxWcPbiFTk4mCztK5QK0OSh7fXJxBUaNmGFAvy8NGt0JxUrnN/S5qct+9R96BzLzx0HTpck9elYR307vZw+AzMYdbW9V2uWVeSVOI24r6YLxybX9EyKmla8W9PhE1do2Pikms7+tJOKp6jp2x/fV9MB0yUl1bTfm5m/ppL06ktJc3X4pKS6lizqq0VfdUuu64UoOZjuq2vpAE0Z1l6fTFihoeOXJ9X1szdVPDC5ru+0raHrN+L17vB5iom7oUqlC2vRV12VzTlruo/PCNTUPqirfVBX23ul5rOKjI7TqCmr7v6x7QKaP6bLfTW9IpPJ+jPAxI/bacSklfpk4nIF+HlrxuiOls8A5yOitXrLfklStTajrF5rybjuer5skXQa2eMxmc1m8z83yzjat2+v6OhoLVmyJNXnw8LCVKZMGX355ZepPj99+nS98847Onz48APPeXt7y8HBIdXXSEhIkL+/v3r27KlevXo9Ul9jY2Pl7u4u50p9ZHLMGEfOMoPIdUOM7gLwyO57H4GNmCgqANhUwp1Eo7uQ6cTGxsrXO7diYmLk5vbwM0me2GvI/ovY2Fjly5fvgUdERESa6zg6Ourtt9/W6NGjde3atTTbAQAAAMCjeuKOkD1JOEJmHxwhw5OEgzm2xxEyALAtjpDZHkfIAAAAAOAJQCADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgjkZ34GlwdGl/ubm5Gd2NTCPg7R+M7kKmdPTrV43uQqaUkGA2uguZTjanLEZ3IVO6Hp9gdBcynRzOfMzCk+Hm7USju5DpxD9GTTlCBgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGcTS6A7Cf7xZt1vg56xURFavigb4a/l4TPVuiYJrtl63brVGTV+r0hSgVKuClD7s1UI3nSlie/3TqKi1Zs0tnI6LllDWLSgX5qX/neipbwj8dRpMxtKlaWG/VLCovt2w6eCZGgxbs1p6TV1Jt+33PUFUq6vXA8vX7z6vD+F8lSXlcndWvUUlVKZZXbjmyavtflzVoQbhOXIqz6zgymm8Xbda42UlztUSgr0a8//C5unTdbo2cvFKnz0cpwC9prta8O1dvJ9zRiIkrtPa3Azp5NlKuLtkUWj5IH3ZtIB8v9/QakuGm/bBF4+eu16W7v/+fvNtY/1c87ZouX79bo6as0pm7v/8Du9TXi/f9/t+vz+j5mrV0q4b0eEVvNguz0wgypikLNunr2esUERmrkCK+GtX7tYfuA5es3aXhE1fq1PlIBfh5aXD3Rnrp+eS6ms1mjZi0UjOXbFVM3A1VLBWgz/s1U+FnvNNhNBnD9B+2aMK89boUdVXFC+fX0H+cq+H6dGryXP2gS329WLm45fmen8zRwp92WK0TViFYc77obLcxZETMVfugrrY3/cctmnR3H1CscH593PPh+4AVG8L12d19gH8BL33Qub6q37cPkKS/TlzQ8InLtS38byXcSVQR/7yaPKyDfPPmtvdw/pUMeYSsffv2atSoUZrPh4WFyWQyaeTIkQ88V7duXZlMJg0ePNiqfc+ePS0/Hz9+XC1btlT+/PmVLVs2FShQQA0bNtShQ4estrVhwwa9/PLL8vT0VI4cOVS8eHG9//77Onv27H8dot0tWbtLg75arPffqK0103urRBFfNX93vC5FXU21/Y69x9R50Ay1rF9Za2f0UZ2qpdS+71Qd/PucpU2An7eGv/+aNs7up2UTe8ovn4eavTNel6+kvs3Mpl7ZAhrYuJTGrjyguiPW6sDZaM3sXkWeLs6ptn9r8laV77fc8qg59Bcl3EnUql1nLG0mv/Wc/PLkVKdJW1V3+Fqdjbqu2T2qKLtTlvQaluEWr9mlj8YuVq+OtbVuRtJcbdoz7bm6fe8xvfXRDLWqX1nr787Vdn2S5+qNm7e09/AZvfd6La2b0VvTR76hoycj1Lr35PQclqGWrt2lwV8v1vsdaunn73qreGB+tXhvQpq/qzv2HVeXwTPVsl4l/TKtt2pXKanX+3+rQ8fOPdB21aY92vXnSfnkeXrC7T0//vKHBn65WH071tHGWX0VUsRXjbuPS3OubttzTB0HTlfrhpW1aXY/1Q0trda9JuvA0eS6jp25VpPmb9IX/ZtrzbReypHdSY27j9PN+NvpNSxDLV23S0O+WaL3Xq+t1d/2UvFAX7V6b+JD52q3ITPVol4l/fxdL9WqUlJv9P9Wh46dt2pXrWKwdi/92PIYN7htegwnw2Cu2gd1tb1l63Zp6DdL1LN9ba2amrQPaPN+2vuAnfuO6+0hM9W8biX99G3SPqDjB9b7gBNnL+vVbl8p8Jm8WvDV2/pleh+9066WnJ0y7nGoDBnIHoWfn5+mT59utezs2bNat26d8uXLl+Z6t2/fVs2aNRUTE6Mff/xRhw8f1vz581WyZElFR0db2k2aNEk1atSQj4+PfvjhBx04cEATJ05UTEyMPv/8czuNynYmztug1g2eU4t6lRRUKJ8+7dNU2Z2dNG/F76m2n7xgk6pVLKZurV9UUX8f9XurrkoGFdB3i7ZY2jSuVU6hFYLk75tHwQH59PE7r+jqtZtWO5bMrGP1ovr+1+Na+PtJHb1wVQPm7dKNW3fU9Dn/VNvHXL+tS7HxlkeVYG/duHVHK+8GskLeLno2wFMDv9+tvSev6FhEnAZ8v0vZnLKoQTm/dByZsSbO26DWDZ9Ty7tz9bO+TZU9m5PmpjVX529S9UrF9HbrF1W0kI/6v1VXpYIK6Nu7c9XNJbsWfd1NjWo8q8CCeVUupJBG9mqiPYdO68yFqPQcmmEmzd+oVvWfU/O6lRRUyEejez/893/qgk2qVjFYXVsl/f73fbOuSha1/v2XpPOXojVwzA8aN6iNHB2fni8N7hk/d73aNnpOrRpUVnBAPn3Rv7lyZHPS7GW/pdp+0vcb9WLlYurRpoaCCvloQJd6Kh3spykLN0lK+mZ84rwN6tWhll4OLaWQIr6aMKStLlyO0cpNe9JzaIaZ8v1GtaxfWc3qVlTRQj4a2fs1Zc/mpO9XbEu1/bcLNymsYrC6tKyuIv4+6tPpZYUULaBpP1jPVScnR3l7ulkeudxypMdwMgzmqn1QV9ubMn+jWty3DxjR6zVly+ak+SvT2Acs2qSwCsHqfHcf0Ltj0j5gxo/J+4DRk1eqeqXiGtC1gUKKFpC/bx699EKI8uR2Ta9hPbYnNpDVq1dPly9f1q+//mpZNmPGDL300kvy9k77MO+ff/6pv//+W+PHj1elSpVUsGBBPf/88xo2bJgqVaokSTpz5ox69OihHj166LvvvlNYWJj8/f1VtWpVTZ06VR999JHdx/df3LqdoL2HT6tK+SDLMgcHB1UtH6Sd+4+nus4f+0+oavmiVsuqVSyWZvtbtxM0a8lWublkV4kivrbrfAaVNYtJIc/k0q+HIyzLzGbp10MX9Wwhz0faRtPnCmn5H6d149YdSZKTY9KvX/ztO1bbvJWQqPKF89iw9xnXrdsJ2nP4tEJTm6v7Up97O1Obq5WKpdlekmLjbspkMsndNbttOp6BJf/+J9fIwcFBVcoV1R/7T6S6zs4/j6tKuSCrZWEVg/XHn8ntExMT1f3j2erSsrqCAtL+0iuzunU7QeGHTiusgvVcDa0QpB1pzL3t+44rrHyw1bLqlYppx74TkqSTZyN1MTJWYRWS27i7ZFfZEv7asfeEzceQ0dy6naC9R86oSjnrufpCuaJWc+9+f+w/YdVeujtXU8zt33YfVal6A1WlxSfq99kCRcVcs3X3Myzmqn1QV9u7dTtB+46c0QtlU3m/SmMfsGv/Cb2QYh8QWiF5H5CYmKj1vx1QIT8vtXpvgsrUH6j6b36h1Zv32msYNvHEBjInJye1atVK06ZNsyybPn26OnTo8ND1vLy85ODgoEWLFunOnTuptlm4cKFu3bqlPn36pPp8rly5Ul0eHx+v2NhYq4cRoqKv6c6dRHl5WH8T4OXhqojI1A8BR0TGysvD7R/b//K//SpUvZeeCX1fk77fqAVju8ozl4ttB5AB5XZxlmMWB12OvWm1/NLVeHm5ZfvH9UsXzK1gX3fN/zV5p/33has6E3lNfRqGyC17VmXNYlLnmkHKnzuHvN3/eZuZQVpz1Tv3w+eqd8q5+pD2N+Nv6+NxS/VqzWflmjPzB7KH/v6ncVrNpciraewvkvdh38xepyxZHNTxtVDbd/oJEBkdl0Zd3azqdL+IyFh5eaZd14t3/5uyjbena5rbzEyiYpLmap5U5t6lNMZ/KeqqvFJ8y50nt6suRSW3r1axmMYObK35Y7tqQJf6+j38b7XpNUl37iTafhAZEHPVPqir7d3bB6SsaZ7cD98HpNxn5PFI3gdcvhKnazfiNX7OOoVVLKY5X3RW7aql9ObAafpt91H7DMQGnthAJkkdOnTQggULdO3aNW3evFkxMTGqV6/eQ9fx9fXVV199pY8++ki5c+dW9erVNXToUB07dszS5q+//pKbm9tDT31MzYgRI+Tu7m55+PllvtPOni9bROtn9NWKyT1VrVIxdRo4Lc1zp5Gs2XOFdPBstNUNQBISzeo8+TcFeLtq7+cNdfDLV1S5qJc27D+vRLOBnc1EbifcUccB02Q2S5/2bWp0d55Yew6d1tSFmzR2QCuZTCajuwM8VMMaz+qlF0JUrHB+1a5aSjNGdVL4wVPamoE/jAGwjURz0geol14IUadmYSpRpIC6ta6hF58rrtlLf/2HtY2ToQPZnDlz5OLiYnls2WJ9jnjp0qVVpEgRLVq0SN99953atGkjR8d/vmCvW7duunDhgubMmaPKlStr4cKFKlGihNasWSMp6Zzef/Oho3///oqJibE8Tp8+/djbsAWPXDmVJYvDA0HpUtRVeXumfv6st6eb1TeMabXPmd1Zhfy8VC6kkL4c0FKOWbJo7vLUz53OTK7ExSvhTqLypDga5uXqrEspjpqllN0pi+qV89OCrSceeG7/6Wi9PGKtSr63RBX6r1C7cf9T7pzOOnX56bjLYlpzNeLKw+dqRMq5mkr7e2HszIUoLfq621NxdEz6h99/j9Rr6uXpmsb+IulI5LY9f+vylTiVazxYBaq+qwJV39WZC1Ea8s0SlW88xD4DyWA8c7mkUddYS51S8vZ006XItOua9+5/U7aJiLya5jYzEw/3pLl6OZW555XG+L08XHUpxcX+l69cfeAMj/sV9M0jj1w5deLMpf/e6ScAc9U+qKvt3dsHpKzp5SsP3wek3GdcjkreB3i455RjFgcV8fexalOkYF6duxhtu87bWIYOZA0aNFB4eLjlUa5cuQfadOjQQePGjdOiRYv+8XTF+7m6uqp+/fr65JNPtGfPHlWpUkXDhg2TJBUtWlQxMTE6f/78P2zFmrOzs9zc3KweRnDK6qhSQX7asvOIZVliYqK27DysciGFUl2nbIi/VXtJ2rT9UJrtLds1J+rW7YT/3ukM7vYds/afitZzQcnXJ5pM0nNB3tp1PPKh69Z9toCcHR20ePupNNtcvZmgqLhb8vdyUcmCubVm7+PNvSeVU1ZHlQ7y0+YdKebqjsMqVzL1uVcuxF9bdqQyV+9rfy+MHTt9SYu+7iYP95z2GUAGdO/3/38pfv//98cRlQ3xT3WdciUK6X9/WNd0847Dlls5N6ldXutn9tHa6b0tD5887urasrrmPSW3EnfK6qgywX7atOOwZVliYqI27zii8mnM1QolC1m1l6QN2w6pfEl/SVJBX0/l9XSzahMbd0N//HlC5Uv523wMGY1TVkeVKlpA//vjL8syy1xN4zbiZUP89b+df1kt27zjcJpzW5LORUTrSsx15X1K7gzKXLUP6mp7TlkdVbJoAf36GPuAZ0P8rdpL0padyfsAp6yOKl3sGR07FWHV5tjpS/L1yZi3vJcyeCBzdXVVYGCg5ZE9+4PfcLds2VL79u1TSEiIihcvnspW/pnJZFJwcLCuXUu66LdJkyZycnLS6NGjU21//90YM6rOLappzrKtmr9ym46cuKA+oxfo+s1bal6voiTp7SGzNGz8Mkv7N5uGasPvBzVh7nr9deKiPp26SnsOnVaHJlUkSdduxOuTCcu1c/9xnT4fpT2HTumdYXN04VKM6lf/P0PGmN6mrj+iFs8XUuOKBVXYx1WfNH9WOZwdtfC3E5Kkz9uVV5+GIQ+s1/S5QvplzzlFX7v1wHMv/5+vKhXxkp9nTtUslU+ze1TRL3vOasvBi/YeTobRuUU1zV62Vd+v3KYjxy+o99252qJu0lztNmSWht4/V5uFav3vBzV+TtJcHT1llcIPntYbd+fq7YQ76tD/W4UfPKUJQ9rqTqJZFyNjdTEy9qn48kCS3moWpjnLf9OCVdt15MQF9f1sYdLv/92adh86W59MWG5p3/Hu7//Eeev118mL+uzbn6x+/z3ccyo4IL/Vw9Exi7w83BRYMK8hYzRC15bVNXPJVs1b8bsOH7+g90bO17Ub8WpVP+mGUJ0HzdSQb5Za2r/VPEzrfjugb2av05ETFzRy8kqFHzylTnevwzOZTOrcopo++261Vm3aqz+PnlWXwbPkk8dddUNLGzLG9NapeZjmLv9NC37arr9OXFC/zxbqxo1banZ3rvYYOlsjJibP1TdeC9XGbQc1cd4GHT15UZ9/+5P2Hjqt1xvffa+6Hq+h45bqj/0ndPp8pLbsPKIO/abK3zePQisEp9qHzIi5ah/U1fY6NQvTvBW/aeHdfcAHnyftA5q+nLQP6Dlstkbevw9okrQPmPR90j7gi++S9gHtXq1iafNWi+pavn635i77TcfPXNL0H7Zo7dY/1faVF9J9fI8q496Q/xHlzp1b58+fV9asWR+pfXh4uAYNGqQ2bdqoePHicnJy0qZNm/Tdd9+pb9++kpJuqT9mzBi9/fbbio2NVdu2beXv768zZ85o5syZcnFxyfC3vm9U41lFXonT6KmrFBEZqxJFCmjemC6WmyGcvXhFDg7Jp2WWLxWgCUPaaeTklRo+cbkK+Xlr+qiOKlY4vyQpi4ODjp68qAWrtisqJk653XOqTLFntHTCOwp+Su64tuKPM/Jwcda79Ypb/jB0u2/+p8tX4yVJvrlzyJzi4q8AbxdVCMyj1l9tTnWb3u7ZNbBJaeVxzaaImBv6cdspff3TAbuPJSN5peazioyO06gpq+7+oc0Cmj+mi+V0jTMXrlidQlyhVIAmftxOIyat1CcTlyvAz1szRifP1fMR0Vq9Zb8kqVqbUVavtWRcdz1ftkg6jcw4DWsk1XT01FW6FJX0+z/3886WUzrOXrwih/tqWr5kIY0f3FajJq/SiEkrVKiAl6aNeEPBAfmNGkKG9OpLZXU5Ok7DJ61URORVlSzqq0VfdbtvrkZZ1bVi6QBNGdZen0xYoaHjlyvAz0uzP3tTxQOT6/pO2xq6fiNe7w6fp5i4G6pUurAWfdVV2Zwf7T3tSdfwxWcVFX1Nn039KWmuBvpq9udvWS7yP5fyvapkIX0zqK1GT1mpUZOT5uq3I96wvA85ZDHp4N/ntPCnHYqNu6G8edwUWj5YvTu9nKH/BpGtMVftg7raXoO7+4DPv03aBxQP9NWsz5L3AWcvWn8GKFeykL4e1FafTlmp0ZNXyL+Al6YOf8Pqs2idqqU0vNdrGjd7rT4a+6MKP+OlSUNfV4VSAek+vkdlMpvNGe72Ae3bt1d0dLSWLFmS6vNhYWEqU6aMvvzyy1SfL1OmjBo1amT549D3t798+bKGDh2q9evX68SJEzKZTPL391e7du307rvvysEh+aDh2rVr9dlnn2n79u26ceOG/P39Va9ePb333nuPdMOP2NhYubu76/TFK4advpgZFe3xo9FdyJSOfv2q0V3IlBLuZLhd7BMv21P0h9PT0/X4p+MIcnrK4fz0hEA82eJu8vtva1djYxXg66mYmJh/zAEZMpBlFgQy+yCQ2QeBzD4IZLZHILMPApntEcjwpCCQ2d7jBLIMfQ0ZAAAAAGRmBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgzga3YGngYMp6QHbOPr1q0Z3IVPyqjPS6C5kSpGr+xvdBeCR5HDmIwHwtMqWlWM0tnbrMWpK9QEAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIM4Gt0B2M+3izZr3Oz1ioiKVYlAX414v4meLVEwzfZL1+3WyMkrdfp8lAL8vPRhtwaq+VwJSdLthDsaMXGF1v52QCfPRsrVJZtCywfpw64N5OPlnl5DMhw1tY+O9cuq+2sV5e3hov3HLqrvuF+06/D5VNs6ZnHQu82fU4uaJZUvj6uOno7U4G83aN3OY5Y2Heo9qw71npVf3qQ6Hjp5SZ/O+Z/W7jiW6jYzo6kLN+ubOesUERmrEkV8NfL9Jipbwj/N9kvX7dbwSSssc3VQt4aq+XwJy/PLN4Rr+o+/as+hU7oSe10bZ/VVyaIF0mEkGcuUBZv09eykuoYU8dWo3q89tK5L1u7S8Ikrdep8pAL8vDS4eyO9dF9dzWazRkxaqZlLtiom7oYqlgrQ5/2aqfAz3ukwmoyBmtoHdbUP6mp7fLbiCFmmtXjNLn00drF6daytdTN6q0QRXzXtOV6Xoq6m2n773mN666MZalW/stbP6KM6VUupXZ+pOvj3OUnSjZu3tPfwGb33ei2tm9Fb00e+oaMnI9S69+T0HJahqKl9vBJaTMPeelGjZv9PYV2/0/5jEfpheHPlyZUj1fYD24eqfd3/U99xv6hSx8matnK3Zg1qrJKF81ranLscqyHfblC1bt+p+tvTtCX8pOYMfk3BBfOk17AMtXjNH/pw7GL1fqOO1s/oo5BAX732zsPnaqcPp6t1/craMLOvXq5aSm36TLHMVUm6fuOWKpUO0KC3G6bXMDKcH3/5QwO/XKy+Heto46y+Ciniq8bdx6VZ1217jqnjwOlq3bCyNs3up7qhpdW612QdOJpc17Ez12rS/E36on9zrZnWSzmyO6lx93G6GX87vYZlKGpqH9TVPqir7fHZKskTFcjat2+vRo0apfl8WFiYevbsmebzUVFR6tmzpwoWLCgnJyflz59fHTp00KlTpx5oe+HCBXXv3l0BAQFydnaWn5+f6tevr3Xr1tlgJPY3cd4GtW74nFrWq6SgQvn0Wd+myp7NSXNX/J5q+8nzN6l6pWJ6u/WLKlrIR/3fqqtSQQX07aItkiQ3l+xa9HU3NarxrAIL5lW5kEIa2auJ9hw6rTMXotJzaIahpvbRtXEFzfwpXHN/2avDpy7rvbE/6Xp8glrXKp1q+6Y1QjRm3lat2fG3Tl6I1ncrdmnN9r/1dpOKljarfz+qNTv+1rFzV/T32SgNm75J127cUrlivuk1LEONn7dBbRpWVqv6lRQckE+f92um7NmcNGf5b6m2nzR/o16sVEzd29RQUCEffdC5nkoF+Wnqws2WNs1erqDeHesotHxQeg0jwxk/d73aNnpOrRpUVnBAPn3Rv7lyZHPS7GVp1PX7jXqxcjH1uFvXAV3qqXSwn6Ys3CQp6ZvxifM2qFeHWno5tJRCivhqwpC2unA5Ris37UnPoRmGmtoHdbUP6mp7fLZK8kQFsv8iKipKlSpV0tq1azVx4kQdPXpU33//vY4ePary5cvr2LHkU5lOnDihsmXLav369fr000+1b98+rV69WtWqVVO3bt0MHMWjuXU7QXsOn7b64OTg4KCq5YO0c9/xVNfZuf+EqpYvarWsWqViabaXpNi4mzKZTHJ3zW6bjmdg1NQ+sjo6qEyRfNq4+4Rlmdksbdp9XOXTCE/OWbPo5u0Eq2U3byWoUonUT59zcDDp1bDiypEtq3YcOGuzvmdUt24naM+h0wqtYD1XQ8sHace+E6mus2PfiQeCVvVKwdrxkLn6tLl1O0Hhh04rLGVdKwSlWaft+44rrHyw1bLqlYpZ/h1Ono3UxchYhVVIbuPukl1lS/hrx94TNh9DRkNN7YO62gd1tT0+WyV7aq4hGzBggM6dO6ejR4/Kx8dHkvTMM8/o559/VpEiRdStWzf99NNPkqSuXbvKZDJp+/btypkzp2UbJUqUUIcOHQzp/+OIir6mO3cS5eXharXcO7erjp64mOo6EZGx8vZws1rmldtVEZGpHzK+GX9bH49bqldrPivXnBl3gtsKNbUPT7cccszioEtXrlktv3Tlmor4eaa6zvqdx9X11QrauveUjp+/otD/81e954OUxcFk1a64v5d+HttO2Zwcde3GLbUZ8oMOn7pst7FkFJF352rKueft4aq/TqY9V1PObS+PtOfq0ygyOi7VfYCXh5v+esg+wMsztbrGSpIu3v1vyjbensltMjNqah/U1T6oq+3x2SrZU3GELDExUd9//71atWplCWP3ZM+eXV27dtXPP/+sqKgoRUVFafXq1erWrZtVGLsnV65cab5OfHy8YmNjrR6Z0e2EO+o4YJrMZunTvk2N7k6mQE0fXb8Ja3TsXJS2f/uWIlb10+hutTT3l71KNJut2v11JlJVu3yrGj2m67sVuzS+d30FPfN0XEMGAMDT7kn6bPVUBLJLly4pOjpaxYoVS/X5YsWKyWw26+jRozp69KjMZrOCg4NTbfswI0aMkLu7u+Xh5+f3X7v+r3jkyqksWRweuCAy4spVeaf4FuYeb083RURZB8hLqbS/N7nPXIjSoq+7ZehvG2yJmtpHZOx1JdxJlFdu6y8/vHLnVETUtdTXibmu1oN/kG+DT1Wq9Teq8MYkXbtxSyfOR1u1u52QqOPnrmjPXxf08Xcbtf/YRXV+pby9hpJheN6dqynnXkTU1Qe+VbzH29Ptgbl9KSrtuf008szlkuo+4FJUrLw9H1LXyNTqmtQ+793/pmwTEXk1zW1mJtTUPqirfVBX2+OzVbInMpDNmTNHLi4ulseWLVseaT1zim/Q/22btPTv318xMTGWx+nTp//1tv4Lp6yOKh3kp807jliWJSYmasuOwypXslCq65QL8deW+9pL0qbth6za35vcx05f0qKvu8nD/cEjiJkVNbWP2wmJCv/rvELL+FuWmUxS1TL+2nHw4dd7xd++o/ORcXLM4qD6LwTpp9+OPLS9g4NJTlmz2KLbGZpTVkeVDn5wrm7ecUTlS/qnuk75kv7avNO6fhu3H1b5NOb208gpq6PKBPtp047DlmXJdU29ThVKFrJqL0kbth2y/DsU9PVUXk83qzaxcTf0x58nVL6Uv83HkNFQU/ugrvZBXW2Pz1bJnshA1qBBA4WHh1se5cqVe2h7Ly8v5cqVSwcPHkz1+YMHD8pkMikwMFBFihSRyWTSoUOHHrtfzs7OcnNzs3oYpXOLapq9bKu+X7lNR45fUO/RC3T95i21qJt0J7puQ2Zp6PhllvZvNgvV+t8Pavyc9frrxEWNnrJK4QdP640mVSQlTe4O/b9V+MFTmjCkre4kmnUxMlYXI2N1K8UNFjIramof43/YrrYvl1HzmiVV1M9TX/Soo5zZsmrOz3slSRN619dHHcIs7csG51e954NU0CeXKof4adHw5nJwMGnsguQ7Mn3UIUzPlfSTX153Fff30kcdwvRCqYJauH5/eg/PEF1bVNOspVs1b+U2HT5+Qb1GLdD1m/FqWa+SJKnL4Jn6eFzyXH2rWZjW/XZA4+as05ETFzRqyiqFHzyljq9VtbS5EnNN+46c0eHjFyRJR09e1L4jZyzXQDwNurasrplLtmreit91+PgFvTdyvq7diFer+kl17TxopoZ8s9TS/q3mSXX9ZnZSXUdOXqnwg6fU6bVQSZLJZFLnFtX02XertWrTXv159Ky6DJ4lnzzuqhua+l1GMxtqah/U1T6oq+3x2SrJE3lTD1dXV7m6PvqpNA4ODmratKnmzJmjjz/+2Oo6shs3bmj8+PGqVauWPDw8JEm1atXSuHHj1KNHjweuI4uOjn7odWQZxSs1n1VkdJxGTVl1948XFtD8MV0sh8DPXLgikyn5JggVSgVo4sftNGLSSn0ycbkC/Lw1Y3RHFSucX5J0PiJaq7ckfZit1maU1WstGdddz5ctkk4jMw41tY/Fmw4qj3sOfdC2qrxz59S+YxfVZMB8XYpOOmWxgLeb1fVhzlkdNaB9qPzz5dK1G7e0Zvvf6jxqmWKvxVva5MmVQxN611deDxfFXo/Xn8ci1PiDedq460R6D88Qr9Qsq8vRcRo5eaUiIq8qpKivFnzZ1TJXz168IgcH67k6eWh7fTJxhYZNWKEAPy/NGt3JMlcl6act+9R96BzLzx0HTpck9elYR307vZw+AzPYqy8l1XX4pKS6lizqq0VfdbtvHxAlh/v2ARVLB2jKsPb6ZMIKDR2/XAF+Xpr92ZsqHphc13fa1tD1G/F6d/g8xcTdUKXShbXoq67K5pw13cdnBGpqH9TVPqir7fHZKonJ/F/O0Utn7du3V3R0tJYsWZLq82FhYfL19VXv3r2tlufLl0+Ojo6qWLGismfPrtGjRyskJETHjx/XwIEDdfjwYf32228KCAiQJB07dkzPP/+8PDw89PHHH6tUqVJKSEjQmjVrNGHChDSPtKUUGxsrd3d3nY24YujRMuBReNUZaXQXMqXI1f2N7kKm45DijpoAgP8m4U6i0V3IdGJjY+XrnVsxMTH/mAOeyFMWH2bu3Ln6v//7P6vHlClT5Onpqd9//13VqlXTW2+9pcKFC6tp06YqXLiwduzYYQljkhQQEKBdu3apWrVqev/99xUSEqKaNWtq3bp1mjBhgoGjAwAAAJCZPFFHyJ40HCHDk4QjZPbBETLb4wgZANgWR8hs76k+QgYAAAAATwoCGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABnE0ugNPgywOJmVxMBndjUzDbDa6B5nTlV8+MLoLmVLuF/oY3YVM58r/RhvdBQDIVPicanuPU1OOkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQRyN7gDsZ+rCzfp69jpFRMaqRBFfjerVRGVL+KfZfsna3RoxaYVOnY9SgJ+XBr/dUDWfL2F5fvmGcE378VftOXhKV2Kva9PsvipZtEA6jCTjmLpws76Zk1zTke8/vKZL1+3W8EkrdPpuTQd1e7Cm03/8VXsOJdV046ynr6aSNGXBJstcDSniq1G9X/uHubpLwyeu1KnzkUlztXsjvXRfXc1ms0ZMWqmZS7YqJu6GKpYK0Of9mqnwM97pMJqMoeOrldW9Rai8PVy1/+/z6jtmqXYdPJ1qW8csDnq3TXW1qFNW+fK46ejpSxo8YZXWbTtiabNnYT89k8/jgXWn/rhVvb9YYq9hZDjMVdujpvZBXe2Dutoen1cz+BGy9u3by2QyWR6enp6qXbu29u7dm+Y6J06ceGCdl156Sbt377a0CQsLs2pz79G5c2dLm/uXu7m5qXz58lq6dKldx2tLP675QwO/XKw+Hetow8w+CiniqyY9xutS1NVU22/be0ydPpyuVg0qa+Osvno5tJRa956iA3+fs7S5fuOWKpUO0KC3G6bXMDKUxWv+0IdjF6v3G3W0fkYfhQT66rV30q7p9rs1bV2/sjbM7KuXq5ZSmz5TdJCaWvnxl6S52rdjHW2c1VchRXzVuPu4tOfqnmPqOHC6WjesrE2z+6luaGm17jVZB44m13XszLWaNH+TvujfXGum9VKO7E5q3H2cbsbfTq9hGeqV6qU17O36GjVtrcLeGKv9R8/rhy/eUJ5cOVNtP/DNWmrfsKL6jlmqSm0+17Qlv2vW8HYqWSS/pU31Tl8rqMHHlkejnpMlSUs2pL0/zmyYq7ZHTe2DutoHdbU9Pq8mydCBTJJq166t8+fP6/z581q3bp0cHR1Vr169f1xv7dq1On/+vH7++WfFxcWpTp06io6OtjzfqVMny3bvPUaPHm21jWnTpun8+fPauXOnnn/+eTVp0kT79u2z9RDtYvzcDWrbqLJa1a+k4IB8+qJfM+XI5qQ5y39Ltf2k7zfqxUrF1KNNDQUV8tGAzvVUKthPUxdstrRp9nIF9elYR2EVgtJrGBnK+Hkb1KZhck0/79dM2R9W0/lJNe1+t6YfdK6nUkF+mrrQuqa9O9ZRaPmns6aSNH7uerVt9JxaNaicNFf7N1eObE6avewhc7XyfXO1Sz2VDvbTlIWbJCV92zhx3gb16lBLL4eWUkgRX00Y0lYXLsdo5aY96Tk0w3RtXkUzl2/T3FU7dfhEhN779Eddv3lbreuVT7V901plNWbWeq35/ZBOnovSd0t+15rfDunt5lUtbSKjrykiKs7yqPVcMR07c1m/7j6WXsMyHHPV9qipfVBX+6Cutsfn1SQZPpA5OzvLx8dHPj4+KlOmjPr166fTp0/r0qVLD13P09NTPj4+KleunD777DNdvHhR27ZtszyfI0cOy3bvPdzc3Ky2kStXLvn4+Kho0aIaOnSoEhIStGHDBruM05Zu3U7QnkOnrT7kOzg4KLR8kHbsO5HqOjv2nVBoiolbvVKwduw7bs+uPjEsNa3wmDUtT00f5tbtBIUfOm2103RwcFBohaA067R933GFlQ+2Wla9UjHLv8PJs5G6GBmrsArJbdxdsqtsCX/t2HvC5mPIaLI6ZlGZor7auPOoZZnZbNamnX+pfImCqa7jnDWLbsYnWC27GX9blUr5p/kaTV96VnNW7rBZvzM65qrtUVP7oK72QV1tj8+ryTJ8ILtfXFycZs+ercDAQHl6ej7yetmzZ5ck3bp161+9bkJCgr799ltJkpOTU5rt4uPjFRsba/UwQmT0Nd25kygvD+uA6eXhqouRqfcpIjJW3h6uVsu8PVwVkcYh46fNvZp6p6hpUo3SrqlXipp6ebgqIpKa3hMZHXd3rqask5siHjJXvTxTq2tS+3tzPGUbb0/XNLeZmXi655SjY5YHTve4FBUn7xQ1uWf99iPq2ryKAgrkkclkUli5IqoXGqK8nm6ptq9btYTcXbJp7qo/bN7/jIq5anvU1D6oq31QV9vj82qyDB/IVqxYIRcXF7m4uMjV1VXLli3T/Pnz5eDwaF2Pjo7W0KFD5eLiogoVKliWjx8/3rLde485c+ZYrduiRQu5uLjI2dlZ7777rvz9/dW0adM0X2vEiBFyd3e3PPz8/P7doAEgHfUbu0zHTl/W9jm9FLFhuEa/10hzV+1UotmcavvWdctr7bbDuvAUfGAAAMDeMnwgq1atmsLDwxUeHq7t27erVq1aqlOnjk6ePKk6depYwlSJEiWs1nvuuefk4uKi3Llza8+ePZo/f77y5s1reb5Vq1aW7d57NGjQwGobY8aMUXh4uH766ScVL15cU6dOlYfHg3cZu6d///6KiYmxPE6fTv2OZvbmmSunsmRx0KUUR24uRV1N8xtvb0+3B75diIi6+sC3EE+rezVNeTQsqUZp1/TBoxRX0zxK8TTyzOVyd66mrFOsvB8yVy9FplbXpPb35njKNhGRV9PcZmYSGXNNCQl3UvkW1yXNo7OR0dfU+oOZ8q05UKWajFCFlp/q2o14nTgX+UBbv7y5FFauiGYu326X/mdUzFXbo6b2QV3tg7raHp9Xk2X4QJYzZ04FBgYqMDBQ5cuX19SpU3Xt2jVNmTJFU6dOtYSpVatWWa03f/587dmzR1euXNHff/+tl19+2ep5d3d3y3bvPVxdrf8xfXx8FBgYqJdeeknTpk1Ts2bNFBERkWZfnZ2d5ebmZvUwglNWR5UO9tPmHcm3rE5MTNSmnUdUvqR/quuUL+lv1V6SNm47rPIlC9mzq0+MtGq6ecc/1HRnippup6b3c8rqqDLBftq047BlWXJdU69ThZKFrNpL0oZthyz/DgV9PZXX082qTWzcDf3x5wmVT+OaqMzkdsIdhR85q9CygZZlJpNJVcsGasefJx+6bvytBJ2/HCvHLA6qH1pSP2058ECblnXL69KVOP3y2yGb9z0jY67aHjW1D+pqH9TV9vi8mizDB7KUTCaTHBwcdOPGDfn6+lrCVMGC1her+/n5qXDhwsqVK5dNXrdChQoqW7asPvnkE5tsz966tqymmUu3at6KbTp8/ILeH7VA12/Eq2W9SpKkLoNm6uNxyyzt32oepnW/HdA3c9bpyIkLGjl5lcIPnlLHpsl3WbsSc037jpzR4eMXJEl/nbyofUfO6OLlp+O0pa4tqmnW0q2atzKppr1GLdD1m/fVdHCKmjZLqum4uzUdNeVuTV9Lu6ZH79X0KToVrGvL6pq5ZKvmrfhdh49f0Hsj5+vajXi1qp9U186DZmrIN8l/csIyV2ffm6srFX7wlDq9FiopaR/RuUU1ffbdaq3atFd/Hj2rLoNnySePu+qGljZkjOlt/Pdb1LZ+BTWvXVZFC3rri16vKGd2J81ZuVOSNGFgM330Vm1L+7LF/VSvaogK5vdQ5VL+WvT5G3JwMGns3I1W2zWZTGr1cjl9v/oP3bmTmJ5DyhCYq7ZHTe2DutoHdbU9Pq8myfB/GDo+Pl4XLiQV9MqVK/rmm28UFxen+vXr/6ftXr9+3bLde5ydnZU7d+401+nZs6deeeUV9enTR76+vv/p9e3t1ZplFXklTiMmr1RE5FWFFPXVwrFdLYfAz1y8IgcHk6V9xVIBmjy0vYZPXKFh41cowM9Lsz/tpOKFk/8O0U9b9untj5Ovs+s4YLokqU/HOur3pvURyMzolZpldTk6TiPvq+mCL5NrejZFTSvcreknE1do2ISkms4a3UnFUtS0+9D7ajpwuqSkmvbtlPlrKkmvvpRU1+GTkupasqivFn3VLXmuXoiSg+m+uVo6QFOGtdcnE1Zo6PjlSXP1szdVPDC5ru+0raHrN+L17vB5iom7oUqlC2vRV12VzTlruo/PCIvX71GeXDn1QceX5O3hqn1Hz6nJ+9/q0pU4SVKBvLmUmJh8fZizU1YN6FRL/vk9dO3GLa35/ZA6D52v2LibVtsNKxcoP5/cmv0U3V3xfsxV26Om9kFd7YO62h6fV5OYzOY0rtrOANq3b68ZM2ZYfnZ1dVVwcLD69u2rxo0bp7rOiRMnVKhQIe3evVtlypRJtU1YWJg2bdr0wPJatWpp9erVkpK+tVi8eLEaNWpked5sNqt48eKqVq2axo8f/4/9j42Nlbu7uy5cjjbs9MXMKOPO2Cfb/Ts82E7uF/oY3YVM58r/Rv9zIwDAI8vAceCJFRsbK588uRQTE/OPOSBDB7InHYHMPpix9kEgsw8Cme0RyADAtogDtvc4geyJu4YMAAAAADILAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQRyN7gDwuBwcTEZ3AXhkV/432uguZDq5X+hjdBcypagto4zuQqZjNhvdg8yJzwG2x1y1vcepKUfIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwiOOjNFq2bNkjb7BBgwb/ujMAAAAA8DR5pEDWqFGjR9qYyWTSnTt3/kt/AAAAAOCp8UiBLDEx0d79AAAAAICnzn+6huzmzZu26gcAAAAAPHUeO5DduXNHQ4cOla+vr1xcXHTs2DFJ0ocffqhvv/3W5h0EAAAAgMzqsQPZJ598ounTp2v06NFycnKyLA8JCdHUqVNt2jkAAAAAyMweO5DNnDlTkydPVqtWrZQlSxbL8tKlS+vQoUM27RwAAAAAZGaPHcjOnj2rwMDAB5YnJibq9u3bNukUAAAAADwNHjuQFS9eXFu2bHlg+aJFi/R///d/NukUAAAAADwNHum29/f76KOP1K5dO509e1aJiYn68ccfdfjwYc2cOVMrVqywRx8BAAAAIFN67CNkDRs21PLly7V27VrlzJlTH330kQ4ePKjly5erZs2a9ugjAAAAAGRKj32ETJKqVKmiNWvW2LovAAAAAPBU+VeBTJJ27typgwcPSkq6rqxs2bI26xQAAAAAPA0eO5CdOXNGLVq00K+//qpcuXJJkqKjo/Xcc8/p+++/V4ECBWzdRwAAAADIlB77GrKOHTvq9u3bOnjwoKKiohQVFaWDBw8qMTFRHTt2tEcfAQAAACBTeuwjZJs2bdLWrVsVFBRkWRYUFKSvv/5aVapUsWnnAAAAACAze+wjZH5+fqn+Aeg7d+4of/78NukUAAAAADwNHjuQffrpp+revbt27txpWbZz50698847+uyzz2zaOQAAAADIzB7plMXcuXPLZDJZfr527ZoqVqwoR8ek1RMSEuTo6KgOHTqoUaNGdukoAAAAAGQ2jxTIvvzySzt3AwAAAACePo8UyNq1a2fvfgAAAADAU+df/2FoSbp586Zu3bpltczNze0/dQgAAAAAnhaPfVOPa9eu6e2335a3t7dy5syp3LlzWz0AAAAAAI/msQNZnz59tH79ek2YMEHOzs6aOnWqhgwZovz582vmzJn26CMAAAAAZEqPfcri8uXLNXPmTIWFhen1119XlSpVFBgYqIIFC2rOnDlq1aqVPfoJAAAAAJnOYx8hi4qKUkBAgKSk68WioqIkSS+88II2b95s294BAAAAQCb22EfIAgICdPz4cT3zzDMKDg7WggULVKFCBS1fvly5cuWyQxfxb01duFlfz16niMhYlSjiq1G9mqhsCf802y9Zu1sjJq3QqfNRCvDz0uC3G6rm8yUszy/fEK5pP/6qPQdP6UrsdW2a3VclixZIh5FkHFMWbLLUNKSIr0b1fu0farpLwyeu1KnzkUk17d5IL91XU7PZrBGTVmrmkq2KibuhiqUC9Hm/Zir8jHc6jCbjoK62R03to+OrldW9Rai8PVy1/+/z6jtmqXYdPJ1qW8csDnq3TXW1qFNW+fK46ejpSxo8YZXWbTtiabNnYT89k8/jgXWn/rhVvb9YYq9hZCi8V9nH1IWb9c2c5LqOfP/hdV26breGT1qh03frOqjbg3Wd/uOv2nMoqa4bZz2ddWXfanvM1X9xhOz111/Xnj17JEn9+vXTuHHjlC1bNr377rvq3bu3zTuIf+fHNX9o4JeL1adjHW2Y2UchRXzVpMd4XYq6mmr7bXuPqdOH09WqQWVtnNVXL4eWUuveU3Tg73OWNtdv3FKl0gEa9HbD9BpGhvLjL0k17duxjjbO6quQIr5q3H1c2jXdc0wdB05X64aVtWl2P9UNLa3WvSbrwNHkmo6duVaT5m/SF/2ba820XsqR3UmNu4/Tzfjb6TUsw1FX26Om9vFK9dIa9nZ9jZq2VmFvjNX+o+f1wxdvKE+unKm2H/hmLbVvWFF9xyxVpTafa9qS3zVreDuVLJLf0qZ6p68V1OBjy6NRz8mSpCUb9qbLmIzGe5V9LF7zhz4cu1i936ij9TP6KCTQV6+9k3Zdt9+ta+v6lbVhZl+9XLWU2vSZooPU1Qr7VttjriZ57ED27rvvqkePHpKkGjVq6NChQ5o7d652796td955x6ada9++vRo1avTQNjdu3NCgQYNUtGhROTs7K0+ePHrttdf0559/WrUbPHiwTCaTTCaTsmTJIj8/P7355puWUy7vt3v3bjVr1kz58uWTs7OzChYsqHr16mn58uUym822HKLdjJ+7QW0bVVar+pUUHJBPX/RrphzZnDRn+W+ptp/0/Ua9WKmYerSpoaBCPhrQuZ5KBftp6oLk01CbvVxBfTrWUViFoPQaRoYyfu56tW30nFo1qJxU0/7NlSObk2Yve0hNK99X0y71VDrYT1MWbpKU9K3YxHkb1KtDLb0cWkohRXw1YUhbXbgco5Wb9qTn0AxFXW2PmtpH1+ZVNHP5Ns1dtVOHT0TovU9/1PWbt9W6XvlU2zetVVZjZq3Xmt8P6eS5KH235Het+e2Q3m5e1dImMvqaIqLiLI9azxXTsTOX9evuY+k1LEPxXmUf4+dtUJuGyXX9vF8zZX9YXecn1bX73bp+0LmeSgX5aepC67r27lhHoeWf4rqyb7U55mqSxw5kKRUsWFCvvvqqSpUqZYv+PJb4+HjVqFFD3333nYYNG6YjR45o1apVSkhIUMWKFfX7779btS9RooTOnz+vU6dOadq0aVq9erW6dOli1Wbp0qWqVKmS4uLiNGPGDB08eFCrV6/WK6+8ooEDByomJiY9h/iv3LqdoD2HTltNRAcHB4WWD9KOfSdSXWfHvhMKTfHmVb1SsHbsO27Prj4xbt1OUPih01Zv8A4ODgqtEJRmjbbvO66w8sFWy6pXKmb5Nzh5NlIXI2MVViG5jbtLdpUt4a8de0/YfAwZEXW1PWpqH1kds6hMUV9t3HnUssxsNmvTzr9UvkTBVNdxzppFN+MTrJbdjL+tSqX803yNpi89qzkrd9is3xkZ71X2Yalryn3AP9W1PHV9GPattsdcTfZI15B99dVXj7zBe0fP0sOXX36p3377Tbt371bp0qUlJQXEH374QRUrVtQbb7yh/fv3y2QySZIcHR3l4+MjSfL19dVrr72madOmWbZ37do1vfHGG6pbt65+/PFHq9cqVqyY3njjjSfiCFlk9DXduZMoLw/rP9Lt5eGqIycvprpORGSsvD1crZZ5e7gqIo1Dxk+byOi4uzW1rpGXh5v+OpF2Tb08U7Z3VURkrCTp4t3/pmzj7ZncJrOjrrZHTe3D0z2nHB2zPHAazaWoOBUpmPq1Huu3H1HX5lW0dc9xHT8bqdCygaoXGqIsDql/F1q3agm5u2TT3FV/2Lz/GRHvVfZxr67eKerq7eGqvx5S1wf3Ga6KiKSu97BvtT3marJHCmRjxox5pI2ZTKZ0DWRz585VzZo1LWHsHgcHB7377rtq1aqV9uzZozJlyjyw7okTJ/Tzzz/LycnJsuyXX35RZGSk+vTpk+Zr3gt3qYmPj1d8fLzl59jYzP/LBABIXb+xyzS2T2Ntn9NLZrNZx89Fae6qnWpVN/VTHFvXLa+12w7rwlPwQQwAkOyRAtnx4xnzMOCRI0dUrVq1VJ8rVqyYpc29QLZv3z65uLjozp07unnzpiTpiy++sNqeJAUFJR8K3bFjh9VrfP/996pXr16qrzlixAgNGTLk3w/IRjxz5VSWLA66FGX9pn4p6qryerqluo63p9sD3zBGRF194JvIp5VnLpe7NU357XisvB9S00uRKdtftbS/929xKfKqfPK4W9pERF7N8HcDshXqanvU1D4iY64pIeFOKt/MuqT5zWxk9DW1/mCmnJ0c5eGWQ+cvx2pwlzo6cS7ygbZ+eXMprFwRtRkw0y79z4h4r7KPe3WNSFHXpDo9ZB/wwD7jqrw9qes97Fttj7ma7D9fQ5Ye5syZIxcXF8tjy5Ytluce5xTCoKAghYeHa8eOHerbt69q1aql7t27P3SdUqVKKTw8XOHh4bp27ZoSEhLSbNu/f3/FxMRYHqdPp34rZHtzyuqo0sF+2rwj+dbKiYmJ2rTziMqX9E91nfIl/a3aS9LGbYdVvmQhe3b1ieGU1VFlgv20acdhy7LExERt3nEkzRpVKFnIqr0kbdh2yPJvUNDXU3k93azaxMbd0B9/nlD5NK4xyWyoq+1RU/u4nXBH4UfOKrRsoGWZyWRS1bKB2vHnyYeuG38rQecvx8oxi4Pqh5bUT1sOPNCmZd3yunQlTr/8dsjmfc+oeK+yj7TqmrQP8E91nfIl/bV5Z4q6bqeu92PfanvM1WRPRCBr0KCBJRSFh4erXLlykqSiRYvq4MGDqa5zb3nRokUty5ycnBQYGKiQkBCNHDlSWbJksTqiVaRIEUnS4cPJvxjOzs4KDAxUYGDym3BanJ2d5ebmZvUwSteW1TRz6VbNW7FNh49f0PujFuj6jXi1rFdJktRl0Ex9PG6Zpf1bzcO07rcD+mbOOh05cUEjJ69S+MFT6tg0+W5gV2Kuad+RMzp8/IIk6a+TF7XvyBldvPx0nF7TtWV1zVyyVfNW/K7Dxy/ovZHzde1GvFrVT6pp50EzNeSbpZb2lprOvlfTlQo/eEqdXguVlPRhrnOLavrsu9VatWmv/jx6Vl0Gz5JPHnfVDS2dah8yI+pqe9TUPsZ/v0Vt61dQ89plVbSgt77o9YpyZnfSnJU7JUkTBjbTR2/VtrQvW9xP9aqGqGB+D1Uu5a9Fn78hBweTxs7daLVdk8mkVi+X0/er/9CdO4npOSTD8V5lH11bVNOspVs1b2VSXXuNWqDrN++r6+AUdW2WVNdxd+s6asrdur6Wdl2P3qvrU3SKLftW22OuJnnsPwxtBFdXV7m6Pngosnnz5howYID27NljdR1ZYmKixowZo+LFiz9wfdn9Bg4cqOrVq6tLly7Knz+/XnrpJXl4eGjUqFFavHixXcaSXl6tWVaRV+I0YvJKRUReVUhRXy0c29VymPzMxStycEi+Hq5iqQBNHtpewyeu0LDxKxTg56XZn3ZS8cLJfy/npy379PbHcyw/dxwwXZLUp2Md9Xvz5fQZmIFefamsLkfHafiklXdPJ/DVoq+6Jdf0QpQc7rvGsGLpAE0Z1l6fTFihoeOXJ9X0szdVPDC5pu+0raHrN+L17vB5iom7oUqlC2vRV12VzTlruo/PKNTV9qipfSxev0d5cuXUBx1fkreHq/YdPacm73+rS1fiJEkF8uZSYmLyWRvOTlk1oFMt+ef30LUbt7Tm90PqPHS+YuNuWm03rFyg/Hxya/ZTcnfF+/FeZR+v1EzaB4y8r64Lvkyu69kUda1wt66fTFyhYROS6jprdCcVS1HX7kPvq+vA6ZKS6tq309NRV/attsdcTWIyZ+DbBrZv317R0dFasmRJqs/fvHlTYWFhOnfunD7//HNVrFhRFy9e1PDhw7VmzRqtXbtWlSolJezBgwdryZIlCg8Pt9pGxYoVVb58eX3zzTeSpMWLF6tZs2aqWbOmevTooSJFiiguLk6rV69W3759tWzZMtWvX/+R+h8bGyt3d3dduBxt6NGyzOZhN1YBkPnlfiHtGy/h34vaMsroLmQ6GfcT1pPt/g/osI37v0yCbcTGxiqfVy7FxMT8Yw54Ik5ZTEu2bNm0fv16tW3bVh988IECAwNVu3ZtZcmSRb///rsljD3Mu+++q6lTp1qu93rllVe0detW5ciRQ23btlVQUJCqV6+u9evXP/SGHgAAAADwuP7VEbItW7Zo0qRJ+vvvv7Vo0SL5+vpq1qxZKlSokF544QV79POJxBEy++AIGfB04wiZfXCEzPY4QmYfHCGzPY6Q2Z5dj5D98MMPqlWrlrJnz67du3db/u5WTEyMhg8f/u96DAAAAABPoccOZMOGDdPEiRM1ZcoUZc2afMHh888/r127dtm0cwAAAACQmT12IDt8+LCqVq36wHJ3d3dFR0fbok8AAAAA8FR47EDm4+Ojo0ePPrD8f//7nwICAmzSKQAAAAB4Gjx2IOvUqZPeeecdbdu2TSaTSefOndOcOXPUq1cvdenSxR59BAAAAIBM6bH/MHS/fv2UmJioF198UdevX1fVqlXl7OysXr16qXv37vboIwAAAABkSo8dyEwmkwYMGKDevXvr6NGjiouLU/HixeXi4mKP/gEAAABApvXYgeweJycnFS9e3JZ9AQAAAICnymMHsmrVqj30D/OuX7/+P3UIAAAAAJ4Wjx3IypQpY/Xz7du3FR4erv3796tdu3a26hcAAAAAZHqPHcjGjBmT6vLBgwcrLi7uP3cIAAAAAJ4Wj33b+7S0bt1a3333na02BwAAAACZns0C2W+//aZs2bLZanMAAAAAkOk99imLr776qtXPZrNZ58+f186dO/Xhhx/arGMAAAAAkNk9diBzd3e3+tnBwUFBQUH6+OOP9dJLL9msYwAAAACQ2T1WILtz545ef/11lSxZUrlz57ZXnwAAAADgqfBY15BlyZJFL730kqKjo+3UHQAAAAB4ejz2TT1CQkJ07Ngxe/QFAAAAAJ4qjx3Ihg0bpl69emnFihU6f/68YmNjrR4AAAAAgEfzyNeQffzxx3r//ff18ssvS5IaNGggk8lked5sNstkMunOnTu27yUAAAAAZEKPHMiGDBmizp07a8OGDfbsDwAAAAA8NR45kJnNZklSaGio3ToDAAAAAE+Tx7qG7P5TFAEAAAAA/81j/R2yokWL/mMoi4qK+k8dAgAAAICnxWMFsiFDhsjd3d1efQEAAACAp8pjBbLmzZvL29vbXn0BAAAAgKfKI19DxvVjAAAAAGBbjxzI7t1lEQAAAABgG498ymJiYqI9+wEAAAAAT53Huu09AAAAAMB2CGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGOSR/w4Z/j2TySSTyWR0NzKNhDv8TTx7cMzC9zN4MkRtGWV0FzIlj9ojje5CphP5Uz+juwA8ktt8trK5x6kpn8AAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDOBrdAdjPlAWb9PXsdYqIjFVIEV+N6v2aypbwT7P9krW7NHziSp06H6kAPy8N7t5ILz1fwvK82WzWiEkrNXPJVsXE3VDFUgH6vF8zFX7GOx1GkzF8u2izxs1er4ioWJUI9NWI95vo2RIF02y/dN1ujZy8UqfPRynAz0sfdmugms8l1fR2wh2NmLhCa387oJNnI+Xqkk2h5YP0YdcG8vFyT68hZQjMVdujpvYxdeFmS11LFPHVqF5N/qGuuzVi0gqdursPGPx2Q9VMWdfJqzTLUtdC+qzv01XXjvWfVfcmFeXt4aL9xyLUd/wv2nX4fKptHbM46N3mldWiRknly+Oqo2ciNfjbjVq385ilTYd6/6cOdZ+VX96k/eihk5f16Zz/ae19bZ4GUxdu1jdzkufqyPcfPleXrtut4ZNWWN6vBnWznqvLN4Rr+o+/as+hU7oSe10bZ/VVyaIF0mEkGQv7Vtv77octGj9nvS5Fxap4oK8+ea+xni2e9merZet3a/TkVTp9IUqFCnhpYNf6qvFcck0/nfqTlq7dpbMR0XLKmkWlgvzU/626evYh/05GyzBHyNq3by+TyWR5eHp6qnbt2tq7d+8/rvvnn3+qadOm8vLykrOzs4oWLaqPPvpI169ft2rn7+9v2X6OHDlUsmRJTZ069YHtmc1mTZkyRZUrV5abm5tcXFxUokQJvfPOOzp69KjNxmxPP/7yhwZ+uVh9O9bRxll9FVLEV427j9OlqKuptt+255g6Dpyu1g0ra9PsfqobWlqte03WgaPnLG3GzlyrSfM36Yv+zbVmWi/lyO6kxt3H6Wb87fQalqEWr9mlj8YuVq+OtbVuRm+VKOKrpj3Hp1nT7XuP6a2PZqhV/cpaP6OP6lQtpXZ9purg30k1vXHzlvYePqP3Xq+ldTN6a/rIN3T0ZIRa956cnsMyHHPV9qipffy4JqmufTrW0YaZfRRSxFdNeqS9D9i295g6fThdrRpU1sZZffVyaCm17j1FB/5OrutXM9dq8vxN+rxfM6357n3lyO6sJj3GPzV1fSW0mIa9+aJGzfmfwrp9p/3HLuqHT5opj3uOVNsPbF9V7V/+P/Udv0aVOk3RtJW7NeujV1WycF5Lm3OXrmrIdxtV7e1pqt59urbsOaE5g5souGCe9BqW4Rav+UMfjl2s3m/U0foZfRQS6KvX3nn4+1WnD6erdf3K2jCzr16uWkpt+kyxvF9J0vUbt1SpdIAGvd0wvYaR4bBvtb0la3dp8FeL9X6HWvplWm+VCMyvFu9OSLOmO/YdV5dBM9WifiWtmd5bdaqW1Ov9vrWaq4Wf8dLw95to46y+WjrhHfnl81CznhN0+Upceg3rsWWYQCZJtWvX1vnz53X+/HmtW7dOjo6Oqlev3kPX+f3331WxYkXdunVLK1eu1JEjR/TJJ59o+vTpqlmzpm7dumXV/uOPP9b58+e1f/9+tW7dWp06ddJPP/1ked5sNqtly5bq0aOHXn75Zf3yyy86cOCAvv32W2XLlk3Dhg2zy9htbfzc9Wrb6Dm1alBZwQH59EX/5sqRzUmzl/2WavtJ32/Ui5WLqUebGgoq5KMBXeqpdLCfpizcJCmpLhPnbVCvDrX0cmgphRTx1YQhbXXhcoxWbtqTnkMzzMR5G9S64XNqWa+Sggrl02d9myp7NifNXfF7qu0nz9+k6pWK6e3WL6poIR/1f6uuSgUV0LeLtkiS3Fyya9HX3dSoxrMKLJhX5UIKaWSvJtpz6LTOXIhKz6EZirlqe9TUPsbP3aC2jSqrVf1KSXXt10w5sjlpzvKH1LXSfXXtXE+lgv00dcFmSXfr+v1GvX+3riWK+GrC4DZ36/rPX0ZmBl1fraCZq/do7i/7dPhUpN77arWuxyeoda1SqbZv+mKIxny/VWt2/K2TF6L13YrdWrPjb73duIKlzeptR7Vmx986du6K/j4bpWHTN+vazVsqF5w/vYZluPHzNqhNw+S5+nm/Zsr+sLk6P2mudr87Vz/oXE+lgvw0deFmS5tmL1dQ7451FFo+KL2GkeGwb7W9Sd9vVKsGz6lFvUoKKuSj0X2aKruzk75P47PVlAWbVK1isLq1elFF/X3U9826KhlUQNN+2GJp8+pL5VS1fJAK+uZRcEA+Denxiq5eu6mDf59Nr2E9tgwVyJydneXj4yMfHx+VKVNG/fr10+nTp3Xp0qVU25vNZr3xxhsqVqyYfvzxR1WoUEEFCxbUa6+9puXLl+u3337TmDFjrNZxdXWVj4+PAgIC1LdvX3l4eGjNmjWW5+fPn6/vv/9e8+fP14cffqhKlSrpmWeeUaVKlTRq1ChNmzbNrjWwhVu3ExR+6LTCKiTvNB0cHBRaIUg79h1PdZ3t+44rrHyw1bLqlYppx74TkqSTZyN1MTJWYRWS27i7ZFfZEv7asfeEzceQ0dy6naA9h09bvRE5ODioavkg7Uyjpjv3n1DV8kWtllWrVCzN9pIUG3dTJpNJ7q7ZbdPxDI65anvU1D5u3U7QnkMP7gNCywdZ6pTSjn0nFFrB+sNr9UrBln+Hk+fu1TW5jdu9uj5kP5FZZHV0UJkiPtq4K3msZrO0afcJlS/um+o6zlkddfNWgtWym/EJqlQi9VPnHBxMejW0mHI4Z9WOgxn3w5gtWeZqyn3AP83V8mnPVbBvtYdbtxO09/BpVS2X/FnJwcFBVcoX1c79J1Jd54/9x1U1xVwNqxicZvtbtxM0a+lWublkV/HA1PcrGUGGvYYsLi5Os2fPVmBgoDw9PVNtEx4ergMHDmju3LlycLDOlqVLl1aNGjU0b9489e3b94F1ExMTtXjxYl25ckVOTk6W5fPmzVNQUJAaNGiQ6muaTKY0+xwfH6/4+HjLz7GxsQ8do71ERsfpzp1EeXm4Wi338nDTXycuprpORGSsvDxTtndVRGTSGC7e/W/KNt6eyW0ys6joa6nW1Du3q44+pKbeHm5Wy7xyuyoiMvXD8Dfjb+vjcUv1as1n5Zrz6QhkzFXbo6b2EWnZB6T4nfZw1ZGTD9sHpKiZh6si7p6KY6nrA/9WT0ddPd1yyDGLgy5FW19ecOnKNRXxS/19f/0fx9S1cQVt3Xdax89fUej/+ave80HK4mD93lzc30s/f9lW2Zwcde3GLbX5+EcdPhVpt7FkJPfmasr3H28PV/31kLma+jxM/f3qacS+1fbS+mzl5eGqoycjUl0nIvKqvHKnaJ/7wXr98ut+df5ohm7cvK28nm6a/2UXeeZyse0AbChDHSFbsWKFXFxc5OLiIldXVy1btkzz589/IGzdc+TIEUlSsWLFUn2+WLFiljb39O3bVy4uLnJ2dlaTJk2UO3dudezY0WqbQUHWybtnz56WfhUokPYFrCNGjJC7u7vl4efn90jjBm4n3FHHAdNkNkuf9m1qdHcAIEPqN2GNjp29ou1T31TEyr4a3fUlzf1lrxLNZqt2f52JVNWu36lGjxn6bsUuje9VT0HPpB7yAGQ+zz9bROtm9NGKST1VrVKw3vxweprXpWUEGSqQVatWTeHh4QoPD9f27dtVq1Yt1alTRydPnlSdOnUsoahEiRJW65lT7Igfpnfv3goPD9f69etVsWJFjRkzRoGBgQ9dZ8CAAQoPD9dHH32kuLi0Lwjs37+/YmJiLI/Tp08/cr9syTOXi7JkcXhg4l2KipW3p1uq63h7uulSZMr2Vy3t8979b8o2EZFX09xmZuKRK2eqNY24clXeKb7Zusfb000RUdbf2FxKpf29MHbmQpQWfd3tqTk6JjFX7YGa2oenZR+Q4nc66qqlPikl7QNS1CzqquWomaWuD/xbPR11jYy9roQ7ifLKZX0DD6/cORWRxsX3kTE31HrID/Jt+JlKtRmnCh0n69rNWzpxIdqq3e2ERB0/d0V7jl7Qx9M2af/xi+rcqLy9hpKh3JurKd9/kubeQ/YBqc7D1N/fnkbsW20vrc9Wl+7bT6bk7emqS1dStL/yYL1yZndWoQJeKhvirzEftJRjFgfNS+O6tIwgQwWynDlzKjAwUIGBgSpfvrymTp2qa9euacqUKZo6daolrK1atUqSVLRo0jmnBw8eTHV7Bw8etLS5J0+ePAoMDFSVKlW0cOFC9ejRQwcOHLA8X6RIER0+fNhqHS8vLwUGBsrb++G3IHV2dpabm5vVwwhOWR1VJthPm3YkjyMxMVGbdxxR+ZKFUl2nQslCVu0lacO2Qypf0l+SVNDXU3k93azaxMbd0B9/nlD5Uv42H0NG45TVUaWD/LR5R/IR18TERG3ZcVjl0qhpuRB/bdlhfYR20/ZDVu3vhbFjpy9p0dfd5OGe0z4DyKCYq7ZHTe3DKaujSgc/uA/YtPOIpU4plS/pb9VekjZuO2z5dyiY/yF1TePfKjO5nZCo8L8uKPT//C3LTCapapmC2nHg4dd7xd++o/ORcXLM4qD6LwTrp9/+emh7B5NJTlmz2KLbGV5aczVpH+Cf6jrlS/pr884Uc3X74adiHj4q9q2255TVUaWC/LTlD+u5+r+dR1QuxD/VdcqGFNKWFHN18/bDabZP3q5Z8SmuP81IMlQgS8lkMsnBwUE3btyQr6+vJawVLJj0twnKlCmj4OBgjRkzRomJiVbr7tmzR2vXrlWLFi3S3L6fn5+aNWum/v37W5a1aNFChw8f1tKlS+0zqHTStWV1zVyyVfNW/K7Dxy/ovZHzde1GvFrVryRJ6jxopoZ8kzzGt5qHad1vB/TN7HU6cuKCRk5eqfCDp9TptVBJSf8WnVtU02ffrdaqTXv159Gz6jJ4lnzyuKtuaGlDxpjeOreoptnLtur7ldt05PgF9R69QNdv3lKLuhUlSd2GzNLQ8css7d9sFqr1vx/U+Dnr9deJixo9ZZXCD57WG02qSEoKYx36f6vwg6c0YUhb3Uk062JkrC5GxurW7Yy707A15qrtUVP76NqymmYu3ap5K7bp8PELen/UAl2/Ea+W9ZLq2mXQTH08LnkfYKnrnHt1XaXwg6fUsWlVSXfr2jxMn3/3s37avE8Hjp5TV0tdU7/LYGYz/sftalunjJrXKKmifp76ontt5cyWVXN+SbrL5ITe9fTR66GW9mWD8qve80VV0CeXKocU0KJPmsnBJI1dkPzN90evh+q5ED/55XVXcX8vffR6qF4oVVALN/yZ7uMzStcW1TRr6VbNW5k0V3uNWqDrN++bq4NTzNVmSXN13N25OmrK3bn6WlVLmysx17TvyBkdPn5BknT05EXtO3LGch3U04B9q+291TxMc5b9pvmrtuvIiQvq++lCXb95S83rJX22evvj2fpkwnJL+05NQ7Xh94OaMDfps9WnU3/SnkOn9XrjpM9W127Ea/jE5fpj/wmdPh+lPYdOq+cnc3XhcozqVy9jxBAfSYa6qUd8fLwuXEj6Rb9y5Yq++eYbxcXFqX79+qm2N5lM+vbbb1WzZk01btxY/fv3l4+Pj7Zt26b3339flStXVs+ePR/6mu+8845CQkK0c+dOlStXTs2bN9ePP/6o5s2bq3///qpVq5by5s2rkydPav78+cqS5cn4hu3Vl8rqcnSchk9aqYjIqypZ1FeLvupmOaR75kKUHO67QUnF0gGaMqy9PpmwQkPHL1eAn5dmf/amigcm3yb4nbY1dP1GvN4dPk8xcTdUqXRhLfqqq7I5Z0338RnhlZrPKjI6TqOmrLr7ByELaP6YLvfV9IrVTV8qlArQxI/bacSklfpk4nIF+HlrxuiOKlY4qabnI6K1est+SVK1NqOsXmvJuO56vmyRdBqZsZirtkdN7ePVmmUVeSVOIyYn1TWkqK8Wju2aXNeLV+Rw380lKpYK0OSh7TV84goNG78iqa6fdlLxwsl17dG2hq7dvHVfXQO0cOzTU9fFmw4qj3sOfdC2irxz59S+YxFqMmCB5UYfBbzclJiYfFmCs1MWDWgXKv98uXTtxi2t2fG3Oo9erthryTfUypMrpyb0rqe8Hi6KvR6vP49HqPGA77Vx14n0Hp5hXqmZtA8Yed9cXfBl8lw9m2KuVrg7Vz+ZuELDJiTN1VmjO1neryTppy371H3oHMvPHQdOlyT16VhHfTu9nD4DMxj7VttrVCPps9XoKat0KSpWJYoU0LwvOltuoJRyrpYvWUjjh7TVqMmrNGLSChUq4KVpI9+wzNUsDg46ejJCC1Z9p6iYOOV2z6kywc9oyfgeCg7IZ8gYH4XJ/DgXYNlR+/btNWPGDMvPrq6uCg4OVt++fdW4ceOHrrtv3z4NGTJEGzZs0NWrV/XMM8+oRYsW6t+/v3LkSD433d/fXz179nwgpNWuXVsODg6WUyETExM1ZcoUTZs2Tfv379ft27dVoEABvfjii3r33XfTvIlISrGxsXJ3d9fFyBjDTl/MjBLuJP5zIzw2xywZ+oA5YJFB3rYyHY/aI43uQqYT+VM/o7uQKTk4pH3Ha/w78bfvGN2FTCc2NlbP+HgoJuafc0CGCWSZEYHMPghk9kEgw5OCty37IJDZHoHMPghktkcgs73HCWR8AgMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMuD/27vz+Bru/Y/j7xNZhCwkQog0loiExlJ79RYtDWppS+3FTelVamspLi1qbW9brZbys1bt2qqt1aK2VouqoBJEiTWWRiRiSZD5/aFOHVmInmPSeD0fj/PwODOfmTPfj8nkvM/MmQAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEmczd4AIKec8/E5AvAgSzfM3oK86dzqwWZvQp7jU3+Y2ZuQJyVuHGP2JuQ5vLeyv5z0lO4DAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZDlYdMWb1SlFm/Kv24/Nez6P+3YG5dt/Vdrf1XN1qPkX7efHm03Rt/9uNdmvmEYGjtlpUIb/1fFH+uvZ3p+pN+PnnHgCHIfeuoY9NX+6KljzFiySVWfGa6Af/XXU5Hv6tc79HXZup2q3WaUAv7VX//qMFZrbuvryvVRat17kso1GqQitXprz4HjDtz63Gn6kk2q3HK4ij/WXw3//e5d7Ks7Vev5USr+WH/VbZ+xpyvWR+m53pNUtuEg+dR8MHsqSd2eqaVdCwco/rsRWvNJDz0SWjLLWud8ThrYpYF+nf+q4r8boc0zXtGTNctlqCtexEtThz6v35cP1cnvRujHWb1VpXyAI4eR63BstT+OqwSyPOvL73Zo2AdLNahbE234bJAeLhegVr0n6ey5C5nWb911SN2GzVanlnW0ce5gPV2vsjoN+D9FHzxprflwzlpNXbRR7w9ppzWzBqiAu6ta9Z6kK6lX79ewTEVPHYO+2h89dYyla3bojQ+XauCLTfT9p6+rYnCAnu87Ocu+btt9SC+9MVsdm9fR+jmD1PTxSur8+jTF/P5XXy9dTlOtymX05ist79cwcpUv19zYV1/v1kTr57yuh8sFqHWfrHu6dfchdX9jtjq2qKMNnw1S03qV1GngNEXf1tPalcto+APaU0l6tkG4Rvdqqrc//V71u0/Sb7+f0hfvdlWRQgUzrR/WrZG6Nq+pQR+uVO0uH2rW8m36bHRHhZcrbq3x9siv1R+/pKvXr+v51z9V7c4fatikb3T+wuX7NSzTcWy1P46rNxDIsnDs2DFFRkaqRIkScnV1VVBQkPr27auEhASzN+2uTJ7/vTo/86g6tqij0DLF9f6QdiqQ31Vzl/+Uaf3UhRv0ZJ0w9XmhocqX9tfQl5upcmigpi3ZKOnGJzhTFqzXgMgINa1XSQ+XC9AnIzvr1B9JWrVx1/0cmmnoqWPQV/ujp47xyYL1eqFlHXVoXlvlyxTXe4Pbyj2/q+avyKKvizboidph6v1CQ4WU9teQHs1UqXygpi/ZZK1p07SmBnZrono1yt+vYeQqk+evV+dn6qhj89o39tXBbVUgv6vmZdXThRv0ZO1b9tUezVQpNFDTF//V07ZNa+r1bk1Uv+aD2VNJ6tmmruas/EXzv/lV+4+c1avvLdOlK1fVqWm1TOvbPFVFE+Zu0JqtB3QkPlEzl23Tmp/365U2j1lr+nV4XCfOJumV8V/q133HdfRUotb/clBxJ8/dr2GZjmOr/XFcvYFAlolDhw6pevXqio2N1YIFC3Tw4EFNmTJF69atU506dXTuXO4++KRdvaaofcdsfhk5OTmpXs3y2r7ncKbLbNtzWPVrhNpMe6J2mLbviZMkHTmRoNMJyapf868abw93VatYStt3x9l9DLkNPXUM+mp/9NQx0q5e0659x1Tv9r7WKG/t0+1+2ROX4Q1Bg9qh+iWL/4cHjbWnNe6+p9v3xNn8H0jSE7VDs9y3H0QuzvlUJaSENuw4aJ1mGIY27jioGhUfynQZNxdnXUm7ZjPtSuo11Q4Psj5vXDdMO/ed0KyR7XTgqyHaOL2XOjer7phB5EIcW+2P4+pfCGSZ6NWrl1xdXfXdd9+pXr16euihh9SkSROtXbtWJ06c0NChQ83exGwlnE/R9evp8vPxtJnu5+OlMwnJmS5zJiFZfr6313ta60//+e/tNUV9PbNcZ15CTx2DvtofPXWMhPMX/+yrl810Px9PnTmXTV9v+38o6uOpMwmZX4rzoMmup6ez2VeLZtbTLC5vehD5eheQs3M+nU1MsZl+NjFFRX08Ml3m++2x6tmmrsoE+Mpisah+9bJq9ngFFbvlZ75U8cKKbFlTh44nqNXA2Zq5bJvG92mmdhFVHTqe3IJjq/1xXP0Lgew2586d07fffquePXvK3d3dZp6/v786duyoRYsWyTCMDMumpqYqOTnZ5gEAAJCbDZ64UoeOJ2jbZ/10Zu1IvdO3ueZ/86vSb3mv4+Rk0e7Ykxo1bY32xMbr0xXbNWfldv27ZU0TtxzIGwhkt4mNjZVhGAoLC8t0flhYmBITE3X27NkM88aNGydvb2/rIzAw0NGbmynfQh7Kl88pwxciz55LVlFfr0yXKerrpbMJt9dfsNYX+/Pf22vOJFzIcp15CT11DPpqf/TUMXwLFfyzr7YftJ09d0FFfbLp623/D2fOXVDR2z4Nf1Bl19Ni2eyrt58NO3PuQoazZg+yhKRLunbtuvwK254N8yvsoTPnUrJcptOweQpoPFKV2r6rmi98oIuX02y+H3Y64YL2xdm+9zlw5KxKFi1k9zHkRhxb7Y/j6l8IZFnI7AzYnQwZMkRJSUnWx7FjxxywZXfm6uKsKqGB2rh9v3Vaenq6Nm0/oBrhpTNdpmZ4aZt6SVq/dZ9qhJeSJAUF+KqYr5dNTXLKZe3YG6calUrZfQy5DT11DPpqf/TUMVxdnFU5NFCbth+wTvurr6UyXaZ6eClt+uWAzbSN2/arehb/Dw+arHq68Zese1ojvJRNvSRt2Lo/y337QXT12nVFHTipetXKWqdZLBY9/khZbd97NNtlU9OuKf6PZDnnc1Lzxyvqmx9jrPO2/nZU5R4qYlNftmQRHT+daN8B5FIcW+2P4+pfCGS3CQ4OlsViUUxMTKbzY2JiVLhwYfn5+WWY5+bmJi8vL5uHWXp2eEJzvtqiBSt/1v7Dp/Tq+EW6eDlVHZvXliT1GD5HIz9eZq3/T7v6WvdTtD6eu04H4k5p/P+tUlTMUXV/vp6kGwfzHu0b6N2Zq/X1xt3ae/CEXh7xmfyLeOvpepVNGeP9Rk8dg77aHz11jJfbN9Bny7Zo4aqtOnD4lAa8vViXrqSqfbMbfe05Yo5GTVpurf9P2/r6/qdoTZq3TrFxp/T2tK8VFXNU3Z5/3FqTmHRRew4c1/7DpyRJB4+c1p4Dx7P8DlVe07NDA81ZtkULVm7V/sOn9Nrbi3Xpcqo6/NnTl4fP0Vu39vTmvjrv5r76Z0/bZN3T2Js9/ePB6KkkTV78ozo/XV3tIqoqJMhP77/aQgXdXTXvmx2SpE/+21pvdn/KWl8trKSa/auCgooXVp1KQfr8f13l5GTRhws2/7XOJT+qeoVAvdqpnkoH+Kh1w0rq0ryGpi/det/HZxaOrfbHcfUGZ7M3ILfx9fVVo0aNNHnyZPXv39/me2SnTp3SvHnz1LlzZ1ksFhO38s6ee6qa/jiforFTV+lMwgWFhwTo84m9rKfAj586J6dbxlCrchlNG91VYz5ZqVGTV6hMoJ/mvvuSKgSXsNb07dxQly6nqv/YBUpKuazalcvq84k9ld/N5b6Pzwz01DHoq/3RU8d4tlE1JZxP0fj/u9HXh0MCtPiDnn/19XSinJz+6mvNSmU0dVRXjZ2yUmM+WakygX6a8053hZX9q6+rN+9R71HzrM+7D5stSRrYrYkGdW96fwZmoucaVVNCYorG3dLTJR9m3dNalcro//7s6ejJN3o693/dVeGWnn6zeY9eeeuvnnYbOluS9Hq3Jhr8Ut7vqSQtXb9HRQoV1H8jn1RRH0/tORiv1gNn62ziRUlSyaLeSk//60ogN1dnDe3WSKWKF9bFy2las/WAeoxZouSUK9aanftO6IVh8/TmS09pYOcGOnIqUf/9eJWWrH0wbs8ucWx1BI6rN1iMe7k2L4+LjY3Vo48+qrCwMI0ePVqlS5fW3r17NXDgQKWmpurnn3+Wj4/PHdeTnJwsb29vnU5IMvVsGQDkJdfT+bXlCE65+3PGfySf+sPM3oQ8KXHjGLM3Ic/huGp/ycnJKuFXSElJd84BXLKYiXLlyumXX35RmTJl1KZNG5UtW1YvvfSSGjRooJ9++umuwhgAAAAA3AmXLGYhKChIs2fPNnszAAAAAORhnCEDAAAAAJMQyAAAAADAJAQyAAAAADAJgQwAAAAATEIgAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAExCIAMAAAAAkxDIAAAAAMAkBDIAAAAAMAmBDAAAAABMQiADAAAAAJMQyAAAAADAJAQyAAAAADAJgQwAAAAATEIgAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAExCIAMAAAAAkxDIAAAAAMAkBDIAAAAAMAmBDAAAAABMQiADAAAAAJMQyAAAAADAJAQyAAAAADAJgQwAAAAATEIgAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAExCIAMAAAAAkxDIAAAAAMAkBDIAAAAAMAmBDAAAAABM4mz2BgAAkBP5nCxmbwJwVxI3jjF7E/KkwjV7m70JeU7ito/M3oQ8Jye/qzhDBgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRAlodNW7xRlVq8Kf+6/dSw6/+0Y29ctvVfrf1VNVuPkn/dfnq03Rh99+Nem/mGYWjslJUKbfxfFX+sv57p+ZF+P3rGgSPIfeipY9BX+6OnjkFf7Y+eOgZ9tb9urf+lXV+NUPzm97Vm5mt6pEJQlrXO+Zw08MXG+vXLNxW/+X1tnjdYT9YOs6nxKOCmsf2f0+5lI3Vy03v6dnp/VQ17yNHDyHXYVwlkedaX3+3QsA+WalC3Jtrw2SA9XC5ArXpP0tlzFzKt37rrkLoNm61OLeto49zBerpeZXUa8H+KPnjSWvPhnLWaumij3h/STmtmDVABd1e16j1JV1Kv3q9hmYqeOgZ9tT966hj01f7oqWPQV/t7tuEjGt3vWb09/RvV7/yOfos9oS8m9lSRwh6Z1g97uZm6PltXg979XLXbjtGsL3/QZ+90U3hISWvNh0M7qH6tUPUYMUd1O4zT91v36atJr6i4n/f9Gpbp2FdvyLWBrGvXrrJYLNaHr6+vGjdurN27d2e5TFxcnCwWi6KiorKs2bJli5o2barChQsrf/78Cg8P1/vvv6/r169nqF2/fr2aNm0qX19fFShQQBUqVNBrr72mEydO2GOIDjV5/vfq/Myj6tiijkLLFNf7Q9qpQH5XzV3+U6b1Uxdu0JN1wtTnhYYqX9pfQ19upsqhgZq2ZKOkG582TFmwXgMiI9S0XiU9XC5An4zsrFN/JGnVxl33c2imoaeOQV/tj546Bn21P3rqGPTV/np2aKA5X/2k+Su3av/hU3p1/CJdupKmTs3rZFrfpklNTZj9ndZsidaRkwma+cUPWrMlWq90fEKSlN/NRS0aVNaIj5Zpy87fdfj4H3p72jc6dOysIls9dj+HZir21RtybSCTpMaNGys+Pl7x8fFat26dnJ2d1axZs3te39KlS1WvXj2VLFlS69ev1759+9S3b1+NHj1a7dq1k2EY1tqpU6eqYcOG8vf31xdffKHo6GhNmTJFSUlJeu+99+wxPIdJu3pNUfuOqX7N8tZpTk5OqlezvLbvOZzpMtv2HFb9GqE2056oHabte+IkSUdOJOh0QrLq1/yrxtvDXdUqltL23XF2H0NuQ08dg77aHz11DPpqf/TUMeir/bk451OV0EBt2L7fOs0wDG3cvl81wktluoybq7OupNmekbmSelW1K5eRdOOSRmfnfFnUlLXvAHIp9tW/OJu9Adlxc3OTv7+/JMnf31+DBw/Wv/71L509e1Z+fn45WtfFixfVvXt3tWjRQv/3f/9nnd6tWzcVK1ZMLVq00OLFi9W2bVsdP35cffr0UZ8+fTRhwgRrbalSpfT444/r/Pnzmb5GamqqUlNTrc+Tk5NztI32knA+Rdevp8vPx9Nmup+Pl2LjTme6zJmEZPn53l7vqTMJN8Zw+s9/b68p6vtXTV5GTx2DvtofPXUM+mp/9NQx6Kv9+RYqKGfnfDp7znasZ89dULmgYpku8/3PMerZ4Qnr2a96NULUrEFl5XOySJJSLqVq2+5DGhjZWAcOn9KZcxfU+qlqqhFeWoeOn3X4mHID9tW/5OozZLdKSUnR3LlzFRwcLF9f3xwv/9133ykhIUEDBgzIMK958+YKCQnRggULJElLlixRWlqaXn/99UzXVahQoUynjxs3Tt7e3tZHYGBgjrcTAAAA/2yD3/tCh46d1bbFw3Tmxwl6Z+Dzmr/iZ6Wn/3U11n+GfyaLRYr5eoxO/zBBL7Wtry++22FTgwdDrg5kK1eulIeHhzw8POTp6anly5dr0aJFcnLK+WYfOHBAkhQWFpbp/NDQUGtNbGysvLy8VLx48Ry9xpAhQ5SUlGR9HDt2LMfbaQ++hTyUL59Thi9Enj2XrKK+XpkuU9TXS2cTbq+/YK0v9ue/t9ecSbiQ5TrzEnrqGPTV/uipY9BX+6OnjkFf7S/h/EVdu3Zdfj62Y731zEzGZVLUaeA0BdR7TZVaDlfN50fr4uVUxZ1MsNbEnfhDzXpMVMDjr+nh5m+q4b/flbNzPh05kZDpOvMa9tW/5OpA1qBBA0VFRSkqKkrbtm1TRESEmjRpoiNHjqhJkybWsFaxYsW7Xuet3xPLrsZiseR4e93c3OTl5WXzMIOri7OqhAZq4y3XOqenp2vT9gOqEV4602Vqhpe2qZek9Vv3Wa+NDgrwVTFfL5ua5JTL2rE3TjUqlbL7GHIbeuoY9NX+6Klj0Ff7o6eOQV/t7+q164rad0z1aoRYp1ksFj1ePcT63aWspKZdU/zZJDnnc1LzBlX0zcY9GWouXUnT6YRkeXu668naofp6U9Y3sMtL2Ff/kqu/Q1awYEEFBwdbn0+fPl3e3t6aNm2apk+frsuXL0uSXFxc7riukJAbP0QxMTF69NFHM8yPiYlRhQoVrLVJSUmKj4/P8Vmy3KJnhyfUc+Rnqhr2kB6pWEqfLFivi5dT1bF5bUlSj+FzVNzPW8NfaSlJ+k+7+mr2nw/08dx1euqxivryux2KijmqD/7bXtKNA0+P9g307szVKhPop6AAX42dskr+Rbz1dL3Kpo3zfqKnjkFf7Y+eOgZ9tT966hj01f4mz1+vycM7aWfMUf2694hebldfBd3dNG/lz5KkT0a8oPgz5/XW5BWSpGoVg1Tcr5D2HDiuEkULaVD3JnJysujDz9Za1/lE7VBZZFHs0TMqU7KI3urzjA7Enda8FT+bMkYzsK/ekKsD2e0sFoucnJx0+fJlBQQE5GjZp556Sj4+PnrvvfcyBLLly5crNjZWo0aNkiS1bt1agwcP1jvvvGNzU4+bzp8/n+X3yHKL556qpj/Op2js1FU6k3BB4SEB+nxiL+vp2uOnzsnplrOAtSqX0bTRXTXmk5UaNXmFygT6ae67L6lCcAlrTd/ODXXpcqr6j12gpJTLql25rD6f2FP53e4ciPMCeuoY9NX+6Klj0Ff7o6eOQV/tb+naX1WksIf++9LTKurrqT0HTqh138nWy+1KFits890vN1cXDe3xtEoFFNHFy6lasyVaPYbPUXLKZWuNl4e73uzZXCWKFlJi8iWt+H6XRn+yQteup9/38ZmFffUGi3E31/CZoGvXrjp9+rRmzZolSUpMTNTHH3+sTz75RN9//73q16+fYZm4uDiVLl1aCxcuVPny5W3mVaxYUcuWLVO7du0UGRmpV155RV5eXlq3bp0GDhyoJ598UosXL7Zeqjh58mS98sor+ve//63OnTurVKlSOn78uObMmSMPD4+7uvV9cnKyvL29dTohybTLFwEAAPKSwjV7m70JeU7ito/M3oQ8Jzk5WcV8vZWUdOcckKvPkK1evdp6yaCnp6dCQ0O1ZMmSTMPYrdq1a5dh2rFjx9S6dWutX79eY8aM0b/+9S9duXJF5cqV09ChQ9WvXz+b74317NlTISEhevfdd/Xss8/q8uXLKlWqlJo1a6ZXX33VruMEAAAA8GDKtWfI8gLOkAEAANgXZ8jsjzNk9peTM2S5+i6LAAAAAJCXEcgAAAAAwCQEMgAAAAAwCYEMAAAAAExCIAMAAAAAkxDIAAAAAMAkBDIAAAAAMAmBDAAAAABMQiADAAAAAJMQyAAAAADAJAQyAAAAADAJgQwAAAAATEIgAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAExCIAMAAAAAkxDIAAAAAMAkBDIAAAAAMAmBDAAAAABMQiADAAAAAJMQyAAAAADAJAQyAAAAADAJgQwAAAAATEIgAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAExCIAMAAAAAkxDIAAAAAMAkBDIAAAAAMAmBDAAAAABMQiADAAAAAJMQyAAAAADAJAQyAAAAADCJs9kbAAAAANytxG0fmb0JeU7hGq+YvQl5jnE97a5rOUMGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkOVh0xZvVKUWb8q/bj817Po/7dgbl239V2t/Vc3Wo+Rft58ebTdG3/2412a+YRgaO2WlQhv/V8Uf669nen6k34+eceAIch966hj01f7oqWPQV/ujp45BXx2DvtrXo1XLasH7/1H012OUuP1jNa1X6Y7L1H2knDZ8NkinfpygHV8OV/tmtTLUdHv+ce1aNlLxP0zQmlkD9EiFIEdsvt2YGsi6du0qi8Viffj6+qpx48bavXt3lsvExcXJYrEoX758OnHihM28+Ph4OTs7y2KxKC4uzqY+KirKWrd06VLVrl1b3t7e8vT0VMWKFdWvXz+bdaWlpemdd95R5cqVVaBAARUpUkR169bVrFmzdPXqVXu1wGG+/G6Hhn2wVIO6NdGGzwbp4XIBatV7ks6eu5Bp/dZdh9Rt2Gx1allHG+cO1tP1KqvTgP9T9MGT1poP56zV1EUb9f6Qdloza4AKuLuqVe9JupKa+/thD/TUMeir/dFTx6Cv9kdPHYO+OgZ9tb8C7m767cAJDXxn0V3VP1TCV4s+6KHNOw7o8Y7jNWXBek0c2kFP1A6z1jzb6BGN7ves3p7+jeq/8LZ+iz2hLz7qpSKFPRw1jL/N9DNkjRs3Vnx8vOLj47Vu3To5OzurWbNmd1wuICBAc+bMsZn26aefKiAgINvl1q1bp7Zt26pVq1batm2bduzYoTFjxtiErLS0NEVERGj8+PF66aWXtGXLFm3btk29evXSRx99pL1792bzCrnD5Pnfq/Mzj6pjizoKLVNc7w9ppwL5XTV3+U+Z1k9duEFP1glTnxcaqnxpfw19uZkqhwZq2pKNkm58gjNlwXoNiIxQ03qV9HC5AH0ysrNO/ZGkVRt33c+hmYaeOgZ9tT966hj01f7oqWPQV8egr/a3dku0xkxZqVUbsj4Zc6vI5x7T0ZMJeuODpToQd1rTlmzS8u+j9HKHBtaanh2e0Jyvtmj+ip+1//ApvTpuoS5dSVOnFnUcNYy/zfRA5ubmJn9/f/n7+6tKlSoaPHiwjh07prNnz2a7XJcuXTRr1iybabNmzVKXLl2yXW7FihWqW7euBg4cqPLlyyskJETPPPOMJk2aZK354IMPtGnTJq1bt069evVSlSpVVKZMGXXo0EFbt25VuXLl7n3A90Ha1WuK2ndM9WuWt05zcnJSvZrltX3P4UyX2bbnsOrXCLWZ9kTtMG3fEydJOnIiQacTklW/5l813h7uqlaxlLbvjrP7GHIbeuoY9NX+6Klj0Ff7o6eOQV8dg77mDjXCS2vDtv0209b9HKOa4aUlSS7O+VQlNNCmxjAMbdy2XzX+rMmNTA9kt0pJSdHcuXMVHBwsX1/fbGtbtGihxMRE/fDDD5KkH374QYmJiWrevHm2y/n7+2vv3r367bffsqyZN2+eGjZsqKpVq2aY5+LiooIFC2a6XGpqqpKTk20eZkg4n6Lr19Pl5+NpM93Px0tnEjLfpjMJyfLzvb3e01p/+s9/b68p6uuZ5TrzEnrqGPTV/uipY9BX+6OnjkFfHYO+5g5Ffb0yXCJ6NiFZXh7uyu/mIt9CHnJ2zpex5lyyivp63c9NzRHTA9nKlSvl4eEhDw8PeXp6avny5Vq0aJGcnLLfNBcXF3Xq1EkzZ86UJM2cOVOdOnWSi4tLtsv17t1bNWrUUHh4uEqVKqV27dpp5syZSk1NtdbExsYqNDQ0m7Vkbty4cfL29rY+AgMDc7wOAAAAAA8O0wNZgwYNFBUVpaioKG3btk0RERFq0qSJjhw5oiZNmljDWsWKFTMsGxkZqSVLlujUqVNasmSJIiMj7/h6BQsW1KpVq3Tw4EENGzZMHh4eeu2111SzZk1dunRJ0o1Tm/diyJAhSkpKsj6OHTt2T+v5u3wLeShfPqccfTpQ1NdLZxNur79grS/257+315xJuJCrP3GwF3rqGPTV/uipY9BX+6OnjkFfHYO+5g5nEpIznqX09VJyymVdSb2qhPMpunbteo7OZOYGpgeyggULKjg4WMHBwapRo4amT5+uixcvatq0aZo+fbo1rH399dcZlg0PD1doaKjat2+vsLAwPfzww3f9umXLllW3bt00ffp0/frrr4qOjtaiRTfu8BISEqJ9+/bleCxubm7y8vKyeZjB1cVZVUIDtXH7X9fPpqena9P2A1leP1szvLRNvSSt37pPNcJLSZKCAnxVzNfLpiY55bJ27I1TjUql7D6G3IaeOgZ9tT966hj01f7oqWPQV8egr7nD9j2HVa9GeZtpDWqGatuf3+O7eu26ovYds6mxWCx6vEZIlt/1yw1MD2S3s1gscnJy0uXLlxUQEGANa0FBmf/9gMjISG3YsOGuzo5lpVSpUipQoIAuXrwoSerQoYPWrl2rnTt3Zqi9evWqtS43u3mHmQUr/7zDzPhFung5VR2b15Yk9Rg+RyM/Xmat/0+7+lr3U7Q+nrtOB+JOafz/rVJUzFF1f76epBv/Lz3aN9C7M1fr6427tffgCb084jP5F/HW0/UqmzLG+42eOgZ9tT966hj01f7oqWPQV8egr/ZX0N1VD4cE6OGQG3dJDyrhq4dDAlSyWGFJ0pu9WuiTES9Y62d++YOCAnw1sndLlQsqphdb/0vPNKyqT+avt9bcvBtmu6drKaRUMb0/uK0Kurtp3oqf7+/gcsDZ7A1ITU3VqVOnJEmJiYn6+OOPlZKScsebc9zUvXt3Pf/88ypUqNBd1Y8YMUKXLl1S06ZNFRQUpPPnz2vixIm6evWqGjVqJEnq16+fVq1apSeffFKjRo3SY489Jk9PT/3yyy96++23NWPGDFWpUuVehnvfPPdUNf1xPkVjp67SmYQLCg8J0OcTe1lPgR8/dU5OFou1vlblMpo2uqvGfLJSoyavUJlAP8199yVVCC5hrenbuaEuXU5V/7ELlJRyWbUrl9XnE3sqv1v239vLK+ipY9BX+6OnjkFf7Y+eOgZ9dQz6an9VwoK0cmpf6/Oxr7aSJM1f+bN6jZyrYkW8VNLfxzr/6MkEte03RWNffU7/aVdfJ8+cV58x8/X9zzHWmqVrflWRQh7673+eVlFfT+05cEKt+2T99+JyA4txr1+YsoOuXbvq008/tT739PRUaGioBg0apFatWmW6TFxcnEqXLq2dO3dmGoqioqJUtWpVHT58WKVKlcpQv379ek2aNEnbtm3T6dOnVbhwYVWtWlVDhw7VY489Zl1PamqqJkyYoPnz5ys2NlYFChRQWFiYunfvro4dO8rZ+c5ZNjk5Wd7e3jqdkGTa5YsAAABAdgrXeMXsTchzjOtpSt0zTUlJd84BpgayvI5ABgAAgNyOQGZ/OQlkue47ZAAAAADwoCCQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEmezNyAvMwxDknQhOdnkLQEAAAAyZ1xPM3sT8pybPb2ZB7JDIHOgCxcuSJKCSweavCUAAAAA7rcLFy7I29s72xqLcTexDfckPT1dJ0+elKenpywWi9mbk63k5GQFBgbq2LFj8vLyMntz8gz6an/01DHoq/3RU8egr/ZHTx2DvjrGP6WvhmHowoULKlGihJycsv+WGGfIHMjJyUklS5Y0ezNyxMvLK1fv3P9U9NX+6Klj0Ff7o6eOQV/tj546Bn11jH9CX+90ZuwmbuoBAAAAACYhkAEAAACASQhkkCS5ublp+PDhcnNzM3tT8hT6an/01DHoq/3RU8egr/ZHTx2DvjpGXuwrN/UAAAAAAJNwhgwAAAAATEIgAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEsD+nataueeeaZLOfXr19fFotFFotF+fPnV4UKFTR58mTr/NmzZ1vn3/rInz+/zWvcnO7i4qLSpUvr9ddf15UrVxw5tPvmbns4fvz4DPOefvppWSwWjRgxwqa+X79+1ueHDx9Whw4dVKJECeXPn18lS5ZUy5YttW/fPpt1rV+/Xk2bNpWvr68KFCigChUq6LXXXtOJEyf+7hBNdzc9vrVntzt37pz69eunoKAgubq6qkSJEoqMjNTRo0cz1J46dUq9e/dWmTJl5ObmpsDAQDVv3lzr1q2zw0jMd+vPo8Vika+vrxo3bqzdu3dnuUxcXFyGZZ566int3LnTWnPrseLWR48ePaw1t0738vJSjRo1tGzZMoeO93650z4qSZcvX9bw4cMVEhIiNzc3FSlSRM8//7z27t1rUzdixAhrn/Lly6fAwEC99NJLOnfuXIZ17ty5U23btlXx4sXl5uamoKAgNWvWTCtWrNA/9f5b97KP3rR37161adNGfn5+cnNzU0hIiN58801dunTJpq5UqVLW9RcoUEDh4eGaPn16hvUZhqFp06apTp068vLykoeHhypWrKi+ffvq4MGDdhvzP82xY8cUGRmpEiVKyNXVVUFBQerbt68SEhLM3jRT/J3jalRUVJY1W7ZsUdOmTVW4cGHlz59f4eHhev/993X9+vUMtXnhPcDf6WO+fPkyjDU+Pl7Ozs6yWCyKi4uzqb+170uXLlXt2rXl7e0tT09PVaxYMcN7irS0NL3zzjuqXLmyChQooCJFiqhu3bqaNWuWrl69aq8W5AiB7AHTvXt3xcfHKzo6Wm3atFGvXr20YMEC63wvLy/Fx8fbPI4cOWKzjsaNGys+Pl6HDh3ShAkTNHXqVA0fPvx+D8U0gYGBmj17ts20EydOaN26dSpevHiWy129elWNGjVSUlKSvvzyS+3fv1+LFi1SeHi4zp8/b62bOnWqGjZsKH9/f33xxReKjo7WlClTlJSUpPfee89Bo/pnOHfunGrXrq21a9dqypQpOnjwoBYuXKiDBw+qRo0aOnTokLU2Li5O1apV0/fff6///e9/2rNnj1avXq0GDRqoV69eJo7Cvm7+PMbHx2vdunVydnZWs2bN7rjc2rVrFR8fr2+//VYpKSlq0qSJzX5481hx6+Odd96xWcesWbMUHx+vX375RXXr1lXr1q21Z88eew8x10lNTVXDhg01c+ZMjR49WgcOHNDXX3+ta9euqVatWvr5559t6itWrKj4+HgdPXpUs2bN0urVq/Xyyy/b1Cxbtky1a9dWSkqKPv30U8XExGj16tV69tlnNWzYMCUlJd3PIdrVveyjP//8s2rVqqW0tDStWrVKBw4c0JgxYzR79mw1atRIaWlpNvVvvfWW4uPj9dtvv6lTp07q3r27vvnmG+t8wzDUoUMH9enTR02bNtV3332n6OhozZgxQ/nz59fo0aMdMvbc7tChQ6pevbpiY2O1YMECHTx4UFOmTNG6detUp06dTD84eBDc63E1K0uXLlW9evVUsmRJrV+/Xvv27VPfvn01evRotWvXzuYDl7z0HuBe+xgQEKA5c+bYTPv0008VEBCQ7XLr1q1T27Zt1apVK23btk07duzQmDFjbEJWWlqaIiIiNH78eL300kvasmWLtm3bpl69eumjjz7K8KHafWMgz+jSpYvRsmXLLOfXq1fP6Nu3r820cuXKGe3atTMMwzBmzZpleHt75/g1nnvuOaNq1ar3sMW5z9308OWXXzZ8fX2NH374wTp9zJgxRvPmzY3KlSsbw4cPt6m/2fOdO3cakoy4uLgs13/s2DHD1dXV6NevX6bzExMTczKcXOle9tObevToYRQsWNCIj4+3mX7p0iUjICDAaNy4sXVakyZNjICAACMlJSXDevJCHw0j815u3rzZkGScOXMm02UOHz5sSDJ27txpnfbjjz8akozVq1cbhpH9/8FNkoylS5danycnJxuSjA8//PBehpKr3GkfHT9+vGGxWIyoqCib6devXzeqV69uVKhQwUhPTzcMwzCGDx9uVK5c2abu1VdfNQoXLmx9npKSYvj6+hrPPvtslq95c33/NPeyj6anpxsVKlQwqlevbly/ft1mXlRUlGGxWIzx48dbpwUFBRkTJkywqfPx8TH69+9vfb5gwQJDkrFs2bIsX/NB1LhxY6NkyZLGpUuXbKbHx8cbBQoUMHr06GHSlpnHXsfVm27+fD/33HMZ5i1fvtyQZCxcuNAwjLz1HuDv9HHYsGFGuXLlbOaFhIQYb7zxhiHJOHz4sE39zb737dvXqF+/frbb9fbbbxtOTk7Gr7/+mmFeWlpapu8Z7gfOkD3g3N3dM3zSmBO//fabtmzZIldXVztuVe7m6uqqjh07atasWdZps2fPVmRkZLbL+fn5ycnJSZ9//nmmlyhI0pIlS5SWlqbXX3890/mFChW65+3+p0tPT9fChQvVsWNH+fv728xzd3dXz5499e233+rcuXM6d+6cVq9erV69eqlgwYIZ1pVX+5iSkqK5c+cqODhYvr6+d72cu7u7JN3zseDatWuaMWOGJD0Qx4L58+erUaNGqly5ss10Jycn9e/fX9HR0dq1a1emy8bFxenbb7+16dN3332nhISELH/upRuXiOYFd7OPRkVFKTo6Wq+++qqcnGzfplSuXFkNGza0ubLjVunp6friiy+UmJho0+MFCxaofPnyatGiRabL5ZX+5sS5c+f07bffqmfPntZjwE3+/v7q2LGjFi1a9I+9XNZe7vW4etPNn+8BAwZkmNe8eXOFhIRY9+e8/B4gJ31s0aKFEhMT9cMPP0iSfvjhByUmJqp58+bZLufv76+9e/fqt99+y7Jm3rx5atiwoapWrZphnouLS6bvGe4HAtkD6vr165o7d652796tJ554wjo9KSlJHh4eNo8mTZrYLLty5Up5eHhYr4E+c+aMBg4ceL+HYKrIyEgtXrxYFy9e1KZNm5SUlHTH0/ABAQGaOHGi3nzzTRUuXFhPPPGERo0aZXOZXWxsrLy8vLK99PFBdfbsWZ0/f15hYWGZzg8LC5NhGDp48KAOHjwowzAUGhp6n7fy/rv58+jh4SFPT08tX75cixYtyvBGNivnz5/XqFGj5OHhoZo1a1qnT548OcOxYN68eTbLtm/fXh4eHnJzc1P//v1VqlQptWnTxq7jy40OHDiQ7X54s+amPXv2yMPDQ+7u7ipdurT27t2rQYMG2axPksqXL2+dtn37dpver1y50hFDuS9yuo/e7Ed2Pb61v5I0aNAg677YunVrFS5cWN26dbNZ5639laR+/fpZt6tkyZJ/Z4j/SLGxsTIMI9s+JyYm6uzZs/d5y8z3d4+rt7rT/hwaGmqtyWvvAe61jy4uLurUqZNmzpwpSZo5c6Y6deokFxeXbJfr3bu3atSoofDwcJUqVUrt2rXTzJkzlZqaaq2JjY3Nle8NCGR50Lx582x+kW/evNk67+abLHd3d3Xv3l39+/e3+S6Dp6enoqKibB63fzm6QYMGioqK0tatW9WlSxf9+9//VqtWre7b+O6H7Hoo3fiUtly5cvr88881c+ZMvfDCC3J2dr7jenv16qVTp05p3rx5qlOnjpYsWaKKFStqzZo1km58z+FB+aT2Tj3Oyt18WvsgfaJ78+cxKipK27ZtU0REhJo0aaIjR46oSZMm1v5WrFjRZrlHH31UHh4eKly4sHbt2qVFixapWLFi1vkdO3bMcCy4/ezChAkTFBUVpW+++UYVKlTQ9OnT5ePjc1/GfT9kt4/mZB8rX768oqKitH37dg0aNEgRERHq3bt3tstUqlTJ2veLFy/q2rVr9zwOs93rPpqTHg8cOFBRUVH6/vvvVatWLU2YMEHBwcHZLjN06FBFRUXpzTffVEpKyj2NLS94kI6Xd+te99ns3O3vrrz0HuDv9DEyMlJLlizRqVOntGTJkjtehSRJBQsW1KpVq3Tw4EENGzZMHh4eeu2111SzZk3rzYBy6/5+53eQ+Mdp0aKFatWqZX1+65cgO3bsqKFDh8rd3V3FixfP8CmFk5PTHX+JFSxY0Fozc+ZMVa5cWTNmzNCLL75ox1GYK7se3hQZGalJkyYpOjpa27Ztu+t1e3p6qnnz5mrevLlGjx6tiIgIjR49Wo0aNVJISIiSkpIUHx+fZz4hy8rd9PhWfn5+KlSokGJiYjKdHxMTI4vFYt03LRZLhrtX5kW3/jxK0vTp0+Xt7a1p06Zp+vTpunz5siRl+GRx0aJFqlChgnx9fTO9DMbb2/uOxwJ/f38FBwcrODhYs2bNUtOmTRUdHa2iRYv+/YHlAlntoyEhIdnuhzdrbnJ1dbX2cvz48Xr66ac1cuRIjRo1SpJUrlw5SdL+/ftVu3ZtSZKbm9sd+/9PkdN99GbvYmJiMr2sKCYmxqa/klSkSBHrvrhkyRKFh4erevXqqlChgqQbPd6/f7/NMn5+fvLz88sz+2tOBQcHy2KxKCYmRs8++2yG+TExMSpcuLD8/PxM2Dpz3etxNTO37s+PPvpohvkxMTHW/TSvvQf4O30MDw9XaGio2rdvr7CwMD388MPZ3sXyVmXLllXZsmXVrVs3DR06VCEhIVq0aJH+/e9/KyQkJFe+N+AMWR7k6elp/cUUHBxsc234zTdZAQEB93Tq/XZOTk7673//q2HDhll/sPKC7Hp4U4cOHbRnzx49/PDD1oNpTlksFoWGhurixYuSpNatW8vV1TXD3exuuvUueP90d9PjWzk5OalNmzaaP3++Tp06ZTPv8uXLmjx5siIiIuTj4yMfHx9FRERo0qRJ1t7eKi/18XYWi0VOTk66fPmyAgICrP0NCgqyqQsMDFTZsmXt9p2EmjVrqlq1ahozZoxd1pcbZLWPtmvXTmvXrs3wPbH09HRNmDBBFSpUyPD9slsNGzZM7777rk6ePClJeuqpp+Tj46O3337bcYPJRe60j1apUkWhoaGaMGGC0tPTbZbdtWuX1q5dq/bt22e5/sDAQLVt21ZDhgyxTmvfvr3279+fZ/40gz34+vqqUaNGmjx5cobf3zev5Gjbtm2eOmNzr+72uJqZmz/fmd0hcfny5YqNjbXuz3n9PUBO+xgZGakNGzbc1dmxrJQqVUoFChSwvhfo0KGD1q5da/OnXm66evVqpu8Z7gcCGWwYhqFTp05leNz+S/FWzz//vPLly6dJkybdxy01X+HCha23cr0bUVFRatmypT7//HNFR0fr4MGDmjFjhmbOnKmWLVtKuvFGYsKECfrwww/14osvauPGjTpy5Ih+/PFH/ec//7F+op7XnT17NsPlcqdPn9bYsWPl7++vRo0a6ZtvvtGxY8e0adMmRURE6OrVqzb74KRJk3T9+nXVrFlTX3zxhWJjYxUTE6OJEyeqTp06Jo7OvlJTU60/pzExMerdu7dSUlLu+OXnO7l06VKG40BiYmK2y/Tr109Tp079R/2tnHvRv39/1axZU82bN9eSJUt09OhRbd++Xa1atVJMTIxmzJiR7ZvYOnXqqFKlSho7dqwkycPDQ9OnT9eqVav09NNP69tvv9WhQ4e0e/du6xuzfPny3ZexOUJO91GLxaIZM2YoOjraevvqo0ePasmSJWrevLnq1KmT7d8qlKS+fftqxYoV+uWXXyTdCNGtW7dWu3bt9NZbb2nr1q2Ki4vTxo0btWjRon90f/+Ojz/+WKmpqYqIiNCmTZt07NgxrV69Wo0aNVJAQECe+oAlJ+71uLp///4Mv7tcXV01depULVu2TC+99JJ2796tuLg4zZgxQ127dlXr1q2t373Na+8B/u7vp+7du+vs2bM23wfNzogRI/T6669rw4YNOnz4sHbu3KnIyEjrnx2Sbvyeqlu3rp588klNmjRJu3bt0qFDh7R48WLVrl1bsbGx9zzev8WMWzvCMf7O7cQN48Zt7yVl+rh5m/GsXmPcuHGGn5+fabcLtZe/28Psbnt/9uxZo0+fPsbDDz9seHh4GJ6enkZ4eLjx7rvvZri185o1a4yIiAijcOHCRv78+Y3Q0FBjwIABxsmTJ//G6HKHu+lxZvvgqFGjDMO40cfevXsbgYGBhouLi1GsWDGja9euxpEjRzKs6+TJk0avXr2MoKAgw9XV1QgICDBatGhhrF+/3kGju7+6dOli0yNPT0+jRo0axueff57lMtndnvmmrP4PIiIirDW67bb3hnHj1uGhoaHGyy+//HeHZqo77aOGYRgXL140hg4dagQHBxsuLi6Gj4+P0apVK2PPnj02dZnd9t4wbtyG3c3NzTh69Kh12vbt243WrVsbRYsWNZydnQ1fX18jIiLCWLhw4T/2tuz3so/etHv3bqNVq1aGj4+P4eLiYpQtW9YYNmyYcfHiRZu6zG57bxiGERERYTRp0sT6/Pr168aUKVOMWrVqGQULFjRcXV2NMmXKGN27dzeio6P/9lj/qeLi4owuXboYxYoVM1xcXIzAwECjd+/exh9//GH2ppni7xxXM3scO3bMMAzD2LRpkxEREWF4eXkZrq6uRsWKFY13333XuHbtWob15YX3AI74/XTzzwdlddv777//3mjVqpURGBhouLq6GsWKFTMaN25sbN682WY9V65cMcaNG2eEh4cb+fPnN3x8fIy6desas2fPNq5evWqP4eeYxTBy6bfbAAAAACCP45JFAAAAADAJgQwAAAAATEIgAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAExCIAMAAAAAkxDIAAAAAMAkBDIAwAOta9eueuaZZ6zP69evr379+t337diwYYMsFovOnz+fZY3FYtFXX3111+scMWKEqlSp8re2Ky4uThaLRVFRUX9rPQCAzBHIAAC5TteuXWWxWGSxWOTq6qrg4GC99dZbunbtmsNf+8svv9SoUaPuqvZuQhQAANlxNnsDAADITOPGjTVr1iylpqbq66+/Vq9eveTi4qIhQ4ZkqE1LS5Orq6tdXtfHx8cu6wEA4G5whgwAkCu5ubnJ399fQUFBevnll9WwYUMtX75c0l+XGY4ZM0YlSpRQ+fLlJUnHjh1TmzZtVKhQIfn4+Khly5aKi4uzrvP69et69dVXVahQIfn6+ur111+XYRg2r3v7JYupqakaNGiQAgMD5ebmpuDgYM2YMUNxcXFq0KCBJKlw4cKyWCzq2rWrJCk9PV3jxo1T6dKl5e7ursqVK+vzzz+3eZ2vv/5aISEhcnd3V4MGDWy2824NGjRIISEhKlCggMqUKaM33nhDV69ezVA3depUBQYGqkCBAmrTpo2SkpJs5k+fPl1hYWHKnz+/QkNDNXny5BxvCwDg3hDIAAD/CO7u7kpLS7M+X7dunfbv3681a9Zo5cqVunr1qiIiIuTp6anNmzfrxx9/lIeHhxo3bmxd7r333tPs2bM1c+ZM/fDDDzp37pyWLl2a7et27txZCxYs0MSJExUTE6OpU6fKw8NDgYGB+uKLLyRJ+/fvV3x8vD788ENJ0rhx4zRnzhxNmTJFe/fuVf/+/dWpUydt3LhR0o3g+Nxzz6l58+aKiopSt27dNHjw4Bz3xNPTU7Nnz1Z0dLQ+/PBDTZs2TRMmTLCpOXjwoBYvXqwVK1Zo9erV2rlzp3r27GmdP2/ePL355psaM2aMYmJiNHbsWL3xxhv69NNPc7w9AIB7YAAAkMt06dLFaNmypWEYhpGenm6sWbPGcHNzMwYMGGCdX6xYMSM1NdW6zGeffWaUL1/eSE9Pt05LTU013N3djW+//dYwDMMoXry48c4771jnX7161ShZsqT1tQzDMOrVq2f07dvXMAzD2L9/vyHJWLNmTabbuX79ekOSkZiYaJ125coVo0CBAsaWLVtsal988UWjffv2hmEYxpAhQ4wKFSrYzB80aFCGdd1OkrF06dIs5//vf/8zqlWrZn0+fPhwI1++fMbx48et07755hvDycnJiI+PNwzDMMqWLWvMnz/fZj2jRo0y6tSpYxiGYRw+fNiQZOzcuTPL1wUA3Du+QwYAyJVWrlwpDw8PXb16Venp6erQoYNGjBhhnR8eHm7zvbFdu3bp4MGD8vT0tFnPlStX9PvvvyspKUnx8fGqVauWdZ6zs7OqV6+e4bLFm6KiopQvXz7Vq1fvrrf74MGDunTpkho1amQzPS0tTVWrVpUkxcTE2GyHJNWpU+euX+OmRYsWaeLEifr999+VkpKia9euycvLy6bmoYceUkBAgM3rpKena//+/fL09NTvv/+uF198Ud27d7fWXLt2Td7e3jneHgBAzhHIAAC5UoMGDfTJJ5/I1dVVJUqUkLOz7a+sggUL2jxPSUlRtWrVNG/evAzr8vPzu6dtcHd3z/EyKSkpkqRVq1bZBCHpxvfi7OWnn35Sx44dNXLkSEVERMjb21sLFy7Ue++9l+NtnTZtWoaAmC9fPrttKwAgawQyAECuVLBgQQUHB991/SOPPKJFixapaNGiGc4S3VS8eHFt3bpVjz/+uKQbZ4J27NihRx55JNP68PBwpaena+PGjWrYsGGG+TfP0F2/ft06rUKFCnJzc9PRo0ezPLMWFhZmvUHJTT///POdB3mLLVu2KCgoSEOHDrVOO3LkSIa6o0eP6uTJkypRooT1dZycnFS+fHkVK1ZMJUqU0KFDh9SxY8ccvT4AwD64qQcAIE/o2LGjihQpopYtW2rz5s06fPiwNmzYoD59+uj48eOSpL59+2r8+PH66quvtG/fPvXs2TPbvyFWqlQpdenSRZGRkfrqq6+s61y8eLEkKSgoSBaLRStXrtTZs2eVkpIiT09PDRgwQP3799enn36q33//Xb/++qs++ugj640yevToodjYWA0cOFD79+/X/PnzNXv27ByNt1y5cjp69KgWLlyo33//XRMnTsz0BiX58+dXly5dtGvXLm3evFl9+vRRmzZt5O/vL0kaOXKkxo0bp4kTJ+rAgQPas2ePZs2apffffz9H2wMAuDcEMgBAnlCgQAFt2rRJDz30kJ577jmFhYXpxRdf1JUrV6xnzF577TW98MIL6tKli+rUqSNPT089++yz2a73k08+UevWrdWzZ0+Fhoaqe/fuunjxoiQpICBAI0eO1ODBg1WsWDG98sorkqRRo0bpjTfe0Lhx4xQWFqbGjRtr1apVKl26tKQb3+v64osv9NVXX6ly5cqaMmWKxo4dm6PxtmjRQv3799crr7yiKlWqaMuWLXrjjTcy1AUHB+u5555T06ZN9dRTT6lSpUo2t7Xv1q2bpk+frlmzZik8PFz16tXT7NmzrdsKAHAsi5HVN5kBAAAAAA7FGTIAAAAAMAmBDAAAAABMQiADAAAAAJMQyAAAAADAJAQyAAAAADAJgQwAAAAATEIgAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAExCIAMAAAAAk/w/Cz9lDfleFoEAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 1000x1000 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "eval_token_list = list(set(eval_df_tokens['labels']))\n",
    "\n",
    "plot_confusion_matrix(eval_df_tokens[\"labels\"], eval_df_tokens[\"predicted_label\"],\n",
    "                      eval_token_list)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define & Call Function to Display Example Token Sequences Along With Labels & Losses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "      <th>14</th>\n",
       "      <th>15</th>\n",
       "      <th>16</th>\n",
       "      <th>17</th>\n",
       "      <th>18</th>\n",
       "      <th>19</th>\n",
       "      <th>20</th>\n",
       "      <th>21</th>\n",
       "      <th>22</th>\n",
       "      <th>23</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>tokens</th>\n",
       "      <td>He</td>\n",
       "      <td>was</td>\n",
       "      <td>selected</td>\n",
       "      <td>to</td>\n",
       "      <td>go</td>\n",
       "      <td>on</td>\n",
       "      <td>the</td>\n",
       "      <td>2002</td>\n",
       "      <td>New</td>\n",
       "      <td>Zealand</td>\n",
       "      <td>rugby</td>\n",
       "      <td>league</td>\n",
       "      <td>tour</td>\n",
       "      <td>of</td>\n",
       "      <td>Great</td>\n",
       "      <td>Britain</td>\n",
       "      <td>and</td>\n",
       "      <td>France</td>\n",
       "      <td>,</td>\n",
       "      <td>playing</td>\n",
       "      <td>at</td>\n",
       "      <td>centre</td>\n",
       "      <td>.</td>\n",
       "      <td>[SEP]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>labels</th>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>preds</th>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>I-LOC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>I-LOC</td>\n",
       "      <td>O</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>losses</th>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.08</td>\n",
       "      <td>6.88</td>\n",
       "      <td>7.26</td>\n",
       "      <td>4.40</td>\n",
       "      <td>4.73</td>\n",
       "      <td>6.53</td>\n",
       "      <td>7.19</td>\n",
       "      <td>8.20</td>\n",
       "      <td>8.14</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          0     1         2     3     4     5     6     7       8        9    \n",
       "tokens    He   was  selected    to    go    on   the  2002     New  Zealand  \\\n",
       "labels     O     O         O     O     O     O     O     O  I-MISC   I-MISC   \n",
       "preds      O     O         O     O     O     O     O     O   B-LOC    I-LOC   \n",
       "losses  0.00  0.00      0.00  0.00  0.00  0.00  0.00  0.08    6.88     7.26   \n",
       "\n",
       "            10      11      12      13      14       15    16      17    18   \n",
       "tokens   rugby  league    tour      of   Great  Britain   and  France     ,  \\\n",
       "labels  I-MISC  I-MISC  I-MISC  I-MISC  I-MISC   I-MISC     O   B-LOC     O   \n",
       "preds        O       O       O       O   B-LOC    I-LOC     O   B-LOC     O   \n",
       "losses    4.40    4.73    6.53    7.19    8.20     8.14  0.00    0.00  0.00   \n",
       "\n",
       "             19    20      21    22     23  \n",
       "tokens  playing    at  centre     .  [SEP]  \n",
       "labels        O     O       O     O    IGN  \n",
       "preds         O     O       O     O      O  \n",
       "losses     0.00  0.00    0.00  0.00   0.00  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "      <th>14</th>\n",
       "      <th>15</th>\n",
       "      <th>16</th>\n",
       "      <th>17</th>\n",
       "      <th>18</th>\n",
       "      <th>19</th>\n",
       "      <th>20</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>tokens</th>\n",
       "      <td>The</td>\n",
       "      <td>Port</td>\n",
       "      <td>of</td>\n",
       "      <td>Dartmouth</td>\n",
       "      <td>Royal</td>\n",
       "      <td>Reg</td>\n",
       "      <td>##att</td>\n",
       "      <td>##a</td>\n",
       "      <td>takes</td>\n",
       "      <td>place</td>\n",
       "      <td>annually</td>\n",
       "      <td>over</td>\n",
       "      <td>three</td>\n",
       "      <td>days</td>\n",
       "      <td>at</td>\n",
       "      <td>the</td>\n",
       "      <td>end</td>\n",
       "      <td>of</td>\n",
       "      <td>August</td>\n",
       "      <td>.</td>\n",
       "      <td>[SEP]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>labels</th>\n",
       "      <td>O</td>\n",
       "      <td>B-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>preds</th>\n",
       "      <td>O</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>I-LOC</td>\n",
       "      <td>I-LOC</td>\n",
       "      <td>I-LOC</td>\n",
       "      <td>I-LOC</td>\n",
       "      <td>I-LOC</td>\n",
       "      <td>I-LOC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>losses</th>\n",
       "      <td>0.00</td>\n",
       "      <td>8.25</td>\n",
       "      <td>9.42</td>\n",
       "      <td>9.26</td>\n",
       "      <td>8.76</td>\n",
       "      <td>8.63</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          0       1       2          3       4       5      6      7      8    \n",
       "tokens   The    Port      of  Dartmouth   Royal     Reg  ##att    ##a  takes  \\\n",
       "labels     O  B-MISC  I-MISC     I-MISC  I-MISC  I-MISC    IGN    IGN      O   \n",
       "preds      O   B-LOC   I-LOC      I-LOC   I-LOC   I-LOC  I-LOC  I-LOC      O   \n",
       "losses  0.00    8.25    9.42       9.26    8.76    8.63   0.00   0.00   0.00   \n",
       "\n",
       "           9         10    11     12    13    14    15    16    17      18   \n",
       "tokens  place  annually  over  three  days    at   the   end    of  August  \\\n",
       "labels      O         O     O      O     O     O     O     O     O       O   \n",
       "preds       O         O     O      O     O     O     O     O     O       O   \n",
       "losses   0.00      0.00  0.00   0.00  0.00  0.00  0.00  0.00  0.00    0.00   \n",
       "\n",
       "          19     20  \n",
       "tokens     .  [SEP]  \n",
       "labels     O    IGN  \n",
       "preds      O      O  \n",
       "losses  0.00   0.00  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "      <th>14</th>\n",
       "      <th>15</th>\n",
       "      <th>16</th>\n",
       "      <th>17</th>\n",
       "      <th>18</th>\n",
       "      <th>19</th>\n",
       "      <th>20</th>\n",
       "      <th>21</th>\n",
       "      <th>22</th>\n",
       "      <th>23</th>\n",
       "      <th>24</th>\n",
       "      <th>25</th>\n",
       "      <th>26</th>\n",
       "      <th>27</th>\n",
       "      <th>28</th>\n",
       "      <th>29</th>\n",
       "      <th>30</th>\n",
       "      <th>31</th>\n",
       "      <th>32</th>\n",
       "      <th>33</th>\n",
       "      <th>34</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>tokens</th>\n",
       "      <td>Into</td>\n",
       "      <td>the</td>\n",
       "      <td>latter</td>\n",
       "      <td>1940</td>\n",
       "      <td>s</td>\n",
       "      <td>the</td>\n",
       "      <td>Nashville</td>\n",
       "      <td>,</td>\n",
       "      <td>Chattanooga</td>\n",
       "      <td>&amp;</td>\n",
       "      <td>St</td>\n",
       "      <td>.</td>\n",
       "      <td>Louis</td>\n",
       "      <td>Railway</td>\n",
       "      <td>'</td>\n",
       "      <td>s</td>\n",
       "      <td>#</td>\n",
       "      <td>3</td>\n",
       "      <td>/</td>\n",
       "      <td>#</td>\n",
       "      <td>4</td>\n",
       "      <td>(</td>\n",
       "      <td>Memphis</td>\n",
       "      <td>-</td>\n",
       "      <td>Nashville</td>\n",
       "      <td>-</td>\n",
       "      <td>Atlanta</td>\n",
       "      <td>)</td>\n",
       "      <td>served</td>\n",
       "      <td>Mu</td>\n",
       "      <td>##rf</td>\n",
       "      <td>##ree</td>\n",
       "      <td>##sboro</td>\n",
       "      <td>.</td>\n",
       "      <td>[SEP]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>labels</th>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>O</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>I-LOC</td>\n",
       "      <td>I-LOC</td>\n",
       "      <td>IGN</td>\n",
       "      <td>I-LOC</td>\n",
       "      <td>I-LOC</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>O</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>O</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>preds</th>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-ORG</td>\n",
       "      <td>I-ORG</td>\n",
       "      <td>I-ORG</td>\n",
       "      <td>I-ORG</td>\n",
       "      <td>I-ORG</td>\n",
       "      <td>I-ORG</td>\n",
       "      <td>I-ORG</td>\n",
       "      <td>I-ORG</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>O</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>O</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>I-LOC</td>\n",
       "      <td>I-LOC</td>\n",
       "      <td>I-LOC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>losses</th>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>3.57</td>\n",
       "      <td>5.49</td>\n",
       "      <td>8.39</td>\n",
       "      <td>6.05</td>\n",
       "      <td>6.41</td>\n",
       "      <td>0.00</td>\n",
       "      <td>6.31</td>\n",
       "      <td>5.82</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          0     1       2     3     4     5          6      7            8    \n",
       "tokens  Into   the  latter  1940     s   the  Nashville      ,  Chattanooga  \\\n",
       "labels     O     O       O     O     O     O      B-LOC      O        B-LOC   \n",
       "preds      O     O       O     O     O     O      B-ORG  I-ORG        I-ORG   \n",
       "losses  0.00  0.00    0.00  0.00  0.00  0.00       3.57   5.49         8.39   \n",
       "\n",
       "           9      10     11     12       13    14    15    16    17    18   \n",
       "tokens      &     St      .  Louis  Railway     '     s     #     3     /  \\\n",
       "labels  I-LOC  I-LOC    IGN  I-LOC    I-LOC     O   IGN     O     O     O   \n",
       "preds   I-ORG  I-ORG  I-ORG  I-ORG    I-ORG     O     O     O     O     O   \n",
       "losses   6.05   6.41   0.00   6.31     5.82  0.00  0.00  0.00  0.00  0.00   \n",
       "\n",
       "          19    20    21       22    23         24    25       26    27   \n",
       "tokens     #     4     (  Memphis     -  Nashville     -  Atlanta     )  \\\n",
       "labels   IGN     O     O    B-LOC     O      B-LOC     O    B-LOC     O   \n",
       "preds      O     O     O    B-LOC     O      B-LOC     O    B-LOC     O   \n",
       "losses  0.00  0.00  0.00     0.00  0.00       0.00  0.00     0.00  0.00   \n",
       "\n",
       "            28     29     30     31       32    33     34  \n",
       "tokens  served     Mu   ##rf  ##ree  ##sboro     .  [SEP]  \n",
       "labels       O  B-LOC    IGN    IGN      IGN     O    IGN  \n",
       "preds        O  B-LOC  I-LOC  I-LOC    I-LOC     O      O  \n",
       "losses    0.00   0.00   0.00   0.00     0.00  0.00   0.00  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def get_samples(df):\n",
    "    for _, row in df.iterrows():\n",
    "        labels, preds, tokens, losses = [], [], [], []\n",
    "        for i, mask in enumerate(row[\"attention_mask\"]):\n",
    "            if i not in {0, len(row[\"attention_mask\"])}:\n",
    "                labels.append(row[\"labels\"][i])\n",
    "                preds.append(row[\"predicted_label\"][i])\n",
    "                tokens.append(row[\"input_tokens\"][i])\n",
    "                losses.append(f\"{row['loss'][i]:.2f}\")\n",
    "        eval_df_tmp = pd.DataFrame({\"tokens\": tokens, \"labels\": labels, \n",
    "                               \"preds\": preds, \"losses\": losses}).T\n",
    "        yield eval_df_tmp\n",
    "\n",
    "eval_df[\"total_loss\"] = eval_df[\"loss\"].apply(sum)\n",
    "eval_df_tmp = eval_df.sort_values(by=\"total_loss\", ascending=False).head(3)\n",
    "\n",
    "pd.set_option('display.max_columns', None)\n",
    "\n",
    "for sample in get_samples(eval_df_tmp):\n",
    "    display(sample)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Notes & Other Takeaways From This Project\n",
    "\n",
    "****\n",
    "- This model is part of a comparison of Token Classification Models using this particular dataset.\n",
    "\n",
    "****"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Citations\n",
    "\n",
    "- Model Checkpoint\n",
    "\n",
    "    > @article{DBLP:journals/corr/abs-1810-04805, author = {Jacob Devlin and Ming{-}Wei Chang and Kenton Lee and Kristina Toutanova}, title = {{BERT:} Pre-training of Deep Bidirectional Transformers for Language Understanding}, journal = {CoRR}, volume = {abs/1810.04805}, year = {2018}, url = {http://arxiv.org/abs/1810.04805}, archivePrefix = {arXiv}, eprint = {1810.04805}, timestamp = {Tue, 30 Oct 2018 20:39:56 +0100}, biburl = {https://dblp.org/rec/journals/corr/abs-1810-04805.bib}, bibsource = {dblp computer science bibliography, https://dblp.org}}\n",
    "\n",
    "- Dataset\n",
    "\n",
    "    > @inproceedings{tedeschi-etal-2021-wikineural-combined, title = \"{W}iki{NE}u{R}al: {C}ombined Neural and Knowledge-based Silver Data Creation for Multilingual {NER}\", author = \"Tedeschi, Simone and Maiorca, Valentino and Campolungo, Niccol{\\`o} and Cecconi, Francesco and Navigli, Roberto\", booktitle = \"Findings of the Association for Computational Linguistics: EMNLP 2021\", month = nov, year = \"2021\", address = \"Punta Cana, Dominican Republic\", publisher = \"Association for Computational Linguistics\", url = \"https://aclanthology.org/2021.findings-emnlp.215\", pages = \"2521--2533\", abstract = \"Multilingual Named Entity Recognition (NER) is a key intermediate task which is needed in many areas of NLP. In this paper, we address the well-known issue of data scarcity in NER, especially relevant when moving to a multilingual scenario, and go beyond current approaches to the creation of multilingual silver data for the task. We exploit the texts of Wikipedia and introduce a new methodology based on the effective combination of knowledge-based approaches and neural models, together with a novel domain adaptation technique, to produce high-quality training corpora for NER. We evaluate our datasets extensively on standard benchmarks for NER, yielding substantial improvements up to 6 span-based F1-score points over previous state-of-the-art systems for data creation.\",}\n",
    "\n",
    "- Metric (PosEval)\n",
    "\n",
    "    > @article{scikit-learn, title={Scikit-learn: Machine Learning in {P}ython}, author={Pedregosa, F. and Varoquaux, G. and Gramfort, A. and Michel, V. and Thirion, B. and Grisel, O. and Blondel, M. and Prettenhofer, P. and Weiss, R. and Dubourg, V. and Vanderplas, J. and Passos, A. and Cournapeau, D. and Brucher, M. and Perrot, M. and Duchesnay, E.}, journal={Journal of Machine Learning Research}, volume={12}, pages={2825--2830}, year={2011}}"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "dl",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}