Token Classification/Monolingual/WikiNeural - Transformer Comparison/POS Project with Wikineural Dataset - FNet Transformer.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Monolingual Named Entity Recognition Token Classification (WikiNeural Dataset)\n",
    "\n",
    "### FNet-Base Transformer\n",
    "\n",
    "Dataset Source: https://huggingface.co/datasets/Babelscape/wikineural"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Import Necessary Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Git LFS initialized.\n"
     ]
    }
   ],
   "source": [
    "import os, sys\n",
    "os.environ['TOKENIZERS_PARALLELISM']='false'\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import sklearn\n",
    "from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay\n",
    "\n",
    "import datasets\n",
    "from datasets import load_dataset, DatasetDict\n",
    "\n",
    "import torch\n",
    "from torch import nn\n",
    "from torch.nn.functional import cross_entropy\n",
    "\n",
    "import transformers\n",
    "from transformers import AutoTokenizer, AutoModelForTokenClassification\n",
    "from transformers import DataCollatorForTokenClassification\n",
    "from transformers import Trainer, TrainingArguments\n",
    "\n",
    "import evaluate\n",
    "\n",
    "!git lfs install"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Display Library Versions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "          Python : 3.9.12\n",
      "           NumPy : 1.25.0\n",
      "           Torch : 2.0.1\n",
      "          Pandas : 2.0.2\n",
      "    Transformers : 4.28.1\n",
      "        datasets : 2.13.0\n",
      "         Sklearn : 1.2.2\n"
     ]
    }
   ],
   "source": [
    "print(\"Python :\".rjust(18), sys.version[0:6])\n",
    "print(\"NumPy :\".rjust(18), np.__version__)\n",
    "print(\"Torch :\".rjust(18), torch.__version__)\n",
    "print(\"Pandas :\".rjust(18), pd.__version__)\n",
    "print(\"Transformers :\".rjust(18), transformers.__version__)\n",
    "print(\"datasets :\".rjust(18), datasets.__version__)\n",
    "print(\"Sklearn :\".rjust(18), sklearn.__version__)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Ingest English Subset of Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Found cached dataset parquet (/Users/briandunn/.cache/huggingface/datasets/Babelscape___parquet/Babelscape--wikineural-579d1dc98d2a6b93/0.0.0/14a00e99c0d15a23649d0db8944380ac81082d4b021f398733dd84f3a6c569a7)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1cecf899a98a43cc8e36891d26d9d3a2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/27 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    test_de: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 12372\n",
       "    })\n",
       "    test_en: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 11597\n",
       "    })\n",
       "    test_es: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 9618\n",
       "    })\n",
       "    test_fr: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 12678\n",
       "    })\n",
       "    test_it: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 11069\n",
       "    })\n",
       "    test_nl: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 10547\n",
       "    })\n",
       "    test_pl: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 13585\n",
       "    })\n",
       "    test_pt: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 10160\n",
       "    })\n",
       "    test_ru: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 11580\n",
       "    })\n",
       "    train_de: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 98640\n",
       "    })\n",
       "    train_en: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 92720\n",
       "    })\n",
       "    train_es: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 76320\n",
       "    })\n",
       "    train_fr: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 100800\n",
       "    })\n",
       "    train_it: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 88400\n",
       "    })\n",
       "    train_nl: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 83680\n",
       "    })\n",
       "    train_pl: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 108160\n",
       "    })\n",
       "    train_pt: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 80560\n",
       "    })\n",
       "    train_ru: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 92320\n",
       "    })\n",
       "    val_de: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 12330\n",
       "    })\n",
       "    val_en: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 11590\n",
       "    })\n",
       "    val_es: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 9540\n",
       "    })\n",
       "    val_fr: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 12600\n",
       "    })\n",
       "    val_it: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 11050\n",
       "    })\n",
       "    val_nl: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 10460\n",
       "    })\n",
       "    val_pl: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 13520\n",
       "    })\n",
       "    val_pt: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 10070\n",
       "    })\n",
       "    val_ru: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 11540\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = load_dataset(\"Babelscape/wikineural\")\n",
    "\n",
    "data = data.remove_columns([\"lang\"])\n",
    "\n",
    "data"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Split Dataset into DatasetDict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training data shape: (92720, 2)\n",
      "Testing data shape: (11597, 2)\n",
      "Validation data shape: (11590, 2)\n"
     ]
    }
   ],
   "source": [
    "ds = DatasetDict({\n",
    "    'train': data['train_en'], \n",
    "    'test': data['test_en'], \n",
    "    'eval': data['val_en']})\n",
    "\n",
    "print('Training data shape:', ds['train'].shape)\n",
    "print('Testing data shape:', ds['test'].shape)\n",
    "print('Validation data shape:', ds['eval'].shape)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'tokens': ['It',\n",
       "  'has',\n",
       "  'a',\n",
       "  'leaf',\n",
       "  'pattern',\n",
       "  'similar',\n",
       "  'to',\n",
       "  'the',\n",
       "  'members',\n",
       "  'of',\n",
       "  'the',\n",
       "  'genera',\n",
       "  '\"',\n",
       "  'Kedrostis',\n",
       "  '\"',\n",
       "  ',',\n",
       "  '\"',\n",
       "  'Melothria',\n",
       "  '\"',\n",
       "  'and',\n",
       "  '\"',\n",
       "  'Zehneria',\n",
       "  '\"',\n",
       "  '.'],\n",
       " 'ner_tags': [0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  7,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  7,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  7,\n",
       "  0,\n",
       "  0]}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example = ds['train'][12]\n",
    "\n",
    "example"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Display Feature Information About Each Feature"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tokens: \n",
      "Sequence(feature=Value(dtype='string', id=None), length=-1, id=None)\n",
      "\n",
      "ner_tags: \n",
      "Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for k, v in ds[\"train\"].features.items():\n",
    "    print(f\"{k}: \\n{v}\\n\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define Tag Values & Conversions Between String & Integer Values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "List of tag values: \n",
      "['O', 'B-PER', 'I-PER', 'B-ORG', 'I-ORG', 'B-LOC', 'I-LOC', 'B-MISC', 'I-MISC']\n",
      "Number of NER Tags: \n",
      "9\n",
      "id2label: \n",
      "{0: 'O', 1: 'B-PER', 2: 'I-PER', 3: 'B-ORG', 4: 'I-ORG', 5: 'B-LOC', 6: 'I-LOC', 7: 'B-MISC', 8: 'I-MISC'}\n",
      "label2id: \n",
      "{'O': 0, 'B-PER': 1, 'I-PER': 2, 'B-ORG': 3, 'I-ORG': 4, 'B-LOC': 5, 'I-LOC': 6, 'B-MISC': 7, 'I-MISC': 8}\n"
     ]
    }
   ],
   "source": [
    "label2id = {\n",
    "    'O': 0, \n",
    "    'B-PER': 1, \n",
    "    'I-PER': 2, \n",
    "    'B-ORG': 3, \n",
    "    'I-ORG': 4, \n",
    "    'B-LOC': 5, \n",
    "    'I-LOC': 6, \n",
    "    'B-MISC': 7, \n",
    "    'I-MISC': 8\n",
    "    }\n",
    "\n",
    "id2label = {tag: idx for idx, tag in label2id.items()}\n",
    "\n",
    "pos_tag_values = list(label2id.keys())\n",
    "NUM_OF_LABELS = len(pos_tag_values)\n",
    "\n",
    "print(f\"List of tag values: \\n{pos_tag_values}\")\n",
    "print(f\"Number of NER Tags: \\n{NUM_OF_LABELS}\")\n",
    "print(f\"id2label: \\n{id2label}\")\n",
    "print(f\"label2id: \\n{label2id}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Basic Values/Constants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "MODEL_CKPT = \"google/fnet-base\"\n",
    "MODEL_NAME = f\"{MODEL_CKPT.split(f'/')[-1]}-finetuned-WikiNeural\"\n",
    "\n",
    "NUM_OF_EPOCHS = 2\n",
    "BATCH_SIZE = 16\n",
    "\n",
    "STRATEGY = \"epoch\"\n",
    "REPORTS_TO = \"tensorboard\"\n",
    "\n",
    "WEIGHT_DECAY = 0.01\n",
    "LR = 2e-5\n",
    "\n",
    "DEVICE = torch.device(\"cpu\")\n",
    "STEPS = 1250"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Function to Tokenize & Align Inputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = AutoTokenizer.from_pretrained(MODEL_CKPT)\n",
    "\n",
    "def tokenize_and_align_labels(samples):\n",
    "    tokenized_inputs = tokenizer(samples[\"tokens\"], \n",
    "                                      truncation=True, \n",
    "                                      is_split_into_words=True)\n",
    "    \n",
    "    labels = []\n",
    "    \n",
    "    for idx, label in enumerate(samples[f\"ner_tags\"]):\n",
    "        word_ids = tokenized_inputs.word_ids(batch_index=idx)\n",
    "        prev_word_idx = None\n",
    "        label_ids = []\n",
    "        for word_idx in word_ids: # set special tokens to -100\n",
    "            if word_idx is None or word_idx == prev_word_idx:\n",
    "                label_ids.append(-100)\n",
    "            else:\n",
    "                label_ids.append(label[word_idx])\n",
    "            prev_word_idx = word_idx\n",
    "        labels.append(label_ids)\n",
    "    \n",
    "    tokenized_inputs[\"labels\"] = labels\n",
    "    return tokenized_inputs"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Apply Above Function to Tokenize Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c5ff17fb4a1c4324b23ec05a3d83e013",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/92720 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bc20544e042143fd800bdcf5ee19a7a7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/11597 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7d90db018b92417eb1e058153042eeeb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/11590 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "encoded_ds = ds.map(tokenize_and_align_labels, \n",
    "                    batched=True, \n",
    "                    remove_columns=\n",
    "                        [\n",
    "                            'ner_tags', \n",
    "                            'tokens'\n",
    "                        ]\n",
    "                    )"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define Data Collator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_collator = DataCollatorForTokenClassification(tokenizer=tokenizer)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load Custom Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at google/fnet-base were not used when initializing FNetForTokenClassification: ['cls.predictions.decoder.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias']\n",
      "- This IS expected if you are initializing FNetForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing FNetForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of FNetForTokenClassification were not initialized from the model checkpoint at google/fnet-base and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "model = (AutoModelForTokenClassification.from_pretrained(\n",
    "    MODEL_CKPT,\n",
    "    num_labels=NUM_OF_LABELS,\n",
    "    id2label=id2label,\n",
    "    label2id=label2id\n",
    "    ).to(DEVICE))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create Compute Metrics Function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "label_list = pos_tag_values\n",
    "\n",
    "seqeval = evaluate.load(\"seqeval\")\n",
    "\n",
    "labels = [label_list[i] for i in example[f'ner_tags']]\n",
    "\n",
    "def compute_metrics(eval_preds):\n",
    "    predictions, labels = eval_preds\n",
    "    predictions = np.argmax(predictions, \n",
    "                            axis=2)\n",
    "    \n",
    "    true_predictions = [\n",
    "        [label_list[p] for (p, l) in zip(prediction, label) if l != -100]\n",
    "        for prediction, label in zip(predictions, labels)\n",
    "    ]\n",
    "    \n",
    "    true_labels = [\n",
    "        [label_list[l] for (p, l) in zip(prediction, label) if l != -100]\n",
    "        for prediction, label in zip(predictions, labels)\n",
    "    ]\n",
    "    \n",
    "    results = seqeval.compute(predictions=true_predictions, \n",
    "                              references=true_labels)\n",
    "    \n",
    "    return results"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define TrainingArguments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "args = TrainingArguments(\n",
    "    MODEL_NAME,\n",
    "    log_level=\"error\",\n",
    "    logging_first_step=True,\n",
    "    learning_rate=LR,\n",
    "    num_train_epochs=NUM_OF_EPOCHS,\n",
    "    per_device_train_batch_size=BATCH_SIZE,\n",
    "    per_device_eval_batch_size=BATCH_SIZE,\n",
    "    evaluation_strategy=STRATEGY,\n",
    "    report_to=REPORTS_TO,\n",
    "    disable_tqdm=False,\n",
    "    logging_steps=STEPS,\n",
    "    weight_decay=WEIGHT_DECAY,\n",
    "    save_strategy=STRATEGY,\n",
    "    hub_private_repo=True,\n",
    "    push_to_hub=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Subclass Trainer to Handle Class Imbalance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CustomTrainer(Trainer):\n",
    "    def compute_loss(self, \n",
    "                     model, \n",
    "                     inputs, \n",
    "                     return_outputs=False):\n",
    "        \n",
    "        labels = inputs.get(\"labels\")\n",
    "        # forward pass\n",
    "        outputs = model(**inputs)\n",
    "        logits = outputs.get(\"logits\")\n",
    "        # compute custom loss \n",
    "        loss_fct = nn.CrossEntropyLoss(weight=torch.tensor(\n",
    "            [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0], \n",
    "            device=model.device)\n",
    "        )\n",
    "        loss = loss_fct(logits.view(-1, \n",
    "                                    self.model.config.num_labels), \n",
    "                        labels.view(-1)\n",
    "                        )\n",
    "        return (loss, outputs) if return_outputs else loss"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define Trainer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Cloning https://huggingface.co/DunnBC22/fnet-base-finetuned-WikiNeural into local empty directory.\n"
     ]
    }
   ],
   "source": [
    "trainer = CustomTrainer(model, \n",
    "                  args=args,\n",
    "                  data_collator=data_collator,\n",
    "                  compute_metrics=compute_metrics,\n",
    "                  tokenizer=tokenizer,\n",
    "                  train_dataset=encoded_ds[\"train\"],\n",
    "                  eval_dataset=encoded_ds[\"test\"]\n",
    "                  )"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Train Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/briandunn/Documents/deep_learning/audio_projects/lib/python3.9/site-packages/transformers/optimization.py:391: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "66966912fd514b0c9802f8dac87dffd7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/11590 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 2.2708, 'learning_rate': 1.9998274374460744e-05, 'epoch': 0.0}\n",
      "{'loss': 0.307, 'learning_rate': 1.7842968075927526e-05, 'epoch': 0.22}\n",
      "{'loss': 0.1818, 'learning_rate': 1.5685936151855047e-05, 'epoch': 0.43}\n",
      "{'loss': 0.1609, 'learning_rate': 1.3528904227782573e-05, 'epoch': 0.65}\n",
      "{'loss': 0.151, 'learning_rate': 1.1371872303710096e-05, 'epoch': 0.86}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c4204344fe424b918726990a59e695b7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/725 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.14341209828853607, 'eval_LOC': {'precision': 0.8661224489795918, 'recall': 0.890848026868178, 'f1': 0.8783112582781456, 'number': 5955}, 'eval_MISC': {'precision': 0.7294915254237289, 'recall': 0.8504248172297965, 'f1': 0.7853298056746647, 'number': 5061}, 'eval_ORG': {'precision': 0.8141543605461131, 'recall': 0.8472020875616121, 'f1': 0.8303495311167945, 'number': 3449}, 'eval_PER': {'precision': 0.9204172300560169, 'recall': 0.9145873320537428, 'f1': 0.9174930201213054, 'number': 5210}, 'eval_overall_precision': 0.8318984175845318, 'eval_overall_recall': 0.8790851334180432, 'eval_overall_f1': 0.8548411011713537, 'eval_overall_accuracy': 0.9833355168347963, 'eval_runtime': 613.3702, 'eval_samples_per_second': 18.907, 'eval_steps_per_second': 1.182, 'epoch': 1.0}\n",
      "{'loss': 0.1339, 'learning_rate': 9.214840379637619e-06, 'epoch': 1.08}\n",
      "{'loss': 0.1088, 'learning_rate': 7.057808455565142e-06, 'epoch': 1.29}\n",
      "{'loss': 0.1056, 'learning_rate': 4.900776531492667e-06, 'epoch': 1.51}\n",
      "{'loss': 0.0999, 'learning_rate': 2.74374460742019e-06, 'epoch': 1.73}\n",
      "{'loss': 0.102, 'learning_rate': 5.867126833477136e-07, 'epoch': 1.94}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a148d644b84f45a3b996dc1bbe82ee1c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/725 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.12971119582653046, 'eval_LOC': {'precision': 0.8777669270833334, 'recall': 0.9056255247691016, 'f1': 0.8914786345979007, 'number': 5955}, 'eval_MISC': {'precision': 0.7579235908857289, 'recall': 0.8741355463347165, 'f1': 0.8118920902917965, 'number': 5061}, 'eval_ORG': {'precision': 0.8336120401337793, 'recall': 0.8672078863438678, 'f1': 0.8500781583060963, 'number': 3449}, 'eval_PER': {'precision': 0.9424901185770751, 'recall': 0.9153550863723608, 'f1': 0.9287244401168452, 'number': 5210}, 'eval_overall_precision': 0.8520529351883271, 'eval_overall_recall': 0.8933672172808133, 'eval_overall_f1': 0.8722211194918619, 'eval_overall_accuracy': 0.9853306133143681, 'eval_runtime': 612.8561, 'eval_samples_per_second': 18.923, 'eval_steps_per_second': 1.183, 'epoch': 2.0}\n",
      "{'train_runtime': 34055.2573, 'train_samples_per_second': 5.445, 'train_steps_per_second': 0.34, 'train_loss': 0.14889413781779096, 'epoch': 2.0}\n"
     ]
    }
   ],
   "source": [
    "train_results = trainer.train()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Push Model To Hub"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Several commits (2) will be pushed upstream.\n",
      "The progress bars may be unreliable.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7304ddc23cd54c928eb7928c384485be",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Upload file pytorch_model.bin:   0%|          | 1.00/316M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "96ae52ff2bf149b5af644e6a8e48a61e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Upload file runs/Jul13_21-37-02_Brians-Mac-mini/events.out.tfevents.1689302227.Brians-Mac-mini.9520.0:   0%|  …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "To https://huggingface.co/DunnBC22/fnet-base-finetuned-WikiNeural\n",
      "   cf8fb6b..13b9308  main -> main\n",
      "\n",
      "To https://huggingface.co/DunnBC22/fnet-base-finetuned-WikiNeural\n",
      "   13b9308..a9df428  main -> main\n",
      "\n"
     ]
    }
   ],
   "source": [
    "kwargs = {\n",
    "    \"tasks\": \"token-classification\",\n",
    "    \"tags\": ['token-classification'],\n",
    "}\n",
    "\n",
    "if args.push_to_hub:\n",
    "    trainer.push_to_hub('All DUNN!!!', **kwargs)\n",
    "else:\n",
    "    trainer.create_model_card(**kwargs)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Save & Log Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "To https://huggingface.co/DunnBC22/fnet-base-finetuned-WikiNeural\n",
      "   a9df428..6cc374f  main -> main\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "***** train metrics *****\n",
      "  epoch                    =        2.0\n",
      "  train_loss               =     0.1489\n",
      "  train_runtime            = 9:27:35.25\n",
      "  train_samples_per_second =      5.445\n",
      "  train_steps_per_second   =       0.34\n"
     ]
    }
   ],
   "source": [
    "trainer.save_model()\n",
    "trainer.log_metrics(\"train\", train_results.metrics)\n",
    "trainer.save_metrics(\"train\", train_results.metrics)\n",
    "trainer.save_state()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Error Analysis"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define Method to Apply to Validation Dataset (& Then Apply it)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def forward_pass_with_label(batch):\n",
    "    # Convert dict of lists to list of dicts suitable for data collator\n",
    "    features = [dict(zip(batch, t)) for t in zip(*batch.values())]\n",
    "    # Pad inputs and labels and put all tensors on device\n",
    "    batch = data_collator(features)\n",
    "    input_ids = batch[\"input_ids\"].to(DEVICE)\n",
    "    attention_mask = batch[\"attention_mask\"].to(DEVICE)\n",
    "    labels = batch[\"labels\"].to(DEVICE)\n",
    "    with torch.no_grad():\n",
    "        # Pass data through model  \n",
    "        output = trainer.model(input_ids, \n",
    "                               attention_mask\n",
    "                               )\n",
    "        # Logit.size: [batch_size, sequence_length, classes]\n",
    "        predicted_label = torch.argmax(output.logits, \n",
    "                                       axis=-1\n",
    "                                       ).cpu().numpy()\n",
    "        \n",
    "    # Calculate loss per token after flattening batch dimension with view\n",
    "    loss = cross_entropy(output.logits.view(-1, 9), \n",
    "                         labels.view(-1), \n",
    "                         reduction=\"none\")\n",
    "    # Unflatten batch dimension and convert to numpy array\n",
    "    loss = loss.view(len(input_ids), -1).cpu().numpy()\n",
    "\n",
    "    return {\"loss\":loss, \"predicted_label\": predicted_label}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Apply Above Function to Entire Evaluation Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Parameter 'function'=<function forward_pass_with_label at 0x7fbbce8dd160> of the transform datasets.arrow_dataset.Dataset._map_single couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.\n",
      "Loading cached processed dataset at /Users/briandunn/.cache/huggingface/datasets/Babelscape___parquet/Babelscape--wikineural-579d1dc98d2a6b93/0.0.0/14a00e99c0d15a23649d0db8944380ac81082d4b021f398733dd84f3a6c569a7/cache-1c80317fa3b1799d.arrow\n"
     ]
    }
   ],
   "source": [
    "eval_set = encoded_ds['eval']\n",
    "\n",
    "eval_set = eval_set.map(forward_pass_with_label,\n",
    "                        batched=True,\n",
    "                        batch_size=32)\n",
    "\n",
    "eval_df = eval_set.to_pandas()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Clean Up Padding Tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>input_ids</th>\n",
       "      <th>attention_mask</th>\n",
       "      <th>labels</th>\n",
       "      <th>loss</th>\n",
       "      <th>predicted_label</th>\n",
       "      <th>input_tokens</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[0, 1405, 825, 7, 9688, 6486, 257, 21, 847, 76...</td>\n",
       "      <td>[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...</td>\n",
       "      <td>[IGN, O, O, O, B-LOC, IGN, IGN, O, O, O, O, O,...</td>\n",
       "      <td>[0.0, 1.8596476e-05, 2.0861407e-05, 3.921909e-...</td>\n",
       "      <td>[O, O, O, O, B-LOC, I-LOC, I-LOC, O, O, O, O, ...</td>\n",
       "      <td>[&lt;unk&gt;, ▁thing, ▁On, ▁t, ▁Environmental, ▁plat...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           input_ids  \\\n",
       "0  [0, 1405, 825, 7, 9688, 6486, 257, 21, 847, 76...   \n",
       "\n",
       "                                      attention_mask  \\\n",
       "0  [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...   \n",
       "\n",
       "                                              labels  \\\n",
       "0  [IGN, O, O, O, B-LOC, IGN, IGN, O, O, O, O, O,...   \n",
       "\n",
       "                                                loss  \\\n",
       "0  [0.0, 1.8596476e-05, 2.0861407e-05, 3.921909e-...   \n",
       "\n",
       "                                     predicted_label  \\\n",
       "0  [O, O, O, O, B-LOC, I-LOC, I-LOC, O, O, O, O, ...   \n",
       "\n",
       "                                        input_tokens  \n",
       "0  [<unk>, ▁thing, ▁On, ▁t, ▁Environmental, ▁plat...  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "id2label[-100] = \"IGN\"\n",
    "eval_df[\"input_tokens\"] = eval_df[\"input_ids\"].apply(\n",
    "    lambda x: tokenizer.convert_ids_to_tokens(x))\n",
    "eval_df[\"predicted_label\"] = eval_df[\"predicted_label\"].apply(\n",
    "    lambda x: [id2label[i] for i in x])\n",
    "eval_df[\"labels\"] = eval_df[\"labels\"].apply(\n",
    "    lambda x: [id2label[i] for i in x])\n",
    "eval_df['loss'] = eval_df.apply(\n",
    "    lambda x: x['loss'][:len(x['input_ids'])], axis=1)\n",
    "eval_df['predicted_label'] = eval_df.apply(\n",
    "    lambda x: x['predicted_label'][:len(x['input_ids'])], axis=1)\n",
    "eval_df.head(1)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Unwrap Each Token Within Sample Separately"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>input_ids</th>\n",
       "      <th>attention_mask</th>\n",
       "      <th>labels</th>\n",
       "      <th>loss</th>\n",
       "      <th>predicted_label</th>\n",
       "      <th>input_tokens</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1405</td>\n",
       "      <td>1</td>\n",
       "      <td>O</td>\n",
       "      <td>0.0</td>\n",
       "      <td>O</td>\n",
       "      <td>▁thing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>825</td>\n",
       "      <td>1</td>\n",
       "      <td>O</td>\n",
       "      <td>0.0</td>\n",
       "      <td>O</td>\n",
       "      <td>▁On</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7</td>\n",
       "      <td>1</td>\n",
       "      <td>O</td>\n",
       "      <td>0.0</td>\n",
       "      <td>O</td>\n",
       "      <td>▁t</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>9688</td>\n",
       "      <td>1</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>0.0</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>▁Environmental</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>21</td>\n",
       "      <td>1</td>\n",
       "      <td>O</td>\n",
       "      <td>0.0</td>\n",
       "      <td>O</td>\n",
       "      <td>it</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>847</td>\n",
       "      <td>1</td>\n",
       "      <td>O</td>\n",
       "      <td>0.0</td>\n",
       "      <td>O</td>\n",
       "      <td>▁list</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>765</td>\n",
       "      <td>1</td>\n",
       "      <td>O</td>\n",
       "      <td>0.0</td>\n",
       "      <td>O</td>\n",
       "      <td>ci</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  input_ids attention_mask labels  loss predicted_label    input_tokens\n",
       "0      1405              1      O   0.0               O          ▁thing\n",
       "0       825              1      O   0.0               O             ▁On\n",
       "0         7              1      O   0.0               O              ▁t\n",
       "0      9688              1  B-LOC   0.0           B-LOC  ▁Environmental\n",
       "0        21              1      O   0.0               O              it\n",
       "0       847              1      O   0.0               O           ▁list\n",
       "0       765              1      O   0.0               O              ci"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "eval_df_tokens = eval_df.apply(pd.Series.explode)\n",
    "eval_df_tokens = eval_df_tokens.query(\"labels != 'IGN'\")\n",
    "eval_df_tokens[\"loss\"] = eval_df_tokens[\"loss\"].astype(float).round(2)\n",
    "eval_df_tokens.head(7)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### See Which Tokens Have Accumulated Most Loss in Evaluation Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>input_tokens</th>\n",
       "      <td>録</td>\n",
       "      <td>▁c</td>\n",
       "      <td>rou</td>\n",
       "      <td>pp</td>\n",
       "      <td>▁(</td>\n",
       "      <td>in</td>\n",
       "      <td>les</td>\n",
       "      <td>▁J</td>\n",
       "      <td>▁bis</td>\n",
       "      <td>tern</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>104</td>\n",
       "      <td>2491</td>\n",
       "      <td>219</td>\n",
       "      <td>486</td>\n",
       "      <td>198</td>\n",
       "      <td>7358</td>\n",
       "      <td>118</td>\n",
       "      <td>146</td>\n",
       "      <td>10</td>\n",
       "      <td>114</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>1.888</td>\n",
       "      <td>0.056</td>\n",
       "      <td>0.53</td>\n",
       "      <td>0.21</td>\n",
       "      <td>0.446</td>\n",
       "      <td>0.012</td>\n",
       "      <td>0.608</td>\n",
       "      <td>0.485</td>\n",
       "      <td>6.236</td>\n",
       "      <td>0.501</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sum</th>\n",
       "      <td>196.39</td>\n",
       "      <td>139.38</td>\n",
       "      <td>116.06</td>\n",
       "      <td>102.01</td>\n",
       "      <td>88.21</td>\n",
       "      <td>87.72</td>\n",
       "      <td>71.78</td>\n",
       "      <td>70.74</td>\n",
       "      <td>62.36</td>\n",
       "      <td>57.17</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   0       1       2       3      4      5      6      7  \\\n",
       "input_tokens       録      ▁c     rou      pp     ▁(     in    les     ▁J   \n",
       "count            104    2491     219     486    198   7358    118    146   \n",
       "mean           1.888   0.056    0.53    0.21  0.446  0.012  0.608  0.485   \n",
       "sum           196.39  139.38  116.06  102.01  88.21  87.72  71.78  70.74   \n",
       "\n",
       "                  8      9  \n",
       "input_tokens   ▁bis   tern  \n",
       "count            10    114  \n",
       "mean          6.236  0.501  \n",
       "sum           62.36  57.17  "
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(\n",
    "    eval_df_tokens.groupby(\"input_tokens\")[[\"loss\"]]\n",
    "    .agg([\"count\", \"mean\", \"sum\"])\n",
    "    .droplevel(level=0, axis=1)  # Get rid of multi-level columns\n",
    "    .sort_values(by=\"sum\", ascending=False)\n",
    "    .reset_index()\n",
    "    .round(3)\n",
    "    .head(10)\n",
    "    .T\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### See Which Label IDs Have Most Loss in Evaluation Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>labels</th>\n",
       "      <td>B-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>B-ORG</td>\n",
       "      <td>B-PER</td>\n",
       "      <td>I-LOC</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>I-ORG</td>\n",
       "      <td>I-PER</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>4173</td>\n",
       "      <td>4555</td>\n",
       "      <td>3512</td>\n",
       "      <td>5593</td>\n",
       "      <td>2370</td>\n",
       "      <td>5904</td>\n",
       "      <td>2963</td>\n",
       "      <td>4107</td>\n",
       "      <td>234662</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.285</td>\n",
       "      <td>0.273</td>\n",
       "      <td>0.264</td>\n",
       "      <td>0.188</td>\n",
       "      <td>0.164</td>\n",
       "      <td>0.157</td>\n",
       "      <td>0.15</td>\n",
       "      <td>0.034</td>\n",
       "      <td>0.012</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sum</th>\n",
       "      <td>1189.15</td>\n",
       "      <td>1244.76</td>\n",
       "      <td>926.57</td>\n",
       "      <td>1051.03</td>\n",
       "      <td>387.71</td>\n",
       "      <td>927.17</td>\n",
       "      <td>444.84</td>\n",
       "      <td>139.22</td>\n",
       "      <td>2786.6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              0        1       2        3       4       5       6       7  \\\n",
       "labels   B-MISC   I-MISC   B-ORG    B-PER   I-LOC   B-LOC   I-ORG   I-PER   \n",
       "count      4173     4555    3512     5593    2370    5904    2963    4107   \n",
       "mean      0.285    0.273   0.264    0.188   0.164   0.157    0.15   0.034   \n",
       "sum     1189.15  1244.76  926.57  1051.03  387.71  927.17  444.84  139.22   \n",
       "\n",
       "             8  \n",
       "labels       O  \n",
       "count   234662  \n",
       "mean     0.012  \n",
       "sum     2786.6  "
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(\n",
    "    eval_df_tokens.groupby(\"labels\")[[\"loss\"]] \n",
    "    .agg([\"count\", \"mean\", \"sum\"])\n",
    "    .droplevel(level=0, axis=1)\n",
    "    .sort_values(by=\"mean\", ascending=False)\n",
    "    .reset_index()\n",
    "    .round(3)\n",
    "    .fillna(0)\n",
    "    .T\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create Function to Display Confusion Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay\n",
    "\n",
    "def plot_confusion_matrix(y_preds, y_true, labels):\n",
    "    cm = confusion_matrix(y_true, y_preds, normalize=\"true\")\n",
    "    fig, ax = plt.subplots(figsize=(10, 10))\n",
    "    disp = ConfusionMatrixDisplay(confusion_matrix=cm, display_labels=labels)\n",
    "    disp.plot(cmap=\"Blues\", values_format=\".2f\", ax=ax, colorbar=False)\n",
    "    plt.title(\"Normalized Confusion Matrix\")\n",
    "    plt.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Display Confusion Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA2QAAANXCAYAAABaBpzLAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAADGL0lEQVR4nOzdd1xV9R/H8fdFBAdDQQhFEnHhtsxVKWiae5Tm3mmapi1nWmqusmnlNvdMS8uRuUc/y70qR6aYGwQBcaDI/f2BXr1wUal7OQSv5+NxH8W533Pu9/vxy73nfc/AZDabzQIAAAAApDsnozsAAAAAAFkVgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDADgUKGhoQoNDbX8HBYWJpPJpFmzZqVrPzp37qzAwMB0fc1/au7cuQoODlb27NmVJ08eu29/+PDhMplMdt/uf5VRcxIAJAIZABhu1qxZMplMypEjh86ePZvi+dDQUJUpU8aAnmVty5YtU/369ZUvXz65uLioQIECatmypTZu3OjQ1z1y5Ig6d+6sIkWKaNq0aZo6dapDXy+9mUwmmUwmdevWzebzQ4YMsbS5dOlSmre/evVqDR8+/F/2EgDSD4EMADKI+Ph4ffDBB0Z3w+EKFSqk69evq0OHDkZ3xSaz2awuXbroxRdf1MWLF/XWW29p8uTJ6t27t06cOKHnnntO27dvd9jrb968WYmJiRo/frw6d+6sli1b2v01hg4dquvXr9t9u48qR44c+vbbb3Xz5s0Uzy1cuFA5cuT4x9tevXq1RowYkaZ1MvqcBJC5EcgAIIOoUKGCpk2bpnPnzjnsNcxms6E74pIsRwOzZctmaD9S88knn2jWrFl64403tGfPHr3zzjvq2rWrhgwZot27d2vOnDlydnZ22OuHh4dLkkNOVbzL2dn5X4Wef6tevXqKjY3Vjz/+aLV8+/btOnnypBo2bJgu/UhISNDNmzcz/JwEkLkRyAAgg3jnnXd0+/btRzpKlpCQoJEjR6pIkSJydXVVYGCg3nnnHcXHx1u1CwwMVKNGjfTTTz/pqaeeUs6cOTVlyhRt3rxZJpNJ33zzjUaMGCF/f3+5u7urRYsWiomJUXx8vN544w35+vrKzc1NXbp0SbHtmTNnqlatWvL19ZWrq6tKlSqlSZMmPbTvya/XudsXW4/k13z9+OOPql69unLnzi13d3c1bNhQv//+e4rXWL58ucqUKaMcOXKoTJkyWrZs2UP7JUnXr1/X2LFjFRwcrI8//tjmdVYdOnRQ5cqVLT+fOHFCL730kry8vJQrVy5VrVpVq1atslrn/nqPHj1aBQsWVI4cOfTcc8/p+PHjlnaBgYEaNmyYJMnHx0cmk8ly+t39/3+/wMBAde7c2fLzrVu3NGLECBUrVkw5cuSQt7e3nn32Wa1bt87SxtY1ZGmdUz///LMqV66sHDlyKCgoSHPmzHlwce/j7++vGjVqaMGCBVbL58+fr7Jly9o8RXfbtm166aWX9Pjjj8vV1VUBAQF68803rb5g6Ny5syZMmGCp192HdG/effzxx/r8888t4/zjjz9SzMnw8HD5+PgoNDRUZrPZsv3jx48rd+7catWq1SOPFQAexnFf8QEA0qRw4cLq2LGjpk2bpkGDBqlAgQKptu3WrZtmz56tFi1a6O2339aOHTs0duxYHT58OEX4OHr0qNq0aaMePXqoe/fuKlGihOW5sWPHKmfOnBo0aJCOHz+uL7/8UtmzZ5eTk5MuX76s4cOH69dff9WsWbNUuHBhvffee5Z1J02apNKlS6tJkyZydnbWihUr1KtXLyUmJqp3796PPO6SJUtq7ty5Vsuio6P11ltvydfX17Js7ty56tSpk+rWrasPP/xQ165d06RJk/Tss89q3759lvC2du1aNW/eXKVKldLYsWMVGRmpLl26qGDBgg/ty88//6yoqCi98cYbj3S05OLFi3r66ad17do19e3bV97e3po9e7aaNGmipUuX6oUXXrBq/8EHH8jJyUn9+vVTTEyMxo0bp3bt2mnHjh2SpM8//1xz5szRsmXLNGnSJLm5ualcuXIP7cf9hg8frrFjx6pbt26qXLmyYmNjtXv3bu3du1d16tRJdb20zKnjx4+rRYsWevnll9WpUyfNmDFDnTt3VsWKFVW6dOlH6mfbtm31+uuvKy4uTm5ubkpISNCSJUv01ltv6caNGynaL1myRNeuXdOrr74qb29v7dy5U19++aXOnDmjJUuWSJJ69Oihc+fOad26dSnm1F0zZ87UjRs39Morr8jV1VVeXl5KTEy0auPr66tJkybppZde0pdffqm+ffsqMTFRnTt3lru7uyZOnPhIYwSAR2IGABhq5syZZknmXbt2mf/66y+zs7OzuW/fvpbnQ0JCzKVLl7b8vH//frMkc7du3ay2069fP7Mk88aNGy3LChUqZJZkXrNmjVXbTZs2mSWZy5QpY75586ZleZs2bcwmk8lcv359q/bVqlUzFypUyGrZtWvXUoylbt265qCgIKtlISEh5pCQEMvPJ0+eNEsyz5w502Y9EhMTzY0aNTK7ubmZf//9d7PZbDZfuXLFnCdPHnP37t2t2l64cMHs6elptbxChQrm/Pnzm6Ojoy3L1q5da5aUYgzJjR8/3izJvGzZsge2u+uNN94wSzJv27bNsuzKlSvmwoULmwMDA823b982m8336l2yZElzfHx8itc7dOiQZdmwYcPMkswRERFWryXJPGzYsBR9KFSokLlTp06Wn8uXL29u2LDhA/t99zXu+idzauvWrZZl4eHhZldXV/Pbb7/9wNe9O47evXubo6KizC4uLua5c+eazWazedWqVWaTyWQOCwuzWQNb823s2LFmk8lkPnXqlGVZ7969zbZ2b+7OOw8PD3N4eLjN55LPyTZt2phz5cplPnbsmPmjjz4ySzIvX778oWMEgLTglEUAyECCgoLUoUMHTZ06VefPn7fZZvXq1ZKkt956y2r522+/LUkpTpcrXLiw6tata3NbHTt2VPbs2S0/V6lSRWazWV27drVqV6VKFZ0+fVoJCQmWZTlz5rT8f0xMjC5duqSQkBCdOHFCMTExDxtqqkaOHKmVK1dq1qxZKlWqlCRp3bp1io6OVps2bXTp0iXLI1u2bKpSpYo2bdokSTp//rz279+vTp06ydPT07LNOnXqWLb1ILGxsZIkd3f3R+rr6tWrVblyZT377LOWZW5ubnrllVcUFhamP/74w6p9ly5d5OLiYvm5evXqkpJOe7SXPHny6Pfff9eff/75yOukdU6VKlXK0ncp6fTKEiVKpGkcefPmVb169bRw4UJJ0oIFC/T000+rUKFCNtvfP9+uXr2qS5cu6emnn5bZbNa+ffse+XWbN28uHx+fR2r71VdfydPTUy1atNC7776rDh06qGnTpo/8WgDwKAhkAJDBDB06VAkJCaleS3bq1Ck5OTmpaNGiVsv9/PyUJ08enTp1ymp54cKFU32txx9/3OrnuyEmICAgxfLExESroPW///1PtWvXVu7cuZUnTx75+PjonXfekaR/HMjWrFmjESNGaPDgwWrevLll+d1wUatWLfn4+Fg91q5da7kRxt2xFytWLMW27z9VMzUeHh6SpCtXrjxSf0+dOmVzuyVLlrTqz13J6503b15J0uXLlx/p9R7F+++/r+joaBUvXlxly5ZV//79dfDgwQeuk9Y5lXwcUtJY0jqOtm3bat26dfr777+1fPlytW3bNtW2f//9tzp37iwvLy+5ubnJx8dHISEhktI23x70+5Ccl5eXvvjiCx08eFCenp764osvHnldAHhUXEMGABlMUFCQ2rdvr6lTp2rQoEGptnvUP+x7/5GF5FK7Tiq15eY7Nzj466+/9Nxzzyk4OFiffvqpAgIC5OLiotWrV+uzzz5LcU3Oozh58qTatWunOnXqaNSoUVbP3d3e3Llz5efnl2Jde931MDg4WJJ06NAhNWvWzC7bvN/D6vpP3L592+rnGjVq6K+//tL333+vtWvXavr06frss880efLkVP/2112POqfsNY4mTZrI1dVVnTp1Unx8fKq3+L99+7bq1KmjqKgoDRw4UMHBwcqdO7fOnj2rzp07p2m+Pej3wZaffvpJUlJoPnPmjEPvfgkgayKQAUAGNHToUM2bN08ffvhhiucKFSqkxMRE/fnnn5YjMVLSDSaio6NTPeXLnlasWKH4+Hj98MMPVkdL7p46mFbXr1/Xiy++qDx58mjhwoVycrI+gaNIkSKSkm62ULt27VS3c3fstk7XO3r06EP78eyzzypv3rxauHCh3nnnnYfe2KNQoUI2t3vkyBGr/thD3rx5FR0dbbXs5s2bNk9t9fLyUpcuXdSlSxfFxcWpRo0aGj58eKqBzKg5lTNnTjVr1kzz5s2z/BFuWw4dOqRjx45p9uzZ6tixo2X5/XeOvOtRQ+WjWLNmjaZPn64BAwZo/vz56tSpk3bs2OHQP3sAIOvhlEUAyICKFCmi9u3ba8qUKbpw4YLVcw0aNJCUdEe++3366aeSlC5/w+luULn/iEhMTIxmzpz5j7bXs2dPHTt2TMuWLbOcxne/unXrysPDQ2PGjNGtW7dSPB8RESFJyp8/vypUqKDZs2dbnca2bt26FNdz2ZIrVy4NHDhQhw8f1sCBA20e8Zk3b5527twpKenfYufOnfrll18sz1+9elVTp05VYGDgI1239qiKFCmirVu3Wi2bOnVqiiNkkZGRVj+7ubmpaNGiKW5ffz8j51S/fv00bNgwvfvuu6m2sTXfzGazxo8fn6Jt7ty5JSlFeE2r6Ohoy50qx4wZo+nTp2vv3r0aM2bMv9ouACTHVzwAkEENGTJEc+fO1dGjR61uJV6+fHl16tRJU6dOVXR0tEJCQrRz507Nnj1bzZo1U82aNR3et+eff14uLi5q3LixevToobi4OE2bNk2+vr6p3owkNatWrdKcOXPUvHlzHTx40Op6Jzc3NzVr1kweHh6aNGmSOnTooCeffFKtW7eWj4+P/v77b61atUrPPPOMvvrqK0lJt/Jv2LChnn32WXXt2lVRUVH68ssvVbp0acXFxT20P/3799fvv/+uTz75RJs2bVKLFi3k5+enCxcuaPny5dq5c6e2b98uSRo0aJAWLlyo+vXrq2/fvvLy8tLs2bN18uRJffvttymO9P0b3bp1U8+ePdW8eXPVqVNHBw4c0E8//ZTiqFKpUqUUGhqqihUrysvLS7t379bSpUv12muvpbptI+dU+fLlVb58+Qe2CQ4OVpEiRdSvXz+dPXtWHh4e+vbbb21es1axYkVJUt++fVW3bl1ly5ZNrVu3TnO/Xn/9dUVGRmr9+vXKli2b6tWrp27dumnUqFFq2rTpQ/sMAI+KQAYAGVTRokXVvn17zZ49O8Vz06dPV1BQkGbNmqVly5bJz89PgwcPtvxRYUcrUaKEli5dqqFDh6pfv37y8/PTq6++Kh8fnxR3aHyYu0e3vv32W3377bdWzxUqVMhyLVfbtm1VoEABffDBB/roo48UHx8vf39/Va9eXV26dLGsU69ePS1ZskRDhw7V4MGDVaRIEc2cOVPff/+9Nm/e/ND+ODk5ac6cOWratKmmTp2qjz/+WLGxsfLx8VGNGjU0btw4VatWTZL02GOPafv27Ro4cKC+/PJL3bhxQ+XKldOKFSvsflSpe/fuOnnypL7++mutWbNG1atX17p16/Tcc89Ztevbt69++OEHrV27VvHx8SpUqJBGjRql/v37P3D7Rs+pB8mePbtWrFihvn37auzYscqRI4deeOEFvfbaaymC0Ysvvqg+ffpo0aJFmjdvnsxmc5oD2Q8//KA5c+bok08+sVxXKCUdMVy3bp06deqkXbt2Wd2hFAD+KZP531xJDAAAAAD4x7iGDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACD8HfIHCgxMVHnzp2Tu7u7TCaT0d0BAAAAkA7MZrOuXLmiAgUKyMnpwcfACGQOdO7cOQUEBBjdDQAAAAAGOH36tAoWLPjANgQyB3J3d5ckuVR+UyZnV4N7k3mc/H6Q0V3IlDiGi/8KJydmKwDYU8LtRKO7kOlcuRKr4CKFLHngQQhkDnT3NEWTs6tMzjkM7k3m4eHhYXQXMiV2cfFfQSADAPsikDnOo1y2xE09AAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEsE+vWtJIOzH9d538conVfvawnSxRIta1zNif171BDe+f20fkfh2jb1B56rlKRVNu/0foZXd4wTGN61XVE1zOsr5ds1RPNhsm/+pt6vuvH2vt72APbf79hn6q2HCn/6m+qetsxWve/362eX7lpv1r0maBidQYqX5U+OnTsjAN7n3FNX7JVFZoNU4Hqb6pO14+15xHqWqXlSBWo/qaetVHXFZv2q3mfCSpaZ6C8s2hdqaljTPtmi8o1eU9+z7yh2p0/emhdl6/fq8otRsrvmTf0dOvRWpusrmazWWMmr1RwvXeU/9k31azXl/rr73AHjiDjoaaOQV0dg7ra39dLt+rJZsNVsMZbqtv1E+39/dQD23+/YZ+qtRqlgjXeUo12Y7Vu+72a3kq4rfe/+l412o1VodB+KtNoqHqPmKsLETGOHsa/QiDLpF4ILa1RPZ/Xh3O2KLTnFP3210V9+2F75cuTy2b7oV1rqXOjihr45Y+q2nWCZq7Yo7kjWqlsUb8UbZ8oUUCdG1XUb39dcPQwMpRl6/bo3fHL1P/l+to4e4BKF/XXS69PVETUFZvtdx48oVfenaV2jatp05yBalCjnDoOmKbDf52ztLl2/aaqlA/Se681Ta9hZDjJ61rmEera/d1Zan9fXTvYqGvV8kEalkXrSk0d47u1ezT082Ua2K2+Ns8dqDLF/NW8z4RU67rjwAl1GzpL7ZtW05Z5g9QwpLza95uqP47fq+v4Oes1ZfEWfTq4tdbN7KdcOV3UvM8E3Yi/lV7DMhQ1dQzq6hjU1f6Wrdur98YvU79u9bRhdn+VLuavlm88+POqx3uz1a5xNW2cPUD1a5RTpwHTLZ9X12/c1MGjZ/RWl7raMLu/Zn3wso6fClf7/lPTc1hp9p8KZJ07d1azZs1SfT40NFRvvPFGqs9HRUXpjTfeUKFCheTi4qICBQqoa9eu+vvvv1O0vXDhgvr06aOgoCC5uroqICBAjRs31oYNG+wwEsfr1aKq5qzeqwU/7dfRU5f01ucrdS3+ltrXe8Jm+5a1y+mzBT9r3c7jOnU+WjNW7Na6HX/qtZeqWbXLnSO7pr7zol7/dIWir9xIj6FkGJMWblKHptXUtnFVlQjKr08GtVLOHC5asOIXm+2nLN6sWlVLqk+H2ipe2E+DezZSuRIBmr5kq6VNywaV1b9bfYVUKpFew8hwJt6pa7vGVRV8X13nP6Cuz92pa4nCfnrHRl1bZfG6UlPHmLhgozo2e1rtmlRTcFB+fTq4tXLlcNG8H1Kp66LNeq5aSfW9U9chrzZS+eAATVuyRVLSN+OTF25Sv6511SCknMoU89ekER114VKMVm05kJ5DMww1dQzq6hjU1f4mL9yk9k2fVttGVVWicH59PLBl0r7Vyl9ttp+6eItqVS2p19o/l7Rv1aOhypUoqK+XbpMkebjl1NIve6tZ7SdVtNBjeqpMYX3Qr4UOHDmtMxei0nNoafKfCmT/RlRUlKpWrar169dr8uTJOn78uBYtWqTjx4+rUqVKOnHihKVtWFiYKlasqI0bN+qjjz7SoUOHtGbNGtWsWVO9e/c2cBSPJruzkyoUL6DNe++NyWyWtuw9oUqlCtpcx9Ulm27cTLBaduNmgqqWedxq2UevN9DaX//Ulr0n7d/xDOzmrQQdOHJaIZXv7Yw6OTkppFIJ7ToUZnOd3YfCUuy81qwarN2HslbtHuSf1HWXjbrWqhqsXdRVEjV1lJu3ErT/yGmFJq9r5RKp1mnnoZMKrRRstaxW1ZKWf4dTZyN1MTJWoZXvtfF0y6mKpQO162CY3ceQ0VBTx6CujkFd7e/mrQQdOHra6vPHyclJNSqVSHVfafdvYapRqbjVsppVSz5w3yo27oZMJpM83XPap+MO4Gx0B9LLkCFDdO7cOR0/flx+fkmn4T3++OP66aefVKxYMfXu3Vs//vijJKlXr14ymUzauXOncufObdlG6dKl1bVrV0P6nxbenrnknM1JEZevWi2PuHxVxQLy2Vxn466/1KtFVW0/eEonz0Up5MkgNXq2pLI5mSxtXqxZWuWL5letXtMc2v+MKDL6qm7fTpSPl4fVch8vd/156qLNdcIjY+Xj5W61zNfLXeGRtg/DZ0V36+qbrK6+aayrD3W1oKaOERkdd+c9IHmdPPRn2APq6m2rrrGSpIt3/pu8ja/3vTaZGTV1DOrqGNTV/qIs+1bJxp/XXccfUNPkn28+eVP/vLoRf0vvT/heL9Z5Uu65M24gyxJHyBITE7Vo0SK1a9fOEsbuypkzp3r16qWffvpJUVFRioqK0po1a9S7d2+rMHZXnjx5Un2d+Ph4xcbGWj3+KwZNWKMTZ6O0c2Zvhf/0rsb1qa8FP+1XotksSfL38dDY3vX0ytjvFH/rtsG9BQAAAFJ3K+G2ug2ZKbNZ+mhgS6O780BZ4ghZRESEoqOjVbJkSZvPlyxZUmazWcePH5eUdE5vcHCwzbYPMnbsWI0YMeJf9dUeImOuKeF2onzyWgdKn7y5FR4Vl+o67d9bLNfs2eTlmUvnL13R8O61FXb+siSpfPH88s3rps2Te1jWcc7mpKfLFVL3ZpX1WL1RSkw0O25QBvPOk1vZsjkpIso6ZEdEXUnxTc1dvt4eKS5KDY+6It9k34RlZXfrGp6sruFprGsEdbWgpo7hncftzntA8jrFytf7AXWNtFXXpPaP3flvROQV+eXztLQJj7yissVtn16emVBTx6CujkFd7c/Lsm+VbF/pcuqfP77eHik+3yJstL8bxs5ciNJ3E/pk6KNj0n/0CNn8+fPl5uZmeWzbtu2R1jObHx4YHqVNagYPHqyYmBjL4/Tp0/94W//GrYRE7T92TiFPBFmWmUxSjSeCtOuPB9+qOv7WbZ2/dEXO2ZzUuHpJ/bj9qCRp696TevrliarxymTLY++Rs1qy4aBqvDI5U4cxSXLJ7qzywQHauuuYZVliYqK27jqmSmUDba7zVNlAbd19zGrZlp1H9VTZwo7s6n/KP6lrJRt13bzzqCpRV0nU1FFcsjurQnCAtuw6all2r66261S5bGGr9pK0accRy79DIX9vPebtYdUmNu669vwepkrlAu0+hoyGmjoGdXUM6mp/LtmdVb5Eys+rbbtS31d6qkygtu1Kvm91xKr93TB24nSEln7ZW16eKc94y2j+k0fImjRpoipVqlh+9vf3f2B7Hx8f5cmTR4cPH7b5/OHDh2UymVS0aFFJkslk0pEjR9LcL1dXV7m6uqZ5PUeYuPRXTRzYTPuOndPeI2f1avOqyp0ju+b/tF+SNGlgM52/dEXvf51018iKwf7Kn89dh/66oAL5PDSwY4icTCaNX/Q/SVLc9Zs6HBZh9RrXbtxSVOz1FMszq1fb1NRr789ThZKP68lShTR50WZduxGvNo2qSpJ6DZ+j/D559G7vJpKkHq1C1aTneE2Yv0HPP1Na363bq/2H/9ang1tbtnk55qrOXLxs+fsYx+9c4+Pr7WH55iyz69WmpnrfV9cpd+ra9k5dX71T1/fuq2vjO3Wt80xpLbtT18+oqwU1dYxebWup14i5eqLk43qydKAmLdykq9fj1a5xUl17Dpuj/D6elj8N0KN1qBr1+Fxfzdug558tre/W7tH+w3/r83faSEr6rOnZpqY+nrFGQQE+KuTvrTGTV8kvn6cahpQ3bJzpiZo6BnV1DOpqfz3b1FSfkfNUoWRA0ufV4s26duOm2jRM2s/vPWKu/Hw89W6vpM+rV1qFqOmrX2ji/I13Pq/2aP/h0/pkUNLn1a2E2+o6+GsdPHpG8z/poduJZsu1enk9cskle8aMPhmzVw/h7u4ud/dHP5XGyclJLVu21Pz58/X+++9bXUd2/fp1TZw4UXXr1pWXl5ckqW7dupowYYL69u2b4jqy6OjoB15HllEs2/y78nnm0judQ+Wb102H/rqgFoPmW270UdDX03J9mCS5ujhrSNdaCsyfV1ev39S6HX+q5wfLFHs13qghZDgv1KmoyOg4fTB1lcIjr6hMcX9983kvy6kHZy5eltN9N0GpXC5IU0Z21pjJKzV60koFBfhozrjuKlnk3h/oXrPtkPqMnG/5ufvQWZKk/t3qa2D3BukzMIO9UKeiLj2grmdt1HXqyM4aPXmlRt2p69xkdf0xWV273anrgCxSV2rqGC8+n1TXMVNW3TmlyF9Lv+h97z3gQpScTPfqWqV8kKaN6qzRk1Zq5MQVCgrw0byPX1Gpovfq+nrH2rp2PV5vjlmomLjrqlq+iJZ+0Us5XLOn+/iMQE0dg7o6BnW1vxfqPKnI6Dh9OG21wiNjVaZYQS3+7NX7anpZJpP159Xk9ztp7JRVGj15hYICfDV7XDfL59X58Git2fabJKlmhw+tXmv5hD56pmKxdBpZ2pjM/+YcvXTWuXNnRUdHa/ny5TafDw0Nlb+/v/r372+1PH/+/HJ2dlaVKlWUM2dOjRs3TmXKlNHJkyc1dOhQHT16VL/88ouCgpJO8Ttx4oSeeeYZeXl56f3331e5cuWUkJCgdevWadKkSakeaUsuNjZWnp6ecn16kEzOOf7V2HHPpXXvGd2FTMn08CZAhnB/mAQA/HsJtxON7kKmExsbK3/fvIqJiZGHx4PPJPlPXkP2IAsWLNATTzxh9Zg2bZq8vb3166+/qmbNmurRo4eKFCmili1bqkiRItq1a5cljElSUFCQ9u7dq5o1a+rtt99WmTJlVKdOHW3YsEGTJk0ycHQAAAAAMpP/1BGy/xqOkDkGR8gcg2MO+K/gCBkA2BdHyOwvSx8hAwAAAID/CgIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGcTa6A1nB8WUD5eHhYXQ3Mo1ifZcZ3YVM6dj4ZkZ3IVNKSDQb3YVMJ4dTNqO7kCldi08wuguZTi5XdrPw33A1/rbRXch0rqWhphwhAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDOBvdATjOjKVbNXH+RoVHxapUUX+NeauFnixdKNX2P2zYpw+nrtLpC1EqXNBH7/ZuotpPl7Y8/9H01Vq+bq/OhkfLJXs2lSsRoME9G6li6cB0GE3G0KFGkLo/V0w+Hjl0+GyMhi85oIOnLttsu+D16qpazCfF8k2/XdDLk7dLkvK5u2pA0zKqXtJXHjmza+fxSI1Ysl9hEVcdOo6M5uulWzVhXtJcLV3UX2PffvBc/X7DPn0wdZVOn49SUEDSXK1zZ67eSritsZNXav0vf+jU2Ui5u+VQSKUSerdXE/n5eKbXkAw389ttmjh/oyLu/P6Pfqu5niiVek1XbNynD6eu1pk7v/9DezXWc/f9/t9vwLjFmrt8u0a8/oJeaRXqoBFkTNO+2aIv521QeGSsyhTz14f9X3rge+Dy9Xs1ZvIq/X0+UkEBPhrep5mef+ZeXc1ms8ZOWaU5y7crJu66qpQL0ieDWqnI477pMJqMYda32zRp4UZFRF1RqSIFNPLNh83V/fpo+r25+s6rjfVctVKW598YPV9LftxltU5o5WDN/7Snw8aQETFXHYO62t+cZT9ryqKk94CSRQpoxOsvqkLJ1N8DVm3ar09m/Jj0HuDvo0E9G6lm1XvvAYEhb9pcb3DPxurRppbd+28PHCFLxenTp9W1a1cVKFBALi4uKlSokF5//XVFRkYa3bVHsnz9Xg37Ypnefrme1s3qr9LF/NX6zYmKiLpis/2ugyfUc9hstW1cTetnD1D9GuXUeeB0Hf7rnKVNUICvxrz9kjbPG6QfJr+hgPxeavX6RF26bHubmU3DJ/31zgtl9cWPR9T4w406fDZGs3s/I283V5vtX532qyoPXmV51B21Tgm3E7V63xlLm8mvVNXj+XKrx5Rf1eiDjTobdU1z+1RXTpds6TUswy1bt1fvjV+mft3qacPspLna8o3U5+rOgyfU473Zate4mjbemaudBtybq9dv3NTBo2f0Vpe62jC7v2Z98LKOnwpX+/5T03NYhvp+/V4N/2KZ3u5aVz/N7K9SRQuozZuTdCm13/9DJ/XqsDlq27iq1s7qr3o1yqrLoK915L7f/7tWbzmgvb+fkl++rBNu7/pu7R4N/XyZBnarr81zB6pMMX817zMh1bm648AJdRs6S+2bVtOWeYPUMKS82vebqj+O36vr+DnrNWXxFn06uLXWzeynXDld1LzPBN2Iv5VewzLU9xv2asRXy/VWl3pa83U/lSrqr3ZvTU71c2XXoZPqPWKO2jSqqp9m9FPd6mX18uCvdeTEeat2NasEa9/371seE4Z3TI/hZBjMVcegrva3YuM+jZqwXK93qqtV095WqSIF1LHflFTfA/b8dlJ9R85VqwZVtHpaPz1fvYxeGTJDR+97D9j53Qirx7iBrWUymVQ/pFx6DSvNCGQ2nDhxQk899ZT+/PNPLVy4UMePH9fkyZO1YcMGVatWTVFRUUZ38aEmL9yk9k2eVptGVVWicH59NKClcrq6aOHKX222n/rNFtWsUlK92z+n4oF+GtSjocqWKKgZS7dZ2jSv+5RCKpdQoH8+BQfl1/uvv6ArV29YvbFkZi/XKqbF28O09NdTOn7hioYu2qfrN2/rpWq2v8WJuXZLl67EWx7PBvvq+s3bWr3vrCSpsK+bnizsrXcX7dPBvy/rZHic3l28T67ZndS4YkB6Ds1QkxduUvumT6vtnbn68cCWypnDRQtSm6uLt6hW1ZJ6rf1zKl7YT4N7NFS5EgX19Z256uGWU0u/7K1mtZ9U0UKP6akyhfVBvxY6cOS0zlzI+L+79jBl0Wa1a/K0WjeqqhKF/TTuIb//07/ZoppVgtWrXdLv/8BX7vz+f7vNqt35iGgN/fRbTRjWQc7OWedLg7smLtiojs2eVrsm1RQclF+fDm6tXDlcNO+HX2y2n7Jos56rVlJ9O9RWicJ+GvJqI5UPDtC0JVskJX0zPnnhJvXrWlcNQsqpTDF/TRrRURcuxWjVlgPpOTTDTFu0WW0bV1OrhlVUvLCfPuj/knLmcNGilTtstv96yRaFVgnWq21rqVignwZ0b6AyxQtqZrK56uLiLF9vD8sjj0eu9BhOhsFcdQzqan/Tv9ms1o2qqWWDKioW6KfRbye9B3yz2vZ7wIylWxVSOVg92tRS0cDH9PbLDVS6eEHNXnbvPeD+331fbw+t+99vqvZEUT1eIF96DSvNCGQ29O7dWy4uLlq7dq1CQkL0+OOPq379+lq/fr3Onj2rIUOGGN3FB7p5K0EHj55W9UolLMucnJxUo1IJ7f7tpM119vwWphqVilstq1mlZKrtb95K0Nzl2+XhllOli/nbr/MZVPZsJpUJyKP/HQ23LDObpf8dDdcThb0eaRstnw7Uyr1ndP3mbUmSi3PSr198QqLVNm8mJOqpIt527H3GdfNWgg4cPa0QW3P1kO25t9vWXK1aMtX2khQbd0Mmk0me7jnt0/EMzPL7/9S9Gjk5Oal6peLa81uYzXV2/3bS6v1CkkKrBFu1T0xMVJ8R8/Rq21oqEZTfEV3P0G7eStD+I6cVWtl6roZULqFdqcy9nYdOKrRSsNWyWlVLatehMEnSqbORuhgZq9DK99p4uuVUxdKB2nUwzO5jyGhu3krQwWNnUszVZ58qrj2/h9lcZ89vYVbtpZRzVZJ+2Xdc5RoNVfU2ozXo428UFZN1TgNnrjoGdbW/m7cS9NuxM3qmovV7wDMVi2nv76dsrrPv9zCr9pJUo1KJVNtHRF3Rpl/+UKsGVezXcQcgkCUTFRWln376Sb169VLOnNY7b35+fmrXrp0WL14ss9mcYt34+HjFxsZaPYwQFX1Vt28nysfL3Wq5j5e7wiNtHwIOj4yVj5fHQ9uv/fk3Fa7VT4+HvK0pizbrm/G95J3Hzb4DyIDyurnKOZuTLl2Jt1p+KTZePh45Hrp+uUJ5VaKApxZvD7Ms++vCFZ2Nuqb+TUrLI2d2Zc9mUo/axVUgby75ej58m5lBanPVN++D56pv8rn6gPY34m/p/Qnf68U6T8o9d+YPZA/8/U/ltJqIyCvyyZusfV53hUfeew/7at4GZcvmpG4tQ+zf6f+AyOi4VOrqYVWn+4VHxsrH29b7cFL7i3f+m7yNr7d7qtvMTKJikuZqPhtzNSKV8UdEpZyr+fK6KyLqXvuaVUpq/ND2Wjy+l4a82li/7v9LHfpN0e3bick3lykxVx2Dutrf5bvvATY+f+7/nb5fRNQVm+0vpdL+2zU7lTtXDtWtkXFPV5S4qUcKf/75p8xms0qWLGnz+ZIlS+ry5cuKiIiQr6/1BZdjx47ViBEj0qObhnmmYjFtnD1QkTFxmvf9L+o+dKZ+nP52ijcoWGtZLVBHzsZY3QAkIdGsV6f9qg/aPan9HzVWwu1E/e9ohDb/fsHAnmYutxJuq9uQmTKbpY8GtjS6O/9ZB46c1vRvtmjtzP4ymUxGdwd4oKa1n7T8f8kiBVSySAE93WqUtu87nuLoGoDM7Zsfd6pZ7SeVwzW70V15II6QpcLWEbCHGTx4sGJiYiyP06dPO6BnD+eVJ7eyZXNKcZFpRNQV+XrbDk6+3h4pvo2w1T53TlcVDvDRU2UK6/MhbeWcLZsWrLB97nRmcjkuXgm3E5XP3foGHvk8XBURe+OB6+Z0yabGFQvqm1/CUjz32+loNfpgo8r3+0FVh6xWl4n/U57cLjodmTVOr0ltroZffvBcDU8+V220vxvGzlyI0tIve2eJo2PSQ37/U/nixMfbXRHJLqBOqmnSkcgdB/7SpctxeurF4SpY/U0VrP6mzlyI0ogvl6vSi5n7S6i7vPO4pVLXWEudkvP19lBEpK334aT2j935b/I24ZFXUt1mZuLlmTRXk99sJiLqinxSGb+PV8q5eunylRRneNyvkH8+eeXJrbAzEf++0/8BzFXHoK72l/fue4CNz5/Ufqd9vNxtts9no/3OA3/pxN/hatWoqv067SAEsmSKFi0qk8mkw4cP23z+8OHDyps3r3x8Ut7O3NXVVR4eHlYPI7hkd1a5EgHatvuYZVliYqK27T6qp8oUtrlOxTKBVu0lacvOI6m2t2zXnKibtxL+faczuFu3zfrtdLSeLnHvqKjJJD1d3Ff7Tj74RhENnvCXi7OTlu9KPaBfuZGgqLibCvTJrbKP59W6g+dTbZuZuGR3VvkSAdq6K9lc3XVUT5W1PfeeKhOobbtszNX72t8NYydOR2jpl73l5ZnbMQPIgO7+/v+8x7qmP+8+poplAm2u81SZwvo52e//1p1HLe1b1KukjXMGaP2s/paHXz5P9WpbSws/yxq3EnfJ7qwKwQHasuuoZVliYqK27jqmSqnM1cplC1u1l6RNO46oUtlASVIhf2895u1h1SY27rr2/B6mSuUC7T6GjMYlu7PKFS+on/f8aVmWmJion/ccS/U24hXLBOrn3X9aLdu662iqc1uSzoVH63LMNT2WRe4Mylx1DOpqfy7ZnVWmeEFtT/Z5tX3vn6n+6ZsnSgdatZekn3cfs9l+8eodKluioEoVzfj3OiCQJePt7a06depo4sSJun79utVzFy5c0Pz589WqVasMf9pOzzY1Nf+H7Vq8aoeOhV3QgHHf6NqNm2rdKOmixtdGzNWoiT9Y2r/SMkSbfj2sSQs26s+wi/po+modOHJaXVtUlyRdvR6v0ZNWaPdvJ3X6fJQOHPlbr4+arwsRMWpc6wlDxpjevt74p1o/HagXqzyuIo+5a2SrJ5TLNZuW/pp0IenHHSqqf5OUf7epZbVArT14TtFXb6Z4rv4T/qpSLJ8CvHOpdtn8mvPas1p38Jx+PhKeom1m1bNNTc37YbsWrdqhYycvqP+dudqmYdJc7T1irkbeP1dbhWjjr4c1cX7SXB03bbX2Hz6tl+/M1VsJt9V18Nfaf/hvTRrRUbcTzboYGauLkbFZ4ssDSerROlTzf/hF36zeqWNhFzTwoyVWv/993p+n0ZNWWNp3u/P7P/nO7//H039M+v1vnlRTL8/cCi5SwOrh7JxNPt4eKlroMUPGaIRebWtpzvLtWrjyVx09eUFvfbBYV6/Hq13jpG9few6boxFffW9p36N1qDb88oe+mrdBx8Iu6IOpq7T/8N/q/lLSdXgmk0k929TUxzPWaPWWg/r9+Fm9Onyu/PJ5qmFIeUPGmN66tw7VghW/6Jsfd+rPsAsa9PESXb9+U63u/P73HTlPYyffm6svvxSizTsOa/LCTTp+6qI++fpHHTxyWl3uzNWr1+I1csL32vNbmE6fj9S23cfUddB0BfrnU0jlYJt9yIyYq45BXe2vW8tQLVz1q5au2anjYRc15NOlunb9pl6qn/Qe8Nbo+fpw6kpL+64tamjLziOatjjpPeCzmWt06OhpdXqhutV2r1y9odWbD6hVw4x/dEziGjKbvvrqKz399NOqW7euRo0apcKFC+v3339X//795e/vr9GjRxvdxYdqVvtJRV6O07jpqxUeGavSxQpq4WevWm6GcPbiZTk53QuVlcoFadKITvpg6iqNmbxChQN8NevDbipZpIAkKZuTk46fuqhvVu9UVEyc8nrmVoWSj+v7Sa8rOIvccW3V3rPycnPVmw1LKZ+7qw6fjVHnCf+z3OijgFcuJSY707Wwr5sqFc2njl/9bHObvh45NOTFssrnnkMRsTf03Y6/9dUa20dnM6sX6jypyOg4fTht9Z0/tFlQiz971XK6xpkLl62+AKlcLkiT3++ksVNWafTkFQoK8NXscffm6vnwaK3Z9pskqWaHD61ea/mEPnqmYrF0GplxmtZOqum4aasVEZX0+7/g056WU0BS/P6XLayJIzrqw6mrNXbKShUu6KOZH7ys4Ds1RZIXn6+oS9FxGjNllcIjr6hscX8t/aL3fXM1Sk73zdUq5YM0bVRnjZ60UiMnrlBQgI/mffyKShW9V9fXO9bWtevxenPMQsXEXVfV8kW09IteGf56B3tp+tyTioq+qo+n/5g0V4v6a94nPSzXJZ+zMVe/GtZR46at0odTk+bq12NftnwOOWUz6fBf57Tkx12Kjbuux/J5KKRSsPp3byBXl6yzy8NcdQzqan+Naz2hqOg4fTZjjSKiYlWyqL9mf3TvPeBs+GWZ7nsPqFimsMa/20GffL1aH01bpcCCPpo6umuKu/+u2LBXZrNZTZ57Uv8FJvM/uVgqCzh16pSGDRumNWvWKCoqSn5+fmrWrJmGDRsmb+9HuyV5bGysPD09dfriZcNOX8yMgt9YbnQXMqVj45sZ3YVMKSF5Sse/liN71vsbaOnhWnzWOIKcnnK5Zp0QiP+2mGtZ4w9Rp6crsbEqFpBPMTExD80BvFOkolChQpo1a5bR3QAAAACQiXENGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBBnozuQFTiZkh6wj2PjmxndhUzJp8GHRnchU4r8cZDRXQAeSU6XbEZ3AYBBcrvy+29vt9NQU46QAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABnE2ugNwnK+XbtWEeRsVHhWr0kX9NfbtFnqydKFU23+/YZ8+mLpKp89HKSjAR+/2bqI6T5eWJN1KuK2xk1dq/S9/6NTZSLm75VBIpRJ6t1cT+fl4pteQDEdNHaNb4yfVp0UV+Xq56bcT4Ro4ca32Hj1vs61zNie92bqa2tQuq/z53HX8TKSGf71ZG3afsLTp2ugJdW34pAIeS6rjkVOX9NH8n7X+vjaZ3fQlW/XV/A0Kj4xV6WL++uDtFqpYOjDV9t9v2KcxU1Za5uqw3k1V55nSludXbNqvWd/9TweO/K3Lsde0ee5AlS1eMB1GkrFM+2aLvpyXVNcyxfz1Yf+XHljX5ev3aszkVfr7fKSCAnw0vE8zPX9fXc1ms8ZOWaU5y7crJu66qpQL0ieDWqnI477pMJqMYfqSrZaali7mrw/7PXiuLl+/T2OnrNTfd+bq8Nes56rZbNbYqas111LTwvp4YNaqqcRcdRTqan/sW3GELNNatm6v3hu/TP261dOG2f1Vupi/Wr4xURFRV2y233nwhHq8N1vtGlfTxtkDVL9GOXUaMF2H/zonSbp+46YOHj2jt7rU1YbZ/TXrg5d1/FS42vefmp7DMhQ1dYwXQkpq1CvP6cP5Pyu09wz9duKivh3dSvk8c9lsP7RzDXVu8IQGTlynqt2naeaqfZr73osqW+QxS5tzEVc0YsZm1Xxtpmr1maVtB8I0f3gLBRfKl17DMtSydXv07vhl6v9yfW2cPUBlivrrpdcfPFe7vztL7RtX06Y5A9WgRjl1GDDNMlcl6dr1m6paPkjDXmuaXsPIcL5bu0dDP1+mgd3qa/PcgSpTzF/N+0xIta47DpxQt6Gz1L5pNW2ZN0gNQ8qrfb+p+uP4vbqOn7NeUxZv0aeDW2vdzH7KldNFzftM0I34W+k1LEN9ty6ppgO61demOQNUppi/WvRNfa7uuDNX2zWpps1zB6pBSDm17z9Nf9w3V7+Ys15TF2/RJ4Naad2Mt5Urp6ta9J2YZWoqMVcdhbraH/tWSf5zgaxz585q1qxZqs+HhobKZDLJZDIpR44cKlWqlCZOnGh5ftasWZbn73/kyJHD6jXuLs+ePbsKFy6sAQMG6MaNG44cml1NXrhJ7Zs+rbaNqqpE4fz6eGBL5czhogUrf7XZfuriLapVtaRea/+cihf20+AeDVWuREF9vXSbJMnDLaeWftlbzWo/qaKFHtNTZQrrg34tdODIaZ25EJWeQzMMNXWMXi9W1pw1B7Rg7SEd/TtSb32xRtfiE9S+bjmb7Vs+V0afLdqudbv+0qkL0Zqxcp/W7fpLrzWvbGmzZsdxrdv1l06cu6y/zkZp1Kytunrjpp4KLpBewzLUxIWb1KFpNbVrXFXBQfn1yaBWypnDRfNX/GKz/ZTFm/Vc1ZLq06G2ShT20zs9G6lciQBNX7LV0qZVg8rq362+QiqVSK9hZDgTF2xUx2ZPq12TagoOyq9PB7dWrhwumvdDKnVdtFnPVSupvnfqOuTVRiofHKBpS7ZISvpmfPLCTerXta4ahJRTmWL+mjSioy5citGqLQfSc2iGmbhgkzo2uzdXPx3USrkeNFcXJc1VS017NlK54ABN/yZprprNZk1etFlv36lp6WL+mjS8w52aHkzPoRmKueoY1NX+2LdK8p8LZI+ie/fuOn/+vP744w+1bNlSvXv31sKFCy3Pe3h46Pz581aPU6dOWW2jXr16On/+vE6cOKHPPvtMU6ZM0bBhw9J7KP/IzVsJOnD0tNWOk5OTk2pUKqHdh07aXGf3b2GqUam41bKaVUum2l6SYuNuyGQyydM9p306noFRU8fI7uykCsX8tHnvvZqYzdKWfWGqVMrf5jqu2Z1142aC1bIb8QmqWtr26XNOTia9GFJSuVyza9fhs/brfAZ181aCDhw5rZDK1nM1pFIJ7ToUZnOdXYfCUgStWlWDtesBczWruXkrQfuPnFZo8rpWLpFqnXYeOqnQSsFWy2pVLWn5dzh1NlIXI2MVWvleG0+3nKpYOlC7DobZfQwZjWWuVkrjXK2c+lw9de5uTe+18bhb0ywyn5mrjkFd7Y99q3syZSDLlSuX/Pz8FBQUpOHDh6tYsWL64YcfLM+bTCb5+flZPR577DGrbbi6usrPz08BAQFq1qyZateurXXr1qX3UP6RqOirun07UT5e7lbLffO6KzzS9iHg8MhY+Xp5WC3zeUD7G/G39P6E7/VinSflnjvjTnB7oaaO4e2RS87ZnBQRfc1qecTlq/LN62ZznY17TqhX88oKKpBXJpMU+mSgGj1TQo95WbcvFeij08vf1sWVA/Rp33rq8P53Ovp3pMPGklFE3pmryeeer5e7wqNiba4THhmbYm77eKU+V7OiyOg4m+8BPl4eCo98QF29bdU1qf3FO/9N3sbX2z3VbWYmkZb31WTvk17ultokl/S+mqxeXu4Kv3N6k6WmNudz5q+pxFx1FOpqf+xb3ZMlbuqRM2dO3bx58x+v/9tvv2n79u0qVCj1CwwlKT4+XvHx8ZafY2Mz5y/TrYTb6jZkpsxm6aOBLY3uTqZATR/doEnrNP6NBto5/RWZJZ08d1kL1h5Uu2SnOP55JlI1es2QRy5XNa1eQhP7NVKj/vOyRCgDACCr+y/tW2XKI2R33b59W/PmzdPBgwdVq1Yty/KYmBi5ublZPerXr2+17sqVK+Xm5qYcOXKobNmyCg8PV//+/R/4emPHjpWnp6flERAQ4JBxPYxXntzKls0pxQWR4ZevyDfZtzB3+Xp7pPj2PMJG+7uT+8yFKC39sneG/rbBnqipY0TGXlPC7UT55LG+gYdP3twKvxxne52Y62o/4lv5N/1Y5TpMUOVuU3X1xk2FXYi2ancrIVEnz13WgeMX9P7MLfrt5EX1bFbJUUPJMLzvzNXkcy886kqKbxXv8vX2SDG3I6JSn9tZkXceN5vvARFRsfL1fkBdI23VNan9Y3f+m7xNeOSVVLeZmXhb3leTvU9GXbHUJrmk99Vk9Yq6YjlqZqmpzfmc+WsqMVcdhbraH/tW9/xnA9n8+fOtAtW2bdssz02cOFFubm7KmTOnunfvrjfffFOvvvqq5Xl3d3ft37/f6jF9+nSr7desWVP79+/Xjh071KlTJ3Xp0kXNmzd/YJ8GDx6smJgYy+P06dP2HfQjcsnurPIlArR11zHLssTERG3bdVRPlS1sc52nygRq233tJWnLziNW7e9O7hOnI7T0y97y8sztmAFkQNTUMW4lJGr/nxcU8kSgZZnJJNWoUEi7/njw9V7xt27rfGScnLM5qfGzwfrxlz8f2N7JZJJL9mz26HaG5pLdWeWDU87VrbuOqVLZQJvrVCobqK27refq5p1HVSmVuZ0VuWR3VoXgAG3ZddSy7F5dbdepctnCVu0ladOOI5Z/h0L+3nrM28OqTWzcde35PUyVygXafQwZTWpzdcvuh8zVZO+rm3fcm6uFCjygpllkPjNXHYO62h/7Vvf8Z09ZbNKkiapUqWL52d//3g0A2rVrpyFDhihnzpzKnz+/nJysc6eTk5OKFi36wO3nzp3b0mbGjBkqX768vv76a7388supruPq6ipXV9d/Mhy769mmpvqMnKcKJQP0ZKlCmrJ4s67duKk2DZNq1nvEXPn5eOrdXk0kSa+0ClHTV7/QxPkbVeeZ0lq2bo/2Hz6tTwa1lpQ0ubsO/loHj57R/E966Hai2XLuc16PXHLJ/p+dSo+MmjrGxO92amK/Rtp37IL2Hj2nV1+opNw5smv+2qQ7ok3q30jnL13R+zOT7kpVsUQB5c/npkN/hatAPjcNbF9dTiZp/Df37sj0XpcQrd91QqcjYuWe00UtapbSs+UKqfmQRYaMMb31alNTvd+fpwolH0+aq4s269qNeLVtVFWS9OrwOcrvk0fv9U6aqz1ahapxz/GaMH/Dnbm6V/sP/63PBre2bPNyzFWduXhZFyJiJEnHT12UlPRtZWpHMzKbXm1rqdeIuXqi5ON6snSgJi3cpKvX49WucVJdew6bo/w+npY/DdCjdaga9fhcX83boOefLa3v1u7R/sN/6/N32khKup65Z5ua+njGGgUF+KiQv7fGTF4lv3yeahhS3rBxpqdebWuq94g7c7V0IU1etFnXrt83V4fNUX7f++Zq61A17jFeX83foOefKa3v1t6Zq+8kzVWTyaSerUP1yYyfVCTAV4UKeGvM5JV3amr7zq2ZEXPVMair/bFvlSRj9uoRuLu7y93d9uFMT0/PhwautHByctI777yjt956S23btlXOnBn7sKckvVDnSUVGx+nDaavv/PHCglr82auWQ+BnLlyWyWSytK9cLkiT3++ksVNWafTkFQoK8NXscd1UskjSbcLPh0drzbbfJEk1O3xo9VrLJ/TRMxWLpdPIjENNHWPZlsPK55lL73SsLt+8uXXoRLhaDPnGcqOPgj4eSkw0W9q7umTTkE4hCsyfR1ev39S6XX+p57gVir167/rNfHlya1L/RnrMy02x1+L1+8lwNR+ySJv3hqX38AzxQp2KuhQdpw+mrlJ45BWVKe6vbz7vZZmrZy9elpOT9VydOrKzRk9eqVGTVioowEdzx3W3zFVJ+nHbIfUZOd/yc7ehsyRJA7rV18DuDdJnYAZ78fmkuo6ZklTXssX9tfSL3ve9B0TJ6b73gCrlgzRtVGeNnrRSIyeuUFCAj+Z9/IpKFb1X19c71ta16/F6c8xCxcRdV9XyRbT0i17K4Zo93cdnhBfrVFTk5TiNvW+uLhl/b66eSTZXq9yZq2Mmr9SoiUlzdd5H3VXqvrnat2NtXb1x876aBmnJ+KxTU4m56ijU1f7Yt0piMpvN5oc3yzg6d+6s6OhoLV++3ObzoaGhqlChgj7//HObz8+aNUuvv/66jh49muI5X19fOTk52XyNhIQEBQYG6o033lC/fv0eqa+xsbHy9PTU2fDL8vDIGt8g47/Lp8GHD2+ENIv8cZDRXch07t9Bh/38x3YH/hPu35EEMrKE24lGdyHTiY2Nlb9vXsXExDw0B/xnryH7N2JjY5U/f/4Uj/Dw8FTXcXZ21muvvaZx48bp6tWr6dhbAAAAAJnVf+4I2X8JR8jwX8IRMsfgCJn9cYTMMdgdsD+OkOG/giNk9scRMgAAAAD4DyCQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEGejO5AVZHMyKZuTyehuZBpms9E9yJyi1gwyuguZklf1gUZ3IdO5/PM4o7uQKZlMfE4BWRX7qfaXlppyhAwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADOJsdAfgONOXbNWX8zYoPDJWpYv568N+LVSxdGCq7Zev36exU1bq7/NRCgrw0fDXmqrOM6Utz6/YtF8zv/ufDhz+W5djr2nLvIEqW7xgOowk45i+ZKu+mn+vph+8/eCafr9hn8ZMWanTd2o6rHfKms767n86cCSpppvnZr2aSvafq2azWWOnrtbc5dsVE3ddVcoV1scDW6nI477pMJqModuL1dSnTYh8vdz121/nNfCz77X38GmbbZ2zOenNDrXUpn5F5c/noeOnIzR80mpt2HHM0ubAkkF6PL9XinWnf7dd/T9d7qhhZDjTvtlimatlivnrw/4vPWSu7tWYyav09/nIpLnap5meTz5Xp6zSHMtcDdIng7LWXKWmjkFdHYO62h/7qxn4CFnnzp1lMpksD29vb9WrV08HDx5MdZ2wsDCZTCbt378/1Tbbt29XgwYNlDdvXuXIkUNly5bVp59+qtu3b6dou2nTJjVo0EDe3t7KlSuXSpUqpbfffltnz561xxAd6rt1ezT082Ua0K2+Ns0ZoDLF/NWi70RFRF2x2X7HwRPq/u4stWtSTZvnDlSDkHJq33+a/vjrnKXNtes3VbV8kIa91jS9hpGhLFu3R++OX6b+L9fXxtkDVKaov156PfWa7rxT0/aNq2nTnIFqUKOcOgyYpsPU1Ioj5uoXc9Zr6uIt+mRQK62b8bZy5XRVi74TdSP+VnoNy1Av1CqvUa811ocz1yv05fH67fh5ffvpy8qXJ7fN9kNfqavOTato4Gffq2qHTzRz+a+aO6aTyhYrYGlTq/uXKtHkfcuj2RtTJUnLN6X+npzZfLc2aa4O7FZfm+cOVJli/mreZ0Lqc/XACXUbOkvtm1bTlnmD1DCkvNr3m6o/jt+bq+PnrNeUxVv06eDWWjezn3LldFHzPhOyzFylpo5BXR2Dutof+6tJMmwgk6R69erp/PnzOn/+vDZs2CBnZ2c1atToH29v2bJlCgkJUcGCBbVp0yYdOXJEr7/+ukaNGqXWrVvLbDZb2k6ZMkW1a9eWn5+fvv32W/3xxx+aPHmyYmJi9Mknn9hjeA41ccEmdWxWTe0aV1VwUH59OqiVcuVw0fwVv9hsP2XRZj1XtaT6dqitEoX9NKRnI5ULDtD0b7Za2rRqUFkDutVXaOUS6TWMDGXiwk3q0PReTT8Z1Eo5H1TTxUk17XOnpu/0bKRyJQI0fYl1Tft3q6+QSlmzppL956rZbNbkRZv1dte6ahBSTqWL+WvS8A66cClGq7ZkjfDQq3V1zVmxQwtW79bRsHC99dF3unbjlto3qmSzfcu6FfXZ3I1a9+sRnToXpRnLf9W6X47otdY1LG0io68qPCrO8qj7dEmdOHNJ/9t3Ir2GZbiJCzaqY7On1a5JtaS5Ori1cuVw0bwfHjBXq903V19tpPLBAZq2ZIukO3N14Sb1uzNXyxTz16QRHe/M1QPpOTTDUFPHoK6OQV3tj/3VJBk6kLm6usrPz09+fn6qUKGCBg0apNOnTysiIiLN27p69aq6d++uJk2aaOrUqapQoYICAwPVrVs3zZ49W0uXLtU333wjSTpz5oz69u2rvn37asaMGQoNDVVgYKBq1Kih6dOn67333rP3UO3q5q0EHThy2mon38nJSSGVSmjXoTCb6+w6FKaQZBO3VtVg7Tp00pFd/c+w1LRyGmtaiZo+iCPm6qlzkboYGWv1RuzhllMVSwdmidpnd86mCsX9tXn3ccsys9msLbv/VKXShWyu45o9m27EJ1gtuxF/S1XLBab6Gi2ff1LzV+2yW78zupu3ErT/yGmreeXk5KSQyiVSnVc7D51UaKVgq2W1qpa0zO1TZ+/O1XttPO/O1YNhdh9DRkNNHYO6OgZ1tT/2V+/J0IHsfnFxcZo3b56KFi0qb2/vNK+/du1aRUZGql+/fimea9y4sYoXL66FCxdKkpYsWaKbN29qwIABNreVJ08em8vj4+MVGxtr9TBCZPRV3b6dKB8vD6vlPl7uuhhpu0/hkbHy9XK3Wubr5a7wVA4ZZzV3a+qbrKZJNUq9pj7Jaurj5a7wSGp6lyPm6t31bNfemN/J9OTtmVvOztlSnO4RERUnX293m+ts3HlMvVpXV1DBfDKZTAp9qpgahZTRY94eNts3rFFanm45tGD1Hrv3P6OKjI67M1eTzyuPVOdVeGSsfLxTn4eWuZqsja931pir1NQxqKtjUFf7Y3/1ngx9U4+VK1fKzc1NUtIRrvz582vlypVyckp7jjx2LOni9JIlS9p8Pjg42NLmzz//lIeHh/Lnz5+m1xg7dqxGjBiR5r4BgJEGjf9B4wc01875/WQ2m3XyXJQWrN6tdg1tn+LYvmElrd9xVBeywA4DAACOlqGPkNWsWVP79+/X/v37tXPnTtWtW1f169fXqVOnVL9+fbm5ucnNzU2lS5d++MbuuP86sQe1MZlMae7v4MGDFRMTY3mcPm37jmaO5p0nt7Jlc1JEsiM3EVFXUv3G29fbI8W3C+FRV1J8C5FV3a1p8qNhSTVKvaYpj1JcSfUoRVbkiLl6dz3btbe9zcwkMuaqEhJu2/gW1y3Vo7OR0VfV/p058q8zVOVajFXlth/p6vV4hZ2LTNE24LE8Cn2qmOas2OmQ/mdU3nnc7szV5PMqNtV55evtoYjI1OehZa4maxMemTXmKjV1DOrqGNTV/thfvSdDB7LcuXOraNGiKlq0qCpVqqTp06fr6tWrmjZtmqZPn24Ja6tXr37otooXLy5JOnz4sM3nDx8+bGlTvHhxxcTE6Pz582nqr6urqzw8PKweRnDJ7qzywQHauuveLasTExO1ZfcxVSobaHOdSmUDrdpL0uYdR1WpbGFHdvU/I7Wabt31kJruTlbTndT0fo6Yq4UKeOsxbw9t2XXU8nxs3HXt+T0sS9T+VsJt7T92ViEVi1qWmUwm1ahYVLt+P/XAdeNvJuj8pVg5Z3NS45Cy+nHbHynatG1YSRGX47T2lyN273tG5pLdWRWCA6zm1b33ANvzqnLZwlbtJWnTjiOWuV3I/wFzNZXr9zITauoY1NUxqKv9sb96T4YOZMmZTCY5OTnp+vXr8vf3t4S1QoVsX6h+v+eff15eXl4275D4ww8/6M8//1SbNm0kSS1atJCLi4vGjRtnc1vR0dH/ahzpoVfbmprz/XYtXLlDR09e0NsffqNr1+PVtlFVSdKrw+bo/Qk/WNr3aB2qDb/8oa/mb9CxsAv6YOpq7T/8t7q1vHeXtcsxV3Xo2BkdPXlBkvTnqYs6dOyMLl7KGqct9WpTU3O/366Fq5Jq2u/Db3Ttxn01HZ6spq2SajrhTk0/nHanpi+lXtPjd2uahU4Fs/dcNZlM6tk6VJ/M+Ek/bj2kP46fU6/hc+WXz1MNQ8oZMsb0NnHRNnVsXFmt61VU8UK++rTfC8qd00XzV+2WJE0a2krv9ahnaV+xVIAa1SijQgW8VK1coJZ+8rKcnEwav2Cz1XZNJpPaNXhKi9bs0e3biek5pAyhV9tamrN8uxau/FVHT17QWx8s1tXr8WrXOGmu9hw2RyO++t7S3jJX592dq6u0//Df6v5SiKQ7c7VNTX08Y41Wbzmo34+f1auWuVrekDGmN2rqGNTVMair/bG/miRDX0MWHx+vCxeSinn58mV99dVXiouLU+PGjR+43tGjR1MsK126tKZMmaLWrVvrlVde0WuvvSYPDw9t2LBB/fv3V4sWLdSyZUtJUkBAgD777DO99tprio2NVceOHRUYGKgzZ85ozpw5cnNzy/C3vn+xTkVFXo7T2KmrFB55RWWK+2vJ+F6WQ+BnLl6Wk9O90zKrlAvS1JGdNWbySo2auFJBAT6a91F3lSpy7+8Q/bjtkF57f77l525DZkmSBnSrr0GvNEifgRnohToVdSk6Th/cV9NvPr9X07PJalr5Tk1HT16pUZOSajp3XHeVTFbTPiPvq+nQWZKSajqwe+avqeSYudq3Y21dvXFTb45ZqJi466paPkhLxvdSDtfs6T4+IyzbeED58uTWO92el6+Xuw4dP6cWb3+tiMtxkqSCj+VRYuK907ddXbJrSPe6CizgpavXb2rdr0fUc+RixcbdsNpu6FNFFeCXV/Oy0N0V7/fi80nvAWOmJM3VssX9tfSL3vfm6oUoOd13unuV8kGaNqqzRk9aqZETVyTN1Y9fUami9+bq6x1r69r1+PvmahEt/SLrzFVq6hjU1TGoq/2xv5rEZH6Ui6oM0LlzZ82ePdvys7u7u4KDgzVw4EA1b97c5jphYWEqXNj2IcvTp0+rYMGC2rZtm0aPHq1ffvlFN27cULFixdSlSxe98cYbypYtm9U669ev18cff6ydO3fq+vXrCgwMVKNGjfTWW2890g0/YmNj5enpqQuXog07fTEzypgz9r/vH1w2iUfgVX2g0V3IdC7/bPvsBQDAP5NB48B/WmxsrPzy5VFMTMxDc0CGDWSZAYHMMZixjkEgcwwCmf0RyADAvogD9peWQPafuoYMAAAAADITAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQZyN7gCQVk5OJqO7ADyyyz+PM7oLmU7eZwcY3YVMiblqf4mJZqO7kCmxH2B/Zqaq3aWlphwhAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIM6P0uiHH3545A02adLkH3cGAAAAALKSRwpkzZo1e6SNmUwm3b59+9/0BwAAAACyjEcKZImJiY7uBwAAAABkOf/qGrIbN27Yqx8AAAAAkOWkOZDdvn1bI0eOlL+/v9zc3HTixAlJ0rvvvquvv/7a7h0EAAAAgMwqzYFs9OjRmjVrlsaNGycXFxfL8jJlymj69Ol27RwAAAAAZGZpDmRz5szR1KlT1a5dO2XLls2yvHz58jpy5IhdOwcAAAAAmVmaA9nZs2dVtGjRFMsTExN169Ytu3QKAAAAALKCNAeyUqVKadu2bSmWL126VE888YRdOgUAAAAAWcEj3fb+fu+99546deqks2fPKjExUd99952OHj2qOXPmaOXKlY7oIwAAAABkSmk+Qta0aVOtWLFC69evV+7cufXee+/p8OHDWrFiherUqeOIPgIAAABAppTmI2SSVL16da1bt87efQEAAACALOUfBTJJ2r17tw4fPiwp6bqyihUr2q1TAAAAAJAVpDmQnTlzRm3atNH//vc/5cmTR5IUHR2tp59+WosWLVLBggXt3UcAAAAAyJTSfA1Zt27ddOvWLR0+fFhRUVGKiorS4cOHlZiYqG7dujmijwAAAACQKaX5CNmWLVu0fft2lShRwrKsRIkS+vLLL1W9enW7dg4AAAAAMrM0HyELCAiw+Qegb9++rQIFCtilUwAAAACQFaQ5kH300Ufq06ePdu/ebVm2e/duvf766/r444/t2jkAAAAAyMwe6ZTFvHnzymQyWX6+evWqqlSpImfnpNUTEhLk7Oysrl27qlmzZg7pKAAAAABkNo8UyD7//HMHdwMAAAAAsp5HCmSdOnVydD8AAAAAIMv5x38YWpJu3LihmzdvWi3z8PD4Vx0CAAAAgKwizTf1uHr1ql577TX5+voqd+7cyps3r9UDAAAAAPBo0hzIBgwYoI0bN2rSpElydXXV9OnTNWLECBUoUEBz5sxxRB8BAAAAIFNK8ymLK1as0Jw5cxQaGqouXbqoevXqKlq0qAoVKqT58+erXbt2jugnAAAAAGQ6aT5CFhUVpaCgIElJ14tFRUVJkp599llt3brVvr0DAAAAgEwszUfIgoKCdPLkST3++OMKDg7WN998o8qVK2vFihXKkyePA7qIf2r6kq36ct4GhUfGqnQxf33Yr4Uqlg5Mtf3y9fs0dspK/X0+SkEBPhr+WlPVeaa05fkVm/Zr5nf/04HDf+ty7DVtmTdQZYsXTIeRZBzTvtliqWmZYv76sP9LD6npXo2ZvEp/n49MqmmfZnr+vpqazWaNnbJKc5ZvV0zcdVUpF6RPBrVSkcd902E0GQd1tT9q6hjdXqymPm1C5Ovlrt/+Oq+Bn32vvYdP22zrnM1Jb3aopTb1Kyp/Pg8dPx2h4ZNWa8OOY5Y2B5YM0uP5vVKsO/277er/6XJHDSNDYa46xvQlW/XV/Hv7AB+8/eB9gO837NOYKSt1+s4+wLDeKfcBZn33Px04krQPsHlu1tsHkJivjsBc/QdHyLp06aIDBw5IkgYNGqQJEyYoR44cevPNN9W/f3+7dxD/zHfr9mjo58s0oFt9bZozQGWK+atF34mKiLpis/2OgyfU/d1ZatekmjbPHagGIeXUvv80/fHXOUuba9dvqmr5IA17rWl6DSND+W5tUk0HdquvzXMHqkwxfzXvMyH1mh44oW5DZ6l902raMm+QGoaUV/t+U/XH8Xs1HT9nvaYs3qJPB7fWupn9lCuni5r3maAb8bfSa1iGo672R00d44Va5TXqtcb6cOZ6hb48Xr8dP69vP31Z+fLkttl+6Ct11blpFQ387HtV7fCJZi7/VXPHdFLZYgUsbWp1/1IlmrxveTR7Y6okafmmg+kyJqMxVx1j2bo9enf8MvV/ub42zh6gMkX99dLrqe8D7LyzD9C+cTVtmjNQDWqUU4cB03SYfQArzFf7Y64mSXMge/PNN9W3b19JUu3atXXkyBEtWLBA+/bt0+uvv27XznXu3Fkmk8ny8Pb2Vr169XTwYOofVGFhYSnWef7557Vv3z5Lm9DQUKs2dx89e/a0tLl/uYeHhypVqqTvv//eruNzpIkLNqljs2pq17iqgoPy69NBrZQrh4vmr/jFZvspizbruaol1bdDbZUo7KchPRupXHCApn9z7zTUVg0qa0C3+gqtXCK9hpGhTFywUR2bPa12Taol1XRwa+XK4aJ5PzygptXuq+mrjVQ+OEDTlmyRlPSt2OSFm9Sva101CCmnMsX8NWlER124FKNVWw6k59AMRV3tj5o6Rq/W1TVnxQ4tWL1bR8PC9dZH3+najVtq36iSzfYt61bUZ3M3at2vR3TqXJRmLP9V6345otda17C0iYy+qvCoOMuj7tMldeLMJf1v34n0GpahmKuOMXHhJnVoem8f4JNBrZTzQfsAi5P2Afrcqes7PRupXIkATV9ivQ/Qv1t9hVTKmvsAEvPVEZirSdIcyJIrVKiQXnzxRZUrV84e/UmhXr16On/+vM6fP68NGzbI2dlZjRo1euh669ev1/nz5/XTTz8pLi5O9evXV3R0tOX57t27W7Z79zFu3DirbcycOVPnz5/X7t279cwzz6hFixY6dOiQvYdodzdvJejAkdNWE9HJyUkhlUpo16Ewm+vsOhSmkGRBq1bVYO06dNKRXf3PuHkrQfuPnLYKo05OTgqpXCLVGu08dFKhlYKtltWqWtLyb3DqbKQuRsYqtPK9Np5uOVWxdKB2HQyz+xgyIupqf9TUMbI7Z1OF4v7avPu4ZZnZbNaW3X+qUulCNtdxzZ5NN+ITrJbdiL+lquUCU32Nls8/qfmrdtmt3xkZc9UxLPsAyev6sH2ASuwDPAjz1f6Yq/c80jVkX3zxxSNv8O7RM3txdXWVn5+fJMnPz0+DBg1S9erVFRERIR8fn1TX8/b2lp+fn/z8/PTxxx/rmWee0Y4dO1S3bl1JUq5cuSzbTU2ePHks2xg5cqTGjx+vTZs2qWzZsvYboANERl/V7duJ8vGy/iPdPl7uOnbqos11wiNj5evlbrXM18td4akcMs5qIqPj7tTUukY+Xh76Myz1mvp4J2/vrvDIWEnSxTv/Td7G1/tem8yOutofNXUMb8/ccnbOluI0moioOBUrZPtaj407j6lX6+rafuCkTp6NVEjFomoUUkbZnGx/F9qwRml5uuXQgtV77N7/jIi56hh39wF8k+0D+Hq5688H7AOk/HdwV3gk+wB3MV/tj7l6zyMFss8+++yRNmYymeweyO4XFxenefPmqWjRovL29n7k9XLmzClJunnz5j963YSEBH399deSJBcXl1TbxcfHKz4+3vJzbGzm/2UCANg2aPwPGj+guXbO7yez2ayT56K0YPVutWto+xTH9g0raf2Oo7qQBXbEAAD3PFIgO3nSuMOAK1eulJubmyTp6tWryp8/v1auXCmnVL5hTC46OlojR46Um5ubKleubFk+ceJETZ8+3artlClTrP6OWps2bZQtWzZdv35diYmJCgwMVMuWLVN9rbFjx2rEiBFpGZ5DeOfJrWzZnBQRZf2hHhF1RY95e9hcx9fbI8XRsPCoKymOmmVV3nnc7tQ0+bfjsfJ9QE0jIpO3v2Jpf/ffIiLyivzyeVrahEdeyfB3A7IX6mp/1NQxImOuKiHhto1vZt1S/WY2Mvqq2r8zR64uzvLyyKXzl2I1/NX6CjsXmaJtwGN5FPpUMXUYMsch/c+ImKuOcXcfIDzZPkDSZ/oD6pri3+GKfL3ZB7iL+Wp/zNV7/vU1ZI5Ws2ZN7d+/X/v379fOnTtVt25d1a9fX6dOnVL9+vXl5uYmNzc3lS5d2mq9p59+Wm5ubsqbN68OHDigxYsX67HHHrM8365dO8t27z6aNGlitY3PPvtM+/fv148//qhSpUpp+vTp8vJKeXviuwYPHqyYmBjL4/Rp27dCdjSX7M4qHxygrbvu3Vo5MTFRW3YfU6WygTbXqVQ20Kq9JG3ecVSVyhZ2ZFf/M1yyO6tCcIC27DpqWZaYmKitu46lWqPKZQtbtZekTTuOWP4NCvl76zFvD6s2sXHXtef3MFVK5RqTzIa62h81dYxbCbe1/9hZhVQsallmMplUo2JR7fr91APXjb+ZoPOXYuWczUmNQ8rqx21/pGjTtmElRVyO09pfjti97xkVc9UxUtsHSKproM11KpUN1NbdyfYBdrIPcD/mq/0xV+9J898hS2+5c+dW0aL3PgCnT58uT09PTZs2TdOnT9f169clSdmzZ7dab/HixSpVqpS8vb1t/n00T09Pq+3a4ufnp6JFi6po0aKaOXOmGjRooD/++EO+vravF3B1dZWrq2saR+gYvdrWVO8R81Sh5ON6snQhTV60Wdeux6tto6qSpFeHzVF+3zx6r3dSCO3ROlSNe4zXV/M36PlnSuu7tXu1//Df+uyd1pZtXo65qjMXL+tCRIwkWc7v9fXy0GP5bH+TkZn0altLvUbM1RMlH9eTpQM1aeEmXb0er3aNk2rac9gc5ffxtNxmtUfrUDXq8bm+mrdBzz9bWt+t3aP9h//W5++0kZS0M9ezTU19PGONggJ8VMjfW2Mmr5JfPk81DClv2DjTG3W1P2rqGBMXbdPEIS2178gZ7T18Wq+2fFa5c7po/qrdkqRJQ1vpfESM3p+yRpJUsVSA8ufz1KHj51Qgn4cGdq0jJyeTxi/YbLVdk8mkdg2e0qI1e3T7dmJ6D8tQzFXH6NWmpnq/f2cfoFQhTVm0Wddu3LcPMHyO8vvctw/QKlSNe47XhPkbVOeZ0lq27s4+wODU9wGO390H8PZI9eybzIb5an/M1SQZPpAlZzKZ5OTkpOvXr8vf3z/VdgEBASpSpIjdXrdy5cqqWLGiRo8erfHjx9ttu47yYp2Kirwcp7FTVyk88orKFPfXkvG9LIfJz1y8LCcnk6V9lXJBmjqys8ZMXqlRE1cqKMBH8z7qrlJF7v29nB+3HdJr78+3/NxtyCxJ0oBu9TXolQbpMzADvfh8RV2KjtOYKavunE7gr6Vf9L5X0wtRcjLdV9PyQZo2qrNGT1qpkRNXJNX041dUqui9mr7esbauXY/Xm2MWKibuuqqWL6KlX/RSDtfsKV4/s6Ku9kdNHWPZxgPKlye33un2vHy93HXo+Dm1ePtrRVyOkyQVfCyPEhPNlvauLtk1pHtdBRbw0tXrN7Xu1yPqOXKxYuNuWG039KmiCvDLq3lZ5O6K92OuOsYLdZLq+sF9+wDffH5vH+Bssn2Aynf2AUZPXqlRk5L2AeaO666SyfYB+oy8bx9g6CxJSfsAA7tn/n0AifnqCMzVJCaz2Wx+eDNjdO7cWRcvXtTMmTMlSZcvX9ZXX32lSZMmaePGjQoNDU2xTlhYmAoXLqx9+/apQoUKNrcbGhqq4sWL6/3337da7urqqrx580pKCn7Lli1Ts2bNLM//+OOPeuGFF/TXX389MAzeFRsbK09PT124FC0Pj4yZyP+LTPe92QHIevI+O8DoLmRKl38e9/BGSJP7Azrs5/4ddNgHc9X+YmNjld8nj2JiYh6aAzL8NWRr1qxR/vz5lT9/flWpUkW7du3SkiVLbIaxtJg2bZplu3cfbdq0eeA69erVU+HChTV69Oh/9doAAAAAIP3DI2Tbtm3TlClT9Ndff2np0qXy9/fX3LlzVbhwYT377LOO6Od/EkfIHIMjZEDWxhEyx+AImf1x1MExOEJmf8xV+3PoEbJvv/1WdevWVc6cObVv3z7L392KiYnRmDFj/lmPAQAAACALSnMgGzVqlCZPnqxp06ZZ3dnwmWee0d69e+3aOQAAAADIzNIcyI4ePaoaNWqkWO7p6ano6Gh79AkAAAAAsoQ0BzI/Pz8dP348xfKff/5ZQUFBdukUAAAAAGQFaQ5k3bt31+uvv64dO3bIZDLp3Llzmj9/vvr166dXX33VEX0EAAAAgEwpzX8YetCgQUpMTNRzzz2na9euqUaNGnJ1dVW/fv3Up08fR/QRAAAAADKlNAcyk8mkIUOGqH///jp+/Lji4uJUqlQpubm5OaJ/AAAAAJBppTmQ3eXi4qJSpUrZsy8AAAAAkKWkOZDVrFnzgX+Yd+PGjf+qQwAAAACQVaQ5kFWoUMHq51u3bmn//v367bff1KlTJ3v1CwAAAAAyvTQHss8++8zm8uHDhysuLu5fdwgAAAAAsoo03/Y+Ne3bt9eMGTPstTkAAAAAyPTsFsh++eUX5ciRw16bAwAAAIBML82nLL744otWP5vNZp0/f167d+/Wu+++a7eOAQAAAEBml+ZA5unpafWzk5OTSpQooffff1/PP/+83ToGAAAAAJldmgLZ7du31aVLF5UtW1Z58+Z1VJ8AAAAAIEtI0zVk2bJl0/PPP6/o6GgHdQcAAAAAso4039SjTJkyOnHihCP6AgAAAABZSpoD2ahRo9SvXz+tXLlS58+fV2xsrNUDAAAAAPBoHvkasvfff19vv/22GjRoIElq0qSJTCaT5Xmz2SyTyaTbt2/bv5cAAAAAkAk9ciAbMWKEevbsqU2bNjmyPwAAAACQZTxyIDObzZKkkJAQh3UGAAAAALKSNF1Ddv8pigAAAACAfydNf4esePHiDw1lUVFR/6pDAAAAAJBVpCmQjRgxQp6eno7qCwAAAABkKWkKZK1bt5avr6+j+gIAAAAAWcojX0PG9WMAAAAAYF+PHMju3mURAAAAAGAfj3zKYmJioiP7AQAAAABZTppuew8AAAAAsB8CGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGeeS/Q4Z/zmQyyWQyGd2NTCPhNn8TzxGcs/H9DP4bLv88zuguZEp56441uguZTuSPg4zuAvBIbrFvZXdpqSl7YAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEGcje4AHGfaN1v05bwNCo+MVZli/vqw/0uqWDow1fbL1+/VmMmr9Pf5SAUF+Gh4n2Z6/pnSlufNZrPGTlmlOcu3KybuuqqUC9Ing1qpyOO+6TCajOHrpVs1Yd5GhUfFqnRRf419u4WeLF0o1fbfb9inD6au0unzUQoK8NG7vZuoztNJNb2VcFtjJ6/U+l/+0KmzkXJ3y6GQSiX0bq8m8vPxTK8hZQjMVfujpo5BXe2vW+Mn1adFFfl6uem3E+EaOHGt9h49b7OtczYnvdm6mtrULqv8+dx1/Eykhn+9WRt2n7C06droCXVt+KQCHkt6Hz1y6pI+mv+z1t/XJiuYvmSrvpqfNFdLF/PXB2+3eOBc/X7DPo2ZstLyeTWsd1PVuW+urti0X7O++58OHPlbl2OvafPcgSpbvGA6jCRj4T3A/mZ8u00T529URFSsShX11+i3muvJUqnvW/2wcZ/GTV2t0xeiVLigj4b2aqzaT9+r6UfTf9T36/fqbHi0XLJnU7kSARrco6GefMC/k9EMPULWuXNnmUwmy8Pb21v16tXTwYMHU10nLCxMJpNJ2bJl09mzZ62eO3/+vJydnWUymRQWFmbVfv/+/ZZ2y5YtU9WqVeXp6Sl3d3eVLl1ab7zxhtW2bt68qXHjxql8+fLKlSuX8uXLp2eeeUYzZ87UrVu37FUCh/lu7R4N/XyZBnarr81zB6pMMX817zNBEVFXbLbfceCEug2dpfZNq2nLvEFqGFJe7ftN1R/Hz1najJ+zXlMWb9Gng1tr3cx+ypXTRc37TNCN+IxfD3tYtm6v3hu/TP261dOG2f1Vupi/Wr4xMdWa7jx4Qj3em612jatp4+wBql+jnDoNmK7DfyXV9PqNmzp49Ize6lJXG2b316wPXtbxU+Fq339qeg7LcMxV+6OmjkFd7e+FkJIa9cpz+nD+zwrtPUO/nbiob0e3Uj7PXDbbD+1cQ50bPKGBE9epavdpmrlqn+a+96LKFnnM0uZcxBWNmLFZNV+bqVp9ZmnbgTDNH95CwYXypdewDLds3R69O36Z+r9cXxtnD1CZov566fUHf151f3eW2jeupk1zBqpBjXLqMGCa5fNKkq5dv6mq5YM07LWm6TWMDIf3APtbvn6vhn+xTG93rau1M/urdNECavPmpFRruuvQSb06bI7aNK6qdbP6q36Nsuoy6GuruVrkcR+NebuFNs8dqO8nva6A/F5q9cYkXbocl17DSjPDT1msV6+ezp8/r/Pnz2vDhg1ydnZWo0aNHrqev7+/5syZY7Vs9uzZ8vf3f+B6GzZsUKtWrdS8eXPt3LlTe/bs0ejRo61C1s2bN1W3bl198MEHeuWVV7R9+3bt3LlTvXv31pdffqnff//9nw02HU1csFEdmz2tdk2qKTgovz4d3Fq5crho3g+/2Gw/ZdFmPVetpPp2qK0Shf005NVGKh8coGlLtkhK+gZn8sJN6te1rhqElFOZYv6aNKKjLlyK0aotB9JzaIaZvHCT2jd9Wm0bVVWJwvn18cCWypnDRQtW/mqz/dTFW1Srakm91v45FS/sp8E9GqpciYL6euk2SZKHW04t/bK3mtV+UkULPaanyhTWB/1a6MCR0zpzISo9h2Yo5qr9UVPHoK721+vFypqz5oAWrD2ko39H6q0v1uhafILa1y1ns33L58ros0XbtW7XXzp1IVozVu7Tul1/6bXmlS1t1uw4rnW7/tKJc5f119kojZq1VVdv3NRTwQXSa1iGm7hwkzo0raZ2jasqOCi/PhnUSjlzuGj+ilTm6uLNeq5qSfW5M1ff6dlI5UoEaPqSrZY2rRpUVv9u9RVSqUR6DSPD4T3A/qYs2qx2TZ5Wm0ZVVaKwn8YNaKmcri5alMq+1bRvtqhmlWD1bvecigf6aeArDVW2REHN/Habpc2Lzz+lGpVKqJB/PgUH5deIvi/oytUbOvzXWZvbzAgMD2Surq7y8/OTn5+fKlSooEGDBun06dOKiIh44HqdOnXSzJkzrZbNnDlTnTp1euB6K1as0DPPPKP+/furRIkSKl68uJo1a6YJEyZY2nz++efaunWrNmzYoN69e6tChQoKCgpS27ZttWPHDhUrVuyfDzgd3LyVoP1HTiu08r03TScnJ4VULqFdh07aXGfnoZMKrRRstaxW1ZLadShMknTqbKQuRsYqtPK9Np5uOVWxdKB2HQyz+xgympu3EnTg6GmrDyInJyfVqFRCu1Op6e7fwlSjUnGrZTWrlky1vSTFxt2QyWSSp3tO+3Q8g2Ou2h81dQzqan/ZnZ1UoZifNu+9Vz+zWdqyL0yVStn+ctU1u7Nu3EywWnYjPkFVS9s+dc7JyaQXQ0oql2t27TqccXfG7OnmrQQdOHJaIcnnaqUSlrmX3K5DYSmCVq2qwanO7ayI9wD7u3krQQePnlaNp+7tKzk5Oal6peLa/VuYzXX2/HZSNZLN1dAqwam2v3krQXO/3y4Pt5wqVfTBB22MZHggu19cXJzmzZunokWLytvb+4FtmzRposuXL+vnn3+WJP3888+6fPmyGjdu/MD1/Pz89Pvvv+u3335Ltc38+fNVu3ZtPfHEEymey549u3Lnzm1zvfj4eMXGxlo9jBAZHafbtxPl4+VutdzHy0Phkbb7FB4ZKx/v5O3dLe0v3vlv8ja+3u6pbjMziYq+arOmvnndFR5p+7B6eGSsfL08rJb5PKD9jfhben/C93qxzpNyz501Ahlz1f6oqWNQV/vz9sgl52xOioi+ZrU84vJV+eZ1s7nOxj0n1Kt5ZQUVyCuTSQp9MlCNnimhx7ys25cK9NHp5W/r4soB+rRvPXV4/zsd/TvSYWPJSCLvfF4l//zx9XJXeNQD5mqKuZ3651VWxHuA/aW2b+Xj5a7wVE5ZDI+8Ip+8ydrnTVmvtf/7TUHP9Veh0H6aumizFn/+qrzz2H5fyQgMD2QrV66Um5ub3Nzc5O7urh9++EGLFy+Wk9ODu5Y9e3a1b99eM2bMkCTNmDFD7du3V/bs2R+4Xp8+fVSpUiWVLVtWgYGBat26tWbMmKH4+HhLmz///FPBwcEP2IptY8eOlaenp+UREBCQ5m0ga7qVcFvdhsyU2Sx9NLCl0d0BgAxp0KR1OnH2snZOf0XhqwZqXK/ntWDtQSWazVbt/jwTqRq9Zqh239masXKvJvZrpBKPP/iLXgCZxzNPFtOG2QO0csobqlk1WK+8OyvV69IyAsMDWc2aNbV//37t379fO3fuVN26dVW/fn2dOnVK9evXt4S10qVLp1i3a9euWrJkiS5cuKAlS5aoa9euD3293Llza9WqVTp+/LiGDh0qNzc3vf3226pcubKuXUv6ls6c7I39UQ0ePFgxMTGWx+nTp//Rdv4t7zxuypbNKcXEi4iKla+3h811fL09FBGZvP0VS/vH7vw3eZvwyCupbjMz8cqT22ZNwy9fkW+yb7bu8vX2SPFtZISN9nfD2JkLUVr6Ze8sc3RMYq46AjV1DOpqf5Gx15RwO1E+eaxv4OGTN7fCU7n4PjLmutqP+Fb+TT9WuQ4TVLnbVF29cVNhF6Kt2t1KSNTJc5d14PgFvT9zi347eVE9m1Vy1FAyFO87n1fJP3/Co66kOGp2l6+3h425nfrnW1bEe4D9pbZvFRF1Rb5eqe1buSvicrL2l1PWK3dOVxUu6KOKZQL12Ttt5ZzNSQtTuS4tIzA8kOXOnVtFixZV0aJFValSJU2fPl1Xr17VtGnTNH36dEtYW716dYp1y5Ytq+DgYLVp00YlS5ZUmTJlHvl1ixQpom7dumn69Onau3ev/vjjDy1evFiSVLx4cR05ciTNY3F1dZWHh4fVwwgu2Z1VIThAW3YdtSxLTEzU1l3HVKlsYZvrVC5b2Kq9JG3acUSVygZKkgr5e+sxbw+rNrFx17Xn9zBVKhdo9zFkNC7ZnVW+RIC27jpmWZaYmKhtu47qqVRq+lSZQG27r70kbdl5xKr93TB24nSEln7ZW16etk+HzayYq/ZHTR2DutrfrYRE7f/zgkKeCLQsM5mkGhUKadcfD77eK/7WbZ2PjJNzNic1fjZYP/7y5wPbO5lMcsmezR7dzvBcsjurfHDKz6ukuRpoc51KZQO1dbf159XmnUdTndtZEe8B9ueS3VnlSgRo2x7rufrz7mN6qkygzXUqlimsbcnm6tadR1Ntf2+7ZsUnu/40IzE8kCVnMpnk5OSk69evy9/f3xLWChWy/fcIunbtqs2bNz/S0bHUBAYGKleuXLp69aokqW3btlq/fr327duXou2tW7cs7TKyXm1rac7y7Vq48lcdPXlBb32wWFevx6td46qSpJ7D5mjEV99b2vdoHaoNv/yhr+Zt0LGwC/pg6irtP/y3ur8UIinp36Vnm5r6eMYard5yUL8fP6tXh8+VXz5PNQwpb8gY01vPNjU174ftWrRqh46dvKD+477RtRs31aZhFUlS7xFzNXLiD5b2r7QK0cZfD2vi/I36M+yixk1brf2HT+vlFtUlJYWxroO/1v7Df2vSiI66nWjWxchYXYyM1c1bGfdNw96Yq/ZHTR2DutrfxO92qmP9Cmpdu6yKB3jr0z71lDtHds1fm/Tnbyb1b6T3uoRY2lcsUUCNnimuQn55VK1MQS0d3UpOJmn8N/e++X6vS4ieLhOggMc8VSrQR+91CdGz5QppyaaMf4dke+nVpqbmfr9dC1ft0NGTF9Tvw2907Ua82jZKmquvDp+j9yfc+7zq0Spprk6YnzRXP5y2WvsP/61uL9WwtLkcc1WHjp3R0ZMXJEnHT13UoWNnLNdBZQW8B9hfj9ahmv/DL1q8eqeOhV3QwI+W6NqNm2rdKGnf6rX352n0pBWW9t1bhmjTr4c1aUHSvtVH03/UgSOn1aV50r7V1evxGjN5hfb8FqbT56N04MhpvTF6gS5cilHjWhWMGOIjMfwPQ8fHx+vChaRf7suXL+urr75SXFzcQ2/OcVf37t310ksvKU+ePI/Ufvjw4bp27ZoaNGigQoUKKTo6Wl988YVu3bqlOnXqSJLeeOMNrVq1Ss8995xGjhypZ599Vu7u7tq9e7c+/PBDff3116pQocI/GW66efH5iroUHacxU1YpPPKKyhb319IvelsO6Z65ECUnk8nSvkr5IE0b1VmjJ63UyIkrFBTgo3kfv6JSRe/dJvj1jrV17Xq83hyzUDFx11W1fBEt/aKXcrg++Lq9zOKFOk8qMjpOH05bfecPQhbU4s9eva+ml2W6r6aVywVp8vudNHbKKo2evEJBAb6aPa6bShZJqun58Git2ZZ0c5maHT60eq3lE/romYoZ+26e9sJctT9q6hjU1f6WbTmsfJ659E7H6vLNm1uHToSrxZBvLDf6KOjjocTEe5cRuLpk05BOIQrMn0dXr9/Uul1/qee4FYq9eu868Hx5cmtS/0Z6zMtNsdfi9fvJcDUfskib94al9/AM80KdpLn6wdSkuVqmuL+++byXZa6evXhZTk7Wn1dTR3bW6MkrNWrSSgUF+GjuuO6WzytJ+nHbIfUZOd/yc7ehsyRJA7rV18DuDdJnYAbjPcD+mtVO2rcaN221IqJiVbpYQS38tKd8vGzP1UplC2viiI76cOpqjZ2yUoUL+mjmBy9b5mo2JycdPxWub1bPUFRMnPJ65laF4Me1fGJfBQflN2SMj8Jk/qcXTNlB586dNXv2bMvP7u7uCg4O1sCBA9W8eXOb64SFhalw4cLat2+fzVC0f/9+PfHEEzp58qQCAwNTtN+0aZMmTJignTt36uLFi8qbN6+eeOIJDRkyRM8++6xlO/Hx8frss8+0YMEC/fnnn8qVK5dKliyp7t27q127dnJ2fniWjY2Nlaenpy5Gxhh2+mJmlHA70eguZErO2TLcAXMA6Shv3bFGdyHTifxxkNFdyJTu30GHfcTfum10FzKd2NhYPe7npZiYh+cAQwNZZkcgcwwCmWMQyICsjUBmfwQyxyCQ2R+BzP7SEsjYAwMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAzibHQHgLRyzsb3CEBWlphoNroLmVLUmkFGdyHT8aox2OguZEqXt31gdBcyHfat7C8tNaX6AAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAD+396dh1VV7m0cvzcgOAAqiKJIqCGCiljmVO/J4Wg4D2nOlcehHHIqi0xzyLFOZVmaHsfMCWdTy3LWModK1IRQU9QU01BBHEBlvX8YW7cMDu3tQvp+rmtfXqz1rLWf58fy2dysAQAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASF7M7AMeZunCzPpmzXqcTklSxrJ/eff05ValQKsv2y9f9rDGTV+tYfILK+PtoeJ8WeuapCtb1hmFo7JTVmr18mxKTL6t6pTL64M22evSRog9gNDkDNXUM6mp/1NQxpi3aok/n3qhrhbJ+Gvda62zrumL9bo2ZskrH48+qjL+PhvVurvq31HXlxijNWvq99vx6TOeSLmnTFxEKDSr5AEaSc0xbtMV6rFYo66d3B2Zf0+XrdmvslFU69ldNh7+SsaYzl36vPTE3arp5zj+vppLUrWUN9WlfS0W93PXLb/GK+OhL/Rzze6ZtXZydNOD5Omrf4HEVL+KpQ8f/1PDPvtb6nQesbfYsjNAjxQtn2Hba0h/0+vgVDhtHTsPcan/TM5lXH7/DvDr2lnl16G3z6qrb5tWND8G8yhmyXGrptz9pyEfLFNGtoTZ9EaGKZf3Uqs9EnTl7IdP2O/YcVrchs9SpeU1tnvOmGtcKU6eB/1P0oZPWNh/PXqcpkZv14aB2WjtzoPLnc1WrPhN1JeXqgxqWqaipY1BX+6OmjrFs7U96++Nler1rQ234/A1VDPTTc/0mZVnXnXsPq/vbs9SpaU1tnB2hRk9X0vNvTFXMbzfreulyqmqEldGwV5o/qGHkKEvX3jhW3+jWUBtnv6GKZf3Uum/WNd3xV007NqupTV9EqFGtSur0+lRFU1MbLetW0qhXmujdWetUu9sn+uVQvJZ80FVFChXItP2Q7s+oc7NqivjoS9V4frxmrtiuL8Y8r9CyJaxt6r70qco1H2V9teg/TZK0fOO+BzKmnIC51f5un1cr3MW8+tLbs9Txlnn1hUzm1ephZTT0IZoDcnQg69y5s1q0aJFtm8uXL2vYsGEKCgqSm5ubihQpoueee0779++3aTd8+HBZLBZZLBY5OzvL399fL730ks6ePZthn7t371bbtm1VvHhxubm5KSAgQE2aNNHKlStlGIY9h+gwk+Zt0AstnlTHZjUVXKa4PhzUTvnzumrOlz9k2n7Kgk36d80Q9X2+nsqV9tXgnk0UFuyvqYs2S7rxG5zJ8zdqYJdwNapVSRXL+umzES/o1J+JWr15z4McmmmoqWNQV/ujpo4xaf5GPd+8pjo2raHgMsX1wZttlS+vq+auzKKukZv07xoh6vNXXd/q0USVyvlr2qIt1jZtG1XT690aqlbVcg9qGDnKpHkb9UKLmzX98M22yp9dTRfcqKn1WO3RRJWC/TVtoW1N3+jWULWr/TNrKkm92v6fZq/cqXlf/aTYuNN69f3lunQlVZ0aP5Fp+zbhj2v8Fxu1dnusjsaf1YzlO7T2h1i90u5f1jYJ5y/q9Nlk6yv8yWAd/v1PfR91+EENy3TMrfb32V/zaoemNVTulnl1Xjbzat2/5tWg0r4alMm82uYhnFdzdCC7k5SUFNWrV08zZszQqFGjdODAAX311Ve6du2aqlevru3bt9u0r1ChguLj43Xs2DHNnDlTa9asUc+ePW3arFixQjVq1FBycrI+//xzxcTEaM2aNWrZsqWGDBmixMTEBznE+5J69Zqifj1u82Hk5OSkWtXKade+I5lus3PfEdWuGmyzrG6NEO3aFydJOnoiQX8kJKl2tZttCrrnU5UKpbRrb5zdx5DTUFPHoK72R00dI/XqNe359bhq3V7XquWsdbrdrn1xGX4gqFsjOMvvwz+NtaZV77Gm1ahpdvK4OKtykJ82/XTIuswwDG3+8ZCqVgjIdBu3PM66knrNZtmV1KuqEVoqy/do88xjmvvVj3brd07H3Gp/9zOv/pjJvFqnRrB+fMjngIf6HrKPPvpIP/zwg3bv3q2wsDBJUkBAgJYsWaLq1aura9eu+uWXX2SxWCRJLi4u8vX1lST5+fnpueee08yZM637u3jxorp27arGjRtr6dKlNu8VEhKirl27PhRnyBLOJ+v69TT5eHnYLPfx8tTBuD8y3eZ0QpJ8vG9v76HTCUmSpD/++vf2NkW9b7bJzaipY1BX+6OmjpFw/qKuX09TUS9Pm+VFvTx08Gg2dc3wffDQ6YTML8X5p0mvqc9tNfXx8tCBbGpa9LaaFvXy0OksLm/6J/IumF8uLs46czbZZvmZc8kqG+CT6TYbdh5Ur7b/0rY9R3TkxFnVqvKomjxdQc5Omf/evvG/yquge17N++onu/c/p2Jutb/s5oB7mVeL5oJ59aE+QzZv3jzVr1/fGsbSOTk5acCAAYqOjtaePZmf8o2Li9M333wjV1dX67Jvv/1WCQkJeuONN7J8z/Rwl5mUlBQlJSXZvAAAAHKyNyes1OHf/9TOOa/p9IZRem9Ac8376ielZfFL6E5NqmrdjgM69ZD/EAzkFA91IDtw4IBCQkIyXZe+/MCBm08I2rdvn9zd3ZUvXz6VLl1a+/fvV0REhM3+JKlcuZunQnft2iV3d3fra9WqVVn2Z+zYsSpYsKD15e/v/7fGd7+8C7nL2dkpww2RZ84mqai3Z6bbFPX21JmE29tfsLYv9te/t7c5nXAhy33mJtTUMair/VFTx/AuVEDOzk46fdb2F22nz17IcNYsXVFvz0y+DxdU9Lbfhv9Tpdf0zG01PXP2gvWYu11Rb88MZ8NufA+oabqExEu6du26fLzcbZb7FHbX6YTkzLc5f1Gd3vpCfs8MVaXn3lW1jh/o4uUUxZ3MeJ+9f7FCql0lULNX7XJI/3Mq5lb7y24OuJd59XQumFcfikA2d+5cm1C0detW67p7uYSwXLlyioqK0q5duxQREaHw8HD16dMn220qVaqkqKgoRUVF6eLFi7p27VqWbQcNGqTExETr6/jx43fdN3tyzeOiysH+2rwr1rosLS1NW3YdUNXQ0pluUy20tE17Sdq441dV/ev68QA/bxXz9rRpk5R8WT/tj1PVSqXsPoachpo6BnW1P2rqGK55XBQW7K8tu27+ku9mXUtluk3V0FLa8uMBm2WbdsZm+X34p8mqppt/vENNd91W0x3U9FZXr11X1IETqlUl0LrMYrHo6SqB2rX/aLbbpqReU/yfSXJxdlLTWhX19XfRGdp0aPSEzpxP1rc//Gr3vudkzK32dz/z6hOZzKubd8bqiYd8DngoAlmzZs2soSgqKkpPPHHjKUFBQUGKiYnJdJv05UFBQdZlrq6uCgwMVMWKFTVu3Dg5OztrxIgR1vVly5aVJMXG3vyP4ebmpsDAQAUG3pzYsuLm5iZPT0+bl1l6dair2cu3af6q7Yo9ckqvjovUxcsp6ti0hiSpx7DZGvHpzb8b8nK72lr/Q7Q+nbNeB+JOadz/Visq5pi6P1dL0o3JvEf7Onp/xhp9tXmv9h86oZ7Dv5BvkYJqXCss0z7kNtTUMair/VFTx+jVvo6+WLFN81fvUOyRUxr47kJdupKiDk1u1LXn8Nl6Z+KX1vYvt71R14lzb9T13alfKSrmmLo997S1zbnEi9p34HfFHjklSTp09A/tO/C79d6S3K5XhzqavWKb5q+6UdPX3l2oS5dvqemw22qafqzOTT9W/6ppm6xrejC9pn/+M2oqSZMiv9MLTaqqXYPHFRTgow9fa6EC+Vw19697vj4b3EZDXw63tq9S3l9Nnq6ggOJeqlmplBZ/0EVOThZ9PG+zzX4tFos6NqqiBV//rOvX0x7omHIC5lb76/nXvLpg9Q4duGVebf/XHNBr+GyNvG1e3fDXvHowF82rD8VDPTw8POThkfFUZLt27TR48GDt2bPH5j6ytLQ0jR8/XuXLl89wf9mthgwZorp166pnz54qUaKEnnnmGXl5eendd9/VsmXLHDKWB+XZZ6roz/PJGjNltU4nXFBokJ8WT+htPQX++6mzcrrlfrjqYWU0dVRnjf5slUZOWqky/j6a8/5LKh9482+Q9Huhni5dTtGAMfOVmHxZNcIe1eIJvZTXLc8DH58ZqKljUFf7o6aO0bL+jbqO+9+NulYM8tPCj3pZ63rij3NycrpZ12qVyuh/Iztr9ORVGvXZKpXx99EX73VXyKM36/r11n3qM3Ku9etuQ2ZJkt7o1lAR3Rs9mIGZ6Nn6VZRwLlljb6npoo9v1vT322pa/a+ajpm8SqMm3ajpnP92V/nbavrKO7fUdPAsSTdq+uZLub+mkrRsw14VKVRAb3Wtr6JeHtp36KRaD5yhM+duXLJYslghm/vD3FxdNLj7MypV3EsXL6dq7fZY9RgZqaTkKzb7rf1EoPx9C2vOP+jpirdibrW/lvWrKCGbefX2OaBapTKa8tccMPqveXX2bfPqmtvm1e5/zauv5+B51WLk4McGdu7cWefPn9fy5cszXX/lyhXVrl1bJ0+e1AcffKDq1avrjz/+0JgxY7R27VqtW7dONWrcSNjDhw/X8uXLFRUVZbOP6tWrq2rVqvr0008lScuWLVPbtm1Vv3599e3bV2XLllVycrLWrFmjiIgIffnll2ratOld9T8pKUkFCxbUHwmJpp4tA4DcJC0tx35sPdSyeWYV7pPX04PM7kKudG7rOLO7kOtcZ161u6SkJJXwKaTExDvngIfiksWs5M2bVxs2bNALL7ygt956S4GBgWrQoIGcnZ21fft2axjLzoABAzRt2jTr/V4tW7bUtm3blD9/fr3wwgsqV66c6tatqw0bNmjBggVq0qSJo4cFAAAA4B8iR58he9hxhgwA7I8zZI7BGTL74wyZY3CGzP44Q2Z//5gzZAAAAADwMCOQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJjExewOAABwL5ycLGZ3Abgr57aOM7sLuVLhan3M7kKuc27nJ2Z3IddxvofPKs6QAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQ5WJTF25WpWZD5ftUf9Xr/F/9tD8u2/bL1/2saq1Hyvep/nqy3Wh9+/1+m/WGYWjM5FUKbvCWiv/fALXo9Yl+O3bagSPIeaipY1BX+6OmjkFd7Y+aOgZ1tb9urf+lPcuHK37rh1o74zU9Xj4gy7Yuzk56vWsD/bx0qOK3fqitc9/Uv2uE2LRxz++mMQOe1d4VI3Ryywf6ZtoAPRbyiKOHkeNwrBLIcq2l3/6kIR8tU0S3htr0RYQqlvVTqz4TdebshUzb79hzWN2GzFKn5jW1ec6balwrTJ0G/k/Rh05a23w8e52mRG7Wh4Paae3Mgcqfz1Wt+kzUlZSrD2pYpqKmjkFd7Y+aOgZ1tT9q6hjU1f5a1ntco/q31LvTvlbtF97TLwdPaMmEXipS2D3T9kN6NlHnlk8p4v3FqtF2tGYu/U5fvNdNoUElrW0+HtxBtasHq8fw2Xqqw1ht2PGrlk98RcV9Cj6oYZmOY/WGHBnIOnfurBYtWmS5vnbt2rJYLBo3blyGdY0bN5bFYtHw4cNt2vfv39/69ZEjR9ShQweVKFFCefPmVcmSJdW8eXP9+uuvNvvauHGjGjVqJG9vb+XPn1/ly5fXa6+9phMnTvzdITrcpHkb9EKLJ9WxWU0FlymuDwe1U/68rprz5Q+Ztp+yYJP+XTNEfZ+vp3KlfTW4ZxOFBftr6qLNkm78tmHy/I0a2CVcjWpVUsWyfvpsxAs69WeiVm/e8yCHZhpq6hjU1f6oqWNQV/ujpo5BXe2vV4c6mr38B81btUOxR07p1XGRunQlVZ2a1sy0fZuG1TR+1rdauy1aR08maMaS77R2W7Re6VhXkpTXLY+a1QnT8E9WaNvu33Tk9z/17tSvdfj4GXVp9X8Pcmim4li9IUcGsrvh7++vWbNm2Sw7ceKE1q9fr+LFi2e53dWrV1W/fn0lJiZq6dKlio2NVWRkpEJDQ3X+/HlruylTpqhevXry9fXVkiVLFB0drcmTJysxMVEffPCBg0ZlH6lXrynq1+OqXa2cdZmTk5NqVSunXfuOZLrNzn1HVLtqsM2yujVCtGtfnCTp6IkE/ZGQpNrVbrYp6J5PVSqU0q69cXYfQ05DTR2DutofNXUM6mp/1NQxqKv95XFxVuVgf23aFWtdZhiGNu+KVdXQUplu4+bqoiuptmdkrqRcVY2wMpJuXNLo4uKcRZtH7TuAHIpj9SYXsztwv5o0aaKFCxfq+++/11NPPSVJ+vzzz/XMM8/o2LFjWW63f/9+/fbbb1q/fr0CAm5c+xsQEGDdhyT9/vvv6tu3r/r27avx48dbl5cqVUpPP/20TXC7VUpKilJSUqxfJyUl/Z0h3reE88m6fj1NPl4eNst9vDx1MO6PTLc5nZAkH+/b23vodMKNMfzx17+3tynqfbNNbkZNHYO62h81dQzqan/U1DGoq/15FyogFxdnnTlrO9YzZy+obECxTLfZsD1GvTrUtZ79qlU1SE3qhMnZySJJSr6Uop17D+v1Lg104MgpnT57Qa2fqaKqoaV1+PczDh9TTsCxetNDe4bM1dVVHTt21MyZM63LZs2apS5dumS7nY+Pj5ycnLR48WJdv3490zaLFi1Samqq3njjjUzXFypUKNPlY8eOVcGCBa0vf3//uxsMAAAAco03P1iiw8fPaOfCITr9/Xi99/pzmrdyu9LSDGubl4d9IYtFivlqtP74brxealtbS779yaYN/hke2kAmSV26dNHChQt18eJFbdmyRYmJiWrSpEm22/j5+WnChAkaOnSoChcurLp162rkyJE6fPiwtc3Bgwfl6emZ7aWPmRk0aJASExOtr+PHj9/XuP4u70LucnZ2ynBD5JmzSSrq7ZnpNkW9PXUm4fb2F6zti/317+1tTidcyHKfuQk1dQzqan/U1DGoq/1RU8egrvaXcP6irl27Lh8v27HeemYm4zbJ6vT6VPnVek2Vmg9TtedG6eLlFMWdTLC2iTvxp5r0mCC/p19TxaZDVe8/78vFxVlHTyRkus/chmP1phwdyObOnSt3d3fra+vWrTbrw8LCVLZsWS1evFgzZszQ888/LxeXO1+F2bt3b506dUpz585VzZo1tWjRIlWoUEFr166VdOO6YIvFcs/9dXNzk6enp83LDK55XFQ52F+bb7nWOS0tTVt2HVDV0NKZblMttLRNe0nauONX67XRAX7eKubtadMmKfmyftofp6qVStl9DDkNNXUM6mp/1NQxqKv9UVPHoK72d/XadUX9ely1qgZZl1ksFj39RJD13qWspKReU/yZRLk4O6lpncr6evO+DG0uXUnVHwlJKuiRT/+uEayvtuy19xByJI7Vm3L0PWTNmjVT9erVrV/7+fllaNOlSxdNnDhR0dHR2rlz513v28PDQ02bNlXTpk01atQohYeHa9SoUapfv76CgoKUmJio+Pj4ez5LllP06lBXvUZ8ocdCHtHjFUrps/kbdfFyijo2rSFJ6jFstor7FNSwV5pLkl5uV1tNXv5In85Zr2f+r4KWfvuTomKO6aO32ku6MfH0aF9H789YozL+Pgrw89aYyavlW6SgGtcKM22cDxI1dQzqan/U1DGoq/1RU8egrvY3ad5GTRrWSbtjjunn/UfVs11tFcjnprmrtkuSPhv+vOJPn9c7k1ZKkqpUCFBxn0Lad+B3lShaSBHdG8rJyaKPv1hn3WfdGsGyyKKDx06rTMkieqdvCx2I+0NzV243ZYxm4Fi9IUcHMg8PD3l4eGTbpkOHDho4cKDCwsJUvnz5+3ofi8Wi4OBgbdu2TZLUunVrvfnmm3rvvfdsHuqR7vz581neR5ZTPPtMFf15PlljpqzW6YQLCg3y0+IJva2na38/dVZOt5wFrB5WRlNHddboz1Zp5KSVKuPvoznvv6TygSWsbfq9UE+XLqdowJj5Sky+rBphj2rxhF7K65bngY/PDNTUMair/VFTx6Cu9kdNHYO62t+ydT+rSGF3vfVSYxX19tC+AyfUut8k6+V2JYsVtrn3y801jwb3aKxSfkV08XKK1m6LVo9hs5WUfNnaxtM9n4b2aqoSRQvpXNIlrdywR6M+W6lr19Me+PjMwrF6g8UwjBx352Dnzp11/vx5LV++PNP1tWvXVuXKlfXRRx9JuhGQ8uTJowIFCkiSKleurBYtWlj/Ftmt7aOiojRs2DA9//zzKl++vFxdXbV582b169dPERERevvttyVJkyZN0iuvvKL//Oc/euGFF1SqVCn9/vvvmj17ttzd3e/q0fdJSUkqWLCg/khINO3yRQAAgNykcLU+Znch1zm38xOzu5DrJCUlqZh3QSUm3jkH5OgzZHfrXs5WlSxZUqVKldKIESMUFxcni8Vi/XrAgAHWdr169VJQUJDef/99tWzZUpcvX1apUqXUpEkTvfrqqw4YBQAAAIB/mhx5hiy34AwZAACAfXGGzP44Q2Z/93KGLEc/ZREAAAAAcjMCGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJnExuwMAAADA3Tq38xOzu5DrFK76itldyHWM66l33ZYzZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgJZLjZ14WZVajZUvk/1V73O/9VP++Oybb983c+q1nqkfJ/qryfbjda33++3WW8YhsZMXqXgBm+p+P8NUIten+i3Y6cdOIKch5o6BnW1P2rqGNTV/qipY1BXx6Cu9vXkY49q/ocvK/qr0Tq361M1qlXpjts89XhZbfoiQqe+H6+flg5T+ybVM7Tp9tzT2rNihOK/G6+1Mwfq8fIBjui+3eSYQNa5c2dZLBbry9vbWw0aNNDevXvvuO3+/fvVpk0b+fj4yM3NTUFBQRo6dKguXbpk065UqVLW/efPn1+hoaGaNm1ahv0ZhqGpU6eqZs2a8vT0lLu7uypUqKB+/frp0KFDdhuzIy399icN+WiZIro11KYvIlSxrJ9a9ZmoM2cvZNp+x57D6jZkljo1r6nNc95U41ph6jTwf4o+dNLa5uPZ6zQlcrM+HNROa2cOVP58rmrVZ6KupFx9UMMyFTV1DOpqf9TUMair/VFTx6CujkFd7S9/Pjf9cuCEXn8v8q7aP1LCW5Ef9dDWnw7o6Y7jNHn+Rk0Y3EF1a4RY27Ss/7hG9W+pd6d9rdrPv6tfDp7Qkk96q0hhd0cN42/LMYFMkho0aKD4+HjFx8dr/fr1cnFxUZMmTbLdZvv27apevbpSU1O1evVqHThwQKNHj9asWbNUv359paam2rR/5513FB8fr19++UWdOnVS9+7d9fXXX1vXG4ahDh06qG/fvmrUqJG+/fZbRUdHa/r06cqbN69GjRrlkLHb26R5G/RCiyfVsVlNBZcprg8HtVP+vK6a8+UPmbafsmCT/l0zRH2fr6dypX01uGcThQX7a+qizZJu1GXy/I0a2CVcjWpVUsWyfvpsxAs69WeiVm/e8yCHZhpq6hjU1f6oqWNQV/ujpo5BXR2Dutrfum3RGj15lVZvuvMJGEnq8uz/6djJBL390TIdiPtDUxdt0ZcbotSzQx1rm14d6mr28m2at3K7Yo+c0qtjF+jSlVR1albTUcP423JUIHNzc5Ovr698fX1VuXJlvfnmmzp+/LjOnDmTaXvDMNS1a1eFhIRo6dKlqlatmgICAvTcc89p5cqV+uGHHzR+/HibbTw8POTr66syZcooIiJCXl5eWrt2rXV9ZGSkFixYoMjISL399tuqUaOGHnnkEdWoUUPvvvuuZs6c6dAa2EPq1WuK+vW4alcrZ13m5OSkWtXKade+I5lus3PfEdWuGmyzrG6NEO3aFydJOnoiQX8kJKl2tZttCrrnU5UKpbRrb5zdx5DTUFPHoK72R00dg7raHzV1DOrqGNQ1Z6gaWlqbdsbaLFu/PUbVQktLkvK4OKtysL9NG8MwtHlnrKr+1SYnylGB7FbJycmaM2eOAgMD5e3tnWmbqKgoRUdH69VXX5WTk+1QwsLCVK9ePc2fPz/TbdPS0rRkyRKdO3dOrq6u1uXz589XuXLl1KxZs0y3s1gsWfY5JSVFSUlJNi8zJJxP1vXrafLx8rBZ7uPlqdMJmffpdEKSfLxvb+9hbf/HX//e3qaot0eW+8xNqKljUFf7o6aOQV3tj5o6BnV1DOqaMxT19sxwieiZhCR5uudTXrc88i7kLhcX54xtziapqLfng+zqPclRgWzVqlVyd3eXu7u7PDw89OWXXyoyMjJD2Ep34MABSVJISEim60NCQqxt0kVERMjd3V1ubm5q3bq1ChcurG7dutnss1y5cjbb9O/f39qvkiVLZtn/sWPHqmDBgtaXv7//XY0bAAAAwD9TjgpkderUUVRUlKKiorRz506Fh4erYcOGOnr0qBo2bGgNRRUqVLDZzjCMu36P119/XVFRUdqwYYOqV6+u8ePHKzAwMNttBg8erKioKA0dOlTJyclZths0aJASExOtr+PHj991v+zJu5C7nJ2d7um3A0W9PXUm4fb2F6zti/317+1tTidcyNG/cbAXauoY1NX+qKljUFf7o6aOQV0dg7rmDKcTkjKepfT2VFLyZV1JuaqE88m6du36PZ3JzAlyVCArUKCAAgMDFRgYqKpVq2ratGm6ePGipk6dqmnTplnD2ldffSVJCgoKkiTFxMRkur+YmBhrm3RFihRRYGCg/vWvf2nRokXq27evoqOjrevLli2r2Fjba1N9fHwUGBiookWLZtt/Nzc3eXp62rzM4JrHRZWD/bV5181xpKWlacuuA1leP1sttLRNe0nauONXVQ0tJUkK8PNWMW9PmzZJyZf10/44Va1Uyu5jyGmoqWNQV/ujpo5BXe2PmjoGdXUM6poz7Np3RLWq2l7JVqdasHb+dR/f1WvXFfXrcZs2FotFT1cNyvJev5wgRwWy21ksFjk5Oeny5cvy8/OzhrWAgBt/S6By5coKDg7W+PHjlZaWZrPtnj17tG7dOrVv3z7L/fv7+6tt27YaNGiQdVn79u0VGxurFStWOGZQD0j6E2bmr/rrCTPjInXxcoo6Nq0hSeoxbLZGfHpzjC+3q631P0Tr0znrdSDulMb9b7WiYo6p+3O1JN34XvRoX0fvz1ijrzbv1f5DJ9Rz+BfyLVJQjWuFmTLGB42aOgZ1tT9q6hjU1f6oqWNQV8egrvZXIJ+rKgb5qWKQnyQpoIS3Kgb5qWSxwpKkob2b6bPhz1vbz1j6nQL8vDWiT3OVDSimrq3/pRb1HtNn8zZa26Q/DbNd4+oKKlVMH77ZVgXyuWnuyu0PdnD3wMXsDtwqJSVFp06dkiSdO3dOn376qZKTk9W0adNM21ssFk2fPl3169dXq1atNGjQIPn6+mrHjh167bXXVLNmTfXv3z/b9+zXr58qVqyoH3/8UU888YTatWunpUuXql27dho0aJDCw8NVrFgxHT16VJGRkXJ2drb3sB3i2Weq6M/zyRozZbVOJ1xQaJCfFk/obT0F/vups3K65QEl1cPKaOqozhr92SqNnLRSZfx9NOf9l1Q+sIS1Tb8X6unS5RQNGDNficmXVSPsUS2e0Et53fI88PGZgZo6BnW1P2rqGNTV/qipY1BXx6Cu9lc5JECrpvSzfj3m1VaSpHmrtqv3iDkqVsRTJX29rOuPnUxQ2/6TNebVZ/Vyu9o6efq8+o6epw3bb14tt2ztzypSyF1vvdxYRb09tO/ACbXum/Xfi8sJLMa93IDlQJ07d9bnn39u/drDw0PBwcGKiIhQq1atst123759GjFihDZu3KgLFy7okUceUfv27TVo0CDlz5/f2q5UqVLq379/hpDWoEEDOTk5WS+FTEtL09SpUzVz5kz98ssvunr1qkqWLKl///vfGjBgQJYPEbldUlKSChYsqD8SEk27fBEAAADITuGqr5jdhVzHuJ6qlH1TlZh45xyQYwJZbkQgAwAAQE5HILO/ewlkOfoeMgAAAADIzQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYxMXsDuRmhmFIki4kJZncEwAAACBzxvVUs7uQ66TXND0PZIdA5kAXLlyQJAWW9je5JwAAAAAetAsXLqhgwYLZtrEYdxPbcF/S0tJ08uRJeXh4yGKxmN2dbCUlJcnf31/Hjx+Xp6en2d3JNair/VFTx6Cu9kdNHYO62h81dQzq6hgPS10Nw9CFCxdUokQJOTllf5cYZ8gcyMnJSSVLljS7G/fE09MzRx/cDyvqan/U1DGoq/1RU8egrvZHTR2DujrGw1DXO50ZS8dDPQAAAADAJAQyAAAAADAJgQySJDc3Nw0bNkxubm5mdyVXoa72R00dg7raHzV1DOpqf9TUMairY+TGuvJQDwAAAAAwCWfIAAAAAMAkBDIAAAAAMAmBDAAAAABMQiADAAAAAJMQyHKJzp07q0WLFlmur127tvr375/l+rNnz6p///4KCAiQq6urSpQooS5duujYsWMZ2p46dUp9+vRRmTJl5ObmJn9/fzVt2lTr16+3w0hyh+PHj6tLly4qUaKEXF1dFRAQoH79+ikhIcHsruUId3O8WiwWWSwW5c2bV+XLl9ekSZOs62fNmmVdf+srb968Nu+RvjxPnjwqXbq03njjDV25csWRQ3ugbh2jxWKRt7e3GjRooL1792a5TVxcnCwWi6KiorJss23bNjVq1EiFCxdW3rx5FRoaqg8//FDXr1/P0Hbjxo1q1KiRvL29lT9/fpUvX16vvfaaTpw4YY8hmuLv1PXWbZ555hnt3r3b2ubW4/rWV48ePaxtbl3u6empqlWrasWKFQ4dr6P8nTo6OztnOIbi4+Pl4uIii8WiuLg4m/a3Hs/Lli1TjRo1VLBgQXl4eKhChQoZPv9SU1P13nvvKSwsTPnz51eRIkX01FNPaebMmbp69aq9SvBA3Gk+laTLly9r2LBhCgoKkpubm4oUKaLnnntO+/fvt2k3fPhw6/fL2dlZ/v7+eumll3T27NkM+9y9e7fatm2r4sWLy83NTQEBAWrSpIlWrlyph/l5cXf7+TRu3LgM6xo3biyLxaLhw4fbtL/1+Dty5Ig6dOigEiVKKG/evCpZsqSaN2+uX3/91WZfuWVuvZ95IN3+/fvVpk0b+fj4yM3NTUFBQRo6dKguXbpk065UqVLW/efPn1+hoaGaNm1ahv0ZhqGpU6eqZs2a8vT0lLu7uypUqKB+/frp0KFDdhvz/SCQQWfPnlWNGjW0bt06TZ48WYcOHdKCBQt06NAhVa1aVYcPH7a2jYuLU5UqVbRhwwb997//1b59+7RmzRrVqVNHvXv3NnEUOcfhw4f1xBNP6ODBg5o/f74OHTqkyZMna/369apZs2amH2zIqHv37oqPj1d0dLTatGmj3r17a/78+db1np6eio+Pt3kdPXrUZh8NGjRQfHy8Dh8+rPHjx2vKlCkaNmzYgx6KQ6WPMT4+XuvXr5eLi4uaNGly3/tbtmyZatWqpZIlS2rjxo369ddf1a9fP40aNUrt2rWz+UFrypQpqlevnnx9fbVkyRJFR0dr8uTJSkxM1AcffGCP4Znmfuu6bt06xcfH65tvvlFycrIaNmyo8+fPW9enH9e3vt577z2bfcycOVPx8fH68ccf9dRTT6l169bat2+fvYf4QNxvHf38/DR79mybZZ9//rn8/Pyy3W79+vVq27atWrVqpZ07d+qnn37S6NGjbUJWamqqwsPDNW7cOL300kvatm2bdu7cqd69e+uTTz7JEFIedikpKapXr55mzJihUaNG6cCBA/rqq6907do1Va9eXdu3b7dpX6FCBcXHx+vYsWOaOXOm1qxZo549e9q0WbFihWrUqKHk5GR9/vnniomJ0Zo1a9SyZUsNGTJEiYmJD3KID5y/v79mzZpls+zEiRNav369ihcvnuV2V69eVf369ZWYmKilS5cqNjZWkZGRCg0NtZknctvcej/zwPbt21W9enWlpqZq9erVOnDggEaPHq1Zs2apfv36Sk1NtWn/zjvvKD4+Xr/88os6deqk7t276+uvv7auNwxDHTp0UN++fdWoUSN9++23io6O1vTp05U3b16NGjXKIWO/awZyhRdffNFo3rx5lutr1apl9OvXL9N1PXr0MAoUKGDEx8fbLL906ZLh5+dnNGjQwLqsYcOGhp+fn5GcnJxhP+fOnbufruc6DRo0MEqWLGlcunTJZnl8fLyRP39+o0ePHib1LOe4n+O1bNmyRrt27QzDMIyZM2caBQsWvOf3ePbZZ43HHnvsPnqcM2U2xq1btxqSjNOnT2e6zZEjRwxJxu7duzOsS05ONry9vY1nn302w7ovv/zSkGQsWLDAMAzDOH78uOHq6mr0798/0/d5mOcDe9X1+++/NyQZa9asMQwj+3k4nSRj2bJl1q+TkpIMScbHH398P0Mx1d+p45AhQ4yyZcvarAsKCjLefvttQ5Jx5MgRm/bpde/Xr59Ru3btbPv17rvvGk5OTsbPP/+cYV1qamqmn2852Z3m03HjxhkWi8WIioqyWX79+nXjiSeeMMqXL2+kpaUZhmEYw4YNM8LCwmzavfrqq0bhwoWtX6fPEy1btszyPdP39zC6m8+nnj17Gt7e3sZ3331nXT569GijadOmRlhYmDFs2DCb9un/73fv3m1IMuLi4rLcf26bW+9nHkhLSzPKly9vPPHEE8b169dt1kVFRRkWi8UYN26cdVlAQIAxfvx4m3ZeXl7GgAEDrF/Pnz/fkGSsWLEiy/c0E2fI/uHS0tK0YMECdezYUb6+vjbr8uXLp169eumbb77R2bNndfbsWa1Zs0a9e/dWgQIFMuyrUKFCD6jXOdfZs2f1zTffqFevXsqXL5/NOl9fX3Xs2FGRkZEP9eUcZsmXL1+G34jdi19++UXbtm2Tq6urHXuVsyQnJ2vOnDkKDAyUt7f3PW//7bffKiEhQQMHDsywrmnTpgoKCrKepVy0aJFSU1P1xhtvZLqv3DQf3G9d0+eA+z1ur127punTp0tSrjhu76WOzZo107lz5/Tdd99Jkr777judO3dOTZs2zXY7X19f7d+/X7/88kuWbebOnat69erpsccey7AuT548mX6+PczmzZun+vXrKywszGa5k5OTBgwYoOjoaO3ZsyfTbePi4vTNN9/YHH/p80RW//elG5fe5maurq7q2LGjZs6caV02a9YsdenSJdvtfHx85OTkpMWLF2d6CbiU++fWu5kHoqKiFB0drVdffVVOTrZRJSwsTPXq1bO5YuZWaWlpWrJkic6dO2dz3M6fP1/lypVTs2bNMt3O7GOWQPYPd+bMGZ0/f14hISGZrg8JCZFhGDp06JAOHTokwzAUHBz8gHv58Dh48KAMw8i2nufOndOZM2cecM8eXtevX9ecOXO0d+9e1a1b17o8MTFR7u7uNq+GDRvabLtq1Sq5u7tb74M6ffq0Xn/99Qc9BIdKH6O7u7s8PDz05ZdfKjIyMsOH2N04cOCAJGV5/AYHB1vbHDx4UJ6entlenvMw+7t1PX/+vEaOHCl3d3dVq1bNunzSpEkZjtu5c+fabNu+fXu5u7vLzc1NAwYMUKlSpdSmTRu7ju9Bud865smTR506ddKMGTMkSTNmzFCnTp2UJ0+ebLfr06ePqlatqtDQUJUqVUrt2rXTjBkzlJKSYm1z8ODBf9Tn2IEDB7L9TEpvk27fvn1yd3dXvnz5VLp0ae3fv18RERE2+5OkcuXKWZft2rXL5phetWqVI4aSo3Tp0kULFy7UxYsXtWXLFiUmJt7xMjw/Pz9NmDBBQ4cOVeHChVW3bl2NHDnS5taQ3Di33us8cKfPopCQEJtjVpIiIiKs82br1q1VuHBhdevWzWaftx6zktS/f39rv0qWLPl3hvi3Echymblz59pMilu3br2r7e7mjA1nde4etbo72R2v6T+45suXT927d9eAAQNs7mPw8PBQVFSUzev2m3jr1KmjqKgo7dixQy+++KL+85//qFWrVg9sfA9C+hijoqK0c+dOhYeHq2HDhjp69KgaNmxorW2FChXuep93Ox+Y/RtFR7rfuj755JNyd3dX4cKFtWfPHkVGRqpYsWLW9R07dsxw3N7+G9vx48crKipKX3/9tcqXL69p06bJy8vrgYzb3v7O8dmlSxctWrRIp06d0qJFi+549kGSChQooNWrV+vQoUMaMmSI3N3d9dprr6latWrWBwHk1vk5u/n0XsZcrlw5RUVFadeuXYqIiFB4eLj69OmT7TaVKlWyfp8vXryoa9eu3fc4coo7/TwVFhamsmXLavHixZoxY4aef/55ubi43HG/vXv31qlTpzR37lzVrFlTixYtUoUKFbR27VpJuXNuvd954F6O29dff11RUVHasGGDqlevrvHjxyswMDDbbQYPHqyoqCgNHTpUycnJ9zU2e7nzkYOHSrNmzVS9enXr13e6AdrHx0eFChVSTExMputjYmJksVisB7XFYsnwJCDcFBgYKIvFopiYGLVs2TLD+piYGBUuXFg+Pj4m9C7nye547dixowYPHqx8+fKpePHiGX6T5uTkdMfJtkCBAtY2M2bMUFhYmKZPn66uXbvacRTmunWMkjRt2jQVLFhQU6dO1bRp03T58mVJuuOZBUkKCgqSdOM4ffLJJzOsj4mJUfny5a1tExMTFR8fn6t+k5vufusaGRmp8uXLy9vbO9NLiwoWLHjH49bX11eBgYEKDAzUzJkz1ahRI0VHR6to0aJ/f2AP2N85PkNDQxUcHKz27dsrJCREFStWzPbpoLd69NFH9eijj6pbt24aPHiwgoKCFBkZqf/85z8KCgrKlZ9jWc2nQUFB2X7Gp7dJ5+rqav2ejRs3To0bN9aIESM0cuRISVLZsmUlSbGxsapRo4Ykyc3N7Y7H9cPmbn6e6tKliyZOnKjo6Gjt3Lnzrvft4eGhpk2bqmnTpho1apTCw8M1atQo1a9fP1fOrfc6D9z6WZTZpcUxMTE2x6wkFSlSxDpvLlq0SKGhoXriiSesn1lly5ZVbGyszTY+Pj7y8fHJEXMrZ8hyGQ8PD+sBGRgYmOE+pts5OTmpTZs2mjdvnk6dOmWz7vLly5o0aZLCw8Pl5eUlLy8vhYeHa+LEibp48WKGfd36hKB/Km9vb9WvX1+TJk2yTjDp0n8j1rZt21z326/7ld3xmv6Dq5+f331dfnc7JycnvfXWWxoyZEiG701uYrFY5OTkpMuXL8vPz89a24CAgDtu+8wzz8jLyyvTp3h9+eWXOnjwoNq3by9Jat26tVxdXTM8ITBdbpsP7rau/v7+evTRR+12n0e1atVUpUoVjR492i77M9u9Hp9dunTRpk2b7ursWFZKlSql/PnzWz+3OnTooHXr1tn8SYJ0V69ezfTz7WGQ1Xzarl07rVu3LsN9YmlpaRo/frzKly+f4f6yWw0ZMkTvv/++Tp48KenmPPHuu+86bjA5wN38PNWhQwft27dPFStWtP7gf68sFouCg4Otx90/YW690zxQuXJlBQcHa/z48UpLS7PZds+ePVq3bp31sygz/v7+atu2rQYNGmRd1r59e8XGxubYPyNCIPsHOXPmTIZLZf744w+NGTNGvr6+ql+/vr7++msdP35cW7ZsUXh4uK5evaqJEyda9zFx4kRdv35d1apV05IlS3Tw4EHFxMRowoQJqlmzpomjyzk+/fRTpaSkKDw8XFu2bNHx48e1Zs0a1a9fX35+frnmByuzGYahU6dOZXjdPnnf6rnnnpOzs7PNMf2wS0lJsY49JiZGffr0UXJy8h0ffhAbG5thPnB1ddWUKVO0YsUKvfTSS9q7d6/i4uI0ffp0de7cWa1bt7bey+Tv76/x48fr448/VteuXbV582YdPXpU33//vV5++WXrb9MfVvdb1zu5dOlShmP23Llz2W7Tv39/TZky5aH7+0PS369j9+7ddebMGZt7QbIzfPhwvfHGG9q0aZOOHDmi3bt3q0uXLtbHjUs36vnUU0/p3//+tyZOnKg9e/bo8OHDWrhwoWrUqKGDBw/e93hzogEDBqhatWpq2rSpFi1apGPHjmnXrl1q1aqVYmJiNH369Gx/SVizZk1VqlRJY8aMkSS5u7tr2rRpWr16tRo3bqxvvvlGhw8f1t69e60hwtnZ+YGMzWyFCxe2Psr9bkRFRal58+ZavHixoqOjdejQIU2fPl0zZsxQ8+bNJeXOufVe5wGLxaLp06crOjra+icsjh07pkWLFqlp06aqWbNmtn9bV5L69eunlStX6scff5R04xcTrVu3Vrt27fTOO+9ox44diouL0+bNmxUZGWn+MWvGox1hf3fzmFZJGV4jR440DMMwzpw5Y/Tp08fw9/c38uTJYxQrVszo3LmzcfTo0Qz7OnnypNG7d28jICDAcHV1Nfz8/IxmzZoZGzdudNDoHj5xcXHGiy++aBQrVszIkyeP4e/vb/Tp08f4888/ze5ajvB3/kyDYdx47H1mx7Mk659vyOo9xo4da/j4+Dx0j7bOzIsvvmgzdg8PD6Nq1arG4sWLs9wm/THhmb2OHz9uGIZhbNmyxQgPDzc8PT0NV1dXo0KFCsb7779vXLt2LcP+1q5da4SHhxuFCxc28ubNawQHBxsDBw40Tp486bBxO9rfqWtmf04gXVbzcHh4uLWNbnvsvWHceBxzcHCw0bNnz787tAfKEXVMf2x4Vo+937Bhg9GqVSvD39/fcHV1NYoVK2Y0aNDA2Lp1q81+rly5YowdO9YIDQ018ubNa3h5eRlPPfWUMWvWLOPq1av2GP4Dc6f51DAM4+LFi8bgwYONwMBAI0+ePIaXl5fRqlUrY9++fTbtMnvsvWHceGS4m5ubcezYMeuyXbt2Ga1btzaKFi1quLi4GN7e3kZ4eLixYMEC0x8h/nf83c+n7B57f+bMGaNv375GxYoVDXd3d8PDw8MIDQ013n///QyPd88tc+v9zAPp9u7da7Rq1crw8vIy8uTJYzz66KPGkCFDjIsXL9q0y+yx94ZhGOHh4UbDhg2tX1+/ft2YPHmyUb16daNAgQKGq6urUaZMGaN79+5GdHT03x7r32ExjFx6dysAAAAA5HBcsggAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgD4R+vcubNatGhh/bp27drq37//A+/Hpk2bZLFYdP78+SzbWCwWLV++/K73OXz4cFWuXPlv9SsuLk4Wi0VRUVF/az8AgMwRyAAAOU7nzp1lsVhksVjk6uqqwMBAvfPOO7p27ZrD33vp0qUaOXLkXbW9mxAFAEB2XMzuAAAAmWnQoIFmzpyplJQUffXVV+rdu7fy5MmjQYMGZWibmpoqV1dXu7yvl5eXXfYDAMDd4AwZACBHcnNzk6+vrwICAtSzZ0/Vq1dPX375paSblxmOHj1aJUqUULly5SRJx48fV5s2bVSoUCF5eXmpefPmiouLs+7z+vXrevXVV1WoUCF5e3vrjTfekGEYNu97+yWLKSkpioiIkL+/v9zc3BQYGKjp06crLi5OderUkSQVLlxYFotFnTt3liSlpaVp7NixKl26tPLly6ewsDAtXrzY5n2++uorBQUFKV++fKpTp45NP+9WRESEgoKClD9/fpUpU0Zvv/22rl69mqHdlClT5O/vr/z586tNmzZKTEy0WT9t2jSFhIQob968Cg4O1qRJk+65LwCA+0MgAwA8FPLly6fU1FTr1+vXr1dsbKzWrl2rVatW6erVqwoPD5eHh4e2bt2q77//Xu7u7mrQoIF1uw8++ECzZs3SjBkz9N133+ns2bNatmxZtu/7wgsvaP78+ZowYYJiYmI0ZcoUubu7y9/fX0uWLJEkxcbGKj4+Xh9//LEkaezYsZo9e7YmT56s/fv3a8CAAerUqZM2b94s6UZwfPbZZ9W0aVNFRUWpW7duevPNN++5Jh4eHpo1a5aio6P18ccfa+rUqRo/frxNm0OHDmnhwoVauXKl1qxZo927d6tXr17W9XPnztXQoUM1evRoxcTEaMyYMXr77bf1+eef33N/AAD3wQAAIId58cUXjebNmxuGYRhpaWnG2rVrDTc3N2PgwIHW9cWKFTNSUlKs23zxxRdGuXLljLS0NOuylJQUI1++fMY333xjGIZhFC9e3Hjvvfes669evWqULFnS+l6GYRi1atUy+vXrZxiGYcTGxhqSjLVr12baz40bNxqSjHPnzlmXXblyxcifP7+xbds2m7Zdu3Y12rdvbxiGYQwaNMgoX768zfqIiIgM+7qdJGPZsmVZrv/vf/9rVKlSxfr1sGHDDGdnZ+P333+3Lvv6668NJycnIz4+3jAMw3j00UeNefPm2exn5MiRRs2aNQ3DMIwjR44Ykozdu3dn+b4AgPvHPWQAgBxp1apVcnd319WrV5WWlqYOHTpo+PDh1vWhoaE2943t2bNHhw4dkoeHh81+rly5ot9++02JiYmKj49X9erVretcXFz0xBNPZLhsMV1UVJScnZ1Vq1atu+73oUOHdOnSJdWvX99meWpqqh577DFJUkxMjE0/JKlmzZp3/R7pIiMjNWHCBP32229KTk7WtWvX5OnpadPmkUcekZ+fn837pKWlKTY2Vh4eHvrtt9/UtWtXde/e3drm2rVrKliw4D33BwBw7whkAIAcqU6dOvrss8/k6uqqEiVKyMXF9iOrQIECNl8nJyerSpUqmjt3boZ9+fj43Fcf8uXLd8/bJCcnS5JWr15tE4SkG/fF2csPP/ygjh07asSIEQoPD1fBggW1YMECffDBB/fc16lTp2YIiM7OznbrKwAgawQyAECOVKBAAQUGBt51+8cff1yRkZEqWrRohrNE6YoXL64dO3bo6aeflnTjTNBPP/2kxx9/PNP2oaGhSktL0+bNm1WvXr0M69PP0F2/ft26rHz58nJzc9OxY8eyPLMWEhJifUBJuu3bt995kLfYtm2bAgICNHjwYOuyo0ePZmh37NgxnTx5UiVKlLC+j5OTk8qVK6dixYqpRIkSOnz4sDp27HhP7w8AsA8e6gEAyBU6duyoIkWKqHnz5tq6dauOHDmiTZs2qW/fvvr9998lSf369dO4ceO0fPly/frrr+rVq1e2f0OsVKlSevHFF9WlSxctX77cus+FCxdKkgICAmSxWLRq1SqdOXNGycnJ8vDw0MCBAzVgwAB9/vnn+u233/Tzzz/rk08+sT4oo0ePHjp48KBef/11xcbGat68eZo1a9Y9jbds2bI6duyYFixYoN9++00TJkzI9AElefPm1Ysvvqg9e/Zo69at6tu3r9q0aSNfX19J0ogRIzR27FhNmDBBBw4c0L59+zRz5kx9+OGH99QfAMD9IZABAHKF/Pnza8uWLXrkkUf07LPPKiQkRF27dtWVK1esZ8xee+01Pf/883rxxRdVs2ZNeXh4qGXLltnu97PPPlPr1q3Vq1cvBQcHq3v37rp48aIkyc/PTyNGjNCbb76pYsWK6ZVXXpEkjRw5Um+//bbGjh2rkJAQNWjQQKtXr1bp0qUl3biva8mSJVq+fLnCwsI0efJkjRkz5p7G26xZMw0YMECvvPKKKleurG3btuntt9/O0C4wMFDPPvusGjVqpGeeeUaVKlWyeax9t27dNG3aNM2cOVOhoaGqVauWZs2aZe0rAMCxLEZWdzIDAAAAAByKM2QAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJvl/AFt0IDYU/LIAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 1000x1000 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "eval_token_list = list(set(eval_df_tokens['labels']))\n",
    "\n",
    "plot_confusion_matrix(eval_df_tokens[\"labels\"], eval_df_tokens[\"predicted_label\"],\n",
    "                      eval_token_list)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define & Call Function to Display Example Token Sequences Along With Labels & Losses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "      <th>14</th>\n",
       "      <th>15</th>\n",
       "      <th>16</th>\n",
       "      <th>17</th>\n",
       "      <th>18</th>\n",
       "      <th>19</th>\n",
       "      <th>20</th>\n",
       "      <th>21</th>\n",
       "      <th>22</th>\n",
       "      <th>23</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>tokens</th>\n",
       "      <td>▁with</td>\n",
       "      <td>it</td>\n",
       "      <td>standing</td>\n",
       "      <td>▁t</td>\n",
       "      <td>▁Th</td>\n",
       "      <td>▁s</td>\n",
       "      <td>[SEP]</td>\n",
       "      <td>▁thr</td>\n",
       "      <td>▁u</td>\n",
       "      <td>mat</td>\n",
       "      <td>▁oppon</td>\n",
       "      <td>▁learn</td>\n",
       "      <td>1,</td>\n",
       "      <td>in</td>\n",
       "      <td>▁attack</td>\n",
       "      <td>▁—</td>\n",
       "      <td>▁a</td>\n",
       "      <td>▁others</td>\n",
       "      <td>▁lay</td>\n",
       "      <td>ollow</td>\n",
       "      <td>or</td>\n",
       "      <td>itchen</td>\n",
       "      <td>▁4</td>\n",
       "      <td>&lt;/s&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>labels</th>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>preds</th>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>I-LOC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>I-LOC</td>\n",
       "      <td>O</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>losses</th>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.39</td>\n",
       "      <td>3.07</td>\n",
       "      <td>4.28</td>\n",
       "      <td>3.30</td>\n",
       "      <td>3.66</td>\n",
       "      <td>3.95</td>\n",
       "      <td>7.51</td>\n",
       "      <td>9.71</td>\n",
       "      <td>9.13</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           0     1         2     3     4     5      6     7       8       9   \\\n",
       "tokens  ▁with    it  standing    ▁t   ▁Th    ▁s  [SEP]  ▁thr      ▁u     mat   \n",
       "labels      O     O         O     O     O     O      O     O  I-MISC  I-MISC   \n",
       "preds       O     O         O     O     O     O      O     O   B-LOC   I-LOC   \n",
       "losses   0.00  0.00      0.00  0.00  0.00  0.00   0.00  0.39    3.07    4.28   \n",
       "\n",
       "            10      11      12      13       14      15    16       17    18  \\\n",
       "tokens  ▁oppon  ▁learn      1,      in  ▁attack      ▁—    ▁a  ▁others  ▁lay   \n",
       "labels  I-MISC  I-MISC  I-MISC  I-MISC   I-MISC  I-MISC     O    B-LOC     O   \n",
       "preds        O       O       O       O    B-LOC   I-LOC     O    B-LOC     O   \n",
       "losses    3.30    3.66    3.95    7.51     9.71    9.13  0.00     0.00  0.00   \n",
       "\n",
       "           19    20      21    22    23  \n",
       "tokens  ollow    or  itchen    ▁4  </s>  \n",
       "labels      O     O       O     O   IGN  \n",
       "preds       O     O       O     O     O  \n",
       "losses   0.00  0.00    0.00  0.00  0.00  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "      <th>14</th>\n",
       "      <th>15</th>\n",
       "      <th>16</th>\n",
       "      <th>17</th>\n",
       "      <th>18</th>\n",
       "      <th>19</th>\n",
       "      <th>20</th>\n",
       "      <th>21</th>\n",
       "      <th>22</th>\n",
       "      <th>23</th>\n",
       "      <th>24</th>\n",
       "      <th>25</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>tokens</th>\n",
       "      <td>絲</td>\n",
       "      <td>re</td>\n",
       "      <td>ARD</td>\n",
       "      <td>▁lay</td>\n",
       "      <td>es</td>\n",
       "      <td>▁w</td>\n",
       "      <td>[SEP]</td>\n",
       "      <td>▁will</td>\n",
       "      <td>in</td>\n",
       "      <td>[SEP]</td>\n",
       "      <td>▁administration</td>\n",
       "      <td>ics</td>\n",
       "      <td>▁oppon</td>\n",
       "      <td>▁machine</td>\n",
       "      <td>ust</td>\n",
       "      <td>▁a</td>\n",
       "      <td>al</td>\n",
       "      <td>▁be</td>\n",
       "      <td>urn</td>\n",
       "      <td>▁administration</td>\n",
       "      <td>ics</td>\n",
       "      <td>▁next</td>\n",
       "      <td>ust</td>\n",
       "      <td>lud</td>\n",
       "      <td>▁4</td>\n",
       "      <td>&lt;/s&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>labels</th>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-ORG</td>\n",
       "      <td>I-ORG</td>\n",
       "      <td>I-ORG</td>\n",
       "      <td>I-ORG</td>\n",
       "      <td>I-ORG</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-ORG</td>\n",
       "      <td>I-ORG</td>\n",
       "      <td>I-ORG</td>\n",
       "      <td>I-ORG</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>preds</th>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>losses</th>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>5.62</td>\n",
       "      <td>4.43</td>\n",
       "      <td>4.60</td>\n",
       "      <td>3.47</td>\n",
       "      <td>3.82</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>5.50</td>\n",
       "      <td>4.59</td>\n",
       "      <td>5.24</td>\n",
       "      <td>4.90</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          0     1      2     3     4     5      6      7     8      9   \\\n",
       "tokens     絲    re    ARD  ▁lay    es    ▁w  [SEP]  ▁will    in  [SEP]   \n",
       "labels     O     O  B-LOC     O     O     O      O      O     O      O   \n",
       "preds      O     O  B-LOC     O     O     O      O      O     O      O   \n",
       "losses  0.00  0.00   0.00  0.00  0.00  0.00   0.00   0.00  0.00   0.00   \n",
       "\n",
       "                     10     11      12        13     14    15    16    17  \\\n",
       "tokens  ▁administration    ics  ▁oppon  ▁machine    ust    ▁a    al   ▁be   \n",
       "labels            B-ORG  I-ORG   I-ORG     I-ORG  I-ORG     O     O     O   \n",
       "preds             B-LOC      O       O         O      O     O     O     O   \n",
       "losses             5.62   4.43    4.60      3.47   3.82  0.00  0.00  0.00   \n",
       "\n",
       "          18               19     20     21     22    23    24    25  \n",
       "tokens   urn  ▁administration    ics  ▁next    ust   lud    ▁4  </s>  \n",
       "labels     O            B-ORG  I-ORG  I-ORG  I-ORG     O     O   IGN  \n",
       "preds      O            B-LOC      O      O      O     O     O     O  \n",
       "losses  0.00             5.50   4.59   5.24   4.90  0.00  0.00  0.00  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "      <th>14</th>\n",
       "      <th>15</th>\n",
       "      <th>16</th>\n",
       "      <th>17</th>\n",
       "      <th>18</th>\n",
       "      <th>19</th>\n",
       "      <th>20</th>\n",
       "      <th>21</th>\n",
       "      <th>22</th>\n",
       "      <th>23</th>\n",
       "      <th>24</th>\n",
       "      <th>25</th>\n",
       "      <th>26</th>\n",
       "      <th>27</th>\n",
       "      <th>28</th>\n",
       "      <th>29</th>\n",
       "      <th>30</th>\n",
       "      <th>31</th>\n",
       "      <th>32</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>tokens</th>\n",
       "      <td>▁spe</td>\n",
       "      <td>▁kn</td>\n",
       "      <td>ress</td>\n",
       "      <td>▁lay</td>\n",
       "      <td>▁ab</td>\n",
       "      <td>▁lay</td>\n",
       "      <td>es</td>\n",
       "      <td>it</td>\n",
       "      <td>▁ac</td>\n",
       "      <td>er</td>\n",
       "      <td>▁c</td>\n",
       "      <td>影</td>\n",
       "      <td>hood</td>\n",
       "      <td>ou</td>\n",
       "      <td>osure</td>\n",
       "      <td>▁January</td>\n",
       "      <td>▁lay</td>\n",
       "      <td>▁u</td>\n",
       "      <td>▁address</td>\n",
       "      <td>ou</td>\n",
       "      <td>▁t</td>\n",
       "      <td>▁av</td>\n",
       "      <td>▁a</td>\n",
       "      <td>es</td>\n",
       "      <td>use</td>\n",
       "      <td>赋</td>\n",
       "      <td>▁s</td>\n",
       "      <td>▁gu</td>\n",
       "      <td>▁ro</td>\n",
       "      <td>▁lay</td>\n",
       "      <td>▁pri</td>\n",
       "      <td>▁4</td>\n",
       "      <td>&lt;/s&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>labels</th>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-ORG</td>\n",
       "      <td>I-ORG</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>I-LOC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>preds</th>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-ORG</td>\n",
       "      <td>I-ORG</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>losses</th>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>8.10</td>\n",
       "      <td>0.00</td>\n",
       "      <td>8.77</td>\n",
       "      <td>10.88</td>\n",
       "      <td>10.25</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          0     1     2     3     4     5     6     7     8     9      10  \\\n",
       "tokens  ▁spe   ▁kn  ress  ▁lay   ▁ab  ▁lay    es    it   ▁ac    er     ▁c   \n",
       "labels     O     O     O     O     O     O     O     O     O     O  B-ORG   \n",
       "preds      O     O     O     O     O     O     O     O     O     O  B-ORG   \n",
       "losses  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00   0.00   \n",
       "\n",
       "           11    12    13      14        15      16      17        18    19  \\\n",
       "tokens      影  hood    ou   osure  ▁January    ▁lay      ▁u  ▁address    ou   \n",
       "labels  I-ORG     O     O   B-LOC       IGN       O   B-LOC     I-LOC     O   \n",
       "preds   I-ORG     O     O  B-MISC    I-MISC  I-MISC  I-MISC    I-MISC     O   \n",
       "losses   0.00  0.00  0.00    8.10      0.00    8.77   10.88     10.25  0.00   \n",
       "\n",
       "          20    21    22    23    24    25    26    27    28    29    30  \\\n",
       "tokens    ▁t   ▁av    ▁a    es   use     赋    ▁s   ▁gu   ▁ro  ▁lay  ▁pri   \n",
       "labels     O     O     O     O     O     O     O     O     O     O     O   \n",
       "preds      O     O     O     O     O     O     O     O     O     O     O   \n",
       "losses  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00   \n",
       "\n",
       "          31    32  \n",
       "tokens    ▁4  </s>  \n",
       "labels     O   IGN  \n",
       "preds      O     O  \n",
       "losses  0.00  0.00  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def get_samples(df):\n",
    "    for _, row in df.iterrows():\n",
    "        labels, preds, tokens, losses = [], [], [], []\n",
    "        for i, mask in enumerate(row[\"attention_mask\"]):\n",
    "            if i not in {0, len(row[\"attention_mask\"])}:\n",
    "                labels.append(row[\"labels\"][i])\n",
    "                preds.append(row[\"predicted_label\"][i])\n",
    "                tokens.append(row[\"input_tokens\"][i])\n",
    "                losses.append(f\"{row['loss'][i]:.2f}\")\n",
    "        eval_df_tmp = pd.DataFrame({\"tokens\": tokens, \"labels\": labels, \n",
    "                               \"preds\": preds, \"losses\": losses}).T\n",
    "        yield eval_df_tmp\n",
    "\n",
    "eval_df[\"total_loss\"] = eval_df[\"loss\"].apply(sum)\n",
    "eval_df_tmp = eval_df.sort_values(by=\"total_loss\", ascending=False).head(3)\n",
    "\n",
    "pd.set_option('display.max_columns', None)\n",
    "\n",
    "for sample in get_samples(eval_df_tmp):\n",
    "    display(sample)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Notes & Other Takeaways From This Project\n",
    "\n",
    "****\n",
    "- This model is part of a comparison of Token Classification Models using this particular dataset.\n",
    "\n",
    "****"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Citations\n",
    "\n",
    "- Model Checkpoint\n",
    "\n",
    "    > @article{DBLP:journals/corr/abs-2105-03824, author = {James Lee{-}Thorp and Joshua Ainslie and Ilya Eckstein and Santiago Onta{\\~{n}}{\\'{o}}n}, title = {FNet: Mixing Tokens with Fourier Transforms}, journal = {CoRR}, volume = {abs/2105.03824}, year = {2021}, url = {https://arxiv.org/abs/2105.03824}, archivePrefix = {arXiv}, eprint = {2105.03824}, timestamp = {Fri, 14 May 2021 12:13:30 +0200}, biburl = {https://dblp.org/rec/journals/corr/abs-2105-03824.bib}, bibsource = {dblp computer science bibliography, https://dblp.org}}\n",
    "\n",
    "- Dataset\n",
    "\n",
    "    > @inproceedings{tedeschi-etal-2021-wikineural-combined, title = \"{W}iki{NE}u{R}al: {C}ombined Neural and Knowledge-based Silver Data Creation for Multilingual {NER}\", author = \"Tedeschi, Simone and Maiorca, Valentino and Campolungo, Niccol{\\`o} and Cecconi, Francesco and Navigli, Roberto\", booktitle = \"Findings of the Association for Computational Linguistics: EMNLP 2021\", month = nov, year = \"2021\", address = \"Punta Cana, Dominican Republic\", publisher = \"Association for Computational Linguistics\", url = \"https://aclanthology.org/2021.findings-emnlp.215\", pages = \"2521--2533\", abstract = \"Multilingual Named Entity Recognition (NER) is a key intermediate task which is needed in many areas of NLP. In this paper, we address the well-known issue of data scarcity in NER, especially relevant when moving to a multilingual scenario, and go beyond current approaches to the creation of multilingual silver data for the task. We exploit the texts of Wikipedia and introduce a new methodology based on the effective combination of knowledge-based approaches and neural models, together with a novel domain adaptation technique, to produce high-quality training corpora for NER. We evaluate our datasets extensively on standard benchmarks for NER, yielding substantial improvements up to 6 span-based F1-score points over previous state-of-the-art systems for data creation.\",}\n",
    "\n",
    "- Metric (PosEval)\n",
    "\n",
    "    > @article{scikit-learn, title={Scikit-learn: Machine Learning in {P}ython}, author={Pedregosa, F. and Varoquaux, G. and Gramfort, A. and Michel, V. and Thirion, B. and Grisel, O. and Blondel, M. and Prettenhofer, P. and Weiss, R. and Dubourg, V. and Vanderplas, J. and Passos, A. and Cournapeau, D. and Brucher, M. and Perrot, M. and Duchesnay, E.}, journal={Journal of Machine Learning Research}, volume={12}, pages={2825--2830}, year={2011}}"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "dl",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}