Token Classification/Monolingual/WikiNeural - Transformer Comparison/POS Project with Wikineural Dataset - I-BERT Transformer.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Monolingual Named Entity Recognition Token Classification (WikiNeural Dataset)\n",
    "\n",
    "### IBert-Roberta-Base Transformer\n",
    "\n",
    "Dataset Source: https://huggingface.co/datasets/Babelscape/wikineural"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Import Necessary Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Git LFS initialized.\n"
     ]
    }
   ],
   "source": [
    "import os, sys\n",
    "os.environ['TOKENIZERS_PARALLELISM']='false'\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import sklearn\n",
    "from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay\n",
    "\n",
    "import datasets\n",
    "from datasets import load_dataset, DatasetDict\n",
    "\n",
    "import torch\n",
    "from torch import nn\n",
    "from torch.nn.functional import cross_entropy\n",
    "\n",
    "import transformers\n",
    "from transformers import AutoTokenizer, AutoModelForTokenClassification\n",
    "from transformers import DataCollatorForTokenClassification\n",
    "from transformers import Trainer, TrainingArguments\n",
    "\n",
    "import evaluate\n",
    "\n",
    "!git lfs install"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Display Library Versions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "          Python : 3.9.12\n",
      "           NumPy : 1.25.0\n",
      "           Torch : 2.0.1\n",
      "          Pandas : 2.0.2\n",
      "    Transformers : 4.28.1\n",
      "        datasets : 2.13.0\n",
      "         Sklearn : 1.2.2\n"
     ]
    }
   ],
   "source": [
    "print(\"Python :\".rjust(18), sys.version[0:6])\n",
    "print(\"NumPy :\".rjust(18), np.__version__)\n",
    "print(\"Torch :\".rjust(18), torch.__version__)\n",
    "print(\"Pandas :\".rjust(18), pd.__version__)\n",
    "print(\"Transformers :\".rjust(18), transformers.__version__)\n",
    "print(\"datasets :\".rjust(18), datasets.__version__)\n",
    "print(\"Sklearn :\".rjust(18), sklearn.__version__)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Ingest English Subset of Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Found cached dataset parquet (/Users/briandunn/.cache/huggingface/datasets/Babelscape___parquet/Babelscape--wikineural-579d1dc98d2a6b93/0.0.0/14a00e99c0d15a23649d0db8944380ac81082d4b021f398733dd84f3a6c569a7)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a69f2136ec7540ffa5336733221896e6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/27 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    test_de: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 12372\n",
       "    })\n",
       "    test_en: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 11597\n",
       "    })\n",
       "    test_es: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 9618\n",
       "    })\n",
       "    test_fr: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 12678\n",
       "    })\n",
       "    test_it: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 11069\n",
       "    })\n",
       "    test_nl: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 10547\n",
       "    })\n",
       "    test_pl: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 13585\n",
       "    })\n",
       "    test_pt: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 10160\n",
       "    })\n",
       "    test_ru: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 11580\n",
       "    })\n",
       "    train_de: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 98640\n",
       "    })\n",
       "    train_en: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 92720\n",
       "    })\n",
       "    train_es: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 76320\n",
       "    })\n",
       "    train_fr: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 100800\n",
       "    })\n",
       "    train_it: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 88400\n",
       "    })\n",
       "    train_nl: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 83680\n",
       "    })\n",
       "    train_pl: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 108160\n",
       "    })\n",
       "    train_pt: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 80560\n",
       "    })\n",
       "    train_ru: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 92320\n",
       "    })\n",
       "    val_de: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 12330\n",
       "    })\n",
       "    val_en: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 11590\n",
       "    })\n",
       "    val_es: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 9540\n",
       "    })\n",
       "    val_fr: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 12600\n",
       "    })\n",
       "    val_it: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 11050\n",
       "    })\n",
       "    val_nl: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 10460\n",
       "    })\n",
       "    val_pl: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 13520\n",
       "    })\n",
       "    val_pt: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 10070\n",
       "    })\n",
       "    val_ru: Dataset({\n",
       "        features: ['tokens', 'ner_tags'],\n",
       "        num_rows: 11540\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = load_dataset(\"Babelscape/wikineural\")\n",
    "\n",
    "data = data.remove_columns([\"lang\"])\n",
    "\n",
    "data"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Split Dataset into DatasetDict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training data shape: (92720, 2)\n",
      "Testing data shape: (11597, 2)\n",
      "Validation data shape: (11590, 2)\n"
     ]
    }
   ],
   "source": [
    "ds = DatasetDict({\n",
    "    'train': data['train_en'], \n",
    "    'test': data['test_en'], \n",
    "    'eval': data['val_en']})\n",
    "\n",
    "print('Training data shape:', ds['train'].shape)\n",
    "print('Testing data shape:', ds['test'].shape)\n",
    "print('Validation data shape:', ds['eval'].shape)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'tokens': ['It',\n",
       "  'has',\n",
       "  'a',\n",
       "  'leaf',\n",
       "  'pattern',\n",
       "  'similar',\n",
       "  'to',\n",
       "  'the',\n",
       "  'members',\n",
       "  'of',\n",
       "  'the',\n",
       "  'genera',\n",
       "  '\"',\n",
       "  'Kedrostis',\n",
       "  '\"',\n",
       "  ',',\n",
       "  '\"',\n",
       "  'Melothria',\n",
       "  '\"',\n",
       "  'and',\n",
       "  '\"',\n",
       "  'Zehneria',\n",
       "  '\"',\n",
       "  '.'],\n",
       " 'ner_tags': [0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  7,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  7,\n",
       "  0,\n",
       "  0,\n",
       "  0,\n",
       "  7,\n",
       "  0,\n",
       "  0]}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example = ds['train'][12]\n",
    "\n",
    "example"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Display Feature Information About Each Feature"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tokens: \n",
      "Sequence(feature=Value(dtype='string', id=None), length=-1, id=None)\n",
      "\n",
      "ner_tags: \n",
      "Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for k, v in ds[\"train\"].features.items():\n",
    "    print(f\"{k}: \\n{v}\\n\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define Tag Values & Conversions Between String & Integer Values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "List of tag values: \n",
      "['O', 'B-PER', 'I-PER', 'B-ORG', 'I-ORG', 'B-LOC', 'I-LOC', 'B-MISC', 'I-MISC']\n",
      "Number of NER Tags: \n",
      "9\n",
      "id2label: \n",
      "{0: 'O', 1: 'B-PER', 2: 'I-PER', 3: 'B-ORG', 4: 'I-ORG', 5: 'B-LOC', 6: 'I-LOC', 7: 'B-MISC', 8: 'I-MISC'}\n",
      "label2id: \n",
      "{'O': 0, 'B-PER': 1, 'I-PER': 2, 'B-ORG': 3, 'I-ORG': 4, 'B-LOC': 5, 'I-LOC': 6, 'B-MISC': 7, 'I-MISC': 8}\n"
     ]
    }
   ],
   "source": [
    "label2id = {\n",
    "    'O': 0, \n",
    "    'B-PER': 1, \n",
    "    'I-PER': 2, \n",
    "    'B-ORG': 3, \n",
    "    'I-ORG': 4, \n",
    "    'B-LOC': 5, \n",
    "    'I-LOC': 6, \n",
    "    'B-MISC': 7, \n",
    "    'I-MISC': 8\n",
    "    }\n",
    "\n",
    "id2label = {tag: idx for idx, tag in label2id.items()}\n",
    "\n",
    "pos_tag_values = list(label2id.keys())\n",
    "NUM_OF_LABELS = len(pos_tag_values)\n",
    "\n",
    "print(f\"List of tag values: \\n{pos_tag_values}\")\n",
    "print(f\"Number of NER Tags: \\n{NUM_OF_LABELS}\")\n",
    "print(f\"id2label: \\n{id2label}\")\n",
    "print(f\"label2id: \\n{label2id}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Basic Values/Constants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "MODEL_CKPT = \"kssteven/ibert-roberta-base\"\n",
    "MODEL_NAME = f\"{MODEL_CKPT.split(f'/')[-1]}-finetuned-WikiNeural\"\n",
    "\n",
    "NUM_OF_EPOCHS = 2\n",
    "BATCH_SIZE = 16\n",
    "\n",
    "STRATEGY = \"epoch\"\n",
    "REPORTS_TO = \"tensorboard\"\n",
    "\n",
    "WEIGHT_DECAY = 0.01\n",
    "LR = 2e-5\n",
    "\n",
    "DEVICE = torch.device(\"cpu\")\n",
    "STEPS = 1250"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Function to Tokenize & Align Inputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = AutoTokenizer.from_pretrained(MODEL_CKPT,\n",
    "                                          add_prefix_space=True)\n",
    "\n",
    "def tokenize_and_align_labels(samples):\n",
    "    tokenized_inputs = tokenizer(samples[\"tokens\"], \n",
    "                                      truncation=True, \n",
    "                                      is_split_into_words=True)\n",
    "    \n",
    "    labels = []\n",
    "    \n",
    "    for idx, label in enumerate(samples[f\"ner_tags\"]):\n",
    "        word_ids = tokenized_inputs.word_ids(batch_index=idx)\n",
    "        prev_word_idx = None\n",
    "        label_ids = []\n",
    "        for word_idx in word_ids: # set special tokens to -100\n",
    "            if word_idx is None or word_idx == prev_word_idx:\n",
    "                label_ids.append(-100)\n",
    "            else:\n",
    "                label_ids.append(label[word_idx])\n",
    "            prev_word_idx = word_idx\n",
    "        labels.append(label_ids)\n",
    "    \n",
    "    tokenized_inputs[\"labels\"] = labels\n",
    "    return tokenized_inputs"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Apply Above Function to Tokenize Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading cached processed dataset at /Users/briandunn/.cache/huggingface/datasets/Babelscape___parquet/Babelscape--wikineural-579d1dc98d2a6b93/0.0.0/14a00e99c0d15a23649d0db8944380ac81082d4b021f398733dd84f3a6c569a7/cache-cfed22b25add3bd9.arrow\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "867da0cb44f0404a827c8d65159a1462",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/11597 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading cached processed dataset at /Users/briandunn/.cache/huggingface/datasets/Babelscape___parquet/Babelscape--wikineural-579d1dc98d2a6b93/0.0.0/14a00e99c0d15a23649d0db8944380ac81082d4b021f398733dd84f3a6c569a7/cache-ed94097c75bbcf47.arrow\n"
     ]
    }
   ],
   "source": [
    "encoded_ds = ds.map(tokenize_and_align_labels, \n",
    "                    batched=True, \n",
    "                    remove_columns=\n",
    "                        [\n",
    "                            'ner_tags', \n",
    "                            'tokens'\n",
    "                        ]\n",
    "                    )"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define Data Collator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_collator = DataCollatorForTokenClassification(tokenizer=tokenizer)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load Custom Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at kssteven/ibert-roberta-base were not used when initializing IBertForTokenClassification: ['lm_head.dense.bias', 'lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.decoder.weight', 'lm_head.dense.weight', 'lm_head.layer_norm.weight']\n",
      "- This IS expected if you are initializing IBertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing IBertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of IBertForTokenClassification were not initialized from the model checkpoint at kssteven/ibert-roberta-base and are newly initialized: ['ibert.encoder.layer.1.intermediate.output_activation.x_min', 'ibert.embeddings.embeddings_act1.x_max', 'ibert.encoder.layer.8.attention.self.query.fc_scaling_factor', 'ibert.encoder.layer.5.intermediate.output_activation.x_min', 'ibert.encoder.layer.8.output.dense.weight_integer', 'ibert.encoder.layer.6.attention.output.output_activation.x_max', 'ibert.encoder.layer.0.attention.self.value_activation.x_max', 'ibert.encoder.layer.3.attention.output.output_activation.x_max', 'ibert.encoder.layer.9.pre_output_act.act_scaling_factor', 'ibert.encoder.layer.1.attention.self.key.fc_scaling_factor', 'ibert.encoder.layer.5.attention.output.dense.bias_integer', 'ibert.encoder.layer.5.attention.self.key.fc_scaling_factor', 'ibert.encoder.layer.8.attention.self.query.bias_integer', 'ibert.encoder.layer.0.attention.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.9.pre_intermediate_act.x_min', 'ibert.encoder.layer.9.output.dense.weight_integer', 'ibert.encoder.layer.3.attention.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.9.attention.self.key.bias_integer', 'ibert.encoder.layer.1.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.1.attention.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.9.attention.self.value.bias_integer', 'ibert.encoder.layer.7.output.dense.fc_scaling_factor', 'ibert.encoder.layer.9.attention.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.7.attention.self.softmax.act.act_scaling_factor', 'ibert.encoder.layer.0.intermediate.output_activation.x_max', 'ibert.encoder.layer.8.pre_intermediate_act.x_min', 'ibert.encoder.layer.6.attention.self.query_activation.x_min', 'ibert.encoder.layer.0.output.output_activation.x_max', 'ibert.encoder.layer.2.output.LayerNorm.shift', 'ibert.encoder.layer.3.attention.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.6.pre_output_act.x_max', 'ibert.encoder.layer.8.attention.self.value_activation.act_scaling_factor', 'ibert.encoder.layer.11.pre_intermediate_act.x_max', 'ibert.encoder.layer.1.attention.self.query_activation.act_scaling_factor', 'ibert.encoder.layer.1.pre_intermediate_act.x_max', 'ibert.encoder.layer.5.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.10.attention.self.softmax.act.x_min', 'ibert.encoder.layer.3.output.ln_input_act.x_max', 'ibert.encoder.layer.4.intermediate.output_activation.x_max', 'ibert.encoder.layer.0.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.8.attention.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.9.intermediate.output_activation.x_min', 'ibert.encoder.layer.7.attention.self.key_activation.x_min', 'ibert.encoder.layer.5.pre_intermediate_act.x_min', 'ibert.encoder.layer.2.attention.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.2.attention.self.softmax.act.act_scaling_factor', 'ibert.encoder.layer.4.attention.output.LayerNorm.shift', 'ibert.encoder.layer.5.attention.self.query_activation.x_min', 'ibert.encoder.layer.11.attention.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.5.intermediate.output_activation.x_max', 'ibert.encoder.layer.11.attention.self.value.bias_integer', 'ibert.encoder.layer.11.intermediate.output_activation.act_scaling_factor', 'ibert.encoder.layer.1.attention.self.value_activation.x_max', 'ibert.encoder.layer.10.attention.self.softmax.act.x_max', 'ibert.encoder.layer.6.attention.output.dense.bias_integer', 'ibert.encoder.layer.4.attention.output.ln_input_act.x_max', 'ibert.encoder.layer.3.attention.self.output_activation.x_min', 'ibert.encoder.layer.3.output.dense.weight_integer', 'ibert.encoder.layer.7.attention.self.key_activation.act_scaling_factor', 'ibert.encoder.layer.10.intermediate.output_activation.x_min', 'ibert.encoder.layer.8.attention.output.LayerNorm.activation.act_scaling_factor', 'ibert.embeddings.embeddings_act2.act_scaling_factor', 'ibert.encoder.layer.0.attention.output.LayerNorm.shift', 'ibert.encoder.layer.1.attention.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.1.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.4.intermediate.output_activation.act_scaling_factor', 'ibert.encoder.layer.6.attention.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.7.intermediate.output_activation.x_max', 'ibert.encoder.layer.10.pre_output_act.x_min', 'ibert.encoder.layer.2.attention.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.5.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.0.attention.output.output_activation.x_min', 'ibert.encoder.layer.0.attention.self.output_activation.x_min', 'ibert.encoder.layer.0.attention.self.softmax.act.x_max', 'ibert.encoder.layer.8.attention.self.softmax.act.act_scaling_factor', 'ibert.encoder.layer.2.attention.output.dense.bias_integer', 'ibert.encoder.layer.10.output.output_activation.x_max', 'ibert.encoder.layer.1.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.4.attention.output.output_activation.x_max', 'ibert.encoder.layer.10.attention.self.query.bias_integer', 'ibert.encoder.layer.1.output.ln_input_act.x_max', 'ibert.encoder.layer.11.attention.self.softmax.act.x_min', 'ibert.encoder.layer.8.output.output_activation.x_max', 'ibert.encoder.layer.5.attention.self.softmax.act.act_scaling_factor', 'ibert.encoder.layer.1.attention.output.dense.fc_scaling_factor', 'ibert.encoder.layer.9.attention.output.LayerNorm.shift', 'ibert.encoder.layer.4.attention.self.softmax.act.act_scaling_factor', 'ibert.encoder.layer.3.attention.self.value.weight_integer', 'ibert.encoder.layer.3.intermediate.dense.weight_integer', 'ibert.encoder.layer.5.attention.self.output_activation.x_max', 'ibert.encoder.layer.9.attention.output.ln_input_act.x_min', 'ibert.encoder.layer.4.attention.self.value_activation.x_min', 'ibert.encoder.layer.6.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.8.attention.self.softmax.act.x_max', 'ibert.encoder.layer.8.output.ln_input_act.x_max', 'ibert.encoder.layer.6.pre_intermediate_act.x_max', 'ibert.encoder.layer.8.attention.self.query_activation.x_min', 'ibert.encoder.layer.9.attention.output.ln_input_act.x_max', 'ibert.encoder.layer.4.output.LayerNorm.shift', 'ibert.encoder.layer.4.attention.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.8.pre_intermediate_act.act_scaling_factor', 'ibert.encoder.layer.0.pre_intermediate_act.x_max', 'ibert.encoder.layer.5.attention.self.query_activation.act_scaling_factor', 'ibert.encoder.layer.2.attention.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.8.attention.output.output_activation.x_max', 'ibert.embeddings.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.4.attention.self.output_activation.act_scaling_factor', 'ibert.encoder.layer.9.pre_output_act.x_min', 'ibert.encoder.layer.10.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.1.attention.self.output_activation.x_max', 'ibert.encoder.layer.2.intermediate.output_activation.x_min', 'ibert.encoder.layer.3.pre_intermediate_act.x_min', 'ibert.encoder.layer.6.attention.output.dense.weight_integer', 'ibert.encoder.layer.5.attention.self.output_activation.x_min', 'ibert.encoder.layer.7.intermediate.dense.bias_integer', 'ibert.encoder.layer.2.pre_output_act.x_min', 'ibert.encoder.layer.9.output.output_activation.x_min', 'ibert.encoder.layer.10.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.2.intermediate.dense.bias_integer', 'ibert.encoder.layer.11.attention.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.1.attention.self.output_activation.x_min', 'ibert.encoder.layer.7.pre_output_act.act_scaling_factor', 'ibert.encoder.layer.3.pre_output_act.x_min', 'ibert.encoder.layer.3.intermediate.output_activation.x_max', 'ibert.encoder.layer.11.output.output_activation.x_min', 'ibert.encoder.layer.5.attention.self.query_activation.x_max', 'ibert.encoder.layer.8.attention.self.key.bias_integer', 'ibert.encoder.layer.9.pre_output_act.x_max', 'ibert.encoder.layer.4.pre_intermediate_act.x_max', 'ibert.encoder.layer.2.attention.output.dense.fc_scaling_factor', 'ibert.encoder.layer.4.attention.self.value.bias_integer', 'ibert.embeddings.LayerNorm.activation.x_max', 'ibert.encoder.layer.0.output.ln_input_act.x_min', 'ibert.encoder.layer.3.attention.self.value.fc_scaling_factor', 'ibert.encoder.layer.9.intermediate.dense.weight_integer', 'ibert.encoder.layer.2.attention.self.query.fc_scaling_factor', 'ibert.encoder.layer.10.attention.self.query.weight_integer', 'ibert.encoder.layer.10.attention.output.ln_input_act.x_max', 'ibert.encoder.layer.5.attention.self.softmax.act.x_min', 'ibert.encoder.layer.8.attention.self.value.fc_scaling_factor', 'ibert.encoder.layer.1.intermediate.output_activation.x_max', 'ibert.encoder.layer.7.attention.self.query_activation.x_max', 'ibert.encoder.layer.10.intermediate.output_activation.x_max', 'classifier.weight', 'ibert.encoder.layer.7.attention.self.value.weight_integer', 'ibert.encoder.layer.0.attention.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.10.attention.self.query.fc_scaling_factor', 'ibert.encoder.layer.11.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.0.attention.output.output_activation.x_max', 'ibert.encoder.layer.11.attention.output.ln_input_act.x_min', 'ibert.encoder.layer.10.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.4.attention.output.dense.bias_integer', 'ibert.encoder.layer.3.attention.self.output_activation.act_scaling_factor', 'ibert.encoder.layer.1.attention.output.dense.bias_integer', 'ibert.encoder.layer.1.attention.self.softmax.act.x_max', 'ibert.encoder.layer.4.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.5.attention.self.value.weight_integer', 'ibert.encoder.layer.9.attention.self.query_activation.x_min', 'ibert.encoder.layer.11.intermediate.dense.weight_integer', 'ibert.encoder.layer.11.output.dense.bias_integer', 'ibert.encoder.layer.11.attention.self.key.weight_integer', 'ibert.encoder.layer.3.attention.self.value_activation.x_min', 'ibert.encoder.layer.5.output.output_activation.x_min', 'ibert.encoder.layer.8.pre_intermediate_act.x_max', 'ibert.encoder.layer.2.attention.self.output_activation.act_scaling_factor', 'ibert.encoder.layer.1.attention.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.8.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.11.attention.output.ln_input_act.x_max', 'ibert.encoder.layer.9.pre_intermediate_act.x_max', 'ibert.encoder.layer.3.attention.self.key.fc_scaling_factor', 'ibert.encoder.layer.1.attention.output.dense.weight_integer', 'ibert.encoder.layer.7.attention.self.query_activation.act_scaling_factor', 'ibert.encoder.layer.10.intermediate.output_activation.act_scaling_factor', 'ibert.encoder.layer.10.attention.self.value.fc_scaling_factor', 'ibert.encoder.layer.5.pre_output_act.x_min', 'ibert.encoder.layer.0.attention.self.softmax.act.x_min', 'ibert.encoder.layer.2.attention.self.value.bias_integer', 'ibert.encoder.layer.4.output.output_activation.x_max', 'ibert.encoder.layer.7.output.output_activation.x_min', 'ibert.encoder.layer.8.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.9.attention.output.output_activation.x_min', 'ibert.encoder.layer.3.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.9.attention.self.key_activation.x_max', 'ibert.encoder.layer.7.attention.self.query.weight_integer', 'ibert.encoder.layer.9.output.dense.bias_integer', 'ibert.encoder.layer.2.attention.output.ln_input_act.x_max', 'ibert.encoder.layer.10.attention.self.key_activation.act_scaling_factor', 'ibert.encoder.layer.2.output.output_activation.x_min', 'ibert.encoder.layer.9.attention.self.query.fc_scaling_factor', 'ibert.encoder.layer.10.attention.output.dense.bias_integer', 'ibert.encoder.layer.0.attention.output.ln_input_act.x_min', 'ibert.encoder.layer.11.attention.self.value_activation.x_max', 'ibert.encoder.layer.7.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.1.attention.output.output_activation.x_max', 'ibert.encoder.layer.2.attention.self.output_activation.x_max', 'ibert.encoder.layer.4.attention.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.11.attention.output.dense.fc_scaling_factor', 'ibert.encoder.layer.5.attention.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.7.attention.self.key.fc_scaling_factor', 'ibert.encoder.layer.8.output.dense.bias_integer', 'ibert.encoder.layer.9.attention.self.softmax.act.x_max', 'ibert.encoder.layer.11.attention.self.key.fc_scaling_factor', 'ibert.encoder.layer.11.output.output_activation.x_max', 'ibert.encoder.layer.5.output.ln_input_act.x_min', 'ibert.encoder.layer.10.attention.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.5.attention.self.softmax.act.x_max', 'ibert.encoder.layer.7.attention.self.key_activation.x_max', 'ibert.encoder.layer.11.attention.self.query.bias_integer', 'ibert.encoder.layer.0.attention.self.query.weight_integer', 'ibert.encoder.layer.4.attention.self.softmax.act.x_min', 'ibert.encoder.layer.4.intermediate.dense.weight_integer', 'ibert.encoder.layer.4.attention.self.key_activation.x_max', 'ibert.encoder.layer.5.attention.self.value_activation.x_min', 'ibert.encoder.layer.10.attention.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.3.output.output_activation.x_min', 'ibert.encoder.layer.10.attention.self.key.bias_integer', 'ibert.encoder.layer.5.attention.self.key.bias_integer', 'ibert.encoder.layer.0.intermediate.output_activation.act_scaling_factor', 'ibert.encoder.layer.9.attention.self.value_activation.x_max', 'ibert.embeddings.output_activation.x_min', 'ibert.encoder.layer.5.intermediate.output_activation.act_scaling_factor', 'ibert.encoder.layer.11.attention.self.key.bias_integer', 'ibert.encoder.layer.5.attention.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.3.attention.self.key_activation.x_max', 'ibert.encoder.layer.4.pre_intermediate_act.act_scaling_factor', 'ibert.encoder.layer.6.intermediate.output_activation.x_max', 'ibert.encoder.layer.0.attention.output.dense.weight_integer', 'ibert.encoder.layer.0.attention.self.value_activation.x_min', 'ibert.encoder.layer.6.output.LayerNorm.shift', 'ibert.encoder.layer.3.attention.self.query_activation.act_scaling_factor', 'ibert.encoder.layer.9.attention.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.6.attention.self.value_activation.act_scaling_factor', 'ibert.encoder.layer.6.attention.output.ln_input_act.x_min', 'ibert.encoder.layer.8.output.LayerNorm.shift', 'ibert.encoder.layer.10.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.2.pre_output_act.x_max', 'ibert.encoder.layer.8.pre_output_act.act_scaling_factor', 'ibert.encoder.layer.2.attention.self.softmax.act.x_min', 'ibert.encoder.layer.2.pre_intermediate_act.x_min', 'ibert.encoder.layer.10.attention.self.value_activation.act_scaling_factor', 'ibert.encoder.layer.9.output.dense.fc_scaling_factor', 'ibert.encoder.layer.0.attention.self.query_activation.x_max', 'ibert.encoder.layer.4.attention.self.value_activation.act_scaling_factor', 'ibert.encoder.layer.9.intermediate.dense.fc_scaling_factor', 'ibert.encoder.layer.10.attention.output.ln_input_act.x_min', 'ibert.encoder.layer.7.pre_intermediate_act.x_min', 'ibert.encoder.layer.5.attention.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.0.attention.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.6.output.dense.weight_integer', 'ibert.encoder.layer.5.attention.self.query.bias_integer', 'ibert.encoder.layer.1.attention.output.ln_input_act.x_max', 'ibert.encoder.layer.7.attention.output.dense.bias_integer', 'ibert.encoder.layer.5.intermediate.dense.bias_integer', 'ibert.encoder.layer.7.pre_output_act.x_min', 'ibert.encoder.layer.5.output.output_activation.x_max', 'ibert.encoder.layer.1.attention.self.key_activation.x_max', 'ibert.encoder.layer.1.intermediate.dense.bias_integer', 'ibert.encoder.layer.2.pre_output_act.act_scaling_factor', 'ibert.embeddings.output_activation.x_max', 'ibert.encoder.layer.1.pre_output_act.act_scaling_factor', 'ibert.encoder.layer.2.attention.output.dense.weight_integer', 'ibert.encoder.layer.10.attention.self.value_activation.x_min', 'ibert.encoder.layer.6.attention.self.key.bias_integer', 'ibert.encoder.layer.4.intermediate.output_activation.x_min', 'ibert.encoder.layer.1.output.LayerNorm.shift', 'ibert.encoder.layer.7.output.ln_input_act.x_max', 'ibert.encoder.layer.8.attention.output.ln_input_act.x_min', 'ibert.encoder.layer.7.attention.self.output_activation.act_scaling_factor', 'ibert.encoder.layer.0.attention.self.key_activation.x_max', 'ibert.encoder.layer.1.attention.self.query.weight_integer', 'ibert.encoder.layer.8.attention.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.1.attention.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.11.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.11.output.dense.weight_integer', 'ibert.encoder.layer.9.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.5.output.ln_input_act.x_max', 'ibert.encoder.layer.0.attention.self.query_activation.x_min', 'ibert.encoder.layer.5.attention.self.query.fc_scaling_factor', 'ibert.encoder.layer.4.output.ln_input_act.x_min', 'ibert.encoder.layer.3.intermediate.output_activation.act_scaling_factor', 'ibert.encoder.layer.1.output.dense.weight_integer', 'ibert.encoder.layer.1.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.8.attention.self.output_activation.x_min', 'ibert.encoder.layer.9.attention.self.key_activation.x_min', 'ibert.encoder.layer.9.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.4.attention.self.value_activation.x_max', 'ibert.encoder.layer.6.attention.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.5.intermediate.dense.weight_integer', 'ibert.encoder.layer.3.pre_intermediate_act.act_scaling_factor', 'ibert.encoder.layer.8.attention.self.key.weight_integer', 'ibert.encoder.layer.0.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.11.attention.output.dense.bias_integer', 'ibert.encoder.layer.4.attention.self.key.weight_integer', 'ibert.encoder.layer.9.attention.self.output_activation.x_max', 'ibert.encoder.layer.6.attention.self.value.fc_scaling_factor', 'ibert.encoder.layer.6.attention.self.softmax.act.x_min', 'ibert.encoder.layer.0.output.output_activation.x_min', 'ibert.encoder.layer.0.intermediate.output_activation.x_min', 'ibert.encoder.layer.2.intermediate.dense.weight_integer', 'ibert.encoder.layer.0.output.dense.fc_scaling_factor', 'ibert.encoder.layer.7.attention.self.value_activation.act_scaling_factor', 'ibert.encoder.layer.10.attention.self.output_activation.x_min', 'ibert.encoder.layer.3.attention.self.key.bias_integer', 'ibert.encoder.layer.0.attention.self.key.weight_integer', 'ibert.encoder.layer.6.attention.self.value_activation.x_min', 'ibert.encoder.layer.10.attention.self.key.fc_scaling_factor', 'ibert.encoder.layer.11.attention.self.query_activation.x_min', 'ibert.encoder.layer.7.output.ln_input_act.x_min', 'ibert.encoder.layer.6.intermediate.dense.fc_scaling_factor', 'ibert.encoder.layer.3.attention.self.value_activation.x_max', 'ibert.encoder.layer.6.attention.output.output_activation.x_min', 'ibert.encoder.layer.4.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.0.intermediate.dense.bias_integer', 'ibert.encoder.layer.7.intermediate.dense.weight_integer', 'ibert.encoder.layer.9.attention.output.output_activation.x_max', 'ibert.encoder.layer.6.output.dense.fc_scaling_factor', 'ibert.encoder.layer.7.attention.output.ln_input_act.x_min', 'ibert.encoder.layer.8.intermediate.output_activation.act_scaling_factor', 'ibert.encoder.layer.9.attention.self.key.fc_scaling_factor', 'ibert.encoder.layer.5.attention.output.output_activation.x_max', 'ibert.encoder.layer.11.attention.self.key_activation.x_min', 'ibert.encoder.layer.6.output.output_activation.x_max', 'ibert.encoder.layer.9.attention.self.value.weight_integer', 'ibert.encoder.layer.6.attention.self.key_activation.x_max', 'ibert.encoder.layer.9.attention.self.key.weight_integer', 'ibert.encoder.layer.0.attention.self.query.bias_integer', 'ibert.encoder.layer.5.attention.output.dense.weight_integer', 'ibert.encoder.layer.7.attention.self.value_activation.x_min', 'ibert.encoder.layer.4.output.dense.bias_integer', 'ibert.encoder.layer.7.output.dense.weight_integer', 'ibert.encoder.layer.5.attention.output.LayerNorm.shift', 'ibert.encoder.layer.0.pre_intermediate_act.x_min', 'ibert.encoder.layer.7.attention.self.output_activation.x_min', 'ibert.encoder.layer.11.output.ln_input_act.x_min', 'ibert.embeddings.embeddings_act2.x_max', 'ibert.encoder.layer.11.attention.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.8.attention.output.output_activation.x_min', 'ibert.encoder.layer.7.attention.self.softmax.act.x_max', 'ibert.encoder.layer.7.attention.self.output_activation.x_max', 'ibert.encoder.layer.1.attention.self.value_activation.x_min', 'ibert.encoder.layer.9.output.LayerNorm.shift', 'ibert.encoder.layer.0.attention.self.key_activation.x_min', 'ibert.encoder.layer.7.pre_intermediate_act.x_max', 'ibert.encoder.layer.9.pre_intermediate_act.act_scaling_factor', 'ibert.encoder.layer.11.attention.self.softmax.act.act_scaling_factor', 'ibert.encoder.layer.8.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.9.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.6.attention.self.value.bias_integer', 'ibert.encoder.layer.6.attention.self.query_activation.x_max', 'ibert.encoder.layer.8.attention.self.query_activation.act_scaling_factor', 'ibert.encoder.layer.9.attention.output.dense.weight_integer', 'ibert.encoder.layer.6.output.ln_input_act.x_min', 'ibert.encoder.layer.11.attention.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.11.pre_output_act.act_scaling_factor', 'ibert.encoder.layer.6.attention.self.key.fc_scaling_factor', 'ibert.encoder.layer.0.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.2.attention.self.output_activation.x_min', 'ibert.encoder.layer.8.pre_output_act.x_min', 'ibert.encoder.layer.3.attention.self.key_activation.x_min', 'ibert.encoder.layer.3.output.dense.bias_integer', 'ibert.encoder.layer.7.attention.output.dense.fc_scaling_factor', 'ibert.encoder.layer.10.pre_intermediate_act.x_max', 'ibert.encoder.layer.4.attention.self.output_activation.x_min', 'ibert.encoder.layer.0.attention.self.output_activation.act_scaling_factor', 'ibert.encoder.layer.1.intermediate.dense.weight_integer', 'ibert.encoder.layer.5.attention.self.query.weight_integer', 'ibert.encoder.layer.7.attention.self.value.fc_scaling_factor', 'ibert.encoder.layer.11.output.dense.fc_scaling_factor', 'ibert.encoder.layer.4.attention.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.8.output.output_activation.x_min', 'ibert.encoder.layer.3.attention.self.query_activation.x_min', 'ibert.encoder.layer.10.attention.output.output_activation.x_min', 'ibert.encoder.layer.4.attention.self.softmax.act.x_max', 'ibert.encoder.layer.3.attention.self.softmax.act.act_scaling_factor', 'ibert.encoder.layer.11.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.9.attention.output.dense.fc_scaling_factor', 'ibert.encoder.layer.1.attention.self.key_activation.act_scaling_factor', 'ibert.encoder.layer.6.intermediate.dense.weight_integer', 'ibert.encoder.layer.9.intermediate.dense.bias_integer', 'ibert.encoder.layer.8.attention.self.output_activation.act_scaling_factor', 'ibert.encoder.layer.7.intermediate.output_activation.act_scaling_factor', 'ibert.encoder.layer.3.attention.self.query.bias_integer', 'ibert.encoder.layer.0.pre_intermediate_act.act_scaling_factor', 'ibert.encoder.layer.11.attention.self.query_activation.act_scaling_factor', 'ibert.encoder.layer.1.attention.self.key_activation.x_min', 'ibert.encoder.layer.6.attention.self.key_activation.x_min', 'ibert.encoder.layer.10.output.dense.fc_scaling_factor', 'ibert.embeddings.token_type_embeddings.weight_integer', 'ibert.encoder.layer.5.attention.self.value.bias_integer', 'ibert.encoder.layer.4.pre_output_act.act_scaling_factor', 'ibert.encoder.layer.3.attention.output.output_activation.x_min', 'ibert.encoder.layer.1.intermediate.output_activation.act_scaling_factor', 'ibert.encoder.layer.10.attention.self.value_activation.x_max', 'ibert.encoder.layer.5.output.LayerNorm.shift', 'ibert.encoder.layer.9.attention.self.query.bias_integer', 'ibert.encoder.layer.2.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.6.intermediate.dense.bias_integer', 'ibert.encoder.layer.1.pre_output_act.x_min', 'ibert.encoder.layer.7.output.output_activation.x_max', 'ibert.encoder.layer.10.attention.self.query_activation.x_min', 'ibert.encoder.layer.6.attention.self.value.weight_integer', 'ibert.encoder.layer.9.attention.self.query_activation.act_scaling_factor', 'ibert.encoder.layer.2.attention.self.key.weight_integer', 'ibert.embeddings.word_embeddings.weight_scaling_factor', 'ibert.encoder.layer.7.intermediate.dense.fc_scaling_factor', 'ibert.encoder.layer.9.attention.self.key_activation.act_scaling_factor', 'ibert.encoder.layer.1.attention.self.query.bias_integer', 'ibert.encoder.layer.5.intermediate.dense.fc_scaling_factor', 'ibert.encoder.layer.5.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.7.attention.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.7.attention.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.4.intermediate.dense.bias_integer', 'ibert.encoder.layer.6.attention.self.output_activation.act_scaling_factor', 'ibert.encoder.layer.10.output.output_activation.x_min', 'ibert.encoder.layer.8.output.dense.fc_scaling_factor', 'ibert.encoder.layer.6.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.7.attention.self.key.weight_integer', 'ibert.encoder.layer.10.intermediate.dense.bias_integer', 'ibert.encoder.layer.7.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.4.attention.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.10.attention.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.11.intermediate.dense.fc_scaling_factor', 'ibert.encoder.layer.11.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.2.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.10.attention.self.query_activation.act_scaling_factor', 'ibert.encoder.layer.6.attention.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.8.attention.output.LayerNorm.shift', 'ibert.encoder.layer.3.attention.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.5.attention.output.dense.fc_scaling_factor', 'ibert.encoder.layer.6.output.output_activation.x_min', 'ibert.encoder.layer.11.attention.self.value_activation.x_min', 'ibert.encoder.layer.10.attention.output.dense.weight_integer', 'ibert.encoder.layer.11.pre_output_act.x_min', 'ibert.encoder.layer.7.attention.output.LayerNorm.shift', 'ibert.encoder.layer.4.attention.self.query_activation.act_scaling_factor', 'ibert.encoder.layer.5.pre_intermediate_act.x_max', 'ibert.encoder.layer.6.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.2.attention.self.key_activation.act_scaling_factor', 'ibert.encoder.layer.5.attention.self.output_activation.act_scaling_factor', 'ibert.encoder.layer.7.pre_intermediate_act.act_scaling_factor', 'ibert.encoder.layer.8.intermediate.dense.bias_integer', 'ibert.encoder.layer.6.attention.self.key_activation.act_scaling_factor', 'ibert.encoder.layer.8.intermediate.output_activation.x_min', 'ibert.encoder.layer.2.attention.self.key_activation.x_min', 'ibert.encoder.layer.9.attention.self.query.weight_integer', 'ibert.encoder.layer.6.pre_intermediate_act.act_scaling_factor', 'ibert.encoder.layer.7.attention.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.7.attention.output.output_activation.x_min', 'ibert.encoder.layer.0.attention.self.softmax.act.act_scaling_factor', 'ibert.encoder.layer.1.attention.self.value.bias_integer', 'ibert.encoder.layer.10.intermediate.dense.fc_scaling_factor', 'ibert.encoder.layer.10.attention.self.output_activation.x_max', 'ibert.encoder.layer.2.output.dense.weight_integer', 'ibert.encoder.layer.10.pre_output_act.x_max', 'ibert.encoder.layer.2.attention.self.query.weight_integer', 'ibert.encoder.layer.4.pre_output_act.x_min', 'ibert.encoder.layer.6.attention.output.dense.fc_scaling_factor', 'ibert.encoder.layer.1.attention.self.value.fc_scaling_factor', 'ibert.embeddings.embeddings_act2.x_min', 'ibert.encoder.layer.7.pre_output_act.x_max', 'ibert.encoder.layer.9.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.3.intermediate.dense.fc_scaling_factor', 'ibert.encoder.layer.6.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.10.output.ln_input_act.x_max', 'ibert.encoder.layer.1.attention.self.query_activation.x_max', 'ibert.encoder.layer.11.attention.self.query_activation.x_max', 'ibert.encoder.layer.9.attention.self.softmax.act.act_scaling_factor', 'ibert.encoder.layer.4.attention.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.0.pre_output_act.x_min', 'ibert.encoder.layer.7.attention.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.7.output.dense.bias_integer', 'ibert.encoder.layer.0.attention.self.key.bias_integer', 'ibert.encoder.layer.1.attention.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.0.attention.self.value_activation.act_scaling_factor', 'ibert.encoder.layer.10.pre_intermediate_act.act_scaling_factor', 'ibert.encoder.layer.3.output.ln_input_act.x_min', 'ibert.encoder.layer.4.intermediate.dense.fc_scaling_factor', 'ibert.encoder.layer.0.attention.self.output_activation.x_max', 'ibert.encoder.layer.9.attention.self.softmax.act.x_min', 'ibert.encoder.layer.4.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.6.attention.self.key.weight_integer', 'ibert.encoder.layer.9.attention.self.value_activation.x_min', 'ibert.encoder.layer.10.attention.self.output_activation.act_scaling_factor', 'ibert.encoder.layer.4.attention.output.dense.fc_scaling_factor', 'ibert.encoder.layer.1.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.0.attention.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.6.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.8.attention.self.query.weight_integer', 'ibert.encoder.layer.10.pre_intermediate_act.x_min', 'ibert.encoder.layer.4.attention.self.query_activation.x_max', 'ibert.encoder.layer.8.attention.self.key_activation.x_max', 'ibert.encoder.layer.1.attention.self.softmax.act.act_scaling_factor', 'ibert.encoder.layer.6.pre_output_act.x_min', 'ibert.encoder.layer.8.intermediate.dense.weight_integer', 'ibert.encoder.layer.5.pre_output_act.x_max', 'ibert.encoder.layer.7.attention.output.dense.weight_integer', 'ibert.encoder.layer.2.pre_intermediate_act.x_max', 'ibert.encoder.layer.3.attention.output.dense.fc_scaling_factor', 'ibert.encoder.layer.2.attention.self.value_activation.x_min', 'ibert.encoder.layer.7.attention.self.query_activation.x_min', 'ibert.encoder.layer.3.pre_output_act.act_scaling_factor', 'ibert.encoder.layer.10.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.5.attention.self.value.fc_scaling_factor', 'ibert.encoder.layer.2.output.dense.fc_scaling_factor', 'ibert.encoder.layer.9.attention.self.value_activation.act_scaling_factor', 'ibert.encoder.layer.9.attention.self.value.fc_scaling_factor', 'ibert.encoder.layer.5.attention.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.3.attention.output.LayerNorm.shift', 'ibert.encoder.layer.7.attention.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.0.attention.output.dense.fc_scaling_factor', 'ibert.encoder.layer.10.attention.output.LayerNorm.shift', 'ibert.encoder.layer.8.output.ln_input_act.x_min', 'ibert.encoder.layer.6.attention.self.output_activation.x_max', 'ibert.encoder.layer.2.output.ln_input_act.x_min', 'ibert.encoder.layer.10.output.LayerNorm.shift', 'ibert.encoder.layer.7.attention.self.key.bias_integer', 'ibert.encoder.layer.5.attention.self.key_activation.x_max', 'ibert.encoder.layer.8.attention.self.softmax.act.x_min', 'ibert.encoder.layer.5.attention.output.ln_input_act.x_min', 'ibert.encoder.layer.6.output.dense.bias_integer', 'ibert.encoder.layer.7.attention.self.value_activation.x_max', 'ibert.embeddings.position_embeddings.weight_scaling_factor', 'ibert.encoder.layer.4.attention.self.value.weight_integer', 'ibert.encoder.layer.3.attention.self.softmax.act.x_max', 'ibert.encoder.layer.0.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.2.pre_intermediate_act.act_scaling_factor', 'ibert.encoder.layer.8.attention.self.query_activation.x_max', 'ibert.encoder.layer.4.attention.output.ln_input_act.x_min', 'ibert.encoder.layer.8.pre_output_act.x_max', 'ibert.encoder.layer.6.pre_intermediate_act.x_min', 'ibert.encoder.layer.1.output.ln_input_act.x_min', 'ibert.encoder.layer.5.attention.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.8.attention.output.dense.fc_scaling_factor', 'ibert.encoder.layer.8.attention.output.dense.bias_integer', 'ibert.encoder.layer.11.attention.output.output_activation.x_max', 'ibert.encoder.layer.4.attention.self.value.fc_scaling_factor', 'ibert.encoder.layer.2.attention.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.1.pre_intermediate_act.act_scaling_factor', 'ibert.embeddings.LayerNorm.activation.x_min', 'ibert.encoder.layer.8.attention.self.key_activation.act_scaling_factor', 'ibert.encoder.layer.2.attention.self.query_activation.act_scaling_factor', 'ibert.encoder.layer.3.attention.self.query_activation.x_max', 'ibert.encoder.layer.6.pre_output_act.act_scaling_factor', 'ibert.encoder.layer.3.attention.self.query.weight_integer', 'ibert.encoder.layer.4.attention.self.key.bias_integer', 'ibert.encoder.layer.2.attention.self.key_activation.x_max', 'ibert.encoder.layer.0.attention.self.value.fc_scaling_factor', 'ibert.encoder.layer.5.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.11.output.ln_input_act.x_max', 'ibert.encoder.layer.11.attention.self.query.fc_scaling_factor', 'ibert.encoder.layer.10.attention.output.dense.fc_scaling_factor', 'ibert.encoder.layer.1.attention.self.softmax.act.x_min', 'ibert.encoder.layer.2.intermediate.output_activation.x_max', 'ibert.encoder.layer.2.attention.self.query.bias_integer', 'ibert.encoder.layer.4.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.4.pre_intermediate_act.x_min', 'ibert.encoder.layer.4.attention.self.query.weight_integer', 'ibert.encoder.layer.6.intermediate.output_activation.x_min', 'ibert.encoder.layer.3.attention.output.ln_input_act.x_min', 'ibert.encoder.layer.11.attention.self.output_activation.x_min', 'ibert.encoder.layer.2.attention.self.value_activation.act_scaling_factor', 'ibert.encoder.layer.4.attention.self.key.fc_scaling_factor', 'ibert.encoder.layer.4.output.dense.fc_scaling_factor', 'ibert.encoder.layer.2.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.2.intermediate.dense.fc_scaling_factor', 'ibert.encoder.layer.3.attention.self.value_activation.act_scaling_factor', 'ibert.encoder.layer.5.output.dense.bias_integer', 'ibert.encoder.layer.7.attention.self.query.bias_integer', 'ibert.encoder.layer.7.intermediate.output_activation.x_min', 'ibert.encoder.layer.9.attention.self.output_activation.act_scaling_factor', 'ibert.encoder.layer.8.attention.self.output_activation.x_max', 'ibert.encoder.layer.2.output.output_activation.x_max', 'ibert.encoder.layer.8.attention.self.key.fc_scaling_factor', 'ibert.encoder.layer.10.attention.self.value.weight_integer', 'ibert.encoder.layer.11.attention.self.key_activation.act_scaling_factor', 'ibert.encoder.layer.7.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.10.attention.self.key_activation.x_max', 'ibert.encoder.layer.8.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.3.attention.self.key_activation.act_scaling_factor', 'ibert.encoder.layer.1.attention.output.output_activation.x_min', 'ibert.encoder.layer.2.attention.self.key.bias_integer', 'ibert.encoder.layer.2.attention.output.ln_input_act.x_min', 'ibert.encoder.layer.1.output.dense.bias_integer', 'ibert.encoder.layer.11.attention.self.value.weight_integer', 'ibert.encoder.layer.1.attention.self.query.fc_scaling_factor', 'ibert.encoder.layer.4.attention.self.key_activation.act_scaling_factor', 'ibert.encoder.layer.3.attention.self.softmax.act.x_min', 'ibert.encoder.layer.9.intermediate.output_activation.x_max', 'ibert.encoder.layer.1.output.output_activation.x_min', 'ibert.encoder.layer.2.attention.output.LayerNorm.shift', 'ibert.embeddings.position_embeddings.weight_integer', 'ibert.encoder.layer.7.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.9.attention.output.dense.bias_integer', 'ibert.encoder.layer.2.attention.self.value.weight_integer', 'ibert.encoder.layer.6.attention.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.6.intermediate.output_activation.act_scaling_factor', 'ibert.encoder.layer.5.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.6.attention.self.query.bias_integer', 'ibert.encoder.layer.2.attention.self.key.fc_scaling_factor', 'ibert.encoder.layer.1.pre_intermediate_act.x_min', 'ibert.encoder.layer.0.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.1.output.dense.fc_scaling_factor', 'ibert.encoder.layer.11.intermediate.dense.bias_integer', 'ibert.encoder.layer.11.attention.output.LayerNorm.shift', 'ibert.encoder.layer.11.attention.output.dense.weight_integer', 'ibert.encoder.layer.8.attention.output.ln_input_act.x_max', 'ibert.encoder.layer.6.attention.self.query.weight_integer', 'ibert.encoder.layer.9.attention.self.query_activation.x_max', 'ibert.encoder.layer.10.attention.self.softmax.act.act_scaling_factor', 'ibert.encoder.layer.9.output.output_activation.x_max', 'ibert.encoder.layer.11.attention.self.query.weight_integer', 'ibert.encoder.layer.9.attention.self.output_activation.x_min', 'ibert.encoder.layer.11.attention.self.key_activation.x_max', 'ibert.encoder.layer.6.attention.self.value_activation.x_max', 'ibert.encoder.layer.8.attention.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.9.attention.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.9.attention.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.11.attention.self.softmax.act.x_max', 'ibert.encoder.layer.3.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.3.output.output_activation.x_max', 'ibert.encoder.layer.5.attention.output.output_activation.x_min', 'ibert.encoder.layer.7.attention.self.softmax.act.x_min', 'ibert.encoder.layer.1.attention.output.ln_input_act.x_min', 'ibert.encoder.layer.0.attention.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.10.attention.self.key.weight_integer', 'ibert.encoder.layer.10.attention.self.query_activation.x_max', 'ibert.encoder.layer.2.attention.self.query_activation.x_max', 'ibert.encoder.layer.3.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.7.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.8.output.LayerNorm.activation.act_scaling_factor', 'ibert.encoder.layer.10.output.dense.bias_integer', 'ibert.encoder.layer.6.attention.self.softmax.act.x_max', 'ibert.encoder.layer.4.pre_output_act.x_max', 'ibert.encoder.layer.6.attention.self.output_activation.x_min', 'ibert.encoder.layer.8.attention.self.value_activation.x_max', 'ibert.encoder.layer.1.attention.self.key.bias_integer', 'ibert.encoder.layer.3.intermediate.output_activation.x_min', 'ibert.encoder.layer.4.attention.self.query_activation.x_min', 'ibert.encoder.layer.6.attention.self.query.fc_scaling_factor', 'ibert.encoder.layer.10.output.dense.weight_integer', 'ibert.encoder.layer.11.pre_intermediate_act.act_scaling_factor', 'ibert.encoder.layer.5.pre_intermediate_act.act_scaling_factor', 'ibert.encoder.layer.0.attention.self.query_activation.act_scaling_factor', 'ibert.encoder.layer.7.attention.output.ln_input_act.x_max', 'ibert.encoder.layer.11.attention.self.output_activation.x_max', 'ibert.encoder.layer.2.attention.output.output_activation.x_max', 'ibert.encoder.layer.11.attention.self.output_activation.act_scaling_factor', 'ibert.encoder.layer.7.attention.output.output_activation.x_max', 'ibert.encoder.layer.0.output.dense.weight_integer', 'ibert.encoder.layer.0.intermediate.dense.weight_integer', 'ibert.encoder.layer.9.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.0.output.dense.bias_integer', 'ibert.encoder.layer.3.attention.self.value.bias_integer', 'ibert.encoder.layer.0.output.LayerNorm.shift', 'ibert.encoder.layer.11.pre_output_act.x_max', 'ibert.encoder.layer.11.attention.self.value.fc_scaling_factor', 'ibert.encoder.layer.0.output.ln_input_act.x_max', 'ibert.encoder.layer.5.attention.self.value_activation.x_max', 'ibert.encoder.layer.3.attention.output.dense.bias_integer', 'ibert.encoder.layer.3.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.6.attention.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.1.attention.self.query_activation.x_min', 'ibert.encoder.layer.1.attention.self.value.weight_integer', 'ibert.encoder.layer.2.attention.self.query_activation.x_min', 'ibert.encoder.layer.4.attention.self.query.bias_integer', 'ibert.encoder.layer.11.attention.output.output_activation.x_min', 'ibert.encoder.layer.3.attention.self.output_activation.x_max', 'ibert.encoder.layer.0.intermediate.dense.fc_scaling_factor', 'ibert.encoder.layer.6.attention.self.softmax.act.act_scaling_factor', 'ibert.encoder.layer.3.attention.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.2.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.3.attention.self.key.weight_integer', 'ibert.encoder.layer.4.attention.output.dense.weight_integer', 'ibert.encoder.layer.10.intermediate.dense.weight_integer', 'ibert.encoder.layer.8.attention.self.value.bias_integer', 'ibert.encoder.layer.4.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.0.attention.self.value.bias_integer', 'ibert.embeddings.output_activation.act_scaling_factor', 'ibert.encoder.layer.9.intermediate.output_activation.act_scaling_factor', 'ibert.encoder.layer.4.output.output_activation.x_min', 'ibert.encoder.layer.11.output.LayerNorm.activation.x_min', 'ibert.embeddings.word_embeddings.weight_integer', 'ibert.encoder.layer.1.attention.self.output_activation.act_scaling_factor', 'ibert.encoder.layer.10.output.ln_input_act.x_min', 'ibert.embeddings.token_type_embeddings.weight_scaling_factor', 'ibert.encoder.layer.0.attention.self.query.fc_scaling_factor', 'ibert.encoder.layer.2.attention.output.output_activation.x_min', 'ibert.encoder.layer.6.output.ln_input_act.x_max', 'ibert.encoder.layer.2.output.dense.bias_integer', 'ibert.encoder.layer.2.intermediate.output_activation.act_scaling_factor', 'ibert.encoder.layer.8.attention.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.1.attention.self.key.weight_integer', 'ibert.encoder.layer.10.attention.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.4.attention.self.output_activation.x_max', 'ibert.encoder.layer.10.pre_output_act.act_scaling_factor', 'ibert.encoder.layer.0.attention.output.ln_input_act.x_max', 'ibert.encoder.layer.5.pre_output_act.act_scaling_factor', 'ibert.encoder.layer.4.attention.self.key_activation.x_min', 'ibert.encoder.layer.0.pre_output_act.x_max', 'ibert.encoder.layer.5.attention.self.key_activation.x_min', 'ibert.encoder.layer.3.attention.self.query.fc_scaling_factor', 'ibert.encoder.layer.4.attention.self.query.fc_scaling_factor', 'ibert.encoder.layer.10.attention.output.LayerNorm.activation.x_min', 'ibert.encoder.layer.5.output.dense.weight_integer', 'ibert.encoder.layer.1.output.output_activation.x_max', 'ibert.encoder.layer.11.intermediate.output_activation.x_min', 'ibert.encoder.layer.10.attention.output.output_activation.x_max', 'ibert.encoder.layer.2.attention.self.softmax.act.x_max', 'ibert.encoder.layer.8.intermediate.output_activation.x_max', 'ibert.encoder.layer.3.pre_intermediate_act.x_max', 'ibert.encoder.layer.11.intermediate.output_activation.x_max', 'ibert.encoder.layer.7.attention.self.query.fc_scaling_factor', 'ibert.encoder.layer.1.intermediate.dense.fc_scaling_factor', 'ibert.encoder.layer.11.pre_intermediate_act.x_min', 'ibert.encoder.layer.6.attention.output.ln_input_act.x_max', 'ibert.encoder.layer.2.attention.self.value_activation.x_max', 'ibert.encoder.layer.3.attention.output.ln_input_act.x_max', 'ibert.encoder.layer.11.output.LayerNorm.shift', 'ibert.encoder.layer.0.pre_output_act.act_scaling_factor', 'ibert.encoder.layer.1.attention.output.LayerNorm.shift', 'ibert.encoder.layer.7.output.LayerNorm.shift', 'classifier.bias', 'ibert.encoder.layer.10.attention.self.key_activation.x_min', 'ibert.encoder.layer.0.attention.self.value.weight_integer', 'ibert.encoder.layer.4.output.dense.weight_integer', 'ibert.encoder.layer.1.pre_output_act.x_max', 'ibert.encoder.layer.5.attention.output.ln_input_act.x_max', 'ibert.encoder.layer.3.pre_output_act.x_max', 'ibert.encoder.layer.10.attention.self.value.bias_integer', 'ibert.embeddings.embeddings_act1.x_min', 'ibert.encoder.layer.9.output.ln_input_act.x_max', 'ibert.encoder.layer.3.output.LayerNorm.shift', 'ibert.encoder.layer.2.attention.self.value.fc_scaling_factor', 'ibert.encoder.layer.0.attention.self.key_activation.act_scaling_factor', 'ibert.encoder.layer.4.output.ln_input_act.x_max', 'ibert.encoder.layer.3.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.8.attention.self.key_activation.x_min', 'ibert.encoder.layer.8.attention.output.dense.weight_integer', 'ibert.encoder.layer.9.attention.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.8.attention.self.value.weight_integer', 'ibert.encoder.layer.3.intermediate.dense.bias_integer', 'ibert.encoder.layer.11.attention.self.value_activation.act_scaling_factor', 'ibert.encoder.layer.0.attention.output.dense.bias_integer', 'ibert.encoder.layer.5.attention.self.key.weight_integer', 'ibert.encoder.layer.3.attention.output.LayerNorm.activation.x_max', 'ibert.encoder.layer.6.attention.output.LayerNorm.shift', 'ibert.embeddings.LayerNorm.shift', 'ibert.encoder.layer.1.attention.self.value_activation.act_scaling_factor', 'ibert.encoder.layer.6.attention.self.query_activation.act_scaling_factor', 'ibert.encoder.layer.2.output.output_activation.act_scaling_factor', 'ibert.encoder.layer.5.output.dense.fc_scaling_factor', 'ibert.encoder.layer.11.attention.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.0.attention.self.key.fc_scaling_factor', 'ibert.encoder.layer.9.output.ln_input_act.x_min', 'ibert.encoder.layer.3.attention.output.dense.weight_integer', 'ibert.encoder.layer.3.output.dense.fc_scaling_factor', 'ibert.embeddings.embeddings_act1.act_scaling_factor', 'ibert.encoder.layer.2.output.ln_input_act.x_max', 'ibert.encoder.layer.8.intermediate.dense.fc_scaling_factor', 'ibert.encoder.layer.5.attention.self.key_activation.act_scaling_factor', 'ibert.encoder.layer.5.attention.self.value_activation.act_scaling_factor', 'ibert.encoder.layer.2.attention.output.ln_input_act.act_scaling_factor', 'ibert.encoder.layer.8.attention.self.value_activation.x_min', 'ibert.encoder.layer.7.attention.self.value.bias_integer', 'ibert.encoder.layer.4.attention.output.output_activation.x_min']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "model = (AutoModelForTokenClassification.from_pretrained(\n",
    "    MODEL_CKPT,\n",
    "    num_labels=NUM_OF_LABELS,\n",
    "    id2label=id2label,\n",
    "    label2id=label2id\n",
    "    ).to(DEVICE))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create Compute Metrics Function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "label_list = pos_tag_values\n",
    "\n",
    "seqeval = evaluate.load(\"seqeval\")\n",
    "\n",
    "labels = [label_list[i] for i in example[f'ner_tags']]\n",
    "\n",
    "def compute_metrics(eval_preds):\n",
    "    predictions, labels = eval_preds\n",
    "    predictions = np.argmax(predictions, \n",
    "                            axis=2)\n",
    "    \n",
    "    true_predictions = [\n",
    "        [label_list[p] for (p, l) in zip(prediction, label) if l != -100]\n",
    "        for prediction, label in zip(predictions, labels)\n",
    "    ]\n",
    "    \n",
    "    true_labels = [\n",
    "        [label_list[l] for (p, l) in zip(prediction, label) if l != -100]\n",
    "        for prediction, label in zip(predictions, labels)\n",
    "    ]\n",
    "    \n",
    "    results = seqeval.compute(predictions=true_predictions, \n",
    "                              references=true_labels)\n",
    "    \n",
    "    return results"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define TrainingArguments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "args = TrainingArguments(\n",
    "    MODEL_NAME,\n",
    "    log_level=\"error\",\n",
    "    logging_first_step=True,\n",
    "    learning_rate=LR,\n",
    "    num_train_epochs=NUM_OF_EPOCHS,\n",
    "    per_device_train_batch_size=BATCH_SIZE,\n",
    "    per_device_eval_batch_size=BATCH_SIZE,\n",
    "    evaluation_strategy=STRATEGY,\n",
    "    report_to=REPORTS_TO,\n",
    "    disable_tqdm=False,\n",
    "    logging_steps=STEPS,\n",
    "    weight_decay=WEIGHT_DECAY,\n",
    "    save_strategy=STRATEGY,\n",
    "    hub_private_repo=True,\n",
    "    push_to_hub=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Subclass Trainer to Handle Class Imbalance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CustomTrainer(Trainer):\n",
    "    def compute_loss(self, \n",
    "                     model, \n",
    "                     inputs, \n",
    "                     return_outputs=False):\n",
    "        \n",
    "        labels = inputs.get(\"labels\")\n",
    "        # forward pass\n",
    "        outputs = model(**inputs)\n",
    "        logits = outputs.get(\"logits\")\n",
    "        # compute custom loss \n",
    "        loss_fct = nn.CrossEntropyLoss(weight=torch.tensor(\n",
    "            [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0], \n",
    "            device=model.device)\n",
    "        )\n",
    "        loss = loss_fct(logits.view(-1, \n",
    "                                    self.model.config.num_labels), \n",
    "                        labels.view(-1)\n",
    "                        )\n",
    "        return (loss, outputs) if return_outputs else loss"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define Trainer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Cloning https://huggingface.co/DunnBC22/ibert-roberta-base-finetuned-WikiNeural into local empty directory.\n"
     ]
    }
   ],
   "source": [
    "trainer = CustomTrainer(model, \n",
    "                  args=args,\n",
    "                  data_collator=data_collator,\n",
    "                  compute_metrics=compute_metrics,\n",
    "                  tokenizer=tokenizer,\n",
    "                  train_dataset=encoded_ds[\"train\"],\n",
    "                  eval_dataset=encoded_ds[\"test\"]\n",
    "                  )"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Train Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/briandunn/Documents/deep_learning/audio_projects/lib/python3.9/site-packages/transformers/optimization.py:391: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "33690103144846e6b78909b8d87451e6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/11590 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'loss': 2.1466, 'learning_rate': 1.9998274374460744e-05, 'epoch': 0.0}\n",
      "{'loss': 0.2122, 'learning_rate': 1.7842968075927526e-05, 'epoch': 0.22}\n",
      "{'loss': 0.1322, 'learning_rate': 1.5685936151855047e-05, 'epoch': 0.43}\n",
      "{'loss': 0.1173, 'learning_rate': 1.3528904227782573e-05, 'epoch': 0.65}\n",
      "{'loss': 0.1092, 'learning_rate': 1.1371872303710096e-05, 'epoch': 0.86}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7e9e7e86a819481aa138133196403ad8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/725 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.09867024421691895, 'eval_LOC': {'precision': 0.9124507227332457, 'recall': 0.9328295549958019, 'f1': 0.9225276093996512, 'number': 5955}, 'eval_MISC': {'precision': 0.8003130979300748, 'recall': 0.9091088717644734, 'f1': 0.8512488436632748, 'number': 5061}, 'eval_ORG': {'precision': 0.9142857142857143, 'recall': 0.9278051609162076, 'f1': 0.9209958267376601, 'number': 3449}, 'eval_PER': {'precision': 0.9714229013693193, 'recall': 0.9395393474088292, 'f1': 0.9552151429407748, 'number': 5210}, 'eval_overall_precision': 0.8957106399685905, 'eval_overall_recall': 0.9276238881829734, 'eval_overall_f1': 0.9113879803250855, 'eval_overall_accuracy': 0.9889951526267523, 'eval_runtime': 862.4046, 'eval_samples_per_second': 13.447, 'eval_steps_per_second': 0.841, 'epoch': 1.0}\n",
      "{'loss': 0.0979, 'learning_rate': 9.214840379637619e-06, 'epoch': 1.08}\n",
      "{'loss': 0.0804, 'learning_rate': 7.057808455565142e-06, 'epoch': 1.29}\n",
      "{'loss': 0.0741, 'learning_rate': 4.900776531492667e-06, 'epoch': 1.51}\n",
      "{'loss': 0.0755, 'learning_rate': 2.74374460742019e-06, 'epoch': 1.73}\n",
      "{'loss': 0.0723, 'learning_rate': 5.867126833477136e-07, 'epoch': 1.94}\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "babcdc6aea3c4e4093e611a6d219f91b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/725 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'eval_loss': 0.0877607986330986, 'eval_LOC': {'precision': 0.9249338624338624, 'recall': 0.9393786733837112, 'f1': 0.9321003082562693, 'number': 5955}, 'eval_MISC': {'precision': 0.8304751697034656, 'recall': 0.9185931634064414, 'f1': 0.8723144760296463, 'number': 5061}, 'eval_ORG': {'precision': 0.9283453237410072, 'recall': 0.9353435778486517, 'f1': 0.9318313113807049, 'number': 3449}, 'eval_PER': {'precision': 0.9698098412076064, 'recall': 0.9495201535508637, 'f1': 0.9595577538551062, 'number': 5210}, 'eval_overall_precision': 0.9106913262783107, 'eval_overall_recall': 0.936010165184244, 'eval_overall_f1': 0.9231771812416973, 'eval_overall_accuracy': 0.9908854410361027, 'eval_runtime': 1370.1951, 'eval_samples_per_second': 8.464, 'eval_steps_per_second': 0.529, 'epoch': 2.0}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Adding files tracked by Git LFS: ['.DS_Store']. This may take a bit of time if the files are large.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'train_runtime': 48176.6313, 'train_samples_per_second': 3.849, 'train_steps_per_second': 0.241, 'train_loss': 0.10698214953474386, 'epoch': 2.0}\n"
     ]
    }
   ],
   "source": [
    "train_results = trainer.train()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Push Model To Hub"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Several commits (2) will be pushed upstream.\n",
      "The progress bars may be unreliable.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5c482533645d46c0b7abb3756080ece9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Upload file pytorch_model.bin:   0%|          | 1.00/947M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "54324099535e43bcbd5a528fa25b33de",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Upload file runs/Jul12_21-58-22_Brians-Mac-mini/events.out.tfevents.1689217120.Brians-Mac-mini.4106.0:   0%|  …"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "To https://huggingface.co/DunnBC22/ibert-roberta-base-finetuned-WikiNeural\n",
      "   c3c8464..a6532d1  main -> main\n",
      "\n",
      "To https://huggingface.co/DunnBC22/ibert-roberta-base-finetuned-WikiNeural\n",
      "   a6532d1..cc297ba  main -> main\n",
      "\n"
     ]
    }
   ],
   "source": [
    "kwargs = {\n",
    "    \"tasks\": \"token-classification\",\n",
    "    \"tags\": ['token-classification'],\n",
    "}\n",
    "\n",
    "if args.push_to_hub:\n",
    "    trainer.push_to_hub('All DUNN!!!', **kwargs)\n",
    "else:\n",
    "    trainer.create_model_card(**kwargs)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Save & Log Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "To https://huggingface.co/DunnBC22/ibert-roberta-base-finetuned-WikiNeural\n",
      "   cc297ba..d4f8422  main -> main\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "***** train metrics *****\n",
      "  epoch                    =         2.0\n",
      "  train_loss               =       0.107\n",
      "  train_runtime            = 13:22:56.63\n",
      "  train_samples_per_second =       3.849\n",
      "  train_steps_per_second   =       0.241\n"
     ]
    }
   ],
   "source": [
    "trainer.save_model()\n",
    "trainer.log_metrics(\"train\", train_results.metrics)\n",
    "trainer.save_metrics(\"train\", train_results.metrics)\n",
    "trainer.save_state()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Error Analysis"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define Method to Apply to Validation Dataset (& Then Apply it)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def forward_pass_with_label(batch):\n",
    "    # Convert dict of lists to list of dicts suitable for data collator\n",
    "    features = [dict(zip(batch, t)) for t in zip(*batch.values())]\n",
    "    # Pad inputs and labels and put all tensors on device\n",
    "    batch = data_collator(features)\n",
    "    input_ids = batch[\"input_ids\"].to(DEVICE)\n",
    "    attention_mask = batch[\"attention_mask\"].to(DEVICE)\n",
    "    labels = batch[\"labels\"].to(DEVICE)\n",
    "    with torch.no_grad():\n",
    "        # Pass data through model  \n",
    "        output = trainer.model(input_ids, \n",
    "                               attention_mask\n",
    "                               )\n",
    "        # Logit.size: [batch_size, sequence_length, classes]\n",
    "        predicted_label = torch.argmax(output.logits, \n",
    "                                       axis=-1\n",
    "                                       ).cpu().numpy()\n",
    "        \n",
    "    # Calculate loss per token after flattening batch dimension with view\n",
    "    loss = cross_entropy(output.logits.view(-1, 9), \n",
    "                         labels.view(-1), \n",
    "                         reduction=\"none\")\n",
    "    # Unflatten batch dimension and convert to numpy array\n",
    "    loss = loss.view(len(input_ids), -1).cpu().numpy()\n",
    "\n",
    "    return {\"loss\":loss, \"predicted_label\": predicted_label}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Apply Above Function to Entire Evaluation Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Parameter 'function'=<function forward_pass_with_label at 0x7fb1a416d700> of the transform datasets.arrow_dataset.Dataset._map_single couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.\n",
      "Loading cached processed dataset at /Users/briandunn/.cache/huggingface/datasets/Babelscape___parquet/Babelscape--wikineural-579d1dc98d2a6b93/0.0.0/14a00e99c0d15a23649d0db8944380ac81082d4b021f398733dd84f3a6c569a7/cache-1c80317fa3b1799d.arrow\n"
     ]
    }
   ],
   "source": [
    "eval_set = encoded_ds['eval']\n",
    "\n",
    "eval_set = eval_set.map(forward_pass_with_label,\n",
    "                        batched=True,\n",
    "                        batch_size=32)\n",
    "\n",
    "eval_df = eval_set.to_pandas()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Clean Up Padding Tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>input_ids</th>\n",
       "      <th>attention_mask</th>\n",
       "      <th>labels</th>\n",
       "      <th>loss</th>\n",
       "      <th>predicted_label</th>\n",
       "      <th>input_tokens</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[0, 1405, 825, 7, 9688, 6486, 257, 21, 847, 76...</td>\n",
       "      <td>[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...</td>\n",
       "      <td>[IGN, O, O, O, B-LOC, IGN, IGN, O, O, O, O, O,...</td>\n",
       "      <td>[0.0, 1.8596476e-05, 2.0861407e-05, 3.921909e-...</td>\n",
       "      <td>[O, O, O, O, B-LOC, I-LOC, I-LOC, O, O, O, O, ...</td>\n",
       "      <td>[&lt;s&gt;, ĠHer, Ġvisit, Ġto, ĠTu, val, u, Ġwas, Ġc...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           input_ids  \\\n",
       "0  [0, 1405, 825, 7, 9688, 6486, 257, 21, 847, 76...   \n",
       "\n",
       "                                      attention_mask  \\\n",
       "0  [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...   \n",
       "\n",
       "                                              labels  \\\n",
       "0  [IGN, O, O, O, B-LOC, IGN, IGN, O, O, O, O, O,...   \n",
       "\n",
       "                                                loss  \\\n",
       "0  [0.0, 1.8596476e-05, 2.0861407e-05, 3.921909e-...   \n",
       "\n",
       "                                     predicted_label  \\\n",
       "0  [O, O, O, O, B-LOC, I-LOC, I-LOC, O, O, O, O, ...   \n",
       "\n",
       "                                        input_tokens  \n",
       "0  [<s>, ĠHer, Ġvisit, Ġto, ĠTu, val, u, Ġwas, Ġc...  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "id2label[-100] = \"IGN\"\n",
    "eval_df[\"input_tokens\"] = eval_df[\"input_ids\"].apply(\n",
    "    lambda x: tokenizer.convert_ids_to_tokens(x))\n",
    "eval_df[\"predicted_label\"] = eval_df[\"predicted_label\"].apply(\n",
    "    lambda x: [id2label[i] for i in x])\n",
    "eval_df[\"labels\"] = eval_df[\"labels\"].apply(\n",
    "    lambda x: [id2label[i] for i in x])\n",
    "eval_df['loss'] = eval_df.apply(\n",
    "    lambda x: x['loss'][:len(x['input_ids'])], axis=1)\n",
    "eval_df['predicted_label'] = eval_df.apply(\n",
    "    lambda x: x['predicted_label'][:len(x['input_ids'])], axis=1)\n",
    "eval_df.head(1)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Unwrap Each Token Within Sample Separately"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>input_ids</th>\n",
       "      <th>attention_mask</th>\n",
       "      <th>labels</th>\n",
       "      <th>loss</th>\n",
       "      <th>predicted_label</th>\n",
       "      <th>input_tokens</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1405</td>\n",
       "      <td>1</td>\n",
       "      <td>O</td>\n",
       "      <td>0.0</td>\n",
       "      <td>O</td>\n",
       "      <td>ĠHer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>825</td>\n",
       "      <td>1</td>\n",
       "      <td>O</td>\n",
       "      <td>0.0</td>\n",
       "      <td>O</td>\n",
       "      <td>Ġvisit</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7</td>\n",
       "      <td>1</td>\n",
       "      <td>O</td>\n",
       "      <td>0.0</td>\n",
       "      <td>O</td>\n",
       "      <td>Ġto</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>9688</td>\n",
       "      <td>1</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>0.0</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>ĠTu</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>21</td>\n",
       "      <td>1</td>\n",
       "      <td>O</td>\n",
       "      <td>0.0</td>\n",
       "      <td>O</td>\n",
       "      <td>Ġwas</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>847</td>\n",
       "      <td>1</td>\n",
       "      <td>O</td>\n",
       "      <td>0.0</td>\n",
       "      <td>O</td>\n",
       "      <td>Ġcut</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>765</td>\n",
       "      <td>1</td>\n",
       "      <td>O</td>\n",
       "      <td>0.0</td>\n",
       "      <td>O</td>\n",
       "      <td>Ġshort</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  input_ids attention_mask labels  loss predicted_label input_tokens\n",
       "0      1405              1      O   0.0               O         ĠHer\n",
       "0       825              1      O   0.0               O       Ġvisit\n",
       "0         7              1      O   0.0               O          Ġto\n",
       "0      9688              1  B-LOC   0.0           B-LOC          ĠTu\n",
       "0        21              1      O   0.0               O         Ġwas\n",
       "0       847              1      O   0.0               O         Ġcut\n",
       "0       765              1      O   0.0               O       Ġshort"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "eval_df_tokens = eval_df.apply(pd.Series.explode)\n",
    "eval_df_tokens = eval_df_tokens.query(\"labels != 'IGN'\")\n",
    "eval_df_tokens[\"loss\"] = eval_df_tokens[\"loss\"].astype(float).round(2)\n",
    "eval_df_tokens.head(7)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### See Which Tokens Have Accumulated Most Loss in Evaluation Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>input_tokens</th>\n",
       "      <td>ĠBatman</td>\n",
       "      <td>ĠThe</td>\n",
       "      <td>ĠD</td>\n",
       "      <td>ĠAssy</td>\n",
       "      <td>ĠâĢĵ</td>\n",
       "      <td>ĠB</td>\n",
       "      <td>Ġof</td>\n",
       "      <td>ĠV</td>\n",
       "      <td>ĠC</td>\n",
       "      <td>ĠLatino</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>104</td>\n",
       "      <td>2491</td>\n",
       "      <td>219</td>\n",
       "      <td>86</td>\n",
       "      <td>486</td>\n",
       "      <td>198</td>\n",
       "      <td>7358</td>\n",
       "      <td>118</td>\n",
       "      <td>146</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>1.888</td>\n",
       "      <td>0.056</td>\n",
       "      <td>0.53</td>\n",
       "      <td>1.322</td>\n",
       "      <td>0.21</td>\n",
       "      <td>0.446</td>\n",
       "      <td>0.012</td>\n",
       "      <td>0.608</td>\n",
       "      <td>0.485</td>\n",
       "      <td>6.236</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sum</th>\n",
       "      <td>196.39</td>\n",
       "      <td>139.38</td>\n",
       "      <td>116.06</td>\n",
       "      <td>113.72</td>\n",
       "      <td>102.01</td>\n",
       "      <td>88.21</td>\n",
       "      <td>87.72</td>\n",
       "      <td>71.78</td>\n",
       "      <td>70.74</td>\n",
       "      <td>62.36</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    0       1       2       3       4      5      6      7  \\\n",
       "input_tokens  ĠBatman    ĠThe      ĠD   ĠAssy    ĠâĢĵ     ĠB    Ġof     ĠV   \n",
       "count             104    2491     219      86     486    198   7358    118   \n",
       "mean            1.888   0.056    0.53   1.322    0.21  0.446  0.012  0.608   \n",
       "sum            196.39  139.38  116.06  113.72  102.01  88.21  87.72  71.78   \n",
       "\n",
       "                  8        9  \n",
       "input_tokens     ĠC  ĠLatino  \n",
       "count           146       10  \n",
       "mean          0.485    6.236  \n",
       "sum           70.74    62.36  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(\n",
    "    eval_df_tokens.groupby(\"input_tokens\")[[\"loss\"]]\n",
    "    .agg([\"count\", \"mean\", \"sum\"])\n",
    "    .droplevel(level=0, axis=1)  # Get rid of multi-level columns\n",
    "    .sort_values(by=\"sum\", ascending=False)\n",
    "    .reset_index()\n",
    "    .round(3)\n",
    "    .head(10)\n",
    "    .T\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### See Which Label IDs Have Most Loss in Evaluation Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>labels</th>\n",
       "      <td>B-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>B-ORG</td>\n",
       "      <td>B-PER</td>\n",
       "      <td>I-LOC</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>I-ORG</td>\n",
       "      <td>I-PER</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>4173</td>\n",
       "      <td>4555</td>\n",
       "      <td>3512</td>\n",
       "      <td>5593</td>\n",
       "      <td>2370</td>\n",
       "      <td>5904</td>\n",
       "      <td>2963</td>\n",
       "      <td>4107</td>\n",
       "      <td>234662</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.285</td>\n",
       "      <td>0.273</td>\n",
       "      <td>0.264</td>\n",
       "      <td>0.188</td>\n",
       "      <td>0.164</td>\n",
       "      <td>0.157</td>\n",
       "      <td>0.15</td>\n",
       "      <td>0.034</td>\n",
       "      <td>0.012</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sum</th>\n",
       "      <td>1189.15</td>\n",
       "      <td>1244.76</td>\n",
       "      <td>926.57</td>\n",
       "      <td>1051.03</td>\n",
       "      <td>387.71</td>\n",
       "      <td>927.17</td>\n",
       "      <td>444.84</td>\n",
       "      <td>139.22</td>\n",
       "      <td>2786.6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              0        1       2        3       4       5       6       7  \\\n",
       "labels   B-MISC   I-MISC   B-ORG    B-PER   I-LOC   B-LOC   I-ORG   I-PER   \n",
       "count      4173     4555    3512     5593    2370    5904    2963    4107   \n",
       "mean      0.285    0.273   0.264    0.188   0.164   0.157    0.15   0.034   \n",
       "sum     1189.15  1244.76  926.57  1051.03  387.71  927.17  444.84  139.22   \n",
       "\n",
       "             8  \n",
       "labels       O  \n",
       "count   234662  \n",
       "mean     0.012  \n",
       "sum     2786.6  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(\n",
    "    eval_df_tokens.groupby(\"labels\")[[\"loss\"]] \n",
    "    .agg([\"count\", \"mean\", \"sum\"])\n",
    "    .droplevel(level=0, axis=1)\n",
    "    .sort_values(by=\"mean\", ascending=False)\n",
    "    .reset_index()\n",
    "    .round(3)\n",
    "    .fillna(0)\n",
    "    .T\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create Function to Display Confusion Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay\n",
    "\n",
    "def plot_confusion_matrix(y_preds, y_true, labels):\n",
    "    cm = confusion_matrix(y_true, y_preds, normalize=\"true\")\n",
    "    fig, ax = plt.subplots(figsize=(10, 10))\n",
    "    disp = ConfusionMatrixDisplay(confusion_matrix=cm, display_labels=labels)\n",
    "    disp.plot(cmap=\"Blues\", values_format=\".2f\", ax=ax, colorbar=False)\n",
    "    plt.title(\"Normalized Confusion Matrix\")\n",
    "    plt.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Display Confusion Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA2QAAANXCAYAAABaBpzLAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAADGe0lEQVR4nOzdd1xV9R/H8fdFBAdDQQhFEnGA2zJXpahpbrO03DMtR9pypqXmtmnlNvfIUW6z3NrPcqSomTPFrSgIOFHk/v5AL164mNa9HILX8/G4j+Lc7zn3+/345d7zvmdgMpvNZgEAAAAA0pyT0R0AAAAAgMyKQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgBwqGrVqqlatWqWn8PDw2UymTRjxow07Uf79u0VGBiYpq/5T82ePVshISHKmjWrcuXKZfftDx48WCaTye7b/a8yak4CgEQgAwDDzZgxQyaTSdmyZdPZs2dTPF+tWjWVLFnSgJ5lbkuWLFHdunWVJ08eubi4KF++fHrttde0YcMGh77uoUOH1L59exUqVEhTpkzR5MmTHfp6ac1kMslkMqlTp042nx8wYIClzeXLlx97+6tXr9bgwYP/ZS8BIO0QyAAgnYiLi9OoUaOM7obDFShQQDdv3lSbNm2M7opNZrNZHTp00CuvvKKLFy/qvffe08SJE9W9e3cdP35cL7zwgrZt2+aw19+0aZMSEhI0duxYtW/fXq+99prdX2PgwIG6efOm3bf7qLJly6bvv/9et2/fTvHc/PnzlS1btn+87dWrV2vIkCGPtU56n5MAMjYCGQCkE2XLltWUKVN07tw5h72G2Ww2dEdckuVoYJYsWQztR2o+++wzzZgxQ++8845+//13ffDBB+rYsaMGDBigXbt2adasWXJ2dnbY60dEREiSQ05VvM/Z2flfhZ5/q06dOoqNjdWPP/5otXzbtm06ceKE6tevnyb9iI+P1+3bt9P9nASQsRHIACCd+OCDD3T37t1HOkoWHx+voUOHqlChQnJ1dVVgYKA++OADxcXFWbULDAxUgwYN9NNPP+mZZ55R9uzZNWnSJG3atEkmk0kLFy7UkCFD5O/vL3d3dzVt2lQxMTGKi4vTO++8I19fX7m5ualDhw4ptj19+nTVqFFDvr6+cnV1VfHixTVhwoS/7Xvy63Xu98XWI/k1Xz/++KOqVKminDlzyt3dXfXr19eBAwdSvMbSpUtVsmRJZcuWTSVLltSSJUv+tl+SdPPmTY0cOVIhISH69NNPbV5n1aZNG1WoUMHy8/Hjx/Xqq6/Ky8tLOXLkUKVKlbRq1SqrdR6s9/Dhw5U/f35ly5ZNL7zwgo4dO2ZpFxgYqEGDBkmSfHx8ZDKZLKffPfj/DwoMDFT79u0tP9+5c0dDhgxRkSJFlC1bNnl7e+v555/X2rVrLW1sXUP2uHPql19+UYUKFZQtWzYFBQVp1qxZDy/uA/z9/VW1alXNmzfPavncuXNVqlQpm6fobt26Va+++qqefPJJubq6KiAgQO+++67VFwzt27fXuHHjLPW6/5CS5t2nn36qL7/80jLOP//8M8WcjIiIkI+Pj6pVqyaz2WzZ/rFjx5QzZ041a9bskccKAH/HcV/xAQAeS8GCBdW2bVtNmTJF/fr1U758+VJt26lTJ82cOVNNmzbV+++/r+3bt2vkyJE6ePBgivBx+PBhtWjRQm+++aY6d+6s4OBgy3MjR45U9uzZ1a9fPx07dkxff/21smbNKicnJ125ckWDBw/Wb7/9phkzZqhgwYL66KOPLOtOmDBBJUqUUKNGjeTs7KwVK1aoW7duSkhIUPfu3R953MWKFdPs2bOtlkVHR+u9996Tr6+vZdns2bPVrl071a5dW6NHj9aNGzc0YcIEPf/889qzZ48lvP38889q0qSJihcvrpEjRyoyMlIdOnRQ/vz5/7Yvv/zyi6KiovTOO+880tGSixcv6tlnn9WNGzfUs2dPeXt7a+bMmWrUqJEWL16sl19+2ar9qFGj5OTkpF69eikmJkZjxoxRq1attH37dknSl19+qVmzZmnJkiWaMGGC3NzcVLp06b/tx4MGDx6skSNHqlOnTqpQoYJiY2O1a9cu7d69W7Vq1Up1vceZU8eOHVPTpk31+uuvq127dpo2bZrat2+vcuXKqUSJEo/Uz5YtW+rtt9/WtWvX5Obmpvj4eC1atEjvvfeebt26laL9okWLdOPGDXXt2lXe3t7asWOHvv76a505c0aLFi2SJL355ps6d+6c1q5dm2JO3Td9+nTdunVLb7zxhlxdXeXl5aWEhASrNr6+vpowYYJeffVVff311+rZs6cSEhLUvn17ubu7a/z48Y80RgB4JGYAgKGmT59ulmTeuXOn+a+//jI7Ozube/bsaXk+NDTUXKJECcvPYWFhZknmTp06WW2nV69eZknmDRs2WJYVKFDALMm8Zs0aq7YbN240SzKXLFnSfPv2bcvyFi1amE0mk7lu3bpW7StXrmwuUKCA1bIbN26kGEvt2rXNQUFBVstCQ0PNoaGhlp9PnDhhlmSePn26zXokJCSYGzRoYHZzczMfOHDAbDabzVevXjXnypXL3LlzZ6u2Fy5cMHt6elotL1u2rDlv3rzm6Ohoy7Kff/7ZLCnFGJIbO3asWZJ5yZIlD2133zvvvGOWZN66datl2dWrV80FCxY0BwYGmu/evWs2m5PqXaxYMXNcXFyK19u/f79l2aBBg8ySzJcuXbJ6LUnmQYMGpehDgQIFzO3atbP8XKZMGXP9+vUf2u/7r3HfP5lTW7ZssSyLiIgwu7q6mt9///2Hvu79cXTv3t0cFRVldnFxMc+ePdtsNpvNq1atMptMJnN4eLjNGtiabyNHjjSbTCbzyZMnLcu6d+9utrV7c3/eeXh4mCMiImw+l3xOtmjRwpwjRw7zkSNHzJ988olZknnp0qV/O0YAeBycsggA6UhQUJDatGmjyZMn6/z58zbbrF69WpL03nvvWS1///33JSnF6XIFCxZU7dq1bW6rbdu2ypo1q+XnihUrymw2q2PHjlbtKlasqNOnTys+Pt6yLHv27Jb/j4mJ0eXLlxUaGqrjx48rJibm74aaqqFDh2rlypWaMWOGihcvLklau3atoqOj1aJFC12+fNnyyJIliypWrKiNGzdKks6fP6+wsDC1a9dOnp6elm3WqlXLsq2HiY2NlSS5u7s/Ul9Xr16tChUq6Pnnn7csc3Nz0xtvvKHw8HD9+eefVu07dOggFxcXy89VqlSRlHjao73kypVLBw4c0NGjRx95ncedU8WLF7f0XUo8vTI4OPixxpE7d27VqVNH8+fPlyTNmzdPzz77rAoUKGCz/YPz7fr167p8+bKeffZZmc1m7dmz55Fft0mTJvLx8Xmktt988408PT3VtGlTffjhh2rTpo1eeumlR34tAHgUBDIASGcGDhyo+Pj4VK8lO3nypJycnFS4cGGr5X5+fsqVK5dOnjxptbxgwYKpvtaTTz5p9fP9EBMQEJBieUJCglXQ+t///qeaNWsqZ86cypUrl3x8fPTBBx9I0j8OZGvWrNGQIUPUv39/NWnSxLL8frioUaOGfHx8rB4///yz5UYY98depEiRFNt+8FTN1Hh4eEiSrl69+kj9PXnypM3tFitWzKo/9yWvd+7cuSVJV65ceaTXexQff/yxoqOjVbRoUZUqVUq9e/fWvn37HrrO486p5OOQEsfyuONo2bKl1q5dq1OnTmnp0qVq2bJlqm1PnTql9u3by8vLS25ubvLx8VFoaKikx5tvD/t9SM7Ly0tfffWV9u3bJ09PT3311VePvC4APCquIQOAdCYoKEitW7fW5MmT1a9fv1TbPeof9n3wyEJyqV0nldpy870bHPz111964YUXFBISos8//1wBAQFycXHR6tWr9cUXX6S4JudRnDhxQq1atVKtWrU0bNgwq+fub2/27Nny8/NLsa697noYEhIiSdq/f78aN25sl20+6O/q+k/cvXvX6ueqVavqr7/+0rJly/Tzzz9r6tSp+uKLLzRx4sRU//bXfY86p+w1jkaNGsnV1VXt2rVTXFxcqrf4v3v3rmrVqqWoqCj17dtXISEhypkzp86ePav27ds/1nx72O+DLT/99JOkxNB85swZh979EkDmRCADgHRo4MCBmjNnjkaPHp3iuQIFCighIUFHjx61HImREm8wER0dneopX/a0YsUKxcXFafny5VZHS+6fOvi4bt68qVdeeUW5cuXS/Pnz5eRkfQJHoUKFJCXebKFmzZqpbuf+2G2drnf48OG/7cfzzz+v3Llza/78+frggw/+9sYeBQoUsLndQ4cOWfXHHnLnzq3o6GirZbdv37Z5aquXl5c6dOigDh066Nq1a6pataoGDx6caiAzak5lz55djRs31pw5cyx/hNuW/fv368iRI5o5c6batm1rWf7gnSPve9RQ+SjWrFmjqVOnqk+fPpo7d67atWun7du3O/TPHgDIfDhlEQDSoUKFCql169aaNGmSLly4YPVcvXr1JCXeke9Bn3/+uSSlyd9wuh9UHjwiEhMTo+nTp/+j7XXp0kVHjhzRkiVLLKfxPah27dry8PDQiBEjdOfOnRTPX7p0SZKUN29elS1bVjNnzrQ6jW3t2rUprueyJUeOHOrbt68OHjyovn372jziM2fOHO3YsUNS4r/Fjh079Ouvv1qev379uiZPnqzAwMBHum7tURUqVEhbtmyxWjZ58uQUR8giIyOtfnZzc1PhwoVT3L7+QUbOqV69emnQoEH68MMPU21ja76ZzWaNHTs2RducOXNKUorw+riio6Mtd6ocMWKEpk6dqt27d2vEiBH/arsAkBxf8QBAOjVgwADNnj1bhw8ftrqVeJkyZdSuXTtNnjxZ0dHRCg0N1Y4dOzRz5kw1btxY1atXd3jfXnzxRbm4uKhhw4Z68803de3aNU2ZMkW+vr6p3owkNatWrdKsWbPUpEkT7du3z+p6Jzc3NzVu3FgeHh6aMGGC2rRpo6efflrNmzeXj4+PTp06pVWrVum5557TN998IynxVv7169fX888/r44dOyoqKkpff/21SpQooWvXrv1tf3r37q0DBw7os88+08aNG9W0aVP5+fnpwoULWrp0qXbs2KFt27ZJkvr166f58+erbt266tmzp7y8vDRz5kydOHFC33//fYojff9Gp06d1KVLFzVp0kS1atXS3r179dNPP6U4qlS8eHFVq1ZN5cqVk5eXl3bt2qXFixfrrbfeSnXbRs6pMmXKqEyZMg9tExISokKFCqlXr146e/asPDw89P3339u8Zq1cuXKSpJ49e6p27drKkiWLmjdv/tj9evvttxUZGal169YpS5YsqlOnjjp16qRhw4bppZde+ts+A8CjIpABQDpVuHBhtW7dWjNnzkzx3NSpUxUUFKQZM2ZoyZIl8vPzU//+/S1/VNjRgoODtXjxYg0cOFC9evWSn5+funbtKh8fnxR3aPw7949uff/99/r++++tnitQoIDlWq6WLVsqX758GjVqlD755BPFxcXJ399fVapUUYcOHSzr1KlTR4sWLdLAgQPVv39/FSpUSNOnT9eyZcu0adOmv+2Pk5OTZs2apZdeekmTJ0/Wp59+qtjYWPn4+Khq1aoaM2aMKleuLEl64okntG3bNvXt21dff/21bt26pdKlS2vFihV2P6rUuXNnnThxQt9++63WrFmjKlWqaO3atXrhhRes2vXs2VPLly/Xzz//rLi4OBUoUEDDhg1T7969H7p9o+fUw2TNmlUrVqxQz549NXLkSGXLlk0vv/yy3nrrrRTB6JVXXlGPHj303Xffac6cOTKbzY8dyJYvX65Zs2bps88+s1xXKCUeMVy7dq3atWunnTt3Wt2hFAD+KZP531xJDAAAAAD4x7iGDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACD8HfIHCghIUHnzp2Tu7u7TCaT0d0BAAAAkAbMZrOuXr2qfPnyycnp4cfACGQOdO7cOQUEBBjdDQAAAAAGOH36tPLnz//QNgQyB3J3d5ckuVR4VyZnV4N7k3GcWNbP6C5kSBzDxX+FkxOzFQDsKf5ugtFdyHCuXo1VSKECljzwMAQyB7p/mqLJ2VUm52wG9ybj8PDwMLoLGRK7uPivIJABgH0RyBznUS5b4qYeAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhECWgXV6qbz2zn1b538coLXfvK6ng/Ol2tY5i5N6t6mq3bN76PyPA7R18pt6oXyhVNu/0/w5XVk/SCO61XZE19Otbxdt0VONB8m/yrt6seOn2n0g/KHtl63fo0qvDZV/lXdVpeUIrf3fAavnV24MU9Me41SkVl/lqdhD+4+ccWDv06+pi7aobONBylflXdXq+Kl+f4S6VnxtqPJVeVfP26jrio1hatJjnArX6ivvTFpXauoYUxZuVulGH8nvuXdUs/0nf1vXpet2q0LTofJ77h0923y4fk5WV7PZrBETVyqkzgfK+/y7atzta/11KsKBI0h/qKljUFfHoK729+3iLXq68WDlr/qeanf8TLsPnHxo+2Xr96hys2HKX/U9VW01Umu3JdX0TvxdffzNMlVtNVIFqvVSyQYD1X3IbF24FOPoYfwrBLIM6uVqJTSsy4saPWuzqnWZpD/+uqjvR7dWnlw5bLYf2LGG2jcop75f/6hKHcdp+orfNXtIM5Uq7Jei7VPB+dS+QTn98dcFRw8jXVmy9nd9OHaJer9eVxtm9lGJwv569e3xuhR11Wb7HfuO640PZ6hVw8raOKuv6lUtrbZ9pujgX+csbW7cvK2KZYL00VsvpdUw0p3kdS35CHXt/OEMtX6grm1s1LVSmSANyqR1paaO8cPPv2vgl0vUt1NdbZrdVyWL+KtJj3Gp1nX73uPqNHCGWr9UWZvn9FP90DJq3Wuy/jyWVNexs9Zp0oLN+rx/c62d3ks5sruoSY9xuhV3J62GZShq6hjU1TGoq/0tWbtbH41dol6d6mj9zN4qUcRfr73z8M+rNz+aqVYNK2vDzD6qW7W02vWZavm8unnrtvYdPqP3OtTW+pm9NWPU6zp2MkKte09Oy2E9tv9cIGvfvr0aN26c6vPVqlWTyWSSyWRStmzZVLx4cY0fP97y/IwZMyzPP/jIli2b1WvcX541a1YVLFhQffr00a1btxw5NLvq1rSSZq3erXk/henwyct678uVuhF3R63rPGWz/Ws1S+uLeb9o7Y5jOnk+WtNW7NLa7Uf11quVrdrlzJZVkz94RW9/vkLRV/879bCHCfM3qs1LldWyYSUFB+XVZ/2aKXs2F81b8avN9pMWbFKNSsXUo01NFS3op/5dGqh0cICmLtpiafNavQrq3amuQssHp9Uw0p3x9+raqmElhTxQ17kPqesL9+oaXNBPH9ioa7NMXldq6hjj521Q28bPqlWjygoJyqvP+zdXjmwumrM8lbp+t0kvVC6mnvfqOqBrA5UJCdCURZslJX4zPnH+RvXqWFv1QkurZBF/TRjSVhcux2jV5r1pOTTDUFPHoK6OQV3tb+L8jWr90rNq2aCSggvm1ad9X0vct1r5m832kxdsVo1KxfRW6xcS963erK/Swfn17eKtkiQPt+xa/HV3Na75tAoXeELPlCyoUb2aau+h0zpzISoth/ZY/nOB7FF07txZ58+f159//qnXXntN3bt31/z58y3Pe3h46Pz581aPkyetD4/WqVNH58+f1/Hjx/XFF19o0qRJGjRoUFoP5R/J6uykskXzadPu45ZlZrO0efdxlS+e3+Y6ri5ZdOt2vNWyW7fjVankk1bLPnm7nn7+7ag27z5h/46nY7fvxGvvodMKrZC0M+rk5KTQ8sHauT/c5jq79oen2HmtXilEu/Znrto9zD+p604bda1RKUQ7qaskauoot+/EK+zQaVVLXtcKwanWacf+E6pWPsRqWY1KxSz/DifPRupiZKyqVUhq4+mWXeVKBGrnvnC7jyG9oaaOQV0dg7ra3+078dp7+LTV54+Tk5Oqlg9OdV9p1x/hqlq+qNWy6pWKPXTfKvbaLZlMJnm6Z7dPxx0gQwayHDlyyM/PT0FBQRo8eLCKFCmi5cuXW543mUzy8/OzejzxxBNW23B1dZWfn58CAgLUuHFj1axZU2vXrk3rofwj3p455JzFSZeuXLdafunKdfl6udlcZ8POv9StaSUF+XvJZJKqlQtSg+eL6YkH2r9SvYTKFM6rj6euc2j/06PI6Ou6ezdBPl4eVst9vNwVERVrc52IyFj5eLlbLfP1cldEpO3D8JnR/br6Jqur72PW1Ye6WlBTx4iMvnbvPSB5nTwUEfmQunrbqmti+4v3/pu8ja+3e6rbzEioqWNQV8egrvYXZdm3Sjb+3Kl//kRExqb4fPN5SPtbcXf08bhleqXW03LPmX4DmbPRHUgL2bNn1+3bt//x+n/88Ye2bdumAgUKPLRdXFyc4uLiLD/Hxv53fpn6jVujse831I7p3WWWdOJclOb9FKZWdcpKkvx9PDSyex290me24u7cNbSvAAAAwMPcib+rTgOmy2yWPun7mtHdeagMHcju3r2r+fPna9++fXrjjTcsy2NiYuTmZn2kqEqVKvrxxx8tP69cuVJubm6Kj49XXFycnJyc9M033zz09UaOHKkhQ4bYdxD/QGTMDcXfTZBP7pxWy31y51RE1LVU12n90QK5Zs0iL88cOn/5qgZ3rqnw81ckSWWK5pVvbjdtmvimZR3nLE56tnQBdW5cQU/UGaaEBLPjBmUw71w5lSWLky4lO8JwKepqim9q7vP19khxUWpE1FX5JvsmLDO7X9fkR24iHrOul6irBTV1DO9cbvfeA5LXKVa+3g+pa6Stuia2f+Lefy9FXpVfHk9Lm4jIqypV1Pbp5RkJNXUM6uoY1NX+vCz7Vsn2la6k/vnj6+2R4vPtko3298PYmQtR+mFcj3R9dEz6D5+yOHfuXLm5uVkeW7dutTw3fvx4ubm5KXv27OrcubPeffddde3a1fK8u7u7wsLCrB5Tp0612n716tUVFham7du3q127durQoYOaNGny0D71799fMTExlsfp06ftO+hHdCc+QWFHzin0qSDLMpNJqvpUkHb++fBbVcfduavzl6/KOYuTGlYpph+3HZYkbdl9Qs++Pl5V35hoeew+dFaL1u9T1TcmZugwJkkuWZ1VJiRAW3YesSxLSEjQlp1HVL5UoM11nikVqC27jlgt27zjsJ4pVdCRXf1P+Sd1LW+jrpt2HFZ56iqJmjqKS1ZnlQ0J0Oadhy3Lkupqu04VShW0ai9JG7cfsvw7FPD31hPeHlZtYq/d1O8HwlW+dKDdx5DeUFPHoK6OQV3tzyWrs8oEp/y82roz9X2lZ0oGauvO5PtWh6za3w9jx09f0uKvu8vLM2fyzaQ7/9kjZI0aNVLFihUtP/v7+1v+v1WrVhowYICyZ8+uvHnzysnJOnc6OTmpcOHCD91+zpw5LW2mTZumMmXK6Ntvv9Xrr7+e6jqurq5ydXX9J8Oxu/GLf9P4vo2158g57T50Vl2bVFLObFk196cwSdKEvo11/vJVffzteklSuRB/5c3jrv1/XVC+PB7q2zZUTiaTxn73P0nStZu3dTD8ktVr3Lh1R1GxN1Msz6i6tqiutz6eo7LFntTTxQto4nebdONWnFo0qCRJ6jZ4lvL65NKH3RtJkt5sVk2NuozVuLnr9eJzJfTD2t0KO3hKn/dvbtnmlZjrOnPxiuXvYxw7eVFS4jdAT6TyjVtG061FdXV/oK6T7tW15b26dr1X148eqGvDe3Wt9VwJLblX1y+oqwU1dYxuLWuo25DZeqrYk3q6RKAmzN+o6zfj1KphYl27DJqlvD6elj8N8Gbzamrw5pf6Zs56vfh8Cf3w8+8KO3hKX37QQlLi9cxdWlTXp9PWKCjARwX8vTVi4ir55fFU/dAyho0zLVFTx6CujkFd7a9Li+rqMXSOyhYLSPy8WrBJN27dVov6ifv43YfMlp+Ppz7slvh59UazUL3U9SuNn7vh3ufV7wo7eFqf9Uv8vLoTf1cd+3+rfYfPaO5nb+pugtlyrV5ujxxyyZo+o0/67NUjcHd3l7u77cOZnp6efxu4HoeTk5M++OADvffee2rZsqWyZ0/fhz0lacmmA8rjmUMftK8m39xu2v/XBTXtN9dyo4/8vp5KMCcd1XJ1cdaAjjUUmDe3rt+8rbXbj6rLqCWKvR6X2ktkOi/XKqfI6GsaNXmVIiKvqmRRfy38spvl1IMzF6/IyclkaV+hdJAmDW2vERNXaviElQoK8NGsMZ1VrFDSH+hes3W/egyda/m588AZkqTeneqqb+d6aTMwg71cq5wuP6SuZ23UdfLQ9ho+caWG3avr7GR1/TFZXTvdq2ufTFJXauoYr7yYWNcRk1bdO6XIX4u/6p70HnAhSk6mpLpWLBOkKcPaa/iElRo6foWCAnw059M3VLxwUl3fbltTN27G6d0R8xVz7aYqlSmkxV91UzbXrGk+PiNQU8egro5BXe3v5VpPKzL6mkZPWa2IyFiVLJJfC77o+kBNr8hksv68mvhxO42ctErDJ65QUICvZo7pZPm8Oh8RrTVb/5AkVW8z2uq1lo7roefKFUmjkT0ek9ls/k+da9a+fXtFR0dr6dKlNp+vVq2aypYtqy+//NLm8zNmzNDbb7+tw4cPp3jO19dXTk5ONl8jPj5egYGBeuedd9SrV69H6mtsbKw8PT3l+mw/mZyz/f0KeCSX135kdBcyJNPfNwHShQfDJADg34u/m2B0FzKc2NhY+fvmVkxMjDw8Hn4myX/2GrJ/IzY2Vnnz5k3xiIiISHUdZ2dnvfXWWxozZoyuX7+eajsAAAAAeFT/uSNk/yUcIXMMjpA5Bscc8F/BETIAsC+OkNkfR8gAAAAA4D+AQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEGcje5AZnBsSV95eHgY3Y0Mo0jPJUZ3IUM6Mrax0V3IkOITzEZ3IcPJ5pTF6C5kSDfi4o3uQoaTw5XdLPw3XI+7a3QXMpwbj1FTjpABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEGcje4AHGfa4i0aP3eDIqJiVbywv0a811RPlyiQavvl6/do9ORVOn0hSgXz++jD7o1U89kSluc/mbpaS9fu1tmIaLlkzaLSwQHq36WBypUITIPRpA9tqgap8wtF5OORTQfPxmjwor3ad/KKzbbz3q6iSkV8Uizf+McFvT5xmyQpj7ur+rxUUlWK+coje1btOBapIYvCFH7pukPHkd58u3iLxs1JnKslCvtr5PsPn6vL1u/RqMmrdPp8lIICEudqrXtz9U78XY2cuFLrfv1TJ89Gyt0tm0LLB+vDbo3k5+OZVkMy3PTvt2r83A26dO/3f/h7TfRU8dRrumLDHo2evFpn7v3+D+zWUC888Pv/oD5jFmj20m0a8vbLeqNZNQeNIH2asnCzvp6zXhGRsSpZxF+je7/60PfApet2a8TEVTp1PlJBAT4a3KOxXnwuqa5ms1kjJ63SrKXbFHPtpiqWDtJn/Zqp0JO+aTCa9GHG91s1Yf4GXYq6quKF8mnou383V8P0ydSkufpB14Z6oXJxy/PvDJ+rRT/utFqnWoUQzf28i8PGkB4xVx2DutrfrCW/aNJ3ie8BxQrl05C3X1HZYqm/B6zaGKbPpv2Y+B7g76N+XRqoeqWk94DA0Hdtrte/S0O92aKG3ftvD4YeIWvfvr1MJpPl4e3trTp16mjfvn2prhMeHi6TyaQsWbLo7NmzVs+dP39ezs7OMplMCg8Pt2ofFhZmabdkyRJVqlRJnp6ecnd3V4kSJfTOO+9Ybev27dsaM2aMypQpoxw5cihPnjx67rnnNH36dN25c8deJXCYpet2a9BXS/T+63W0dkZvlSjir+bvjtelqKs22+/cd1xdBs1Uy4aVtW5mH9WtWlrt+07Vwb/OWdoEBfhqxPuvatOcflo+8R0F5PVSs7fH6/IV29vMaOo/7a8PXi6lr348pIajN+jg2RjN7P6cvN1cbbbvOuU3Vei/yvKoPWyt4u8maPWeM5Y2E9+opCfz5NSbk35Tg1EbdDbqhmb3qKLsLlnSaliGW7J2tz4au0S9OtXR+pmJc/W1d1Kfqzv2HdebH81Uq4aVteHeXG3XJ2mu3rx1W/sOn9F7HWpr/czemjHqdR07GaHWvSen5bAMtWzdbg3+aone71hbP03vreKF86nFuxN0ObXf//0n1HXQLLVsWEk/z+itOlVLqUO/b3Xogd//+1Zv3qvdB07KL0/mCbf3/fDz7xr45RL17VRXm2b3Vcki/mrSY1yqc3X73uPqNHCGWr9UWZvn9FP90DJq3Wuy/jyWVNexs9Zp0oLN+rx/c62d3ks5sruoSY9xuhWX/j9n7GHZ+t0a8s1SvdehjtZ820vFC/ur1XsTU/1c2bn/hLoPmaUWDSrpp2m9VLtKKb3e/1sdOn7eql31iiHas+xjy2Pc4LZpMZx0g7nqGNTV/lZs2KNh45bq7Xa1tWrK+ypeKJ/a9pqU6nvA73+cUM+hs9WsXkWtntJLL1YpqTcGTNPhB94DdvwwxOoxpm9zmUwm1Q0tnVbDemyGn7JYp04dnT9/XufPn9f69evl7OysBg0a/O16/v7+mjVrltWymTNnyt/f/6HrrV+/Xs2aNVOTJk20Y8cO/f777xo+fLhVyLp9+7Zq166tUaNG6Y033tC2bdu0Y8cOde/eXV9//bUOHDjwzwabhibO36jWjZ5ViwaVFFwwrz7p85qyu7po/srfbLafvHCzqlcspu6tX1DRQD/1e7O+SgXn17TFWy1tmtR+RqEVghXon0chQXn18dsv6+r1W1ZvLBnZ6zWKaMG2cC3+7aSOXbiqgd/t0c3bd/VqZdvf4sTcuKPLV+Msj+dDfHXz9l2t3pP4RUJBXzc9XdBbH363R/tOXdGJiGv6cMEeuWZ1UsNyAWk5NENNnL9RrV96Vi3vzdVP+76m7NlcNC+1ubpgs2pUKqa3Wr+gogX91P/N+iodnF/f3purHm7Ztfjr7mpc82kVLvCEnilZUKN6NdXeQ6d15kJUWg7NMJO+26RWjZ5V8waVFFzQT2P+5vd/6sLNql4xRN1aJf7+933j3u//91ut2p2/FK2Bn3+vcYPayNk583xpcN/4eRvUtvGzatWoskKC8urz/s2VI5uL5iz/1Wb7Sd9t0guVi6lnm5oKLuinAV0bqExIgKYs2iwp8ZvxifM3qlfH2qoXWloli/hrwpC2unA5Rqs2703LoRlmyneb1LJhZTWrX1FFC/ppVO9XlT2bi75bud1m+28XbVa1iiHq2rKGigT6qU/neipZNL+mJ5urLi7O8vX2sDxyeeRIi+GkG8xVx6Cu9jd14SY1b1BZr9WrqCKBfhr+fuJ7wMLVtt8Dpi3eotAKIXqzRQ0VDnxC779eTyWK5tfMJUnvAQ/+7vt6e2jt//5Q5acK68l8edJqWI/N8EDm6uoqPz8/+fn5qWzZsurXr59Onz6tS5cuPXS9du3aafr06VbLpk+frnbt2j10vRUrVui5555T7969FRwcrKJFi6px48YaN26cpc2XX36pLVu2aP369erevbvKli2roKAgtWzZUtu3b1eRIkX++YDTwO078dp3+LSqlA+2LHNyclLV8sHa9ccJm+v8/ke4qpYvarWsesViqba/fSdes5duk4dbdpUo8vAQnBFkzWJSyYBc+t/hCMsys1n63+EIPVXQ65G28dqzgVq5+4xu3r4rSXJxTvz1i4tPsNrm7fgEPVPI2469T79u34nX3sOnFWprru63Pfd22ZqrlYql2l6SYq/dkslkkqd7dvt0PB2z/P4/k1QjJycnVSlfVL//EW5znV1/nLB6v5CkahVDrNonJCSox5A56tqyhoKD8jqi6+na7TvxCjt0WtUqWM/V0ArB2pnK3Nux/4SqlQ+xWlajUjHt3B8uSTp5NlIXI2NVrUJSG0+37CpXIlA794XbfQzpze078dp35EyKufr8M0X1+4Fwm+v8/ke4VXsp5VyVpF/3HFPpBgNVpcVw9ft0oaJiMs9p4MxVx6Cu9nf7Trz+OHJGz5Wzfg94rlwR7T5w0uY6ew6EW7WXpKrlg1Ntfynqqjb++qea1atov447gOGB7EHXrl3TnDlzVLhwYXl7P3yHtFGjRrpy5Yp++eUXSdIvv/yiK1euqGHDhg9dz8/PTwcOHNAff/yRapu5c+eqZs2aeuqpp1I8lzVrVuXMmdPmenFxcYqNjbV6GCEq+rru3k2Qj5e71XIfL3dFRNo+BBwRGSsfL4+/bf/zL3+oYI1eejL0fU36bpMWju0m71xu9h1AOpTbzVXOWZx0+Wqc1fLLsXHy8cj2t+uXLpBbwfk8tWBbuGXZXxeu6mzUDfVuVEIe2bMqaxaT3qxZVPly55Cv599vMyNIba765n74XPVNPlcf0v5W3B19PG6ZXqn1tNxzZvxA9tDf/1ROq7kUeVU+uZO1z+2uiMik97Bv5qxXlixO6vRaqP07/R8QGX0tlbp6WNXpQRGRsfLxtvU+nNj+4r3/Jm/j6+2e6jYzkqiYxLmax8ZcvZTK+C9FpZyreXK761JUUvvqFYtp7MDWWjC2mwZ0bajfwv5Sm16TdPduQvLNZUjMVcegrvZ35f57gI3Pnwd/px90KeqqzfaXU2n//Zodypkjm2pXTb+nK0rpIJCtXLlSbm5ucnNzk7u7u5YvX64FCxbIyenhXcuaNatat26tadOmSZKmTZum1q1bK2vWrA9dr0ePHipfvrxKlSqlwMBANW/eXNOmTVNcXNKO9tGjRxUSEvKQrdg2cuRIeXp6Wh4BARnvtLPnyhXRhpl9tXLyO6peqZg6D5ye6rnTSPJa5UAdOhtjdQOQ+ASzuk75TQV93RT2SUMd+PwlVSrqo00HLighwWxgbzOOO/F31WnAdJnN0id9XzO6O/9Zew+d1tSFmzV2YCuZTCajuwM81Es1n9aLz5dUsUL5VKdqac0c3VlhB09p255jRncNQBpb+OMONa75tLK5PjwfGM3wQFa9enWFhYUpLCxMO3bsUO3atVW3bl2dPHlSdevWtYS1EiVS3u2rY8eOWrRokS5cuKBFixapY8eOf/t6OXPm1KpVq3Ts2DENHDhQbm5uev/991WhQgXduHFDUuI5vf9E//79FRMTY3mcPn36H23n3/LKlVNZsjilCEqXoq7KN9m3MPf5enuk+DbCVvuc2V1VMMBHz5QsqC8HtJRzliyat8L2udMZyZVrcYq/m6A87tY38Mjj4apLsbceum52lyxqWC6/Fv4anuK5P05Hq8GoDSrTa7kqDVitDuP/p1w5XXQ6MnOcXpPaXI248vC5GpF8rtpofz+MnbkQpcVfd88UR8ekv/n997JdUx9vd11KdgF1Yk0Tj0Ru3/uXLl+5pmdeGaz8Vd5V/irv6syFKA35eqnKvzLEIeNIb7xzuaVS11hLnZLz9fbQpUhb78OJ7Z+499/kbSIir6a6zYzEyzNxria/2cylqKvySWX8Pl4p5+rlK1dTnOHxoAL+eeSVK6fCzzz8UoiMgrnqGNTV/nLffw+w8fmT2u+0j5e7zfZ5bLTfsfcvHT8VoWYNKtmv0w5ieCDLmTOnChcurMKFC6t8+fKaOnWqrl+/rilTpmjq1KmWsLZ69eoU65YqVUohISFq0aKFihUrppIlSz7y6xYqVEidOnXS1KlTtXv3bv35559asGCBJKlo0aI6dOjQY4/F1dVVHh4eVg8juGR1VungAG3ddcSyLCEhQVt3HdYzJQvaXKdcyUCr9pK0ecehVNtbtmtO0O078f++0+ncnbtm/XE6Ws8GJ92G1mSSni3qqz0nHn6jiHpP+cvF2UlLd6Ye0K/eilfUtdsK9MmpUk/m1tp951Ntm5G4ZHVWmeAAbdmZbK7uPKxnStmee8+UDNTWnTbm6gPt74ex46cvafHX3eXlafs044zo/u//L79b1/SXXUdUrmSgzXWeKVlQvyT7/d+y47ClfdM65bVhVh+tm9Hb8vDL46luLWto/heZ41biLlmdVTYkQJt3HrYsS0hI0JadR1Q+lblaoVRBq/aStHH7IZUvFShJKuDvrSe8PazaxF67qd8PhKt86UC7jyG9ccnqrNJF8+uX349aliUkJOiX34+kehvxciUD9cuuo1bLtuw8nOrclqRzEdG6EnNDT2SSO4MyVx2DutqfS1ZnlSyaX9uSfV5t23001T9981SJQKv2kvTLriM22y9YvV2lgvOreOH0f68DwwNZciaTSU5OTrp586b8/f0tYa1AAdv/MB07dtSmTZse6ehYagIDA5UjRw5dv554VKJly5Zat26d9uzZk6LtnTt3LO3Ssy4tqmvu8m1asGq7joRfUJ8xC3Xj1m01b5B4UeNbQ2Zr2PjllvZvvBaqjb8d1IR5G3Q0/KI+mbpaew+dVsemVSRJ12/GafiEFdr1xwmdPh+lvYdO6e1hc3XhUowa1kh5rV1G9O2Go2r+bKBeqfikCj3hrqHNnlIO1yxa/FvihaSftimn3o1SHsl9rXKgft53TtHXb6d4ru5T/qpYJI8CvHOoZqm8mvXW81q775x+ORSRom1G1aVFdc1Zvk3frdquIycuqPe9udqifuJc7T5ktoY+OFebhWrDbwc1fm7iXB0zZbXCDp7W6/fm6p34u+rY/1uFHTylCUPa6m6CWRcjY3UxMjZTfHkgSW82r6a5y3/VwtU7dCT8gvp+ssjq97/Hx3M0fMIKS/tO937/J977/f906o+Jv/9NEmvq5ZlTIYXyWT2cnbPIx9tDhQs8YcgYjdCtZQ3NWrpN81f+psMnLui9UQt0/WacWjVM/Pa1y6BZGvLNMkv7N5tX0/pf/9Q3c9brSPgFjZq8SmEHT6nzq4nX4ZlMJnVpUV2fTluj1Zv36cCxs+o6eLb88niqfmgZQ8aY1jo3r6Z5K37Vwh936Gj4BfX7dJFu3rytZvd+/3sOnaORE5Pm6uuvhmrT9oOaOH+jjp28qM++/VH7Dp1Wh3tz9fqNOA0dt0y//xGu0+cjtXXXEXXsN1WB/nkUWuHxL0X4r2KuOgZ1tb9Or1XT/FW/afGaHToWflEDPl+sGzdv69W6ie8B7w2fq9GTV1rad2xaVZt3HNKUBYnvAV9MX6P9h0+r3ctVrLZ79fotrd60V83qp/+jY1I6+MPQcXFxunDhgiTpypUr+uabb3Tt2rW/vTnHfZ07d9arr76qXLlyPVL7wYMH68aNG6pXr54KFCig6OhoffXVV7pz545q1aolSXrnnXe0atUqvfDCCxo6dKief/55ubu7a9euXRo9erS+/fZblS1b9p8MN800rvm0Iq9c05ipqxURGasSRfJr/hddLTdDOHvxipyckq4FKV86SBOGtNOoyas0YuIKFQzw1YzRnVSsUD5JUhYnJx07eVELV+9QVMw15fbMqbLFntSyCW8rJJPccW3V7rPycnPVu/WLK4+7qw6ejVH7cf+z3Ogjn1cOJb/0q6Cvm8oXzqO23/xic5u+Htk04JVSyuOeTZdib+mH7af0zZqDjh5KuvJyracVGX1No6esvveHNvNrwRddLadrnLlwxeq6pQqlgzTx43YaOWmVhk9coaAAX80ckzRXz0dEa83WxJv2VG8z2uq1lo7roefKpe+7pNrDSzUTazpmympdikr8/Z/3eRfLKSApfv9LFdT4IW01evJqjZy0UgXz+2j6qNcVcq+mSPTKi+V0OfqaRkxapYjIqypV1F+Lv+r+wFyNktMDc7VimSBNGdZewyes1NDxKxQU4KM5n76h4oWT6vp225q6cTNO746Yr5hrN1WpTCEt/qpbur/ewV5eeuFpRUVf16dTf0ycq4X9NeezNy03TjhnY65+M6itxkxZpdGTE+fqtyNft3wOOWUx6eBf57Tox52KvXZTT+TxUGj5EPXuXE+uLobv8qQZ5qpjUFf7a1jjKUVFX9MX09boUlSsihX218xPkt4DzkZckemB94ByJQtq7Idt9Nm3q/XJlFUKzO+jycM7prj774r1u2U2m9XohafTdDz/lMn8Ty+YsoP27dtr5syZlp/d3d0VEhKivn37qkmTJjbXCQ8PV8GCBbVnzx6boSgsLExPPfWUTpw4ocDAwBTtN27cqHHjxmnHjh26ePGicufOraeeekoDBgzQ888/b9lOXFycvvjiC82bN09Hjx5Vjhw5VKxYMXXu3FmtWrWSs/Pfv7HHxsbK09NTpy9eMez0xYwo5J2lRnchQzoytrHRXciQ4rlBi91ly5r5/gZaWrgRlzmOIKelHK6ZJwTivy3mRub4Q9Rp6WpsrIoE5FFMTMzf5gBDA1lGRyBzDAKZYxDIHINAZn8EMscgkNkfgQz/FQQy+3ucQJburiEDAAAAgMyCQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEGejO5AZOJkSH7CPI2MbG92FDMmn3miju5AhRf7Yz+guAI8ku0sWo7sAwCA5Xfn9t7e7j1FTjpABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGcTa6A3Ccbxdv0bg5GxQRFasShf018v2merpEgVTbL1u/R6Mmr9Lp81EKCvDRh90bqdazJSRJd+LvauTElVr36586eTZS7m7ZFFo+WB92ayQ/H8+0GpLhqKljdGr4tHo0rShfLzf9cTxCfcf/rN2Hz9ts65zFSe82r6wWNUspbx53HTsTqcHfbtL6XcctbTo2eEod6z+tgCcS63jo5GV9MvcXrXugTUY3ddEWfTN3vSIiY1WiiL9Gvd9U5UoEptp+2fo9GjFppWWuDur+kmo9V8Ly/IqNYZrxw/+099ApXYm9oU2z+6pU0fxpMJL0ZcrCzfp6TmJdSxbx1+jerz60rkvX7daIiat06nykggJ8NLhHY734QF3NZrNGTlqlWUu3KebaTVUsHaTP+jVToSd902A06cPURVssNS1RxF+jez18ri5dt0cjJ63UqXtzdfBb1nPVbDZr5OTVmm2paUF92jdz1VRirjoKdbU/9q04QpZhLVm7Wx+NXaJenepo/czeKlHEX6+9M16Xoq7abL9j33G9+dFMtWpYWRtm9lHdqqXVrs9UHfzrnCTp5q3b2nf4jN7rUFvrZ/bWjFGv69jJCLXuPTkth2UoauoYL4cW07A3XtDoub+oWvdp+uP4RX0/vJnyeOaw2X5g+6pqX+8p9R2/VpU6T9H0VXs0+6NXVKrQE5Y25y5d1ZBpm1T9remq0WOGtu4N19zBTRVSIE9aDctQS9b+rg/HLlHv1+tqw8w+KlnYX6++/fC52vnDGWrdsLI2zuqrelVLq02fKZa5Kkk3bt5WpTJBGvTWS2k1jHTnh59/18Avl6hvp7raNLuvShbxV5Me41Kt6/a9x9Vp4Ay1fqmyNs/pp/qhZdS612T9eSyprmNnrdOkBZv1ef/mWju9l3Jkd1GTHuN0K+5OWg3LUD+sTaxpn051tXFWH5Us4q+mPVOfq9vvzdVWjSpr0+y+qhdaWq17T9GfD8zVr2at0+QFm/VZv2ZaO+195cjuqqY9x2eamkrMVUehrvbHvlWidBvI2rdvL5PJZHl4e3urTp062rdvX6rrhIeHy2QyKSwsLNU227ZtU7169ZQ7d25ly5ZNpUqV0ueff667d++maLtx40bVq1dP3t7eypEjh4oXL673339fZ8+etccQHWri/I1q/dKzatmgkoIL5tWnfV9T9mwumrfyN5vtJy/YrBqViumt1i+oaEE/9X+zvkoH59e3i7dKkjzcsmvx193VuObTKlzgCT1TsqBG9WqqvYdO68yFqLQcmmGoqWN0e6WCZq3Zq3k/79fhU5F676s1uhEXr9a1S9ts/9oLJfXFd9u0dudfOnkhWtNW7tHanX/prSYVLG3WbD+mtTv/0vFzV/TX2SgNm7FF12/d1jMh+dJqWIYaP3+j2rxUWa0aVlJIUF591q+Zsmdz0dwVv9psP2nBJr1QqZh6tKmp4IJ++qBLA5UODtDURVssbZrVq6DeneoqtHxwWg0j3Rk/b4PaNn5WrRpVVkhQXn3ev7lyZHPRnOWp1PW7TXqhcjH1vFfXAV0bqExIgKYs2iwp8ZvxifM3qlfH2qoXWloli/hrwpC2unA5Rqs2703LoRlm/LyNats4aa5+3q+Zcjxsrn6XOFctNe3SQKVDAjR1YeJcNZvNmvjdJr1/r6YlivhrwuA292qa+v5DRsNcdQzqan/sWyVKt4FMkurUqaPz58/r/PnzWr9+vZydndWgQYN/vL0lS5YoNDRU+fPn18aNG3Xo0CG9/fbbGjZsmJo3by6z2WxpO2nSJNWsWVN+fn76/vvv9eeff2rixImKiYnRZ599Zo/hOcztO/Hae/i01Y6Tk5OTqpYP1q79J2yus+uPcFUtX9RqWfVKxVJtL0mx127JZDLJ0z27fTqejlFTx8jq7KSyRfy0aXdSTcxmafOecJUv7m9zHdeszrp1O95q2a24eFUqYfv0OScnk14JLaYcrlm182D6/zLl37p9J157D51WaAXruRpaPlg794fbXGfn/vAUQatGpRDtfMhczWxu34lX2KHTqpa8rhWCU63Tjv0nVK18iNWyGpWKWf4dTp6N1MXIWFWrkNTG0y27ypUI1M594XYfQ3pjmavlH3OuVkh9rp48d7+mSW087tc0k8xn5qpjUFf7Y98qSbq+hszV1VV+fn6SJD8/P/Xr109VqlTRpUuX5OPj81jbun79ujp37qxGjRpp8uSkw5adOnXSE088oUaNGmnhwoVq1qyZzpw5o549e6pnz5764osvLG0DAwNVtWpVRUdH22V8jhIVfV137ybIx8vdarlvbncdC79oc52IyFj5enlYLfPJ7a6ISNuHjG/F3dHH45bplVpPyz1n+p3g9kJNHcPbI4ecszjpUvQNq+WXrlxXkQBvm+ts+P24ujWpoG37T+vE+SsKfSpQDZ4LVhYnk1W74oE++unLtsrm4qzrN2+rzcc/6PCpSIeNJb2IvDdXk889Xy93HT2Z+lxNPrd9vFKfq5lRZPQ1m+8BPl4eOvqQ9wAfb1t1jZUkXbz33+RtfL2T2mRkkZb31WTvk17uOvKQueqb/H3Yy10R905vstTU5nzO+DWVmKuOQl3tj32rJOn6CNmDrl27pjlz5qhw4cLy9ra9o/YwP//8syIjI9WrV68UzzVs2FBFixbV/PnzJUmLFi3S7du31adPH5vbypUrl83lcXFxio2NtXpkRHfi76rTgOkym6VP+r5mdHcyBGr66PpNWKvjZ69ox9Q3FLGqr8Z0e1Hzft6nhAeOcEvS0TORqtptmmr2nKlpK3drfK8GCn7y8d87AADAf89/ad8qXQeylStXys3NTW5ubnJ3d9fy5cu1YMECOTk9frePHDkiSSpWrJjN50NCQixtjh49Kg8PD+XNm/exXmPkyJHy9PS0PAICAh67n/bglSunsmRxSnFBZMSVq/JN9i3Mfb7eHoqIsg6Ql2y0vz+5z1yI0uKvu6frbxvsiZo6RmTsDcXfTZBPLusbePjkzqmIK9dsrxNzU62HfC//lz5V6TbjVKHTZF2/dVvhF6Kt2t2JT9CJc1e099gFfTx9s/44cVFdGpd31FDSDe97czX53IuIupriW8X7fL09UsztS1Gpz+3MyDuXm833gEtRsfL1fkhdI23VNbH9E/f+m7xNROTVVLeZkXhb3leTvU9GXbXUJrnE99Vk9Yq6ajlqZqmpzfmc8WsqMVcdhbraH/tWSdJ1IKtevbrCwsIUFhamHTt2qHbt2qpbt65OnjypunXrWsJaiRIl/n5j95iTfYueWhuTyfS37ZLr37+/YmJiLI/Tp08/9jbswSWrs8oEB2jLziOWZQkJCdq687CeKVXQ5jrPlAzU1gfaS9LmHYes2t+f3MdPX9Lir7vLyzOnYwaQDlFTx7gTn6CwoxcU+lSgZZnJJFUtW0A7/3z49V5xd+7qfOQ1OWdxUsPnQ/Tjr0cf2t7JZJJL1iz26Ha65pLVWWVCUs7VLTuPqHypQJvrlC8VqC27rOfqph2HVT6VuZ0ZuWR1VtmQAG3eediyLKmututUoVRBq/aStHH7Icu/QwF/bz3h7WHVJvbaTf1+IFzlSwfafQzpTWpzdfOuv5mryd5XN21PmqsF8j2kpplkPjNXHYO62h/7VknSdSDLmTOnChcurMKFC6t8+fKaOnWqrl+/rilTpmjq1KmWsLZ69eq/3VbRookXAB48eNDm8wcPHrS0KVq0qGJiYnT+vO2/g5QaV1dXeXh4WD2M0qVFdc1Zvk3frdquIycuqPeYhbpx67Za1K8oSeo+ZLaGjl9uaf9Gs1Bt+O2gxs/doKPhFzVmymqFHTyt15tWkZQ4uTv2/1ZhB09pwpC2uptg1sXIWF2MjNXtO/E2+5DRUFPHGP/DDrWtW1bNa5ZS0QBvfd6jjnJmy6q5PyfeEW1C7wb6qEOopX254Hxq8FxRFfDLpcol82vx8GZyMkljFybdkemjDqF6tmSAAp7wVPFAH33UIVTPly6gRRsPpPn4jNCtRXXNXrZN81dt1+ETF9Rr9ELduBWnlg0qSZK6Dp6lj8clzdU3m1XT+l//1Li563Uk/IJGT1mtsIOn1OnVqpY2V2Kua/+RMzp84oIk6djJi9p/5IzlGojMoFvLGpq1dJvmr/xNh09c0HujFuj6zTi1aphY1y6DZmnIN8ss7d9snljXb+Yk1nXU5FUKO3hKnV9NnM8mk0ldWlTXp9PWaPXmfTpw7Ky6Dp4tvzyeqh9axpAxprVuLatr1rJtmr8yca6+P3qhbtx8YK4OSjZX79d07v2a3purryXOVZPJpC7Nq+mzaT/pxy379eexc+pmqantO7dmRMxVx6Cu9se+VaJ0fVOP5Ewmk5ycnHTz5k35+9u+A1tqXnzxRXl5eemzzz7Ts88+a/Xc8uXLdfToUQ0dOlSS1LRpU/Xr109jxoyxuqnHfdHR0aleR5ZevFzraUVGX9PoKavv/fHC/FrwRVfLIfAzF65YHQWsUDpIEz9up5GTVmn4xBUKCvDVzDGdVKxQ4m3Cz0dEa83WPyRJ1duMtnqtpeN66LlyRdJoZMahpo6xZPNB5fHMoQ/aVpFv7pzafzxCTQcstNzoI7+PhxISko5su7pk0YB2oQrMm0vXb97W2p1/qcuYFYq9HmdpkydXTk3o3UBPeLkp9kacDpyIUJMB32nT7vC0Hp4hXq5VTpejr2nU5FWKiLyqkkX9tfDLbpa5evbiFTk5Wc/VyUPba/jElRo2YaWCAnw0e0xny1yVpB+37lePoXMtP3caOEOS1KdTXfXtXC9tBmawV15MrOuISYl1LVXUX4u/6v7Ae0CUnB54D6hYJkhThrXX8AkrNXT8CgUF+GjOp2+oeOGkur7dtqZu3IzTuyPmK+baTVUqU0iLv+qmbK5Z03x8RnilVjlFXrmmkQ/M1UVjk+bqmWRzteK9uTpi4koNG584V+d80lnFH5irPdvW1PVbtx+oaZAWjc08NZWYq45CXe2PfatEJvOjnMNngPbt2+vixYuaPn26JOnKlSv65ptvNGHCBG3YsEHVqlVLsU54eLgKFiyo7777TsHB1rfFLVGihJYtW6bmzZurY8eOeuutt+Th4aH169erd+/eeuGFF7Rw4ULLP/r48eP11ltvqUOHDmrbtq0CAwN15swZzZo1S25ubo906/vY2Fh5enrqbMQVQ4+WAY/Cp97ov2+Exxb5Yz+ju5DhODk9/inl+HvpdHfgP+2fXP4AGCH+boLRXchwYmNj5e+bWzExMX+bA9L1EbI1a9ZYbqzh7u6ukJAQLVq0yGYYe1Dz5s1TLDt9+rSaNm2qjRs3avjw4apSpYpu3bqlIkWKaMCAAXrnnXes3ji7deumokWL6tNPP9XLL7+smzdvKjAwUA0aNNB7771n13ECAAAAyJzS7RGyjIAjZPgv4QiZY3CEzP44QuYY7A7YH0fI8F/BETL7e5wjZOn6ph4AAAAAkJERyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMIiz0R3IDLI4mZTFyWR0NzIMs9noHmRMUWv6Gd2FDMmrSl+ju5DhXPlljNFdyJBMJj6ngMyK/VT7e5yacoQMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAzibHQH4DhTF23R13PWKyIyViWK+Gt0r6YqVyIw1fZL1+3RyEkrdep8lIICfDT4rZdU67kSludXbAzT9B/+p70HT+lK7A1tntNXpYrmT4ORpB9TF23RN3OTajrq/YfXdNn6PRoxaaVO36vpoO4pazrjh/9p76HEmm6anflqKtl/rprNZo2cvFqzl25TzLWbqli6oD7t20yFnvRNg9GkD51eqaweLULl6+WuP/46r75fLNPug6dttnXO4qR329RQi7rllDePh46dvqTBE1Zr/fYjljZ7F/XTk3m9Uqw79Ydt6v35UkcNI92ZsnCzZa6WLOKv0b1f/Zu5ulsjJq7SqfORiXO1R2O9mHyuTlqlWZa5GqTP+mWuuUpNHYO6OgZ1tT/2VzlClqrTp0+rY8eOypcvn1xcXFSgQAG9/fbbioyMNLprj+SHtb9r4JdL1KdTXW2c1Ucli/irac/xuhR11Wb77fuOq/OHM9SqUWVtmt1X9UJLq3XvKfrzr3OWNjdu3lalMkEa9NZLaTWMdGXJ2t/14dgl6v16XW2Y2UclC/vr1bdTr+mOezVt3bCyNs7qq3pVS6tNnyk6SE2tOGKufjVrnSYv2KzP+jXT2mnvK0d2VzXtOV634u6k1bAM9XKNMhr2VkONnr5O1V4fqz+Ondf3n7+uPLly2mw/8I3aav9SRfX9YpkqtflM05f+ptkj2qlUkXyWNjU6f63gRh9bHo3fmSxJWrpxX5qMKT344efEudq3U11tmt1XJYv4q0mPcanP1b3H1WngDLV+qbI2z+mn+qFl1LrXZP15LGmujp21TpMWbNbn/Ztr7fReypHdRU16jMs0c5WaOgZ1dQzqan/sryYikNlw/PhxPfPMMzp69Kjmz5+vY8eOaeLEiVq/fr0qV66sqKgoo7v4t8bP26i2jSurVcNKCgnKq8/7NVOObC6au+JXm+0nfbdJL1Qqpp5taiq4oJ8GdGmg0iEBmrpwi6VNs3oV1KdTXVWrEJxWw0hXxs/fqDYvJdX0s37NlP1hNV2QWNMe92r6QZcGKh0coKmLrGvau1NdhZbPnDWV7D9XzWazJn63Se93rK16oaVVooi/JgxuowuXY7Rqc+YID92aV9GsFds1b/UuHQ6P0Huf/KAbt+6odYPyNtu/Vrucvpi9QWt/O6ST56I0belvWvvrIb3VvKqlTWT0dUVEXbM8aj9bTMfPXNb/9hxPq2EZbvy8DWrb+Fm1alQ5ca72b64c2Vw0Z/lD5mrlB+Zq1wYqExKgKYs2S7o3V+dvVK97c7VkEX9NGNL23lzdm5ZDMww1dQzq6hjU1f7YX01EILOhe/fucnFx0c8//6zQ0FA9+eSTqlu3rtatW6ezZ89qwIABRnfxoW7fidfeQ6etdvKdnJwUWj5YO/eH21xn5/5whSabuDUqhWjn/hOO7Op/hqWmFR6zpuWp6cM4Yq6ePBepi5GxVm/EHm7ZVa5EYKaofVbnLCpb1F+bdh2zLDObzdq866jKlyhgcx3XrFl0Ky7eatmtuDuqVDow1dd47cWnNXfVTrv1O727fSdeYYdOW80rJycnhVYITnVe7dh/QtXKh1gtq1GpmGVunzx7f64mtfG8P1f3hdt9DOkNNXUM6uoY1NX+2F9NQiBLJioqSj/99JO6deum7NmzWz3n5+enVq1aacGCBTKbzSnWjYuLU2xsrNXDCJHR13X3boJ8vDyslvt4uetipO0+RUTGytfL3WqZr5e7IlI5ZJzZ3K+pb7KaJtYo9Zr6JKupj5e7IiKp6X2OmKv317Nde2N+J9OSt2dOOTtnSXG6x6Woa/L1dre5zoYdR9SteRUF5c8jk8mkas8UUYPQknrC28Nm+/pVS8jTLZvmrf7d7v1PryKjr92bq8nnlUeq8yoiMlY+3qnPQ8tcTdbG1ztzzFVq6hjU1TGoq/2xv5qEQJbM0aNHZTabVaxYMZvPFytWTFeuXNGlS5dSPDdy5Eh5enpaHgEBAY7uLgD8a/3GLtfx05e1Y24vRWwcoTHvNda81buUYOOLJ0lqXb+81m0/rAuZYIcBAABHI5ClwtYRsL/Tv39/xcTEWB6nT9u+o5mjeefKqSxZnHQp2ZGbS1FXU/3G29fbI8W3CxFRV1N8C5FZ3a9p8qNhiTVKvaYpj1JcTfUoRWbkiLl6fz3btbe9zYwkMua64uPv2vgW1y3Vo7OR0dfV+oNZ8q81UKWbjlSFlp/o+s04hZ9LeROjgCdyqdozRTRrxQ6H9D+98s7ldm+uJp9XsanOK19vD12KTH0eWuZqsjYRkZljrlJTx6CujkFd7Y/91SQEsmQKFy4sk8mkgwcP2nz+4MGDyp07t3x8fFI85+rqKg8PD6uHEVyyOqtMSIC27Ey6ZXVCQoI27zqi8qUCba5TvlSgVXtJ2rT9sMqXKujIrv5npFbTLTv/pqa7ktV0BzV9kCPmaoF83nrC20Obdx62PB977aZ+PxCeKWp/J/6uwo6cVWi5wpZlJpNJVcsV1s4DJx+6btzteJ2/HCvnLE5qGFpKP279M0WblvXL69KVa/r510N273t65pLVWWVDAqzmVdJ7gO15VaFUQav2krRx+yHL3C7g/5C5msr1exkJNXUM6uoY1NX+2F9NQiBLxtvbW7Vq1dL48eN18+ZNq+cuXLiguXPnqlmzZjKZTAb18NF0a1lds5Zt0/yV23X4xAW9P3qhbtyMU8sGlSRJXQfN0sfjllvav9m8mtb/+qe+mbteR8IvaNTk1Qo7eEqdXku6y9qVmOvaf+SMDp+4IEk6evKi9h85o4uXM8dpS91aVNfsZds0f1ViTXuNXqgbtx6o6eBkNW2WWNNx92o6esq9mr6aek2P3a9pJjoVzN5z1WQyqUvzavps2k/6cct+/XnsnLoNni2/PJ6qH1rakDGmtfHfbVXbhhXUvE45FS3gq897vayc2V00d9UuSdKEgc300Zt1LO3LFQ9Qg6olVSCflyqXDtTiz16Xk5NJY+dtstquyWRSq3rP6Ls1v+vu3YS0HFK60K1lDc1auk3zV/6mwycu6L1RC3T9ZpxaNUycq10GzdKQb5ZZ2lvm6pz7c3WVwg6eUudXQyXdm6stquvTaWu0evM+HTh2Vl0tc7WMIWNMa9TUMairY1BX+2N/NRF/GNqGb775Rs8++6xq166tYcOGqWDBgjpw4IB69+4tf39/DR8+3Ogu/q1XapVT5JVrGjl5lSIir6pkUX8tGtvNcgj8zMUrcnJKCpUVSwdp8tD2GjFxpYaNX6mgAB/N+aSzihdK+jtEP27dr7c+nmv5udOAGZKkPp3qqt8b9dJmYAZ6uVY5XY6+plEP1HThl0k1PZusphXu1XT4xJUaNiGxprPHdFaxZDXtMfSBmg6cISmxpn07Z/yaSo6Zqz3b1tT1W7f17oj5irl2U5XKBGnR2G7K5po1zcdnhCUb9ipPrpz6oNOL8vVy1/5j59T0/W916co1SVL+J3IpISHptGxXl6wa0Lm2AvN56frN21r72yF1GbpAsdduWW232jOFFeCXW3My0d0VH/TKi4nvASMmJc7VUkX9tfir7klz9UKUnB74sq5imSBNGdZewyes1NDxKxLn6qdvqHjhpLn6dtuaunEz7oG5WkiLv8o8c5WaOgZ1dQzqan/sryYymf/JxVKZwMmTJzVo0CCtWbNGUVFR8vPzU+PGjTVo0CB5e3s/0jZiY2Pl6empC5ejDTt9MSNixjpGOj/o+5/lVaWv0V3IcK78MsboLgBAhkIcsL/Y2Fj55cmlmJiYv80BHCFLRYECBTRjxgyjuwEAAAAgA+MaMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCDORncAeFxOTiajuwA8siu/jDG6CxlO7uf7GN2FDIm5an8JCWaju5AhsR9gf2amqt09Tk05QgYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEGcH6XR8uXLH3mDjRo1+sedAQAAAIDM5JECWePGjR9pYyaTSXfv3v03/QEAAACATOORAllCQoKj+wEAAAAAmc6/uobs1q1b9uoHAAAAAGQ6jx3I7t69q6FDh8rf319ubm46fvy4JOnDDz/Ut99+a/cOAgAAAEBG9diBbPjw4ZoxY4bGjBkjFxcXy/KSJUtq6tSpdu0cAAAAAGRkjx3IZs2apcmTJ6tVq1bKkiWLZXmZMmV06NAhu3YOAAAAADKyxw5kZ8+eVeHChVMsT0hI0J07d+zSKQAAAADIDB47kBUvXlxbt25NsXzx4sV66qmn7NIpAAAAAMgMHum29w/66KOP1K5dO509e1YJCQn64YcfdPjwYc2aNUsrV650RB8BAAAAIEN67CNkL730klasWKF169YpZ86c+uijj3Tw4EGtWLFCtWrVckQfAQAAACBDeuwjZJJUpUoVrV271t59AQAAAIBM5R8FMknatWuXDh48KCnxurJy5crZrVMAAAAAkBk8diA7c+aMWrRoof/973/KlSuXJCk6OlrPPvusvvvuO+XPn9/efQQAAACADOmxryHr1KmT7ty5o4MHDyoqKkpRUVE6ePCgEhIS1KlTJ0f0EQAAAAAypMc+QrZ582Zt27ZNwcHBlmXBwcH6+uuvVaVKFbt2DgAAAAAyssc+QhYQEGDzD0DfvXtX+fLls0unAAAAACAzeOxA9sknn6hHjx7atWuXZdmuXbv09ttv69NPP7Vr5wAAAAAgI3ukUxZz584tk8lk+fn69euqWLGinJ0TV4+Pj5ezs7M6duyoxo0bO6SjAAAAAJDRPFIg+/LLLx3cDQAAAADIfB4pkLVr187R/QAAAACATOcf/2FoSbp165Zu375ttczDw+NfdQgAAAAAMovHvqnH9evX9dZbb8nX11c5c+ZU7ty5rR4AAAAAgEfz2IGsT58+2rBhgyZMmCBXV1dNnTpVQ4YMUb58+TRr1ixH9BEAAAAAMqTHPmVxxYoVmjVrlqpVq6YOHTqoSpUqKly4sAoUKKC5c+eqVatWjugnAAAAAGQ4j32ELCoqSkFBQZISrxeLioqSJD3//PPasmWLfXsHAAAAABnYYx8hCwoK0okTJ/Tkk08qJCRECxcuVIUKFbRixQrlypXLAV3EPzV10RZ9PWe9IiJjVaKIv0b3aqpyJQJTbb903R6NnLRSp85HKSjAR4Pfekm1nitheX7FxjBN/+F/2nvwlK7E3tDmOX1Vqmj+NBhJ+jFl4WZLTUsW8dfo3q/+TU13a8TEVTp1PjKxpj0a68UHamo2mzVy0irNWrpNMdduqmLpIH3Wr5kKPembBqNJP6ir/VFTx+j0SmX1aBEqXy93/fHXefX9Ypl2Hzxts61zFie926aGWtQtp7x5PHTs9CUNnrBa67cfsbTZu6ifnszrlWLdqT9sU+/PlzpqGOkKc9Uxpi7aom/mJu0DjHr/4fsAy9bv0YhJK3X63j7AoO4p9wFm/PA/7T2UuA+waXbm2weQmK+OwFz9B0fIOnTooL1790qS+vXrp3Hjxilbtmx699131bt3b7t3EP/MD2t/18Avl6hPp7raOKuPShbxV9Oe43Up6qrN9tv3HVfnD2eoVaPK2jS7r+qFllbr3lP051/nLG1u3LytSmWCNOitl9JqGOnKDz8n1rRvp7raNLuvShbxV5Me41Kv6d7j6jRwhlq/VFmb5/RT/dAyat1rsv48llTTsbPWadKCzfq8f3Otnd5LObK7qEmPcboVdyethmU46mp/1NQxXq5RRsPeaqjR09ep2utj9cex8/r+89eVJ1dOm+0HvlFb7V+qqL5fLFOlNp9p+tLfNHtEO5Uqks/SpkbnrxXc6GPLo/E7kyVJSzfuS5MxGY256hhL1v6uD8cuUe/X62rDzD4qWdhfr76d+j7Ajnv7AK0bVtbGWX1Vr2pptekzRQfZB7DCfLU/5mqixw5k7777rnr27ClJqlmzpg4dOqR58+Zpz549evvtt+3aufbt26tx48YPbXPz5k0NGjRIRYsWlaurq/LkyaNXX31VBw4csGo3ePBgmUwmmUwmZcmSRQEBAXrjjTcsp1w+aM+ePWrWrJny5s0rV1dXFShQQA0aNNCKFStkNpvtOUSHGT9vo9o2rqxWDSspJCivPu/XTDmyuWjuil9ttp/03Sa9UKmYerapqeCCfhrQpYFKhwRo6sKk01Cb1augPp3qqlqF4LQaRroyft4GtW38rFo1qpxY0/7NlSObi+Ysf0hNKz9Q064NVCYkQFMWbZaU+K3YxPkb1atjbdULLa2SRfw1YUhbXbgco1Wb96bl0AxFXe2PmjpGt+ZVNGvFds1bvUuHwyP03ic/6MatO2rdoLzN9q/VLqcvZm/Q2t8O6eS5KE1b+pvW/npIbzWvamkTGX1dEVHXLI/azxbT8TOX9b89x9NqWIZirjrG+Pkb1ealpH2Az/o1U/aH7QMsSNwH6HGvrh90aaDSwQGaush6H6B3p7oKLZ859wEk5qsjMFcTPXYgS65AgQJ65ZVXVLp0aXv057HExcWpZs2amjZtmoYNG6YjR45o9erVio+PV8WKFfXbb79ZtS9RooTOnz+vU6dOafr06VqzZo26du1q1WbZsmWqVKmSrl27ppkzZ+rgwYNas2aNXn75ZQ0cOFAxMTFpOcR/5PadeO09dNpqIjo5OSm0fLB27g+3uc7O/eEKTRa0alQK0c79JxzZ1f+M23fiFXbotFUYdXJyUmiF4FRrtGP/CVUrH2K1rEalYpZ/g5NnI3UxMlbVKiS18XTLrnIlArVzX7jdx5AeUVf7o6aOkdU5i8oW9demXccsy8xmszbvOqryJQrYXMc1axbdiou3WnYr7o4qlQ5M9TVee/FpzV210279Ts+Yq45h2QdIXte/2wcozz7AwzBf7Y+5muSRriH76quvHnmD94+epYUvv/xSv/76q/bs2aMyZcpISgyI33//vSpWrKjXX39df/zxh0wmkyTJ2dlZfn5+kiR/f3+9+uqrmj59umV7169f1+uvv6769evrhx9+sHqtYsWK6fXXX/9PHCGLjL6uu3cT5ONl/Ue6fbzcdeTkRZvrRETGytfL3WqZr5e7IlI5ZJzZREZfu1dT6xr5eHnoaHjqNfXxTt7eXRGRsZKki/f+m7yNr3dSm4yOutofNXUMb8+ccnbOkuI0mktR11SkgO1rPTbsOKJuzato294TOnE2UqHlCqtBaEllcbL9XWj9qiXk6ZZN81b/bvf+p0fMVce4vw/gm2wfwNfLXUcfsg+Q8t/BXRGR7APcx3y1P+ZqkkcKZF988cUjbcxkMqVpIJs3b55q1aplCWP3OTk56d1331WrVq20d+9elS1bNsW64eHh+umnn+Ti4mJZ9vPPPysyMlJ9+vRJ9TXvhztb4uLiFBcXZ/k5Njbj/zIBAGzrN3a5xvZpoh1ze8lsNuvEuSjNW71LrerbPsWxdf3yWrf9sC5kgh0xAECSRwpkJ06kz8OAR44cUfXq1W0+V6xYMUub+4Fs//79cnNz0927d3Xr1i1J0ueff261PUkKDk46FLpz506r1/juu+/UoEEDm685cuRIDRky5J8PyE68c+VUlixOuhRl/aF+KeqqnvD2sLmOr7dHiqNhEVFXUxw1y6y8c7ndq2nyb8dj5fuQml6KTN7+qqX9/X+LS5FX5ZfH09ImIvJqur8bkL1QV/ujpo4RGXNd8fF3bXwz65bqN7OR0dfV+oNZcnVxlpdHDp2/HKvBXesq/FxkirYBT+RStWeKqM2AWQ7pf3rEXHWM+/sAEcn2ARI/0x9S1xT/Dlfl680+wH3MV/tjrib519eQpYW5c+fKzc3N8ti6davlucc5hTA4OFhhYWHauXOn+vbtq9q1a6tHjx4PXad06dIKCwtTWFiYrl+/rvj4+FTb9u/fXzExMZbH6dO2b4XsaC5ZnVUmJEBbdibdWjkhIUGbdx1R+VKBNtcpXyrQqr0kbdp+WOVLFXRkV/8zXLI6q2xIgDbvPGxZlpCQoC07j6RaowqlClq1l6SN2w9Z/g0K+HvrCW8Pqzax127q9wPhKp/KNSYZDXW1P2rqGHfi7yrsyFmFlitsWWYymVS1XGHtPHDyoevG3Y7X+cuxcs7ipIahpfTj1j9TtGlZv7wuXbmmn389ZPe+p1fMVcdIbR8gsa6BNtcpXypQW3Yl2wfYwT7Ag5iv9sdcTfKfCGSNGjWyhKKwsDA988wzkqSiRYvq4MGDNte5v7xo0aKWZS4uLipcuLBKliypUaNGKUuWLFZHtIoUKSJJOnw46RfD1dVVhQsXVuHCSR/CqXF1dZWHh4fVwyjdWlbXrGXbNH/ldh0+cUHvj16oGzfj1LJBJUlS10Gz9PG45Zb2bzavpvW//qlv5q7XkfALGjV5tcIOnlKn15LuBnYl5rr2HzmjwycuSJKOnryo/UfO6OLlzHF6TbeWNTRr6TbNX/mbDp+4oPdGLdD1m3Fq1TCxpl0GzdKQb5ZZ2ltqOud+TVcp7OApdX41VFLizlyXFtX16bQ1Wr15nw4cO6uug2fLL4+n6oeWsdmHjIi62h81dYzx321V24YV1LxOORUt4KvPe72snNldNHfVLknShIHN9NGbdSztyxUPUIOqJVUgn5cqlw7U4s9el5OTSWPnbbLarslkUqt6z+i7Nb/r7t2EtByS4ZirjtGtRXXNXrZN81cl7gP0Gr1QN249sA8wONk+QLPEuo67tw8wesq9fYBXU98HOHZ/HyATnWLLfLU/5mqix/7D0EZwd3eXu3vKQ5HNmzfXgAEDtHfvXqvryBISEvTFF1+oePHiKa4ve9DAgQNVo0YNde3aVfny5dOLL74oLy8vjR49WkuWLHHIWNLKK7XKKfLKNY2cvEoRkVdVsqi/Fo3tZjlMfubiFTk5JV0PV7F0kCYPba8RE1dq2PiVCgrw0ZxPOqt4oaS/l/Pj1v166+O5lp87DZghSerTqa76vVEvbQZmoFdeLKfL0dc0YtKqe6cT+GvxV92TanohSk4PXGNYsUyQpgxrr+ETVmro+BWJNf30DRUvnFTTt9vW1I2bcXp3xHzFXLupSmUKafFX3ZTNNWuaj88o1NX+qKljLNmwV3ly5dQHnV6Ur5e79h87p6bvf6tLV65JkvI/kUsJCUlnbbi6ZNWAzrUVmM9L12/e1trfDqnL0AWKvXbLarvVnimsAL/cmpNJ7q74IOaqY7xcK7Guox7YB1j4ZdI+wNlk+wAV7u0DDJ+4UsMmJO4DzB7TWcWS7QP0GPrAPsDAGZIS9wH6ds74+wAS89URmKuJTOZ0fNvA9u3bKzo6WkuXLrX5/K1bt1StWjWdO3dOn332mSpWrKiLFy9qxIgRWrt2rdatW6dKlRIT9uDBg7V06VKFhYVZbaNixYoqX768vvnmG0nSkiVL1KxZM9WqVUs9e/ZUkSJFdO3aNa1Zs0Z9+/bV8uXL1bBhw0fqf2xsrDw9PXXhcrShR8symofdWAVAxpf7+dRvvIR/7sovY4zuQobzYECH/Ty4gw77YK7aX2xsrPL65FJMTMzf5oD/xCmLqcmWLZs2bNigtm3b6oMPPlDhwoVVp04dZcmSRb/99psljD3Mu+++q6lTp1qu93r55Ze1bds25ciRQ23btlVwcLBq1KihDRs2PPSGHgAAAADwuP7REbKtW7dq0qRJ+uuvv7R48WL5+/tr9uzZKliwoJ5//nlH9PM/iSNkjsERMiBz4wiZY3CEzP446uAYHCGzP+aq/Tn0CNn333+v2rVrK3v27NqzZ4/l727FxMRoxIgR/6zHAAAAAJAJPXYgGzZsmCZOnKgpU6Yoa9akCw6fe+457d69266dAwAAAICM7LED2eHDh1W1atUUyz09PRUdHW2PPgEAAABApvDYgczPz0/Hjh1LsfyXX35RUFCQXToFAAAAAJnBYweyzp076+2339b27dtlMpl07tw5zZ07V7169VLXrl0d0UcAAAAAyJAe+w9D9+vXTwkJCXrhhRd048YNVa1aVa6ururVq5d69OjhiD4CAAAAQIb02IHMZDJpwIAB6t27t44dO6Zr166pePHicnNzc0T/AAAAACDDeuxAdp+Li4uKFy9uz74AAAAAQKby2IGsevXqD/3DvBs2bPhXHQIAAACAzOKxA1nZsmWtfr5z547CwsL0xx9/qF27dvbqFwAAAABkeI8dyL744gubywcPHqxr16796w4BAAAAQGbx2Le9T03r1q01bdo0e20OAAAAADI8uwWyX3/9VdmyZbPX5gAAAAAgw3vsUxZfeeUVq5/NZrPOnz+vXbt26cMPP7RbxwAAAAAgo3vsQObp6Wn1s5OTk4KDg/Xxxx/rxRdftFvHAAAAACCje6xAdvfuXXXo0EGlSpVS7ty5HdUnAAAAAMgUHusasixZsujFF19UdHS0g7oDAAAAAJnHY9/Uo2TJkjp+/Lgj+gIAAAAAmcpjB7Jhw4apV69eWrlypc6fP6/Y2FirBwAAAADg0TzyNWQff/yx3n//fdWrV0+S1KhRI5lMJsvzZrNZJpNJd+/etX8vAQAAACADeuRANmTIEHXp0kUbN250ZH8AAAAAINN45EBmNpslSaGhoQ7rDAAAAABkJo91DdmDpygCAAAAAP6dx/o7ZEWLFv3bUBYVFfWvOgQAAAAAmcVjBbIhQ4bI09PTUX0BAAAAgEzlsQJZ8+bN5evr66i+AAAAAECm8sjXkHH9GAAAAADY1yMHsvt3WQQAAAAA2Mcjn7KYkJDgyH4AAAAAQKbzWLe9BwAAAADYD4EMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIM88t8hwz9nMplkMpmM7kaGEX+Xv4nnCM5Z+H4G/w1XfhljdBcypNy1RxrdhQwn8sd+RncBeCR32Leyu8epKXtgAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQZyN7gAcZ8rCzfp6znpFRMaqZBF/je79qsqVCEy1/dJ1uzVi4iqdOh+poAAfDe7RWC8+V8LyvNls1shJqzRr6TbFXLupiqWD9Fm/Zir0pG8ajCZ9+HbxFo2bs0ERUbEqUdhfI99vqqdLFEi1/bL1ezRq8iqdPh+loAAffdi9kWo9m1jTO/F3NXLiSq379U+dPBspd7dsCi0frA+7NZKfj2daDSldYK7aHzV1DOpqf50aPq0eTSvK18tNfxyPUN/xP2v34fM22zpncdK7zSurRc1SypvHXcfORGrwt5u0ftdxS5uODZ5Sx/pPK+CJxPfRQycv65O5v2jdA20yg6mLtuibuYlztUQRf416v+lD5+qy9Xs0YtJKy+fVoO4vqdYDc3XFxjDN+OF/2nvolK7E3tCm2X1Vqmj+NBhJ+sJ7gP1N+36rxs/doEtRsSpe2F/D32uip4unvm+1fMMejZm8WqcvRKlgfh8N7NZQNZ9NquknU3/UsnW7dTYiWi5Zs6h0cID6v1lfTz/k38lo/6kjZO3bt1fjxo1Tfb5atWp65513Un0+KipK77zzjgoUKCAXFxfly5dPHTt21KlTp1K0vXDhgnr06KGgoCC5uroqICBADRs21Pr16+0wEsf74effNfDLJerbqa42ze6rkkX81aTHOF2Kumqz/fa9x9Vp4Ay1fqmyNs/pp/qhZdS612T9eeycpc3YWes0acFmfd6/udZO76Uc2V3UpMc43Yq7k1bDMtSStbv10dgl6tWpjtbP7K0SRfz12jvjU63pjn3H9eZHM9WqYWVtmNlHdauWVrs+U3Xwr8Sa3rx1W/sOn9F7HWpr/czemjHqdR07GaHWvSen5bAMx1y1P2rqGNTV/l4OLaZhb7yg0XN/UbXu0/TH8Yv6fngz5fHMYbP9wPZV1b7eU+o7fq0qdZ6i6av2aPZHr6hUoScsbc5duqoh0zap+lvTVaPHDG3dG665g5sqpECetBqW4Zas/V0fjl2i3q/X1YaZfVSysL9effvhn1edP5yh1g0ra+OsvqpXtbTa9Jli+bySpBs3b6tSmSANeuultBpGusN7gP0tXbdbg79aovc71tbP03urROF8avHuhFRrunP/CXUdNEstGlbS2hm9VbdqKXXo963VXC30pI9GvN9Um2b31bIJbysgr5eavTNBl69cS6thPbb/VCD7N6KiolSpUiWtW7dOEydO1LFjx/Tdd9/p2LFjKl++vI4fT/rmLDw8XOXKldOGDRv0ySefaP/+/VqzZo2qV6+u7t27GziKRzd+3ga1bfysWjWqrJCgvPq8f3PlyOaiOct/tdl+0neb9ELlYurZpqaCC/ppQNcGKhMSoCmLNktK/AZn4vyN6tWxtuqFllbJIv6aMKStLlyO0arNe9NyaIaZOH+jWr/0rFo2qKTggnn1ad/XlD2bi+at/M1m+8kLNqtGpWJ6q/ULKlrQT/3frK/Swfn17eKtkiQPt+xa/HV3Na75tAoXeELPlCyoUb2aau+h0zpzISoth2Yo5qr9UVPHoK721+2VCpq1Zq/m/bxfh09F6r2v1uhGXLxa1y5ts/1rL5TUF99t09qdf+nkhWhNW7lHa3f+pbeaVLC0WbP9mNbu/EvHz13RX2ejNGzGFl2/dVvPhORLq2EZbvz8jWrzUmW1alhJIUF59Vm/ZsqezUVzV6QyVxds0guViqnHvbn6QZcGKh0coKmLtljaNKtXQb071VVo+eC0Gka6w3uA/U36bpNaNXpWLRpUUnBBP43p85qyu7rou1T2raYs3KzqFUPUvdULKhrop75v1Fep4Pya/v1WS5tXXnxGVcsHq4B/HoUE5dWQni/r6vVbOvjX2bQa1mPLNIFswIABOnfunNatW6e6devqySefVNWqVfXTTz8pa9asVkGrW7duMplM2rFjh5o0aaKiRYuqRIkSeu+99/Tbb7YnSHpy+068wg6dVrUKSW+aTk5OCq0QrJ37T9hcZ8f+E6pWPsRqWY1KxbRzf7gk6eTZSF2MjFW1CkltPN2yq1yJQO3cF273MaQ3t+/Ea+/h01YfRE5OTqpaPli7Uqnprj/CVbV8Uatl1SsVS7W9JMVeuyWTySRP9+z26Xg6x1y1P2rqGNTV/rI6O6lsET9t2p1UP7NZ2rwnXOWL+9tcxzWrs27djrdadisuXpVK2D51zsnJpFdCiymHa1btPJh+d8bs6fadeO09dFqhyedq+WDL3Etu5/7wFEGrRqWQVOd2ZsR7gP3dvhOvfYdPq+ozSftKTk5OqlK+qHb9EW5znd//OKGqyeZqtYohqba/fSdes5dtk4dbdhUvbPt9JT3IFNeQJSQk6LvvvlOrVq3k5+dn9Vz27NnVrVs3DRw4UFFRiUcl1qxZo+HDhytnzpwptpUrV65UXycuLk5xcXGWn2NjY+0zgMcUGX1Nd+8myMfL3Wq5j5eHjoZftLlORGSsfLyTt3dXRGTiGC7e+2/yNr7eSW0ysqjo6zZr6pvbXcceUlNfLw+rZT653RURafsw/K24O/p43DK9UutpuefMHIGMuWp/1NQxqKv9eXvkkHMWJ12KvmG1/NKV6yoS4G1znQ2/H1e3JhW0bf9pnTh/RaFPBarBc8HK4mSyalc80Ec/fdlW2Vycdf3mbbX5+AcdPhXpsLGkJ5H3Pq+Sf/74ernr6MmHzNUUczv1z6vMiPcA+0tt38rHy13HTkbYXCci8qp8cidrnztlvX7+3x/q8tFM3bx1R094e2jBl13lncvNvgOwo0xxhOzSpUuKjo5WsWLFbD5frFgxmc1mHTt2TMeOHZPZbFZISIjNtg8zcuRIeXp6Wh4BAQH/tuvIJO7E31WnAdNlNkuf9H3N6O4AQLrUb8JaHT97RTumvqGIVX01ptuLmvfzPiWYzVbtjp6JVNVu01Sz50xNW7lb43s1UPCTtkMegIznuaeLaP3MPlo56R1VrxSiNz6ckep1aenBfzKQzZ07V25ubpbH1q1b/34lJZ6ra482qenfv79iYmIsj9OnT//jbf0b3rnclCWLU4qJdykqVr7eHjbX8fX20KXI5O2vWto/ce+/ydtERF5NdZsZiVeunDZrGnHlqnyTfbN1n6+3hyKirL+xuWSj/f0wduZClBZ/3T3THB2TmKuOQE0dg7raX2TsDcXfTZBPLusbePjkzqmIVC6+j4y5qdZDvpf/S5+qdJtxqtBpsq7fuq3wC9FW7e7EJ+jEuSvae+yCPp6+WX+cuKgujcs7aijpive9z6vknz8RUVdTHDW7z9fbw8bcTv3zLTPiPcD+Utu3uhR1Vb5eqe1buevSlWTtr6SsV87sriqY30flSgbqiw9ayjmLk+ancl1aevCfDGSNGjVSWFiY5fHMM888tL2Pj49y5cqlgwcP2nz+4MGDMplMKly4sIoUKSKTyaRDhw49dr9cXV3l4eFh9TCCS1ZnlQ0J0Oadhy3LEhIStGXnEZUvVdDmOhVKFbRqL0kbtx9S+VKBkqQC/t56wtvDqk3stZv6/UC4ypcOtPsY0huXrM4qExygLTuPWJYlJCRo687DeiaVmj5TMlBbH2gvSZt3HLJqfz+MHT99SYu/7i4vz5SnyWZkzFX7o6aOQV3t7058gsKOXlDoU4GWZSaTVLVsAe388+HXe8XduavzkdfknMVJDZ8P0Y+/Hn1oeyeTSS5Zs9ij2+meS1ZnlQlJ+XmVOFcDba5TvlSgtuyy/rzatONwqnM7M+I9wP5csjqrdHCAtv5uPVd/2XVEz5QMtLlOuZIFtTXZXN2y43Cq7ZO2a1ZcsutP05P/ZCBzd3dX4cKFLY/s2R9+RMHJyUmvvfaa5s2bpwsXLlg9d/PmTY0fP161a9eWl5eXvLy8VLt2bY0bN07Xr19Psa3o6Gh7DsVhurWsoVlLt2n+yt90+MQFvTdqga7fjFOrhpUkSV0GzdKQb5ZZ2r/ZvJrW//qnvpmzXkfCL2jU5FUKO3hKnV8NlSSZTCZ1aVFdn05bo9Wb9+nAsbPqOni2/PJ4qn5oGUPGmNa6tKiuOcu36btV23XkxAX1HrNQN27dVov6FSVJ3YfM1tDxyy3t32gWqg2/HdT4uRt0NPyixkxZrbCDp/V60yqSEsNYx/7fKuzgKU0Y0lZ3E8y6GBmri5Gxun0n/b5p2Btz1f6oqWNQV/sb/8MOta1bVs1rllLRAG993qOOcmbLqrk/75MkTejdQB91CLW0LxecTw2eK6oCfrlUuWR+LR7eTE4maezCpG++P+oQqmdLBijgCU8VD/TRRx1C9XzpAlq08UCaj88o3VpU1+xl2zR/1XYdPnFBvUYv1I1bcWrZIHGudh08Sx+PS/q8erNZ4lwdNzdxro6eslphB0+p06tVLW2uxFzX/iNndPhE4n7UsZMXtf/IGct1UJkB7wH292bzapq7/FctWL1DR8IvqO8ni3Tj1m01b5C4b/XWx3M0fMIKS/vOr4Vq428HNWFe4r7VJ1N/1N5Dp9WhSeK+1fWbcRoxcYV+/yNcp89Hae+h03pn+DxduByjhjXKGjHER5Lhbupx6dIlhYWFWS3LmzevRowYofXr16tWrVoaM2aMSpYsqRMnTmjgwIG6c+eOxo0bZ2k/btw4Pffcc6pQoYI+/vhjlS5dWvHx8Vq7dq0mTJiQ6pG29OSVF8vpcvQ1jZi0ShGRV1WqqL8Wf9Xdckj3zIUoOZmSLoKuWCZIU4a11/AJKzV0/AoFBfhozqdvqHjhpNsEv922pm7cjNO7I+Yr5tpNVSpTSIu/6qZsrlnTfHxGeLnW04qMvqbRU1bf+4OQ+bXgi64P1PSKTA/UtELpIE38uJ1GTlql4RNXKCjAVzPHdFKxQok1PR8RrTVb/5AkVW8z2uq1lo7roefKFUmjkRmLuWp/1NQxqKv9Ldl8UHk8c+iDtlXkmzun9h+PUNMBCy03+sjv46GEhKRLCVxdsmhAu1AF5s2l6zdva+3Ov9RlzArFXk+6oVaeXDk1oXcDPeHlptgbcTpwIkJNBnynTbvD03p4hnm5VuJcHTU5ca6WLOqvhV92s8zVsxevyMnJ+vNq8tD2Gj5xpYZNWKmgAB/NHtPZ8nklST9u3a8eQ+dafu40cIYkqU+nuurbuV7aDMxgvAfYX+OaiftWY6as1qWoWJUokl/zP+8iHy/bc7V8qYIaP6StRk9erZGTVqpgfh9NH/W6Za5mcXLSsZMRWrh6mqJirim3Z06VDXlSS8f3VEhQXkPG+ChM5n9z0VQaa9++vaKjo7V06VKbz1erVk2bN29OsXzo0KEaOHCgLl++rI8//lhLly7VhQsX5OXlpbp162rIkCF68sknrdY5f/68hg8frpUrV+r8+fPy8fFRuXLl9O6776patWqP1N/Y2Fh5enrqYmSMYacvZkTxdxOM7kKG5JzlP3nAHICd5K490uguZDiRP/YzugsZklOyu2ri34u7c9foLmQ4sbGxetLPSzExf58D/lOB7L+GQOYYBDLHIJABmRuBzP4IZI5BILM/Apn9PU4gYw8MAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwiLPRHQAel3MWvkcAMrOEBLPRXciQotb0M7oLGY5X1f5GdyFDurJ1lNFdyHDYt7K/x6kp1QcAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAPy/vTuPj+ne/zj+nojEkgURQqQJIiJEtPa6t2gR1FKl9partEptrRalllrb21arpfysVVtqbymtnda1tBVUUntQYmmQiCVBzu8PMowsls44Ea/n4zEPj5zz/Z4534+T78x7zpkTmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASZzN3gE4zuRvN+iLWWt0Oi5B5Ur56sN3XlLFsgEZtl+y+neNmrhcR2PjVMLPW0N7vKB6Ncpa1xuGodGTlmvmks2KT7ysquVL6JP+rVTyiUIPYTRZAzV1DOpqf9TUMabM36gvZ9+oa9lSvhrzdotM67p0zQ6NmrRMx2LPqoSft4Z0b6q6t9X1+3WRmrHoF+3886jOJVzS+m/6KTSo2EMYSdYxZf5G67FatpSvPuybeU2XrN6h0ZOW6ejNmg59M21Npy/6RTujb9R0w6zHr6aS1LlZNfVoU1OFCrjpj4Ox6vfZd/o9+q902zrncFKfl2urTf2nVKSghw4c+1tDv1qhNdv2Wdvs/LafniiSP03fKYv+p3fGLnXYOLIa5lb7m5rOvPrUXebV0bfNq4PvmFeX3TGvrnsE5lXOkGVTi376TYM+W6x+nRto/Tf9VK6Ur5r3GK8zZy+k237rzkPqPGiG2jetrg2z+uv5mmFq3/f/FHXghLXN5zNXa1LEBn06oLVWTe+rPLld1LzHeF1JuvqwhmUqauoY1NX+qKljLF71m97/fLHeebWB1n79rsoF+uqlXhMyrOu2XYfU5f0Zat+4utbN7KeGz5TXy+9OVvTBW3W9dDlZ1cJKaMibTR/WMLKURatuHKvvdm6gdTPfVblSvmrRM+Oabr1Z03ZNqmv9N/3UsGZ5tX9nsqKoqY1mz5bXiDcb6cMZq1Wr8xf640CsFn7yqgrmy5tu+0Fd6qljkyrq99l3qvbyWE1fukXfjHpZoaWKWts8+9qXKt10hPXxQu8pkqQl63Y/lDFlBcyt9nfnvFr2HubV196foXa3zauvpDOvVg0rocGP0ByQZQJZx44dZbFYrA8vLy/Vr19fu3btumvfPXv2qGXLlvL29parq6uCgoI0ePBgXbp0yaZdQECAdft58uRRaGiopkyZkmZ7hmFo8uTJql69ujw8POTm5qayZcuqV69eOnDggN3G7EgT5qzVKy88rXZNqiu4RBF9OqC18uRy0azv/pdu+0nz1uu56mXU8+U6Kl3cRwPfaKSwYD9Nnr9B0o2aTJy7Tn07hathzfIqV8pXXw17RSf/jtfyDTsf5tBMQ00dg7raHzV1jAlz1+nlptXVrnE1BZcook/6t1LuXC6a/X0GdY1Yr+eqlVGPm3V9r2sjlS/tpynzN1rbtGpYRe90bqCalUs/rGFkKRPmrNMrL9yq6af9WylPZjWdd6Om1mO1ayOVD/bTlG9ta/pu5waqVeXxrKkkdWv1L838fpvm/PCb9sac1lsfL9GlK8lq/3yldNu3DH9KY79Zp1Vb9upI7FlNW7JVq/63V2+2/re1Tdz5izp9NtH6CH86WIf++lu/RB56WMMyHXOr/X11c15t27iaSt82r87JZF599ua8GlTcRwPSmVdbPoLzapYJZJJUv359xcbGKjY2VmvWrJGzs7MaNWqUaZ8tW7aoatWqSk5O1vLly7Vv3z6NHDlSM2bMUN26dZWcnGzT/oMPPlBsbKz++OMPtW/fXl26dNGKFSus6w3DUNu2bdWzZ081bNhQP/30k6KiojR16lTlypVLI0aMcMjY7Sn56jVF/nnM5sXIyclJNauU1vbdh9Pts233YdWqHGyz7NlqZbR9d4wk6cjxOJ2KS1CtKrfaeLrlVsWyAdq+K8buY8hqqKljUFf7o6aOkXz1mnb+eUw176xr5dLWOt1p++6YNG8Inq0WnOH/w+PGWtPK91nTKtQ0Mzmdc6hCkK/W/3brA2TDMLTh1wOqXNY/3T6uOXPoSvI1m2VXkq+qWmhAhs/Rst6Tmv3Dr3bb76yOudX+HmRe/TWdebV2tWD9+ojPAVnqO2Surq7y8fGRJPn4+Kh///7697//rTNnzsjb2ztNe8Mw9Oqrr6pMmTJatGiRnJxu5Et/f38FBQXpySef1NixY9WvXz9rH3d3d+tz9OvXTx999JFWrVqlBg0aSJIiIiI0b948LV26VE2aNLH2e+KJJ1StWjUZhuGw8dtL3PlEXb+eIu8C7jbLvQt4aH/MqXT7nI5LkLfXne3ddTouQZJ06ua/d7Yp5HWrTXZGTR2DutofNXWMuPMXdf16igoV8LBZXqiAu/YfyaSuaf4f3HU6Lv1LcR43qTX1vqOm3gXctS+Tmha6o6aFCrjrdAaXNz2OvDzzyNk5h86cTbRZfuZcokr5p30vJUlrt+1Xt1b/1uadh3X4+FnVrFhSjZ4pqxxO6X9u//y/Q+TplktzfvjN7vufVTG32l9mc8D9zKuFssG8mqXOkN0uMTFRs2bNUmBgoLy8vNJtExkZqaioKL311lvWMJYqLCxMderU0dy5c9Ptm5KSooULF+rcuXNycXGxLp87d65Kly5tE8ZuZ7FYMtznpKQkJSQk2DwAAACysv7jvtehv/7Wtllv6/TaEfqoT1PN+eE3pWTwIXT7RpW1eus+nXzE3wQDWUWWCmTLli2Tm5ub3Nzc5O7uru+++04RERFpwlaqfftu3P2nTJky6a4vU6aMtU2qfv36yc3NTa6urmrRooXy58+vzp0722yzdGnbU6G9e/e27lexYhnfpWX06NHy9PS0Pvz8/O5p3Pbmlc9NOXI4pflC5JmzCSrk5ZFun0JeHjoTd2f7C9b2hW/+e2eb03EXMtxmdkJNHYO62h81dQyvfHmVI4eTTp+1/aDt9NkLac6apSrk5ZHO/8MFFbrj0/DHVWpNz9xR0zNnL1iPuTsV8vJIczbsxv8BNU0VF39J165dl3cBN5vl3vnddDouMf0+5y+q/XvfyLfeYJV/6UNVafeJLl5OUsyJs2na+hXOp1oVAzVz2XaH7H9Wxdxqf5nNAfczr57OBvNqlgpktWvXVmRkpCIjI7Vt2zaFh4erQYMGOnLkiBo0aGANRWXLlrXpdz+XEb7zzjuKjIzU2rVrVbVqVY0dO1aBgYGZ9hk4cKAiIyM1ePBgJSamP5lJ0oABAxQfH299HDt27J73y55ccjqrQrCfNmzfa12WkpKijdv3qXJo8XT7VAktbtNektZt/VOVb14/7u/rpcJeHjZtEhIv67c9MapcPsDuY8hqqKljUFf7o6aO4ZLTWWHBftq4/daHfLfqGpBun8qhAdr4q+2Hguu37c3w/+Fxk1FNN/x6l5puv6OmW6np7a5eu67IfcdVs+Kt9zYWi0XPVAzU9j1HMu2blHxNsX8nyDmHkxrXLKcVP0eladO2YSWdOZ+on/73p933PStjbrW/B5lXK6Uzr27YtleVHvE5IEt9hyxv3rw24WjKlCny9PTU5MmTNWXKFF2+fFmSlDNnTklSUFCQJCk6OlpPPvlkmu1FR0db26QqWLCgAgMDFRgYqPnz5ys0NFSVKlVSSEiIJKlUqVLau9f2l8fb21ve3t4qVCjzvwnh6uoqV1fX+xy1Y3Rr+6y6DftGT5Z5Qk+VDdBXc9fp4uUktWtcTZLUdchMFfH2tN4W+PXWtdTo9c/05aw1qvevslr002+KjD6qz95rI+nGZN61TW19PG2lSvh5y9/XS6MmLpdPQU89XzPMtHE+TNTUMair/VFTx+jWpra6fzBLFco8oadC/DVp3npdupKkto1u1PWNoTNVxDufBne/ccn7661qqXHXzzV+9hrVrVFWi1f9rsjooxo7oLV1m+fiL+qvU+d08ky8JOnAze9NFPLyyPAsUXbSrW1tdR92s6Zl/TVx3npdunxbTYfMVJFCt9W0dS01fv1zfTl7jerVKKtFP92s6XsZ1zT1uyiFCniocMHsX1NJmhDxsya895J2/PmXfo8+pjde+pfy5nbR7Jvf+fpqYEvF/h2vDyb9KEmqGOKnIgU9tHt/rIp6e6hfpzpycrLo8zkbbLZrsVjUrmFFzVvxu65fT3no4zIbc6v9vdGmtt68bV6deHNebXNzDuh2c159/7Z5tcnNebVejbJadHNe/fQRn1ezVCC7k8VikZOTky5fvixfX9806ytUqKDg4GCNHTtWrVu3trm0cefOnVq9erVGjx6d4fb9/PzUqlUrDRgwQEuX3vijhm3atFHbtm21dOlSNW366Pz9gju9WK+i/j6fqFGTlut03AWFBvlqwbju1lPgf508K6fbvg9XNayEJo/oqJFfLdPwCd+rhJ+3Zn38mkICb/0Nkl6v1NGly0nqM2qu4hMvq1pYSS0Y1025XHM+9PGZgZo6BnW1P2rqGM3q3qjrmP+7UddyQb769rNu1roeP3VOTk636lqlfAn93/COGjlxmUZ8tUwl/Lz1zUddVKbkrbqu2LRbPYbPtv7cedAMSdK7nRuoX5eGD2dgJnqxbkXFnUvU6NtqOv/zWzX9646aVr1Z01ETl2nEhBs1nfXfLgq5o6ZvfnBbTQfOkHSjpv1fy/41laTFa3epYL68eu/VuipUwF27D5xQi77TdObcjat8ihXOZ/P9MFcXZw3sUk8BRQro4uVkrdqyV12HRygh8YrNdmtVCpSfT37Neozurng75lb7a1a3ouIymVfvnAOqlC+hSTfngJE359WZd8yrK++YV7vcnFffycLzqsXIIrcN7Nixo06dOqXp06dLks6dO6cvv/xSX331ldauXatatWql22/z5s2qW7eu6tWrpwEDBsjHx0dbt27V22+/LT8/P61du9Z61iogIEC9e/dW7969rf2joqJUrlw5bdu2TZUqVZJhGGrZsqWWLVumAQMGKDw8XIULF9aRI0c0ZswYbdu2TXFxcfc0poSEBHl6eupUXLw8PLJmIgeAR01KSpZ42cp2MrlnFR5QgWcGmL0L2dK5TWPM3oVs5zrzqt0lJCSoqHc+xcffPQdkqe+QrVy5UkWKFFGRIkVUtWpVbd++XfPnz88wjEnS008/rS1btihHjhxq0KCBAgMDNWDAAHXo0EGrVq266yWEISEhqlevngYPHizpxlm5iIgIffbZZ/rhhx/03HPPqXTp0urUqZP8/Pz0888/23PIAAAAAB5jWeYMWXbEGTIAsD/OkDkGZ8jsjzNkjsEZMvvjDJn9PbJnyAAAAADgcUIgAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAExCIAMAAAAAkxDIAAAAAMAkBDIAAAAAMAmBDAAAAABMQiADAAAAAJMQyAAAAADAJAQyAAAAADAJgQwAAAAATEIgAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAExCIAMAAAAAkxDIAAAAAMAkBDIAAAAAMAmBDAAAAABMQiADAAAAAJMQyAAAAADAJAQyAAAAADAJgQwAAAAATEIgAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAExCIAMAAAAAkxDIAAAAAMAkBDIAAAAAMAmBDAAAAABMQiADAAAAAJMQyAAAAADAJAQyAAAAADCJs9k7AADA/XByspi9C8A9ObdpjNm7kC3lr9LD7F3Ids5t+8LsXch2ctzHaxVnyAAAAADAJAQyAAAAADAJgQwAAAAATEIgAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAExCIAMAAAAAkxDIAAAAAMAkBDIAAAAAMAmBDAAAAABMQiADAAAAAJMQyAAAAADAJAQyAAAAADAJgQwAAAAATEIgAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAExCIAMAAAAAkxDIAAAAAMAkBDIAAAAAMAmBDAAAAABMQiADAAAAAJMQyAAAAADAJAQyAAAAADAJgQwAAAAATEIgAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAExCIAMAAAAAkxDIAAAAAMAkBDIAAAAAMAmBDAAAAABMQiADAAAAAJMQyLKxyd9uUPkmg+VTo7fqdPyvftsTk2n7Jat/V5UWw+VTo7eebj1SP/2yx2a9YRgaNXGZguu/pyL/6qMXun2hg0dPO3AEWQ81dQzqan/U1DGoq/1RU8egrvbXucW/tXPJUMVu+lSrpr2tp0L8M2zrnMNJ77xaX78vGqzYTZ9q0+z+eq5aGZs2bnlcNarPi9q1dJhObPxEP07poyfLPOHoYWQ5HKsEsmxr0U+/adBni9WvcwOt/6afypXyVfMe43Xm7IV022/deUidB81Q+6bVtWFWfz1fM0zt+/6fog6csLb5fOZqTYrYoE8HtNaq6X2VJ7eLmvcYrytJVx/WsExFTR2DutofNXUM6mp/1NQxqKv9NavzlEb0bqYPp6xQrVc+0h/7j2vhuG4qmN8t3faD3mikjs1qqN/HC1St1UhNX/Szvvmos0KDilnbfD6wrWpVDVbXoTNVo+1ord36p5aMf1NFvD0f1rBMx7F6Q5YMZB07dtQLL7yQ4fpatWrJYrFozJgxadY9//zzslgsGjp0qE373r17W38+fPiw2rZtq6JFiypXrlwqVqyYmjZtqj///NNmW+vWrVPDhg3l5eWlPHnyKCQkRG+//baOHz/+T4focBPmrNUrLzytdk2qK7hEEX06oLXy5HLRrO/+l277SfPW67nqZdTz5ToqXdxHA99opLBgP02ev0HSjU8bJs5dp76dwtWwZnmVK+Wrr4a9opN/x2v5hp0Pc2imoaaOQV3tj5o6BnW1P2rqGNTV/rq1ra2ZS/6nOcu2au/hk3prTIQuXUlW+8bV023fskEVjZ3xk1ZtjtKRE3GatvBnrdocpTfbPStJyuWaU01qh2noF0u1ecdBHf7rb304eYUOHTujTs3/9TCHZiqO1RuyZCC7F35+fpoxY4bNsuPHj2vNmjUqUqRIhv2uXr2qunXrKj4+XosWLdLevXsVERGh0NBQnT9/3tpu0qRJqlOnjnx8fLRw4UJFRUVp4sSJio+P1yeffOKgUdlH8tVrivzzmGpVKW1d5uTkpJpVSmv77sPp9tm2+7BqVQ62WfZstTLavjtGknTkeJxOxSWoVpVbbTzdcqti2QBt3xVj9zFkNdTUMair/VFTx6Cu9kdNHYO62l9O5xyqEOyn9dv3WpcZhqEN2/eqcmhAun1cXZx1Jdn2jMyVpKuqFlZC0o1LGp2dc2TQpqR9B5BFcaze4mz2DjyoRo0a6dtvv9Uvv/yiGjVqSJK+/vpr1atXT0ePHs2w3549e3Tw4EGtWbNG/v43rv319/e3bkOS/vrrL/Xs2VM9e/bU2LFjrcsDAgL0zDPP2AS32yUlJSkpKcn6c0JCwj8Z4gOLO5+o69dT5F3A3Wa5dwEP7Y85lW6f03EJ8va6s727TsfdGMOpm//e2aaQ16022Rk1dQzqan/U1DGoq/1RU8egrvbnlS+vnJ1z6MxZ27GeOXtBpfwLp9tn7ZZodWv7rPXsV83KQWpUO0w5nCySpMRLSdq265De6VRf+w6f1OmzF9SiXkVVDi2uQ3+dcfiYsgKO1Vse2TNkLi4uateunaZPn25dNmPGDHXq1CnTft7e3nJyctKCBQt0/fr1dNvMnz9fycnJevfdd9Ndny9fvnSXjx49Wp6entaHn5/fvQ0GAAAA2Ub/Txbq0LEz2vbtIJ3+Zaw+euclzfl+i1JSDGub14d8I4tFiv5hpE79PFavtaqlhT/9ZtMGj4dHNpBJUqdOnfTtt9/q4sWL2rhxo+Lj49WoUaNM+/j6+mrcuHEaPHiw8ufPr2effVbDhw/XoUOHrG32798vDw+PTC99TM+AAQMUHx9vfRw7duyBxvVPeeVzU44cTmm+EHnmbIIKeXmk26eQl4fOxN3Z/oK1feGb/97Z5nTchQy3mZ1QU8egrvZHTR2DutofNXUM6mp/cecv6tq16/IuYDvW28/MpO2TqPbvTJZvzbdVvukQVXlphC5eTlLMiThrm5jjf6tR13HyfeZtlWs8WHX+87GcnXPoyPG4dLeZ3XCs3pKlA9ns2bPl5uZmfWzatMlmfVhYmEqVKqUFCxZo2rRpevnll+XsfPerMLt3766TJ09q9uzZql69uubPn6+yZctq1apVkm5cF2yxWO57f11dXeXh4WHzMINLTmdVCPbThtuudU5JSdHG7ftUObR4un2qhBa3aS9J67b+ab022t/XS4W9PGzaJCRe1m97YlS5fIDdx5DVUFPHoK72R00dg7raHzV1DOpqf1evXVfkn8dUs3KQdZnFYtEzlYKs313KSFLyNcWeiZdzDic1rl1BKzbsTtPm0pVknYpLkKd7bj1XLVg/bNxl7yFkSRyrt2Tp75A1adJEVatWtf7s6+ubpk2nTp00fvx4RUVFadu2bfe8bXd3dzVu3FiNGzfWiBEjFB4erhEjRqhu3boKCgpSfHy8YmNj7/ssWVbRre2z6jbsGz1Z5gk9VTZAX81dp4uXk9SucTVJUtchM1XE21ND3mwqSXq9dS01ev0zfTlrjer9q6wW/fSbIqOP6rP32ki6MfF0bVNbH09bqRJ+3vL39dKoicvlU9BTz9cMM22cDxM1dQzqan/U1DGoq/1RU8egrvY3Yc46TRjSXjuij+r3PUf0RutaypvbVbOXbZEkfTX0ZcWePq8PJnwvSapY1l9FvPNp976/VLRQPvXr0kBOThZ9/s1q6zafrRYsiyzaf/S0ShQrqA96vqB9Mac0+/stpozRDByrN2TpQObu7i53d/dM27Rt21Z9+/ZVWFiYQkJCHuh5LBaLgoODtXnzZklSixYt1L9/f3300Uc2N/VIdf78+Qy/R5ZVvFivov4+n6hRk5brdNwFhQb5asG47tbTtX+dPCun284CVg0rockjOmrkV8s0fML3KuHnrVkfv6aQwKLWNr1eqaNLl5PUZ9RcxSdeVrWwklowrptyueZ86OMzAzV1DOpqf9TUMair/VFTx6Cu9rd49e8qmN9N7732vAp5uWv3vuNq0WuC9XK7YoXz23z3y9UlpwZ2fV4BvgV18XKSVm2OUtchM5WQeNnaxsMttwZ3a6yihfLpXMIlfb92p0Z89b2uXU956OMzC8fqDRbDMLLcNwc7duyo8+fPa8mSJemur1WrlipUqKDPPvtM0o2AlDNnTuXNm1eSVKFCBb3wwgvWv0V2e/vIyEgNGTJEL7/8skJCQuTi4qINGzaoV69e6tevn95//31J0oQJE/Tmm2/qP//5j1555RUFBATor7/+0syZM+Xm5nZPt75PSEiQp6enTsXFm3b5IgAAQHaSv0oPs3ch2zm37QuzdyHbSUhIUGEvT8XH3z0HZOkzZPfqfs5WFStWTAEBARo2bJhiYmJksVisP/fp08farlu3bgoKCtLHH3+sZs2a6fLlywoICFCjRo301ltvOWAUAAAAAB43WfIMWXbBGTIAAAD74gyZ/XGGzP7u5wxZlr7LIgAAAABkZwQyAAAAADAJgQwAAAAATEIgAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAExCIAMAAAAAkxDIAAAAAMAkBDIAAAAAMAmBDAAAAABMQiADAAAAAJMQyAAAAADAJAQyAAAAADAJgQwAAAAATEIgAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAExCIAMAAAAAkxDIAAAAAMAkBDIAAAAAMAmBDAAAAABMQiADAAAAAJMQyAAAAADAJAQyAAAAADAJgQwAAAAATEIgAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAExCIAMAAAAAkxDIAAAAAMAkBDIAAAAAMAmBDAAAAABM4mz2DgAAAAD36ty2L8zehWwnf+U3zd6FbMe4nnzPbTlDBgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZBlY5O/3aDyTQbLp0Zv1en4X/22JybT9ktW/64qLYbLp0ZvPd16pH76ZY/NesMwNGriMgXXf09F/tVHL3T7QgePnnbgCLIeauoY1NX+qKljUFf7o6aOQV0dg7ra19NPltTcT19X1A8jdW77l2pYs/xd+9R4qpTWf9NPJ38Zq98WDVGbRlXTtOn80jPauXSYYn8eq1XT++qpEH9H7L7dZOlA1rFjR1ksFuvDy8tL9evX165duzLsExMTk6ZPvXr1tGPHDmubWrVq2bRJfXTt2tXa5vblHh4eqly5spYuXerQ8drTop9+06DPFqtf5wZa/00/lSvlq+Y9xuvM2Qvptt+685A6D5qh9k2ra8Os/nq+Zpja9/0/RR04YW3z+czVmhSxQZ8OaK1V0/sqT24XNe8xXleSrj6sYZmKmjoGdbU/auoY1NX+qKljUFfHoK72lye3q/7Yd1zvfBRxT+2fKOqliM+6atNv+/RMuzGaOHedxg1sq2erlbG2aVb3KY3o3UwfTlmhWi9/qD/2H9fCL7qrYH43Rw3jH8vSgUyS6tevr9jYWMXGxmrNmjVydnZWo0aN7tpv9erVio2N1Y8//qjExEQ1aNBA58+ft67v0qWLdbupj48++shmG9OnT1dsbKx+/fVX1ahRQy1atNDu3bvtPUSHmDBnrV554Wm1a1JdwSWK6NMBrZUnl4tmffe/dNtPmrdez1Uvo54v11Hp4j4a+EYjhQX7afL8DZJufIIzce469e0UroY1y6tcKV99NewVnfw7Xss37HyYQzMNNXUM6mp/1NQxqKv9UVPHoK6OQV3tb/XmKI2cuEzL12d8suV2nV78l46eiNP7ny3WvphTmjx/o75bG6k32ta2tunW9lnNXLJZc77for2HT+qt0fN06Uqy2jep7qhh/GNZPpC5urrKx8dHPj4+qlChgvr3769jx47pzJkzmfbz8vKSj4+PKlWqpI8//linTp3S1q1brevz5Mlj3W7qw8PDw2Yb+fLlk4+Pj4KCgjR8+HBdu3ZN69atc8g47Sn56jVF/nlMtaqUti5zcnJSzSqltX334XT7bNt9WLUqB9sse7ZaGW3fHSNJOnI8TqfiElSryq02nm65VbFsgLbvirH7GLIaauoY1NX+qKljUFf7o6aOQV0dg7pmDZVDi2v9tr02y9ZsiVaV0OKSpJzOOVQh2M+mjWEY2rBtryrfbJMVZflAdrvExETNmjVLgYGB8vLyuud+uXPnliQlJyc/0PNeu3ZNU6dOlSS5uLhk2C4pKUkJCQk2DzPEnU/U9esp8i7gbrPcu4CHTselv0+n4xLk7XVne3dr+1M3/72zTSEv9wy3mZ1QU8egrvZHTR2DutofNXUM6uoY1DVrKOTlkeYS0TNxCfJwy61crjnllc9Nzs450rY5m6BCXrYnXrKSLB/Ili1bJjc3N7m5ucnd3V3fffedIiIi5OR0b7t+/vx5DR8+XG5ubqpSpYp1+YQJE6zbTX3Mnj3bpm+bNm3k5uYmV1dX9enTRwEBAWrZsmWGzzV69Gh5enpaH35+fg82aAAAAACPhSwfyGrXrq3IyEhFRkZq27ZtCg8PV4MGDXTkyBE1aNDAGqbKli1r0+/pp5+Wm5ub8ufPr507dyoiIkKFCxe2rm/Xrp11u6mPJk2a2Gxj7NixioyM1IoVKxQSEqIpU6aoQIECGe7rgAEDFB8fb30cO3bMvsW4R1753JQjh9N9fTpQyMtDZ+LubH/B2r7wzX/vbHM67kKW/sTBXqipY1BX+6OmjkFd7Y+aOgZ1dQzqmjWcjktIe5bSy0MJiZd1Jemq4s4n6tq16/d1JjMryPKBLG/evAoMDFRgYKAqV66sKVOm6OLFi5o8ebKmTJliDVM//PCDTb+IiAjt3LlT586d08GDB9WwYUOb9Z6entbtpj7c3W3/83x8fBQYGKh69epp+vTpatWqlU6fzvhWpK6urvLw8LB5mMElp7MqBPtpw/Zb18+mpKRo4/Z9GV4/WyW0uE17SVq39U9VDg2QJPn7eqmwl4dNm4TEy/ptT4wqlw+w+xiyGmrqGNTV/qipY1BX+6OmjkFdHYO6Zg3bdx9WzcqlbZbVrhKsbTe/x3f12nVF/nnMpo3FYtEzlYMy/K5fVpDlA9mdLBaLnJycdPnyZfn6+lrDlL+/7d8X8PPzU8mSJZUvXz67PG+VKlVUsWJFjRw50i7bc7TUO8zMXXbzDjNjInTxcpLaNa4mSeo6ZKaGfXnrNv6vt66lNf+L0pez1mhfzEmN+b/liow+qi4v1ZR0o+5d29TWx9NW6ocNu7TnwHG9MfQb+RT01PM1w0wZ48NGTR2DutofNXUM6mp/1NQxqKtjUFf7y5vbReWCfFUuyFeS5F/US+WCfFWscH5J0uDuTfTV0Jet7act+ln+vl4a1qOpSvkX1qst/q0X6jypr+bcuule6t0wWz9fVUEBhfVp/1bKm9tVs7/f8nAHdx+czd6Bu0lKStLJkyclSefOndOXX36pxMRENW7c+B9t99KlS9btpnJ1dVX+/Pkz7NO7d281a9ZM7777rnx9ff/R8zvai/Uq6u/ziRo1ablOx11QaJCvFozrbj0F/tfJs3KyWKztq4aV0OQRHTXyq2UaPuF7lfDz1qyPX1NIYFFrm16v1NGly0nqM2qu4hMvq1pYSS0Y1025XHM+9PGZgZo6BnW1P2rqGNTV/qipY1BXx6Cu9lehjL+WTepl/XnUW80lSXOWbVH3YbNUuKCHivnc+rrQ0RNxatV7oka99aJeb11LJ06fV8+Rc7R2S7S1zeJVv6tgPje99/rzKuTlrt37jqtFz4z/XlxWYDEMwzB7JzLSsWNHff3119af3d3dFRwcrH79+ql58+bp9omJiVHx4sW1Y8cOVahQId02tWrV0oYNG9IsDw8P18qVKyXd+NRi8eLFeuGFF6zrDcNQSEiIateurQkTJtx1/xMSEuTp6alTcfGmXb4IAAAAZCZ/5TfN3oVsx7ierKTdkxUff/cckKUD2aOOQAYAAICsjkBmf/cTyB6575ABAAAAQHZBIAMAAAAAkxDIAAAAAMAkBDIAAAAAMAmBDAAAAABMQiADAAAAAJMQyAAAAADAJAQyAAAAADAJgQwAAAAATEIgAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAExCIAMAAAAAkxDIAAAAAMAkBDIAAAAAMAmBDAAAAABMQiADAAAAAJMQyAAAAADAJAQyAAAAADAJgQwAAAAATEIgAwAAAACTEMgAAAAAwCQEMgAAAAAwCYEMAAAAAExCIAMAAAAAkxDIAAAAAMAkBDIAAAAAMAmBDAAAAABMQiADAAAAAJMQyAAAAADAJAQyAAAAADAJgQwAAAAATEIgAwAAAACTEMgAAAAAwCTOZu9AdmYYhiTpQkKCyXsCAAAApM+4nmz2LmQ7qTVNzQOZIZA50IULFyRJgcX9TN4TAAAAAA/bhQsX5OnpmWkbi3EvsQ0PJCUlRSdOnJC7u7ssFovZu5OphIQE+fn56dixY/Lw8DB7d7IN6mp/1NQxqKv9UVPHoK72R00dg7o6xqNSV8MwdOHCBRUtWlROTpl/S4wzZA7k5OSkYsWKmb0b98XDwyNLH9yPKupqf9TUMair/VFTx6Cu9kdNHYO6OsajUNe7nRlLxU09AAAAAMAkBDIAAAAAMAmBDJIkV1dXDRkyRK6urmbvSrZCXe2PmjoGdbU/auoY1NX+qKljUFfHyI515aYeAAAAAGASzpABAAAAgEkIZAAAAABgEgIZAAAAAJiEQAYAAAAAJiGQZSMdO3bUCy+8kOH6WrVqyWKxyGKxKFeuXAoJCdGECROs62fMmGFdf/sjV65cNs+RujxnzpwqXry43n33XV25csWRQ3Oo28dksVjk5eWl+vXra9euXRn2iYmJkcViUY4cOXT8+HGbdbGxsXJ2dpbFYlFMTIxN+8jISGu7xYsXq1q1avL09JS7u7vKli2r3r1722wrOTlZH330kcLCwpQnTx4VLFhQNWrU0PTp03X16lV7lcDh/kmNb6/ZnTZv3qyGDRsqf/78ypUrl0JDQ/Xpp5/q+vXradquW7dODRs2lJeXl/LkyaOQkBC9/fbbaf7/HjfHjh1Tp06dVLRoUbm4uMjf31+9evVSXFyc2btmmrvNpZJ0+fJlDRkyREFBQXJ1dVXBggX10ksvac+ePTbthg4daj3uc+TIIT8/P7322ms6e/Zsmm3u2LFDrVq1UpEiReTq6ip/f381atRI33//vbLT/bfu5bXqzrnwdmfPnlXv3r3l7+8vFxcXFS1aVJ06ddLRo0fTtD158qR69OihEiVKyNXVVX5+fmrcuLHWrFljh5GY50Hm1FR79uxRy5Yt5e3tLVdXVwUFBWnw4MG6dOmSTbuAgADr9vPkyaPQ0FBNmTIlzfYMw9DkyZNVvXp1eXh4yM3NTWXLllWvXr104MABu435YbnX91JjxoxJs+7555+XxWLR0KFDbdrffjwfPnxYbdu2VdGiRZUrVy4VK1ZMTZs21Z9//mmzrez2mvVP3gfc3qdevXrasWOHtc3t721vf3Tt2tXa5vblHh4eqly5spYuXerQ8d4vAtljpkuXLoqNjVVUVJRatmyp7t27a+7cudb1Hh4eio2NtXkcOXLEZhv169dXbGysDh06pLFjx2rSpEkaMmTIwx6KXaWOKTY2VmvWrJGzs7MaNWp0136+vr6aOXOmzbKvv/5avr6+mfZbs2aNWrVqpebNm2vbtm367bffNHLkSJuQlZycrPDwcI0ZM0avvfaaNm/erG3btql79+764osv0rzxy+oetMYZWbx4sWrWrKlixYpp3bp1+vPPP9WrVy+NGDFCrVu3tnkDO2nSJNWpU0c+Pj5auHChoqKiNHHiRMXHx+uTTz6xx/AeSYcOHVKlSpW0f/9+zZ07VwcOHNDEiRO1Zs0aVa9ePd3QACkpKUl16tTRtGnTNGLECO3bt08//PCDrl27pqpVq2rLli027cuWLavY2FgdPXpU06dP18qVK/XGG2/YtFm6dKmqVaumxMREff3114qOjtbKlSvVrFkzDRo0SPHx8Q9ziFnW2bNnVa1aNa1evVoTJ07UgQMHNG/ePB04cECVK1fWoUOHrG1jYmJUsWJFrV27Vv/973+1e/durVy5UrVr11b37t1NHIV9PMicumXLFlWtWlXJyclavny59u3bp5EjR2rGjBmqW7eukpOTbdp/8MEHio2N1R9//KH27durS5cuWrFihXW9YRhq27atevbsqYYNG+qnn35SVFSUpk6dqly5cmnEiBEOGbvZ/Pz8NGPGDJtlx48f15o1a1SkSJEM+129elV169ZVfHy8Fi1apL179yoiIkKhoaE6f/68tV12fc160PcBq1evVmxsrH788UclJiaqQYMGNvVKfW97++Ojjz6y2cb06dMVGxurX3/9VTVq1FCLFi20e/duew/xwRnINjp06GA0bdo0w/U1a9Y0evXqZbOsVKlSRuvWrQ3DMIzp06cbnp6e9/0cL774ovHkk08+wB5nDemNadOmTYYk4/Tp0+n2OXz4sCHJGDRokFGqVCmbdUFBQcb7779vSDIOHz5s037Hjh2GYRhGr169jFq1amW6Xx9++KHh5ORk/P7772nWJScnG4mJifc2wCzgn9Q4tWa3S0xMNLy8vIwXX3wxzbrvvvvOkGTMmzfPMAzDOHbsmOHi4mL07t073ec5d+7cfY0lO6lfv75RrFgx49KlSzbLY2NjjTx58hhdu3Y1ac/Mdbe5dMyYMYbFYjEiIyNtll+/ft2oVKmSERISYqSkpBiGYRhDhgwxwsLCbNq99dZbRv78+a0/px7PzZo1y/A5U7eXHTzIa1Wqrl27Gnnz5jViY2Ntll+6dMnw9fU16tevb13WoEEDw9fXN9258lH/vX+QOTUlJcUICQkxKlWqZFy/ft1mXWRkpGGxWIwxY8ZYl/n7+xtjx461aVegQAGjT58+1p/nzp1rSDKWLl2a4XM+au7l+HzjjTcMLy8v4+eff7YuHzlypNG4cWMjLCzMGDJkiE371ON5x44dhiQjJiYmw+1n19cse70P+OWXXwxJxsqVKw3DyHy+SCXJWLx4sfXnhIQEQ5Lx+eefP8hQHIIzZI+53Llzp/lE7H788ccf2rx5s1xcXOy4V+ZKTEzUrFmzFBgYKC8vr0zbNmnSROfOndPPP/8sSfr555917tw5NW7cONN+Pj4+2rNnj/74448M28yePVt16tTRk08+mWZdzpw5lTdv3nsYTdZ0PzVOz08//aS4uDj17ds3zbrGjRsrKCjIeuZ3/vz5Sk5O1rvvvpvutvLly3ffz58dnD17Vj/++KO6deum3Llz26zz8fFRu3btFBERka0ulbOXOXPmqG7dugoLC7NZ7uTkpD59+igqKko7d+5Mt29MTIx+/PFHmzkz9XjO6BiVblxy87hLSUnRvHnz1K5dO/n4+Nisy507t7p166Yff/xRZ8+e1dmzZ7Vy5Up179493bkyu/3e38ucGhkZqaioKL311ltycrJ9+xcWFqY6derYXDFzu5SUFC1cuFDnzp2zOXbnzp2r0qVLq0mTJun2y67HrYuLi9q1a6fp06dbl82YMUOdOnXKtJ+3t7ecnJy0YMGCdC+tlx6f16wHfR+Q+nr1oO9dr127pqlTp0pSlnrvSiB7TF2/fl2zZs3Srl279Oyzz1qXx8fHy83NzebRoEEDm77Lli2Tm5ub9Ts7p0+f1jvvvPOwh2BXqWNyc3OTu7u7vvvuO0VERKR50bpTzpw51b59e02bNk2SNG3aNLVv3145c+bMtF+PHj1UuXJlhYaGKiAgQK1bt9a0adOUlJRkbbN//34FBwf/88FlEQ9a4/Ts27dPklSmTJl01wcHB1vb7N+/Xx4eHpleRvI42r9/vwzDyLCGZcqU0blz53TmzJmHvGdZ3759+zKtW2qbVLt375abm5ty586t4sWLa8+ePerXr5/N9iSpdOnS1mXbt2+3mYeXLVvmiKE8Us6cOaPz589nWnvDMHTgwAEdOHBAhmFkqzn0Tvc7p95t3ixTpozNcStJ/fr1k5ubm1xdXdWiRQvlz59fnTt3ttnm7cetJPXu3du6X8WKFfsnQ8zSOnXqpG+//VYXL17Uxo0bFR8ff9fL73x9fTVu3DgNHjxY+fPn17PPPqvhw4fbXGqbnV+z/un7gPPnz2v48OFyc3NTlSpVrMsnTJiQ5r3r7Nmzbfq2adPGeiz36dNHAQEBatmypV3H908QyLKh2bNn2xyUmzZtsq5LPWhz586tLl26qE+fPjbfZXB3d1dkZKTN484v8dauXVuRkZHaunWrOnTooP/85z9q3rz5QxufI6SOKTIyUtu2bVN4eLgaNGigI0eOqEGDBtZali1bNk3fTp06af78+Tp58qTmz59/10/IJClv3rxavny5Dhw4oEGDBsnNzU1vv/22qlSpYv1idXY7M/FPapyRe6mRYRjZ9lNae8hux5k9ZTaX3k/dSpcurcjISG3fvl39+vVTeHi4evTokWmf8uXLW39fLl68qGvXrj3wOLKqzOqbmXv9vc/uHnROvZ/avPPOO4qMjNTatWtVtWpVjR07VoGBgZn2GThwoCIjIzV48GAlJiY+0Niygrsdn2FhYSpVqpQWLFigadOm6eWXX5azs/Ndt9u9e3edPHlSs2fPVvXq1TV//nyVLVtWq1atkpS9X7Me9Jh9+umn5ebmpvz582vnzp2KiIhQ4cKFrevbtWuX5r3rnWdtx44dq8jISK1YsUIhISGaMmWKChQo8FDGfS/ufuTgkdOkSRNVrVrV+vPtN5ho166dBg4cqNy5c6tIkSJpPpVwcnK662SbN29ea5tp06YpLCxMU6dO1auvvmrHUTxct49JkqZMmSJPT09NnjxZU6ZM0eXLlyUp3TNfoaGhCg4OVps2bVSmTBmVK1cu0zsD3q5kyZIqWbKkOnfurIEDByooKEgRERH6z3/+o6CgoDR3XXqU/ZMa3ykoKEiSFB0draeffjrN+ujoaIWEhFjbxsfHKzY2Nlt+4vigAgMDZbFYFB0drWbNmqVZHx0drfz588vb29uEvcsaMppLg4KCFB0dnW6f1OWpx6h047KY1GN/zJgxev755zVs2DANHz5cklSqVClJ0t69e1WtWjVJkqur613n4kddZq9V6fH29la+fPkyrb3FYrHWzWKxZKs59E73O6fePm+mdyl8dHS0zXErSQULFlRgYKACAwM1f/58hYaGqlKlStb5tVSpUtq7d69NH29vb3l7e6tQoUL2G6wJ7uX47NSpk8aPH6+oqCht27btnrft7u6uxo0bq3HjxhoxYoTCw8M1YsQI1a1bN1u/Zj3o+4CIiAiFhITIy8sr3Us2PT097zpf+vj4WI/l6dOnq2HDhoqKisoyxylnyLIhd3d360EXGBho8/2Q1IPW19f3gS4Vu5OTk5Pee+89DRo0yPqLlB1YLBY5OTnp8uXL8vX1tdbS398/3fadOnXS+vXr7+nsWEYCAgKUJ08eXbx4UZLUtm1brV692ub2rqmuXr1qbfeout8a365evXoqUKBAuneb+u6777R//361adNGktSiRQu5uLikueNSqtvv1PQ48fLyUt26dTVhwoQ0v7upn962atUq235Sey8ymktbt26t1atXp/meWEpKisaOHauQkJA03y+73aBBg/Txxx/rxIkTkm4dzx9++KHjBpMFZfZalR4nJye1bNlSc+bM0cmTJ23WXb58WRMmTFB4eLgKFCigAgUKKDw8XOPHj093rsyOv/d3m1MrVKig4OBgjR07VikpKTZ9d+7cqdWrV1vnzfT4+fmpVatWGjBggHVZmzZttHfv3ix3C3F7uJfjs23bttq9e7fKlStnDan3y2KxKDg42HqcPk6vWff6PsDPz08lS5a02/fnqlSpoooVK2rkyJF22Z49EMhgwzAMnTx5Ms3jzsn7di+99JJy5Mih8ePHP8Q9ta+kpCTrWKOjo9WjRw8lJibe9eYcqbp06aIzZ87YXFufmaFDh+rdd9/V+vXrdfjwYe3YsUOdOnWy3hJXunEdfo0aNfTcc89p/Pjx2rlzpw4dOqRvv/1W1apV0/79+x94vGZ40Brv3bs3zaUILi4umjRpkpYuXarXXntNu3btUkxMjKZOnaqOHTuqRYsW1mvD/fz8NHbsWH3++ed69dVXtWHDBh05ckS//PKLXn/9detZisfRl19+qaSkJIWHh2vjxo06duyYVq5cqbp168rX1zdLvVhlJX369FGVKlXUuHFjzZ8/X0ePHtX27dvVvHlzRUdHa+rUqZkG2erVq6t8+fIaNWqUJMnNzU1TpkzR8uXL9fzzz+vHH3/UoUOHtGvXLuubshw5cjyUsWUVZ86cSfN7f+rUKY0aNUo+Pj6qW7euVqxYoWPHjmnjxo0KDw/X1atXbV6Hxo8fr+vXr6tKlSpauHCh9u/fr+joaI0bN07Vq1c3cXT2cb9zqsVi0dSpUxUVFWX9kytHjx7V/Pnz1bhxY1WvXj3Tv/8mSb169dL333+vX3/9VdKNDydatGih1q1b64MPPtDWrVsVExOjDRs2KCIiItsft/nz57fewv1eREZGqmnTplqwYIGioqJ04MABTZ06VdOmTVPTpk0lZe/XrH/6Xisjly5dSvO+9dy5c5n26d27tyZNmpR1/q6bGbd2hGP8k1sJG8aN295LSveReovhjJ5j9OjRhre39yN1K/ZUHTp0sBmru7u7UblyZWPBggUZ9snsluyGcevWthnd9n7t2rVG8+bNDT8/P8PFxcUoXLiwUb9+fWPTpk0227ly5YoxevRoIzQ01MiVK5dRoEABo0aNGsaMGTOMq1ev2mP4D8U/qXF6j2PHjhmGYRgbN240wsPDDQ8PD8PFxcUoW7as8fHHHxvXrl1Ls71Vq1YZ4eHhRv78+Y1cuXIZwcHBRt++fY0TJ044bNyPgpiYGKNDhw5G4cKFjZw5cxp+fn5Gjx49jL///tvsXTPN3eZSwzCMixcvGgMHDjQCAwONnDlzGgUKFDCaN29u7N6926Zdere9N4wbtwt3dXU1jh49al22fft2o0WLFkahQoUMZ2dnw8vLywgPDzfmzZv3SN4+PCP38lqV3u/98OHDDcMwjDNnzhg9evQw/Pz8jJw5cxqFCxc2OnbsaBw5ciTNtk6cOGF0797d8Pf3N1xcXAxfX1+jSZMmxrp16xw0uofjQebUVLt27TKaN29uFChQwMiZM6dRsmRJY9CgQcbFixdt2qV323vDMIzw8HCjQYMG1p+vX79uTJw40ahataqRN29ew8XFxShRooTRpUsXIyoq6h+P9WH7p++lMrvt/ZkzZ4yePXsa5cqVM9zc3Ax3d3cjNDTU+Pjjj9P8KYLs9prliPdahpHxfBEeHm5toztue28YN/4kQ3BwsPHGG2/806HZhcUwHoNvvgIAAABAFsQliwAAAABgEgIZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAIDHWseOHfXCCy9Yf65Vq5Z69+790Pdj/fr1slgsOn/+fIZtLBaLlixZcs/bHDp0qCpUqPCP9ismJkYWi0WRkZH/aDsAgPQRyAAAWU7Hjh1lsVhksVjk4uKiwMBAffDBB7p27ZrDn3vRokUaPnz4PbW9lxAFAEBmnM3eAQAA0lO/fn1Nnz5dSUlJ+uGHH9S9e3flzJlTAwYMSNM2OTlZLi4udnneAgUK2GU7AADcC86QAQCyJFdXV/n4+Mjf319vvPGG6tSpo++++07SrcsMR44cqaJFi6p06dKSpGPHjqlly5bKly+fChQooKZNmyomJsa6zevXr+utt95Svnz55OXlpXfffVeGYdg8752XLCYlJalfv37y8/OTq6urAgMDNXXqVMXExKh27dqSpPz588tisahjx46SpJSUFI0ePVrFixdX7ty5FRYWpgULFtg8zw8//KCgoCDlzp1btWvXttnPe9WvXz8FBQUpT548KlGihN5//31dvXo1TbtJkybJz89PefLkUcuWLRUfH2+zfsqUKSpTpoxy5cql4OBgTZgw4b73BQDwYAhkAIBHQu7cuZWcnGz9ec2aNdq7d69WrVqlZcuW6erVqwoPD5e7u7s2bdqkX375RW5ubqpfv7613yeffKIZM2Zo2rRp+vnnn3X27FktXrw40+d95ZVXNHfuXI0bN07R0dGaNGmS3Nzc5Ofnp4ULF0qS9u7dq9jYWH3++eeSpNGjR2vmzJmaOHGi9uzZoz59+qh9+/basGGDpBvB8cUXX1Tjxo0VGRmpzp07q3///vddE3d3d82YMUNRUVH6/PPPNXnyZI0dO9amzYEDB/Ttt9/q+++/18qVK7Vjxw5169bNun727NkaPHiwRo4cqejoaI0aNUrvv/++vv766/veHwDAAzAAAMhiOnToYDRt2tQwDMNISUkxVq1aZbi6uhp9+/a1ri9cuLCRlJRk7fPNN98YpUuXNlJSUqzLkpKSjNy5cxs//vijYRiGUaRIEeOjjz6yrr969apRrFgx63MZhmHUrFnT6NWrl2EYhrF3715DkrFq1ap093PdunWGJOPcuXPWZVeuXDHy5MljbN682abtq6++arRp08YwDMMYMGCAERISYrO+X79+abZ1J0nG4sWLM1z/3//+16hYsaL15yFDhhg5cuQw/vrrL+uyFStWGE5OTkZsbKxhGIZRsmRJY86cOTbbGT58uFG9enXDMAzj8OHDhiRjx44dGT4vAODB8R0yAECWtGzZMrm5uenq1atKSUlR27ZtNXToUOv60NBQm++N7dy5UwcOHJC7u7vNdq5cuaKDBw8qPj5esbGxqlq1qnWds7OzKlWqlOayxVSRkZHKkSOHatasec/7feDAAV26dEl169a1WZ6cnKwnn3xSkhQdHW2zH5JUvXr1e36OVBERERo3bpwOHjyoxMREXbt2TR4eHjZtnnjiCfn6+to8T0pKivbu3St3d3cdPHhQr776qrp06WJtc+3aNXl6et73/gAA7h+BDACQJdWuXVtfffWVXFxcVLRoUTk7275k5c2b1+bnxMREVaxYUbNnz06zLW9v7wfah9y5c993n8TEREnS8uXLbYKQdON7cfbyv//9T+3atdOwYcMUHh4uT09PzZs3T5988sl97+vkyZPTBMQcOXLYbV8BABkjkAEAsqS8efMqMDDwnts/9dRTioiIUKFChdKcJUpVpEgRbd26Vc8884ykG2eCfvvtNz311FPptg8NDVVKSoo2bNigOnXqpFmfeobu+vXr1mUhISFydXXV0aNHMzyzVqZMGesNSlJt2bLl7oO8zebNm+Xv76+BAwdalx05ciRNu6NHj+rEiRMqWrSo9XmcnJxUunRpFS5cWEWLFtWhQ4fUrl27+3p+AIB9cFMPAEC20K5dOxUsWFBNmzbVpk2bdPjwYa1fv149e/bUX3/9JUnq1auXxowZoyVLlujPP/9Ut27dMv0bYgEBAerQoYM6deqkJUuWWLf57bffSpL8/f1lsVi0bNkynTlzRomJiXJ3d1ffvn3Vp08fff311zp48KB+//13ffHFF9YbZXTt2lX79+/XO++8o71792rOnDmaMWPGfY23VKlSOnr0qObNm6eDBw9q3Lhx6d6gJFeuXOrQoYN27typTZs2qWfPnmrZsqV8fHwkScOGDdPo0aM1btw47du3T7t379b06dP16aef3tf+AAAeDIEMAJAt5MmTRxs3btQTTzyhF198UWXKlNGrr76qK1euWM+Yvf3223r55ZfVoUMHVa9eXe7u7mrWrFmm2/3qq6/UokULdevWTcHBwerSpYsuXrwoSfL19dWwYcPUv39/FS5cWG+++aYkafjw4Xr//fc1evRolSlTRvXr19fy5ctVvHhxSTe+17Vw4UItWbJEYWFhmjhxokaNGnVf423SpIn69OmjN998UxUqVNDmzZv1/vvvp2kXGBioF198UQ0bNlS9evVUvnx5m9vad+7cWVOmTNH06dMVGhqqmjVrasaMGdZ9BQA4lsXI6JvMAAAAAACH4gwZAAAAAJiEQAYAAAAAJiGQAQAAAIBJCGQAAAAAYBICGQAAAACYhEAGAAAAACYhkAEAAACASQhkAAAAAGASAhkAAAAAmIRABgAAAAAmIZABAAAAgEn+HyZo40UkQllDAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 1000x1000 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "eval_token_list = list(set(eval_df_tokens['labels']))\n",
    "\n",
    "plot_confusion_matrix(eval_df_tokens[\"labels\"], eval_df_tokens[\"predicted_label\"],\n",
    "                      eval_token_list)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define & Call Function to Display Example Token Sequences Along With Labels & Losses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "      <th>14</th>\n",
       "      <th>15</th>\n",
       "      <th>16</th>\n",
       "      <th>17</th>\n",
       "      <th>18</th>\n",
       "      <th>19</th>\n",
       "      <th>20</th>\n",
       "      <th>21</th>\n",
       "      <th>22</th>\n",
       "      <th>23</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>tokens</th>\n",
       "      <td>ĠHe</td>\n",
       "      <td>Ġwas</td>\n",
       "      <td>Ġselected</td>\n",
       "      <td>Ġto</td>\n",
       "      <td>Ġgo</td>\n",
       "      <td>Ġon</td>\n",
       "      <td>Ġthe</td>\n",
       "      <td>Ġ2002</td>\n",
       "      <td>ĠNew</td>\n",
       "      <td>ĠZealand</td>\n",
       "      <td>Ġrugby</td>\n",
       "      <td>Ġleague</td>\n",
       "      <td>Ġtour</td>\n",
       "      <td>Ġof</td>\n",
       "      <td>ĠGreat</td>\n",
       "      <td>ĠBritain</td>\n",
       "      <td>Ġand</td>\n",
       "      <td>ĠFrance</td>\n",
       "      <td>Ġ,</td>\n",
       "      <td>Ġplaying</td>\n",
       "      <td>Ġat</td>\n",
       "      <td>Ġcentre</td>\n",
       "      <td>Ġ.</td>\n",
       "      <td>&lt;/s&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>labels</th>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>preds</th>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>I-LOC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>I-LOC</td>\n",
       "      <td>O</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>losses</th>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.39</td>\n",
       "      <td>3.07</td>\n",
       "      <td>4.28</td>\n",
       "      <td>3.30</td>\n",
       "      <td>3.66</td>\n",
       "      <td>3.95</td>\n",
       "      <td>7.51</td>\n",
       "      <td>9.71</td>\n",
       "      <td>9.13</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          0     1          2     3     4     5     6      7       8   \\\n",
       "tokens   ĠHe  Ġwas  Ġselected   Ġto   Ġgo   Ġon  Ġthe  Ġ2002    ĠNew   \n",
       "labels     O     O          O     O     O     O     O      O  I-MISC   \n",
       "preds      O     O          O     O     O     O     O      O   B-LOC   \n",
       "losses  0.00  0.00       0.00  0.00  0.00  0.00  0.00   0.39    3.07   \n",
       "\n",
       "              9       10       11      12      13      14        15    16  \\\n",
       "tokens  ĠZealand  Ġrugby  Ġleague   Ġtour     Ġof  ĠGreat  ĠBritain  Ġand   \n",
       "labels    I-MISC  I-MISC   I-MISC  I-MISC  I-MISC  I-MISC    I-MISC     O   \n",
       "preds      I-LOC       O        O       O       O   B-LOC     I-LOC     O   \n",
       "losses      4.28    3.30     3.66    3.95    7.51    9.71      9.13  0.00   \n",
       "\n",
       "             17    18        19    20       21    22    23  \n",
       "tokens  ĠFrance    Ġ,  Ġplaying   Ġat  Ġcentre    Ġ.  </s>  \n",
       "labels    B-LOC     O         O     O        O     O   IGN  \n",
       "preds     B-LOC     O         O     O        O     O     O  \n",
       "losses     0.00  0.00      0.00  0.00     0.00  0.00  0.00  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "      <th>14</th>\n",
       "      <th>15</th>\n",
       "      <th>16</th>\n",
       "      <th>17</th>\n",
       "      <th>18</th>\n",
       "      <th>19</th>\n",
       "      <th>20</th>\n",
       "      <th>21</th>\n",
       "      <th>22</th>\n",
       "      <th>23</th>\n",
       "      <th>24</th>\n",
       "      <th>25</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>tokens</th>\n",
       "      <td>ĠLocated</td>\n",
       "      <td>Ġin</td>\n",
       "      <td>ĠCardiff</td>\n",
       "      <td>Ġ,</td>\n",
       "      <td>Ġit</td>\n",
       "      <td>Ġis</td>\n",
       "      <td>Ġthe</td>\n",
       "      <td>Ġhome</td>\n",
       "      <td>Ġof</td>\n",
       "      <td>Ġthe</td>\n",
       "      <td>ĠWales</td>\n",
       "      <td>Ġnational</td>\n",
       "      <td>Ġrugby</td>\n",
       "      <td>Ġunion</td>\n",
       "      <td>Ġteam</td>\n",
       "      <td>Ġand</td>\n",
       "      <td>Ġhas</td>\n",
       "      <td>Ġalso</td>\n",
       "      <td>Ġheld</td>\n",
       "      <td>ĠWales</td>\n",
       "      <td>Ġnational</td>\n",
       "      <td>Ġfootball</td>\n",
       "      <td>Ġteam</td>\n",
       "      <td>Ġgames</td>\n",
       "      <td>Ġ.</td>\n",
       "      <td>&lt;/s&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>labels</th>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-ORG</td>\n",
       "      <td>I-ORG</td>\n",
       "      <td>I-ORG</td>\n",
       "      <td>I-ORG</td>\n",
       "      <td>I-ORG</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-ORG</td>\n",
       "      <td>I-ORG</td>\n",
       "      <td>I-ORG</td>\n",
       "      <td>I-ORG</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>preds</th>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>losses</th>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>5.62</td>\n",
       "      <td>4.43</td>\n",
       "      <td>4.60</td>\n",
       "      <td>3.47</td>\n",
       "      <td>3.82</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>5.50</td>\n",
       "      <td>4.59</td>\n",
       "      <td>5.24</td>\n",
       "      <td>4.90</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              0     1         2     3     4     5     6      7     8     9   \\\n",
       "tokens  ĠLocated   Ġin  ĠCardiff    Ġ,   Ġit   Ġis  Ġthe  Ġhome   Ġof  Ġthe   \n",
       "labels         O     O     B-LOC     O     O     O     O      O     O     O   \n",
       "preds          O     O     B-LOC     O     O     O     O      O     O     O   \n",
       "losses      0.00  0.00      0.00  0.00  0.00  0.00  0.00   0.00  0.00  0.00   \n",
       "\n",
       "            10         11      12      13     14    15    16     17     18  \\\n",
       "tokens  ĠWales  Ġnational  Ġrugby  Ġunion  Ġteam  Ġand  Ġhas  Ġalso  Ġheld   \n",
       "labels   B-ORG      I-ORG   I-ORG   I-ORG  I-ORG     O     O      O      O   \n",
       "preds    B-LOC          O       O       O      O     O     O      O      O   \n",
       "losses    5.62       4.43    4.60    3.47   3.82  0.00  0.00   0.00   0.00   \n",
       "\n",
       "            19         20         21     22      23    24    25  \n",
       "tokens  ĠWales  Ġnational  Ġfootball  Ġteam  Ġgames    Ġ.  </s>  \n",
       "labels   B-ORG      I-ORG      I-ORG  I-ORG       O     O   IGN  \n",
       "preds    B-LOC          O          O      O       O     O     O  \n",
       "losses    5.50       4.59       5.24   4.90    0.00  0.00  0.00  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "      <th>14</th>\n",
       "      <th>15</th>\n",
       "      <th>16</th>\n",
       "      <th>17</th>\n",
       "      <th>18</th>\n",
       "      <th>19</th>\n",
       "      <th>20</th>\n",
       "      <th>21</th>\n",
       "      <th>22</th>\n",
       "      <th>23</th>\n",
       "      <th>24</th>\n",
       "      <th>25</th>\n",
       "      <th>26</th>\n",
       "      <th>27</th>\n",
       "      <th>28</th>\n",
       "      <th>29</th>\n",
       "      <th>30</th>\n",
       "      <th>31</th>\n",
       "      <th>32</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>tokens</th>\n",
       "      <td>ĠOn</td>\n",
       "      <td>ĠMay</td>\n",
       "      <td>Ġ11</td>\n",
       "      <td>Ġ,</td>\n",
       "      <td>Ġ2018</td>\n",
       "      <td>Ġ,</td>\n",
       "      <td>Ġit</td>\n",
       "      <td>Ġwas</td>\n",
       "      <td>Ġannounced</td>\n",
       "      <td>Ġthat</td>\n",
       "      <td>ĠThe</td>\n",
       "      <td>ĠCW</td>\n",
       "      <td>Ġordered</td>\n",
       "      <td>Ġ\"</td>\n",
       "      <td>ĠRos</td>\n",
       "      <td>well</td>\n",
       "      <td>Ġ,</td>\n",
       "      <td>ĠNew</td>\n",
       "      <td>ĠMexico</td>\n",
       "      <td>Ġ\"</td>\n",
       "      <td>Ġto</td>\n",
       "      <td>Ġseries</td>\n",
       "      <td>Ġand</td>\n",
       "      <td>Ġit</td>\n",
       "      <td>Ġlater</td>\n",
       "      <td>Ġpremiered</td>\n",
       "      <td>Ġon</td>\n",
       "      <td>ĠJanuary</td>\n",
       "      <td>Ġ15</td>\n",
       "      <td>Ġ,</td>\n",
       "      <td>Ġ2019</td>\n",
       "      <td>Ġ.</td>\n",
       "      <td>&lt;/s&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>labels</th>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-ORG</td>\n",
       "      <td>I-ORG</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>IGN</td>\n",
       "      <td>O</td>\n",
       "      <td>B-LOC</td>\n",
       "      <td>I-LOC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>IGN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>preds</th>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-ORG</td>\n",
       "      <td>I-ORG</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>B-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>I-MISC</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>losses</th>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>8.10</td>\n",
       "      <td>0.00</td>\n",
       "      <td>8.77</td>\n",
       "      <td>10.88</td>\n",
       "      <td>10.25</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          0     1     2     3      4     5     6     7           8      9   \\\n",
       "tokens   ĠOn  ĠMay   Ġ11    Ġ,  Ġ2018    Ġ,   Ġit  Ġwas  Ġannounced  Ġthat   \n",
       "labels     O     O     O     O      O     O     O     O           O      O   \n",
       "preds      O     O     O     O      O     O     O     O           O      O   \n",
       "losses  0.00  0.00  0.00  0.00   0.00  0.00  0.00  0.00        0.00   0.00   \n",
       "\n",
       "           10     11        12    13      14      15      16      17       18  \\\n",
       "tokens   ĠThe    ĠCW  Ġordered    Ġ\"    ĠRos    well      Ġ,    ĠNew  ĠMexico   \n",
       "labels  B-ORG  I-ORG         O     O   B-LOC     IGN       O   B-LOC    I-LOC   \n",
       "preds   B-ORG  I-ORG         O     O  B-MISC  I-MISC  I-MISC  I-MISC   I-MISC   \n",
       "losses   0.00   0.00      0.00  0.00    8.10    0.00    8.77   10.88    10.25   \n",
       "\n",
       "          19    20       21    22    23      24          25    26        27  \\\n",
       "tokens    Ġ\"   Ġto  Ġseries  Ġand   Ġit  Ġlater  Ġpremiered   Ġon  ĠJanuary   \n",
       "labels     O     O        O     O     O       O           O     O         O   \n",
       "preds      O     O        O     O     O       O           O     O         O   \n",
       "losses  0.00  0.00     0.00  0.00  0.00    0.00        0.00  0.00      0.00   \n",
       "\n",
       "          28    29     30    31    32  \n",
       "tokens   Ġ15    Ġ,  Ġ2019    Ġ.  </s>  \n",
       "labels     O     O      O     O   IGN  \n",
       "preds      O     O      O     O     O  \n",
       "losses  0.00  0.00   0.00  0.00  0.00  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def get_samples(df):\n",
    "    for _, row in df.iterrows():\n",
    "        labels, preds, tokens, losses = [], [], [], []\n",
    "        for i, mask in enumerate(row[\"attention_mask\"]):\n",
    "            if i not in {0, len(row[\"attention_mask\"])}:\n",
    "                labels.append(row[\"labels\"][i])\n",
    "                preds.append(row[\"predicted_label\"][i])\n",
    "                tokens.append(row[\"input_tokens\"][i])\n",
    "                losses.append(f\"{row['loss'][i]:.2f}\")\n",
    "        eval_df_tmp = pd.DataFrame({\"tokens\": tokens, \"labels\": labels, \n",
    "                               \"preds\": preds, \"losses\": losses}).T\n",
    "        yield eval_df_tmp\n",
    "\n",
    "eval_df[\"total_loss\"] = eval_df[\"loss\"].apply(sum)\n",
    "eval_df_tmp = eval_df.sort_values(by=\"total_loss\", ascending=False).head(3)\n",
    "\n",
    "pd.set_option('display.max_columns', None)\n",
    "\n",
    "for sample in get_samples(eval_df_tmp):\n",
    "    display(sample)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Notes & Other Takeaways From This Project\n",
    "\n",
    "****\n",
    "- This model is part of a comparison of Token Classification Models using this particular dataset.\n",
    "\n",
    "****"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Citations\n",
    "\n",
    "- Model Checkpoint\n",
    "\n",
    "    > @article{kim2021bert, title={I-BERT: Integer-only BERT Quantization}, author={Kim, Sehoon and Gholami, Amir and Yao, Zhewei and Mahoney, Michael W and Keutzer, Kurt}, \n",
    "  journal={arXiv preprint arXiv:2101.01321}, year={2021}}\n",
    "\n",
    "- Dataset\n",
    "\n",
    "    > @inproceedings{tedeschi-etal-2021-wikineural-combined, title = \"{W}iki{NE}u{R}al: {C}ombined Neural and Knowledge-based Silver Data Creation for Multilingual {NER}\", author = \"Tedeschi, Simone and Maiorca, Valentino and Campolungo, Niccol{\\`o} and Cecconi, Francesco and Navigli, Roberto\", booktitle = \"Findings of the Association for Computational Linguistics: EMNLP 2021\", month = nov, year = \"2021\", address = \"Punta Cana, Dominican Republic\", publisher = \"Association for Computational Linguistics\", url = \"https://aclanthology.org/2021.findings-emnlp.215\", pages = \"2521--2533\", abstract = \"Multilingual Named Entity Recognition (NER) is a key intermediate task which is needed in many areas of NLP. In this paper, we address the well-known issue of data scarcity in NER, especially relevant when moving to a multilingual scenario, and go beyond current approaches to the creation of multilingual silver data for the task. We exploit the texts of Wikipedia and introduce a new methodology based on the effective combination of knowledge-based approaches and neural models, together with a novel domain adaptation technique, to produce high-quality training corpora for NER. We evaluate our datasets extensively on standard benchmarks for NER, yielding substantial improvements up to 6 span-based F1-score points over previous state-of-the-art systems for data creation.\",}\n",
    "\n",
    "- Metric (PosEval)\n",
    "\n",
    "    > @article{scikit-learn, title={Scikit-learn: Machine Learning in {P}ython}, author={Pedregosa, F. and Varoquaux, G. and Gramfort, A. and Michel, V. and Thirion, B. and Grisel, O. and Blondel, M. and Prettenhofer, P. and Weiss, R. and Dubourg, V. and Vanderplas, J. and Passos, A. and Cournapeau, D. and Brucher, M. and Perrot, M. and Duchesnay, E.}, journal={Journal of Machine Learning Research}, volume={12}, pages={2825--2830}, year={2011}}"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "dl",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}