6_multilingual_sbert.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Finetuning a multilingual SBERT model\n",
    "\n",
    "In this notebook we will fintetune a sentence BERT model to output similar sentences from multiple languages in the same vector space. We will closely follow the paper [Making Monolingual Sentence Embeddings Multilingual using\n",
    "Knowledge Distillation](https://arxiv.org/pdf/2004.09813.pdf) by Nils Reimers and Iryna Gurevych.\n",
    "\n",
    "We will use knowledge distillation to train a student model to mimic a monolingual teacher model which has been finetuned to produce good sentence embeddings. As long as we have parallel data in source and target language and a finetuned teacher model in either the source or target language, this technique can easily be used to extend the student models with relatively small amount of training data.\n",
    "\n",
    "| ![](assets/multilingual_sbert.png) | \n",
    "|:--:| \n",
    "| Fig. 1. Illustration of Multilingual SBERT knowledge distillation. (Image source: https://www.sbert.net/examples/training/multilingual/README.html) |\n",
    "<br>\n",
    "\n",
    "In the above figure, the student model should map `Hello World` and the German translation `Hallo Welt` to the vector of teacher model(`Hello World`). Authors recommend using Mean Squared Error loss for finetuning.\n",
    "\n",
    "Formally, for a `batch size B`, `teacher model M` and `student model Mhat` with parallel(translated) pairs of sentences `((s1, t1), (s2, t2), ... , (si, ti))` where `s` is the source language and `t` is the target language, the loss looks like\n",
    "\n",
    "$$ \\frac{1}{|B|} \\sum_{j\\in B} [(M(s_j) - \\hat{M}(s_j))^2 + (M(s_j) - \\hat{M}(t_j))^2] $$\n",
    "\n",
    "It's recommended to use an English SBERT model as the teacher and XLMRoBERTa as the student model. We are using a cross lingual model as the student to take advantage of it's multilingual vocabulary. XLM-R has a vocab size of 250k from 100 different languages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "from typing import Callable, Iterable\n",
    "\n",
    "import datasets\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import torch\n",
    "from torch.utils.data import Dataset, DataLoader, random_split\n",
    "from tqdm import tqdm\n",
    "from transformers import AutoTokenizer, AutoModel\n",
    "from transformers.optimization import get_linear_schedule_with_warmup\n",
    "from scipy import stats\n",
    "from sklearn.metrics.pairwise import cosine_similarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "teacher_model_name = \"sentence-transformers/paraphrase-distilroberta-base-v2\"\n",
    "student_model_name = \"xlm-roberta-base\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "teacher_tokenizer = AutoTokenizer.from_pretrained(teacher_model_name)\n",
    "student_tokenizer = AutoTokenizer.from_pretrained(student_model_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus = [\n",
    "    \"The roads on the highway were bad.\",\n",
    "    \"My car broke down last night\",\n",
    "    \"किसने कुत्तों को खुला छोड़ा\",\n",
    "    \"Where are my dogs?\",\n",
    "    \"Are the dogs outside?\",\n",
    "    \"मेरी कार टूट गई\",\n",
    "    \"मेरी कार हाईवे पर खराब हो गई\",\n",
    "    \"I wonder if they have reached home\",\n",
    "    \"मुझे आश्चर्य है कि क्या वे घर पहुंच गए हैं\"\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['The', 'Ġroads', 'Ġon', 'Ġthe', 'Ġhighway', 'Ġwere', 'Ġbad', '.']\n",
      "['My', 'Ġcar', 'Ġbroke', 'Ġdown', 'Ġlast', 'Ġnight']\n",
      "['à¤', 'ķ', 'à¤', '¿', 'à¤', '¸', 'à¤', '¨', 'à¥', 'ĩ', 'Ġà¤', 'ķ', 'à¥', 'ģ', 'à¤', '¤', 'à¥', 'į', 'à¤', '¤', 'à¥', 'ĭ', 'à¤', 'Ĥ', 'Ġà¤', 'ķ', 'à¥', 'ĭ', 'Ġà¤', 'ĸ', 'à¥', 'ģ', 'à¤', '²', 'à¤¾', 'Ġà¤', 'Ľ', 'à¥', 'ĭ', 'à¤', '¡', 'à¤', '¼', 'à¤¾']\n",
      "['Where', 'Ġare', 'Ġmy', 'Ġdogs', '?']\n",
      "['Are', 'Ġthe', 'Ġdogs', 'Ġoutside', '?']\n",
      "['à¤', '®', 'à¥', 'ĩ', 'à¤', '°', 'à¥', 'Ģ', 'Ġà¤', 'ķ', 'à¤¾', 'à¤', '°', 'Ġà¤', 'Ł', 'à¥', 'Ĥ', 'à¤', 'Ł', 'Ġà¤', 'Ĺ', 'à¤', 'Ī']\n",
      "['à¤', '®', 'à¥', 'ĩ', 'à¤', '°', 'à¥', 'Ģ', 'Ġà¤', 'ķ', 'à¤¾', 'à¤', '°', 'Ġà¤', '¹', 'à¤¾', 'à¤', 'Ī', 'à¤', 'µ', 'à¥', 'ĩ', 'Ġà¤', 'ª', 'à¤', '°', 'Ġà¤', 'ĸ', 'à¤', '°', 'à¤¾', 'à¤', '¬', 'Ġà¤', '¹', 'à¥', 'ĭ', 'Ġà¤', 'Ĺ', 'à¤', 'Ī']\n",
      "['I', 'Ġwonder', 'Ġif', 'Ġthey', 'Ġhave', 'Ġreached', 'Ġhome']\n",
      "['à¤', '®', 'à¥', 'ģ', 'à¤', 'Ŀ', 'à¥', 'ĩ', 'Ġà¤', 'Ĩ', 'à¤', '¶', 'à¥', 'į', 'à¤', 'ļ', 'à¤', '°', 'à¥', 'į', 'à¤', '¯', 'Ġà¤', '¹', 'à¥', 'Ī', 'Ġà¤', 'ķ', 'à¤', '¿', 'Ġà¤', 'ķ', 'à¥', 'į', 'à¤', '¯', 'à¤¾', 'Ġà¤', 'µ', 'à¥', 'ĩ', 'Ġà¤', 'ĺ', 'à¤', '°', 'Ġà¤', 'ª', 'à¤', '¹', 'à¥', 'ģ', 'à¤', 'Ĥ', 'à¤', 'ļ', 'Ġà¤', 'Ĺ', 'à¤', 'ı', 'Ġà¤', '¹', 'à¥', 'Ī', 'à¤', 'Ĥ']\n"
     ]
    }
   ],
   "source": [
    "for text in corpus:\n",
    "    print(teacher_tokenizer.tokenize(text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['▁The', '▁road', 's', '▁on', '▁the', '▁high', 'way', '▁were', '▁bad', '.']\n",
      "['▁My', '▁car', '▁bro', 'ke', '▁down', '▁last', '▁night']\n",
      "['▁किस', 'ने', '▁कु', 'त्त', 'ों', '▁को', '▁खुला', '▁छोड़', 'ा']\n",
      "['▁Where', '▁are', '▁my', '▁dog', 's', '?']\n",
      "['▁Are', '▁the', '▁dog', 's', '▁outside', '?']\n",
      "['▁मेरी', '▁कार', '▁टूट', '▁गई']\n",
      "['▁मेरी', '▁कार', '▁हाई', 'वे', '▁पर', '▁खराब', '▁हो', '▁गई']\n",
      "['▁I', '▁wonder', '▁if', '▁they', '▁have', '▁reached', '▁home']\n",
      "['▁मुझे', '▁आश्चर्य', '▁है', '▁कि', '▁क्या', '▁वे', '▁घर', '▁पहुंच', '▁गए', '▁हैं']\n"
     ]
    }
   ],
   "source": [
    "for text in corpus:\n",
    "    print(student_tokenizer.tokenize(text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can clearly see that the teacher model with just the english vocabulary does a very bad job in tokenizing texts in hindi. The hindi text has been broken down into individual characters which is hardly useful for representation purposes.\n",
    "\n",
    "The XLM-R model on the other hand does a great job in tokenizing both the hindi and english texts. It's clear that common hindi tokens are present in it's vocab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "seed = 10\n",
    "\n",
    "random.seed(seed)\n",
    "np.random.seed(seed)\n",
    "\n",
    "torch.manual_seed(seed)\n",
    "torch.cuda.manual_seed_all(seed)\n",
    "\n",
    "torch.backends.cudnn.deterministic = True\n",
    "torch.backends.cudnn.benchmark = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data preparation\n",
    "\n",
    "Note that we will be using relatively small dataset here as we need to fit two models in gpu memory hence limiting our batch size hence leading to much larger training times on a single GPU(3080 in my case)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def is_valid_len(d):\n",
    "    d = d[\"translation\"]\n",
    "    return len(d[\"en\"]) > 30 and len(d[\"en\"]) < 64 and len(d[\"hi\"]) > 20 and len(d[\"hi\"]) < 64"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using custom data configuration cfilt--iitb-english-hindi-911387c6837f8b91\n",
      "Reusing dataset parquet (/home/utsav/.cache/huggingface/datasets/cfilt___parquet/cfilt--iitb-english-hindi-911387c6837f8b91/0.0.0/7328ef7ee03eaf3f86ae40594d46a1cec86161704e02dd19f232d81eee72ade8)\n",
      "Parameter 'function'=<function <lambda> at 0x7fa196bc6ca0> of the transform datasets.arrow_dataset.Dataset.filter@2.0.1 couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.\n",
      "Loading cached processed dataset at /home/utsav/.cache/huggingface/datasets/cfilt___parquet/cfilt--iitb-english-hindi-911387c6837f8b91/0.0.0/7328ef7ee03eaf3f86ae40594d46a1cec86161704e02dd19f232d81eee72ade8/cache-08577eb1924770d3.arrow\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(133571,\n",
       " {'translation': {'en': 'The default plugin layout for the top panel',\n",
       "   'hi': 'ऊपरी पटल के लिए डिफोल्ट प्लग-इन खाका'}})"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset = datasets.load_dataset(\"cfilt/iitb-english-hindi\", split=\"train\")\n",
    "\n",
    "dataset = dataset.filter(lambda x: True if is_valid_len(x) and (random.random() > 0.9) else False)\n",
    "\n",
    "len(dataset), dataset[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1min 1s, sys: 8.44 s, total: 1min 9s\n",
      "Wall time: 34.4 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "teacher_tokenized_en = teacher_tokenizer([data[\"translation\"][\"en\"] for data in dataset],\n",
    "                                         max_length=64, padding=\"max_length\",\n",
    "                                         truncation=True, verbose=True)\n",
    "\n",
    "student_tokenized_en = student_tokenizer([data[\"translation\"][\"en\"] for data in dataset],\n",
    "                                         max_length=64, padding=\"max_length\",\n",
    "                                         truncation=True, verbose=True)\n",
    "\n",
    "student_tokenized_hi = student_tokenizer([data[\"translation\"][\"hi\"] for data in dataset],\n",
    "                                         max_length=64, padding=\"max_length\",\n",
    "                                         truncation=True, verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will create out custom dataset and dataloaders with train/validation split for training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "class EnHiDataset:\n",
    "    def __init__(\n",
    "        self, teacher_en_tokens: dict, student_en_tokens: dict, student_hi_tokens: dict\n",
    "    ):\n",
    "        self.teacher_en_tokens = teacher_en_tokens\n",
    "        self.student_en_tokens = student_en_tokens\n",
    "        self.student_hi_tokens = student_hi_tokens\n",
    "\n",
    "    def __getitem__(self, ix: int) -> dict[str, torch.tensor]:\n",
    "        return {\n",
    "            \"teacher_en_input_ids\": torch.tensor(\n",
    "                self.teacher_en_tokens[\"input_ids\"][ix], dtype=torch.long\n",
    "            ),\n",
    "            \"teacher_en_attention_mask\": torch.tensor(\n",
    "                self.teacher_en_tokens[\"attention_mask\"][ix], dtype=torch.long\n",
    "            ),\n",
    "            \"student_en_input_ids\": torch.tensor(\n",
    "                self.student_en_tokens[\"input_ids\"][ix], dtype=torch.long\n",
    "            ),\n",
    "            \"student_en_attention_mask\": torch.tensor(\n",
    "                self.student_en_tokens[\"attention_mask\"][ix], dtype=torch.long\n",
    "            ),\n",
    "            \"student_hi_input_ids\": torch.tensor(\n",
    "                self.student_hi_tokens[\"input_ids\"][ix], dtype=torch.long\n",
    "            ),\n",
    "            \"student_hi_attention_mask\": torch.tensor(\n",
    "                self.student_hi_tokens[\"attention_mask\"][ix], dtype=torch.long\n",
    "            )\n",
    "        }\n",
    "\n",
    "    def __len__(self) -> int:\n",
    "        return len(self.teacher_en_tokens[\"input_ids\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 0 ns, sys: 6.19 ms, total: 6.19 ms\n",
      "Wall time: 6.18 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "dataset = EnHiDataset(teacher_tokenized_en, student_tokenized_en, student_tokenized_hi)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "133571"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_ratio = 0.8\n",
    "n_total = len(dataset)\n",
    "n_train = int(n_total * train_ratio)\n",
    "n_val = n_total - n_train\n",
    "\n",
    "train_dataset, val_dataset = random_split(dataset, [n_train, n_val])\n",
    "\n",
    "batch_size = 2\n",
    "train_dataloader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)\n",
    "val_dataloader = DataLoader(val_dataset, batch_size=batch_size, shuffle=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'cuda'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
    "device"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The method `mean_pool()` implements the mean token pooling strategy mentioned in the paper. The implementation has been picked up from the `sentence_transformers` library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def mean_pool(token_embeds: torch.tensor, attention_mask: torch.tensor) -> torch.tensor:\n",
    "    in_mask = attention_mask.unsqueeze(-1).expand(token_embeds.size()).float()\n",
    "    pool = torch.sum(token_embeds * in_mask, 1) / torch.clamp(in_mask.sum(1), min=1e-9)\n",
    "    return pool"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at xlm-roberta-base were not used when initializing XLMRobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.layer_norm.weight', 'lm_head.dense.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight']\n",
      "- This IS expected if you are initializing XLMRobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing XLMRobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
     ]
    }
   ],
   "source": [
    "teacher_model = AutoModel.from_pretrained(teacher_model_name).to(device)\n",
    "student_model = AutoModel.from_pretrained(student_model_name).to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "#  optimizer, lr, num_warmup steps have been picked from the paper\n",
    "optimizer = torch.optim.Adam(student_model.parameters(), lr=2e-5)\n",
    "total_steps = len(train_dataset) // batch_size\n",
    "warmup_steps = int(0.1 * total_steps)\n",
    "scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=warmup_steps,\n",
    "                                            num_training_steps=total_steps - warmup_steps)\n",
    "\n",
    "loss_fn = torch.nn.MSELoss()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_train_step_fn(\n",
    "    teacher_model: torch.nn.Module, student_model: torch.nn.Module,\n",
    "    optimizer: torch.optim.Optimizer, scheduler: torch.optim.lr_scheduler.LambdaLR,\n",
    "    loss_fn: torch.nn.CrossEntropyLoss\n",
    ") -> Callable[[torch.tensor], float]:\n",
    "\n",
    "    def train_step_fn(d: torch.tensor) -> float:\n",
    "        student_model.train()\n",
    "\n",
    "        teacher_model_out = teacher_model(\n",
    "            d[\"teacher_en_input_ids\"].to(device), d[\"teacher_en_attention_mask\"].to(device)\n",
    "        )\n",
    "        teacher_en_pooled = mean_pool(\n",
    "            teacher_model_out.last_hidden_state, d[\"teacher_en_attention_mask\"].to(device)\n",
    "        )\n",
    "\n",
    "        student_en_model_out = student_model(\n",
    "            d[\"student_en_input_ids\"].to(device), d[\"student_en_attention_mask\"].to(device)\n",
    "        )\n",
    "        student_hi_model_out = student_model(\n",
    "            d[\"student_hi_input_ids\"].to(device), d[\"student_hi_attention_mask\"].to(device)\n",
    "        )\n",
    "        student_en_pooled = mean_pool(\n",
    "            student_en_model_out.last_hidden_state, d[\"student_en_attention_mask\"].to(device)\n",
    "        )\n",
    "        student_hi_pooled = mean_pool(\n",
    "            student_hi_model_out.last_hidden_state, d[\"student_hi_attention_mask\"].to(device)\n",
    "        )\n",
    "        \n",
    "        loss = (loss_fn(student_en_pooled, teacher_en_pooled) +\n",
    "                loss_fn(student_hi_pooled, teacher_en_pooled)) / 2\n",
    "\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        scheduler.step()\n",
    "        optimizer.zero_grad()\n",
    "        return loss.item()\n",
    "\n",
    "    return train_step_fn\n",
    "\n",
    "\n",
    "def get_val_step_fn(\n",
    "    teacher_model: torch.nn.Module, student_model: torch.nn.Module,\n",
    "    loss_fn: torch.nn.CrossEntropyLoss\n",
    ") -> Callable[[torch.tensor], float]:\n",
    "\n",
    "    def val_step_fn(d: torch.tensor) -> float:\n",
    "        student_model.eval()\n",
    "\n",
    "        teacher_model_out = teacher_model(\n",
    "            d[\"teacher_en_input_ids\"].to(device), d[\"teacher_en_attention_mask\"].to(device)\n",
    "        )\n",
    "        teacher_en_pooled = mean_pool(\n",
    "            teacher_model_out.last_hidden_state, d[\"teacher_en_attention_mask\"].to(device)\n",
    "        )\n",
    "\n",
    "        student_en_model_out = student_model(\n",
    "            d[\"student_en_input_ids\"].to(device), d[\"student_en_attention_mask\"].to(device)\n",
    "        )\n",
    "        student_hi_model_out = student_model(\n",
    "            d[\"student_hi_input_ids\"].to(device), d[\"student_hi_attention_mask\"].to(device)\n",
    "        )\n",
    "        student_en_pooled = mean_pool(\n",
    "            student_en_model_out.last_hidden_state, d[\"student_en_attention_mask\"].to(device)\n",
    "        )\n",
    "        student_hi_pooled = mean_pool(\n",
    "            student_hi_model_out.last_hidden_state, d[\"student_hi_attention_mask\"].to(device)\n",
    "        )\n",
    "        \n",
    "        loss = (loss_fn(student_en_pooled, teacher_en_pooled) +\n",
    "                loss_fn(student_hi_pooled, teacher_en_pooled)) / 2\n",
    "\n",
    "        return loss.item()\n",
    "\n",
    "    return val_step_fn\n",
    "\n",
    "\n",
    "def mini_batch(\n",
    "    dataloader: DataLoader,\n",
    "    step_fn: Callable[[torch.tensor], float],\n",
    "    is_training: bool = True\n",
    ") -> tuple[np.array, list[float]]:\n",
    "\n",
    "    mini_batch_losses = []\n",
    "\n",
    "    if is_training:\n",
    "        print(\"\\nTraining ...\")\n",
    "    else:\n",
    "        print(\"\\nValidating ...\")\n",
    "    n_steps = len(dataloader)\n",
    "    for i, data in enumerate(dataloader):\n",
    "        loss = step_fn(data)\n",
    "        mini_batch_losses.append(loss)\n",
    "        if i % (batch_size * 1000) == 0:\n",
    "            print(f\"step {i:>5}/{n_steps}, loss = {loss: .3f}\")\n",
    "\n",
    "    return np.mean(mini_batch_losses), mini_batch_losses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Training ...\n",
      "step     0/53428, loss =  0.683\n",
      "step  2000/53428, loss =  0.303\n",
      "step  4000/53428, loss =  0.331\n",
      "step  6000/53428, loss =  0.281\n",
      "step  8000/53428, loss =  0.258\n",
      "step 10000/53428, loss =  0.246\n",
      "step 12000/53428, loss =  0.259\n",
      "step 14000/53428, loss =  0.235\n",
      "step 16000/53428, loss =  0.327\n",
      "step 18000/53428, loss =  0.180\n",
      "step 20000/53428, loss =  0.129\n",
      "step 22000/53428, loss =  0.144\n",
      "step 24000/53428, loss =  0.167\n",
      "step 26000/53428, loss =  0.209\n",
      "step 28000/53428, loss =  0.140\n",
      "step 30000/53428, loss =  0.187\n",
      "step 32000/53428, loss =  0.167\n",
      "step 34000/53428, loss =  0.173\n",
      "step 36000/53428, loss =  0.149\n",
      "step 38000/53428, loss =  0.176\n",
      "step 40000/53428, loss =  0.189\n",
      "step 42000/53428, loss =  0.170\n",
      "step 44000/53428, loss =  0.240\n",
      "step 46000/53428, loss =  0.191\n",
      "step 48000/53428, loss =  0.201\n",
      "step 50000/53428, loss =  0.157\n",
      "step 52000/53428, loss =  0.116\n",
      "\n",
      "Validating ...\n",
      "step     0/13358, loss =  0.200\n",
      "step  2000/13358, loss =  0.198\n",
      "step  4000/13358, loss =  0.241\n",
      "step  6000/13358, loss =  0.177\n",
      "step  8000/13358, loss =  0.175\n",
      "step 10000/13358, loss =  0.190\n",
      "step 12000/13358, loss =  0.155\n",
      "CPU times: user 1h 9min 30s, sys: 12min 53s, total: 1h 22min 24s\n",
      "Wall time: 1h 22min 20s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "n_epochs = 1\n",
    "\n",
    "train_step_fn = get_train_step_fn(teacher_model, student_model, optimizer, scheduler, loss_fn)\n",
    "val_step_fn = get_val_step_fn(teacher_model, student_model, loss_fn)\n",
    "\n",
    "train_losses, train_mini_batch_losses = [], []\n",
    "val_losses, val_mini_batch_losses = [], []\n",
    "\n",
    "for epoch in range(1, n_epochs + 1):\n",
    "    train_loss, _train_mini_batch_losses = mini_batch(train_dataloader, train_step_fn)\n",
    "    train_mini_batch_losses += _train_mini_batch_losses\n",
    "    train_losses.append(train_loss)\n",
    "\n",
    "    with torch.no_grad():\n",
    "        val_loss, _val_mini_batch_losses = mini_batch(val_dataloader, val_step_fn, is_training=False)\n",
    "        val_mini_batch_losses += _val_mini_batch_losses\n",
    "        val_losses.append(val_loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([0.19776925913485568], [0.15786257060084322])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_losses, val_losses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "student_model.save_pretrained(\"models/student_multi_enhi\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Normally we look at losses over multiple epochs, but here we have only 1 epoch. One way to look at the mini batch losses is to use a running mean(smoothing) to reduce noise from per batch loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "window_size = 32\n",
    "\n",
    "train_mb_running_loss = []\n",
    "for i in range(len(train_mini_batch_losses)-window_size):\n",
    "    train_mb_running_loss.append(np.mean(train_mini_batch_losses[i:i+window_size]))\n",
    "\n",
    "val_mb_running_loss = []\n",
    "for i in range(len(val_mini_batch_losses)-window_size):\n",
    "    val_mb_running_loss.append(np.mean(val_mini_batch_losses[i:i+window_size]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7fdff3e8d3d0>"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAzIAAAI/CAYAAACs3OxHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAACBd0lEQVR4nO3dd3hUVf7H8c+ZSYMEQoBQA4TeOwJKt1LsXWHtdfW3uq4Fe991V9d1Xduqa1l7XwsIiqKAIr333nsNkD7398eUTJ9JSJgZ8n49Dw8zd+7cOTO5gfuZc873GMuyBAAAAACJxBbrBgAAAABAeRFkAAAAACQcggwAAACAhEOQAQAAAJBwCDIAAAAAEg5BBgAAAEDCSYrVC9evX9/Kzc2N1csDAAAAiHNz5szZbVlWdrDHYhZkcnNzNXv27Fi9PAAAAIA4Z4zZEOoxhpYBAAAASDgEGQAAAAAJhyADAAAAIOHEbI4MAAAAUF7FxcXavHmzCgoKYt0UVKK0tDTl5OQoOTk56ucQZAAAAJAwNm/erFq1aik3N1fGmFg3B5XAsizt2bNHmzdvVsuWLaN+HkPLAAAAkDAKCgpUr149QsxxxBijevXqlbuXjSADAACAhEKIOf5U5GdKkAEAAACitH//fr300ksVeu7IkSO1f//+sPs89NBDmjRpUoWO7y83N1e7d++ulGPFI4IMAAAAEKVwQaa0tDTsc8ePH686deqE3eexxx7TqaeeWtHmVSsEGQAAACBKY8eO1Zo1a9SjRw/ddddd+umnnzRs2DBdfvnl6tq1qyTp3HPPVe/evdW5c2e9+uqrnue6e0jWr1+vjh076vrrr1fnzp11+umnKz8/X5J01VVX6dNPP/Xs//DDD6tXr17q2rWrli9fLknatWuXTjvtNPXq1Us33nijWrRoEbHn5dlnn1WXLl3UpUsXPffcc5Kkw4cPa9SoUerevbu6dOmijz76yPMeO3XqpG7duunOO+/0vOYFF1ygE044QSeccIJ++eUXSdLPP/+sHj16qEePHurZs6fy8vIq6ZOOjKplAAAAQJSeeuopLV68WPPnz5ck/fTTT5o5c6YWL17sqbj1xhtvqG7dusrPz9cJJ5ygCy64QPXq1fM5zqpVq/TBBx/otdde08UXX6zPPvtMY8aMCXi9+vXra+7cuXrppZf0zDPP6PXXX9ejjz6qk08+Wffee68mTJjgE5aCmTNnjt58803NmDFDlmWpX79+GjJkiNauXasmTZpo3LhxkqQDBw5o7969+uKLL7R8+XIZYzxD4W677Tb98Y9/1MCBA7Vx40adccYZWrZsmZ555hm9+OKLGjBggA4dOqS0tLSj/ISjR5ABAABAQnr06yVauvVgpR6zU5PaeviszuV6Tt++fX3KBj///PP64osvJEmbNm3SqlWrAoJMy5Yt1aNHD0lS7969tX79+qDHPv/88z37fP7555KkadOmeY4/fPhwZWVlhW3ftGnTdN555yk9Pd1zzKlTp2r48OG68847dc899+jMM8/UoEGDVFJSorS0NF133XUaNWqUzjzzTEnSpEmTtHTpUs8xDx48qLy8PA0YMEB33HGHRo8erfPPP185OTnRfGSVgqFlAAAAwFFwBwTJ2UMzadIkTZ8+XQsWLFDPnj2DlhVOTU313Lbb7SopKQl6bPd+3vtYllWu9oXav127dpozZ466du2qe++9V4899piSkpI0c+ZMXXDBBfrf//6n4cOHS5IcDoemT5+u+fPna/78+dqyZYtq1aqlsWPH6vXXX1d+fr769+/vGf52LNAjAwAAgIRU3p6TylCrVq2w80AOHDigrKws1axZU8uXL9dvv/1W6W0YOHCgPv74Y91zzz367rvvtG/fvrD7Dx48WFdddZXGjh0ry7L0xRdf6J133tHWrVtVt25djRkzRhkZGXrrrbd06NAhHTlyRCNHjlT//v3Vpk0bSdLpp5+uF154QXfddZckaf78+erRo4fWrFmjrl27qmvXrpo+fbqWL1+uDh06VPp7DoYgAwAAAESpXr16GjBggLp06aIRI0Zo1KhRPo8PHz5cr7zyirp166b27durf//+ld6Ghx9+WJdddpk++ugjDRkyRI0bN1atWrVC7t+rVy9dddVV6tu3ryTpuuuuU8+ePTVx4kTdddddstlsSk5O1ssvv6y8vDydc845KigokGVZ+sc//iHJOVzulltuUbdu3VRSUqLBgwfrlVde0XPPPafJkyfLbrerU6dOGjFiRKW/31BMebumKkufPn2s2bNnx+S1AQAAkJiWLVumjh07xroZMVVYWCi73a6kpCRNnz5dN998s6f4QCIL9rM1xsyxLKtPsP3pkQEAAAASyMaNG3XxxRfL4XAoJSVFr732WqybFBMEGQAAACCBtG3bVvPmzYt1M2KOqmUAAAAAEg5BBgAAAEDCIcgAAAAASDgEGQAAAAAJhyAjKa+gWLljx+mX1btj3RQAAAAcZzIyMiRJW7du1YUXXhh0n6FDhyrS0iTPPfecjhw54rk/cuRI7d+//6jb98gjj+iZZ5456uMcawQZSY9+vVSSNPr1GTFuCQAAAI5XTZo00aefflrh5/sHmfHjx6tOnTqV0LLERJCRtGjzgVg3AQAAAAngnnvu0UsvveS5/8gjj+jvf/+7Dh06pFNOOUW9evVS165d9eWXXwY8d/369erSpYskKT8/X5deeqm6deumSy65RPn5+Z79br75ZvXp00edO3fWww8/LEl6/vnntXXrVg0bNkzDhg2TJOXm5mr3bueIomeffVZdunRRly5d9Nxzz3ler2PHjrr++uvVuXNnnX766T6vE8z8+fPVv39/devWTeedd5727dvnef1OnTqpW7duuvTSSyVJP//8s3r06KEePXqoZ8+eysvLkyQ9/fTTOuGEE9StWzdP+w8fPqxRo0ape/fu6tKliz766KPyffBBsI6MpBU78mLdBAAAACSASy+9VLfffrt+//vfS5I+/vhjTZgwQWlpafriiy9Uu3Zt7d69W/3799fZZ58tY0zQ47z88suqWbOmFi5cqIULF6pXr16ex5588knVrVtXpaWlOuWUU7Rw4UL94Q9/0LPPPqvJkyerfv36PseaM2eO3nzzTc2YMUOWZalfv34aMmSIsrKytGrVKn3wwQd67bXXdPHFF+uzzz7TmDFjQr6/K664Qv/61780ZMgQPfTQQ3r00Uf13HPP6amnntK6deuUmprqGc72zDPP6MUXX9SAAQN06NAhpaWl6bvvvtOqVas0c+ZMWZals88+W1OmTNGuXbvUpEkTjRs3TpJ04MDRdyQQZCTVSktSXkFJrJsBAACA8pgwQdq+vXKP2aiRNHx4yId79uypnTt3auvWrdq1a5eysrLUvHlzFRcX67777tOUKVNks9m0ZcsW7dixQ40aNQp6nClTpugPf/iDJKlbt27q1q2b57GPP/5Yr776qkpKSrRt2zYtXbrU53F/06ZN03nnnaf09HRJ0vnnn6+pU6fq7LPPVsuWLdWjRw9JUu/evbV+/fqQxzlw4ID279+vIUOGSJKuvPJKXXTRRZ42jh49Wueee67OPfdcSdKAAQN0xx13aPTo0Tr//POVk5Oj7777Tt9995169uwpSTp06JBWrVqlQYMG6c4779Q999yjM888U4MGDQrZjmgxtEzSTUNaS5LO6dEkxi0BAABAvLvwwgv16aef6qOPPvIMs3rvvfe0a9cuzZkzR/Pnz1fDhg1VUFAQ9jjBemvWrVunZ555Rj/88IMWLlyoUaNGRTyOZVkhH0tNTfXcttvtKimp2Jf348aN0y233KI5c+aod+/eKikp0dixY/X6668rPz9f/fv31/Lly2VZlu69917Nnz9f8+fP1+rVq3XttdeqXbt2mjNnjrp27ap7771Xjz32WIXa4Y0eGUm1ayRLktJT+TgAAAASRpiek6p06aWX6vrrr9fu3bv1888/S3L2ZjRo0EDJycmaPHmyNmzYEPYYgwcP1nvvvadhw4Zp8eLFWrhwoSTp4MGDSk9PV2Zmpnbs2KFvv/1WQ4cOlSTVqlVLeXl5AUPLBg8erKuuukpjx46VZVn64osv9M4775T7fWVmZiorK0tTp07VoEGD9M4772jIkCFyOBzatGmThg0bpoEDB+r999/XoUOHtGfPHnXt2lVdu3bV9OnTtXz5cp1xxhl68MEHNXr0aGVkZGjLli1KTk5WSUmJ6tatqzFjxigjI0NvvfVWudvnjyt3AAAAoBw6d+6svLw8NW3aVI0bN5YkjR49WmeddZb69OmjHj16qEOHDmGPcfPNN+vqq69Wt27d1KNHD/Xt21eS1L17d/Xs2VOdO3dWq1atNGDAAM9zbrjhBo0YMUKNGzfW5MmTPdt79eqlq666ynOM6667Tj179gw7jCyUt99+WzfddJOOHDmiVq1a6c0331RpaanGjBmjAwcOyLIs/fGPf1SdOnX04IMPavLkybLb7erUqZNGjBih1NRULVu2TCeeeKIkZ+npd999V6tXr9Zdd90lm82m5ORkvfzyy+Vumz8TriuqKvXp08eKVCv7WHnntw168H+LNbpfcz15XtdYNwcAAAAhLFu2TB07dox1M1AFgv1sjTFzLMvqE2x/5sgAAAAASDgEGUmKUa8UAAAAgIohyEhykGMAAACAhEKQkVRKkgEAAEgYsZrjjapTkZ8pQUaSg18GAACAhJCWlqY9e/YQZo4jlmVpz549SktLK9fzKL+ssh6ZIGsSAQAAII7k5ORo8+bN2rVrV6ybgkqUlpamnJyccj2HICOplEQPAACQEJKTk9WyZctYNwNxgKFlkhzMkQEAAAASCkFGVC0DAAAAEg1BRtKz36+MdRMAAAAAlANBxgs9MwAAAEBiqPZB5khRied2aSlJBgAAAEgE1T7I2G1lNZeLHY4YtgQAAABAtKIKMsaY4caYFcaY1caYsUEev8sYM9/1Z7ExptQYU7fym1v5km1lH0EJPTIAAABAQogYZIwxdkkvShohqZOky4wxnbz3sSzracuyeliW1UPSvZJ+tixrbxW0t9LZvHtkSumRAQAAABJBND0yfSWttixrrWVZRZI+lHROmP0vk/RBZTTuWCumRwYAAABICNEEmaaSNnnd3+zaFsAYU1PScEmfHX3Tjr0S5sgAAAAACSGaIGOCbAvVdXGWpF9CDSszxtxgjJltjJm9a9euaNt4zDBHBgAAAEgM0QSZzZKaed3PkbQ1xL6XKsywMsuyXrUsq49lWX2ys7Ojb+UxsvdwUaybAAAAACAK0QSZWZLaGmNaGmNS5AwrX/nvZIzJlDRE0peV28RjZ+m2g7FuAgAAAIAoRAwylmWVSLpV0kRJyyR9bFnWEmPMTcaYm7x2PU/Sd5ZlHa6apladK09sEesmAAAAACiHpGh2sixrvKTxftte8bv/lqS3Kqthx9Kj53TRnsNF9MgAAAAACSKqBTGrA7vNqNTBZH8AAAAgERBkXAgyAAAAQOIgyLgkEWQAAACAhEGQcbHbbCohyAAAAAAJgSDjYreJHhkAAAAgQRBkXJJsNoIMAAAAkCAIMi5M9gcAAAASB0HGxW4zKnE4Yt0MAAAAAFEgyLjQIwMAAAAkDoKMS5LNULUMAAAASBAEGZckm02WReUyAAAAIBEQZFyS7EaSmCcDAAAAJACCjEuSzRlk6JEBAAAA4h9BxsXuCjLFpQQZAAAAIN4RZFyS7c6Pgh4ZAAAAIP4RZFzcPTIlpcyRAQAAAOIdQcYl2TXZv5geGQAAACDuEWRc7DbnR+EgyAAAAABxjyDj4poiw6KYAAAAQAIgyLi4e2SY7A8AAADEP4KMi92wjgwAAACQKAgyLnYWxAQAAAASBkHGhSADAAAAJA6CjIt7sn+pRZABAAAA4h1BxqVssj8LYgIAAADxjiDjUjbZP8YNAQAAABARQcbFPUemhB4ZAAAAIO4RZFzcQYYcAwAAAMQ/goyLp2oZk/0BAACAuEeQcSkrv0yXDAAAABDvCDIuSTYm+wMAAACJgiDjYjP0yAAAAACJgiDjkmSnRwYAAABIFAQZF3ePDOWXAQAAgPhHkHFxz5FxULUMAAAAiHsEGRd31bIt+/Jj3BIAAAAAkRBk/Dzz3cpYNwEAAABABAQZF5urRwYAAABA/CPIuJBjAAAAgMRBkHFhjj8AAACQOAgyLpk1kmPdBAAAAABRIsi4pKcmSZLqZ6TGuCUAAAAAIiHI+Nl9qDDWTQAAAAAQAUEGAAAAQMIhyAAAAABIOAQZAAAAAAmHIAMAAAAg4STFugHxpFPj2kz2BwAAABIAQcZLkzo1Yt0EAAAAAFFgaJkXm5EclhXrZgAAAACIgCDjxWaMyDEAAABA/CPIeLHZpFKSDAAAABD3CDJebMYwtAwAAABIAAQZLwwtAwAAABIDQcaLzUilDpIMAAAAEO8IMl5sNoaWAQAAAImAIOOFoWUAAABAYiDIeCkscWjL/vxYNwMAAABABAQZL18v2BrrJgAAAACIAkEGAAAAQMIhyAAAAABIOAQZAAAAAAmHIBOERekyAAAAIK4RZILYe7go1k0AAAAAEAZBJoiiUkesmwAAAAAgDIJMEIwsAwAAAOIbQcZLt5xMSVJqEh8LAAAAEM+4Yvdyfs+mkiRjTIxbAgAAACAcgowXm80ZYKhaBgAAAMQ3gowXd0+MgxwDAAAAxDWCjBf3gDJ6ZAAAAID4RpDxYnP1yBBjAAAAgPhGkPHimiIjBz0yAAAAQFwjyHgpcU2OKWWSDAAAABDXogoyxpjhxpgVxpjVxpixIfYZaoyZb4xZYoz5uXKbeWy8PnWtJOl/87bEuCUAAAAAwokYZIwxdkkvShohqZOky4wxnfz2qSPpJUlnW5bVWdJFld/Uqrd+zxFJ0nszNsa4JQAAAADCiaZHpq+k1ZZlrbUsq0jSh5LO8dvnckmfW5a1UZIsy9pZuc08trYdKIh1EwAAAACEEU2QaSppk9f9za5t3tpJyjLG/GSMmWOMuaKyGngstcpOlyR1y8mMcUsAAAAAhBNNkDFBtvnPhk+S1FvSKElnSHrQGNMu4EDG3GCMmW2Mmb1r165yN7aqvXBZL0nS9YNaxbglAAAAAMKJJshsltTM636OpK1B9plgWdZhy7J2S5oiqbv/gSzLetWyrD6WZfXJzs6uaJurTGqy8+PYdiA/xi0BAAAAEE40QWaWpLbGmJbGmBRJl0r6ym+fLyUNMsYkGWNqSuonaVnlNrXqOVxll/88fnmMWwIAAAAgnKRIO1iWVWKMuVXSREl2SW9YlrXEGHOT6/FXLMtaZoyZIGmhJIek1y3LWlyVDa8KLB8DAAAAJIaIQUaSLMsaL2m837ZX/O4/LenpymsaAAAAAAQX1YKY1YXDoksGAAAASAQEGS8EGQAAACAxEGS8kGMAAACAxECQ8UKQAQAAABIDQcZLo8y0WDcBAAAAQBQIMl6ya6XGugkAAAAAokCQAQAAAJBwCDJ+OjSqFesmAAAAAIggqgUxq5Oezetoz+GiWDcDAAAAQBj0yPix24xKHZQvAwAAAOIZQcZPks1GkAEAAADiHEHGDz0yAAAAQPwjyPhJshmVOByxbgYAAACAMAgyfuiRAQAAAOIfQcaPs0eGIAMAAADEM4KMH7vNJsuSHIQZAAAAIG4RZPxsP1ggSSpmngwAAAAQtwgyfj6YuVGStGJ7XoxbAgAAACAUgkwI+48Ux7oJAAAAAEIgyPg5tWMDSc5J/wAAAADiE0HGzwW9ciRJdWqmxLglAAAAAEIhyPhJsjs/EhbFBAAAAOIXQcZPkt05pIy1ZAAAAID4RZDxc7iwRJI0buG2GLcEAAAAQCgEGT9rdx2WJP1n2roYtwQAAABAKAQZP3aqlQEAAABxjyDjh7LLAAAAQPwjyPihRwYAAACIfwQZP8O7NJIk3TC4VYxbAgAAACAUgoyfWqnJkqSGtdNi3BIAAAAAoRBk/LjXkSllQUwAAAAgbhFk/LjnyBSXsiAmAAAAEK8IMn6S7c6PpNRBkAEAAADiFUHGj7to2fxN+2PaDgAAAAChEWT8GONMMj8u3xnjlgAAAAAIhSADAAAAIOEQZMJ4/odVsW4CAAAAgCAIMmE8+/3KWDcBAAAAQBAEGQAAAAAJhyADAAAAIOEQZAAAAAAkHIIMAAAAgIRDkAEAAACQcAgyAAAAABIOQQYAAABAwiHIAAAAAEg4BBkAAAAACYcgE0TTOjVi3QQAAAAAYRBkgtiyP99z27KsGLYEAAAAQDAEmSBGdW3suV3qIMgAAAAA8YYgE4wpu1lCkAEAAADiDkEmiOyMVM9tggwAAAAQfwgyQYwd0UEdGtWSJG3edyTGrQEAAADgjyATRFqyXZf1bS5JGv7c1Bi3BgAAAIA/gkwIdlvZRBkqlwEAAADxhSATQpJXkGGaDAAAABBfCDIhePfIlDgcMWwJAAAAAH8EmRCS7GVBZv+R4hi2BAAAAIA/gkwIdlvZR/PpnM0xbAkAAAAAfwSZELznyCzdejCGLQEAAADgjyATgvccmbzCkhi2BAAAAIA/gkwI3j0yU1buimFLAAAAAPgjyITg3SMDAAAAIL4QZEJIsvHRAAAAAPGKq/UQyDEAAABA/OJyPQSbYWgZAAAAEK8IMgAAAAASDkEGAAAAQMIhyAAAAABIOAQZAAAAAAmHIAMAAAAg4RBkAAAAACQcgkwIjWqnxboJAAAAAEIgyISQWz9dP9811HPfsqzYNQYAAACAD4JMGC3qpXtuO8gxAAAAQNyIKsgYY4YbY1YYY1YbY8YGeXyoMeaAMWa+689Dld/U2HLQIwMAAADEjYhBxhhjl/SipBGSOkm6zBjTKciuUy3L6uH681gltzNmLu6TI4kgAwAAAMSTaHpk+kpabVnWWsuyiiR9KOmcqm1W/Mit7xxeRo4BAAAA4kc0QaappE1e9ze7tvk70RizwBjzrTGmc6W0Lg7YjJFEkAEAAADiSVIU+5gg2/wv6+dKamFZ1iFjzEhJ/5PUNuBAxtwg6QZJat68eflaGiM217tnaBkAAAAQP6LpkdksqZnX/RxJW713sCzroGVZh1y3x0tKNsbU9z+QZVmvWpbVx7KsPtnZ2UfR7GPH3SNDkAEAAADiRzRBZpaktsaYlsaYFEmXSvrKewdjTCNjnFf8xpi+ruPuqezGxoLxBJkYNwQAAACAR8ShZZZllRhjbpU0UZJd0huWZS0xxtzkevwVSRdKutkYUyIpX9Kl1nGygqR7aNlx8nYAAACA40I0c2Tcw8XG+217xev2C5JeqNymxQcbPTIAAABA3IlqQczqjMn+AAAAQPwhyERgmOwPAAAAxB2CTASsIwMAAADEH4JMBIahZQAAAEDcIchEUFa1LLbtAAAAAFCGIBMBc2QAAACA+EOQiYA5MgAAAED8IchEQPllAAAAIP4QZCJgQUwAAAAg/hBkIqBqGQAAABB/CDIRlM2RIcgAAAAA8YIgEwFDywAAAID4Q5CJgMn+AAAAQPwhyETgmSPjiG07AAAAAJQhyETAgpgAAABA/CHIROCeIwMAAAAgfhBkImCODAAAABB/CDIRULUMAAAAiD8EmQhYEBMAAACIPwSZCFgQEwAAAIg/BJkIGFoGAAAAxB+CTASeyf4kGQAAACBuEGQiMPTIAAAAAHGHIBOBu0eGOTIAAABA/CDIRECPDAAAABB/CDIReHpkRJIBAAAA4gVBJgJ6ZAAAAID4Q5CJwMaCmAAAAEDcIchEwIKYAAAAQPwhyETgWRDTEeOGAAAAAPAgyERgGFoGAAAAxB2CTAQ2JvsDAAAAcYcgE4HN9QkxRwYAAACIHwSZCOiRAQAAAOIPQSYC1xQZ5sgAAAAAcYQgE0HZgpgEGQAAACBeEGQicC+ICQAAACB+EGQisNEjAwAAAMQdgkwELIgJAAAAxB+CTAQsiAkAAADEH4JMBDbXJBlyDAAAABA/CDIR2OiRAQAAAOIOQSYCFsQEAAAA4g9BJgLmyAAAAADxhyATgbtHxiLIAAAAAHGDIBOBez1MhpYBAAAA8YMgEwE9MgAAAED8IchEwGR/AAAAIP4QZCIwrk+Iyf4AAABA/CDIRFA2tCzGDQEAAADgQZCJgAUxAQAAgPhDkImAOTIAAABA/CHIRMCCmAAAAED8IchEQPllAAAAIP4QZCJgaBkAAAAQfwgyEbhGljG0DAAAAIgjBJkIyubIxLYdAAAAAMoQZCIwxjjDDD0yAAAAQNwgyETBZgw9MgAAAEAcIchEwWaYIwMAAADEE4JMFAw9MgAAAEBcIchEwWZYRwYAAACIJwSZKDjnyBBkAAAAgHhBkInCkaJSbdqbH+tmAAAAAHAhyERpwpLtsW4CAAAAABeCDAAAAICEQ5ABAAAAkHAIMgAAAAASDkEGAAAAQMIhyAAAAABIOEmxbkAi6JaTqbrpKbFuBgAAAAAXemSiYDNGpQ4WxAQAAADiBUEmCnabkcMiyAAAAADxgiATBTs9MgAAAEBciSrIGGOGG2NWGGNWG2PGhtnvBGNMqTHmwsprYuzZbJLDEetWAAAAAHCLGGSMMXZJL0oaIamTpMuMMZ1C7PdXSRMru5GxZrcZlTK0DAAAAIgb0fTI9JW02rKstZZlFUn6UNI5Qfb7P0mfSdpZie2LC0z2BwAAAOJLNEGmqaRNXvc3u7Z5GGOaSjpP0iuV17T4wWR/AAAAIL5EE2RMkG3+V/XPSbrHsqzSsAcy5gZjzGxjzOxdu3ZF2cTYY7I/AAAAEF+iWRBzs6RmXvdzJG3126ePpA+NMZJUX9JIY0yJZVn/897JsqxXJb0qSX369EmYZGCzEWQAAACAeBJNkJklqa0xpqWkLZIulXS59w6WZbV03zbGvCXpG/8Qk8jshqFlAAAAQDyJGGQsyyoxxtwqZzUyu6Q3LMtaYoy5yfX4cTkvxpudHhkAAAAgrkTTIyPLssZLGu+3LWiAsSzrqqNvVnyx2YzIMQAAAED8iGpBzOrObkSPDAAAABBHCDJRYLI/AAAAEF8IMlFgsj8AAAAQXwgyUWCyPwAAABBfCDJRcE72J8gAAAAA8YIgEwW7oUcGAAAAiCcEmSgwtAwAAACILwSZKNjokQEAAADiCkEmCnabVOKw9K8fVmnd7sOxbg4AAABQ7RFkomCzGRWWOPT371dq9Gu/xbo5AAAAQLVHkImC3RjP7SPFpTFsCQAAAACJIBMVu60syDiYKwMAAADEHEEmCjavHhliDAAAABB7BJkoePfIkGQAAACA2CPIRME7yJBjAAAAgNgjyETBp0cGAAAAQMwRZKLgXbUMAAAAQOwRZKJgo0cGAAAAiCsEmSjYvef6W8ySAQAAAGKNIBMFJvsDAAAA8YUgE4U9h4s8t48UlcawJQAAAAAkgkxUZqzdG+smAAAAAPBCkInCviNFkXcCAAAAcMwQZKJQPyM11k0AAAAA4IUgE4XZGxhaBgAAAMQTgkwUmtetGesmAAAAAPBCkIlCt5w6sW4CAAAAAC8EmSi0zs6IdRMAAAAAeCHIROGGwa30wfX9JUntG9aKcWsAAAAAEGSiYLcZndi6ns7o3FDGxLo1AAAAAAgy5WAzRsu35+mMf0yJdVMAAACAao0gUw42m7M7ZsWOPO05VKj8otIYtwgAAAConggy5WDzGlfW+4lJuuDlX2PYGgAAAKD6IsiUw46DBT73l247GKOWAAAAANUbQaYcZq7bG+smAAAAABBBBgAAAEACIsgAAAAASDgEGQAAAAAJhyADAAAAIOEQZAAAAAAkHIIMAAAAgIRDkAEAAACQcAgy5TCsfXasmwAAAABABJlyaVEvPdZNAAAAACCCTLmM7No41k0AAAAAIIJMufRtWTfWTQAAAAAgggwAAACABESQKaerB+TGugkAAABAtUeQKackm4l1EwAAAIBqjyBTTq9NXRfrJgAAAADVHkEGAAAAQMIhyByFeukpsW4CAAAAUC0RZMrp/pEdPbeT7MyXAQAAAGKBIFNOaSl2z22HFcOGAAAAANUYQaacejarI0mqmWKXZTmTzG9r92jS0h0xbBUAAABQvRBkyqlL00wtf3y4LuiV4+mRufTV33Tdf2fHtmEAAABANUKQqYC0ZLtsRnJYvmPLznvpF7396/rYNAoAAACoRggyFWSMkcNvksy8jfv18FdLYtQiAAAAoPogyFRQXkGJDhaUBH0sd+w4rdyRd4xbBAAAAFQfBJkK+mzuZknO0BLMr6t3H8vmAAAAANUKQaaK2G1GT3yzVEu2Hgi7363vz1Xu2HEBw9Qk6Z3p65U7dpwOFwbv+QEAAACqK4JMFXnwyyV6fdo6jXp+Wtj9vlm4TZK0+3ChJOmzOZu1dX++JOm1qeucjx0qrMKWAgAAAImHIBMnLEsqKC7Vnz5ZoEtf/c25Tay4CQAAAARDkKmgUzs2qNTjeVdy3nGwQJK0aa+zZ8bIVOprAQAAAIkuKdYNSFQjuzbWpGU7y/WcwpJSrdpxSF2aZuoPH8xT/YxUz2Pea9I4LEsv/bTac7/UomcGAAAA8EaQqaAV28tfXvnxb5bq3d82aspdw/TVgq0+j5301I86o3NDSVJxqaW/TVjheSw1iY4zAAAAwBtBpoJ+WRN9eeXiUodKSi0t2nJQkrQrxOT9iUt2BN1uGFkGAAAA+CDIVNBiVyiJxlVvztQvq/d47s9ev7dcr7XnUJEaZ9Yo13MAAACA4xljliooLTn6j847xEjSr2v2hNgzuFA9OAAAAEB1RZCpoPSUindmeU/yj8ZOVxUzAAAAAE4EmQo6mjpin83dXK79OzXOPIpXAwAAAI4/BJkKalqHOSsAAABArBBkKuj96/sFbGtQq3xDxqJ1sKBY8zft99m2bvdhlTpYXwYAAADVE0GmgmqlJQdsy0irmiJwV74xU+e++IsnuKzemadhz/yk5yatrJLXAwAAAOIdQaYSrd11uEqOW+IKMCUOhyTpu6XO9Wb+9ePqKnk9AAAAIN5FFWSMMcONMSuMMauNMWODPH6OMWahMWa+MWa2MWZg5Tc1/tVNT6nS41/26m/6ZfVu/W3CCs+28Yu2ybIYYgYAAIDqJWKQMcbYJb0oaYSkTpIuM8Z08tvtB0ndLcvqIekaSa9Xcjvj0gW9cjy3OzSqJUcVB4q5G/dr9OszfLb9/r25+mbhtip9XQAAACDeRNMj01fSasuy1lqWVSTpQ0nneO9gWdYhq6xbIF1HV504Yfz94u66ekCuJKl3iywdzC+OSTt2sM4MAAAAqplogkxTSZu87m92bfNhjDnPGLNc0jg5e2WqhVbZGZKcyS1WRcQYWQYAAIDqJpogY4JsC7h0tizrC8uyOkg6V9LjQQ9kzA2uOTSzd+3aVa6GxqsLejXV+T2b6o7T2sWsDVU9pA0AAACIN9EEmc2Smnndz5G0NdTOlmVNkdTaGFM/yGOvWpbVx7KsPtnZ2eVubDyqmZKkZy/pofoZVbOGTDT+8u1yhpcBAACgWokmyMyS1NYY09IYkyLpUklfee9gjGljjDGu270kpUjaU9mNRWgPf7kk1k0AAAAAjpmIKzhallVijLlV0kRJdklvWJa1xBhzk+vxVyRdIOkKY0yxpHxJl1jUBAYAAABQRaJait6yrPGSxvtte8Xr9l8l/bVym1a93DqsjV6YzAKXAAAAQDSiWhATVe8Pp7Q9qufP27RPi7cckCRd+up0fTBzoyzLClgsc/uBAu0/UnRUrwUAAADEGkEmTqQk2TTtnmF699p+FXr+joOFOvNf01RYUqrf1u7VvZ8v0jkv/qKW9/p0pKn/X37QiX/5sTKaDAAAAMQMQaYKdMvJrNDzcrJqKi356H4k57zwi+f2ws3OHpqNe4747JNfXHpUrwEAAADEGkGmEl3YO0eSNKZfC8+21tnpPvt898fBYY9hswVbtid6y7fnBWw7UlxyVMcEAAAA4g1BphJ5pqN4ZZEzOjfy3O7UuLbaNawV8LyXRvfy3LaZowsy4dpVQE8MAAAAjhNRVS1DdCw5E0OoKOJwJYpku1FxqaXv/jg4INgcZYdM8Ha5gsyew0zyBwAAwPGBHplKdHGfZpKk/q3q6eahrTWobf2g+7WqnyFJqlMjOeCx4tKyKmOX9W2ueQ+epmWPDVf/VnUr3C53gGJpHwAAABwv6JGpRP1b1dP6p0ZJku4Z3kGS9PTE5Z7H3YHizatP0HdLtqtB7bSAYzi8wsZfzu/quW1C9vNE9sW8LerStGIFCAAAAIB4RI9MFctILet1cbgySpM6NXTVgJZB9y91BO81SU+1V7gN2w8USJJMFcy/AQAAAGKBIFPFrh3YUhf3cVYzc0QxtCtUkLlpSOsKt2HXoUI5HJYmLd0R8nXW7z7M0DMAAAAkDIJMFUtJsulGdwiJIicUlTqCbq+R4tsj06p+etD9gpm5bq/u+Hi+Hv5qiWfbJf+e7rk9eflODX3mJ703Y2PUxwQAAABiiSBzDLgHdEXTI1NUEjzI+M+RuXpAbrna8L/5W33uz96wT5I0aekOXf3WLEnOwCM5iwLQOwMAAIB4RpA5Btxrw4QYNeajOESPTOsG6TohN8tz326rnB/ddf+d7bn91YKt2rI/Xyc8+YP+9PGCSjk+AAAAUBUIMsdAWZCJnGSa1KkRdHtqkl2f3HRSpbYrmOvenq3dhwr1+bwtVf5aAAAAQEVRfvkYaJpVQyO6NCqbKxNGlyaZapKZpifO6xJ2v/u+WFRZzfNxqLC4So4LAAAAVCaCzDFgtxm9PKZ3VPumJNn0672nhHz8rjPaKzsjVXd/trCymufjaNarAQAAAI4VhpYlmFuGtdHFJzSrsuPbyDEAAABIAAQZ+LCFWDSz1GH5FCIoLCnVW7+sC7nuDQAAAFCVCDLV3PDOjXzur919OOh+re8br7b3f6st+/MlSc//sEqPfL1UX86nKAAAAACOPYJMgnr8nM6VcpwQHTAexaUOn16XxVsOSJK2HSiQJB0uKq2UdgAAAADlQZBJUKO6NTnqY1z39mx9u3h7yMfX7jqktvd/q8te+82zzb1Q5udznT0xD/5v8VG3AwAAACgvqpYlKP9J+X+/qLu2HyzQeT2b6qSnfozqGJOW7Qj7+Ml//1mSNHPdXs+2KJbCAQAAAKocQSZB2fySzAW9czy31z81Srljx1XJ61qSDuSz1gwAAABiiyCToOyRJrdUkamrduufk1bF5LUBAAAAN4JMggpVJrmqfTBzY8R95mzYpzYNMpRZI/kYtAgAAADVEZP9E5TN6yc3okujkPvNffC0Y9CaMqUOSxe8/KuuenPmMX1dAAAAVC8EmQRl5OyRSbIZvTymd8j96qanHKsmSXKuNyNJCzcfOKavCwAAgOqFoWUJKtluNLJrI13et0XEfZc/PlwdHpxQZW1xFxY4q3tZSWjvtWcAAACAykaPTIIyxuil0b01sG39iPumJds9t9c/NapS2/F/J7fx3P56wVafx7bsz6/U1wIAAADcCDI4KnkFJSEfW7H9YFTHsCzLs9AmAAAAEA2CTDX09a0DK+1Yb/26PuRjM9buVWFJadjnOxyWWt47Xte+PbvS2gQAAIDjH0GmGuqak6nfD23tsy0tufJPhX9PWasT//Jj2H3mbdonSfpx+c5Kf30AAAAcv5jsf5yactcw7TpUEPJx/4Fc7ipola1mij3s4wXFjip5XQAAABzfCDLHqeb1aqp5vZqe+29c1cdnPov/lJSqWl+zYe20sI8XlRBkAAAAUH4EmWri5A4Nfe5bfn0y6alJOlIUfj5LRczZ4Bw6ZlmW9h4uUt30FFmWtPtQoRrUTtPGvUcq/TUBAABw/CPIQPcM76AzOjfUyX//ucpe47Wpa/Xn8ct9tnVqXFvDuzSqstcEAADA8YvJ/tWUe2jZJX2a6eahrdUqO6PKXmvnwYKAECNJS7cdZOFMAAAAVAhBppo6q1sTSdL1g1tV+Wv1/fMPIR8r9JojU1IaOF/mv9PXa9LSHVXSLgAAACQugkw11TUnU+ufGqU2Dcp6YhY8dLqGtc8+pu3Yf6TIc/vFyWsCHn/oyyW67r+ztWV/voqDBB0AAABUTwQZeGTWTNZNQ8rWl6mRHL50cmX4cNYmz+1xi7aG3G/AUz+q7f3fHtVrHSkq0bJtB4/qGAAAAIgPBBn46NeqnhY8dLrqZ6Tqvev7HdPXXrnjUJUe/8Z35mjEP6dS8hkAAOA4QJBBgMyayZr9wKnq1TxL2bVSj+lr544dp+venq2deQVavj1878mBI8XlOvb0NXskVd2aOQAAADh2CDKISq7X4poPndmpSl9r0rIdGvK3n/SHD+aF3GfHwQJ1f+w7vTZlbdTHLXFVSEuykWQAAAASHUEGUfnghv6e29cMbBl23y9vGXDUr5dfXBpyqFnHByeon6sS2vcVqGhm6JIBAABIeAQZRMVWjov/7s3qVFk7DuQXK7+4tMqODwAAgMRAkEFYkeLLfSM7HJN2uJ374i++G0I0sNRh6XBhSdU3CAAAADFBkEFULCv49tbZGcEfqCLrdh+Oar9OD01Q54cnKr+oVG9MW6fCkrJenM37jvjsuzOvgDVqAAAAEgxBBmF5jyibed8pmnznUJ/HT+nYUGNHHNteGW8z1+0Nur3QVWL5zV/X6bFvlurbRds9jx3y6qkpKnGo75M/6J7PFlZtQwEAAFCpCDII682r+uqyvs3VoFaqGtROU8v66QH7dM+pc+wb5uVAfugyzJOX75QkbTtQEPTxEocz8IxftE2StGV/PuvMAAAAJACCDMLq1KS2/nJ+V9nClCxOSYptFTDLsrRxz5Ggj81av0+S5PAaG3f2C7+o1FWK2fWXHJZUUurQgKd+1B8/nn9U7TlYUOwzlA0AAACVjyCDo5ZkO/rTaHC77Ao/97q3Z2vw05P11YKtkuQJKd6enrjCc7uoxKG8gmKffS3LUpFrnswPy8pf0tlbt0e+08X//u2ojgEAAIDwCDKokDtOa6d7hjvnxgSbgP/YOZ3Ldbx2DSpeNGD2Bmevyx8+mKeC4lI9MW6p57G2IY7788pdksqCjJFRcanztr0S1plZsGn/UR8DAAAAoRFkUCF/OKWtbh7aWpJ0pKhsGNUH1zsXzrzixFx95LWIZiS/rNlTKe16ZuIKvfnLes/9ZnVrBt3vtg/nS5KnWllRqUOz1zsLBxwuKvX02AAAACA+EWRw1E7u0MBz+8TW9Ty3+7Uquz3/odPCHqNmir1S2uI/qOxH12T/YOZs2Kt+f/7Bc//at2d7br/005qoXzOvoFh//25FuUo4L9i0X8u3H4x6fwAAAPgiyOCopSU7T6P0MGGkTs0UndmtccjHdx8qrJS2/GfaOp/7/VrWDbnvxCWh58JMXr5TBwuKNWHxtoiv+czEFfrXj6v1tWuOjtuX87fox+XBX+OcF3/R8Oem+mxbuHm/Ppi5MeLrAQAAgCCDSmQPUtnsratP0A9/GiJJOq9nU0m+PThuG0JUHTtaM0KsMyNJr05ZG/Kx5dvzdPuH83XTu3O1aW/4tuUXO4fW+ffI3PbhfF3z1uxgTwnq7Bd+0b2fL4p6fwAAgOqMIIOjZlyT43ODrDEztH0Dtc52TrhvWDtNktShUa1j17ij4B6W5q5mVlzq0M3vztGK7XnKLyrVPZ8u1N7DRT7PmbU+dHACAABA5UmKdQOQ+DJrJOvl0b10QphhXJLUpWmmvvj9SeraNFNfzNvis0jlpzedqAtfmV7VTa2Qm9+do09uOknrdx/Wt4u3a8OeIxrTv4U+mr1JNpuR1xI1uijCe7AsS3M37qviFgMAABz/6JFBpRjRtbHqZ6RG3K9n8ywl2W2acPtgn+19cutqSBRryXx4Q39d1rdZhdtZESt3HNIHMzd61qlZuu2g3KPoSh1lw8mMwpdtXrr1oFreO14XvFwWdt6Zvr7S21tRpQ5LR4pKYt0MAACAqBBkEBOZNZIDtv22NnIJ5mS7CToXp6o99e1yn0IC7qVmCoodnkIFVkDNNKfDhc5w8PHsTQGPPfjlkgq3qaTUoUlLd8gRZAHQirjzkwXq9NDESjkWAABAVSPIIG6kJUcuweywpPGLth+D1oTn7n35asFWTV7hXFyzJESgePDLxZKk/4bofbEsK+z9YHLHjlOb+7/Vdf+drc/mbo622TpYUKx7P18UtOfli3lboj4OAABArBFkEDP10lN87qcm+Z6O5/dyVjkzXh0w2w4UBEywj4W8wsAg8Mvq3UH3/XzuFm3ed0ShOk78t5c6LO07XKQuD0+Maj7NzrzoS1f/64dV+mDmRr0/I3SZZ8uy5HBYWrvrUNTHBQAAONYIMogZdw9GI1c1s2S77+n47MU9tP6pURrTr0XZc0odAQEoFp6euDxgW7ieoke/Xhrysbkb92nVjjzP/VLL0ox1e3WosERPjlumgwXFYduyeV/0pauPFDlLRfuHRm+lDkuvT1urk//+sxZvORD1sQEAAI4lggxipmmdGpKkcX8YKEnq2bxO0P0ePbuz53aXppmecs+xVFDsiLyTl3C9Jhe9Ml2n/WOK577DUdYLNWfDPp30lx999i8sKfW5/8HMwLk33lbuyPMMa5u+xjkPaem2gyH3L3FYmr9pv6SqW98HAADgaBFkEDNvX9NXr4zprXquamc3Dm7teax7Tqbnts1rcn/zujVVI6XstL1vZIegx77yxBZBt8fKAlcwiEapZWmZV9A4VFii3LHjVFBcqqISh9o/MCHgOeGGgY3851Q95CoqsHb3YUnSTNdCoc9MXKHcseN85sz87j8zPL1j/ot8AgAAxAuCDGImu1aqhndp5Lnv7oXo2Li2/nfLgKDPsRmj96/rL0n65KYTNaJL46D7PXxW56DbE4HDsvTqlLUB2/ceLlK7B74N+pwPZgaf85JXUOwZwudd3cxdWOGFyasl+fa8zFq/Tyt3OINRUUnVBZlvFm4Nu4Dow18u1iNfVbyqGwAAOL6xICbihrtYl80o5PAxu82oWd2aWv/UKElSQXFpwD7ux8Lp2jRTi0LM/6iVlqS8gtitp7J+92HPXBZvr09dF2Rvpxp+Fd9en7pWddNT9LBXeWfvqmolpeEro7l7hIqqsEfm1vfnSQr983p7+gZJ0iNnJ24oBQAAVYceGcSNUleSsYWZA+O/hEy4ks3XDmypDo1qBX2sX8u6Qbf/7cJuMQ0xknT2C78E3f75vNBlllNdn8M/J63Sl/O36Ilxy3THxwt8qqs5vMo6+695c+a/pgU9blX2yFSV0kpaVwcAAMQ3emQQN9wX2rYwC16WZ6L/g2d2kuScP7L7UJGembhCM11DmZLswTN8jqsAQTzafyR09TJ3oPvHpJUh9/lyftk6MQ1dleLcQl38e8+RWbh5v/YdKdaQdtkBz/1o1iZd1CdHyXabfv/eHE1fs0fzHjo9ZFvWVHJp5y/nb1FOVg2l2O0664VpevuavgHtjMbPK3cpxW7Tia3rVWr7AABA5aNHBnGjwDWcqjwT4yWpV/M6uqxvs5CPt8rOUN+WdfXxTSd6tnVtmhl859gXRKuQ3YciryVzz2eLPLfrRlnC2rtH5uwXftGVb8wM2Ofj2Zt03xeL9NpU57ye8Yu2a59f6CopdejJcUs97fznpFVRvX4ohwpLfMLQbR/O1wUvT9eMdc6qbD+t2Fmh4175xkxd9tpvR9U2AABwbBBkEDfsrp6YYMPB+rTICvm8z38/QI+f0yWq12hap4a6Ns3UCK8iA5LU1zXUzCRoknn5pzU6HGSRzlA6NKod1X5//36lp2Ka2yezN+mpb5fr5Z/WSJJ2HnSGkx+W7dSN78wOOMbuQ4WasmqXXpu6Tg/+b7G27M/XVwu2Rt3WYM55YZpO+fvPsqzgPUnhhicCAIDjA0PLEDf65NbVFSe20E1DWgc89tY1fbX9QEHI59rDDEfz9tNdQ2UUOHwts0ay5zhPX9hNd326MPqGxwn3GjHRyEgr369+hwfLSj57fzbdcjI9w9nmbNgX8LyZ6/bq4n9P16iuzupyhSUODXjKd10cy7IiDhksLCnV5OW7PFXu1uw67HnNPrll853cwxOJMQAAHP8IMogbdpvRYyF6VjJSk9SmQUbI50Y7dybZa27M17cOVHqqXRmpSSq1LLVrmKHeLbK0bnflzt84FoyR9ueHnkPj7+NZm9S8bs2jft3Rr88I+/jF/54uSRq3aJskBe1BKS61lJJU9vP7y/hl6t+qbI7KOS/+oswayZqycpdevLyXhrQvm/viX4zAbnP+fEuq+YR/h8OSCVP9DwCA4wFBBtVW1xzfeTJ3neFcXLM4RGniYe2zNXnFrqiP/9nNJ+mCl3+teAPLIclm9NS3y6Pef9GWA0Hnu1S1XUHm8rgLDUxfs8czP+XfXuvoeM+Z+m3tHt3y/lzP/X/+sEoFJWXD3pLtxueYFVVS6ghZEKKyucNdZYUOh8NSq/vG64bBrXTfyI6VckwAAOJRVP9TG2OGG2NWGGNWG2PGBnl8tDFmoevPr8aY7pXfVODYKAmxdsqbV/ct13HcF9XHQnGpFdWE/2NpysrA0Ld4y8GAbSUO5+cdzST7eZt8h6/NWLdX17wVOC/Hv7y0JN3x0Xzljh3nU4ktlDb3f6vCksC1fLw9M3GFzvjHFJ9tN787R5d7vY/Z6/fqo1nBFyt1O+uFaWr/wISw+5RHoauX6s1fQq87BADA8SBikDHG2CW9KGmEpE6SLjPGdPLbbZ2kIZZldZP0uKRXK7uhwLESqkemvMKtcSNJd53RvlJeJ159NHtTwLa05MB/csrTexIsCHmb65qnU1AcGFY+n+csP71yR15UrxXsGN5emLxaK3bkaeOeI57w++3i7fp1zR6t2+2cw3PhK9N9qsUFs3jLwUpdeNS90GtlnccAAMSraHpk+kpabVnWWsuyiiR9KOkc7x0sy/rVsiz3V6W/Scqp3GYCx07TrKNfS6ZL09pqk52h5y7podx6weei9G8VfFHO40Wwnq1g4aAy15T533xnNbRow0o40QaswU9P1pPjl/lsG/bMT0f9+hX11wnRDzEEACCRRRNkmkry/mp1s2tbKNdK+vZoGgVUxEmt6+mxczof9XFGdGmkL28ZIElq1zB0gYFwHjmrs2w2o3N7NtX1g1sF3edo50RcekLotXPiwcQlO6La74KXp2vS0uj2jdbCzQeUO3ac9gQZbvfNwrLCA6GGEUpli4Eu335Qb0wLP0zr5yDD6KpKcalDs10LuwbToFbqMWsLAACxFE2QCXa1FfSrSmPMMDmDzD0hHr/BGDPbGDN7165j9x8/qof3r++vK07MPerjGGPUvVkdTblrmD67+aQKHqPs9pFC51yLhrV9LzCTbWW/flee2KLcrzGya2OtfGJEhdoXygW9YtOZen2Q9Wcqw/lBii24y1Tf98Uitbn/W/11wnJ1fmiCFm0+4LPfV67eneHPTdVj3yz1eWzT3iO+B63AKK59h4uCzmv6aNZG5Y4dp4Wb9wc8tmnvEZ39wi+68JXpQXudXp+61hOqmmSmRWyDw2Fp3+Gi8je+ihw4UqwDR6KvvgcAqN6iCTKbJXl/9ZsjKWA1O2NMN0mvSzrHsqygC1pYlvWqZVl9LMvqk52dHWwXIG40r1dTtdKSlVUzWbcOa1POZ5clmQ9dk713HPS9aK2ZWjaH5qGzOmvBQ6dHffRv/m+gBrfLVlKU6+dE64zODSv1eNEKsa7lUduw54h25hXo1SlrPNvcC4d+MNPZ0fzyT2t0uKhUZ70wzee5T45fpt/9p6y8tHep5wtf8Q1Ia11zYrzlFYS+IJ+3cZ96Pv69+jwxybPN4bBUUFzqmVMzadnOgOcN+ttkLdt20HX8wAVQnxi3TEeKnMG5fhQ9M/d+vkg9H/9eO/NCr9F0LHV/7Dt1f+y7WDcDAJAgogkysyS1Nca0NMakSLpU0lfeOxhjmkv6XNLvLMtaWfnNBGJn3kOn606/ifkPjCora7v+qVEBz2mdne65PaRdA0lSjt/cmyaZZfftNqPMmslRt6lLU2fpaP+FPY9WctKxKTl8LPV98gf9eXzZvJFVO6OfkzN11W7PbXd1NSkwlAbT9RHfC3LLsvT2r+u151ChznspsKfo7s8W+iw8ao8w9DDFqzz06p15WrLVt0cpmnDoLsjwUznKivv7esFWbd2fX+HnAwBQURHXkbEsq8QYc6ukiZLskt6wLGuJMeYm1+OvSHpIUj1JL7nG/ZdYltWn6poNxNZ1g1ops0ayMlJ9f4XuOqO9bvHrvbl/VEelJtt00+DWnm+bHz27s2qk2LX0sTNkDxJGTmpdT7+uCdqxWaVSjtHaKbFWkdLEkaqA5Y4dF/axv13QTQ9/tUSTlgWfD/TpnM0+9yOVf07yKu996rNTAh63ZDl7eUpKVTMl/D/1d3+6UOf0aKLUpPCV9vw5HJb+74N5apKZpl/vPaVczwUA4GhFddViWdZ4y7LaWZbV2rKsJ13bXnGFGFmWdZ1lWVmWZfVw/SHE4Lh3UZ9mGtG1sSRpzgOn6vpBLQNCjOTsbblneAefHpcrT8qVJNVMSfK5eHzz6hN074gOevuasjVrfrpzaNW8gSCOZqjaB9f3r8SWVK1Hv14aeSc/3R/9Tku3hi//HM7dny2UJO2Pcg5IpJ6jYAHYm2VJre4br04PTYxq3slN78zxub9lf74+nhVYQtutpNSh16Y6Fy7deiD6oWlLtx70LAIKAMDRqB5fvwKVZGj7bNVLTwnYXi8jVfeP8l9eqfyGtW+gG4e0VrJXz0hu/XTVSovYeRqVP5wcfq5PpIvjcE5sXa/Cz00UX87fctTHiPYj/n7pDn3stRaP/8X/6f+YorGfLVRBcfCem0KvOT2r/UpcP/vdCk1d5TucbPKKXfq/D+Z57g/664+6+7OFIcPbR7M36S/fRi71/MOyHZ55SZOW7tDI56cG9D75syyrXOsLSdJZ/5oWtlfMW0FxqWeuEQAgcRFkgHJ46+q+mvPgacf8dQe0rl8px7k4Qslm/5LQix6JvgBBdfDvKWujvlgOZYFfdbRw7v9ikd75bYP+/fMazVgXWHL5w1mb1P8vPwR97mqvHp0L/Kq3Pf/jav3uPzMDnvP1gq3alVeonXkFcueIkc9PlSTNWLtHCzbt9+w7b+N+n+fmjh2nZ79fqa8XbFXu2HH6eeUujVu4Tde+PVudH54oyVnKWpLW73EWR1i3+7AGPPWjpviVr27/wAS1vm98yJDmdqSoRDNdn4t7IdBoXP/f2Rrxz6k6kE+FNABIZAQZ4Bj67o+DNe2eYeV+Xtso17NZ++eRQbe3zk7XXWe0V9M64Rf7XO9XfatWWtlwuHN7NImqDag8xaWWHvzfYv3l2+X6akFAsUhJ0Q9Vk5zzbp6ZuCLsPle/NVNfzA3sebrk1d90zou/eO4H61V5/odV+sS1/co3ZuqTOb5D0/Yedrb1B1dFtpcmr9aW/fm64o2Zmrhku2e/ItcaPsHKU3u785MFuvjf07XjYHRD22av36vFWw54ijhc8u/pGvHPqVE9FwAQfwgywDHUrmEt5WTVLPfzbjulrf7rNW/G26kdnVXR/nJ+V9lsRqd1aqiOjWvr/F5N1aNZHUnSaZ0a6ZZhbWSM0dD2oUuf101PUVaI6mmdm2T63O+ekxl0P1SN92dsPKrnHywo1oczN+mFyavD7rd4y8GAIWPzNu6L+nW8e1faZPsG8J9WOgPM8u15euzrpZ7QI0k3+s3RkaSBf50c9rUWbHL2wgQrRe1v5Y48XfjKdJ35r7Iy28u358XNELO5G/d5epcAANGpnIH3AKpUkt2mwe2yNeeBUwMee/3KE3zuv3ZFWa2N3YcKdecnC3TlSWULbubWS5cUvNxudq1U1amZon1BvuV3uOZo9G9VV/eO6Ki66Ska9LfwF5qIH90eqfj6LH+dUBZscseO08uje0X1vAy/uV3uuTKS9EYFKsd523+kSFtcZZ9vfX+uZ/vOvALVSLar6yPfqXuzOvrDyW10qLBEt304/6her6qd7yrJHaycOwAgOHpkgARSLyNV9TIiL3ToVj8jVW9d3VeNM8MPKfP2nyv76OzuTTTjvuDldHs0y1L3ZnXUrG5N/finIZKkOuVYAweJ57e1vj0FN783N8SevtyLiHZoVEtSdOvvRGPv4SL1eOx7z/3l2/M8t/s++YO2uaqoLdi0X9e+PTtiiIm2HHfu2HH640fhjxXJut2H9fnc8MUOEB8Kiku193BRrJsBIAyCDBCnaqclBa2QdrRG92se8rGs9BS1ys7Q85f1VMPaaT6PjenfQpee0Ey/H9basy3FtYBmx0a1y92Om4e2jrjPiC6Nyn1cxI81rmpp3kGjvPYfCbyQrOyLS3c57g9mbtTiLQc0YfH2kCWrv5h3dJXrznx+qu74eEHIx93V6c5/6Rc9Oa58ZcKPFJWopNQReUc5L9LX7op+cdholTqsiEMRLcu5xlG8u+TV39Tr8e8j71jNrdt9WDPWHvt1zwCJIAPErbkPnhayV+RotG1YS5ODrE1TNz0lbDGA9NQkPXVBN9X2KgCQk1VT/7y0h16KMNRoSLtsnzk1658apWsGtIzY1pfH9I64D+LXxCXBF/8sjzs/WaAR/5yqbxaWFTuIdAFekSLiew4V6t7PF+nMf03TTe/O8SxeW9kOFzkrsQULaM7tzgA1d+N+vTa1rKfomYkrNGGxsyBCxwcnBK2e1+mhiZ4S2s9MXKGnJ4Yuj/1/H8zTyX//OeLCq+X1zx9W6byXftV8rwp3/p6btEqtXFXpPpuzOWJ1umNl2qrdmrB4m+e+u0pf7thx+ixCyfDqbNgzP+mSV3+LdTNQTRFkgDiVZLcpyV41v6It66fr/F5NfbY1rxu8CMHTF3bTXy/oGvJY5/RoqixXz9FlfYOXd377mr567tKekqTces7XiXbxzTeuYn3d48HOKCuL+Zu0bKeWbTuoW9+fpxXb85Q7dpxuCFIY4Gj1fmJSwLanJy7X3Z86e0+CXcgWlzqUO3acpxLcvsNFUa9/s9ZVIXD5dt8FQk2IX4sXJq/WTe8633e+68L/QH5xQLu+dYWdFyav1ouT14R8ffc6Qo7oOnCitsJVYnuba/6S2y3vz9Upf/9JkvTf6eslOYPDnz5ZoIe+XFy5jYjS0q0Hdchr3taY/8zQTe8GHzb5p08WaMzrM/Trmt1hj9nugW/1cIzeT6zljh2nORsqp2DF7kOFnqGpknPu27RV4T/7RFZYUqr7vlikPREqNVa2UoelI0WRi6XEM4IMUE094LWAZ9M6NXT38PZB97uoTzNdckLo4WjeHj27i07v1NBn27UDnT0v7txS6rpos9sDr9gu7J0TsK19BYatSc5S1+6KbuX11tUnRN4J5dL3z8HXuykP7wVCj4UXJ6/Rx7OdQeFPn5QNB7MsSx/N2uipePbC5NU6VFiino9/ryf8hoNZlqWL/z1dX8zbrHVe5c0XbtqvcQu3afhzU/WeV0U6v3VPw14c3v3pAv3pkwVaXI41dNzcecuS80ZRiUO78oJfRFmWpX2Hi/TubxskORc5/Xll8IIhSTbnZUWp3xsZt3Cb1uw67PPa7kVbN+49EnW729w3Xvd+vtBzP3fsOP15/LKon+9WUurQyOenatgzP0X9nGmrd+vy12aE3aeoxKG3p28od3uOFxe8PF2StO1AvgqKSzVpafl7ZS3LUp8nJumOj+d7tp31r2ka85/wn30i+2r+Vr0/Y6OeimKR4Qtf/jWq/aJx2au/qdNDE8u9AHE8IcgA1ZTd66vfX8aerJMqYdHNlCSbWnmV3P3whv568ExnYLK7koz7G2B7kK+enzi3i5rV9R3e1rRODf3x1HYBw95O6RA+pGSkJimzhu8co/tGdoj4Hvq3qqveLbIi7heKe2I7Kp931bNwTvvHlCpth8OS7vlskc5+oWxdnQ9cYcR9se9WXGpp5rq9+uNHC3wumg8Vlujhr5ZIkh74X9k3+KWWFVAS2n1xKElneL23lTucQ+x25RXq0lenK5SC4lLPOj2rd+Zpz6FCz4WLO2+M/WyhTnhyUtALmpb3jlfPx7/XA/9brB+X79C1b8/WlW/4LqhaXOrQt4u2ecLauIXbAo7j5u6Bcr/PYEUgCopLg347XeKw9MHMTXI4LG1wLaz66pS1IV8rFPfb3JVXqBvfmV3u5x9rxaUODfrbjz7rLcWrn1fu0ol/+VFDn/5J1/13drl7aYpLnT8cd++iVHaOJPIFdzju92yPYqTC7A379MrPoXtby2PmeufPJpE/V4IMUE1l1kzW+9f308JHTq/U457R2dkjc+PgVurfqp5nu/ub2mLXZOS0ZLvP8x49u7PSku0Ba49I0m2nttWkO4Zo4u2DlZPlDDQ3DG6lFy53Dldr5FeYQJIsSdcN8p2Hc+3AVp7boYKQ3WZ8FgL1NvfB0/Tjn4aod4ssLQrxubUO0v5EEWpYU7z4cNax7ZFxu+3DeT73g/2n/6SrV8B9QSJJv/vPDHV/NPhcm7rpqUEX/OzzxCRd/EroULJiR1nhBHdouPqtWQGV5bw99s1S3fjOHM1av1enPjtFpzz7s+c9uOeBfO4qYjB7fdlx1uw6pL9/57uA6jVvBb/o/9ePq3Xze3O11BVO3BehJaWOgEBy0LXuj3tNo3W7Dyt37DjN8nrtMa/PCBju5x0SW903XkOe/slzf93uw+UaIuPuiZIqZy5XZSp1WAFzl3YfKtSmvfkxG4ZXHu5zartrOOkt780Ls3cg94K4wYYfew83C3Y/UR0scM6NC/Zv3M68Am3ed0Q/LKu689T79yHREGSAauyk1vV9Ju9Xhp7Ns/TB9f1193Df3o+GtVN15Ykt9LZrYU//b57aNgwfAGqk2NW+US1NvXuYJt4+WP1a1dNpnRqqV/M6emlMYLGBRrXTlOw3fM37NZOCDG2Tws8ZqOuq6vbZzSepVlqybh7aOiAQDW2fral3D/PZNqjt0fd2HQv+w5rg9OX8rT732z3wbVTPm7pqt2c+i79wcy3youx5ipZ7MdWLXAFpv1dFthvfmaMHvXqEHvtmqX5Z7WzbKX//Wf/6MfwCqm7B5kD9uHyH7vh4gU8gscKcZC96LdY6e4Oz8pnDYelAfrE6PTTBp+fK37BnftKo56dFvXCs/+95eb6RvumdOXotSC9QZVViG/36b2r/wARJzp67gEp0eXnauHhN1K83ddUuzxdI4TgcloY+PVlfLdiqgwXFnqA9+vXf9NJPvufBlW/M1NjPFnouwL2V+LVrexTz45ZsPaBnv18pSZ4KaMF6J7wD3qdzNqvdA99q457wQxMty1Lu2HExq0C3eMsBPT1xuc+5X1Bcqo9mbfRs+2nFzqDPXbPrkPo++YMG/nWyrn27cnsOvT+3RP63nyADoNKd2LpewH9Cxhg9ek4XdWwcfM5Leopz8cT7R3WUJF11Um7Q/Ywxau8avpWaZNfnvx+gXs3LhoItfewMzX/oNNltJqCEtDd3D5Ek9WtZ13O7Ye3g6/QEWyvnnuEd9KJfxbZN+/LVzK9wQpsG8d9LczTD6eArmiFw34QZenUs5RWW6B2vno4lWw9q9OvRz0XIHTtO932xSHkFge/5mrdm66sFviHQu8cqoC1BjrHtYIG6P/qdjhRFrmy2bvdh3ffFIm07kB9x3yK/C/tL/h26F8zfhCXbPT1w3oq90lF5SoSv2XVIw5+b4qlk5+5dm7B4m7o8PFG3vu/bo7Hj8b/q85sejGrOyMx1e/W7/8xU2/u/1c680IFi8oqdWrv7kNbvOaJ7Pl2oAX/5UX1cAfSX1Xv0twm+PXM/r9ylD2dtUr8ng8x987sqrp8ReRmBc1/8Rc//sEpzNuwNe8HunmclyVNhbrmrwERBcWnQcOcOqbFaE+iyV3/Ti5PXeOaESdLzP6zSPZ8t0lcLturtX9dr6/7gP5sbq6CwieT8TAY/XbagNUEGACqoQa1UfXhDf3VvVkeS1KZBLf127ymeuTXlVTMlSXVqOv/jrJWWrMv6OgsVnOzXc+LdI3Nap4Z6+Czn66Wn+q5G7xZsTo8UOESub64zFI0d0UEPjOqo+0Z20F1nBC+kUBGT7xyq9BR75B3L6Xq/YXiouKcnroi803Fgsutb5PdnbNS4Rc6LysaZob88kKSSMF2ewaYHPOKaR1QewQKRv1en+M4xcPcAlcfcjfuUO3actrsWYPUOab0e/z7qIgzPTVql5dvzAgoouCuoTViy3XOhueNgoRasc/aY/bom9Notpz37sz6YudHn4v36MAHh6jdnacQ/p0pyDjGNtlfQv8exbYMMT4lxt3N6NA3bE3fgSLHns9u8ryyEBguv/52+3nMsm+vfZIdlqaC4VB0enOBT+KGguFQLNu1XpI6r3LHj9J9p0S2KG60jRSXa5/rs3QHXu8z4Plev6G0fztfDXy0JWfDCv0x7RXv9Jizergtf/tXz2fn/3PaGKAefCAgyAGJm6t3D9P0dQ3zm0khSo8y0qCY9evvm/wbqH5d0D9j+l/O76pv/G6h//865Js39I509Pt49MqP7tdCJrZ1tOLWjc45Phl+g8W+jt9SksmN1a+ZcL+emIa113aBWumFwa9VMCR6OaviFoK9uHRCwj/8Cpi3rp2vJY8NDtqWi1u32/Y80UjEFt0l3DAn7eLRlto8nb/26vkLVmiqL92u7Sx2XV7gLT7er35wVsG3bgfDDiIpLQh/XuFYA2uE1FOn7CnyOJa6L4i/mbVbu2HFB19x5L8ohaOGMcfVcfTbXWdnu/Rm+hR7O/Ne0qI5zxBUa0kP8O+Fvk9fFfu7Ycbrjo/k+j1uWpVU7D+nezxf5hMMFmw/o0a+XhFw7KFxvmfu1DuQHXyjWbdXOQwFzov4zbZ1a3jteb/6yTvM27gvoGTrZVZZbkg76H9/hkA6X9cJ8OX+rp+CB+/+IUoe0ylX44vVp6zyB4cH/LdY5L/6iLV6lwC3L0o/Ld3jOb/cQwce/Kd/is5F0emiierqGsrkDl3ePzOTlwYeS+fMfpue+n7t3i/TCC1G356Z352j2hn2eADPgqR99Hl/uV2AkkRBkAMRMs7o1lVmjcubodGmaqfN6BpZvdj+W7FqTx/0fQU1Xr0Zqkk01Uuzq0Ki21v1lpIa5LuAXP3qG5/lf3TrAU1ggGO9rPu9QE8kHN/T3ud8tp47WPzVKg9tle7Z5D2kb2Kbq5trMWr9XQ1yve0qHBvpXmPfrrU2DjLA/Q///iKuL6/4bu0pY3q/90JdLlBVkWGQkLe8dX5lN8nj+x1UhH9t2MF8fz9qkUc9PParX+GSOc8L0Hz8qK5n9fx/MU+7YcZ45HdFWwAvH3WPg7oH78/jylcT9aNZGvfDjKs8ckvV7Dofc13sOT57fvJTP523Rb2vLeme8f+f8vxB685f1+mLuFp9t5fmW/x2vYGx3lKr17k0B45KCHa3JwZ06cM/9uvT5n3wq70nSHq9eo0e/9g0UR954Wyv+9JCSS8ve805XmXB3QDhcVKIv5pW9J/faQAs3O3vE9nn1Nnwye7OueWu2Wt47XrsPFepfYc5HtwmLt2nf4SJtCtJrYlmWFmzar2XbDmrv4SLP/B5v7nYeKSrVBzM3qrjUoe7NMgP2C8b/PHW/t3OX/iTt3i0VF2vr/nw9/s1Sz8/xwJFiXfDyr0EXmnX3VvoH0kgBNZ4RZABUK6e51rm5qI8z9HjPDTF+w8fci3fWz0gNeMzb8C6NPLdTyrGIabOsGkGHnf3XVRBBKushknyrsM178LRKrTJ2pKhE17jW/Ll2UMuQvUjBtKyfXqHXvP3UtlrxxHB9e9sgPXlelwodA5HtOxI/FynhhvBs2puvuz9bqN2HIg9zaRemOMibv6wP2Pa1a65Ot0e+08h/Tg3b+5BfVOopeBCtL+dvCfv4hj2HAybc3/PZIj3z3Uot2er8NjxcGelBf5scsC21pOxzmnL1HTrwnPMbeu8hbTbvfyQsS302L1HOD+OUX1SqW96fq20H8n16CiTnUK1Qnvluped25x1rdNbyKRq6drbSC8su8pv4lcqXpIsXOnsnem1ZFvZ89P/iY+LXv+rbxduUVFp2Uf7fr+fohHs+81TIe+B/i1XqGr7Va8syJU92zttxv3XvoPbGL+t00cLvdeKGBRrxz6meCnqSc+jXiu15+vfPa/Tn8cv06+rd2pVXqJvenauej3+vQX+b7BMYJef5fM6Lv2jEP6eq1+Pf65JXfwsIhu7w8ejXS3Tv54v0n2nrIlbKc/cY+Z+nQ572Ow+M0c3vzdV/pq3T4q3On3v3x77TnA37dO/niwKOG+r/p8OFJZq2arcKS0qjKgwRT6L/nwoAjgNtGmRo/VOjJElf3zpQLbNDX4R/evNJmrZqd9D/mL09c1F3Nc2qofdnbAwbeNxObFVPj5/bWfUyUgMqq/nr2TxL658apVKH5fPtalZ6in64Y4ju/nRhwPj+v13QTTLS3Z8u9D9cSMWlloa0y9aCh0/39LDcMqx10NXhf/zTEM1Yt1dZrrlI5emF8tagVppSk+zq2Li2GtVO0/1flFWlWvXkCLW9P7rqYKh+doZYvNPtnTBD6pZGGEYzZ8O+ci++eNuH80M+1vvx75W/74BslqUFf79Qd3+6UDcMbhWwX0Fx+QJUg0N7tamO80uUlNJivfnprxp97fU676VfPftc/ZZz+F9qcaFunvGpJGn2eun5rB6auX6vZ72flJJiNTuwXWvqNVNBcdmF7KEgPVfu+UV1jzg/xx7bVqrjrvV6uf9FkqR66aEn9zffv13TW3RXQXGp3p+xMWRRFzf3WkGW65++Rgd368yFEyVJzw0cLclZgnm/q0dh8Lq52jtxhzLPGuk5hrv6ms1RqkHFO5V+cKeaHtyp+Y3bSyllc7qOFJXqjOfKeotenbJWv4492ac98zbu9xlm/MS4wKIP3kHQexife/7P7lDnrmWp0851uuuDOfpkwXa9eHlgNU7/+V+z1+3RFtdxvb98un3ae2q4ME0a+U8t/WS8jFVTlrGp5+Pf68c/BQ4HfvDLsrlorbPT9cOfhgZvYxwiyACotrrmhO/er5+RqnN7No14nJQkm+4Z3kH3DA+94OZnN5+oGslJKiwpVYdGtVXDNbTN/Q1kD1exA7d/XNLdZ8hWsDlDrbIz9O51/dThwQmebfMfOs1T7EAKHWYu6p2jT+Zs9tx3D7Xzfs0/ndbeJ8iM6NJI1w1qqVbZGT4Lnz57SY+AMdfR8B4ik+V38ROquEJlO6VDA/0Q5Xh1xI/9EXqZvC/MyqsyV5B/9KPZGjXlM9XNd170z1g7TJ/N3ayFm/cH7HuwoMRTMS698IgOp9TwWdypdsEhXTP7S8/9Zvu3e4KM23avOUrN9m/XyBXT9H6PEUor9u3lmum1Zo8knbp6htrt3qC3e52pfTXL/l3cHmTOU7BiFt69Q7PX+xVO8LqwX1m/hSR5/s0KNoerZlG+kktLdCDNt9cttbhQl7pCjD/vEul7Dxeqpcp62G96d64GrZurrttXq0/jdLn/RWy7Z6MWNm7ned6z3/u+r8z8PKVOnuRMUa5j5ReVaPXOQ9qVV6gv52+RMYEVv7w/H3cZbUnas36LLl41Q4e7jg76Hlrv3azTV03XrPyDUm4P3fbOTDU/uEsb6zQKusjXf6ev15OHflVhUopy927RxpUb1KZBZ8/jOw4W6LunXtPSn2ereedh2pDVRJJ08t9/Dvr6bmt2Hdb/5m2J6v++eMDQMgA4Bnq3qKtOTWqrZ/MsT4iRpFLX0AH/+S/n9czRyR0aKpK0ZLvWPzVK0+89WSufGOETYrwLFvRsXke/eH27OLq/84LCXRr17jMCQ5jNLzy9PKa3ereoG7Bf0zo1NPO+U/SX87tGbG+0/F+7MnlXaHv2kh5RP+9fl0U3bwjxz7+YR2WyOUqV4RpqVfKf/3hCjCTd+K6znO6mfb7zLVKLC3X7tPfUas9mtdqzWdfP+kKD1vuWXW5y0LeqWVpJ4BC8s14oKzBwweIfVKO4UNfO+p9sVvjy1VmuNiY5SlW74JBa7nUOlVuytWyYmrEcSiot8Qx3qlEcvLDDBNdkfLdbpn/suZ2X6luavs9PX6ndrvWqd3i/kkqdvQ03zPxcV8/5SgPXz/fsN2DDAqU4ws9run3ae5Kc6zdZluVTNKL3lmVKKS32CZBZ+Xk+z1+545CSS4s9oWzkil+UOmOGsg+XPaeo1NKpz/6sy177TR/O2uQTYkYtm6oeW1fo30GGCBrLoSvnfqMmebs0a3LwksrtdjmLRZyw2RnCL1n4nc5f8qP6bPGdN+T+nPYeLlLuPmeAO3fpT2r54VsBx/x2ge+Qx2b7t6vDTt/hndmH9qnXFt+epds/ml+uBWZjiSADADFU6vqfsLxV2vw1zqyhFL8hXmnJZfe/+P0ANa1TQ5ee0Ew5WTXUo1kdLX98uKbdc7Lev75fxN6pSBrUTtPprvlHnZsEXyvIn38vzGPndPa5P/uBU/XaFX0q1J62Ydbuadewlud2eaqqVc+yBcenYEOmKsuwtbN13awvlFJSrHpHfEswJ+3do9unvac6e3x7Ad1hp8/mpRroCjC9/S4uW+/Z7HM/vSjyejlug7xCQTDG69bo+d/qnKU/SZJe/ednuuXXj5RaXKiT18zS73/7xPk+SkvUfveGYIdS3SMHdPLqmZ6uimSvANJh13qffXMO7NDIFb/od/PG6YyVvmv5eF/Ad92+WrULAosh1C445DM/x+28K/+hor37Q79hSaWm7N/HrttWadGKrbpm9pe6+bdPJMuSzXLIkqXcfVs8xQb8yyG79du4SG33bNTQtbPVddsqnx6qGkUFunbW/zz3LRll5ufp9mnvqc1uZ/W89MIjAZ9n9mFnz5b/z9372MYrSW3Zl6/D/3xBFy38zrOtxX7f9aouWPyDhq/8VTZHqbpvXSFjOTR6/ngNXjc34D39c1LkQgjxgKFlABBD7v+H/ENIZUixB64389QF3Ty33WvgnNS6cqqh1ctI1az7T1Xd9BS1uX980EXWGtVO81RpOrNbY7/2+n4G9TNSPUPeyuv0zg21auehiPuVJ0BGU5IY6Lp9tSR5woC3NnucFdUuXThRP7Xqo/lNnMU+3BfVdsvh04Pjzb8HpvXezapRVKAbZ34WsU05B8JPLjeWc16MzSr1XCgby6Eha+co2VHimV/jVqMkcJ5HanGh6uUf0GmrZigr/6DmNWkfsF+bPZuUs3+7NtdpFDAmq8X+bbI5QvccZRT5BpYaRQWeoXZv9DnH57Gha2er8441eq/nSIXScec6TWvZU3XyD+qUNTPVYdc61Sh2tvf2X96XJOUVZGjAhgVqcGifxnUcpA9nbQp6rBM3lg3hPWXNTLXYv017amZqevNuAT+f2jWSZFwhpf2uDVpdv7lGLQ9dqrtx3m613r1JZy2fol9bdNeihm08jxmvr1dmb9ireRunq2mQf6eM37Zbp38sm+XQsLWhqytGu5ZQrNEjAwAxdN2glrqsb/OIk14rwh2O+nhVZqtq2bVSZbcZLXtsuH4/tHVAUHjyvC463zX22nstHyn4cLJos8MrY3qptatww4A29XRKx/DD8sb0b653ru1briBTWk1LSaNimh4MnHuVXlQ23Gmo6yIytbhQNteJ3vCQX/le9y+AZanZAd8hW5JChpgr5nxdrra6L3SHe/WKZOXnyWYFr2CVUhI4R+mcpT/r4oXfy+616Km7Wpm3Cxf/oNy9WwJ6q1JKi9U4L/Qin/6v6f3evecOuWUf3qc+m5co+9DegMckKb04X2es+FXN9zs/16Z+Q/ck6ePZzuBSN/+A0guPqMfWFapRFH6tJMkZ2PptWqwLFgfOHfSulJaVf1C3T3tPTfICX9vbWcudRQhO2rDAJ7xI0qmryuZ0lYb4B9PutxBtsJ+r93Gk4AvUxiOCDADEUK20ZP3l/K5Kr4Ix++5qYkUxKKeZlmzX3cM76J7hzm+cu+Vkalj7bA1oU19/vbCbZt5/SkAvlPv/zXN7NPFss0IM6OrQqJbP/YzUZD1+rrOEc41ku3o1z9K6v5R9Gzuya9mk6GS7TU+c21WD2mb7DC2bctcwSeGrLsWr4Z0b6SO/dYnK68oTW1RSayILVna8OuixzXdCefahvbp5xqfqvq2srHFhUtn5d8qamZKktrujW8CzdsEh3T7tvZC9Om7OC/yySfnuIJXl9bwr5n6jpCA9JHZHqX43z3eR0WS7TQ0OOwNDWpDeGn/nLv0p4BiSdNGiwODjVpRU/vWQBq6fr9HzQ1c/7LhrnRrnRa4UZyxL18/6QkPXztaNMz9TakmRMgqPKPvQPnVx9cAFEyx8Wl4D+eof2R/xtf15/0zyUtPVZUfo13dzB6Fw/I9zShRzNOMBQQYAjlPuoFBYHLt1AdbuKhvX/ubVfZWWbHde9NRKC/mcaCb6v+211o4kJdmN+respz+c3MYzfM4Yo79d2E0/3zVUl/d1XqQ3yUzTKK8hbd7lspvXq6klj56hJ88LLFrw7MXd1SgzdJsjeWVMWSnVQW0jD+U7q3uTiPv4HP93vdXPqyxsRTSrWzNg24KHTteM+045quMGM6htfS17bHilH7eqzKyCz0CS5yK7466yCdjecyA67XBOHD9tdXSV1E5eMyvodptf1SvnBX7Zoqf+3/KHc/bSwKpXufXSPRfYKa75JP6T6Y+W//CoytLRb/J7MP7B8ObfPtF1s77Q6PnjdWqUP5tyCfNevXufLlw0qVJftvP21bp92nvK2b897HpC8YQgAwDHqdbZGerdIktPxHCxybauifW9oxje5gkVXv9/Ng4RHuw2o/ev7+e5n2y3yWYzuuP09qqfkerZfnGfZmpRL10D29bX1LuH6dd7T1FymEVL01OT1DzIBX3L+uk6qXV9vXFVH8154FR1beosjvC/WwZEfF+SNLR9A8/tUNXPaqUlhdzn2Yu7R/U6l/RpFvbxcIUYShyW6tT0/dY7s2ZyQA/VqRGG7UWjsMThU70v3lVlFb1wHK65Myml0S1qGmzIlyR1CvFzr1FUoNunvafMgsjzydz8J5A7BV70eg+TS006+p91cmlizNmIpOPOdUqJ8F5qhqgIV9n8v1ByB+YLF/8gszv8cLd4QZABgONUSpJNn918kk7IDSyZXF7e68uUx1ndG6tV/XRdfVLLyDu7eV0ztmlQSz/+aYim3TNMD53ZyWcX7yIFkRYWlYL3OEjSo2d39hmW1alJbT3uV0HNPZ/n5A4NVS8jVW9f01dvXX1CwPo/oaQl2zXpjsFa/vhwnxLZ3u44rZ2Gd26kf17aw2f7+T2b6jyvNR3CveafI5TA9v5m/rZT2vo9Jt15etmQrw9dn0mSV/Bb/9QoNcpMVTRuGdZaOVnBF5NNkC97PbzXNWpQK/T7Tw1SErmi2jTIULKjJKBcbjgbshoH3Z5Tp4Zy6wUu/htNoYBorNp5SE0yfX/W3sHDe2HIikqOUH45UbTds1Gnr5oeecdK1t+rIIFb35ahv2Bq9v6bVdmcSkOQAQCEZIwzEE26I3A16Gg0qJWmH+8cqub1gocIb0UlziFwyX5FAFplZygnq6auGVgWhvyvgyMtkBjOlSflBgzLau1XvtmvSaqbnuLpZbnCb26JuzclxW7T9HtP1n+udJaQbtOglqdS3CkdGsifkXOI2Dk9fBeie/aSHjLGeMpbh+udsduM7jy9XcjHF21xTrC+cXArtXG9x2Hts3XdwJa64sRcjelf9l76hxiq5j9MSQp+cZ+aZNcnN50Y9Bjuwgk3DWkdsq3RuHv40c21uaBXTrmfM/P+U3VCbvALwPvOKPvsux1lSXN3z9fwlb9G/ZwTg1ysSs7y6FW9wOEovyqEvbYuL9fzw5VMl+RZMyWU3/UPP8fL/buXCGxVkPT7b1wUsK1eeuhQnii1TQgyAICQ1v55pFY+MULZYb6FrizuCz//C6Jg6vr1agRbgfxodGjkOxRn8ZYDIfaUHhjVSU+e10Vr/zxS658a5bkA/euFXdU4s0bQCmr/urynXry8l9b+eWTU6+48fVF3vTKml1plZ2j9U6O0/qlRkgIXd7z15LYBpawl6Ylzy4YY1kixe8JgzdQkPXBmp6gv9C7v11wpdme4/ffvektyBrtOjX3fh91mQi486Q4ym/cFrgHiVjstcgGM9q6hi6GCRSR5Bc4A/KfT2gUMqwvGXQ68IMS8syu8qg92bBzdzzWYC3rlVOqFd90oC1gEC6nRSk9NOqpgOqJr+N97d3WxUPyrIPpzl3uvqkBzxYm5lXasYEUCjsbpnRrpuoGtAraHG94ZqtBKvCHIAABCMkdxYVNeXZpmas2fR2pwu+yI+/rPWVi/J3CxvKNRNz3FExQkqX2j0BelKUk2je7XwtOmGil2rX9qlM7rGfrb/popSRrVrbFsNqOezetICnxPo7o19qkkllkjWcO7+F7svX1NX0384+CA4992atuAbd69LV2bZnrWxYnmJ/zmVSfos5udvSsdGtXWyidHqE2DDDXLKutp+/z3J/k8598/rwk4fybdMUTD2merjyt4BBvu5JbkF8YGtPHtIcpITfJ8a1w7rWJDH79b6lxf5eeVu0LOXZKkOjWT9YeT2+irW51zovyH/7kZr7Wb7EfxuxNqGGR5NKydplFdG/v0GLasH/rzlnTUE7yPZmFfmzG6cXBrXdynWcRQEoyJ8JTDhc7hbQXFvsPcjiZweqvMaVT+i4MejfoZqerUpLYygnwxEO7H7YhdjZhyIcgAAOJGRS+ELo4wyb2imriKDUTTO1BRxhUl/C8qXry8lx49J3yhhiHtstW0TuA8FO9v1r+9bZBnaKD7m/nuzep4ih7U8PuG+t1r++mzm31DybAODdS7Rfi5Vv69QOf1bOopAe7WpkGGp3pdJHf7lWi2+13cZtZI9glh3r1U4bwypre65WQqJ6uGp8eoZf101QoRhhrWTpUxzkISbRo4e4BaZWeoVrDeJq/PvbwFAm4cHLk346qTclW7RnJUQ+IuPaGZ2jaspbpew4dCFWo4P0zoLo9whTSiUSPFriZ1augkv9A6KkJvjRS5N+lQYfDhpxXtzfN3tF/6RPqd8B8GFimUup3UOnQ1w3BzC2etD77+TrwhyAAAEsapHX3nlvRy9WYEqzRWGT668UT94ZS2UV80VIS7R6Ztw/BzBMrjtE7Oz+nda/upY+PanvkwJaVl85BO79RQtw5rowdGdfJ57sC29aOqMif5XLcHXLg3yqyhZLst6DA3t+6uwgX1M1J8jmczvsUc/nVZTyW7jv/8ZT2VW6+mnjivi2f/8vQkDO/SSF/dOlDT7jlZl/drLskZsLoHmdOy5s8jNeO+U4Me55d7T9acB07VnAe8HvcOMl63o+kxiqYTomZKkq4Z0NJnSJD3sDHvC/5gF9bpqUlBL2zdXyDU9DpuqCIhbRvU0q3D2gR9LJT0lIp/EfBuz5Fq27CWLu/bPGzFPO/TL9iQRvccPEnaVst5buVk1Yx6ON2oro191p0K9/q3n9rOU7ExWqEKY7j5zzNs2yDy8S/u00ytsgP/XenQqLZuP7VdQK+ntyPnXRjx+PGAIAMASBiv/q6Plj9etv7IW9f01de3Dqyy8rjN6tbUHae1q9Ihduf3ytGUu4b5XLgfrTYNamn9U6M00G/NmtevPEGjujZWrbQkJdltuvOM9sqMYm5IKP75YcUTZT+bfUecFbzSkkNfapzWqaF+HXuyfrv3FE2/92St+4uzV2XtX0b5tOus7k08F9spdpt+umuYhrVvoCAVu/XDn4aEHSbmzV1aum56iowxap3tG1jD9RDWTktWvYxU1fMq962ksgto93MPpGWocMwV2pkRPBic3KGBRnRpHPSC2v9b+rKg57xRp2aKz9yMaHo0uwepelcWosqeP6BN2bnjHSBGdm0U9gI4mJy6ZRfp0cwJa9ewlidUZTdwhuoGtdNCBon01CSf9x6sd8M7WOxOryOprLctGumpSboxzBwg/+M09xp2Ge53292LleYqUd3Oq50Na5eVR/Z/68l2E7K3pVtOHV3Wt7maBOmtlXzXshoSYijvtHX7gm6PN1XXVw4AQCWz2YzSbGUXKbXTktX1KKtDxYNoqrpVhr4t66pvy6Mvx+1Wu4bzMmKg66LXe72QPYecQWZwu2x9s3BbyG+c3Rdbjf3K97oLAmS5As35vZrqu6U71LFx2YWee47Nmd3KFhBtnZ0RUArYrX8r3/d+7cCWqp+R6ilv7b4Yffbi7j4XkVGz2fTcwNG6fdp7njkyX3YaqtlXnKy8Swar9KGHPUP58otLtW1/vjq71iQqCTIpweH6DAa0qa+5G/Z5LtY9Ac6VJHu3yNKcDfs8PRE+17zdukkLy6qZpSbZNax9A01esdOzzV3170iRb4nj0zo2VGGpQ12aZqpLU9/fs6yaKZ6w6j3sqWmdGqqfkaphHRroYH6xZm/YpwFt6mnFducCmd7rPIWSkZqkG1xD7W7/0zDpb86FPkP1u9mM8Tn3+uRmacLi7ZrRrKv6bVrk87pZNVNUZHeeU/7BqFZasqcAhD+7zYQdOuceRun+/erStLbq10rRgSPFKigJPeGkRopdxfkOpbt+dt5tOv+2yzTuxY+0ce8RHSzw/dnUrpGstg1rqW/Lenpu0kqfx04OUhXRm3fPW8/mWdq6P1+rdjrXEhrTv4Xe/W2DNhysvFLiVYkeGQAAUCE5WTU16Y4hGjuiQ8Bj7ovcEa4CBeUtR5ye6rzYusU1jGl4l8Za/9QotfAqENAqO0MLHj5dF/QKX1r4lA4NNPP+U/TW1X19tifZbbqgd46nR899Ddm2QS2fHomKcPdyOIyRzWaUWSNZddNTVCPFrhopdtVNT/GEGEmyBSm74B4y17Vppm4c0trTE+O/54A29TWmfws1qJ2m6we1Kus5OOEEnxDj5t0rE27IVOemmep1101BH5twYdn2Xi3KjndRn2Ya5rqQrl0jWSd3aKDUJLumN+8mybcXr45rfao+55wsdfZdu0mS1Lix5FVAIdQIwtppSVJ2tvJSnF8IuKsOGssZINzDq37Xv4UuOaGZiu1Jnu3eHSnXDMiV5Awll/dt7vMadpsJOf7v9lPbKblpE/3fyW11oqtsuTFGjTNrqEPj2mpQK1VfdxjsEyDczurWRE0fulu9brxUySPO8DnvbFlZOr9Xjm4/tZ227Mv3bB/ZtbFP0D6vnKW1/XuPMmuUDU90D1UstVVNdbfKRpABAAAV1qZBRtChRnsPO4PM0PbZGtCmnu4ZHhh2wklNclZ/u25QYNlYb5k1kgMuzGqk2HVxH+cE9qsH5OqV3/VWg1ppESdUu4NCeYtmzXvwNC146HTP/c87n+w5VpIjusUgA0Y42ctKZPs/5l8gwpaR4elxSE9NKnufo0ZJacF7li7v11yj+7VQ24a1fHrDdmT4DVfq2DHo80d0bSRJunpAS3VuEjmk/uMfNznXgvJ6L+f2bKq+LetqwKFN0olB1hy65hqf4XrBSgKP6NLY2SN3yy0qsZft26FRbXVwzTsb0Lqe1LOn6mWkOj+bZGeA8i8YYYzR/53cVjcOaa3afosAG0m69FJd1KeZsoItanvVVbLbjPNcHDpU6lB2vjeqnaZ+owborHuuCXha7RpJumhoJ6UOHKBb7r/S0zMjSfb2wdeE8u8t9A73nnb/8Y/S8OHSI494foaj+7UICGiSZEtN0SddT9PH3U7znLeR1u2JFwwtAwAAlaZueor2Hi7yLHCYnpqk967rf8zb8bcLu+vWYW3VpE5a1HM6KjoXKstvnZaNWY1lt22UJGUUeq2VM2CAlJIiTZ0qlfgOFQp4bVc42XEwcI2kUleC8XSkjBwpffJJ8MZ17y7NmBGwuUGtsovhjLQkZaQmaUi7bNU+bZhqzf4t+LG8/K5/CxUPbaOUpOg+2yZZNaW0ZLWsn64pK3fphNy6qlMzxTl/5NRTpQZBhkMlJ/t0w7hvLmnYWp13rJEktW9UNtTQ4fUZDj+xrU676nfa//e9qpdipHXrpGbNpE2btDnT+Vr+Q9KksnlGQXupkpPVtE4NXelaL2jzviPO4WaPPOK738CB0rZt0nLnoqC2YUP152FdJUkXjPtWxkifztnsaYN3WPNmKyr03PYe9hdqfSZJuqRPM+nBB509Wf1dv3ft2knLlil7UD9pwQLfrq0LL1RWUm1tecu9YOYGSdLpA8r3xUOs0CMDAAAqzRhXJbBTO4WuMHWsNK9Xs1wT093XrkeznMpXtw7QhNsHeeZTHEjzqhp12mnSkCHSAw8Efe7h5Bpla8hcfbXOGdpZZ3dv4rzYHjTIs1+dmslq17CWRrqrlCWHKdjQzFWavH6IoXI5zp6r6wa1UtuGtdSwVY5qRlFlzBijlDNOK9tQK0wVrcsu83RzZdVM0e/P6Ow7UT0jwxnwrr1WuuceZ++Me0iZVzhxfzYdr7pIp3z4sq4d2NLnZSzvxWRuv132enVV78Q+zvu9e0tXXSXddJO2ZdTXp11Okc0414E686wTdf15vsMO7Taj93qM9NwP9pnkZNUMPpfKbvft1hs2rOw93HSVcrwKAdhN6CFrqllTqh1YHMHes0fZnRRniB7WoYGaZdVUepuWPsPxJEmdOjnDzLBh0m23lW2vU0fq0kUdO/j20tx+ajtdfUXwan3xhiADAAAqzR9OaavnL+up0+MgyMRCt5w6zjka112nH1ufIFt2iAVem/sN8enSRfdP/k/ZGjF16qjm3X9Sqz8/KLVo4ezN+d3vpLZtZTNGI7s2VoPaaVJbv8VP+7ouyFu18n2dfv2Ct+P0033v5+aG3te9//XXO297hSuNCrGGz+WXS+3b+wSSlFS/4YDu+THNmkk1akhnnOHsVXDr4lxPKbNGsm4/tZ3uGN5RXXMyy9b+ae2cE7Qp0+ucc1/Mz5nj/HvhQue2Ro3ksCxtzmwoc9pp0p13qs09/6f0C8/zabYZPly7MrJ0OLmGbhnWxlnyuk6dwPfXLsjwL2PCTOhxBpNzejRRi3rpgUMKvSUlOYeIuRxJTtNH3U6Xzj23bJ+bb5YuvVTdc+rogt45Ut0gxTxSU50/h8xM3/dw++0+u53iXSTgGC6GfDQIMgAAoNIk2W06u3uTKi1ZXVXck6ZDla0tl5wcvf3SLfr57mHBH7/6aumWW8ruhxhepMxM575pac4L9tGjyx674w7nffdFc7t2ZXNN3H/Xru0c+tTSt/cipBo1nEO9JGexAH8nnSQ1DTK53G6XLrkkcPuBA86/vc+HI67hdt27SxdcEL5HSXIO1fJ/LckZ8CRnT5ekn1v1dt4PdjHv5YvfD9B1g1rJNmigszfIv32S5/N7q89ZSv7dGGnsWCkrS2rSxHe/k3wXj/VwB9izzgr6cMv6GTqvZ1OZu+7yfcD1nhzG5uxtcbUrNcmuV/tdoI59O/u2NSvLOR/H3XMzdGjI9x3O4kfP0Cu/612h58YSc2QAAADkLMd8xYm5Uc/7iMR/7owPY8oudqWyINO0qTNMRMN/2FFhofPC1n/OhhR6+FKpXzGCVFd55PvvDx2uvJ1+uvTdd84L8JQU59Awh0N6+mnn4+6QEizY9u4d2DMVTKNGzvfkfl/u93LJJdLq1c7H5Vzt3vboo9IpXr1Ul18uvf++c+K7S49mddTDfz2dEGGq2J7s2+vSpo201WsifJChX5Kcn2Own4N3T02wxy+5RAvHrdfPrXrrDq/PbHDb+vqrpCfP6xL89Ro0kA4eDFncIZKAeTcJ8kUEQQYAAECueR9JMbqAc4cG97Ct8nBfHOfnh96nbl1pxAjp22/LttWpExhk3Lwv7O+6yxlMfv/7wP1OOsm3V8Idwtq0cYaMmq75IMEujMt7sXznndKePWX3a9Z0rpPjMvnOoYHPaddOuu8+z1ySkBo1ks48U/rmG8+mx87pHLgmkftz6dDB+doRen/KrWZN/dimb8Dmzk0zteahkb6LnrqHD0rShRdK27dHF4JHjw69X82azh4zggwAAACi4upViEpyslTstXCjeyJ/wwjzkvr1c+67bJk0e7azNyXIQpwB0tOD9x5Ew/+COCnJOTSssDC61/aWkVE2FKw8IoUYydnOPn18gswVJ+YG7te2rfTDD86hZ+6hbceIT4i5/37fSf1pac75TdHwn1flbcwY57yimjVD7xNHCDIAAACx0rChtGOHZ8J6VO6807cnpX595xCqaC6sW7d2fpPfr59zaJurRLAaNnR+q19ZzjpLmjLFt9fg3HOdE/q//lpav778QSYeuIe5BZOdLe3addQvcdVJuVq85UD4nSLNK6qoJk0C5wHFMYIMAABArIwe7QwT4coX+3PPY/EWrHpWKN7zc9xzPDp39p2zc7QyMwMnuvfo4fzbPcclEYNMODfcEHqonls9V9lp78pjfh45u3Pltek4R5ABAACIldq1y0omx0KTJtJNN0UellaZ4jnI1KwZvCpbNJKTI/eU1KxZ/mF6l17qLOKAAAQZAACA6qw883Mqw+mnO+fIHOM5JlG5++5YtyBQhw6xbkHcIsgAAADg2GnQQLr22li3AscBFsQEAAAAkHAIMgAAAAASDkEGAAAAQMIhyAAAAABIOAQZAAAAAAmHIAMAAAAg4RBkAAAAACQcggwAAACAhEOQAQAAAJBwCDIAAAAAEg5BBgAAAEDCIcgAAAAASDgEGQAAAAAJhyADAAAAIOEQZAAAAAAkHIIMAAAAgIRDkAEAAACQcAgyAAAAABIOQQYAAABAwiHIAAAAAEg4BBkAAAAACYcgAwAAACDhEGQAAAAAJBxjWVZsXtiYXZI2xOTFg6svaXesG4G4wjkBf5wT8Mc5AX+cE/DHOXF0WliWlR3sgZgFmXhjjJltWVafWLcD8YNzAv44J+CPcwL+OCfgj3Oi6jC0DAAAAEDCIcgAAAAASDgEmTKvxroBiDucE/DHOQF/nBPwxzkBf5wTVYQ5MgAAAAASDj0yAAAAABJOtQ8yxpjhxpgVxpjVxpixsW4PKpcx5g1jzE5jzGKvbXWNMd8bY1a5/s7yeuxe17mwwhhzhtf23saYRa7HnjfGGNf2VGPMR67tM4wxucf0DaLcjDHNjDGTjTHLjDFLjDG3ubZzXlRTxpg0Y8xMY8wC1znxqGs750Q1ZoyxG2PmGWO+cd3nfKjmjDHrXT/P+caY2a5tnBcxVK2DjDHGLulFSSMkdZJ0mTGmU2xbhUr2lqThftvGSvrBsqy2kn5w3ZfrZ3+ppM6u57zkOkck6WVJN0hq6/rjPua1kvZZltVG0j8k/bXK3gkqS4mkP1mW1VFSf0m3uH72nBfVV6Gkky3L6i6ph6Thxpj+4pyo7m6TtMzrPucDJGmYZVk9vMopc17EULUOMpL6SlptWdZay7KKJH0o6ZwYtwmVyLKsKZL2+m0+R9LbrttvSzrXa/uHlmUVWpa1TtJqSX2NMY0l1bYsa7rlnFT2X7/nuI/1qaRT3N+sID5ZlrXNsqy5rtt5cl6oNBXnRbVlOR1y3U12/bHEOVFtGWNyJI2S9LrXZs4HBMN5EUPVPcg0lbTJ6/5m1zYc3xpalrVNcl7USmrg2h7qfGjquu2/3ec5lmWVSDogqV6VtRyVytVt31PSDHFeVGuuYUTzJe2U9L1lWZwT1dtzku6W5PDaxvkAS9J3xpg5xpgbXNs4L2IoKdYNiLFgKZcybtVXqPMh3HnCOZSgjDEZkj6TdLtlWQfDfOnFeVENWJZVKqmHMaaOpC+MMV3C7M45cRwzxpwpaadlWXOMMUOjeUqQbZwPx6cBlmVtNcY0kPS9MWZ5mH05L46B6t4js1lSM6/7OZK2xqgtOHZ2uLp25fp7p2t7qPNhs+u2/3af5xhjkiRlKnAoG+KMMSZZzhDznmVZn7s2c15AlmXtl/STnGPWOSeqpwGSzjbGrJdzyPnJxph3xflQ7VmWtdX1905JX8g5RYHzIoaqe5CZJamtMaalMSZFzklZX8W4Tah6X0m60nX7Sklfem2/1FU1pKWcE/BmurqK84wx/V1jVa/we477WBdK+tFicaa45voZ/kfSMsuynvV6iPOimjLGZLt6YmSMqSHpVEnLxTlRLVmWda9lWTmWZeXKeV3wo2VZY8T5UK0ZY9KNMbXctyWdLmmxOC9iy7Ksav1H0khJKyWtkXR/rNvDn0r/+X4gaZukYjm/6bhWzvGmP0ha5fq7rtf+97vOhRWSRnht7yPnP1hrJL2gssVk0yR9IuckvpmSWsX6PfMn4jkxUM6u+oWS5rv+jOS8qL5/JHWTNM91TiyW9JBrO+dENf8jaaikbzgf+COplaQFrj9L3NeMnBex/eP+4AAAAAAgYVT3oWUAAAAAEhBBBgAAAEDCIcgAAAAASDgEGQAAAAAJhyADAAAAIOEQZAAAAAAkHIIMAAAAgIRDkAEAAACQcP4fh6yy7GPiWt8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1008x720 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fix, ax = plt.subplots(figsize=(14, 10))\n",
    "n_train_losses = len(train_mb_running_loss)\n",
    "ax.plot(range(n_train_losses), train_mb_running_loss, label=\"training losses\");\n",
    "ax.plot(range(n_train_losses-len(val_mb_running_loss), n_train_losses),\n",
    "        val_mb_running_loss, c=\"r\", alpha=0.5, label=\"validation losses\")\n",
    "ax.legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_ranked_docs(\n",
    "    tokenizer: AutoTokenizer, model: AutoModel, device: str, query: str, corpus: list[str]\n",
    ") -> None:\n",
    "    \n",
    "    query_embed = mean_pool(model(**tokenizer(query, return_tensors=\"pt\").to(device)).last_hidden_state,\n",
    "                            tokenizer(query, return_tensors=\"pt\")[\"attention_mask\"].to(device))\n",
    "\n",
    "    corpus_last_hidden_state = model(\n",
    "        **tokenizer(corpus, max_length=64, padding=True, truncation=True, return_tensors=\"pt\").to(device)\n",
    "    ).last_hidden_state\n",
    "\n",
    "    corpus_embeds = mean_pool(\n",
    "        corpus_last_hidden_state,\n",
    "        tokenizer(corpus, max_length=64, padding=True,truncation=True, return_tensors=\"pt\")[\"attention_mask\"].to(device)\n",
    "    )\n",
    "\n",
    "    scores = torch.nn.CosineSimilarity()(query_embed, corpus_embeds)\n",
    "    \n",
    "    print(f\"Query - {query}\\n---\")\n",
    "    scores = scores.cpu().detach().numpy()\n",
    "    scores_ix = np.argsort(scores)[::-1]\n",
    "    for ix in scores_ix:\n",
    "        print(f\"{scores[ix]: >.2f}\\t{corpus[ix]}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will load the original student model again to compare it's performance with the finetuned model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at xlm-roberta-base were not used when initializing XLMRobertaModel: ['lm_head.layer_norm.bias', 'lm_head.bias', 'lm_head.layer_norm.weight', 'lm_head.dense.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight']\n",
      "- This IS expected if you are initializing XLMRobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing XLMRobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
     ]
    }
   ],
   "source": [
    "original_student_model = AutoModel.from_pretrained(student_model_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"Who let the dogs out\"\n",
    "\n",
    "corpus = [\n",
    "    \"A frog jumped in the pond\",\n",
    "    \"My car broke down last night\",\n",
    "    \"किसने कुत्तों को खुला छोड़ा\",\n",
    "    \"Where are my dogs?\",\n",
    "    \"Are the dogs outside?\",\n",
    "    \"मेरी कार टूट गई\",\n",
    "    \"I wonder if they have reached home\",\n",
    "    \"मुझे आश्चर्य है कि क्या वे घर पहुंच गए हैं\"\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Query - Who let the dogs out\n",
      "---\n",
      "0.84\tकिसने कुत्तों को खुला छोड़ा\n",
      "0.67\tAre the dogs outside?\n",
      "0.63\tWhere are my dogs?\n",
      "0.40\tA frog jumped in the pond\n",
      "0.39\tमुझे आश्चर्य है कि क्या वे घर पहुंच गए हैं\n",
      "0.38\tI wonder if they have reached home\n",
      "0.17\tमेरी कार टूट गई\n",
      "0.13\tMy car broke down last night\n"
     ]
    }
   ],
   "source": [
    "get_ranked_docs(student_tokenizer, student_model, \"cuda\", query, corpus)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Query - Who let the dogs out\n",
      "---\n",
      "1.00\tAre the dogs outside?\n",
      "1.00\tI wonder if they have reached home\n",
      "1.00\tA frog jumped in the pond\n",
      "1.00\tWhere are my dogs?\n",
      "1.00\tMy car broke down last night\n",
      "1.00\tमेरी कार टूट गई\n",
      "1.00\tकिसने कुत्तों को खुला छोड़ा\n",
      "0.99\tमुझे आश्चर्य है कि क्या वे घर पहुंच गए हैं\n"
     ]
    }
   ],
   "source": [
    "get_ranked_docs(student_tokenizer, original_student_model, \"cpu\", query, corpus)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"My car broke down on the highway\"\n",
    "\n",
    "corpus = [\n",
    "    \"The roads on the highway were bad.\",\n",
    "    \"My car broke down last night\",\n",
    "    \"किसने कुत्तों को खुला छोड़ा\",\n",
    "    \"Where are my dogs?\",\n",
    "    \"Are the dogs outside?\",\n",
    "    \"मेरी कार टूट गई\",\n",
    "    \"मेरी कार हाईवे पर खराब हो गई\",\n",
    "    \"I wonder if they have reached home\",\n",
    "    \"मुझे आश्चर्य है कि क्या वे घर पहुंच गए हैं\"\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Query - My car broke down on the highway\n",
      "---\n",
      "0.94\tमेरी कार हाईवे पर खराब हो गई\n",
      "0.93\tMy car broke down last night\n",
      "0.93\tमेरी कार टूट गई\n",
      "0.68\tThe roads on the highway were bad.\n",
      "0.14\tI wonder if they have reached home\n",
      "0.12\tWhere are my dogs?\n",
      "0.08\tमुझे आश्चर्य है कि क्या वे घर पहुंच गए हैं\n",
      "0.06\tAre the dogs outside?\n",
      "0.06\tकिसने कुत्तों को खुला छोड़ा\n"
     ]
    }
   ],
   "source": [
    "get_ranked_docs(student_tokenizer, student_model, \"cuda\", query, corpus)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Query - My car broke down on the highway\n",
      "---\n",
      "1.00\tThe roads on the highway were bad.\n",
      "1.00\tMy car broke down last night\n",
      "1.00\tमेरी कार हाईवे पर खराब हो गई\n",
      "1.00\tमुझे आश्चर्य है कि क्या वे घर पहुंच गए हैं\n",
      "1.00\tI wonder if they have reached home\n",
      "1.00\tAre the dogs outside?\n",
      "0.99\tकिसने कुत्तों को खुला छोड़ा\n",
      "0.99\tWhere are my dogs?\n",
      "0.99\tमेरी कार टूट गई\n"
     ]
    }
   ],
   "source": [
    "get_ranked_docs(student_tokenizer, original_student_model, \"cpu\", query, corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our finetuned model is doing a great job in retrieving the most relevant documents from the corpus, both in english and hindi. The bert-base model is scoring all the documents exactly the same. Here are the hi-en translations for those who don't have access to internet.\n",
    "\n",
    "- किसने कुत्तों को खुला छोड़ा - Who let the dogs out\n",
    "- मेरी कार हाईवे पर खराब हो गई - My car broke down on the highway\n",
    "- मेरी कार टूट गई - My car broke down\n",
    "- मुझे आश्चर्य है कि क्या वे घर पहुंच गए हैं - I wonder if they've reached home\n",
    "\n",
    "I could not find any STS dataset between en-hi to evaluate this model effectively. \n",
    "\n",
    "---\n",
    "\n",
    "## References\n",
    "\n",
    "[1] Nils Reimers and Iryna Gurevych. \"[Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation](https://arxiv.org/pdf/2004.09813.pdf)\"\n",
    "\n",
    "[2] https://www.sbert.net"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}