Fine-Tuning/PT5_LoRA_Finetuning_per_prot.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "050d7dd4",
   "metadata": {},
   "source": [
    "# Prot T5 Finetuning\n",
    "# per protein prediction\n",
    "\n",
    "This notebook allows you to finetune ProtT5 to your own datasets\n",
    "\n",
    "The protein language model ProtT5 was first published [here](https://ieeexplore.ieee.org/document/9477085) and is available on [github](https://github.com/agemagician/ProtTrans). We use the [huggingface](https://huggingface.co/Rostlab/prot_t5_xl_uniref50) checkpoint.\n",
    "\n",
    "For better perfomance we apply [Parameter-Efficient Fine-Tuning (PEFT)](https://huggingface.co/blog/peft). For this we apply [LoRA: Low-Rank Adaptation of Large Language Models](https://arxiv.org/abs/2106.09685).\n",
    "\n",
    "For higher memory efficiency we also utilize the [deepspeed](https://github.com/microsoft/DeepSpeed) implementation of [huggingface](https://huggingface.co/docs/accelerate/usage_guides/deepspeed).\n",
    "\n",
    "The core training loop is implemented with the pytorch [huggingface trainer](https://huggingface.co/docs/transformers/main_classes/trainer) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dde58a20",
   "metadata": {},
   "source": [
    "## Imports and env. variables\n",
    "\n",
    "\n",
    "**set a working dir below** model weights will be saved there"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "angry-toronto",
   "metadata": {},
   "outputs": [],
   "source": [
    "#import dependencies\n",
    "import os.path\n",
    "os.chdir(\"set a path here\")\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "from torch.nn import BCEWithLogitsLoss, CrossEntropyLoss, MSELoss\n",
    "from torch.utils.data import DataLoader\n",
    "\n",
    "import re\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import copy\n",
    "\n",
    "import transformers, datasets\n",
    "from transformers.modeling_outputs import SequenceClassifierOutput\n",
    "from transformers.models.t5.modeling_t5 import T5Config, T5PreTrainedModel, T5Stack\n",
    "from transformers.utils.model_parallel_utils import assert_device_map, get_device_map\n",
    "from transformers import T5EncoderModel, T5Tokenizer\n",
    "from transformers import TrainingArguments, Trainer, set_seed\n",
    "\n",
    "from evaluate import load\n",
    "from datasets import Dataset\n",
    "\n",
    "from tqdm import tqdm\n",
    "import random\n",
    "\n",
    "from scipy import stats\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "8534fbd4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set environment variables to run Deepspeed from a notebook\n",
    "os.environ[\"MASTER_ADDR\"] = \"localhost\"\n",
    "os.environ[\"MASTER_PORT\"] = \"9994\"  # modify if RuntimeError: Address already in use\n",
    "os.environ[\"RANK\"] = \"0\"\n",
    "os.environ[\"LOCAL_RANK\"] = \"0\"\n",
    "os.environ[\"WORLD_SIZE\"] = \"1\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24e21e96",
   "metadata": {},
   "source": [
    "# Environment to run this notebook\n",
    "\n",
    "\n",
    "These are the versions of the core packages we use to run this notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b2ccc1eb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Torch version:  1.13.1\n",
      "Cuda version:  11.7\n",
      "Numpy version:  1.22.3\n",
      "Pandas version:  1.5.3\n",
      "Transformers version:  4.26.1\n",
      "Datasets version:  2.9.0\n"
     ]
    }
   ],
   "source": [
    "print(\"Torch version: \",torch.__version__)\n",
    "print(\"Cuda version: \",torch.version.cuda)\n",
    "print(\"Numpy version: \",np.__version__)\n",
    "print(\"Pandas version: \",pd.__version__)\n",
    "print(\"Transformers version: \",transformers.__version__)\n",
    "print(\"Datasets version: \",datasets.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fcff648f",
   "metadata": {},
   "source": [
    "**For easy setup of this conda environment you can use the finetuning.yml File provided in this folder**\n",
    "\n",
    "check here for [setting up env from a yml File](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#creating-an-environment-from-an-environment-yml-file)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb2dda19",
   "metadata": {},
   "source": [
    "# Input data\n",
    "\n",
    "Provide your training and validation data in seperate pandas dataframes \n",
    "\n",
    "example shown below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "9584b17b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# For this example we import the \"three_vs_rest\" GB1 dataset from https://github.com/J-SNACKKB/FLIP\n",
    "# For details, see publication here: https://openreview.net/forum?id=p2dMLEwL8tF\n",
    "import requests\n",
    "import zipfile\n",
    "from io import BytesIO\n",
    "\n",
    "# Download the zip file from GitHub\n",
    "url = 'https://github.com/J-SNACKKB/FLIP/raw/main/splits/gb1/splits.zip'\n",
    "response = requests.get(url)\n",
    "zip_file = zipfile.ZipFile(BytesIO(response.content))\n",
    "\n",
    "# Load the `three_vs_rest.csv` file into a pandas dataframe\n",
    "with zip_file.open('splits/three_vs_rest.csv') as file:\n",
    "    df = pd.read_csv(file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "e5b6f186",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop test data\n",
    "df=df[df.set==\"train\"]\n",
    "\n",
    "# Get train and validation data\n",
    "my_train=df[df.validation!=True].reset_index(drop=True)\n",
    "my_valid=df[df.validation==True].reset_index(drop=True)\n",
    "\n",
    "# Set column names to \"sequence\" and \"label\"\n",
    "my_train.columns=[\"sequence\",\"label\"]+list(my_train.columns[2:])\n",
    "my_valid.columns=[\"sequence\",\"label\"]+list(my_valid.columns[2:])\n",
    "\n",
    "# Drop unneeded columns\n",
    "my_train=my_train[[\"sequence\",\"label\"]]\n",
    "my_valid=my_valid[[\"sequence\",\"label\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c012178",
   "metadata": {},
   "source": [
    "**Modify the data loading part above as needed for your data**\n",
    "\n",
    "To run the training you need two dataframes (training and validation) each with the columns \"sequence\" and \"label\"\n",
    "\n",
    "Labels are:\n",
    "+ a float value for **regression**\n",
    "+ the class as an integer for **classification** (from 0 to number of classes - 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "14f14d83",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sequence</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>MQYKLILNGKTLKGETTTEAVDAATAEKVFKQYANDNGVDGEWTYD...</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>MQYKLILNGKTLKGETTTEAVDAATAEKVFKQYANDNGIDGEWTYD...</td>\n",
       "      <td>1.445905</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>MQYKLILNGKTLKGETTTEAVDAATAEKVFKQYANDNGLDGEWTYD...</td>\n",
       "      <td>1.690164</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>MQYKLILNGKTLKGETTTEAVDAATAEKVFKQYANDNGMDGEWTYD...</td>\n",
       "      <td>1.170550</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>MQYKLILNGKTLKGETTTEAVDAATAEKVFKQYANDNGVAGEWTYD...</td>\n",
       "      <td>2.401243</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            sequence     label\n",
       "0  MQYKLILNGKTLKGETTTEAVDAATAEKVFKQYANDNGVDGEWTYD...  1.000000\n",
       "1  MQYKLILNGKTLKGETTTEAVDAATAEKVFKQYANDNGIDGEWTYD...  1.445905\n",
       "2  MQYKLILNGKTLKGETTTEAVDAATAEKVFKQYANDNGLDGEWTYD...  1.690164\n",
       "3  MQYKLILNGKTLKGETTTEAVDAATAEKVFKQYANDNGMDGEWTYD...  1.170550\n",
       "4  MQYKLILNGKTLKGETTTEAVDAATAEKVFKQYANDNGVAGEWTYD...  2.401243"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_train.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "a71fc6d8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sequence</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>MQYKLILNGKTLKGETTTEAVDAATAEKVFKQYANDNGVEGEWTYD...</td>\n",
       "      <td>0.860931</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>MQYKLILNGKTLKGETTTEAVDAATAEKVFKQYANDNGVQGEWTYD...</td>\n",
       "      <td>1.113894</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>MQYKLILNGKTLKGETTTEAVDAATAEKVFKQYANDNGVTGEWTYD...</td>\n",
       "      <td>1.165465</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>MQYKLILNGKTLKGETTTEAVDAATAEKVFKQYANDNGVYGEWTYD...</td>\n",
       "      <td>3.891461</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>MQYKLILNGKTLKGETTTEAVDAATAEKVFKQYANDNGCIGEWTYD...</td>\n",
       "      <td>1.172411</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            sequence     label\n",
       "0  MQYKLILNGKTLKGETTTEAVDAATAEKVFKQYANDNGVEGEWTYD...  0.860931\n",
       "1  MQYKLILNGKTLKGETTTEAVDAATAEKVFKQYANDNGVQGEWTYD...  1.113894\n",
       "2  MQYKLILNGKTLKGETTTEAVDAATAEKVFKQYANDNGVTGEWTYD...  1.165465\n",
       "3  MQYKLILNGKTLKGETTTEAVDAATAEKVFKQYANDNGVYGEWTYD...  3.891461\n",
       "4  MQYKLILNGKTLKGETTTEAVDAATAEKVFKQYANDNGCIGEWTYD...  1.172411"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_valid.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b996723",
   "metadata": {},
   "source": [
    "# PT5 Model and Low Rank Adaptation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "responsible-standing",
   "metadata": {},
   "source": [
    "## LoRA modification definition\n",
    "\n",
    "Implementation taken from https://github.com/r-three/t-few\n",
    "\n",
    "(https://github.com/r-three/t-few/blob/master/src/models/lora.py, https://github.com/r-three/t-few/tree/master/configs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "separated-grenada",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Modifies an existing transformer and introduce the LoRA layers\n",
    "\n",
    "class LoRAConfig:\n",
    "    def __init__(self):\n",
    "        self.lora_rank = 4\n",
    "        self.lora_init_scale = 0.01\n",
    "        self.lora_modules = \".*SelfAttention|.*EncDecAttention\"\n",
    "        self.lora_layers = \"q|k|v|o\"\n",
    "        self.trainable_param_names = \".*layer_norm.*|.*lora_[ab].*\"\n",
    "        self.lora_scaling_rank = 1\n",
    "        # lora_modules and lora_layers are speicified with regular expressions\n",
    "        # see https://www.w3schools.com/python/python_regex.asp for reference\n",
    "        \n",
    "class LoRALinear(nn.Module):\n",
    "    def __init__(self, linear_layer, rank, scaling_rank, init_scale):\n",
    "        super().__init__()\n",
    "        self.in_features = linear_layer.in_features\n",
    "        self.out_features = linear_layer.out_features\n",
    "        self.rank = rank\n",
    "        self.scaling_rank = scaling_rank\n",
    "        self.weight = linear_layer.weight\n",
    "        self.bias = linear_layer.bias\n",
    "        if self.rank > 0:\n",
    "            self.lora_a = nn.Parameter(torch.randn(rank, linear_layer.in_features) * init_scale)\n",
    "            if init_scale < 0:\n",
    "                self.lora_b = nn.Parameter(torch.randn(linear_layer.out_features, rank) * init_scale)\n",
    "            else:\n",
    "                self.lora_b = nn.Parameter(torch.zeros(linear_layer.out_features, rank))\n",
    "        if self.scaling_rank:\n",
    "            self.multi_lora_a = nn.Parameter(\n",
    "                torch.ones(self.scaling_rank, linear_layer.in_features)\n",
    "                + torch.randn(self.scaling_rank, linear_layer.in_features) * init_scale\n",
    "            )\n",
    "            if init_scale < 0:\n",
    "                self.multi_lora_b = nn.Parameter(\n",
    "                    torch.ones(linear_layer.out_features, self.scaling_rank)\n",
    "                    + torch.randn(linear_layer.out_features, self.scaling_rank) * init_scale\n",
    "                )\n",
    "            else:\n",
    "                self.multi_lora_b = nn.Parameter(torch.ones(linear_layer.out_features, self.scaling_rank))\n",
    "\n",
    "    def forward(self, input):\n",
    "        if self.scaling_rank == 1 and self.rank == 0:\n",
    "            # parsimonious implementation for ia3 and lora scaling\n",
    "            if self.multi_lora_a.requires_grad:\n",
    "                hidden = F.linear((input * self.multi_lora_a.flatten()), self.weight, self.bias)\n",
    "            else:\n",
    "                hidden = F.linear(input, self.weight, self.bias)\n",
    "            if self.multi_lora_b.requires_grad:\n",
    "                hidden = hidden * self.multi_lora_b.flatten()\n",
    "            return hidden\n",
    "        else:\n",
    "            # general implementation for lora (adding and scaling)\n",
    "            weight = self.weight\n",
    "            if self.scaling_rank:\n",
    "                weight = weight * torch.matmul(self.multi_lora_b, self.multi_lora_a) / self.scaling_rank\n",
    "            if self.rank:\n",
    "                weight = weight + torch.matmul(self.lora_b, self.lora_a) / self.rank\n",
    "            return F.linear(input, weight, self.bias)\n",
    "\n",
    "    def extra_repr(self):\n",
    "        return \"in_features={}, out_features={}, bias={}, rank={}, scaling_rank={}\".format(\n",
    "            self.in_features, self.out_features, self.bias is not None, self.rank, self.scaling_rank\n",
    "        )\n",
    "\n",
    "\n",
    "def modify_with_lora(transformer, config):\n",
    "    for m_name, module in dict(transformer.named_modules()).items():\n",
    "        if re.fullmatch(config.lora_modules, m_name):\n",
    "            for c_name, layer in dict(module.named_children()).items():\n",
    "                if re.fullmatch(config.lora_layers, c_name):\n",
    "                    assert isinstance(\n",
    "                        layer, nn.Linear\n",
    "                    ), f\"LoRA can only be applied to torch.nn.Linear, but {layer} is {type(layer)}.\"\n",
    "                    setattr(\n",
    "                        module,\n",
    "                        c_name,\n",
    "                        LoRALinear(layer, config.lora_rank, config.lora_scaling_rank, config.lora_init_scale),\n",
    "                    )\n",
    "    return transformer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "electronic-channels",
   "metadata": {},
   "source": [
    "## Classification model definition \n",
    "\n",
    "adding a classifcation or regression head (num_labels = 1) on top of the encoder model\n",
    "\n",
    "modified from https://gist.github.com/sam-writer/723baf81c501d9d24c6955f201d86bbb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "acting-archives",
   "metadata": {},
   "outputs": [],
   "source": [
    "class ClassConfig:\n",
    "    def __init__(self, dropout=0.2, num_labels=1):\n",
    "        self.dropout_rate = dropout\n",
    "        self.num_labels = num_labels\n",
    "\n",
    "class T5EncoderClassificationHead(nn.Module):\n",
    "    \"\"\"Head for sentence-level classification tasks.\"\"\"\n",
    "\n",
    "    def __init__(self, config, class_config):\n",
    "        super().__init__()\n",
    "        self.dense = nn.Linear(config.hidden_size, config.hidden_size)\n",
    "        self.dropout = nn.Dropout(class_config.dropout_rate)\n",
    "        self.out_proj = nn.Linear(config.hidden_size, class_config.num_labels)\n",
    "\n",
    "    def forward(self, hidden_states):\n",
    "\n",
    "        hidden_states =  torch.mean(hidden_states,dim=1)  # avg embedding\n",
    "\n",
    "        hidden_states = self.dropout(hidden_states)\n",
    "        hidden_states = self.dense(hidden_states)\n",
    "        hidden_states = torch.tanh(hidden_states)\n",
    "        hidden_states = self.dropout(hidden_states)\n",
    "        hidden_states = self.out_proj(hidden_states)\n",
    "        return hidden_states\n",
    "\n",
    "class T5EncoderForSimpleSequenceClassification(T5PreTrainedModel):\n",
    "\n",
    "    def __init__(self, config: T5Config, class_config):\n",
    "        super().__init__(config)\n",
    "        self.num_labels = class_config.num_labels\n",
    "        self.config = config\n",
    "\n",
    "        self.shared = nn.Embedding(config.vocab_size, config.d_model)\n",
    "\n",
    "        encoder_config = copy.deepcopy(config)\n",
    "        encoder_config.use_cache = False\n",
    "        encoder_config.is_encoder_decoder = False\n",
    "        self.encoder = T5Stack(encoder_config, self.shared)\n",
    "\n",
    "        self.dropout = nn.Dropout(class_config.dropout_rate) \n",
    "        self.classifier = T5EncoderClassificationHead(config, class_config)\n",
    "\n",
    "        # Initialize weights and apply final processing\n",
    "        self.post_init()\n",
    "\n",
    "        # Model parallel\n",
    "        self.model_parallel = False\n",
    "        self.device_map = None\n",
    "\n",
    "    def parallelize(self, device_map=None):\n",
    "        self.device_map = (\n",
    "            get_device_map(len(self.encoder.block), range(torch.cuda.device_count()))\n",
    "            if device_map is None\n",
    "            else device_map\n",
    "        )\n",
    "        assert_device_map(self.device_map, len(self.encoder.block))\n",
    "        self.encoder.parallelize(self.device_map)\n",
    "        self.classifier = self.classifier.to(self.encoder.first_device)\n",
    "        self.model_parallel = True\n",
    "\n",
    "    def deparallelize(self):\n",
    "        self.encoder.deparallelize()\n",
    "        self.encoder = self.encoder.to(\"cpu\")\n",
    "        self.model_parallel = False\n",
    "        self.device_map = None\n",
    "        torch.cuda.empty_cache()\n",
    "\n",
    "    def get_input_embeddings(self):\n",
    "        return self.shared\n",
    "\n",
    "    def set_input_embeddings(self, new_embeddings):\n",
    "        self.shared = new_embeddings\n",
    "        self.encoder.set_input_embeddings(new_embeddings)\n",
    "\n",
    "    def get_encoder(self):\n",
    "        return self.encoder\n",
    "\n",
    "    def _prune_heads(self, heads_to_prune):\n",
    "        \"\"\"\n",
    "        Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base\n",
    "        class PreTrainedModel\n",
    "        \"\"\"\n",
    "        for layer, heads in heads_to_prune.items():\n",
    "            self.encoder.layer[layer].attention.prune_heads(heads)\n",
    "\n",
    "    def forward(\n",
    "        self,\n",
    "        input_ids=None,\n",
    "        attention_mask=None,\n",
    "        head_mask=None,\n",
    "        inputs_embeds=None,\n",
    "        labels=None,\n",
    "        output_attentions=None,\n",
    "        output_hidden_states=None,\n",
    "        return_dict=None,\n",
    "    ):\n",
    "        return_dict = return_dict if return_dict is not None else self.config.use_return_dict\n",
    "\n",
    "        outputs = self.encoder(\n",
    "            input_ids=input_ids,\n",
    "            attention_mask=attention_mask,\n",
    "            inputs_embeds=inputs_embeds,\n",
    "            head_mask=head_mask,\n",
    "            output_attentions=output_attentions,\n",
    "            output_hidden_states=output_hidden_states,\n",
    "            return_dict=return_dict,\n",
    "        )\n",
    "\n",
    "        hidden_states = outputs[0]\n",
    "        logits = self.classifier(hidden_states)\n",
    "\n",
    "        loss = None\n",
    "        if labels is not None:\n",
    "            if self.config.problem_type is None:\n",
    "                if self.num_labels == 1:\n",
    "                    self.config.problem_type = \"regression\"\n",
    "                elif self.num_labels > 1 and (labels.dtype == torch.long or labels.dtype == torch.int):\n",
    "                    self.config.problem_type = \"single_label_classification\"\n",
    "                else:\n",
    "                    self.config.problem_type = \"multi_label_classification\"\n",
    "\n",
    "            if self.config.problem_type == \"regression\":\n",
    "                loss_fct = MSELoss()\n",
    "                if self.num_labels == 1:\n",
    "                    loss = loss_fct(logits.squeeze(), labels.squeeze())\n",
    "                else:\n",
    "                    loss = loss_fct(logits, labels)\n",
    "            elif self.config.problem_type == \"single_label_classification\":\n",
    "                loss_fct = CrossEntropyLoss()\n",
    "                loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))\n",
    "            elif self.config.problem_type == \"multi_label_classification\":\n",
    "                loss_fct = BCEWithLogitsLoss()\n",
    "                loss = loss_fct(logits, labels)\n",
    "        if not return_dict:\n",
    "            output = (logits,) + outputs[1:]\n",
    "            return ((loss,) + output) if loss is not None else output\n",
    "\n",
    "        return SequenceClassifierOutput(\n",
    "            loss=loss,\n",
    "            logits=logits,\n",
    "            hidden_states=outputs.hidden_states,\n",
    "            attentions=outputs.attentions,\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71a0e217",
   "metadata": {},
   "source": [
    "## Modified ProtT5 model\n",
    "this creates a ProtT5 model with prediction head and LoRA modification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "split-austin",
   "metadata": {},
   "outputs": [],
   "source": [
    "def PT5_classification_model(num_labels, half_precision):\n",
    "    # Load PT5 and tokenizer\n",
    "    # possible to load the half preciion model (thanks to @pawel-rezo for pointing that out)\n",
    "    if not half_precision:\n",
    "        model = T5EncoderModel.from_pretrained(\"Rostlab/prot_t5_xl_uniref50\")\n",
    "        tokenizer = T5Tokenizer.from_pretrained(\"Rostlab/prot_t5_xl_uniref50\")\n",
    "    elif half_precision and torch.cuda.is_available() : \n",
    "        tokenizer = T5Tokenizer.from_pretrained('Rostlab/prot_t5_xl_half_uniref50-enc', do_lower_case=False)\n",
    "        model = T5EncoderModel.from_pretrained(\"Rostlab/prot_t5_xl_half_uniref50-enc\", torch_dtype=torch.float16).to(torch.device('cuda'))\n",
    "    else:\n",
    "          raise ValueError('Half precision can be run on GPU only.')\n",
    "    \n",
    "    # Create new Classifier model with PT5 dimensions\n",
    "    class_config=ClassConfig(num_labels=num_labels)\n",
    "    class_model=T5EncoderForSimpleSequenceClassification(model.config,class_config)\n",
    "    \n",
    "    # Set encoder and embedding weights to checkpoint weights\n",
    "    class_model.shared=model.shared\n",
    "    class_model.encoder=model.encoder    \n",
    "    \n",
    "    # Delete the checkpoint model\n",
    "    model=class_model\n",
    "    del class_model\n",
    "    \n",
    "    # Print number of trainable parameters\n",
    "    model_parameters = filter(lambda p: p.requires_grad, model.parameters())\n",
    "    params = sum([np.prod(p.size()) for p in model_parameters])\n",
    "    print(\"ProtT5_Classfier\\nTrainable Parameter: \"+ str(params))    \n",
    " \n",
    "    # Add model modification lora\n",
    "    config = LoRAConfig()\n",
    "    \n",
    "    # Add LoRA layers\n",
    "    model = modify_with_lora(model, config)\n",
    "    \n",
    "    # Freeze Embeddings and Encoder (except LoRA)\n",
    "    for (param_name, param) in model.shared.named_parameters():\n",
    "                param.requires_grad = False\n",
    "    for (param_name, param) in model.encoder.named_parameters():\n",
    "                param.requires_grad = False       \n",
    "\n",
    "    for (param_name, param) in model.named_parameters():\n",
    "            if re.fullmatch(config.trainable_param_names, param_name):\n",
    "                param.requires_grad = True\n",
    "\n",
    "    # Print trainable Parameter          \n",
    "    model_parameters = filter(lambda p: p.requires_grad, model.parameters())\n",
    "    params = sum([np.prod(p.size()) for p in model_parameters])\n",
    "    print(\"ProtT5_LoRA_Classfier\\nTrainable Parameter: \"+ str(params) + \"\\n\")\n",
    "    \n",
    "    return model, tokenizer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "beautiful-yeast",
   "metadata": {},
   "source": [
    "# Training Definition "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e735e819",
   "metadata": {},
   "source": [
    "## Deepspeed config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "eed91c2e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Deepspeed config for optimizer CPU offload\n",
    "\n",
    "ds_config = {\n",
    "    \"fp16\": {\n",
    "        \"enabled\": \"auto\",\n",
    "        \"loss_scale\": 0,\n",
    "        \"loss_scale_window\": 1000,\n",
    "        \"initial_scale_power\": 16,\n",
    "        \"hysteresis\": 2,\n",
    "        \"min_loss_scale\": 1\n",
    "    },\n",
    "\n",
    "    \"optimizer\": {\n",
    "        \"type\": \"AdamW\",\n",
    "        \"params\": {\n",
    "            \"lr\": \"auto\",\n",
    "            \"betas\": \"auto\",\n",
    "            \"eps\": \"auto\",\n",
    "            \"weight_decay\": \"auto\"\n",
    "        }\n",
    "    },\n",
    "\n",
    "    \"scheduler\": {\n",
    "        \"type\": \"WarmupLR\",\n",
    "        \"params\": {\n",
    "            \"warmup_min_lr\": \"auto\",\n",
    "            \"warmup_max_lr\": \"auto\",\n",
    "            \"warmup_num_steps\": \"auto\"\n",
    "        }\n",
    "    },\n",
    "\n",
    "    \"zero_optimization\": {\n",
    "        \"stage\": 2,\n",
    "        \"offload_optimizer\": {\n",
    "            \"device\": \"cpu\",\n",
    "            \"pin_memory\": True\n",
    "        },\n",
    "        \"allgather_partitions\": True,\n",
    "        \"allgather_bucket_size\": 2e8,\n",
    "        \"overlap_comm\": True,\n",
    "        \"reduce_scatter\": True,\n",
    "        \"reduce_bucket_size\": 2e8,\n",
    "        \"contiguous_gradients\": True\n",
    "    },\n",
    "\n",
    "    \"gradient_accumulation_steps\": \"auto\",\n",
    "    \"gradient_clipping\": \"auto\",\n",
    "    \"steps_per_print\": 2000,\n",
    "    \"train_batch_size\": \"auto\",\n",
    "    \"train_micro_batch_size_per_gpu\": \"auto\",\n",
    "    \"wall_clock_breakdown\": False\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92962861",
   "metadata": {},
   "source": [
    "## Training functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "liberal-learning",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set random seeds for reproducibility of your trainings run\n",
    "def set_seeds(s):\n",
    "    torch.manual_seed(s)\n",
    "    np.random.seed(s)\n",
    "    random.seed(s)\n",
    "    set_seed(s)\n",
    "\n",
    "# Dataset creation\n",
    "def create_dataset(tokenizer,seqs,labels):\n",
    "    tokenized = tokenizer(seqs, max_length=1024, padding=False, truncation=True)\n",
    "    dataset = Dataset.from_dict(tokenized)\n",
    "    dataset = dataset.add_column(\"labels\", labels)\n",
    "\n",
    "    return dataset\n",
    "    \n",
    "# Main training fuction\n",
    "def train_per_protein(\n",
    "        train_df,         #training data\n",
    "        valid_df,         #validation data      \n",
    "        num_labels= 1,    #1 for regression, >1 for classification\n",
    "    \n",
    "        # effective training batch size is batch * accum\n",
    "        # we recommend an effective batch size of 8 \n",
    "        batch= 4,         #for training\n",
    "        accum= 2,         #gradient accumulation\n",
    "    \n",
    "        val_batch = 16,   #batch size for evaluation\n",
    "        epochs= 10,       #training epochs\n",
    "        lr= 3e-4,         #recommended learning rate\n",
    "        seed= 42,         #random seed\n",
    "        deepspeed= True,  #if gpu is large enough disable deepspeed for training speedup\n",
    "        mixed= False,     #enable mixed precision training\n",
    "        gpu= 1 ):         #gpu selection (1 for first gpu)\n",
    "\n",
    "    # Set gpu device\n",
    "    os.environ[\"CUDA_VISIBLE_DEVICES\"]=str(gpu-1)\n",
    "    \n",
    "    # Set all random seeds\n",
    "    set_seeds(seed)\n",
    "    \n",
    "    # load model\n",
    "    model, tokenizer = PT5_classification_model(num_labels=num_labels)\n",
    "\n",
    "    # Preprocess inputs\n",
    "    # Replace uncommon AAs with \"X\"\n",
    "    train_df[\"sequence\"]=train_df[\"sequence\"].str.replace('|'.join([\"O\",\"B\",\"U\",\"Z\"]),\"X\",regex=True)\n",
    "    valid_df[\"sequence\"]=valid_df[\"sequence\"].str.replace('|'.join([\"O\",\"B\",\"U\",\"Z\"]),\"X\",regex=True)\n",
    "    # Add spaces between each amino acid for PT5 to correctly use them\n",
    "    train_df['sequence']=train_df.apply(lambda row : \" \".join(row[\"sequence\"]), axis = 1)\n",
    "    valid_df['sequence']=valid_df.apply(lambda row : \" \".join(row[\"sequence\"]), axis = 1)\n",
    "\n",
    "    # Create Datasets\n",
    "    train_set=create_dataset(tokenizer,list(train_df['sequence']),list(train_df['label']))\n",
    "    valid_set=create_dataset(tokenizer,list(valid_df['sequence']),list(valid_df['label']))\n",
    "\n",
    "    # Huggingface Trainer arguments\n",
    "    args = TrainingArguments(\n",
    "        \"./\",\n",
    "        evaluation_strategy = \"epoch\",\n",
    "        logging_strategy = \"epoch\",\n",
    "        save_strategy = \"no\",\n",
    "        learning_rate=lr,\n",
    "        per_device_train_batch_size=batch,\n",
    "        per_device_eval_batch_size=val_batch,\n",
    "        gradient_accumulation_steps=accum,\n",
    "        num_train_epochs=epochs,\n",
    "        seed = seed,\n",
    "        deepspeed= ds_config if deepspeed else None,\n",
    "        fp16 = mixed,\n",
    "    ) \n",
    "\n",
    "    # Metric definition for validation data\n",
    "    def compute_metrics(eval_pred):\n",
    "        if num_labels>1:  # for classification\n",
    "            metric = load(\"accuracy\")\n",
    "            predictions, labels = eval_pred\n",
    "            predictions = np.argmax(predictions, axis=1)\n",
    "        else:  # for regression\n",
    "            metric = load(\"spearmanr\")\n",
    "            predictions, labels = eval_pred\n",
    "\n",
    "        return metric.compute(predictions=predictions, references=labels)\n",
    "    \n",
    "    # Trainer          \n",
    "    trainer = Trainer(\n",
    "        model,\n",
    "        args,\n",
    "        train_dataset=train_set,\n",
    "        eval_dataset=valid_set,\n",
    "        tokenizer=tokenizer,\n",
    "        compute_metrics=compute_metrics\n",
    "    )    \n",
    "    \n",
    "    # Train model\n",
    "    trainer.train()\n",
    "\n",
    "    return tokenizer, model, trainer.state.log_history\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ac94ab1",
   "metadata": {},
   "source": [
    "# Run Training"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ede09d5",
   "metadata": {},
   "source": [
    "## Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "muslim-hearts",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at Rostlab/prot_t5_xl_uniref50 were not used when initializing T5EncoderModel: ['decoder.block.6.layer.0.SelfAttention.q.weight', 'decoder.block.10.layer.2.DenseReluDense.wo.weight', 'decoder.block.21.layer.1.layer_norm.weight', 'decoder.block.22.layer.1.EncDecAttention.q.weight', 'decoder.block.19.layer.0.SelfAttention.o.weight', 'decoder.block.10.layer.0.SelfAttention.k.weight', 'decoder.block.12.layer.0.SelfAttention.v.weight', 'decoder.block.8.layer.1.EncDecAttention.v.weight', 'decoder.block.19.layer.1.EncDecAttention.q.weight', 'decoder.block.8.layer.2.layer_norm.weight', 'decoder.block.15.layer.1.EncDecAttention.k.weight', 'decoder.block.17.layer.0.layer_norm.weight', 'decoder.block.21.layer.0.SelfAttention.v.weight', 'decoder.block.15.layer.2.DenseReluDense.wi.weight', 'decoder.block.8.layer.2.DenseReluDense.wi.weight', 'decoder.block.13.layer.2.layer_norm.weight', 'decoder.block.20.layer.0.SelfAttention.q.weight', 'decoder.block.17.layer.2.DenseReluDense.wi.weight', 'decoder.block.20.layer.1.EncDecAttention.v.weight', 'decoder.block.14.layer.1.layer_norm.weight', 'decoder.block.13.layer.1.EncDecAttention.q.weight', 'decoder.block.5.layer.2.DenseReluDense.wi.weight', 'decoder.block.1.layer.1.EncDecAttention.q.weight', 'decoder.block.23.layer.1.EncDecAttention.o.weight', 'decoder.block.21.layer.0.SelfAttention.o.weight', 'decoder.block.23.layer.2.layer_norm.weight', 'decoder.block.0.layer.1.EncDecAttention.q.weight', 'decoder.block.9.layer.0.SelfAttention.k.weight', 'decoder.block.20.layer.1.EncDecAttention.q.weight', 'decoder.block.6.layer.1.EncDecAttention.k.weight', 'decoder.block.5.layer.2.DenseReluDense.wo.weight', 'decoder.block.12.layer.2.DenseReluDense.wi.weight', 'decoder.block.3.layer.1.layer_norm.weight', 'decoder.block.2.layer.2.DenseReluDense.wo.weight', 'decoder.block.11.layer.2.layer_norm.weight', 'decoder.block.19.layer.2.layer_norm.weight', 'decoder.block.7.layer.2.DenseReluDense.wo.weight', 'decoder.block.5.layer.1.layer_norm.weight', 'decoder.block.13.layer.1.EncDecAttention.v.weight', 'decoder.block.19.layer.0.SelfAttention.k.weight', 'decoder.block.19.layer.1.EncDecAttention.o.weight', 'decoder.block.17.layer.0.SelfAttention.v.weight', 'decoder.block.4.layer.2.layer_norm.weight', 'decoder.block.12.layer.1.EncDecAttention.o.weight', 'decoder.block.1.layer.2.layer_norm.weight', 'decoder.block.3.layer.2.layer_norm.weight', 'decoder.block.17.layer.1.EncDecAttention.k.weight', 'decoder.block.14.layer.0.layer_norm.weight', 'decoder.block.9.layer.0.SelfAttention.o.weight', 'decoder.block.9.layer.0.layer_norm.weight', 'decoder.block.17.layer.2.layer_norm.weight', 'decoder.block.22.layer.1.EncDecAttention.o.weight', 'decoder.block.5.layer.1.EncDecAttention.q.weight', 'decoder.block.11.layer.0.SelfAttention.k.weight', 'decoder.block.22.layer.2.DenseReluDense.wi.weight', 'decoder.block.10.layer.0.layer_norm.weight', 'decoder.block.15.layer.0.SelfAttention.q.weight', 'decoder.block.14.layer.2.layer_norm.weight', 'decoder.block.4.layer.0.SelfAttention.q.weight', 'decoder.block.7.layer.0.layer_norm.weight', 'decoder.block.20.layer.2.DenseReluDense.wo.weight', 'decoder.block.11.layer.2.DenseReluDense.wi.weight', 'decoder.block.3.layer.1.EncDecAttention.v.weight', 'decoder.block.8.layer.1.EncDecAttention.o.weight', 'decoder.block.7.layer.1.EncDecAttention.o.weight', 'decoder.block.19.layer.2.DenseReluDense.wo.weight', 'decoder.block.1.layer.0.SelfAttention.o.weight', 'decoder.embed_tokens.weight', 'decoder.block.22.layer.0.SelfAttention.v.weight', 'decoder.block.13.layer.0.layer_norm.weight', 'decoder.block.9.layer.1.EncDecAttention.o.weight', 'decoder.block.13.layer.2.DenseReluDense.wi.weight', 'decoder.block.9.layer.2.layer_norm.weight', 'decoder.block.21.layer.1.EncDecAttention.o.weight', 'decoder.block.22.layer.0.SelfAttention.q.weight', 'decoder.block.9.layer.1.EncDecAttention.q.weight', 'decoder.block.16.layer.0.SelfAttention.v.weight', 'decoder.block.15.layer.1.layer_norm.weight', 'decoder.block.5.layer.1.EncDecAttention.o.weight', 'decoder.block.20.layer.0.layer_norm.weight', 'decoder.block.9.layer.1.EncDecAttention.k.weight', 'decoder.block.20.layer.1.EncDecAttention.k.weight', 'decoder.block.22.layer.0.SelfAttention.o.weight', 'decoder.block.15.layer.0.SelfAttention.v.weight', 'decoder.block.15.layer.0.SelfAttention.k.weight', 'decoder.block.4.layer.0.SelfAttention.o.weight', 'decoder.block.10.layer.2.layer_norm.weight', 'decoder.block.20.layer.0.SelfAttention.o.weight', 'decoder.block.3.layer.2.DenseReluDense.wo.weight', 'decoder.block.22.layer.0.layer_norm.weight', 'decoder.block.10.layer.0.SelfAttention.q.weight', 'decoder.block.16.layer.1.EncDecAttention.v.weight', 'decoder.block.23.layer.0.SelfAttention.k.weight', 'decoder.block.14.layer.0.SelfAttention.o.weight', 'decoder.block.9.layer.0.SelfAttention.v.weight', 'decoder.block.23.layer.0.layer_norm.weight', 'decoder.block.15.layer.1.EncDecAttention.v.weight', 'decoder.block.13.layer.1.EncDecAttention.o.weight', 'decoder.block.12.layer.0.layer_norm.weight', 'decoder.block.18.layer.1.EncDecAttention.q.weight', 'decoder.block.7.layer.1.EncDecAttention.v.weight', 'decoder.block.4.layer.1.EncDecAttention.v.weight', 'decoder.block.4.layer.0.SelfAttention.v.weight', 'decoder.block.18.layer.0.SelfAttention.v.weight', 'decoder.block.22.layer.1.EncDecAttention.v.weight', 'decoder.block.6.layer.2.DenseReluDense.wi.weight', 'decoder.block.21.layer.2.layer_norm.weight', 'decoder.block.12.layer.2.layer_norm.weight', 'decoder.block.16.layer.0.SelfAttention.k.weight', 'decoder.block.6.layer.1.layer_norm.weight', 'decoder.block.14.layer.2.DenseReluDense.wo.weight', 'decoder.block.12.layer.0.SelfAttention.k.weight', 'decoder.block.16.layer.2.DenseReluDense.wo.weight', 'decoder.block.17.layer.0.SelfAttention.q.weight', 'decoder.block.10.layer.0.SelfAttention.v.weight', 'decoder.block.13.layer.0.SelfAttention.o.weight', 'decoder.block.16.layer.1.EncDecAttention.q.weight', 'decoder.block.23.layer.1.EncDecAttention.k.weight', 'decoder.block.2.layer.0.SelfAttention.q.weight', 'decoder.block.1.layer.1.layer_norm.weight', 'decoder.block.7.layer.1.EncDecAttention.k.weight', 'decoder.block.16.layer.1.layer_norm.weight', 'decoder.block.12.layer.1.layer_norm.weight', 'decoder.block.14.layer.0.SelfAttention.q.weight', 'decoder.block.1.layer.2.DenseReluDense.wo.weight', 'decoder.block.7.layer.1.layer_norm.weight', 'decoder.block.11.layer.1.EncDecAttention.o.weight', 'decoder.block.12.layer.0.SelfAttention.q.weight', 'decoder.block.8.layer.2.DenseReluDense.wo.weight', 'decoder.block.2.layer.1.EncDecAttention.o.weight', 'decoder.block.21.layer.2.DenseReluDense.wi.weight', 'decoder.block.6.layer.1.EncDecAttention.q.weight', 'decoder.block.14.layer.2.DenseReluDense.wi.weight', 'decoder.block.3.layer.2.DenseReluDense.wi.weight', 'decoder.block.2.layer.0.SelfAttention.o.weight', 'decoder.block.5.layer.1.EncDecAttention.k.weight', 'decoder.block.14.layer.0.SelfAttention.k.weight', 'decoder.block.17.layer.1.EncDecAttention.q.weight', 'decoder.block.1.layer.0.SelfAttention.k.weight', 'decoder.block.11.layer.1.EncDecAttention.q.weight', 'decoder.block.17.layer.0.SelfAttention.k.weight', 'decoder.block.2.layer.0.layer_norm.weight', 'decoder.block.8.layer.1.layer_norm.weight', 'decoder.block.1.layer.1.EncDecAttention.v.weight', 'decoder.block.1.layer.0.SelfAttention.v.weight', 'decoder.block.11.layer.1.EncDecAttention.k.weight', 'decoder.block.18.layer.1.EncDecAttention.o.weight', 'decoder.block.14.layer.1.EncDecAttention.q.weight', 'decoder.block.7.layer.0.SelfAttention.o.weight', 'decoder.block.17.layer.1.layer_norm.weight', 'decoder.block.3.layer.0.SelfAttention.q.weight', 'decoder.block.15.layer.1.EncDecAttention.o.weight', 'decoder.block.10.layer.0.SelfAttention.o.weight', 'decoder.block.16.layer.2.DenseReluDense.wi.weight', 'decoder.block.11.layer.1.layer_norm.weight', 'decoder.block.0.layer.2.layer_norm.weight', 'decoder.block.3.layer.1.EncDecAttention.o.weight', 'decoder.block.0.layer.0.SelfAttention.o.weight', 'decoder.block.4.layer.1.EncDecAttention.k.weight', 'decoder.block.18.layer.2.DenseReluDense.wi.weight', 'decoder.block.4.layer.0.layer_norm.weight', 'decoder.block.15.layer.0.layer_norm.weight', 'decoder.block.0.layer.0.SelfAttention.k.weight', 'decoder.block.23.layer.0.SelfAttention.o.weight', 'decoder.block.4.layer.1.EncDecAttention.q.weight', 'decoder.block.4.layer.0.SelfAttention.k.weight', 'decoder.block.12.layer.1.EncDecAttention.q.weight', 'decoder.block.7.layer.0.SelfAttention.v.weight', 'decoder.block.3.layer.0.layer_norm.weight', 'decoder.block.21.layer.1.EncDecAttention.k.weight', 'decoder.block.2.layer.0.SelfAttention.k.weight', 'decoder.block.10.layer.1.EncDecAttention.k.weight', 'decoder.block.8.layer.0.layer_norm.weight', 'decoder.block.17.layer.1.EncDecAttention.v.weight', 'decoder.block.16.layer.0.SelfAttention.o.weight', 'decoder.block.9.layer.2.DenseReluDense.wi.weight', 'decoder.block.13.layer.2.DenseReluDense.wo.weight', 'decoder.block.19.layer.1.EncDecAttention.v.weight', 'decoder.block.19.layer.0.SelfAttention.q.weight', 'decoder.block.3.layer.0.SelfAttention.k.weight', 'decoder.block.10.layer.1.EncDecAttention.v.weight', 'decoder.block.10.layer.1.layer_norm.weight', 'decoder.block.18.layer.2.DenseReluDense.wo.weight', 'decoder.block.21.layer.1.EncDecAttention.q.weight', 'decoder.block.11.layer.0.SelfAttention.o.weight', 'decoder.block.2.layer.1.EncDecAttention.q.weight', 'decoder.block.2.layer.2.DenseReluDense.wi.weight', 'decoder.block.21.layer.0.SelfAttention.q.weight', 'decoder.block.9.layer.2.DenseReluDense.wo.weight', 'decoder.block.20.layer.2.layer_norm.weight', 'decoder.block.18.layer.1.EncDecAttention.v.weight', 'decoder.block.15.layer.2.layer_norm.weight', 'decoder.block.11.layer.0.layer_norm.weight', 'decoder.block.2.layer.1.EncDecAttention.v.weight', 'decoder.block.3.layer.0.SelfAttention.o.weight', 'decoder.block.19.layer.2.DenseReluDense.wi.weight', 'decoder.block.12.layer.0.SelfAttention.o.weight', 'decoder.block.16.layer.2.layer_norm.weight', 'decoder.block.10.layer.1.EncDecAttention.o.weight', 'decoder.block.4.layer.2.DenseReluDense.wi.weight', 'decoder.block.1.layer.0.layer_norm.weight', 'decoder.block.20.layer.0.SelfAttention.v.weight', 'decoder.block.16.layer.1.EncDecAttention.o.weight', 'decoder.block.2.layer.1.layer_norm.weight', 'decoder.block.4.layer.1.EncDecAttention.o.weight', 'decoder.block.3.layer.1.EncDecAttention.k.weight', 'decoder.block.6.layer.0.layer_norm.weight', 'decoder.block.18.layer.0.SelfAttention.q.weight', 'decoder.block.11.layer.2.DenseReluDense.wo.weight', 'decoder.block.18.layer.0.SelfAttention.k.weight', 'decoder.block.2.layer.1.EncDecAttention.k.weight', 'decoder.block.21.layer.1.EncDecAttention.v.weight', 'decoder.block.20.layer.1.layer_norm.weight', 'decoder.block.15.layer.2.DenseReluDense.wo.weight', 'decoder.block.13.layer.0.SelfAttention.k.weight', 'decoder.block.20.layer.1.EncDecAttention.o.weight', 'decoder.block.0.layer.2.DenseReluDense.wo.weight', 'decoder.block.14.layer.1.EncDecAttention.k.weight', 'decoder.block.16.layer.1.EncDecAttention.k.weight', 'decoder.final_layer_norm.weight', 'decoder.block.21.layer.0.layer_norm.weight', 'decoder.block.6.layer.2.layer_norm.weight', 'decoder.block.13.layer.1.layer_norm.weight', 'decoder.block.19.layer.1.EncDecAttention.k.weight', 'decoder.block.19.layer.0.SelfAttention.v.weight', 'decoder.block.18.layer.0.layer_norm.weight', 'decoder.block.16.layer.0.layer_norm.weight', 'decoder.block.16.layer.0.SelfAttention.q.weight', 'decoder.block.6.layer.2.DenseReluDense.wo.weight', 'decoder.block.5.layer.2.layer_norm.weight', 'decoder.block.12.layer.2.DenseReluDense.wo.weight', 'decoder.block.14.layer.0.SelfAttention.v.weight', 'decoder.block.20.layer.2.DenseReluDense.wi.weight', 'decoder.block.7.layer.2.layer_norm.weight', 'decoder.block.6.layer.0.SelfAttention.o.weight', 'decoder.block.5.layer.0.SelfAttention.q.weight', 'decoder.block.18.layer.1.EncDecAttention.k.weight', 'decoder.block.11.layer.0.SelfAttention.v.weight', 'decoder.block.0.layer.1.EncDecAttention.k.weight', 'decoder.block.2.layer.0.SelfAttention.v.weight', 'decoder.block.18.layer.2.layer_norm.weight', 'decoder.block.17.layer.0.SelfAttention.o.weight', 'decoder.block.10.layer.2.DenseReluDense.wi.weight', 'decoder.block.5.layer.0.layer_norm.weight', 'decoder.block.23.layer.0.SelfAttention.q.weight', 'decoder.block.19.layer.0.layer_norm.weight', 'lm_head.weight', 'decoder.block.0.layer.1.layer_norm.weight', 'decoder.block.6.layer.0.SelfAttention.k.weight', 'decoder.block.1.layer.1.EncDecAttention.o.weight', 'decoder.block.8.layer.0.SelfAttention.v.weight', 'decoder.block.11.layer.1.EncDecAttention.v.weight', 'decoder.block.23.layer.1.EncDecAttention.v.weight', 'decoder.block.5.layer.0.SelfAttention.k.weight', 'decoder.block.11.layer.0.SelfAttention.q.weight', 'decoder.block.8.layer.1.EncDecAttention.k.weight', 'decoder.block.22.layer.2.DenseReluDense.wo.weight', 'decoder.block.12.layer.1.EncDecAttention.k.weight', 'decoder.block.7.layer.2.DenseReluDense.wi.weight', 'decoder.block.9.layer.0.SelfAttention.q.weight', 'decoder.block.9.layer.1.EncDecAttention.v.weight', 'decoder.block.0.layer.0.SelfAttention.q.weight', 'decoder.block.22.layer.0.SelfAttention.k.weight', 'decoder.block.2.layer.2.layer_norm.weight', 'decoder.block.21.layer.0.SelfAttention.k.weight', 'decoder.block.6.layer.1.EncDecAttention.v.weight', 'decoder.block.1.layer.0.SelfAttention.q.weight', 'decoder.block.22.layer.1.EncDecAttention.k.weight', 'decoder.block.5.layer.0.SelfAttention.o.weight', 'decoder.block.0.layer.1.EncDecAttention.v.weight', 'decoder.block.15.layer.1.EncDecAttention.q.weight', 'decoder.block.17.layer.2.DenseReluDense.wo.weight', 'decoder.block.18.layer.0.SelfAttention.o.weight', 'decoder.block.0.layer.1.EncDecAttention.o.weight', 'decoder.block.23.layer.1.EncDecAttention.q.weight', 'decoder.block.9.layer.1.layer_norm.weight', 'decoder.block.8.layer.1.EncDecAttention.q.weight', 'decoder.block.17.layer.1.EncDecAttention.o.weight', 'decoder.block.3.layer.1.EncDecAttention.q.weight', 'decoder.block.5.layer.0.SelfAttention.v.weight', 'decoder.block.13.layer.0.SelfAttention.q.weight', 'decoder.block.13.layer.1.EncDecAttention.k.weight', 'decoder.block.15.layer.0.SelfAttention.o.weight', 'decoder.block.7.layer.0.SelfAttention.k.weight', 'decoder.block.23.layer.2.DenseReluDense.wo.weight', 'decoder.block.20.layer.0.SelfAttention.k.weight', 'decoder.block.23.layer.0.SelfAttention.v.weight', 'decoder.block.7.layer.1.EncDecAttention.q.weight', 'decoder.block.5.layer.1.EncDecAttention.v.weight', 'decoder.block.18.layer.1.layer_norm.weight', 'decoder.block.6.layer.1.EncDecAttention.o.weight', 'decoder.block.1.layer.2.DenseReluDense.wi.weight', 'decoder.block.7.layer.0.SelfAttention.q.weight', 'decoder.block.8.layer.0.SelfAttention.k.weight', 'decoder.block.3.layer.0.SelfAttention.v.weight', 'decoder.block.22.layer.2.layer_norm.weight', 'decoder.block.12.layer.1.EncDecAttention.v.weight', 'decoder.block.21.layer.2.DenseReluDense.wo.weight', 'decoder.block.6.layer.0.SelfAttention.v.weight', 'decoder.block.8.layer.0.SelfAttention.q.weight', 'decoder.block.0.layer.2.DenseReluDense.wi.weight', 'decoder.block.13.layer.0.SelfAttention.v.weight', 'decoder.block.0.layer.0.layer_norm.weight', 'decoder.block.0.layer.0.SelfAttention.relative_attention_bias.weight', 'decoder.block.8.layer.0.SelfAttention.o.weight', 'decoder.block.23.layer.1.layer_norm.weight', 'decoder.block.4.layer.1.layer_norm.weight', 'decoder.block.19.layer.1.layer_norm.weight', 'decoder.block.10.layer.1.EncDecAttention.q.weight', 'decoder.block.22.layer.1.layer_norm.weight', 'decoder.block.4.layer.2.DenseReluDense.wo.weight', 'decoder.block.0.layer.0.SelfAttention.v.weight', 'decoder.block.23.layer.2.DenseReluDense.wi.weight', 'decoder.block.14.layer.1.EncDecAttention.o.weight', 'decoder.block.14.layer.1.EncDecAttention.v.weight', 'decoder.block.1.layer.1.EncDecAttention.k.weight']\n",
      "- This IS expected if you are initializing T5EncoderModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing T5EncoderModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ProtT5_Classfier\n",
      "Trainable Parameter: 1209192449\n",
      "ProtT5_LoRA_Classfier\n",
      "Trainable Parameter: 3558401\n",
      "\n",
      "[2023-07-11 13:05:17,306] [INFO] [comm.py:657:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl\n",
      "[2023-07-11 13:05:17,509] [INFO] [logging.py:75:log_dist] [Rank 0] DeepSpeed info: version=0.8.1, git-hash=unknown, git-branch=unknown\n",
      "[2023-07-11 13:05:19,200] [INFO] [logging.py:75:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using /homes/user/.cache/torch_extensions/py39_cu117 as PyTorch extensions root...\n",
      "Detected CUDA files, patching ldflags\n",
      "Emitting ninja build file /homes/user/.cache/torch_extensions/py39_cu117/cpu_adam/build.ninja...\n",
      "Building extension module cpu_adam...\n",
      "Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)\n",
      "Loading extension module cpu_adam...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ninja: no work to do.\n",
      "Time to load cpu_adam op: 2.7784407138824463 seconds\n",
      "[2023-07-11 13:05:24,835] [INFO] [logging.py:75:log_dist] [Rank 0] Using DeepSpeed Optimizer param name adamw as basic optimizer\n",
      "[2023-07-11 13:05:24,874] [INFO] [logging.py:75:log_dist] [Rank 0] DeepSpeed Basic Optimizer = DeepSpeedCPUAdam\n",
      "[2023-07-11 13:05:24,874] [INFO] [utils.py:53:is_zero_supported_optimizer] Checking ZeRO support for optimizer=DeepSpeedCPUAdam type=<class 'deepspeed.ops.adam.cpu_adam.DeepSpeedCPUAdam'>\n",
      "[2023-07-11 13:05:24,875] [INFO] [logging.py:75:log_dist] [Rank 0] Creating torch.float32 ZeRO stage 2 optimizer\n",
      "[2023-07-11 13:05:24,875] [INFO] [stage_1_and_2.py:144:__init__] Reduce bucket size 200000000\n",
      "[2023-07-11 13:05:24,876] [INFO] [stage_1_and_2.py:145:__init__] Allgather bucket size 200000000\n",
      "[2023-07-11 13:05:24,876] [INFO] [stage_1_and_2.py:146:__init__] CPU Offload: True\n",
      "[2023-07-11 13:05:24,876] [INFO] [stage_1_and_2.py:147:__init__] Round robin gradient partitioning: False\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using /homes/user/.cache/torch_extensions/py39_cu117 as PyTorch extensions root...\n",
      "Emitting ninja build file /homes/user/.cache/torch_extensions/py39_cu117/utils/build.ninja...\n",
      "Building extension module utils...\n",
      "Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)\n",
      "Loading extension module utils...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ninja: no work to do.\n",
      "Time to load utils op: 0.3791632652282715 seconds\n",
      "Rank: 0 partition count [1] and sizes[(3558402, False)] \n",
      "[2023-07-11 13:05:25,513] [INFO] [utils.py:825:see_memory_usage] Before initializing optimizer states\n",
      "[2023-07-11 13:05:25,514] [INFO] [utils.py:826:see_memory_usage] MA 4.52 GB         Max_MA 4.52 GB         CA 4.54 GB         Max_CA 5 GB \n",
      "[2023-07-11 13:05:25,515] [INFO] [utils.py:834:see_memory_usage] CPU Virtual Memory:  used = 38.29 GB, percent = 20.5%\n",
      "[2023-07-11 13:05:25,671] [INFO] [utils.py:825:see_memory_usage] After initializing optimizer states\n",
      "[2023-07-11 13:05:25,672] [INFO] [utils.py:826:see_memory_usage] MA 4.52 GB         Max_MA 4.52 GB         CA 4.54 GB         Max_CA 5 GB \n",
      "[2023-07-11 13:05:25,673] [INFO] [utils.py:834:see_memory_usage] CPU Virtual Memory:  used = 38.32 GB, percent = 20.5%\n",
      "[2023-07-11 13:05:25,674] [INFO] [stage_1_and_2.py:527:__init__] optimizer state initialized\n",
      "[2023-07-11 13:05:25,822] [INFO] [utils.py:825:see_memory_usage] After initializing ZeRO optimizer\n",
      "[2023-07-11 13:05:25,823] [INFO] [utils.py:826:see_memory_usage] MA 4.52 GB         Max_MA 4.52 GB         CA 4.54 GB         Max_CA 5 GB \n",
      "[2023-07-11 13:05:25,824] [INFO] [utils.py:834:see_memory_usage] CPU Virtual Memory:  used = 38.32 GB, percent = 20.5%\n",
      "[2023-07-11 13:05:25,834] [INFO] [logging.py:75:log_dist] [Rank 0] DeepSpeed Final Optimizer = adamw\n",
      "[2023-07-11 13:05:25,835] [INFO] [logging.py:75:log_dist] [Rank 0] DeepSpeed using configured LR scheduler = WarmupLR\n",
      "[2023-07-11 13:05:25,835] [INFO] [logging.py:75:log_dist] [Rank 0] DeepSpeed LR Scheduler = <deepspeed.runtime.lr_schedules.WarmupLR object at 0x7efe3e0f8880>\n",
      "[2023-07-11 13:05:25,835] [INFO] [logging.py:75:log_dist] [Rank 0] step=0, skipped=0, lr=[0.0003], mom=[[0.9, 0.999]]\n",
      "[2023-07-11 13:05:25,837] [INFO] [config.py:1009:print] DeepSpeedEngine configuration:\n",
      "[2023-07-11 13:05:25,837] [INFO] [config.py:1013:print]   activation_checkpointing_config  {\n",
      "    \"partition_activations\": false, \n",
      "    \"contiguous_memory_optimization\": false, \n",
      "    \"cpu_checkpointing\": false, \n",
      "    \"number_checkpoints\": null, \n",
      "    \"synchronize_checkpoint_boundary\": false, \n",
      "    \"profile\": false\n",
      "}\n",
      "[2023-07-11 13:05:25,838] [INFO] [config.py:1013:print]   aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True}\n",
      "[2023-07-11 13:05:25,838] [INFO] [config.py:1013:print]   amp_enabled .................. False\n",
      "[2023-07-11 13:05:25,839] [INFO] [config.py:1013:print]   amp_params ................... False\n",
      "[2023-07-11 13:05:25,839] [INFO] [config.py:1013:print]   autotuning_config ............ {\n",
      "    \"enabled\": false, \n",
      "    \"start_step\": null, \n",
      "    \"end_step\": null, \n",
      "    \"metric_path\": null, \n",
      "    \"arg_mappings\": null, \n",
      "    \"metric\": \"throughput\", \n",
      "    \"model_info\": null, \n",
      "    \"results_dir\": \"autotuning_results\", \n",
      "    \"exps_dir\": \"autotuning_exps\", \n",
      "    \"overwrite\": true, \n",
      "    \"fast\": true, \n",
      "    \"start_profile_step\": 3, \n",
      "    \"end_profile_step\": 5, \n",
      "    \"tuner_type\": \"gridsearch\", \n",
      "    \"tuner_early_stopping\": 5, \n",
      "    \"tuner_num_trials\": 50, \n",
      "    \"model_info_path\": null, \n",
      "    \"mp_size\": 1, \n",
      "    \"max_train_batch_size\": null, \n",
      "    \"min_train_batch_size\": 1, \n",
      "    \"max_train_micro_batch_size_per_gpu\": 1.024000e+03, \n",
      "    \"min_train_micro_batch_size_per_gpu\": 1, \n",
      "    \"num_tuning_micro_batch_sizes\": 3\n",
      "}\n",
      "[2023-07-11 13:05:25,840] [INFO] [config.py:1013:print]   bfloat16_enabled ............. False\n",
      "[2023-07-11 13:05:25,840] [INFO] [config.py:1013:print]   checkpoint_parallel_write_pipeline  False\n",
      "[2023-07-11 13:05:25,840] [INFO] [config.py:1013:print]   checkpoint_tag_validation_enabled  True\n",
      "[2023-07-11 13:05:25,841] [INFO] [config.py:1013:print]   checkpoint_tag_validation_fail  False\n",
      "[2023-07-11 13:05:25,841] [INFO] [config.py:1013:print]   comms_config ................. <deepspeed.comm.config.DeepSpeedCommsConfig object at 0x7efe3e0d3b20>\n",
      "[2023-07-11 13:05:25,841] [INFO] [config.py:1013:print]   communication_data_type ...... None\n",
      "[2023-07-11 13:05:25,842] [INFO] [config.py:1013:print]   compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}}\n",
      "[2023-07-11 13:05:25,842] [INFO] [config.py:1013:print]   curriculum_enabled_legacy .... False\n",
      "[2023-07-11 13:05:25,843] [INFO] [config.py:1013:print]   curriculum_params_legacy ..... False\n",
      "[2023-07-11 13:05:25,843] [INFO] [config.py:1013:print]   data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'curriculum_learning': {'enabled': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}}\n",
      "[2023-07-11 13:05:25,843] [INFO] [config.py:1013:print]   data_efficiency_enabled ...... False\n",
      "[2023-07-11 13:05:25,844] [INFO] [config.py:1013:print]   dataloader_drop_last ......... False\n",
      "[2023-07-11 13:05:25,844] [INFO] [config.py:1013:print]   disable_allgather ............ False\n",
      "[2023-07-11 13:05:25,844] [INFO] [config.py:1013:print]   dump_state ................... False\n",
      "[2023-07-11 13:05:25,845] [INFO] [config.py:1013:print]   dynamic_loss_scale_args ...... None\n",
      "[2023-07-11 13:05:25,845] [INFO] [config.py:1013:print]   eigenvalue_enabled ........... False\n",
      "[2023-07-11 13:05:25,845] [INFO] [config.py:1013:print]   eigenvalue_gas_boundary_resolution  1\n",
      "[2023-07-11 13:05:25,846] [INFO] [config.py:1013:print]   eigenvalue_layer_name ........ bert.encoder.layer\n",
      "[2023-07-11 13:05:25,846] [INFO] [config.py:1013:print]   eigenvalue_layer_num ......... 0\n",
      "[2023-07-11 13:05:25,847] [INFO] [config.py:1013:print]   eigenvalue_max_iter .......... 100\n",
      "[2023-07-11 13:05:25,847] [INFO] [config.py:1013:print]   eigenvalue_stability ......... 1e-06\n",
      "[2023-07-11 13:05:25,847] [INFO] [config.py:1013:print]   eigenvalue_tol ............... 0.01\n",
      "[2023-07-11 13:05:25,848] [INFO] [config.py:1013:print]   eigenvalue_verbose ........... False\n",
      "[2023-07-11 13:05:25,848] [INFO] [config.py:1013:print]   elasticity_enabled ........... False\n",
      "[2023-07-11 13:05:25,849] [INFO] [config.py:1013:print]   flops_profiler_config ........ {\n",
      "    \"enabled\": false, \n",
      "    \"profile_step\": 1, \n",
      "    \"module_depth\": -1, \n",
      "    \"top_modules\": 1, \n",
      "    \"detailed\": true, \n",
      "    \"output_file\": null\n",
      "}\n",
      "[2023-07-11 13:05:25,849] [INFO] [config.py:1013:print]   fp16_auto_cast ............... None\n",
      "[2023-07-11 13:05:25,849] [INFO] [config.py:1013:print]   fp16_enabled ................. False\n",
      "[2023-07-11 13:05:25,850] [INFO] [config.py:1013:print]   fp16_master_weights_and_gradients  False\n",
      "[2023-07-11 13:05:25,850] [INFO] [config.py:1013:print]   global_rank .................. 0\n",
      "[2023-07-11 13:05:25,851] [INFO] [config.py:1013:print]   grad_accum_dtype ............. None\n",
      "[2023-07-11 13:05:25,851] [INFO] [config.py:1013:print]   gradient_accumulation_steps .. 8\n",
      "[2023-07-11 13:05:25,851] [INFO] [config.py:1013:print]   gradient_clipping ............ 1.0\n",
      "[2023-07-11 13:05:25,852] [INFO] [config.py:1013:print]   gradient_predivide_factor .... 1.0\n",
      "[2023-07-11 13:05:25,852] [INFO] [config.py:1013:print]   initial_dynamic_scale ........ 65536\n",
      "[2023-07-11 13:05:25,852] [INFO] [config.py:1013:print]   load_universal_checkpoint .... False\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2023-07-11 13:05:25,853] [INFO] [config.py:1013:print]   loss_scale ................... 0\n",
      "[2023-07-11 13:05:25,853] [INFO] [config.py:1013:print]   memory_breakdown ............. False\n",
      "[2023-07-11 13:05:25,853] [INFO] [config.py:1013:print]   monitor_config ............... tensorboard=TensorBoardConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') enabled=False\n",
      "[2023-07-11 13:05:25,854] [INFO] [config.py:1013:print]   nebula_config ................ {\n",
      "    \"enabled\": false, \n",
      "    \"persistent_storage_path\": null, \n",
      "    \"persistent_time_interval\": 100, \n",
      "    \"num_of_version_in_retention\": 2, \n",
      "    \"enable_nebula_load\": true, \n",
      "    \"load_path\": null\n",
      "}\n",
      "[2023-07-11 13:05:25,854] [INFO] [config.py:1013:print]   optimizer_legacy_fusion ...... False\n",
      "[2023-07-11 13:05:25,855] [INFO] [config.py:1013:print]   optimizer_name ............... adamw\n",
      "[2023-07-11 13:05:25,855] [INFO] [config.py:1013:print]   optimizer_params ............. {'lr': 0.0003, 'betas': [0.9, 0.999], 'eps': 1e-08, 'weight_decay': 0.0}\n",
      "[2023-07-11 13:05:25,855] [INFO] [config.py:1013:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0}\n",
      "[2023-07-11 13:05:25,856] [INFO] [config.py:1013:print]   pld_enabled .................. False\n",
      "[2023-07-11 13:05:25,856] [INFO] [config.py:1013:print]   pld_params ................... False\n",
      "[2023-07-11 13:05:25,856] [INFO] [config.py:1013:print]   prescale_gradients ........... False\n",
      "[2023-07-11 13:05:25,857] [INFO] [config.py:1013:print]   scheduler_name ............... WarmupLR\n",
      "[2023-07-11 13:05:25,857] [INFO] [config.py:1013:print]   scheduler_params ............. {'warmup_min_lr': 0, 'warmup_max_lr': 0.0003, 'warmup_num_steps': 0}\n",
      "[2023-07-11 13:05:25,858] [INFO] [config.py:1013:print]   sparse_attention ............. None\n",
      "[2023-07-11 13:05:25,858] [INFO] [config.py:1013:print]   sparse_gradients_enabled ..... False\n",
      "[2023-07-11 13:05:25,858] [INFO] [config.py:1013:print]   steps_per_print .............. 2000\n",
      "[2023-07-11 13:05:25,859] [INFO] [config.py:1013:print]   train_batch_size ............. 8\n",
      "[2023-07-11 13:05:25,859] [INFO] [config.py:1013:print]   train_micro_batch_size_per_gpu  1\n",
      "[2023-07-11 13:05:25,859] [INFO] [config.py:1013:print]   use_node_local_storage ....... False\n",
      "[2023-07-11 13:05:25,860] [INFO] [config.py:1013:print]   wall_clock_breakdown ......... False\n",
      "[2023-07-11 13:05:25,860] [INFO] [config.py:1013:print]   world_size ................... 1\n",
      "[2023-07-11 13:05:25,860] [INFO] [config.py:1013:print]   zero_allow_untested_optimizer  False\n",
      "[2023-07-11 13:05:25,861] [INFO] [config.py:1013:print]   zero_config .................. stage=2 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=200000000 allgather_partitions=True allgather_bucket_size=200000000 overlap_comm=True load_from_fp32_weights=True elastic_checkpoint=False offload_param=None offload_optimizer=DeepSpeedZeroOffloadOptimizerConfig(device='cpu', nvme_path=None, buffer_count=4, pin_memory=True, pipeline=False, pipeline_read=False, pipeline_write=False, fast_init=False) sub_group_size=1,000,000,000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=50,000,000 param_persistence_threshold=100,000 model_persistence_threshold=sys.maxsize max_live_parameters=1,000,000,000 max_reuse_distance=1,000,000,000 gather_16bit_weights_on_model_save=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False\n",
      "[2023-07-11 13:05:25,861] [INFO] [config.py:1013:print]   zero_enabled ................. True\n",
      "[2023-07-11 13:05:25,862] [INFO] [config.py:1013:print]   zero_optimization_stage ...... 2\n",
      "[2023-07-11 13:05:25,863] [INFO] [config.py:998:print_user_config]   json = {\n",
      "    \"fp16\": {\n",
      "        \"enabled\": false, \n",
      "        \"loss_scale\": 0, \n",
      "        \"loss_scale_window\": 1000, \n",
      "        \"initial_scale_power\": 16, \n",
      "        \"hysteresis\": 2, \n",
      "        \"min_loss_scale\": 1\n",
      "    }, \n",
      "    \"optimizer\": {\n",
      "        \"type\": \"AdamW\", \n",
      "        \"params\": {\n",
      "            \"lr\": 0.0003, \n",
      "            \"betas\": [0.9, 0.999], \n",
      "            \"eps\": 1e-08, \n",
      "            \"weight_decay\": 0.0\n",
      "        }\n",
      "    }, \n",
      "    \"scheduler\": {\n",
      "        \"type\": \"WarmupLR\", \n",
      "        \"params\": {\n",
      "            \"warmup_min_lr\": 0, \n",
      "            \"warmup_max_lr\": 0.0003, \n",
      "            \"warmup_num_steps\": 0\n",
      "        }\n",
      "    }, \n",
      "    \"zero_optimization\": {\n",
      "        \"stage\": 2, \n",
      "        \"offload_optimizer\": {\n",
      "            \"device\": \"cpu\", \n",
      "            \"pin_memory\": true\n",
      "        }, \n",
      "        \"allgather_partitions\": true, \n",
      "        \"allgather_bucket_size\": 2.000000e+08, \n",
      "        \"overlap_comm\": true, \n",
      "        \"reduce_scatter\": true, \n",
      "        \"reduce_bucket_size\": 2.000000e+08, \n",
      "        \"contiguous_gradients\": true\n",
      "    }, \n",
      "    \"gradient_accumulation_steps\": 8, \n",
      "    \"gradient_clipping\": 1.0, \n",
      "    \"steps_per_print\": 2.000000e+03, \n",
      "    \"train_batch_size\": 8, \n",
      "    \"train_micro_batch_size_per_gpu\": 1, \n",
      "    \"wall_clock_breakdown\": false\n",
      "}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using /homes/user/.cache/torch_extensions/py39_cu117 as PyTorch extensions root...\n",
      "No modifications detected for re-loaded extension module utils, skipping build step...\n",
      "Loading extension module utils...\n",
      "***** Running training *****\n",
      "  Num examples = 2691\n",
      "  Num Epochs = 20\n",
      "  Instantaneous batch size per device = 1\n",
      "  Total train batch size (w. parallel, distributed & accumulation) = 8\n",
      "  Gradient Accumulation steps = 8\n",
      "  Total optimization steps = 6720\n",
      "  Number of trainable parameters = 3558401\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time to load utils op: 0.0012099742889404297 seconds\n",
      "Adam Optimizer #0 is created with AVX2 arithmetic capability.\n",
      "Config: alpha=0.000300, betas=(0.900000, 0.999000), weight_decay=0.000000, adam_w=1\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='6720' max='6720' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [6720/6720 3:42:32, Epoch 19/20]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Spearmanr</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1.220000</td>\n",
       "      <td>1.029735</td>\n",
       "      <td>0.456535</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.921400</td>\n",
       "      <td>0.841677</td>\n",
       "      <td>0.666679</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.774900</td>\n",
       "      <td>0.795428</td>\n",
       "      <td>0.730048</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.669700</td>\n",
       "      <td>0.594561</td>\n",
       "      <td>0.765898</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0.598500</td>\n",
       "      <td>0.527124</td>\n",
       "      <td>0.799539</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>0.530000</td>\n",
       "      <td>0.412178</td>\n",
       "      <td>0.810243</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>0.473300</td>\n",
       "      <td>0.500522</td>\n",
       "      <td>0.825762</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>0.437800</td>\n",
       "      <td>0.322964</td>\n",
       "      <td>0.854570</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>0.365200</td>\n",
       "      <td>0.676764</td>\n",
       "      <td>0.855964</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>0.349100</td>\n",
       "      <td>0.359426</td>\n",
       "      <td>0.859090</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>0.339300</td>\n",
       "      <td>0.404395</td>\n",
       "      <td>0.860436</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>0.283000</td>\n",
       "      <td>0.462267</td>\n",
       "      <td>0.882389</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>0.278600</td>\n",
       "      <td>0.273607</td>\n",
       "      <td>0.875810</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>0.255400</td>\n",
       "      <td>0.518614</td>\n",
       "      <td>0.867230</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>0.241900</td>\n",
       "      <td>0.390732</td>\n",
       "      <td>0.887881</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>15</td>\n",
       "      <td>0.233200</td>\n",
       "      <td>0.626555</td>\n",
       "      <td>0.877354</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>16</td>\n",
       "      <td>0.213300</td>\n",
       "      <td>0.231160</td>\n",
       "      <td>0.883552</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>17</td>\n",
       "      <td>0.218000</td>\n",
       "      <td>0.309015</td>\n",
       "      <td>0.871208</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>18</td>\n",
       "      <td>0.207800</td>\n",
       "      <td>0.581621</td>\n",
       "      <td>0.889005</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>19</td>\n",
       "      <td>0.196100</td>\n",
       "      <td>0.413738</td>\n",
       "      <td>0.890522</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running Evaluation *****\n",
      "  Num examples = 299\n",
      "  Batch size = 16\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 299\n",
      "  Batch size = 16\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 299\n",
      "  Batch size = 16\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 299\n",
      "  Batch size = 16\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 299\n",
      "  Batch size = 16\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2023-07-11 14:11:19,953] [INFO] [logging.py:75:log_dist] [Rank 0] step=2000, skipped=0, lr=[0.0003], mom=[[0.9, 0.999]]\n",
      "[2023-07-11 14:11:19,954] [INFO] [timer.py:198:stop] epoch=0/micro_step=16000/global_step=2000, RunningAvgSamplesPerSec=4.165304479449947, CurrSamplesPerSec=4.16520225251198, MemAllocated=4.52GB, MaxMemAllocated=9.11GB\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running Evaluation *****\n",
      "  Num examples = 299\n",
      "  Batch size = 16\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 299\n",
      "  Batch size = 16\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 299\n",
      "  Batch size = 16\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 299\n",
      "  Batch size = 16\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 299\n",
      "  Batch size = 16\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 299\n",
      "  Batch size = 16\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2023-07-11 15:17:27,969] [INFO] [logging.py:75:log_dist] [Rank 0] step=4000, skipped=0, lr=[0.0003], mom=[[0.9, 0.999]]\n",
      "[2023-07-11 15:17:27,970] [INFO] [timer.py:198:stop] epoch=0/micro_step=32000/global_step=4000, RunningAvgSamplesPerSec=4.165851823242893, CurrSamplesPerSec=4.16168419903742, MemAllocated=4.52GB, MaxMemAllocated=9.11GB\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running Evaluation *****\n",
      "  Num examples = 299\n",
      "  Batch size = 16\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 299\n",
      "  Batch size = 16\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 299\n",
      "  Batch size = 16\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 299\n",
      "  Batch size = 16\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 299\n",
      "  Batch size = 16\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 299\n",
      "  Batch size = 16\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2023-07-11 16:23:42,930] [INFO] [logging.py:75:log_dist] [Rank 0] step=6000, skipped=0, lr=[0.0003], mom=[[0.9, 0.999]]\n",
      "[2023-07-11 16:23:42,932] [INFO] [timer.py:198:stop] epoch=0/micro_step=48000/global_step=6000, RunningAvgSamplesPerSec=4.1635646382033755, CurrSamplesPerSec=4.158338530053567, MemAllocated=4.52GB, MaxMemAllocated=9.11GB\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "***** Running Evaluation *****\n",
      "  Num examples = 299\n",
      "  Batch size = 16\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 299\n",
      "  Batch size = 16\n",
      "***** Running Evaluation *****\n",
      "  Num examples = 299\n",
      "  Batch size = 16\n",
      "\n",
      "\n",
      "Training completed. Do not forget to share your model on huggingface.co/models =)\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tokenizer, model, history = train_per_protein(my_train, my_valid, num_labels=1, batch=1, accum=8, epochs=20, seed=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54bab485",
   "metadata": {},
   "source": [
    "## Plot results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "f8465267",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA4EAAAHWCAYAAADJvoyqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/P9b71AAAACXBIWXMAAA9hAAAPYQGoP6dpAADKdklEQVR4nOzdZ3RUVReH8WcmHVKoCQFCr6EGkKo0qSoKNkQBQUVFVBR7QazwKqIoiiiKKAJiA0WQKlWalNB7DS3UQAgkIcm8Hw5JQEJJMpObSf6/tWbNzc2dc3eos+ecs7fN4XA4EBERERERkXzBbnUAIiIiIiIiknOUBIqIiIiIiOQjSgJFRERERETyESWBIiIiIiIi+YiSQBERERERkXxESaCIiIiIiEg+oiRQREREREQkH1ESKCIiIiIiko8oCRQREREREclHlASKiIhL9erVi3LlymXptW+++SY2m825AbnInj17sNlsjB071upQRERErkpJoIhIPmWz2a7rMX/+fKtDtUSvXr3w9/e/4vdtNhtPPvlktu8zcuRIJY4iIpKjPK0OQERErDFu3LhLvv7++++ZPXv2ZeerV6+erfuMHj2alJSULL329ddf5+WXX87W/XNK2bJlOXfuHF5eXpl63ciRIylWrBi9evVyTWAiIiL/oSRQRCSf6t69+yVfL1u2jNmzZ192/r/Onj1LgQIFrvs+mU2KLubp6Ymnp3v8V2Wz2fD19bU6DADi4+Px9vbGbteCHxERuZz+dxARkStq2bIlNWvWZNWqVTRv3pwCBQrw6quvAvD7779z6623UrJkSXx8fKhYsSLvvPMOycnJl4zx3z2BqXvnPvzwQ7766isqVqyIj48PN9xwA//+++8lr81oT2DqMswpU6ZQs2ZNfHx8qFGjBjNmzLgs/vnz59OgQQN8fX2pWLEiX375pcv2GWa0J/Dw4cP07t2b0qVL4+PjQ2hoKHfccQd79uwBoFy5cmzcuJEFCxakLb9t2bJl2ut37drFPffcQ5EiRShQoACNGzdm2rRpl/2MNpuNH3/8kddff51SpUpRoEABIiMjsdlsfPzxx5fFumTJEmw2GxMnTnT6r4OIiOR+7vHxqoiIWOb48eN07NiR++67j+7duxMSEgLA2LFj8ff3Z8CAAfj7+/P333/zxhtvcPr0aYYOHXrNcSdMmEBsbCyPPfYYNpuNDz74gDvvvJNdu3Zdc/Zw8eLF/PbbbzzxxBMEBATw6aefctddd7Fv3z6KFi0KwJo1a+jQoQOhoaG89dZbJCcn8/bbb1O8ePFM/fzHjh3L1PUXu+uuu9i4cSNPPfUU5cqV48iRI8yePZt9+/ZRrlw5hg8fzlNPPYW/vz+vvfYaQNqvb3R0NE2bNuXs2bM8/fTTFC1alO+++47bb7+dX375hS5dulxyr3feeQdvb2+ef/55EhISqFatGs2aNWP8+PE8++yzl1w7fvx4AgICuOOOO7L8s4mIiBtziIiIOByOfv36Of7730KLFi0cgGPUqFGXXX/27NnLzj322GOOAgUKOOLj49POPfjgg46yZcumfb17924H4ChatKjjxIkTaed///13B+CYOnVq2rlBgwZdFhPg8Pb2duzYsSPt3Nq1ax2AY8SIEWnnOnXq5ChQoIDjwIEDaee2b9/u8PT0vGzMjDz44IMO4KqPfv36XfZzffvttw6Hw+E4efKkA3AMHTr0qvepUaOGo0WLFpedf+aZZxyAY9GiRWnnYmNjHeXLl3eUK1fOkZyc7HA4HI558+Y5AEeFChUu+z358ssvHYBj8+bNaecSExMdxYoVczz44IPX/DUQEZG8SctBRUTkqnx8fOjdu/dl5/38/NKOY2NjOXbsGDfddBNnz55ly5Yt1xy3a9euFC5cOO3rm266CTBLIK+lTZs2VKxYMe3r2rVrExgYmPba5ORk5syZQ+fOnSlZsmTadZUqVaJjx47XHD+Vr68vs2fPzvBxLX5+fnh7ezN//nxOnjx53fdMNX36dBo2bMiNN96Yds7f359HH32UPXv2sGnTpkuuf/DBBy/5PQG499578fX1Zfz48WnnZs6cybFjx66591NERPIuLQcVEZGrKlWqFN7e3ped37hxI6+//jp///03p0+fvuR7p06duua4ZcqUueTr1ITwehKm/7429fWprz1y5Ajnzp2jUqVKl12X0bkr8fDwoE2bNtd9/cV8fHx4//33ee655wgJCaFx48bcdttt9OzZkxIlSlzz9Xv37qVRo0aXnU+t1rp3715q1qyZdr58+fKXXVuoUCE6derEhAkTeOeddwCzFLRUqVK0bt06Sz+XiIi4P80EiojIVf13dgkgJiaGFi1asHbtWt5++22mTp3K7Nmzef/99wGuqyWEh4dHhucdDodLX5uTnnnmGbZt28aQIUPw9fVl4MCBVK9enTVr1jj9Xhn9PgH07NmTXbt2sWTJEmJjY/njjz/o1q2bKoeKiORjmgkUEZFMmz9/PsePH+e3336jefPmaed3795tYVTpgoOD8fX1ZceOHZd9L6NzrlSxYkWee+45nnvuObZv307dunUZNmwYP/zwA8AVK5WWLVuWrVu3XnY+dalt2bJlr+v+HTp0oHjx4owfP55GjRpx9uxZevTokcWfRkRE8gJ9DCgiIpmWOhN38cxbYmIiI0eOtCqkS6Qu45wyZQoHDx5MO79jxw7++uuvHInh7NmzxMfHX3KuYsWKBAQEkJCQkHauYMGCxMTEXPb6W265hRUrVrB06dK0c3FxcXz11VeUK1eO8PDw64rD09OTbt268dNPPzF27Fhq1apF7dq1s/ZDiYhInqCZQBERybSmTZtSuHBhHnzwQZ5++mlsNhvjxo3LVcsx33zzTWbNmkWzZs3o27cvycnJfPbZZ9SsWZPIyEiX33/btm3cfPPN3HvvvYSHh+Pp6cnkyZOJjo7mvvvuS7uufv36fPHFF7z77rtUqlSJ4OBgWrduzcsvv8zEiRPp2LEjTz/9NEWKFOG7775j9+7d/Prrr5laztmzZ08+/fRT5s2bl7ZkV0RE8i8lgSIikmlFixblzz//5LnnnuP111+ncOHCdO/enZtvvpn27dtbHR5gkqu//vqL559/noEDBxIWFsbbb7/N5s2br6t6aXaFhYXRrVs35s6dy7hx4/D09KRatWr89NNP3HXXXWnXvfHGG+zdu5cPPviA2NhYWrRoQevWrQkJCWHJkiW89NJLjBgxgvj4eGrXrs3UqVO59dZbMxVL/fr1qVGjBps3b+aBBx5w9o8qIiJuxubITR/bioiIuFjnzp3ZuHEj27dvtzqUHBUREUGRIkWYO3eu1aGIiIjFtCdQRETyrHPnzl3y9fbt25k+fTotW7a0JiCLrFy5ksjISHr27Gl1KCIikgtoJlBERPKs0NBQevXqRYUKFdi7dy9ffPEFCQkJrFmzhsqVK1sdnstt2LCBVatWMWzYMI4dO8auXbvw9fW1OiwREbGY9gSKiEie1aFDByZOnMjhw4fx8fGhSZMmDB48OF8kgAC//PILb7/9NlWrVmXixIlKAEVEBNBMoIiIiIiIyHVbuHAhQ4cOZdWqVRw6dIjJkyfTuXPnq75m/vz5DBgwgI0bNxIWFsbrr79Or169ciTejGhPoIiIiIiIyHWKi4ujTp06fP7559d1/e7du7n11ltp1aoVkZGRPPPMMzzyyCPMnDnTxZFemWYCRUREREREssBms11zJvCll15i2rRpbNiwIe3cfffdR0xMDDNmzMiBKC+X7/YEJiUlsWbNGkJCQjLVaFdERERERPKWlJQU9u3bR3h4OJ6e6amRj48PPj4+TrnH0qVLadOmzSXn2rdvzzPPPOOU8bMi3yWBa9asoWHDhlaHISIiIiIiudSgQYN48803nTLW4cOHCQkJueRcSEgIp0+f5ty5c/j5+TnlPpmR75LA1N+AFStWEBoaanE0IiIiIiJilUOHDtGwYUM2bNhAWFhY2nlnzQLmVvkuCUxdAhoaGkrp0qUtjkZERERERKwWFBREYGCgS8YuUaIE0dHRl5yLjo4mMDDQkllAUHVQERERERERl2nSpAlz58695Nzs2bNp0qSJRREpCRQREREREbluZ86cITIyksjISMC0gIiMjGTfvn0AvPLKK/Ts2TPt+scff5xdu3bx4osvsmXLFkaOHMlPP/3Es88+a0X4gJJAERERERGR67Zy5UoiIiKIiIgAYMCAAURERPDGG28AZp9hakIIUL58eaZNm8bs2bOpU6cOw4YN4+uvv6Z9+/aWxA/5sE/g/v37CQsLIyoqSnsCRURERETysfyaG2gmUEREREREJB9REigiIiIiIpKPKAkUERERERHJR5QEioiIiIiI5CNKAkVERERERPIRJYEiIiIiIiL5iJJAERERERGRfERJoIiIiIiISD6iJNBiKSkOkpJTrA5DRERERETyCSWBFhqzeDc3vv8309YfsjoUERERERHJJ5QEWijm3HkOnornt9UHrA5FRERERETyCSWBFuoSUQqARduPciQ23uJoREREREQkP1ASaKHyxQoSUaYQKQ74I/Kg1eGIiIiIiEg+YGkSuHDhQjp16kTJkiWx2WxMmTLlqtf/9ttvtG3bluLFixMYGEiTJk2YOXNmzgTrIndemA2cvEZLQkVERERExPUsTQLj4uKoU6cOn3/++XVdv3DhQtq2bcv06dNZtWoVrVq1olOnTqxZs8bFkbrObbVL4uVhY+PB02yLjrU6HBERERERyeM8rbx5x44d6dix43VfP3z48Eu+Hjx4ML///jtTp04lIiLCydHljMIFvWlZNZjZm6L5bfUBXu5YzeqQREREREQkD3PrPYEpKSnExsZSpEiRK16TkJDA6dOn0x6xsblvti11SejvkQdISXFYHI2IiIiIiORlbp0Efvjhh5w5c4Z77733itcMGTKEoKCgtEd4eHgORnh9WlcPJtDXk0On4lm267jV4YiIiIiISB7mtknghAkTeOutt/jpp58IDg6+4nWvvPIKp06dSnts2rQpB6O8Pj6eHtxauyQAv6lAjIiIiIiIuJBbJoE//vgjjzzyCD/99BNt2rS56rU+Pj4EBgamPQICAnIoysy5s55ZEvrX+kOcS0y2OBoREREREcmr3C4JnDhxIr1792bixInceuutVofjNA3KFiasiB9xicnM2nTY6nBERERERCSPsjQJPHPmDJGRkURGRgKwe/duIiMj2bdvH2CWcvbs2TPt+gkTJtCzZ0+GDRtGo0aNOHz4MIcPH+bUqVNWhO9UNpuNLnXVM1BERERERFzL0iRw5cqVREREpLV3GDBgABEREbzxxhsAHDp0KC0hBPjqq69ISkqiX79+hIaGpj369+9vSfzO1qVeaQAWbT/G0dgEi6MREREREZG8yNI+gS1btsThuHJLhLFjx17y9fz5810bkMXKFytI3bBCREbF8Mfagzx8Y3mrQxIRERERkTzG7fYE5nWpBWImr9lvcSQiIiIiIpIXKQnMZW6rXRJPu40NB06zPTr3NbYXERERERH3piQwlylS0JuWVU3fQ/UMFBERERERZ1MSmAulLgn9fc0BUlKuvGdSREREREQks5QE5kKtqwUT4OvJwVPxLNt93OpwREREREQkD1ESmAv5enlwW+1QACav1pJQERERERFxHiWBuVSXCNMz8K8NhzmXmGxxNCIiIiIiklcoCcylGpQtTOnCfpxJSGL25mirwxERERERkTxCSWAuZbfb6BJxoWfgavUMFBERERER5/C0OgC5si4RpRjx9w4Wbj/G0dgEigf4WB2SiIiIiEjGUlIgKen6H+Hh4Kl0xAr6Vc/FKhT3p05YIdZGxTB17UEeurG81SGJiIiIKzkcEBUF/v5QuDDYbFZHJO4oORl27IC1a83j2LHMJWfXepw/n/F5RyZbm0VHQ3Cwa34N5KqUBOZyd0aUYm1UDJPXHFASKCIiktccOwYrVsDy5eaxYgWcPGm+5+cHpUtf/VGsGNi1uydfi42F9eshMjI96Vu/Hs6etTqydJ6eGT9SUqyOLN9SEpjLdapTknf+3MT6A6fYcSSWSsEBVockIiLu4Px58PKyOgq5WHy8eaOemvAtXw67dl1+naenmVU5dw62bzePK/H2hlKlrp4ohoSAh4fLfiyXSkyEmBiTGMfEXH5cqBCUKwfly0PZsuDra2GwLuZwwL596Yne2rXmz9POnRlf7+cHtWpBnTrmz4GXV8aJ2JXOZ/Xx3/Hsds1o50JKAnO5IgW9aVm1OHM2H+G31Qd4sUM1q0MSEZHcJjkZNm6EJUvgn3/M865d4OMDRYqYZYVXer7SOe3TyR6HwyRvF8/yRUaa5Py/qlWDRo2gYUPzXLu2SQIPHoT9+6/8iI42SdLu3eZxJR4eULLk1RPF0FDXfGiQkmJmqq6UyF3rObOzWSVLmoQwNTG8+FG6tPv8uY6Ph02b0hO91KQvJibj60uWNMle3brmuU4dqFzZfZN/cTk3+ZuQv3WJKM2czUf4PfIgz7erit2uT1NERPK12FiTVKQmfMuWwenTl1+XkACHDplHZgUEXDuBzCiRDAzMn5/6X21Z58WKFzeJXurjhhvMbNZ/eXlBxYrmcSWJieb39mqJ4sGD5kOCqCjzuBKbDUqUuHKSWLKkSUwzm8idOuWcJX+BgebPV6FC6c9BQXDiRHoSHBdnft6DB83fjf/y8ICwsEsTw4uTxRIlrFlae+TI5cne5s3m9+2/PD1NMZXURC/1Ubx4joct7k1JoBu4uXowAb6eHIg5x/LdJ2hSsajVIYmISE5xOGDPHpPspc70rV9/+RvrgABo3BiaNjWPOnXMbMLJk+aN8vU+pyaTsbHmsXdv5uL18DBv0DNKGIsUMW9WQ0IufRQq5F6JY0ICrFmTnuwtX57xkjxfX6hXLz3ha9jQJB3O+lm9vc0SyLJlr3xNUpKZMbxaonjggJmhTP3A4N9/nRPff2NN/cDg4kTuep4DA689o+VwwPHj6Qnh7t3m783Fx4mJ5nnPHpg37/IxfHzM709Gs4jlykHRotn7vUtKgm3bLl/OefhwxtcXKZKe5KXO8FWvbuIUySYlgW7A18uDW2uF8uO/UUxes19JoIhIXpaYaBKM1Fm+JUsynskrX94ke82ameeaNTN+o3y1BCEjSUnpsziZSR5PnDDJUXKyeTN+/Pj139PLy1QI/G9yGBJy+fmiRXN2iZvDYaosXryP70rLOqtWvXSWr3Zt6/dlenqaPYOlSpmYMpKSAkePXpoUZpQoentnPoFLfXb1Xj2bzRTJKVbMzK5m9DMeOnRpYnhxghgVZf78bt1qHhnx97/yLGL58uaDmFSnTsG6dZfO8G3YYD6YySj2SpUuX85ZurR7fTgibsXmcGS2lqt7279/P2FhYURFRVG6dGmrw7luy3cdp+tXywjw8eTf19vg66U13iIiecLRo7B0afos38qVl79R9PIyM0qpCV/TpmYPV25z7tzVk8Xjx83St+ho8zhyxLxZzgy73cwmZpQ0/vdccHDmk7Djxy9f1nnixOXX/XdZZ4MGJtkR93T+vEl2M5pF3L37+pZUFy1qEsPUJaoZKVjQfDhw8QxfzZomwRRLuGtukF2aCXQTN5QrQqlCfhyIOcfsTdF0qlPS6pBERCSzUlJgy5ZLZ/m2bbv8umLF0pO9Zs2gfn1T6S+38/Mzj5KZ+D8qPv7yxDD1+L/njh83v4ap59evv/b4RYpcPWEMDDQzNqlJ39WWdaYWbmnUyLnLOsV6Xl7pM3oZOXfOVObMaBZx9+702e+LZ8DDwi5fzlmxolp6SK6gJNBN2O02ukSU4rN5O5i85oCSQBERdxAXZ2aSUmf5li7NuLpfePils3yVK+efBMPXF8qUMY9rSUoyM6fXShZTj5OT05eqbtly/THlxmWdYi0/P/PnomrVjL9/+nT6fsOAAJPwFSmSkxGKZIqSQDfSpZ5JAhdsO8qxMwkU89fGYBGRXCUq6tJZvsjIyyv8FShgEovUhK9JEy0jvF6enmYZ7PUshU1JMUtQr5YoRkeba6pVS5/lu+EG/X5I5gUGmg8Late2OhKR66Ik0I1ULO5PndJBrN1/iqlrD9K72RWWLIiI5EWpzbNTi48kJaU/X3xsxfeOHjVJ3/79l8cdFnZpARfNKuUMu93s0Spa1My0iohIGiWBbqZLRCnW7j/F5DUHlASKSM5LTjZ7uM6dM8+pj4u/vp7jrLwmo55ZuY2HB0REpM/yNW1qkkAREZFcREmgm+lUpyTvTtvMuv2n2HHkDJWCVU1KxO2lpMDq1bBr1+WzTNfzdVa/dz3XJiZempAlJVn9q5XOw8MsD/T0TD/+7/P1nsvs9f895+9vlhPecIOp/iciIpKLKQl0M0X9fWhRpThztxxh8pr9vNC+mtUhiUhWxMXBnDkwdSpMm3blZsG5maenKerh52eer/c4u9d5e5vEK78UThEREXEyJYFuqEu9UszdcoQpaw7yXNuq2O16IyTiFvbtMwnf1Knw999mb1sqf3+zjNDb+/IZpyvNQGXne5kZI7Xs/8XJmI+P+Z6IiIi4Hf0P7obaVA8hwMeTAzHnWLHnBI0rFLU6JBHJSEoK/PuvSfr+/BPWrr30++XKQadO5tG8uUmsRERERFxMSaAb8vXy4JZaoUxaGcXk1QeUBIrkJmfOwKxZJumbNs2Uo09lt5t2ALfdZhK/8HAtaRQREZEcpyTQTXWpV4pJK6OYvv4Qb91RA18vD6tDEsm/9u5Nn+2bN88UU0kVEAAdOpikr2NHKFbMujhFREREUBLothqWK0KpQn4ciDnHnM3R3Fa7pNUhieQfycmwYoVJ/KZOhQ0bLv1+hQrpyzxvusns8xMRERHJJZQEuim73UbniJJ8Pm8nk1cfUBIo4mqnT6cv85w+3TQHT2W3m0bgnTqZpZ7VqmmZp4iIiORaSgLdWJeI0nw+bycLth3l+JkEivqrqISIU+3enb7Mc/58OH8+/XtBQWZ55223meciRSwLU0RERCQzlAS6sUrB/tQuHcS6/aeYuvYgvZqVtzokEfeWnAxLl5qkb+pU2LTp0u9Xrpy+zLNZM/DysiZOERERkWxQEujmukSUYt3+U0xec0BJoEhWnDoFM2eapO+vv+D48fTveXiYPX2p1TyrVLEuThEREREnURLo5jrVKcm70zazdv8pdh49Q8Xi/laHJJK7JSTA1q2mWfvUqbBwISQlpX+/cGGzvLNTJ2jf3nwtIiIikocoCXRzxfx9aFGlOH9vOcLk1Qd4vn1Vq0MSyR1iY2Hz5ksfmzbBrl2mifvFqlVLn+1r2hQ89U+jiIiI5F16p5MHdIkoZZLANQcY0LYKdruqEko+cvTopUle6vH+/Vd+TVAQNGhgEr/bboNKlXIuXhERERGLKQnMA9qGhxDg48mBmHP8u+cEjSoUtTokEedyOExSd3GSl5r0XbyH779CQiA8HKpXT3+Eh0OJEmrhICIiIvmWksA8wNfLg461SvDTyv1MXnNASaC4r6Qks1zzvzN7W7bAmTNXfl25cpcmeanH2s8nIiIichklgXlEl4jS/LRyP9PWH+LN22vg6+VhdUgiVxYfD9u2XT6zt20bJCZm/BpPT7Ns878ze1WrQsGCORu/iIiIiBtTEphHNCpfhFKF/DgQc465m49wa+1Qq0OS/C4pCQ4ehKgo2L790pm93bsvL86Sys/PFGr578xepUrqyyciIiLiBEoC8wi73cYddUsycv5OJq/ZryRQXCs1wdu/3zyioi5/Pnz4yokeQKFCly/frF4dypYFuz3HfhQRERGR/EZJYB5yZ71SjJy/k/lbj3L8TAJF/X2sDknckTMSvFReXlCqFJQvn57spT6HhKg4i4iIiLilzz//nKFDh3L48GHq1KnDiBEjaNiw4RWvHz58OF988QX79u2jWLFi3H333QwZMgRfX98cjDqdksA8pFJwALVKBbH+wCn+XHeIB5uWszokyW1ckeCVLg1hYeb54uOwMAgO1qyeiIiI5CmTJk1iwIABjBo1ikaNGjF8+HDat2/P1q1bCQ4Ovuz6CRMm8PLLLzNmzBiaNm3Ktm3b6NWrFzabjY8++siCn0BJYJ7TJaIU6w+c4rc1B5QE5kdHj5r9d0rwRERERFzio48+ok+fPvTu3RuAUaNGMW3aNMaMGcPLL7982fVLliyhWbNm3H///QCUK1eObt26sXz58hyN+2JKAvOYTnVK8t70zayNimHn0TNULO5vdUjiaidPwq+/woQJMH++6al3NUrwRERERC4RGxvL6dOn07728fHBx+fyrVWJiYmsWrWKV155Je2c3W6nTZs2LF26NMOxmzZtyg8//MCKFSto2LAhu3btYvr06fTo0cP5P8h1UhKYxxQP8OGmysWYv/UoU9Yc4Ll2Va0OSVwhLg6mTjWJ34wZcP58+vfKljWJnBI8ERERkesSHh5+ydeDBg3izTffvOy6Y8eOkZycTEhIyCXnQ0JC2LJlS4Zj33///Rw7dowbb7wRh8NBUlISjz/+OK+++qrT4s8sJYF5UJeIUszfepTJaw7wbJsq2O0qvpEnJCbCrFkwcSL8/rtJBFPVrg333w/33WeSQBERERG5bps2baJUqVJpX2c0C5hV8+fPZ/DgwYwcOZJGjRqxY8cO+vfvzzvvvMPAgQOddp/MUBKYB7ULL4G/jyf7T55j5d6TNCxfxOqQJKtSUmDhQpP4/fILnDiR/r0KFUzi162bqbgpIiIiIlkSEBBAYGDgNa8rVqwYHh4eREdHX3I+OjqaEiVKZPiagQMH0qNHDx555BEAatWqRVxcHI8++iivvfYadgtWaGlNmNXOxzt9SD9vDzrUNH8IJ6/Z7/TxxcUcDli1Cp57DsqUgVat4KuvTAJYogT07w/Ll8OOHfDOO0oARURERHKIt7c39evXZ+7cuWnnUlJSmDt3Lk2aNMnwNWfPnr0s0fPw8ADAca1aDi6imUArrfsZZr0OXb6Aiq2dOvSdEaX4ZdV+/lx3iEGdauDr5eHU8cUFtmwxM34TJ5oKn6mCguDuu82MX8uW4KHfSxERERGrDBgwgAcffJAGDRrQsGFDhg8fTlxcXFq10J49e1KqVCmGDBkCQKdOnfjoo4+IiIhIWw46cOBAOnXqlJYM5jQlgVY6sArOHIYZr8Dj/4CH8347GlcoSmiQL4dOxfP3liPcUivUaWOLE0VFwaRJpsDLmjXp5/384PbbTeLXoQM4cV26iIiIiGRd165dOXr0KG+88QaHDx+mbt26zJgxI61YzL59+y6Z+Xv99dex2Wy8/vrrHDhwgOLFi9OpUyfee+89q34EbA6r5iAtsn//fsLCwoiKiqJ06dLWBnPuJHxaD86dgI5DodGjTh3+f39tYdSCnbSpHszXD97g1LElG44dM/v7JkyARYvSz3t6Qrt2Zp/f7bdDQIB1MYqIiIjkA7kqN8hB2hNoJb/C0Po1czzvPTh74urXZ9Kd9UyFo/lbj3L8TIJTx5ZMio2FH36AW26B0FDo2zc9AWzeHL74Ag4dgmnT4IEHlACKiIiIiMsoCbRavV4QXAPiY2D+EKcOXSUkgBolA0lKcfDnukNOHVuuQ0ICTJkCXbtCSAj06AF//QVJSVCvHgwdCvv2wYIF8PjjUKyY1RGLiIiISD6gJNBqHp7Q4ULy9+83EL3JqcN3iTCzgb+tOeDUceUKkpNhzhx4+GGT+HXpAj/9BOfOQZUqMGiQKQCzahU8/7xp3i4iIiIikoOUBOYGFVpAtdvAkQwzXzEtApzk9rolsdtgbVQMO4+ecdq4chGHA5YtM60bSpeGtm1hzBg4dQpKlTKtHlauNMnfm29C1apWRywiIiIi+ZiSwNyi3bvg4Q275sPWv5w2bHCALzdVLg7AFM0GOtfGjfDaa1CpEjRpAp9+CocPQ5Ei8NhjMH++We754YdQvz7YbFZHLCIiIiJibRK4cOFCOnXqRMmSJbHZbEyZMuWar5k/fz716tXDx8eHSpUqMXbsWJfHmSOKlIcmT5rjma9CkvMKuaQWiJm85gApKfmqGKzzHTtmkr369aFmTRg8GHbtgoIFTVXPP/80BV5GjYIWLcCuz1lEREREJHex9B1qXFwcderU4fPPP7+u63fv3s2tt95Kq1atiIyM5JlnnuGRRx5h5syZLo40h9w0APxD4ORuWPaF04ZtF16Cgt4e7D95jpV7Tzpt3HwjMdEUeOnSBUqWNMs+V682LR06dTLN3aOjYfx4uPVW8Pa2OmIRERERkSuytFl8x44d6dix43VfP2rUKMqXL8+wYcMAqF69OosXL+bjjz+mffv2Gb4mISGBhIT0WbXY2NjsBe1KPgHQ5k2Y0hcWfgh1ukFASLaH9fP2oEPNUH5dvZ/Ja/bTsHyR7Mea1zkcpnn7d9+Zfn7HjqV/r1496NXLNHJXRU8RERERcTNutVZt6dKltGnT5pJz7du3Z+nSpVd8zZAhQwgKCkp7hIeHuzrM7Kl9H5SsB4mx8PfbThs2dUnon+sOEX8+2Wnj5jmHD8OwYVC7tlny+emnJgEsUcJU81y/3lT2fOopJYAiIiIi4pbcKgk8fPgwISGXzoyFhIRw+vRpzp07l+FrXnnlFU6dOpX22LTJuS0YnM5uh47vm+M14+HgGqcM27hCUUoE+hIbn8TfW444Zcw8Iz4efv7ZLOUsXdokexs2gI8P3HuvaeAeFWX6+tWsaXW0IiIiIiLZ4lZJYFb4+PgQGBiY9ggICLA6pGsLawi17gUc8NfLTmkZ4WG3cUdESQB+W60qoWltHfr2hdBQk+xNn276/DVuDF98YQq8TJoEt9xi9v+JiIiIiOQBbpUElihRgujo6EvORUdHExgYiJ+fn0VRuUibN8GrAEQtgw2/OmXIOyNKAzB/6xFOxCU6ZUy3s38/DBkC1aubtg6jRkFMjJkBfPVV08tv6VJ4/HEoXNjqaEVEREREnM6tksAmTZowd+7cS87Nnj2bJk2aWBSRCwWVghufNcez34DEs9kesmqJAMJDA0lKcfDnuoPZHs9tnD1rKne2bQtlyphkb+tW8POD7t1h9mzYswfee0+N3EVEREQkz7M0CTxz5gyRkZFERkYCpgVEZGQk+/btA8x+vp49e6Zd//jjj7Nr1y5efPFFtmzZwsiRI/npp5949tlnrQjf9Zo+BUFhcPoALPnUKUOmFojJ80tCHQ5YtAgeftgUdeneHebMMeebN4dvvjFFYMaNgzZtwMPD6ohFRERERHKEpUngypUriYiIICIiAoABAwYQERHBG2+8AcChQ4fSEkKA8uXLM23aNGbPnk2dOnUYNmwYX3/99RXbQ7g9Lz9o9445XjwcYqKyPeTtdUpit0FkVAy7jp7J9ni5zu7d8NZbUKmSSfbGjIHYWChfHgYNgp07YcECeOghCAy0OloRERERkRxnczicUHXEjezfv5+wsDCioqIoXbq01eFcm8MB394C+5ZAzbvg7jHZHrLnmBUs3HaUp1tXYkC7PLD8MTYWfvnF9PRbsCD9vL8/3HOP6el3442m8qqIiIiIyAVulxs4id4V53Y2G3T8H2AzBWL2Xrkn4vW6M8IsCZ0ceQC3/QwgJQXmzoWePc1yz4ceMgmgzWaWd44bZ5Z7jhljZgSVAIqIiIiIAKC69+4gtA7U6wmrv4MZL0Gf+dlKatrVCKGAtwdRJ86xcu9JbihXxHmxutq2bWbGb9w407svVZUq8OCD0KMHhIVZF5+IiIiISC6n6RF30Xog+ATCobUQOT5bQxXw9qRDzRKAmxSIOX8eRo+Gpk1N9c7Bg00CGBQEjz0GS5aY1g6vvqoEUERERETkGpQEugv/4tDiRXM89y2IP52t4VJ7Bk5bd5D488nZjc511qyBhg3h0UdN/z67HTp2hB9/NMs9R40y/f5sNqsjFRERERFxC0oC3UnDx6BIRYg7Cos+zNZQTSoWJSTQh9PxSczbcsRJATpRQgIMHGgSwMhIKFIEPvjANHufPh26dgVfX6ujFBERERFxO0oC3YmnN7QfbI6XjoTjO7M8lIfdRue6F3oGrsllS0JXrID69eHddyEpCe6+GzZtghdegNBQq6MTEREREXFrSgLdTZX2UPFmSDkPs17P1lBdLjSOn7/1CCfjEp0RXfacOwcvvmiWd27cCMHB8PPP5hESYnV0IiIiIiJ5gpJAd2OzQYchYPOArdNh599ZHqpaiUCqhwZyPtnBn+sOOjHILPjnH6hbF4YONe0fHnjAJIJ3321tXCIiIiIieYySQHdUvCo07GOOZ7wCyUlZHiq1Z6BlS0Lj4qB/f7jpJtP+ITQUfv8dfvgBihWzJiYRERERkTxMSaC7avky+BWBo1tg5ZgsD3NH3ZLYbbBmXwy7j8U5McDrMG8e1KoFn34KDgf07m32/t1+e87GISIiIiKSjygJdFd+haH1a+Z43ntw9kSWhgkO9KVZJTPjNjmnZgNPn4a+faF1a9i92/T2mzEDxoyBQoVyJgYRERERkXxKSaA7q9cLgsMhPgbmD8nyMHdeKBAzZc0BHA6Hc2K7kpkzoWZN098PTDK4YQO0b+/a+4qIiIiICKAk0L15eJoiMQD/fgNHNmdpmPY1SlDA24N9J86yau9JJwZ4kZMn4aGHoEMHiIqCChXg779h5EgIDHTNPUVERERE5DJKAt1dhZZQ7TZwJJsiMVmYySvg7UmHGiUAFxWImToVatSAb7811U3794d166BVK+ffS0RERERErkpJYF7Q7l3w8IZd82DrX1kaIrVn4LR1h0hISnZOXMePm1YPt98Ohw5BlSqwaBEMHw4FCzrnHiIiIiIikilKAvOCIuWhST9zPPNVSErI9BBNKxYjJNCHU+fOM2/LkezH9MsvEB4OEyaA3W6awEdGQrNm2R9bRERERESyTElgXnHTc+AfAid3w/JRmX65h93GHXUv9AxcnY0lodHRcM895nHkiFkGunQpvP8++PllfVwREREREXEKJYF5hU8A3DzIHC8YCrHRmR6iy4XG8fO2HuFkXGLmXuxwwPjxZvbvl1/AwwNefx1WrYKGDTMdi4iIiIiIuIaSwLykTjcoGQGJsfD325l+efXQQKqVCOB8soOp6w5e/wsPHIA77oDu3eHECahbF/79F955B3x8Mh2HiIiIiIi4jpLAvMRuh44fmOM14+HgmkwPcU+DMAA+mr2NgzHnrn6xw2EqftaoYSqAenmZxG/FCoiIyPS9RURERETE9ZQE5jVhDaHWPYAD/no50y0jujcuQ61SQcScPc/TE9eQlJyS8YX79pmefw89BKdOwQ03wOrVZgmol1f2fw4REREREXEJJYF5UZu3wKsARC2DDb9m6qU+nh58dn8E/j6erNx7ko/nbLv0gpQUGDXKzP7NmmWWe77/PixZAjVrOvGHEBERERERV1ASmBcFlYIbnzXHswdB4tlMvbxs0YIMubMWACPn72TR9qPmG7t2QZs20LcvnDkDTZvC2rWm/YOnpzN/AhERERERcRElgXlV06cgKAxO74cln2b65Z3qlKRbwzI4HDBg4mpi3/8QatWCefNMq4fhw2HhQqha1fmxi4iIiIiIyygJzKu8/KDthQqhi4fDqf2ZHmJQp3Butp1k5FcDCHj5BTh7Flq2hPXroX9/0wZCRERERETcipLAvKxGFyjTFJLOmWWhmZGUhO/wjxg9vA83HNjEGW8/5j3zFsydCxUruiZeERERERFxOSWBeZnNBh3/B9hgwy+wd+n1vW7DBrPf78UXscfHc7hxc9o9/DkP+9Znxd4YV0YsIiIiIiIupiQwrwutA/V6mOMZL5vqnlczfz7Uq2eavQcFwZgxhPwzj0YtIkhxwNMT13AiLtHlYYuIiIiIiGsoCcwPWg8En0A4FAlrJ1z92tdfh/PnTRXQjRuhd29sdjvvdK5JhWIFOXw6nhd+Xosjk/0HRUREREQkd1ASmB/4B0PzF8zxnLcg/nTG1y1ZAv/8A97e8P33UKpU+hA+noy4PwJvTztztxzhm8W7cyBwERERERFxNiWB+UWjx6FIRYg7Aos+zPiaoUPNc48eEBp62bdrlAxi4K3VAXh/xhbWRsW4KFgREREREXEVJYH5hac3tB9sjpd9Acd3Xvr9rVvh99/N8XPPXXGY7o3L0rFmCc4nO3hy4mpOx593UcAiIiIiIuIKSgLzkyrtoWJrSE6EWQMv/d6wYeBwQKdOUL36FYew2Wz8767alC7sR9SJc7zy63rtDxQRERERcSNKAvMTmw3aDwGbB2ydBjvnmfPR0WYPIMCLL15zmCA/L0Z0i8DTbmPa+kNMWLHPhUGLiIiIiIgzKQnMb4KrQcM+5njGK5CcBCNGQEICNG4MzZpd1zARZQrzYoeqALw1dRObD12h2IyIiIiIiOQqSgLzoxYvgV9hOLoZFn0BI0ea8y+8YGYLr9MjN1agVdXiJCal8OSE1cQlJLkoYBERERERcRYlgflRgSLQ6jVzPGwQnDwJlSvDHXdkahi73cawe+sSEujDzqNxvPH7RhcEKyIiIiIizqQkML+q3xuKVoNFMebr554DD49MD1OkoDef3BeB3Qa/rt7Pr6v2OzdOERERERFxKiWB+ZWHJyS3hVMOKGCDDg2zPFTjCkV5pk0VAAb+voGdR884K0oREREREXEyJYH5lcMB300zx428YcFb5lwW9WtViaYVi3I2MZl+41cTfz7ZSYGKiIiIiIgzKQnMr+bOhchIKOAHjQrCrnmwbUaWh/Ow2xjetS5FC3qz5XAs707b5LxYRURERETEaZQE5lcffGCeH34EWj9pjme+CkkJWR4yONCXj7rWBeCHZfuYtu5QNoMUERERERFnUxKYH0VGwuzZphDMgAHQ/HnwD4ETu2D5qGwN3aJKcfq2rAjAy7+uY9/xs04IWEREREREnEVJYH704Yfm+Z57oFw58AmAmweZcwuGwpkj2Rp+QNsq1CtTiNiEJJ6auJrEpJTsxSsiIiIiIk6jJDC/2bsXfvzRHL/wQvr5Ot2gZAQkxsLct7N1Cy8PO592iyDIz4u1+08xdOaWbI0nIiIiIiLOoyQwvxk+HJKT4eaboV699PN2O3R43xyv+QF2zsvWbUoXLsAHd9cGYPSi3fy9JTpb44mIiIiIiHMoCcxPTp6E0aPN8cWzgKnKNIJa9wAOGNcZvm4D636GpMQs3a59jRL0aloOgOd+WsuhU+eyNI6IiIiIiDiPksD85IsvIC4OateGdu0yvuaWoVD7PrB7wf5/4bdHYHhNmP8/iM38bN4rt1SjZqlATp49T/+JkSQla3+giIiIiIiVlATmF/Hx8Omn5viFF8Bmy/g6v8Jw55fw7EZo+aqpGnomGuYPgY9rwK+PQNS/191Y3sfTg8+61cPfx5MVe07w6dztTvqBREREREQkK5QE5hc//ADR0RAWBl27Xvv6gBBo+RI8swHu+gZKN4SU87D+Z/imDYxuBWt/vK6+guWKFWTwnbUAGDFvB//sOJbdn0ZERERERLJISWB+kJKS3hbimWfAy+v6X+vpDbXuhkdmw6Pzoc794OENB9fA5MfM7ODf78Lpg1cd5vY6JbnvhjAcDnhmUiRHY7PelF5ERERExEqff/455cqVw9fXl0aNGrFixYqrXh8TE0O/fv0IDQ3Fx8eHKlWqMH369ByK9nJKAvODqVNh61YICoI+fbI+TskI6PIFDNgMrQdCYCmIOwoLh8LwWvBzL9i79IpLRQd1qkGVEH+OxiYw4KdIUlKub0mpiIiIiEhuMWnSJAYMGMCgQYNYvXo1derUoX379hw5knGv7cTERNq2bcuePXv45Zdf2Lp1K6NHj6ZUqVI5HHk6m8NxnZu78oj9+/cTFhZGVFQUpUuXtjqcnHHjjfDPP/DyyzBkiPPGTU6CLX/C8i9h35L08yVqQcPHzAyil98lL9keHUunzxYTfz6FFztU5YmWlZwXj4iIiIhIJmQlN2jUqBE33HADn332GQApKSmEhYXx1FNP8fLLL192/ahRoxg6dChbtmzBKzMr8lxIM4F53ZIlJgH09oann3bu2B6eUKMzPPQXPL4YInqApy8cXg9/PAkfhcOcNyEmKu0llUMCePv2mgAMm7WNlXtOODcmEREREZFMio2N5fTp02mPhISMty4lJiayatUq2rRpk3bObrfTpk0bli5dmuFr/vjjD5o0aUK/fv0ICQmhZs2aDB48mOTkZJf8LNdDSWBeN3Soee7RA0JDXXefErXgjs/MUtE2b0FQGJw7AYs/hk9qw6TusHsROBzc06A0neuWJDnFwdMT1xBzNmt9CEVEREREnCE8PJygoKC0x5ArrJ47duwYycnJhISEXHI+JCSEw4cPZ/iaXbt28csvv5CcnMz06dMZOHAgw4YN491333X6z3G9PC27s7je1q3w++/m+LnncuaeBYrAjc9A06dg61+wfBTsWQSbp5pHcA1sDfvw7m13sXb/KXYfi+P5n9cxumd9bFdqWyEiIiIi4kKbNm26ZI+ej4+P08ZOSUkhODiYr776Cg8PD+rXr8+BAwcYOnQogwYNctp9MkMzgXnZsGGmSEunTlC9es7e2+4B1W+DXn9C36VQvzd4FYAjG+HPZ/D/rBY/VZhOBY9jzNkczbf/7MnZ+ERERERELggICCAwMDDtcaUksFixYnh4eBAdHX3J+ejoaEqUKJHha0JDQ6lSpQoeHh5p56pXr87hw4dJTLRmRZzlSWBmy6sOHz6cqlWr4ufnR1hYGM8++yzx8fE5FK0biY6G7783xy++aG0sIeHQaTgM2ATt3oPC5SA+huLrvmSuV39Gew1jwYyfWBd10to4RURERESuwtvbm/r16zN37ty0cykpKcydO5cmTZpk+JpmzZqxY8cOUlJS0s5t27aN0NBQvL29XR5zRixNAjNbXnXChAm8/PLLDBo0iM2bN/PNN98wadIkXn311RyO3A2MGAEJCdC4MTRrZnU0hl9haPokPLUauk2Ciq2x4aCtxyq+8xxM4JgbiV/yJSScsTpSEREREZEMDRgwgNGjR/Pdd9+xefNm+vbtS1xcHL179wagZ8+evPLKK2nX9+3blxMnTtC/f3+2bdvGtGnTGDx4MP369bPqR7B2T+BHH31Enz590n7BRo0axbRp0xgzZkyG5VWXLFlCs2bNuP/++wEoV64c3bp1Y/ny5Tkad6535gyMHGmOX3gBctteO7sHVO1gHke3kbBkFMlrxlPOsR9mvYhjwbvYIrrDDY9A0YpWRysiIiIikqZr164cPXqUN954g8OHD1O3bl1mzJiRVixm37592O3pc21hYWHMnDmTZ599ltq1a1OqVCn69+/PSy+9ZNWPYF2fwMTERAoUKMAvv/xC586d084/+OCDxMTE8HtqQZOLTJgwgSeeeIJZs2bRsGFDdu3axa233kqPHj2uOBuYkJBwSYnXAwcOEB4enrf7BH7yCTzzDFSuDJs3w0Xrj3OryB37+GPsh3S3z6SCPbWykg0qtzU9Byu2Brvlq5dFREREJA/Jlz3EsXA5aFbKq95///28/fbb3HjjjXh5eVGxYkVatmx51eWgQ4YMuaTca3h4uFN/jlzn/Hn46CNz/NxzbpEAAtStVIbgtv25OfFDHkl+mTNlWgMO2D4Lxt8FnzWAPYutDlNERERExO251dTK/PnzGTx4MCNHjmT16tX89ttvTJs2jXfeeeeKr3nllVc4depU2mPTpk05GLEFfv4Z9u2D4sWhZ0+ro8mUR2+qQPMqIcw5X5vOMc9w7vEV0Kgv+ATCiZ3w6yOQeNbqMEVERERE3JplSWBWyqsOHDiQHj168Mgjj1CrVi26dOnC4MGDGTJkyCXVdi7m4+NzSbnXgIAAp/8suYbDkd4c/umnwc/P2ngyyW638dG9dQgJ9GHHkTMMWhwPHf9nqooWKgOxh0zfQRERERERyTLLksCslFc9e/bsJZssgbR+GxZtbcxd5s6FyEgoUAD69rU6miwp6u/D8K4R2G3w08r9TFlzAHwCoPVAc8Hi4XD2hKUxioiIiIi4M0uXg2a2vGqnTp344osv+PHHH9m9ezezZ89m4MCBdOrU6ZLmi/nWBx+Y54cfhqJFrY0lG5pULMrTN1cG4LXJ69l19AzUvBtCakHCKVg0zOIIRURERETcl6UtIjJbXvX111/HZrPx+uuvc+DAAYoXL06nTp147733rPoRco/ISJg92xSCGTDA6miy7anWlVm26zjLdp3gyQlr+O2Jpvi2fRN+uAtWfAWNHjNLREVEREREJFMsaxFhlTxbBrZ7dxg/Hu67DyZOtDoap4g+HU/HTxZxIi6RB5uU5a3ba8D3t8PuhVCnG3TR/kARERERybo8mxtcg1tVB5Ur2LsXfvzRHL/wgrWxOFFIoC8f3VsHgO+W7uXP9YegzVvmm2t/hMPrLYxORERERMQ9KQnMC4YPh+RkuPlmqFfP6micqmXVYB5rUQGAZydFMuNkKNS4E3DAnLesDU5ERERExA0pCXR3J0/C6NHmOA/NAl7shXZVub1OSc4nO+g3YQ1zSz4Kdk/YMdssDRURERERkeumJNDdffEFxMVB7drQrp3V0biEp4edj7vW5e76pUlOcdDnzxPsKHOP+ebsQaY/ooiIiIiIXBclge4sPh4+/dQcv/AC2GzWxuNCHnYbH9xVm24Ny5DigG5bm3PeowAcXA2bplgdnoiIiIiI21AS6M5++AGioyEsDLp2tToal7PbbQzuUpNeTctx1BHEZ/EdzTfmvg3J560NTkRERETETSgJdFcpKfDhh+b4mWfAy8vScHKKzWZjUKdwHm1ega+Tb+GoIxBO7ILV31kdmoiIiIiIS/z7778sX778svPLly9n5cqVmR5PSaC7mjoVtm6FoCDo08fqaHKUzWbjlY7VeKh1LT5NuhOAuFmDIeGMxZGJiIiIiDhfv379iIqKuuz8gQMH6NevX6bHUxLoroYONc99+0JAgLWxWMBms/Fcu6qUaPkYe1JCKHj+OP+MewuHisSIiIiISB6zadMm6mXQCi4iIoJNmzZlejwlge5oyRL45x/w9oann7Y6Gkv1axvOjlrPAlAn6ns++WOJEkERERERyVN8fHyIjo6+7PyhQ4fw9PTM9HhKAt1R6ixgjx4QGmptLLlAm7se41hgDfxt8QT9+wlv/rGRlBQlgiIiIiKSN7Rr145XXnmFU6dOpZ2LiYnh1VdfpW3btpkeT0mgu9m6FX7/3Rw/95y1seQWdjvFOg8B4AGPOcxbtoLXpqxXIigiIiIiecKHH35IVFQUZcuWpVWrVrRq1Yry5ctz+PBhhg0blunxMj93KNYaNsw0R+/UCapXtzqa3KNCC6h4M9475/KC5088tSKEhKQUht5dBw973u2fKCIiIiJ5X6lSpVi3bh3jx49n7dq1+Pn50bt3b7p164ZXFroEZCkJjIqKwmazUbp0aQBWrFjBhAkTCA8P59FHH83KkHI9oqPh++/N8YsvWhtLbtT2Ldj5N508lvJ1ym38thrOJzv46N46eHlo0ltERERE3FfBggWdlmtlKQm8//77efTRR+nRoweHDx+mbdu21KhRg/Hjx3P48GHeeOMNpwQn/zFiBCQkQOPG0KyZ1dHkPiVqQe17Yd0kvik1jSYHnmLq2oMkJiUzols9vD2VCIqIiIiIe/jjjz/o2LEjXl5e/PHHH1e99vbbb8/U2DZHFkopFi5cmGXLllG1alU+/fRTJk2axD///MOsWbN4/PHH2bVrV2aHzDH79+8nLCyMqKiotJlMt3DmDJQpAydPwq+/wp13Wh1R7nRyL3zWAJITiWwxhnvnFiAxKYXW1YIZ+UA9fL08rI5QRERERHKJ3Jwb2O12Dh8+THBwMHb7lSczbDYbycnJmRs7KwGdP38eHx8fAObMmZOWeVarVo1Dhw5lZUi5lm++MQlg5cpwxx1WR5N7FS4LNzwCQN2tH/N1j3r4etn5e8sR+ny/knOJmfsLIiIiIiJihZSUFIKDg9OOr/TIbAIIWUwCa9SowahRo1i0aBGzZ8+mQ4cOABw8eJCiRYtmZUi5mvPn4aOPzPFzz4GHZrOu6qbnwScQDq+necJCvu3VkALeHizafoxe364gLiHJ6ghFRERERK7L+fPnufnmm9m+fbvTxsxSEvj+++/z5Zdf0rJlS7p160adOnUAs261YcOGTgtOLvj5Z9i3D4oXh549rY4m9ytYFJr1N8d/v02Tsv6Me7ghAT6eLN99gp5jVnA6/ry1MYqIiIiIXAcvLy/WrVvn1DGzlAS2bNmSY8eOcezYMcaMGZN2/tFHH2XUqFFOC04w7SBSm8M//TT4+Vkbj7to3Bf8S0DMPlg5hvpli/DDI40I9PVk1d6TdP96OTFnE62OUkRERETkmrp3784333zjtPGyVB303LlzOBwOChcuDMDevXuZPHky1atXp3379k4LToC5cyEyEgoUgL59rY7GfXgXhJYvw5/PwIIPoO791AkrxMRHG9P96+Ws23+K+0cvZ9zDDSnq72N1tCIiIiIiV5SUlMSYMWOYM2cO9evXp2DBgpd8/6PUrWPXKUszgXfccQffX+hXFxMTQ6NGjRg2bBidO3fmiy++yMqQciUffGCeH34YtN8ycyJ6QNHKcO4ELBkBQI2SQfz4aBOK+fuw6dBpuo1expHYeIsDFRERERG5sg0bNlCvXj0CAgLYtm0ba9asueSRWVlqEVGsWDEWLFhAjRo1+PrrrxkxYgRr1qzh119/5Y033mDz5s2ZDiSn5OYysJeJjISICFMIZscOKFfO6ojcz+apMKk7eBWAp9dAQAkAdh49w/2jlxF9OoEKxQoyoU9jSgT5WhysiIiIiOQkt8oNnChLM4Fnz54lICAAgFmzZnHnnXdit9tp3Lgxe/fudWqA+dqHH5rne+5RAphV1W6D0jfA+bMw/39ppysW9+enx5pQqpAfu47Fce+XS9l/8qyFgYqIiIiIZOyhhx4iNjb2svNxcXE89NBDmR4vS0lgpUqVmDJlClFRUcycOZN27doBcOTIEQIDA7MypPzX3r3w44/m+IUXrI3Fndls0PZtc7z6eziWXlq3bNGCTHqsMWWKFGDfibN0/XIZe4/HWRSoiIiIiEjGvvvuO86dO3fZ+XPnzqVt08uMLCWBb7zxBs8//zzlypWjYcOGNGnSBDCzghEREVkZUv5r+HBIToabb4Z69ayOxr2VbQpVOoIjGea+fcm3ShcuwE+PNaFCsYIciDnHvV8uZefRMxYFKiIiIiKS7vTp05w6dQqHw0FsbCynT59Oe5w8eZLp06enNZTPjCztCQQ4fPgwhw4dok6dOtjtJpdcsWIFgYGBVKtWLStD5gi3WPd78iSEhUFcHMyYAaq4mn3Rm2BUM3CkwMNzIOyGS759JDae7l8vZ1v0GYr5+zD+kUZULRFgUbAiIiIikhNye25gt9ux2WxX/L7NZuOtt97itddey9S4WWoRAVCiRAlKlCjB/v37AShdurQaxTvLF1+YBLB2bbiw1FayKSQc6twPkT/AnEHQa5pZKnpBcIAvPz7ahO5fL2fTodPc99VSxj3ciJqlgiwMWkRERETys3nz5uFwOGjdujW//vorRYoUSfuet7c3ZcuWpWTJkpkeN0szgSkpKbz77rsMGzaMM2fM0rmAgACee+45XnvttbSZwdwot2f7xMebIjDR0TBuHHTvbnVEecep/TCiPiTFw/0/QZXLZ1hjziby4JgVrN1/ikBfT8Y93Ig6YYVyPlYRERERcblcnxtcsHfvXsqUKXPVWcHMyFK29tprr/HZZ5/xv//9L603xeDBgxkxYgQDBw50SmD51g8/mAQwLAy6drU6mrwlqDQ0eswcz3kTUpIvu6RQAW/GPdKI+mULczo+iQe+Xs6qvSdyNk4RERERkYuULVuWxYsX0717d5o2bcqBAwcAGDduHIsXL870eFlKAr/77ju+/vpr+vbtS+3atalduzZPPPEEo0ePZuzYsVkZUgBSUtLbQjzzDHh5WRpOnnTjs+AbBEc2wbpJGV4S6OvF9w81pHGFIpxJSKLHNytYtut4DgcqIiIiImL8+uuvtG/fHj8/P1avXk1CQgIAp06dYvDgwZkeL0tJ4IkTJzIs/lKtWjVOnNCsSZZNnQpbt0JQEPTpY3U0eZNfYbjpOXP893twPj7Dywr6ePJtr4bcVLkYZxOT6fXtChZtP5qDgYqIiIiIGO+++y6jRo1i9OjReF00UdSsWTNWr16d6fGylATWqVOHzz777LLzn332GbVr187KkAIwdKh57tsXAlSZ0mUaPgqBpeD0fljx1RUv8/P2YHTPBrSuFkz8+RQe/m4lf2+JzsFARURERERg69atNG/e/LLzQUFBxMTEZHq8LCWBH3zwAWPGjCE8PJyHH36Yhx9+mPDwcMaOHcuHqcsZJXOWLIF//gFvb3j6aaujydu8/KDVq+Z40TA4d/KKl/p6eTCqe33a1wghMSmFx8atYsaGwzkUqIjFUpIhKdHqKERERPK9EiVKsGPHjsvOL168mAoVKmR6vCwlgS1atGDbtm106dKFmJgYYmJiuPPOO9m4cSPjxo3LypCSOgvYoweEhlobS35QpxsUrw7xMbB4+FUv9fa089n99ehUpyTnkx30m7CaqWsP5kiYIpaa2A0+qganD1kdiYiISL7Wp08f+vfvz/Lly7HZbBw8eJDx48fz/PPP07dv30yPl+Vm8RlZu3Yt9erVIzn58qqLuUWuLAO7dStUrw4OB2zaZI7F9bbOgIldwdMXnlplqodeRXKKgxd/Wcevq/djt8HQu+twV/1c8mdIxNli9sHwWua47TvQTCsUREQk78mVuUEGHA4HgwcPZsiQIZw9exYAHx8fnn/+ed55551Mj5d7G/rlJ8OGmQSwUyclgDmpSnso09T0DZw/5JqXe9htDL27Nt0ahpHigOd/WcvohbtISXHa5ygiucfWGenHG361Lg4RERHBZrPx2muvceLECTZs2MCyZcs4evRolhJAAE8nxyeZFR0N339vjl980dpY8hubDdq+Bd+0hcgJ0ORJCL56Em632xjcpRY+nh6MXbKH96ZvZvamaP53Vy0qFPfPocBFcsDWaenHhyLh+E4oWtGycERERPKjhx566LquGzNmTKbG1Uyg1UaMgIQEaNwYmjWzOpr8J6whVO8EjhSY+/Z1vcRmszGoUzjv3FGDAt4erNhzgo6fLOKrhTtJ1qyg5AXxp2DPhcazxaqY542/WRePiIhIPjV27FjmzZtHTEwMJ0+evOIjszI1E3jnnXde9ftZKU+ar505AyNHmuMXXjAzU5Lzbh4EW6bD1umwdymUbXLNl9hsNno0KUfLqsG88tt6Fu84xuDpW5i2/jBD765NlRC1+BA3tn02pCSZBLBZf/i9H2yYDM1fsDoyERGRfKVv375MnDiR3bt307t3b7p3706RIkWyPW6mZgKDgoKu+ihbtiw9e/bMdlD5xjffwMmTULky3HGH1dHkX8UqQ70e5nj2G2Z/5nUKK1KAcQ835P27ahHg48naqBhu/XQRI+Zu53xyiosCFnGxrX+Z56ododqtYPeCIxvhyBZr4xIREclnPv/8cw4dOsSLL77I1KlTCQsL495772XmzJlkp75npmYCv/322yzfSDJw8CB4eMBzz5lnsU6Ll2HtJNi/ArZMg+q3XfdLbTYbXW8oQ4sqwbw2eT1ztxxh2Oxt/LXhMB/cXZuapYJcGLiIkyWfNzOBAFVvBb/CUKkNbPvLLAkNftXa+ERERPIZHx8funXrRrdu3di7dy9jx47liSeeICkpiY0bN+Lvn/m6FNoTaKX334ddu0Czp9YLDIUm/czx3LcgOSnTQ5QI8uXrBxswvGtdChXwYtOh09zx+T98OHMrCUm5t22KyCX2/gMJp6BAMSjdwJyreWErwIZfMzVTLiIiIs5lt9ux2Ww4HI5steVTEmi1MmXAz8/qKARMHzS/InBsG0T+kKUhbDYbnSNKMfvZFtxSqwTJKQ4+m7eD2z5dzJp9md+0K5Ljtkw3z1U7gP3CCoWqHU0/zeM74PA662ITERHJhxISEpg4cSJt27alSpUqrF+/ns8++4x9+/ZlaRYQlASKpPMNSi98MW8IJJ7N8lDFA3wY+UB9vnigHsX8vdl+5Ax3fbGE96Zt4lyiZgUll3I4LtoPeEv6eZ8AqNzOHG9QlVAREZGc8sQTTxAaGsr//vc/brvtNqKiovj555+55ZZbsNuznsrZHNnZUeiG9u/fT1hYGFFRUZQuXdrqcCS3SUqAzxpAzD64+Q246blsD3kyLpF3/tzEb2sOAFCuaAHev6s2jSoUzfbYIk51eD2MutHM+r24G7wLpH9v4xT4+UEoVAb6r1M1YxERyRNye25gt9spU6YMERER2K7yf+9vv2XuQ1rNBIpczNMHWg80x4uHw9kT2R6ycEFvPupalzG9GlAi0Jc9x8/S9atlvPH7BuISMr/3UMRlUmcBK7S6NAEEMxPoVdB8QHJgVc7HJiIikg/17NmTVq1aUahQoat2acisTFUHFckXat4N/3wK0eth4YfQYbBThm1dLYRZA4owZPpmJq6I4vule5m7+Qjv31WbGysXc8o9RLJlyzTzXO2Wy7/nXcCcX/+zKRCTWjRGREREXGbs2LEuGVczgSL/ZbdD2zfN8b+j4eRepw0d6OvFkDtr88PDjShd2I8DMefo/s1yXv51HafjzzvtPiKZduoAHIoEbFClQ8bX1LhQJXTjZEhRH0wRERF3pSRQJCMVb4byzSE5EeY5ZybwYjdWLsbMZ5rzYJOyAPz4bxTtPlrI3M3RTr+XyHXZNsM8l74B/IMzvqbSzeATBLGHYN/SnItNREREnEpJoEhGbDZo85Y5XjfJFMxwsoI+nrx1R01+eqwJ5YoW4PDpeB7+biXP/LiGk3GJTr+fyFVtTW0N0fHK13j6QPXbzPFGVQkVERFxV0oCRa6kVL0Ly98cMOdNl92mYfki/NW/OY82r4DdBlMiD9L24wVMX3/IZfcUuURCLOxeaI6r3Xr1a1Mbx2+cAskqbCSSL5zYDSObwD+fWB2JiDiJkkCRq2n9Otg9Yccc2LXAZbfx8/bg1Vuq82vfplQO9ufYmUSeGL+avj+s4mhsgsvuKwLAjrlm6XORClCsytWvLd8C/IrA2WOwZ1HOxCci1lr4IRzZZIqmaT+wSJ6gJFDkaopWhPq9zfGcQaaZtgtFlCnMn0/fyFOtK+Fht/HXhsO0/XgBk9fsJ5+19JScdHGD+Gv1//PwgvA7zPGGX10bl4hY7/RBsy0CzIc/h9dZG4+IOIWSQJFrafESePvDwTWwaYrLb+fj6cFz7arye79mhIcGEnP2PM9OWsvD363k0KlzLr+/5DPJSbB9pjmumkFriIykLgndPBWStH9VJE9b9gWkXFS9euff1sUiIk6jJFDkWvyLQ9OnzPHctyE5Z1o51CwVxO9PNuP5dlXw9rDz95YjtPtoIT+u2KdZQXGeqGVw7qRZ4hnW6PpeU7YZ+IdAfAzsmufS8ETEQvGnYOW35rhSW/OsJFAkT1ASKHI9mvSDgsXhxC5YNTbHbuvlYefJ1pX58+kbqRNWiNiEJF7+bT09vllB1ImzORaH5GFbLlQFrdIePDyv7zV2DwjvbI43qEqoSJ61aiwkxkLxatBhiDm3bxkkxlkalohkn+VJ4Oeff065cuXw9fWlUaNGrFix4qrXx8TE0K9fP0JDQ/Hx8aFKlSpMnz49h6KVfMsnwCwLBVjwPiScydHbVwkJ4Le+TXntlur4eNpZvOMY7Ycv5Lsle0hJ0aygZJHDcVFriOtcCpqq5l3mecs0OB/v3LhExHpJCWYpKJjVMEUrQaEyZmnonsXWxiYi2WZpEjhp0iQGDBjAoEGDWL16NXXq1KF9+/YcOXIkw+sTExNp27Yte/bs4ZdffmHr1q2MHj2aUqVK5XDkki/V72WqJ8YdhaWf5/jtPew2+jSvwIxnmtOwXBHOJiYz6I+N3PfVMnYf06eykgVHt8DJ3eDhAxVbZ+61pW+AwNJmlmDHbNfEJyLWWf8zxB6CgFCodY8pGpX674SWhIq4PUuTwI8++og+ffrQu3dvwsPDGTVqFAUKFGDMmDEZXj9mzBhOnDjBlClTaNasGeXKlaNFixbUqVMnhyOXfMnDC1oPNMdLPoUzRy0Jo3yxgvz4aGPevqMGBbw9WLHnBB2GL2T0wl0ka1ZQMiN1FrBCC/Dxz9xr7Xao2cUcq0qoSN6SkmLaQQA07guePuZYSaBImsyuZkz1448/YrPZ6Ny5s2sDvAbLksDExERWrVpFmzZt0oOx22nTpg1Lly7N8DV//PEHTZo0oV+/foSEhFCzZk0GDx5McnLyFe+TkJDA6dOn0x6xsbFO/1kkHwnvDCUjIPEMLPzAsjDsdhs9m5Rj5jPNubFSMRKSUnhv+mbu/GIJ26L1Z1yuU+p+wKods/b6GheqhG6bqT1CInnJ9llwbCv4BJpVMKnKtwCbHY5tg5goy8ITsVpmVzOm2rNnD88//zw33XRTDkV6ZZYlgceOHSM5OZmQkJBLzoeEhHD48OEMX7Nr1y5++eUXkpOTmT59OgMHDmTYsGG8++67V7zPkCFDCAoKSnuEh4c79eeQfMZuhzZvmeOVY0yhGAuFFSnAuIcb8v5dtQjw8WRtVAy3frqIYbO2En/+yh+OiBAbDQdWmuMqWUwCS0ZA4fJw/mx6r0ERcX//fGKeG/QG36D0836FoFQDc6zZQMnHMruaESA5OZkHHniAt956iwoVKuRgtBmzvDBMZqSkpBAcHMxXX31F/fr16dq1K6+99hqjRo264mteeeUVTp06lfbYtGlTDkYseVKFFlCpDaQkwa+PwLHtloZjs9noekMZZg1oTpvqwZxPdjDi7x10/GQRS3cetzQ2ycW2XUjaStaDwNCsjWGzpfcM3DjZOXGJiLWiVsC+JWD3gkZ9L/++loRKHhUbG3vJ6sGEhIQMr8vKakaAt99+m+DgYB5++GGnx54VliWBxYoVw8PDg+jo6EvOR0dHU6JEiQxfExoaSpUqVfDw8Eg7V716dQ4fPkxiYsYNi318fAgMDEx7BAQEOO+HkPyr7TvgVRAOrIIvmsK8wZZXSAwN8mN0zwaMfKAewQE+7D4WR7fRy3jh57WcjFNDb/mP1Jm7apmsCvpfqVVCt88yPcVExL2lzgLW7prxB0SpSeCu+ZCiFSeSd4SHh1+yenDIkCEZXpeV1YyLFy/mm2++YfTo0U6PO6ssSwK9vb2pX78+c+fOTTuXkpLC3LlzadKkSYavadasGTt27CAlJSXt3LZt2wgNDcXb29vlMYukCQmHJ5aYGcHkRNM24oum5j9FC9lsNm6pFcrsAS14oFEZAH5etZ82Hy3g98gDajIvRmJc+p/VzLaG+K/gcChW1fw92KJ2PSJu7dgO0/YFTFuIjJSqDz5BEB8DByNzKjIRl9u0adMlqwdfeeUVp4wbGxtLjx49GD16NMWKFXPKmM5g6XLQAQMGMHr0aL777js2b95M3759iYuLo3fv3gD07Nnzkt+Avn37cuLECfr378+2bduYNm0agwcPpl+/flb9CJKfFS4HD/wCd38L/iFwYid8fwf89qhllUNTBfl58V6XWvzyeBMqB/tzPC6R/j9G0nPMCvYdV5P5fG/nPEiKh0JlTRKXHTZb+mygqoSKuLelIwCH2SccXC3jazw8oUJzc6wloZKHBAQEXLJ60MfHJ8PrMruacefOnezZs4dOnTrh6emJp6cn33//PX/88Qeenp7s3LnTJT/PtViaBHbt2pUPP/yQN954g7p16xIZGcmMGTPSplf37dvHoUOH0q4PCwtj5syZ/Pvvv9SuXZunn36a/v378/LLL1v1I0h+l7on6sl/4YY+gA3WTYLPGsCqsabMtoUalCvCtKdv4vl2VfD2tLNo+zHaDV/AqAU7OZ9sbWxioYsbxNts2R8vdV/grnlw9kT2xxORnBcbDZETzXGzp69+bdq+wLlXv05yh8gJsO5nq6PIMzK7mrFatWqsX7+eyMjItMftt99Oq1atiIyMJCwsLCfDT2Nz5LP1Yfv37ycsLIyoqChKly5tdTiS1+xfBX/2h8PrzddhjeG2j83yUYvtOnqGVyevZ9ku8ya9emgg/7uzFnXCClkbmOSslGT4sDKcPQ4PToXyzZ0z7qgbzZ/7Tp9cWlJeRNzD3Ldh0TAofQM8PPvqHxCd3AOf1AGbB7y0B3wDcypKyazDG2BUM3PszH/z85Cs5AaTJk3iwQcf5Msvv6Rhw4YMHz6cn376iS1bthASEkLPnj0pVarUFfcV9urVi5iYGKZMmeLEnyRz3Ko6qEiuV7o+9JkP7QebwjFRy+DLm2D2IEi0dhlmheL+TOzTmA/urk2hAl5sPnSaziP/4c0/NnImIcnS2CQHRa0wCaBvEJTJeP91lqT2DNzwm/PGFJGckRAL/35tjpv1v/YKgcLloEhFcCTDnkUuD0+yYdXY9OM/nrb8vUhekdnVjLmRZgJFXOXUfvjrJdjyp/m6UBm49SOo3NbauIBjZxJ4989NTIk8CEBokC/v3FGTNuEh13iluL1ZA2HJp1DrXrjLiVXK0mYG7PDcVvAPdt7YIuJaS0fCzFegaCXotwLsHtd+zbTn4d/RcMMjcOsw18comZd4FoZVhYTT4FXA9HRt+jS0e8fqyHKV/JobaCZQxFWCSsN94+G+CRBYGmL2wfi74acH4bS1nw4V8/dh+H0RfPdQQ8KK+HHoVDyPfL+Svj+sIvq0ta0uxMXS9gNmsUH8lRQuZ6oGOlJg0+/OHVtEXCf5PCz93Bw3efL6EkBI3xe4Q/sCc62Nk00CWKgs3PWNObf0Mziw2tq4JFdQEijiatVuhX7LzX+uNg/YNAU+bwjLv7K8x1KLKsWZ9UwLHmtRAQ+7jb82HKbNsAX8sGwvKSn5apFA/nBsOxzfYZpAV2pz7eszS1VCRdzPht/g9H4oWBzqdLv+15W7EeyecHI3nNjluvgk61Z/Z57rP2h6wta8y3xQ98dTJvmXfE1JoEhO8PGH9u/Bo/PNbEnCafjrBfi6DRxaa2loft4evNKxOn882Yw6pYOITUji9SkbuOfLpWyLjrU0NnGy1P5f5W9yTSGH8M7med9SOHXA+eOLiHM5HOnN4Rs9Dl6+1/9a30AIa2SOd85zfmySPdGbIGq5+fC57gPmXIf3wa8IRG9I/32XfEtJoEhOCq1tqq7d8iH4BMLB1fBVS5jxKiScsTS0GiWD+O2JZgzqFE5Bbw9W7T3JrZ8uYtisrcSft3bGUpxk61/mObsN4q8kqBSUaWqON052zT1ExHl2zoUjG00hsxsezvzrK7a6MI76BeY6qbOAVTtCwIXedf7FocP/zPGC9+HoNmtik1xBSaBITrN7QMM+prdgjS5macayz80S0c1/Whqah91G72blmT2gBW2qB3M+2cGIv3fQ8ZNFLNl5zNLYJJvOHDWfCoPz9wNeLLVn4EZVCRXJ9VJng+o/CH6FM//61H2BuxdqeWFucv4crL3Q87F+70u/V/teqNQWkhPNslCL+xmLdZQEilgloATcMxYe+NVs2j59ACY9ABO7QUyUpaGVLOTH6J4N+OKBegQH+LD7WBz3j17O8z+v5WRcoqWxSRZtnwk4ILSOKVrkKuF3mAqhB1bBid2uu4+IZM+B1SZ5s3lA4yeyNkZoXZM8Jpw2f+cld9j0O8SfgqAy6bO1qWw207/Y29+0sVr5jTUxiuWUBIpYrXIbeGIZ3DjAbLLfOh0+bwRLRkCydf37bDYbHWuFMue5FnRvXAaAX1bt5+aPFjB5zX7yWXcZ97cltSqoi5aCpvIPhnI3mWMtCRXJvZZ8ap5r3Q2FwrI2ht0DKrQ0x1oSmnuk9gas1zPjaq+FwqDNm+Z4zpuWf/As1lASKJIbeBeANoPg8cWmgff5OJj1utkvuH+lpaEF+nrxbuda/Nq3CVVC/DkRl8izk9bSc8wK9h1X01m3cP5c+hs0VyeBcFGVUC0JFcmVTuxOb+XS9OnsjVXxZvOsJDB3OLLFFOeyeUDEA1e+rsHDENYYEs/An8+aIkGSrygJFMlNgqtDr+lw+wjwLQTR600F0WnPmaUdFqpftgh/PnUTz7ergrennUXbj9Fu+AJGLdjJ+WTtKcjVds2HpHMQFAYlarn+ftU7mVnt6PUqPCCSGy393OxHr9QGStTM3lipyw0PrIJzJ7Mfm2RPakGYKh0gsOSVr7PbzXsND2/YMRvW/5wz8UmuoSRQJLex280SjidXQu37AAf8+zV8doPpv2bhp3XennaebF2ZGf1vokmFosSfT+F/f22h04jFREbFWBaXXMPFDeJtNtffr0CR9IIRKhAjkrvEHYM1P5jj7M4CgtljXKyqSSp3Lcj+eJJ15+MvKgjT69rXF68CLV40x3+9ZP5sSL6hJFAkt/IvDnd+CT3/gKKV4Ew0/PIQjL/b8oIbFYr7M6FPI4beXZtCBbzYcjiWLiP/4c0/NnImwbp9jJKBlBTYOsMcu7Iq6H/VuFAl1OIPLkTkP1aMNisDQutC+ebOGTP1Qx8tCbXW5qlmNjawNFS6+fpe0+wZCKkJ506YRFDyDSWBIrldhRbw+D/Q8pULyzbmwMjGsPBDSLKuUqfNZuOeBmHMHdCCLhGlcDhg7JI9tP1oAbM3RVsWl/zHgVUQd8T0pSx7Y87dt9ot4OEDx7ZB9Macu6+IXFniWVjxlTlu1t95KwNSE46d8/Shj5XSCsL0yLggTEY8vMyyUJsdNvyS/qGh5HlKAkXcgZcvtHwZ+i41n9wmxcPf78CXN8HepZaGVtTfh4+71mXcww0pU6QAh07F0+f7lTw+bhXRp+MtjU2ArdPMc6U24Omdc/f1DYLKbc2xloSK5A6R482MT+FyUP12541btqn5kPLUPji+03njyvU7th32LjbJXET3zL22VD1o0s8cTxsA8aedH5/kOkoCRdxJsUpmeWiXL6FAUTi6Bb7tAL8/CWdPWBraTZWLM/OZ5jzeoiIedhszNh6mzbAFjFu2l5QUfTJsma1/medqt+b8vWtqSahIrpGcZFoPATR5Ejw8nTe2d0Eo09gc75zrvHHl+qXOAlZul7VesC1fhcLlTc/iOW86MzLJpZQEirgbmw3q3GcKx9Trac6tGQefNYDIiZa+2fbz9uDljtWY+uSN1AkrRGxCEgOnbKDzyH9YvF0bznPc8Z3mgwK7p5kJzGlVOoBXATi5Bw6uyfn7i0i6zb9DzF7wKwJ1r9I6IKu0L9A6SQkQOcEcX09BmIx4F4DbL/SOXPkN7F3ilNAk91ISKOKuChQx6/h7z4Di1eHscZjyOEzqbuleQYDwkoH81rcpb3YKp6C3B+v2n6L7N8vp9tUyVu1VCfEckzoLWLYZ+BXK+ft7FzSJIJjZQBGxhsMB/3xijhs9Zt7wO1tqv8Ddiyz/Pyjf2TzVLPMNKAmV2mZ9nPLN0z9c/uMpU21U8iwlgSLurmwTeGwh3DzIFOLY8if81gdSki0Ny8Nuo1ez8sx7oSW9mpbD28PO0l3HueuLJTzy3b9sPqQ9By6X1hoiBxrEX0nqktCNk02lUhHJebsXwqG14OkHN/RxzT1CakLB4nA+DvavcM09JGOpvQHr9cj+Mt+274B/CTi+Axa8n/3YJNdSEiiSF3h6w00D4L4JYPeCTVNg6tO54k13cIAvb95eg7+fb8G9DUpjt8GczUfo+Mkinp64ht3H4qwOMW86ewL2XSgalJOtIf6rUlvwDjD7TPTGUHIjiz8wyxGps4AR3aFgUdfcw26HChcax+/QvsAcc3ynSfKxQUSP7I/nVwhuHWaO//nEfHggeZKSQJG8pHIbuOtrUx1szQ8w89VcU5CjdOECfHB3HWYPaMFttUMB+GPtQdp8tICXf13HwZhzFkeYx2ybaZo3h9SEwmWti8PLN70ojZaESm6z9S94NwTm5+EZj8PrTbEWmz29AqSraF9gzkudBazUBgqFOWfM6rdBeGdwJJvCc8nq/5sXKQkUyWtqdIbbPzPHy7+A+UMsDee/Khb357P76zHt6RtpXS2Y5BQHP/4bRcuh83l76iaOnUmwOsS8ITcsBU1V8y7zvHFK/ph1EffgcMDf70HKeZg/OL2wRl7zz4ViH+GdoUh5196r4oWZwENrIU7FwFwuKRHWjDfHWS0IcyW3DAXfQnB4HSwd4dyxJVdQEiiSF0U8AB0/MMcL3k8vC56L1CgZxJheN/DL401oWL4IickpjPlnN80/mMewWVs5de681SG6r/Px6cuxrFwKmqpCS/NmIu4I7FlsdTQixp5FEL0+/es/ns57fz5j9qXPwDd72vX3CyhhVh/ggF3zXX+//G7rNDh7zOzhq9LeuWP7B0OHCx8iz/+f+j/mQUoCRfKqRo9B69fN8azXYeW31sZzBQ3KFWHSo435/qGG1C4dxNnEZEb8vYPmH8xj5PwdnE3UMpRM27PIFGcICIWSEVZHY/ashl9oTK3G8ZJbLP3cPDd4yMySpZw31ZXz0pvdZV+YJX3lm+fcvwWps4E75+XM/fKz1N6AEd3Bw8v549fpZpb4JsWbD0lyQZ0BcR4lgSJ52U3PQ9MLn/7++Sys/8XaeK7AZrPRvEpxfu/XjFHd61M52J9T587zwYytNP9gPt8t2UNCkpYRXrct08xz1Y6mr2RuUONCldBNv0OyZnnFYsd2wLYZ5rhxP+gyCko1gHMnYcK9prCSuzt7AlZd2C/WrH/O3TdtX+DcXLMnPU86sevCbKvNVAV1BZsNbhtu+r3uXQyrx7rmPmIJJYEieZnNBm3fNp9044DJj6X3jsuFbDYbHWqWYMYzzfno3jqEFfHj2JkEBv2xkdYfLuDnlVEkJeuTyKtKSUl/c1v1VmtjuVi5m0z5+HMnYdcCq6OR/G75F+a5SgcoVgm8/KDbRAgKM6Xxf+rp/r3uVn5jVgSE1Ezv4ZcTyjQBT1+IPQRHt+TcffOb1d+b54qtoXA5192ncFm4+Q1zPHsQnDrguntJjlISKJLX2WxwyzCodS+kJMFPD+b6N+Eedht31ivN3AEtebdzTYIDfDgQc44XfllH++ELmbbuECkpWfyE+cxR+LkXTHkibxYpORRp3nx5+0P5m6yOJp2HJ4TfYY5VJVSsdPZEehGYxk+kn/cPhvsnmZYmexbBtAHuO5N1Ph6Wf2mOm/XP2RUBXn5Qtpk5VpVQ10g+77qCMBlp+KiZKU84DdOec9+/F3IJJYEi+YHdDp1Hmpmh5ASY2A2i/rU6qmvy9rTTvXFZFrzQildvqUahAl7sPBpHvwmr6fTZYuZtPYIjM/8ZHVoLX7U0jcsjx8O/37gsdsukVgWtdDN4+lgby3+lVgnd8ickqQqsWGTVWDh/FkJqmb1yFwupAfd8e6HNzrj0/nruZu1EiDsKgaWhRpecv79aRbjW1r9Moa2CwTlT/MvuAXd8ZvoQb/tLe7vzCCWBIvmFhxfcPQbKtzBLhMbfBYc3WB3VdfHz9uDR5hVZ9GIr+t9cGX8fTzYePE3vb//l3i+XsnzX8WsPsv4X+KY9nN5vKlUC/P0OxEa7NPYcl7rcNze0hvivsMYQUNJ8mrxjjtXRSH6UlAgrvjLHTZ7IeIasclvo8D9zPOdN2Dw1x8JzipTk9IrQTfq5pmDItaQmgXv+MbOS4lxpBWEeyLnf3+Dq0Px5czz9xbyxbzafUxIokp94+cJ9E6B0Q4g/BeO6mAIJbiLA14tn21Zh4YuteLR5BXw87fy75yRdv1pGzzErWL//1OUvSkk2b+R+fRiSzkGltvD0GlMpL+E0zHotx38Olzm5B6I3gM0DKrezOprL2e3psxIb9EmyWGDTFLNcumBw+sx0Rho9Bjf0ARzwax84uCanIsy+LdPgxE7zYVe9ntbEEFzdVCdOOgf7lloTQ151ck/6DGtO//7eOACCw01bihmv5Oy9xemUBIrkNz7+8MDPZilU3BH4/g6IibI6qkwpUtCbV2+pzoIXWvFAozJ42m0s3HaUTp8tpu8Pq9geHWsujD9llr4u/th83ay/2fNToAjc+hFgg/U/5/o9ktdt64WCMGWamJ8xN6p5oUro1r8g8ay1sUj+4nCkt4Vo+Oi1l0t3+B9UamMSmQn3uUdBDIcjfQnrDY+Yf++tYLNpSairrB4HOEz/1SIVcvbent5w+wjABut+hO1a0eHOlASK5Ed+haDHZCha2SyP/P4OOHPE6qgyrUSQL+91qcXc51rQJaIUNhv8teEw7YcvZMi4Pzn/ZWvYPtNUqrvza1Mp1e5hXlyqnnmTBGaju7tXAgTTOBigWi5cCpqqVH0oVMYsSd4+0+poJD/Zt9QUTvL0vVAx+Ro8POHub83Mx5nDMLErJJxxeZjZsm8pHFgJHj5mNtNKaUmg+gU6TfJ5WPODOc6JgjAZKd0AGvc1x38+Awmx1sQh2aYkUCS/8i8OPaeYkugndsL3nd12jX/ZogX5uGtdZvRvTvsaIdxkW0u/HX3wOrmDU17BnOj6B9S+5/IXtn7dtC04vh2WfJrzgTvTuZNm/w3kTKGArLLZ0nsGqkqo5KTUWcDaXaFg0et7jW+gWT1QsDgcXg+/9cndVYVTZwHrdjPVTq1UoaV5jl6f9/ZeW2XbTPOBRIFi1rYAav26+TDvVBTMfce6OCRblASK5GdBpaHn7+AfAkc2wvh73PpTvaoh/nxZYQljfYYSaDvLypQqtIl9k6bfn2DIX5s5Gfef2T6/QtDuPXO8cKjZa+Guts8BRzIUr57zS4QyK3Uv1vbZEH/a2lgkfzixy+yVg0vbQlyPQmXgvolmdm3rdJj9hvPjc4Yjmy/0CLVBk6esjgYKFoPQOuZ4l2YDneLigjCe3tbF4V0QOl344HTFV7BvuXWxSJYpCRTJ74pWhB5TwK+wWUY0sZt7VnM7fw4mPwazB2JzpEBED5K6/06ZsuWJP5/Clwt20fyDeXw6dztnEpLSX1f7XtPIPCke/nrJffsfpbaGyM2zgKlK1IKilcyveWo1UxFXWv4l4DB7/IKrZf71YTdAlwsN5pd+Biu/dWp4TpFaEbT6bVCskrWxpEptUq99gdkXE5VeVbneg9bGAlCxFdTtDjjgjyfd831DPqckUEQgJBy6/2oajO9ZZJqpJ5+3Oqrrd/ogfNsR1k0ylTFv+RBuH0HjKiX55fEmjOnVgOqhgcQmJPHR7G00/2Aen/29nT3H4szyxFuHXeh/NCM9mXInSYnpbw6qWbhE6HrZbOmzgeo3Ja52LuZCMQ0yPwt4sZp3QasL1YSnPZe79rqdPgjrfjLHzZ6xNJRLXLwvMCXF2ljc3ZoLBWHKNzcf3uYG7d81lXaPbYNFH1odjWSSkkARMUrVh24/mqIJ2/6CyY/n7r0vqaJWmAbwB9eAXxGzz7Fhn7T+XzabjdbVQpj21I2M6BZB+WIFORGXyIezttHyw/nc8skiRqyzc6LuhSIKf70EiXGW/ThZsnexaXfhHwIl61kdzfVJ3Re4Y67ZzyjiKqu/N4WIildPT0qyqvkLZk+hIxl+ehCObnVOjNm17AtIOQ9lmprCHblFWEPwKmgqUR/ZaHU07is5Kf2DjNwwC5jKrzDceiH5W/yx2/QeFkNJoIikK38T3DsO7J6w4Rf489ncvTxy9TgYeyuciYbgGvDoPPMpaQbsdhud6pRk9rPNGXZPHW6qXAwPu41Nh04zbPY2mi6pz2FbMJyK4vj0d3Hk5p/7v7ZcmL2s0sH04nMHwdXM71nKedj8p9XRSF6VnHRhKShXbg6fGTabKZFfpgkknIIJ90Lc8ezHmR3xp9KXpzbrb20s/+XpA+VuNMc75lobizvbMRtiD5oPOqt3sjqaS4XfAdVug5Qksyw0Oenar5FcwU3eLYhIjqnSDu4cDTY7rP4OZr2e+xLB5PNmxu6PJyE50fyn+PAsKFzumi/19LBzV/3SjHu4EStfa8MHd9WmZdXiJHv48npCDwAC13zJQ0PHMXTmFjYcOJW7E0KHI31fXdVc3BoiIzVVJVRcbPPvpg1OgWJQ617njOnpA13Hm39vTu6BH++HpATnjJ0VK7+FxFgoXg0qt7MujiuppH2B2ZZaEKbu/dfub2mFWz4EnyCzImf5F1ZHI9dJSaCIXK7mnemVv5Z+Zipn5hZnT8APd8LyUebrlq/CPd9nqSly4YLe3HtDGGN7N2Tl62255e6HWFOgKV62ZB47M5LP5+3gthGLaTF0PkP+2kxkVEzuSwgPrzNvcr0KQIUWVkeTOalJ4O6FcOaotbHkJYlnzdKs8ffC8Z1WR2OtpSPN8w2PgJev88YtWBTu/9m88Y1aBn88Zc2HZUkJ6f8WNn06d64ESF2Cu2+p+bMpmXNqP2yfZY6t6g14LYGhZn8gwN/vmWq8kuvlwn8tRCRXqNcD2g8xx/PeM3tOrBa90ez/273QFLHpOh5avuSUNz5Bfl7cWa80EX1G4fD0o7F9M4PKrMfXy86+E2f5csEuOn/+Dze+P493/tzEyj0nSEnJBQlh6ixgxdbg5WdtLJlVpAKUjDD7qzb/bnU07i/5vJkVGlEP5rwJ22dal5zkBlErLjRO94YbHnb++MWrwL3fmWJU6yZZ82HZ+p8h9hAEhEKtDHqh5gZFK5l+tMmJsHeJ1dG4nzU/gCMFyt4IxSpbHc2VRfQw2zGSzsHU/vn33x03oiRQRK6syRNmpg1gxsvpG9OtsHkqfN0WYvZCobLw8GxTCt3ZCpfF1uJFAHrHfcPq529g5AP1uK12KAW8PTgQc45vFu/m7lFLafK/uQz6fQPLdh0n2aqEMLX3mbstBU2V1jh+srVxuDOHAzZOhpGN4c9nTFIQVMYUedr7D2zKpwn20s/Mc617Xdc4vWIrU10YzIdlObm0OSUF/rmwYqNxX2v7xl2NzWZ+nQB2al9gpqQkp/+/m1tnAVPZbNDpE/D0Mx/UrrHw/YJcFyWBInJ1LV6EJk+a46lPmzebOSklBeb/DyZ1NxX+yreAR+ebthau0uRJKFYF4o5SYNFgbqkVymf312P1wLZ81aM+XSJKEeDjSfTpBL5bupf7vlpGo8FzeHXyehZvP0ZScg6VQj+13ywHtdmhSvucuaez1ehinvf+A6cPWRuLO9o5z8yO/9wLju8we986vA9PrUwvEjJ7YP7r4XVyr/ngCMyHWa7UoHf6v5GT+0LUv669X6rtM+HYVvAJzP0JQlqrCO0LzJQdc81yf7/Cua8gTEaKVIDWF9qozHxd/6bnckoCReTqbDZo9y7U62mWpPzaB7bNypl7J5yBn3vC/AvLUhv1he6/QYEirr2vp3f6p/srx8CBVQD4ennQrkYJPu5al5UD2zCmVwPuqV+aID8vjp1JZMLyfXT/Zjk3vDeHl35Zx/ytR0hMcmFCmLoUNKwRFCzmuvu4UqEwEz8O2DTF6mjcx8E18P0dMK4zHIo0y6NbvAz9I6Hx46Z4RLP+EFASYvalz4rlF8u/NP9eVWgJITVcf7+2b5vZ+OQE+LGbSUJdLXUWsEFv8A1y/f2yo3wL82HV0S1w6oDV0biP1IIwde537p5WV2rU17QqSjgF05+3Ohq5CiWBInJtNhvcNtw0S045Dz/1gD2LXXvPE7vhm3bm03wPb7jjc+j4P/DwdO19U5VvbvqB4YA/B1zWM9HH04PW1UIYek8dVr7ehu8faki3hmEUKejNybPnmbQyil7f/kuDd2cz4KdI5myKJv68k/supja2r9rRuePmtNTG8aoSem3Hdpj+dF+1hF3zwe4FjR6HpyOh1SvgE5B+rXdBaPuWOV70Uf75VD7+tOkNCOkzdK5m9zBVlUvUgrijMKGrad3gKlErYN+SC7//fV13H2cpUCS9h+muedbG4i5OH4RtM8xx/VzUG/BaPDxNGxW7J2z5M/8uR3cDSgJF5PrYPaDLl6YXXVI8TLgvbYbM6XYtgNGtTHNh/xDoNQ0iurvmXlfT7l1T/e9QpJkRvAIvDzvNqxRnyJ21WfHqzUzo04gejctSPMCH0/FJ/Lb6AI98v5IG786h/49rmLHhEOcSs5kQxp+C3YvMcdVbszeW1cLvAGyw/9+cmUFxR6cPmWILnze8MGNqg9r3wVOroOP74F8849fVugdK32CWUs99Kycjts6aH0zLhGJVoOLNOXdfH3/oNskUaTm6GX7u7bqeaf98Yp5rdzWVGd1B6pJQ9Qu8PmvGm6JZZZpA8apWR5M5JWrCjc+a42nPm6rekusoCRSR6+fhBfeMhXI3mTdZP9wF0ZucN77DYZZxjesC506aT44fnQ9hDZ13j8zwD4abB5rjue9AbPQ1X+LpYadpxWK807kmy165mZ8fb0LvZuUoEejLmYQkfo88yOM/rKbeO7PpN341f647SFxCFt4o7phrZmWLVoZilTL/+twkoER6Q+mc3nOa2507CbMHwacRZmmYI9l8ENP3H7jzSyhc9uqvt9nMHkGAtRNhv4s+uMktUpLT+5Q17pvzLROCSkG3iaZly865pqCWsx3bkV4QqulTzh/fVVKTwF3zLltZIf+Rkpw+m53b93teSfMXLuytPwKzBlodjWRASaCIZI6Xn3mTU6qBeYM6rrNzepElJZhy9n+9aN7o1u4KvadDYMnsj50dDR6C0Lpmf8Os1zP1Ug+7jRvKFWFQpxosebk1vz3RlD43ladUIT/OnU9m2vpDPDlhDfXemc2j369kxobD19+HMHUpaDU3rQr6X6k9Azf+Zm0cuUVqr79P6sA/w03Z9bDG0HsG3D8pc/vcSteHOt3M8YyX83bp9i1/mj2QfkXMTKkVSkbAnV8BNvh3tPlgy5mWjgAcUKUjBFdz7tiuVLqBKWJz7iQcWmt1NLnbznlwap/Z6xl+h9XRZI2nD9z+GWCDyB9UFCgXUhIoIpnnEwAP/AzBNeBMNHzfOXub/WOj4btOpqS0zW6WYXb5Mnf0vbN7wG0fAzZY/5MpfZ2VYew26pUpzGu3hrP4pVZMffJG+rasSLmiBUhISmHWpmge/2EVj3y3kujT16jkmHw+vXmwu7aG+K/qd5h+a4fW5u8G58lJl/b6iz8FweHQ7Ud4aAaUbZK1cW8eBF4FYf8KWP+LU0POVVKbwzd4CLwLWBdH9U7Q5k1zPONl5xXTio2GyInmOLX6q7vw8DJ7rUEJwbWs+tY81+mWO/4fzKoyjaDho+Z4an9IjLM2HrmEkkARyZoCRaDHZChS0Xxi+f0dcOZo5sc5sNoUuYhabvbfPfCzWeJkszk95CwrVS+92fS05yApMVvD2Ww2apUO4qUO1Zj3fEv+6n8Tj7WogLeHnblbjtD2owX8tnr/lWcF9y4xyUGBYma/V15QsKip5AiwIR/OBjocsHEKjGx0aa+/zqPg8cWm+E92/k4EhsJNF/bozH4jb74Z278KopaZYikN+1gdjUnSIrqbKqW/PATRG7M/5oovTQXS0jdAmcbZHy+npfULVHGYK4o9nF4Qpp4bFYS5kpvfgKAwM0P/93tWRyMXURIoIlkXEAI9f4fA0nB8O/zQBc7FXP/r1/0E33aE2INm70Cfv6FSG5eFmy2tB0LB4nBs24XlWM5hs9moHhrIKx2rM/WpG6lVKojT8UkM+Gktfb5fxZGMZgVTW0NU6WBmKvOK/FoldNd8Uwjp5wcv9PorCh3+Z3r91e3mvN/jJk+axDL2YHphkbxk2efmudbdZp+p1Ww2uPXj9D3UE7pe177iK0qIhX+/NsfN+ueuD8quV+q+wKhl5ueRy0WOh5Qk0zrHlf1wc4qPv6kuDrBsJOxfaWk4kk5JoIhkT6EwkwgWLA6H18OEe689y5CSbDaK/9bHVBqt3B4emZO7C5z4FYJ2Fz7FXDDUJVUsq5YI4LcnmvJ8uyp4ediYszmath8vZMqaA+mzgg4HbL1QFCKv7AdMVe1W0w7k6GbnFhzKrVJ7/X1/hzlO7fX3dKQpauLp49z7eflBu3fM8T+fQEyUc8e30qn9ZiYVoLGLm8Nnhqc3dB0HRSvBqSjTQ/D8uayNtXqcWQFQtJL7LgMvUgEKlzdJjqvbDLmjlBRY9Z05dteCMBmp3ObCHl0H/P5ktlfTiHMoCRSR7CtWCXpMMZvYo5bDj/fD+SvsazsXYxLFJRcaHd84wBSaye3NjgFq3wtlbzRFOv56ySW38PKw82Trykx96kZqlgrk1LnzPDMpksfGreJIbDwc2WSW1Xj6pi+fzCv8CqXPBOflAjHX6vXnG+i6e4ffAWWbmQ9f5gxy3X1y2vIvTUGpcjdBaG2ro7mUX2G4/yfzfGAVTH7cvNnPjOTzsPTCTGfTp9x7BUDqbKD2BV5u93yI2Wu2RoR3tjoa5+owxGxhOLoZFn9kdTSCkkARcZYSNeGBX03xiV3zzR6Y5POXXnN0G3x9M+yYA55+cPcYaDPIfd7Q2Gxw6zDTBHfbX7BlustuVa1EIJOfaMaAtmZWcNamaNp9vJDN8380F1RoZZqB5zU1LlQJ3fBb3qtiecVefyuv3uvPmWw282YMm1l2u2+Z6+/pagln0mdPmvSzNpYrKVoRuo43Cf+mKTAvk3ujNvwGp/dDwWDrqp46i5LAK1s11jzXvtfawkauUKAI3PKBOV74IRzZbG08oiRQRJwo7AYzq+fhY5Ys/t4v/RPvbTNNAnh8h9lD+NCM9D1g7iS4Wnpvrr9ecmmBDS8PO0/fXJk/nryR8NBAYs6eJ2HjnwDElm3rsvtaqmpH8wHBiZ15p4z8uZOm0ufFvf4qtzcFX+78EgqXy9l4QutAvR7m+K+XMj8rldtETjAtXIpUNL+uuVW5ZnD7hRUQiz5Mr/J5LQ5H+h7ORo+Bl69r4ssp5W8ylYCP73DJsnq3deZIev/H+nmgIExGatxpWpuknDfLQtUv0lJKAkXEuSq0gHu/M7Nl6ybB9Odg0UemKELCaSjTxDSAL1nX6kizrvkLpsDGqX2wcKjLb1c9NJDfn2zGazcVoq59FykOG53nBPDnuoMuv3eO8/GHKu3MsbsvCT1/DhYPh0/qmp5/SedMsYfef8EDP5nZc6u0HgjeAXAoEtZOsC6O7EpJNsUmwJrm8JlV936zBB5MX9S9S679mh1z4chGs8oitUqxO/MNgrCG5lizgelSC8KUagAlalkdjWukrqbxCYQDK2HFV1ZHlK/l8n8tRcQtVe1o+vxhg5VjYO5bgMNsdO/5R84se3Ml74Jm+R7AkhFwZIvLb+nlYadPyFYAtnpWYee5gjw5YQ1PjF/FsTMJLr9/jkqrEjrZPZeEJieZGb9PI8y+u/gYKF4d7psID82Esk2tjhD8g6HFi+Z47tvuW6lx2ww4uRt8C5kEyx20Hmj2Zqachx8fuHZfzCUXZgHr9zL7CvMCLQm9VF4tCJORoFLQ9i1zPPdtzQZbSEmgiLhGrbuh03BzbPeEWz+CTp+Yanl5QbVbLixrSYLpz+dMsnJhD2Ll5l3pf3NlPO02pq8/TLuPFzJt3SHX3z+nVG5nKmWe2ude5cQv7vU3tf+FXn9h0PkL6PuP+TOTm8r6N3rcVGs8Ew2LhlkdTdakNoev38t99sja7ab/Y8l6cO6EKZR17mTG1x5YDbsXmuWTjfvmbJyulJoE7l5gPjTJ7/YsMh9meAdAzTutjsb16vUyRdZK1DL/h4olckUS+Pnnn1OuXDl8fX1p1KgRK1asuK7X/fjjj9hsNjp37uzaAEUka+r3godmmb1PeWEZ0391fN/sX9uzyPQ8dKWEM+YNE+BZ/VaebVuFKf2aUa1EACfiEuk3YTX9JqzmeF6YFfTySy+B7y49AzPq9dd+CDy1ysxQ5cbiR57e6W1Pln4OJ3ZbG09mHYyEvYvNh0wNH7U6mszxLgDdfrzQY3UH/NTz8kJakF5Fudbdph1PXlEywiwLjT8FB1dbHY31LikI4yYfZmSH3W5ap/SeYYomiSUsTwInTZrEgAEDGDRoEKtXr6ZOnTq0b9+eI0eOXPV1e/bs4fnnn+emm27KoUhFJEvKNILg6lZH4RqFy0KLF8zxrNdM+wtX2TkXkhPNzE3xqgDULBXEH0/eyNOtK+FhtzFt3SHafbyQv9bngVnB1E/DN07OvcUDkhJM8v912/Ref14FocVLpt1Dkyec3+vP2ap2NK1GkhNh9kCro8mc1L2ANbqYJWbuJiAE7p9kZr13L4RpAy5dUXBiN2z63Rw3fdqaGF3F7pHe4ia/LwmNOwabp5rjvL4U9GIFiuT+Pbx5nOW/+h999BF9+vShd+/ehIeHM2rUKAoUKMCYMWOu+Jrk5GQeeOAB3nrrLSpUqJCD0YqI/EeTp6BYFYg7Cn+/47r7bP3LPFe9dEmht6edAe2qMuWJZlQNCeB4XCJ9x6/mqYlrOBHnxg15K7Y2MwVnDsO+pVZHc6mTe021z4+qw299YP8KU/q/4WPQPxJaveraXn/OZLOZGUub3bwR3b3Q6oiuz+mD6bPEuak5fGaVqGla5djssPp7s8c41dLPwZFiemdaWUTIVSrebJ7zexIYOcHsDy1ZL/f1uJQ8zdIkMDExkVWrVtGmTZu0c3a7nTZt2rB06ZX/03/77bcJDg7m4YevvbwsISGB06dPpz1iY91087uI5E6e3qbaGcC/35g9PM6WnGQKYED6Msn/qFU6iD+easaTrcys4NS1B2n38QJmbDjs/HhygqcPVOtkjnPDktCUFNg+21S5/aSOqfZ59jgElIRWr8GzG0wPLP9gqyPNvJBwaHDh/9MZr+TemdeLrRht9hKVaQql6lkdTfZUaQ/tB5vj2W/A5j/N7NCaH8y5vDYLmKpiK/O8f6VrV1HkZg5H+lLQvNoWQnItS5PAY8eOkZycTEhIyCXnQ0JCOHw44zcuixcv5ptvvmH06NHXdY8hQ4YQFBSU9ggPD8923CIilyjfHGrdCzjgz2ed/yY6arkpHOFX2LQYuAIfTw+eb1+VyU80pXKwP8fOJPL4D6vo/+MaTrrjrGDqktBNv1tXPOLsCdOjbUQEjL/7QjLuMEvZuv4Az6w3VTYDSlgTn7O0etVU2IzeAKu/szqaq0s8C6u+NcdN3HgW8GKNHr+QiDvM7PL0501LkdC65t+XvKhQGSha2fTNdJcZaGfbs9j0RPX2d8++ueLWLF8OmhmxsbH06NGD0aNHU6xYset6zSuvvMKpU6fSHps2bXJxlCKSL7V7F3yCTN+1lVdezp4lW01VUKp0AA/Pa15eu3Qh/nz6Rp5oWRG7DX6PPEjbjxcya6ObzQqWb2EKrJw9nlYUJ0c4HLB/FUzuC8OqmdmZk3vM72/jJ+DJldDzd6je6bp+P9xCgSLQ8hVz/Pe7uXtmZu1E86FI4XJXnBl3OzYbdPzALIM+f9bshQVo1j93VZR1tvzeKiJ1FrDW3eATYGkokv9YmgQWK1YMDw8PoqOjLzkfHR1NiRKXf6q6c+dO9uzZQ6dOnfD09MTT05Pvv/+eP/74A09PT3buvLzXjo+PD4GBgWmPgAD9JRMRFwgIgZsvFNaY+w6cuXpxq+vmcMCWaea4asfrfpmPpwcvdqjGb080o1KwP8fOJPDouFU8OymSmLNuMivo4Wn6qUHONI5PPAurx8FXLeHr1qaJenIClKgNt4+A5zZDhyFQrLLrY7HCDQ9Dsaom6V441OpoMpaSkl4QplHf3Fl1Nas8POGesVC8mvm6cDmofruVEblepdR9gXPdsydodsQdh81/mOP8VBBGcg1Lk0Bvb2/q16/P3Llz086lpKQwd+5cmjRpctn11apVY/369URGRqY9br/9dlq1akVkZCRhYXmofLKIuJ8GD0FoHUg4BbNed86YR7ea/lEe3umFFDKhblgh/nzqRh5vYWYFJ685QNuPFzJnU/S1X5wb1LiwJHTzVEhyUfJ6fCfMeBU+qgZ/PGlmcz28ofZ98PAceGwh1OuZ90u3e3il701bPgqObbc2nozsmG1aKvgEQsQDVkfjfL5B8MDP5s/e7SPyzkzzlZRtZooqxeyDE7usjiZnrfvRVOUNrWNaZojkMMv/dRkwYAAPPvggDRo0oGHDhgwfPpy4uDh69+4NQM+ePSlVqhRDhgzB19eXmjUvrZBVqFAhgMvOi4jkOLsH3PYxjL4Z1k2CiB5QPpttbFKXgpZvAT7+WRrC18uDlztWo12NEF74eS07j8bxyPcrubNeKQbdVoOgAl7Zi9GVyjYF/xKmSujOv6FqB+eMm5wE22fCv19fuhStUBmzNyuiOxS8vm0HeUrlNlC5vfm1mfkaPODi/peZtfRz81yvZ95dPleoDNz5pdVR5AwffyjT2PRa3fl3/ukZd0lBmF5WRiL5mOV7Art27cqHH37IG2+8Qd26dYmMjGTGjBlpxWL27dvHoUN5oOeViOQPpeqbGUGAac9lf/YqNQmslv29T/XKFGba0zfxWPMK2Gzw2+oDtBu+gL+35OJZQbsH1Ohsjp1RJfTMEbPU8ZM68OP9FxJAG1RuB/f/bPr73fhM/kwAU7V/zzRg3z4TdsyxOpp0h9ebvaE2D2j0mNXRiLOkVgnNT/sC9y2FY9tMX9Gad1sdjeRTlieBAE8++SR79+4lISGB5cuX06hRevW7+fPnM3bs2Cu+duzYsUyZMsX1QYqIXK+bB0LB4nBsKyz9LOvj/L+9+45vql4fOP5J0ibdmy7oYBYQKBsKLoYUUDaKyBVQQb2AVy5yL+pPKIiKoiLXBS5QFERRwMGSLSKyN1gByyh0UKB7pE3O74/TBkInpW06nvfrlVdOzvmec56cHkKefFdagjp8OkCzsvcHLImDvY4X+rfgu6e70cjHmYTUHB7/fB9TVxwmJSu3Qs5R4QpGzYteC7lZt76/osC53+G7x2FeS3Xgk9RYcPRSB97410G1CV6zPrWrj1l5+TRV5zwEtZmsqZrcF38sUJ9bDlRry0TtUNDMPebX6nOvVTbLgDDDas6coqKQDz74gNDQUBwcHOjSpQt79uwptuwnn3zCXXfdhaenJ56envTu3bvE8lWhWiSBQghRqzh6qqOFAmyfq/Z3KY9TGwBFnUTYLaDCwgPoEOLJ2mfvYvxdDdFo4Lv9sUS+8ytboytoQJuK1KATuAeBMR1O/VL2/XLS1LkbF3SHxf3UmkRzrnq8IR/BlJNw38vg1bDyYq+p7vmvOjJrUnTFj3ZbHmkJcHSFutx1om1jERXLv416rxnTIXavraOpfJlX4fhqdbn9WFtGIm7DN998w5QpU4iKiuLAgQOEh4cTGRlJYmLR/4du27aNkSNHsnXrVnbt2kVQUBB9+vTh4sWLVRz5dZIECiFEZWgzQh30IC8L1k0r3zH+zG8KWknD4DvY6/i/+1uy4qkIGvo4E5+azWOL9/Lf7w6Tml2NfpHXaOCOIerysTKMEpp4EtZMhbdbwJopkHgc7BzVfmRPbodxmyD8YbB3qNy4azJHD+jxf+ry1tfUL662tPdTdRCNBp0hqJNtYxEVS6uFRnWoSeiRb9RRh/1aQ/32to5GlNO8efMYP348jz32GC1btmThwoU4OTmxaFHRP5otXbqUCRMm0LZtW5o3b86nn35qGQzTViQJFEKIyqDRwP1vq32rotdeT+jKypgJf29VlyugP2BJOoZ6sfZfd/HEnWqt4Lf71FrB7X9drtTz3pKCieP/2gA56YW35xnVBHHx/fBhV9j7CRjTwLsJ9H1dnd5h4HsQ2LZKw67ROowFv1aQnawmgraSmwX7PlOXa8vk8MJawXyBp233hbhKWA0IM6Z2zwFZA6WlpZGammp55OTkFFnOaDSyf/9+evfubVmn1Wrp3bs3u3btKtO5MjMzyc3NxcvLq0JiLw9JAoUQorL4toCISeryumlgzCj7vn9vhbxste+Tb8vKie8Gjnod0x9oyTdPRhDq7URcSjZjFu1h4rIDfPZbDFv/TORsUgZ5JnOlx1KkgLbg1UitWf1r/fX1KRdhy6swvxV89xic+w00Wmj+ADy6Wp3Yves/1Sa64tZodeq8iKA2CU08aZs4jnyrzl3oHgzNB9gmBlG5CpLASwdtX+tcmS7sgct/qi0T2jxk62jETVq2bIm7u7vlMWfOnCLLJSUlYTKZLINYFvDz8yM+Pr5M55o2bRqBgYFWiWRVs/kUEdWVyWQiN7caNYcSFcbe3h6dTgZ/EFXknv+qfdFSzsOvb0HvqLLtVzAqaNj9VfprceeGXqx79m7mbviTxTvPsuZIHGuOXB+h2U6rIdjbiUY+zjT0caahjwsNfZxpVM8ZX1cDmsqKVaNR5wzc8RYc/U4dvXPvp2oNq2JSyzj7qrVXHcaAe4PKiaOuaXi3mlD/+TOsfwEeXVW1tReKcsPk8E/V/nnz6iq3APXHrsQT8Pe26zX/tU1BLWCrYeqckKJaOXHiBPXr17e8NhgMlXKe119/neXLl7Nt2zYcHGzXLUE+TW+iKArx8fEkJyfbOhRRiTw8PPD396+8L6xCFNA7Q7831OkIfn9P7YtWL6zkfcwmiM6v7QqrmFFBb4WjXkfUgDsYEB7IphMJnL2Swd+XMzh7JYPsXDN/X1Zf38xJr8tPDJ3VJLHe9STR3bEC5iJsNUxNAv9apz4KhHSHTuPUZMVOf/vnEdb6vKIOyPP3VrUWtirvyTOb1ZoTvQu0f7TqziuqXuOeahJ4ZkvtTAKzrsHx/D7NMjdgteTq6oqbW+mjtfr4+KDT6UhIsJ5eKSEhAX9//xL3feutt3j99dfZtGkTbdq0ua14b5ckgTcpSAB9fX1xcnKSJKGWURSFzMxMy+hNAQEVO+KiEEUK6w/N+qpfoNc8B2N+Krk2JXYfZCapvxSHdKu6OG/SPtiT9sHXm1GazQrxqdnEJGXwd1IGMZcziElKJyYpgwvXssg0mjh+KZXjl1ILHcvbWW9JEBvWy08SfVwI8XbCwb6MNfN+LdXBFBKOqklB+MPqxO5+ld9ctk7zaghdJ8DO+eoE8o17VV2yXTA5fLtHpeaktmvcQ51S58wWtQa4tn3/OrJCbeLvewc06GjraMRt0Ov1dOjQgc2bNzN48GAAyyAvkyZNKna/uXPn8uqrr7JhwwY6drT9PSBJ4A1MJpMlAfT29rZ1OKKSODo6ApCYmIivr680DRWVT6NRawP/3g5nd6hD3ZfUHyR6jfrctA/oKqAGrYJotRoCPRwJ9HCkexPrydSNeWYuXMvMTwzzk8T8BDEhNYcrGUauZBjZd+6a1X4aDQS6O9KonvP1JNHHmUY+LtT3dESnvemL4Mhlar+hxj3B4FrZb1kUuHsqHP4arp6BPR9Bt2cq/5yJJ9WEQKOVyeHrguBuoDNA6kV1IvXSWkzUJFYDwoytfQluHTRlyhTGjBlDx44d6dy5M/PnzycjI4PHHnsMgNGjR1O/fn1Lv8I33niDGTNmsGzZMkJDQy19B11cXHBxcbHJe5Ak8AYFfQCdnJxsHImobAV/49zcXEkCRdXwDFW/SG+ZDRteVBM8R4+iy0bnN3W0QVPQ8tLbaWlcz4XG9Qr/Z5aek8fZJDU5LHj8nZTB35fTScvO42JyFheTs9hxKsn6mDotwd5O15uX5j+C6/fBx85A9UmP6wCDK/SaAT9MVOe+bPMwuNSr3HMW9AVsfr/M5VgX6J3Ulg9/b1WT/9qUBMbuy5+qxkEGhKklRowYweXLl5kxYwbx8fG0bduW9evXWwaLOX/+PFrt9fE3FyxYgNFoZPjw4VbHiYqKYubMmVUZuoUkgUWQJqC1n/yNhU10ewYOL4crp2DLK3D/W4XLJJ1WfwXX2kMT240aVpFcDHa0qu9Oq/rWzfkUReFqhvGGmsMMS01izJUMjHlmTiemczqxiCkhAE8ne3xcDOrD1YCPix4fFwP1XAzUcy1Yr8fb2YDeTgbDvm3hj8CeTyDukPpjxsB3K+9c6Zfh8DfqckTxzatELdO45/UksOs/bR1NxSmoBbxjSPE//okaZ9KkScU2/9y2bZvV67Nnz1Z+QLdIkkAhhKgqdgZ17sAlA9WRLduNgsB21mUKRgUNvbPW94HSaDR4uxjwdjHQMdR6riSTWSEuJet6zeHl67WIF5OzMJkVrmXmci0zl1PFJIk3cne0tySJPq43Jor6QomkwU5aBxRJq1XnXFzcFw4sUQfjCaikgQ32LVIn1A5sD0FdKuccovpp3BM2Toezv0FejvqZWdNlp6gjRIMMCCOqFUkChRCiKjW6B1o/qPYL/PnfMG6zOh9bgYIksPn9tomvmtBpNTTwdKKBpxN3NbVudmg2K1zLNJKUbiQpPYfLaTnqc3oOSWnquoLHlXQjeWaFlKxcUrJyOVPEqKY3c3Wws9Qk1nMxWCWPPvmvC7aXeVCb2iIkQp2q4/hKdcqIsT9XfP+m3GzY+4m6HDFR+k/VJX53gIsfpCfA+T/Uz8ua7ugKdX7Tes3lBw1RrUgSKAoJDQ1l8uTJTJ48+baPtW3bNnr06MG1a9fw8PC47eMJUSv0eRX+2qAOcLJ/sVqjApCRBBd2q8vN+touvmpOq71egxhGyYPDmM0KyVm5alKYlp8o5ieP11+ryeOVjBxyTQpp2XmkZecVOQ3GzVwNdvi4GgjycuKxbqHcG1av9jc3v+9l9ceKc7/ByR+h5aCKPf6x7yDjMrjVr/hji+pNo1FrAw9/rTYJrelJoKLAvs/VZRkQRlQzkgTWEvfeey9t27Zl/vz5t32svXv34uzsfPtBCSGK5uoHPafDuv/AppehxUBw8VUTQ8UM/m3AI8jWUdYKWq0GL2c9Xs56mvmVnDAqilpjqNYuGvNrFnNuqFm0Th5zTQppOXmk5eQRk5TBr39dpkOIJ8/1aUa3xj4lnqtG8wiC7s/C9jfgl5egaSTYV9CEx4oCu/IHhOn8ZLUaHVdUkRuTwPtm2Tqa23PpgDqljc4AbUbYOhohrEgSWEcoioLJZMLOrvQ/eb16lTzimxACOj0Bh76CuMPwy3QY+tH1pqBh/W0bWx2l0WjwcNLj4aSniW/JZRVFITUrz1KTuOXPRL74/Sz7z13jkU92072JN1P7hNHuhnkWa5Xuz8KBLyH5vDq3291TK+a4MdvVURTtnaDDmIo5pqhZGt2rPscfUQcIquxRaCuTZUCYweDkVVJJIaqcDJdWCkVRyDTm2eShKEqZYhw7dizbt2/nf//7HxqNBo1Gw+eff45Go2HdunV06NABg8HAb7/9xpkzZxg0aBB+fn64uLjQqVMnNm3aZHW80NBQqxpFjUbDp59+ypAhQ3BycqJp06b8+OOP5b6m33//PXfccQcGg4HQ0FDefvttq+0ffvghTZs2xcHBAT8/P6vhdL/77jtat26No6Mj3t7e9O7dm4yM0ptsCVHtaHVw/zuABo4sh1Ob1F++AZpLEljdaTQa3J3saeLrQtdG3rzYvwW//rcHj3YNwV6nYefpKwz58HfGfbGXE5dSbR1uxdM7X6+l2TEPUuMq5riWyeH/AY61NIEWJXPxBf/W6vLfW20by+3IToWjMiCMqL6kJrAUWbkmWs7YYJNzn3g5Eid96X+i//3vf/z111+0atWKl19+GYDjx48D8Pzzz/PWW2/RqFEjPD09uXDhAv379+fVV1/FYDCwZMkSBgwYQHR0NMHBwcWeY9asWcydO5c333yT9957j1GjRnHu3Dm8vG7tl639+/fz0EMPMXPmTEaMGMHvv//OhAkT8Pb2ZuzYsezbt49//etffPnll3Tr1o2rV6+yY8cOAOLi4hg5ciRz585lyJAhpKWlsWPHjjIny0JUOw06QMfH1JEQV4yB3Exwa6A2BxU1jp+bA7MHt+LJuxvx7uZTfH8glk0nE9l0MpEH2gTw7/uaFTmPYo3V+kHY8zHE7oXNL8OQBbd3vMt/walfAA10ebpCQhQ1VONeEH9U/WGsps6rd+w7yM0An2YQHGHraIQoRGoCawF3d3f0ej1OTk74+/vj7+9vmQD95Zdf5r777qNx48Z4eXkRHh7OU089RatWrWjatCmzZ8+mcePGpdbsjR07lpEjR9KkSRNee+010tPT2bNnzy3HOm/ePHr16sX06dNp1qwZY8eOZdKkSbz55puAOrmms7MzDzzwACEhIbRr145//etfgJoE5uXlMXToUEJDQ2ndujUTJkzAxaUWfakSdU+vGeDkA8b8aQ7C+sngATVckJcTbz4YzsYp9/BAmwAAfj4Sx33ztvOfFYe5cDXTxhFWEI0G+r6hLh9eBhf3397xducnkWH9wLvx7R1L1GyNe6rPZ7ao/URrooKmoO3HyGe6qJakJrAUjvY6TrwcabNz366OHTtavU5PT2fmzJmsWbPGklRlZWVx/vz5Eo/Tps31mglnZ2fc3NxITEy85XhOnjzJoEHWo711796d+fPnYzKZuO+++wgJCaFRo0b07duXvn37WpqhhoeH06tXL1q3bk1kZCR9+vRh+PDheHpKkyFRgzl6Qp9XYHV+zUdYP9vGIypM43ouvP9Ieybcm8q8jdFsOpnIiv2xrD50kYc7BTOpZxP83CpoQBVbadABwkeqA3msex6e+KV8X3gzr8Khr9XliIkVG6OoeYK7gp2jOlVE4gl16oia5NJBtb+3Tq/++xCiGpKawFJoNBqc9HY2eVTEMOM3j/I5depUVq1axWuvvcaOHTs4dOgQrVu3xmg0lngce3vrEdo0Gg1ms/m247uZq6srBw4c4OuvvyYgIIAZM2YQHh5OcnIyOp2OjRs3sm7dOlq2bMl7771HWFgYMTExFR6HEFUq/GFoPxqaPwAN77Z1NKKCtQx049MxnVg1oRt3NvEh16Tw5R/nuHvuVl5be5KrGSV//lZ7vaLA3hli98DR78p3jH2L1LnU/NtASPeKjU/UPHYGCL1TXT692baxlMf+L9TnFgPB2du2sQhRDEkCawm9Xo/JZCq13M6dOxk7dixDhgyhdevW+Pv7c/bs2coPMF+LFi3YuXNnoZiaNWtmacJqZ2dH7969mTt3LkeOHOHs2bNs2aIOmKHRaOjevTuzZs3i4MGD6PV6Vq1aVWXxC1EpNBoY+B48vFSGxK/F2gV78tW4Lnw9visdQjzJyTPz8a9/c/fcrczb+Bep2bm2DrF83ALgrn+ry5uiwHiLg3XlGWGPTA4vbtKkl/pcMGBWTZGTrk4QDzIgjKjWpDloLREaGsru3bs5e/YsLi4uxdbSNW3alJUrVzJgwAA0Gg3Tp0+vlBq94jz33HN06tSJ2bNnM2LECHbt2sX777/Phx+q80L9/PPP/P3339x99914enqydu1azGYzYWFh7N69m82bN9OnTx98fX3ZvXs3ly9fpkWLFlUWvxBC3K6Ixt5893QE26Iv89Yv0Ry/lMq7m0/xxe9neeqeRoztFlqmQcGqlYhJsH8JpJyHne9CjxfKvu/xlZAeDy7+cMfQyotR1CwF/QLP/Q65WWDvaNt4yurY92ofb6/G12szhaiGpCawlpg6dSo6nY6WLVtSr169Yvv4zZs3D09PT7p168aAAQOIjIykffv2VRZn+/bt+fbbb1m+fDmtWrVixowZvPzyy4wdOxYADw8PVq5cSc+ePWnRogULFy7k66+/5o477sDNzY1ff/2V/v3706xZM1566SXefvtt+vWTPlRCiJpFo9HQo7kvP026kw9HtaeJrwspWbnMXR/N3XO3sXhnDDl5pbfuqDbsHaHPbHV55/8gJbZs+ynK9WkhOo8HO33lxCdqHp9m4FYfTDlqIlgTKMr1AWE6jJVabVGtaZQ6Nr5+bGwsQUFBXLhwgQYNGlhty87OJiYmhoYNG+LgUMM764sSyd9aCFGdmMwKPxy6yPxNpzifP3pooLsD/+rVlGEdGmCvqwG/2SoKfH4/nNsJrYbD8M9K3+fsb+o+do4w5YRMqC2s/TARDn6l1jRHvmrraEqWdQ1+ehZO/ABae3juT3D2sXVUogxKyg1qsxrwv4oQQghRu+m0Goa2b8Dm5+7h1SGt8Hdz4FJKNs+vPMp987az+uBFTOZq/putRgN95wAadY6083+Uvk9BLWDbkZIAisJunCqiOju7ExbcmZ8A2kG/1yUBFNWeJIHitjz99NO4uLgU+Xj6aZnsVwghboW9TsuoLiFs+8+9vHR/C7yd9Zy9ksnkbw7R73+/sv5YPNW6AU9AOLR/VF1eNw1K6nN+5QxEr1OXu06o/NhEzdOoB6BRp4lIjbN1NIWZcmHLK/DFA5AaC16N1GlSOo2zdWRClKqG9TwX1c3LL7/M1KlTi9zm5uZWxdEIIUTt4GCvY9xdjRjZOZjPfz/LR9vP8FdCOk9/tZ82Ddx5rk8Ydzf1qZCphCpcz+lwbBXEHVLnD2w3quhyuxcCCjSNBJ+mVRmhqCmcvCCwHVw6AH9vhbaP2Dqi667GwPfj4OI+9XXbUdDvDTC42jYuIcpIkkBxW3x9ffH19bV1GEIIUSs5G+yY2KMJ/+gawie//s2inTEciU1hzKI9dA71YmpkGJ0bVrNmlC6+cM9/YeN02DwLWg4s/MU465ra1wsgQmoBRQka91STwDNbqk8SePgbWPMcGNPA4A4D3oFWw2wdlRC3RJqDCiGEENWcu6M9UyPD+PW/PXjizobo7bTsOXuVhz7axaOf7ebwhWRbh2ity9Nq07j0BNgxr/D2/V9Abib4tYKG91R9fKLmuLFfYBVOaVWk7BT4fjyselJNAIMj4J+/SQIoaiRJAoUQQogawsfFwPQHWvLrf3owqkswdloNO04lMeiDnTy5ZB/R8Wm2DlFlp4c++aM57npfbTpXwJQLez5Wl7tOkGH0RcmCOoPeBTKvQPwR28VxYQ8svBOOfgsaHfT4PxjzM3gE2y4mIW6DJIFCCCFEDePv7sCrQ1qz5bl7Gdq+PloN/HIigb7/+5Vnlx8kJinD1iFCWD9odC+YjGrT0AInfoDUi+DsC62H2yw8UUPo7KHh3eqyLUYJNZtg+1xY1BeSz6tJ32Pr1CbPOulVJWouuXuFEEKIGirY24l5D7Vlwr2NmbfxL9YejeeHQ5f4+Ugcg8IDaejjjKNepz7s8x/5y056Oxz1WhwKlu11ONhrK26wGY0GIufAwu5w8ieI2QGhd16fFqLTOLAzVMy5RO3WuCdEr1WTwLumVN15ky/AyifhfP5k9a0fhPvfBgf3qotBiEoiSaAQQghRwzXxdeXDUR04djGFeRv/Ysufiaw8eLFcx7oxUXTU63DS63CwL0gcdYW2X08qbyxXkGDWJ6jVaNyOfo5p3fMofV/H7tIB0Bmg0xMVfBVErVXQL/D8H5CTDgaXyj/nsZXw02TISQG9q5r8hY+o/PMKUUUkCRQAhIaGMnnyZCZPnlxqWY1Gw6pVqxg8eHClxyWEEKLsWtV3Z9HYTuw/d5V1R+NJz8kjK9dEltFU5HNm/rMx7/qAG1m56rqK4kEE2wwr8Eg8xuUvRlFPA7vd7uPMsUzCg1II83PFTie9U0QJvBqBRwgkn4NzO6FZZOWdKyddnePyUP7otfU7wrBP1BiEqEUkCRRCCCFqmQ4hXnQIKfvUESazcj05LEgUc01kGvPIzjWRZTRbljNv2G5V/obEMjv3hmWjHe+ahjPD7gvqaVIAeCn+bk6tOgqoNY+t67sTHuRO2yBP2gZ7EOjuUD3nQBS2odGotYH7F6tNQisrCbx4QJ377+oZQAN3PQf3Pq/2SxSilpEkUAghhKjjdFoNLgY7XAyV9LXA1BNlwR9okqJJ8r+LPg3vwe9CCocvJJOWk8ees1fZc/YqoI4iWs/VQHgDD9oFe9A2yIM2DdxxdZAv4nXajUlgRTOb4ff/wZZXwJwHbvVh6MdqH1Yhailpf1EaRQFjhm0eilKmED/++GMCAwMx3zR/zqBBg3j88cc5c+YMgwYNws/PDxcXFzp16sSmTZsq7BIdPXqUnj174ujoiLe3N08++STp6emW7du2baNz5844Ozvj4eFB9+7dOXfuHACHDx+mR48euLq64ubmRocOHdi3b1+FxSaEEKIa0NmjGfoxhN2Pz9C3+E9kc74a14XDUX3YNOVu3hzehn90DaZVfTfstBoup+Ww6WQCb26IZtSnu2kz6xd6z9vO1BWH+eqPcxy7mEKuycZzxomq1fBudWqGpL/UAVsqSuol+HIQbJqpJoAtB8E/d0oCKGo9qQksTW4mvBZom3O/eAn0zqUWe/DBB3nmmWfYunUrvXr1AuDq1ausX7+etWvXkp6eTv/+/Xn11VcxGAwsWbKEAQMGEB0dTXDw7c1vk5GRQWRkJBEREezdu5fExETGjRvHpEmT+Pzzz8nLy2Pw4MGMHz+er7/+GqPRyJ49eyzNfEaNGkW7du1YsGABOp2OQ4cOYW8vv/YKIUStE9gWRi6zWqXVamji60oTX1ce7BgEQHauiWMXUzh0IdnyiL2WxenEdE4npvPd/lgAHOy1tAp0p22QB+FBao1hA09HaUZaWzl6QIOOcGG3WhvYYcztH/Pkz/DjJMi6BvZO0G8utPuHzF0p6gRJAmsBT09P+vXrx7JlyyxJ4HfffYePjw89evRAq9USHh5uKT979mxWrVrFjz/+yKRJk27r3MuWLSM7O5slS5bg7KwmrO+//z4DBgzgjTfewN7enpSUFB544AEaN24MQIsWLSz7nz9/nv/85z80b94cgKZNm95WPEIIIWo2B3sdHUO96Bh6vU/j5bQcjsQmWyWGadl57Dt3jX3nrlnK+bjo1aSwgQdtgz1o08ADd0f5YbHWaNyzYpJAYyZseFFtXgoQEA7DPgMf+Q4i6g5JAktj76TWyNnq3GU0atQoxo8fz4cffojBYGDp0qU8/PDDaLVa0tPTmTlzJmvWrCEuLo68vDyysrI4f/78bYd48uRJwsPDLQkgQPfu3TGbzURHR3P33XczduxYIiMjue++++jduzcPPfQQAQEBAEyZMoVx48bx5Zdf0rt3bx588EFLsiiEEEKA2kewVws/erXwA8BsVoi5ksGh89eTwpNxqSSlG9l0MpFNJxMt+zau50x4kAftgjxoG+RJmL8rejvpDVMjNe4J2+bA39vUSdy1uls/RtwR+P4JtVkpQPdnocdLYKev0FCFqO4kCSyNRlOmJpm2NmDAABRFYc2aNXTq1IkdO3bwzjvvADB16lQ2btzIW2+9RZMmTXB0dGT48OEYjcYqiW3x4sX861//Yv369XzzzTe89NJLbNy4ka5duzJz5kweeeQR1qxZw7p164iKimL58uUMGTKkSmITQghR82i1GhrXc6FxPReGdWgAqM1Ij19KtSSFhy8kc/5qJmcuZ3DmcgYrD6jzJurttLQKdLOMRNq2gQf1PR3RaaUJYLUX2B4M7pCdDJcOQYMOZd/XbIbdC9S+fyYjuPjDkIXQuEclBStE9SZJYC3h4ODA0KFDWbp0KadPnyYsLIz27dsDsHPnTsaOHWtJrNLT0zl79myFnLdFixZ8/vnnZGRkWGoDd+7ciVarJSwszFKuXbt2tGvXjhdeeIGIiAiWLVtG165dAWjWrBnNmjXj3//+NyNHjmTx4sWSBAohhLglDvY6OoR40iHE07LuSnoOh2OT1RrDWHU00pSsXA6cT+bA+WTYqZaz12kI9HCkgacjDTycqO+Zv+zpRANPR/zcHCRJrA50dtDobjj5E5zZXPYkMC0BVv9T3QcgrD8MfB+cvSsvViGqOUkCa5FRo0bxwAMPcPz4cf7xj39Y1jdt2pSVK1cyYMAANBoN06dPLzSS6O2cMyoqijFjxjBz5kwuX77MM888w6OPPoqfnx8xMTF8/PHHDBw4kMDAQKKjozl16hSjR48mKyuL//znPwwfPpyGDRsSGxvL3r17GTZsWIXEJoQQom7zdjHQs7kfPZurzUgVRSEmKcNSU3joQjIn4lLJNSmcu5LJuSuZwJVCx7HT3pAkejpS38PJstzAywl/SRKrTuNe+UngFrjnv6WX/2sDrJ4AmUlg5wCRr0HHx2XwF1HnSRJYi/Ts2RMvLy+io6N55JFHLOvnzZvH448/Trdu3fDx8WHatGmkpqZWyDmdnJzYsGEDzz77LJ06dcLJyYlhw4Yxb948y/Y///yTL774gitXrhAQEMDEiRN56qmnyMvL48qVK4wePZqEhAR8fHwYOnQos2bNqpDYhBBCiBtpNBoa1XOhUT0XhrZXm5HmmcwkpOUQezWT2GtZXEzOIvaauhx7LYtLyVnkmRXOX83k/NXMIo9rp9UQ4OFAg/zksP4NtYgNPB3xd3PATif9ECtEQfPNC3sgOxUc3Ioul5sNG2fAno/U136t1MFffJtXTZxCVHMaRSnjZHS1RGxsLEFBQVy4cIEGDRpYbcvOziYmJoaGDRvi4OBgowhFVZC/tRBCiLIwmRUSUrPzE8RMYq+qyWFscqYlScw1lfxVSqfVEODuYNXEtL7H9eUAd0kSb8m77eHqGXh4GTS/v/D2hBPw/ThIPK6+7joBekWBvfx/LworKTeozaQmUAghhBCiGLr8pqCBHo6AV6HtJrPC5bScG2oPr9cixl7L5FJyNkaT2bIOrhZ5Dn+360lifU9H6rka8HLS4+V8/eHpZC/JIqijhF49A6c3WyeBigJ7P4VfXoK8bHCuB4MXQNP7bBerENWUJIHCytKlS3nqqaeK3BYSEsLx48erOCIhhBCi+tJpNfi7O+Dv7kDH0MLbzWaFy+k5hZLD2GtZXLyWRWxyFsY8MxeT1aaou2MKJ4k3cne0t0oMvZz0eLnoCyWMBQ8nvQ5Nbev/1qQX7P1E7RdYICMJfpgIf63PL3MfDP4QXHxtE6MQ1ZwkgcLKwIED6dKlS5Hb7O1lwl0hhBDiVmi1GvzcHPBzc6BDSOHtZrNCUnoOF/KTQ7VPYhZX0nO4lpHLlYwcrmXmci3TiKJASlYuKVm5xCRllOn8ejst3s56PJ30eLuoz17OenVdEc+eTvrqP8hN6J2gtYNrMXD1b7h2FlY9DekJoNPDfbOhy1My+IsQJZAkUFhxdXXF1dXV1mEIIYQQdYJWq8HXzQFfNwer6S1uZjIrpGTlcjUjh6sZRT9fyTByLdPI1XQjVzKM5OSZMeaZiUvJJi4lu0zxaDQ31DbeVLsY6uNMywA3mvi64GBfjonaK4rBFYK6wLmdat+/i/vV9fWaq4O/+LeyXWxC1BCSBAohhBBCVHM6rcaSjJVVpjGPqxnGIh/XMo1cSc9/nak+J2fmoiiQnJlLcmYuf1N0baNOq6FxPWdaBLjR3N+NFgGutAxwo56roeqanjbuoSaBBQlgp3HQ5xWwd6ya8wtRw0kSKIQQQghRCznp7XDS29HA06lM5fNMZpKzcrmWodYk3vh8OT2HUwnpnIxPJTkzl78S0vkrIZ0fuGTZ39tZT4sANSlUn9VaQ/vKGMym+QOw9TVw8IBBH0Dz/hV/DiFqMUkChRBCCCEEdjotPi4GfFwMNC2mjKIoxKdmczIulZNxaZyIS+VkXCpnkzK4kmHkt9NJ/HY6yVLeXqehia+rpbawIDm8lRrNIvm2gAl/qAO/OBbfjFYIUTRJAoUQQgghRJloNBoC3B0JcHekZ3M/y/oso4nohLT85FB9/BmXRlpOnuX1Si5ayvu5GSwJYYsAN1oGuBLq7XxrU2DUC6vItyZEnSJJoBBCCCGEuC2Oeh1tgzxoG+RhWacoCrHXsiy1hifjUjkZn8q5K5kkpOaQkHqZbdGXLeUNdlrC/F1p4X+9SWnzADfcHavX6OSKopCTZybTaCLTmEeW0ZS/bCIrNw8Hex2+rgbquTjg5mhX+6boELWCJIECgNDQUCZPnszkyZNtHUqZ3HvvvbRt25b58+fbOhQhhBBCFEGj0RDk5USQlxN97vC3rE/PySM6PpUTcddrDqPj08g0mjgSm8KR2BSr49T3cLTqZ9giwI0QLye0JUxlUZCoZeTk5SdnpkJJW1b+68xcE5k515O4zJu337B/wTqzUrZroNdpqedqwMfVQD0XA/VcDWqCeOMjf71NR1wVdY4kgaLa0Gg0rFq1isGDB5daduXKlTJvoRBCCFEDuRjs6BDiRYcQL8s6s1nh3NVMq+akJ+PSuJicZXlsOploKe+k1xHm74qTXndDclaQqKmJnVLGRO126O20OOl1ONnrcNSrjyyjictpOaRm52E0mS3xl8bVwc4qKbwxSfR1c7Cs93KuAXM5impPkkBRoxiNRvR6PV5eXqUXFkIIIUSNoNVqaOjjTEMfZ/q3DrCsT8nM5WS8dWIYnaDWGh48n1ymYxsKEjW9HY56HU56HY72Oss6p4J1VssF5W5eZ4dz/rKjva7EPozZuSaS0nO4nJb/uHH5hteJaTkY88ykZeeRlp3H35eLnprDcq004O1SdLJ442tfVwMuBmmOKopWLZLADz74gDfffJP4+HjCw8N577336Ny5c5FlP/nkE5YsWcKxY8cA6NChA6+99lqx5W+bokBmZuUcuzROTuqsraX4+OOPmTlzJrGxsWi11z+MBg0ahLe3N//3f//HlClT+OOPP8jIyKBFixbMmTOH3r17lyssjUbDwoUL+emnn9iyZQshISEsWrSIevXqMW7cOPbu3Ut4eDhffvkljRs3tuz3ww8/MGvWLE6cOEFgYCBjxozh//7v/7CzsyM0NBSAIUOGABASEsLZs2eZOXMmq1evZtKkSbz66qucO3cOs9lcqDloTk4OM2bMYNmyZSQmJhIUFMQLL7zAE088Ua73KIQQQgjbc3eyp2sjb7o28rasyzOZiUnKIDohjTyTYknYbkzqChI2R3udzWrNHOx1NPB0KnWKDkVRSM3OKzVZvJyWw5WMHMwKltfElRaDNj8hdMDf3QF/NwcC3B3wy3/2d3fA19UBvV0lTOMhqjWbJ4HffPMNU6ZMYeHChXTp0oX58+cTGRlJdHQ0vr6+hcpv27aNkSNH0q1bNxwcHHjjjTfo06cPx48fp379+hUfYGYmuLhU/HHLIj0dnJ1LLfbggw/yzDPPsHXrVnr16gXA1atXWb9+PWvXriU9PZ3+/fvz6quvYjAYWLJkCQMGDCA6Oprg4OByhTZ79mzmzZvHvHnzmDZtGo888giNGjXihRdeIDg4mMcff5xJkyaxbt06AHbs2MHo0aN59913ueuuuzhz5gxPPvkkAFFRUezduxdfX18WL15M37590emut4s/ffo033//PStXrrRaf6PRo0eza9cu3n33XcLDw4mJiSEpKanIskIIIYSouex0Wpr6udLUz9XWoVQIjUaDu6M97o72NPEt+TtnnsnM1UxjsTWKl9NySMp/TsvJIzvXzIWrWVy4WnJzVB8XA/7uBvzdHC3Job9b/nP+srPB5mmDqEA2/2vOmzeP8ePH89hjjwGwcOFC1qxZw6JFi3j++ecLlV+6dKnV608//ZTvv/+ezZs3M3r06CqJubrx9PSkX79+LFu2zJIEfvfdd/j4+NCjRw+0Wi3h4eGW8rNnz2bVqlX8+OOPTJo0qVznfOyxx3jooYcAmDZtGhEREUyfPp3IyEgAnn32WcvfFGDWrFk8//zzjBkzBoBGjRoxe/Zs/vvf/xIVFUW9evUA8PDwwN/f3+pcRqORJUuWWMrc7K+//uLbb79l48aNltrNRo0alet9CSGEEEJUV3Y6Lb6uau1dabKManPUxLRsElNziEvJJj41m/iU/Ef+stFkJik9h6T0HI5dTC32eK4OdjfVIjoWqln0cLKX5qc1hE2TQKPRyP79+3nhhRcs67RaLb1792bXrl1lOkZmZia5ubnF9hHLyckhJyfH8jotLe3WgnRyUmvkbMGp5OYDNxo1ahTjx4/nww8/xGAwsHTpUh5++GG0Wi3p6enMnDmTNWvWEBcXR15eHllZWZw/f77cobVp08ay7OenzhPUunVrq3XZ2dmkpqbi5ubG4cOH2blzJ6+++qqljMlkIjs7m8zMTJxKeK8hISHFJoAAhw4dQqfTcc8995T7/QghhBBC1CaOep1ldNbiKIrC1QyjJSGMS8kmIVV9vjFRTM/Jy++zmM5fCcV/LzbYaQvVIgZYltWksZ6rQQa2qQZsmgQmJSVhMpksSUQBPz8//vzzzzIdY9q0aQQGBhbbv23OnDnMmjWr/EFqNGVqkmlrAwYMQFEU1qxZQ6dOndixYwfvvPMOAFOnTmXjxo289dZbNGnSBEdHR4YPH47RaCz3+W4cmbPgF5+i1pnNZgDS09OZNWsWQ4cOLXQsB4eSf81yLuX6Ozo6li1oIYQQQghhodFo8HYx4O1i4I5A92LLpWXnWieHKdnEpWaTcEPieCXDSE6emXNXMjl3pfjxNHRaDfVcDPi7O/DBqPbU95DvcbZg8+agt+P1119n+fLlbNu2rdhE4oUXXmDKlCmW1xcvXqRly5ZVFWKVcXBwYOjQoSxdupTTp08TFhZG+/btAdi5cydjx461DLqSnp7O2bNnqzS+9u3bEx0dTZMmTYotY29vj8lkuuVjt27dGrPZzPbt28s92I0QQgghhCiaq4M9rg72NPEtvh9mdq6JxNQc4lOziUvJsqpJLEgUE1KzMZkVdX1qNi76Gp2K1Gg2vfI+Pj7odDoSEhKs1ickJBTqF3azt956i9dff51NmzZZNU28mcFgwGAwWF6nphbf1rmmGzVqFA888ADHjx/nH//4h2V906ZNWblyJQMGDECj0TB9+nRLDV1VmTFjBg888ADBwcEMHz4crVbL4cOHOXbsGK+88gqgTli/efNmunfvjsFgwNPTs0zHDg0NZcyYMTz++OOWgWHOnTtHYmKipd+iEEIIIYSoPA72OoK9nQj2Lr75qcmskJSeY6lRdHOUJNBWbDoerF6vp0OHDmzevNmyzmw2s3nzZiIiIordb+7cucyePZv169fTsWPHqgi1RujZsydeXl5ER0fzyCOPWNbPmzcPT09PunXrxoABA4iMjLTUElaVyMhIfv75Z3755Rc6depE165deeeddwgJCbGUefvtt9m4cSNBQUG0a9fulo6/YMEChg8fzoQJE2jevDnjx48nI6PkeXaEEEIIIUTV0Wk1+Lk50DbIg76t/GUQGRvSKIqi2DKAb775hjFjxvDRRx/RuXNn5s+fz7fffsuff/6Jn58fo0ePpn79+syZMweAN954wzIfXPfu3S3HcXFxwaUMUznExsYSFBTEhQsXaNCggdW27OxsYmJiaNiwYan91ETNJn9rIYQQQghRUm5Qm9m8DnbEiBFcvnyZGTNmEB8fT9u2bVm/fr1lsJjz589bTYC+YMECjEYjw4cPtzpOVFQUM2fOrMrQhRBCCCGEEKLGsXkSCDBp0qRi56vbtm2b1euqHtCkrlm6dClPPfVUkdtCQkI4fvx4FUckhBBCCCGEqEjVIgkU1cfAgQPp0qVLkdtunAJCCCGEEEIIUTNJEiisuLq64upa/PC/QgghhBBCiJrNpqODVlc2HitHVAH5GwshhBBCiLpKksAbFDR3zMzMtHEkorIV/I2liasQQgghhKhrpDnoDXQ6HR4eHiQmJgLg5OQk85fUMoqikJmZSWJiIh4eHuh0OluHJIQQQgghRJWSJPAm/v7+AJZEUNROHh4elr+1EEIIIYQQdYkkgTfRaDQEBATg6+tLbm6urcMRlcDe3l5qAIUQQgghRJ0lSWAxdDqdJApCCCGEEEKIWkcGhhFCCCGEEEKIW/DBBx8QGhqKg4MDXbp0Yc+ePSWWX7FiBc2bN8fBwYHWrVuzdu3aKoq0aJIECiGEEEIIIUQZffPNN0yZMoWoqCgOHDhAeHg4kZGRxY4p8vvvvzNy5EieeOIJDh48yODBgxk8eDDHjh2r4siv0yh1bMK02NhYgoKCuHDhAg0aNLB1OEIIIYQQQggbKU9u0KVLFzp16sT7778PgNlsJigoiGeeeYbnn3++UPkRI0aQkZHBzz//bFnXtWtX2rZty8KFCyvmjdyiOtcn0Gw2AxAXF2fjSIQQQgghhBC2VJATpKSk4ObmZllvMBgwGAyFyhuNRvbv388LL7xgWafVaunduze7du0q8hy7du1iypQpVusiIyNZvXp1BbyD8qlzSWBCQgIAnTt3tnEkQgghhBBCiOqgVatWVq+joqKYOXNmoXJJSUmYTCb8/Pys1vv5+fHnn38Weez4+Pgiy8fHx99e0LehziWB7dq1Y8+ePfj5+aHVSpfIypSWlkbLli05ceIErq6utg6nTpBrXvXkmlctud5VT6551ZNrXrXkele96nTNzWYz58+fp2XLltjZXU+NiqoFrE3qXBJoZ2dHp06dbB1GnZCamgpA/fr1rarXReWRa1715JpXLbneVU+uedWTa1615HpXvep2zYODg8tc1sfHB51OZ2ldWCAhIQF/f/8i9/H397+l8lVBqsKEEEIIIYQQogz0ej0dOnRg8+bNlnVms5nNmzcTERFR5D4RERFW5QE2btxYbPmqUOdqAoUQQgghhBCivKZMmcKYMWPo2LEjnTt3Zv78+WRkZPDYY48BMHr0aOrXr8+cOXMAePbZZ7nnnnt4++23uf/++1m+fDn79u3j448/ttl7kCRQVBqDwUBUVFStb1Ndncg1r3pyzauWXO+qJ9e86sk1r1pyvateTb/mI0aM4PLly8yYMYP4+Hjatm3L+vXrLYO/nD9/3mrskW7durFs2TJeeuklXnzxRZo2bcrq1asLDUZTlercPIFCCCGEEEIIUZdJn0AhhBBCCCGEqEMkCRRCCCGEEEKIOkSSQCGEEEIIIYSoQyQJFEIIIYQQQog6RJJAUS5z5syhU6dOuLq64uvry+DBg4mOji5xn88//xyNRmP1cHBwqKKIa76ZM2cWun7NmzcvcZ8VK1bQvHlzHBwcaN26NWvXrq2iaGuH0NDQQtdco9EwceLEIsvLPX5rfv31VwYMGEBgYCAajYbVq1dbbVcUhRkzZhAQEICjoyO9e/fm1KlTpR73gw8+IDQ0FAcHB7p06cKePXsq6R3UPCVd89zcXKZNm0br1q1xdnYmMDCQ0aNHc+nSpRKPWZ7PprqktPt87Nixha5f3759Sz2u3OfFK+2aF/W5rtFoePPNN4s9ptznxSvLd8Ls7GwmTpyIt7c3Li4uDBs2rNDk6Tcr7/8BomwkCRTlsn37diZOnMgff/zBxo0byc3NpU+fPmRkZJS4n5ubG3FxcZbHuXPnqiji2uGOO+6wun6//fZbsWV///13Ro4cyRNPPMHBgwcZPHgwgwcP5tixY1UYcc22d+9eq+u9ceNGAB588MFi95F7vOwyMjIIDw/ngw8+KHL73Llzeffdd1m4cCG7d+/G2dmZyMhIsrOziz3mN998w5QpU4iKiuLAgQOEh4cTGRlJYmJiZb2NGqWka56ZmcmBAweYPn06Bw4cYOXKlURHRzNw4MBSj3srn011TWn3OUDfvn2trt/XX39d4jHlPi9Zadf8xmsdFxfHokWL0Gg0DBs2rMTjyn1etLJ8J/z3v//NTz/9xIoVK9i+fTuXLl1i6NChJR63PP8HiFugCFEBEhMTFUDZvn17sWUWL16suLu7V11QtUxUVJQSHh5e5vIPPfSQcv/991ut69Kli/LUU09VcGR1x7PPPqs0btxYMZvNRW6Xe7z8AGXVqlWW12azWfH391fefPNNy7rk5GTFYDAoX3/9dbHH6dy5szJx4kTLa5PJpAQGBipz5syplLhrspuveVH27NmjAMq5c+eKLXOrn011WVHXfMyYMcqgQYNu6Thyn5ddWe7zQYMGKT179iyxjNznZXfzd8Lk5GTF3t5eWbFihaXMyZMnFUDZtWtXkcco7/8BouykJlBUiJSUFAC8vLxKLJeenk5ISAhBQUEMGjSI48ePV0V4tcapU6cIDAykUaNGjBo1ivPnzxdbdteuXfTu3dtqXWRkJLt27arsMGslo9HIV199xeOPP45Goym2nNzjFSMmJob4+Hire9jd3Z0uXboUew8bjUb2799vtY9Wq6V3795y35dTSkoKGo0GDw+PEsvdymeTKGzbtm34+voSFhbGP//5T65cuVJsWbnPK1ZCQgJr1qzhiSeeKLWs3Odlc/N3wv3795Obm2t1zzZv3pzg4OBi79ny/B8gbo0kgeK2mc1mJk+eTPfu3WnVqlWx5cLCwli0aBE//PADX331FWazmW7duhEbG1uF0dZcXbp04fPPP2f9+vUsWLCAmJgY7rrrLtLS0oosHx8fj5+fn9U6Pz8/4uPjqyLcWmf16tUkJyczduzYYsvIPV5xCu7TW7mHk5KSMJlMct9XkOzsbKZNm8bIkSNxc3MrttytfjYJa3379mXJkiVs3ryZN954g+3bt9OvXz9MJlOR5eU+r1hffPEFrq6upTZNlPu8bIr6ThgfH49ery/0Y1JJ92x5/g8Qt8bO1gGImm/ixIkcO3as1LbxERERREREWF5369aNFi1a8NFHHzF79uzKDrPG69evn2W5TZs2dOnShZCQEL799tsy/YIpbs9nn31Gv379CAwMLLaM3OOitsjNzeWhhx5CURQWLFhQYln5bLo9Dz/8sGW5devWtGnThsaNG7Nt2zZ69eplw8jqhkWLFjFq1KhSB/GS+7xsyvqdUNie1ASK2zJp0iR+/vlntm7dSoMGDW5pX3t7e9q1a8fp06crKbrazcPDg2bNmhV7/fz9/QuNvJWQkIC/v39VhFernDt3jk2bNjFu3Lhb2k/u8fIruE9v5R728fFBp9PJfX+bChLAc+fOsXHjxhJrAYtS2meTKFmjRo3w8fEp9vrJfV5xduzYQXR09C1/toPc50Up7juhv78/RqOR5ORkq/Il3bPl+T9A3BpJAkW5KIrCpEmTWLVqFVu2bKFhw4a3fAyTycTRo0cJCAiohAhrv/T0dM6cOVPs9YuIiGDz5s1W6zZu3GhVUyXKZvHixfj6+nL//fff0n5yj5dfw4YN8ff3t7qHU1NT2b17d7H3sF6vp0OHDlb7mM1mNm/eLPd9GRUkgKdOnWLTpk14e3vf8jFK+2wSJYuNjeXKlSvFXj+5zyvOZ599RocOHQgPD7/lfeU+v66074QdOnTA3t7e6p6Njo7m/Pnzxd6z5fk/QNwiGw9MI2qof/7zn4q7u7uybds2JS4uzvLIzMy0lHn00UeV559/3vJ61qxZyoYNG5QzZ84o+/fvVx5++GHFwcFBOX78uC3eQo3z3HPPKdu2bVNiYmKUnTt3Kr1791Z8fHyUxMRERVEKX++dO3cqdnZ2yltvvaWcPHlSiYqKUuzt7ZWjR4/a6i3USCaTSQkODlamTZtWaJvc47cnLS1NOXjwoHLw4EEFUObNm6ccPHjQMhLl66+/rnh4eCg//PCDcuTIEWXQoEFKw4YNlaysLMsxevbsqbz33nuW18uXL1cMBoPy+eefKydOnFCefPJJxcPDQ4mPj6/y91cdlXTNjUajMnDgQKVBgwbKoUOHrD7bc3JyLMe4+ZqX9tlU15V0zdPS0pSpU6cqu3btUmJiYpRNmzYp7du3V5o2bapkZ2dbjiH3+a0p7bNFURQlJSVFcXJyUhYsWFDkMeQ+L7uyfCd8+umnleDgYGXLli3Kvn37lIiICCUiIsLqOGFhYcrKlSstr8vyf4AoP0kCRbkART4WL15sKXPPPfcoY8aMsbyePHmyEhwcrOj1esXPz0/p37+/cuDAgaoPvoYaMWKEEhAQoOj1eqV+/frKiBEjlNOnT1u233y9FUVRvv32W6VZs2aKXq9X7rjjDmXNmjVVHHXNt2HDBgVQoqOjC22Te/z2bN26tcjPkYJrajablenTpyt+fn6KwWBQevXqVejvEBISokRFRVmte++99yx/h86dOyt//PFHFb2j6q+kax4TE1PsZ/vWrVstx7j5mpf22VTXlXTNMzMzlT59+ij16tVT7O3tlZCQEGX8+PGFkjm5z29NaZ8tiqIoH330keLo6KgkJycXeQy5z8uuLN8Js7KylAkTJiienp6Kk5OTMmTIECUuLq7QcW7cpyz/B4jy0yiKolROHaMQQgghhBBCiOpG+gQKIYQQQgghRB0iSaAQQgghhBBC1CGSBAohhBBCCCFEHSJJoBBCCCGEEELUIZIECiGEEEIIIUQdIkmgEEIIIYQQQtQhkgQKIYQQQgghRB0iSaAQQgghhBBC1CGSBAohhBC3QKPRsHr1aluHIYQQQpSbJIFCCCFqjLFjx6LRaAo9+vbta+vQhBBCiBrDztYBCCGEELeib9++LF682GqdwWCwUTRCCCFEzSM1gUIIIWoUg8GAv7+/1cPT0xNQm2ouWLCAfv364ejoSKNGjfjuu++s9j969Cg9e/bE0dERb29vnnzySdLT063KLFq0iDvuuAODwUBAQACTJk2y2p6UlMSQIUNwcnKiadOm/Pjjj5X7poUQQogKJEmgEEKIWmX69OkMGzaMw4cPM2rUKB5++GFOnjwJQEZGBpGRkXh6erJ3715WrFjBpk2brJK8BQsWMHHiRJ588kmOHj3Kjz/+SJMmTazOMWvWLB566CGOHDlC//79GTVqFFevXq3S9ymEEEKUl0ZRFMXWQQghhBBlMXbsWL766iscHBys1r/44ou8+OKLaDQann76aRYsWGDZ1rVrV9q3b8+HH37IJ598wrRp07hw4QLOzs4ArF27lgEDBnDp0iX8/PyoX78+jz32GK+88kqRMWg0Gl566SVmz54NqImli4sL69atk76JQgghagTpEyiEEKJG6dGjh1WSB+Dl5WVZjoiIsNoWERHBoUOHADh58iTh4eGWBBCge/fumM1moqOj0Wg0XLp0iV69epUYQ5s2bSzLzs7OuLm5kZiYWN63JIQQQlQpSQKFEELUKM7OzoWaZ1YUR0fHMpWzt7e3eq3RaDCbzZURkhBCCFHhpE+gEEKIWuWPP/4o9LpFixYAtGjRgsOHD5ORkWHZvnPnTrRaLWFhYbi6uhIaGsrmzZurNGYhhBCiKklNoBBCiBolJyeH+Ph4q3V2dnb4+PgAsGLFCjp27Midd97J0qVL2bNnD5999hkAo0aNIioqijFjxjBz5kwuX77MM888w6OPPoqfnx8AM2fO5Omnn8bX15d+/fqRlpbGzp07eeaZZ6r2jQohhBCVRJJAIYQQNcr69esJCAiwWhcWFsaff/4JqCN3Ll++nAkTJhAQEMDXX39Ny5YtAXBycmLDhg08++yzdOrUCScnJ4YNG8a8efMsxxozZgzZ2dm88847TJ06FR8fH4YPH151b1AIIYSoZDI6qBBCiFpDo9GwatUqBg8ebOtQhBBCiGpL+gQKIYQQQgghRB0iSaAQQgghhBBC1CHSJ1AIIUStIT0chBBCiNJJTaAQQgghhBBC1CGSBAohhBBCCCFEHSJJoBBCCCGEEELUIZIECiGEEEIIIUQdIkmgEEIIIYQQQtQhkgQKIYQQQgghRB0iSaAQQgghhBBC1CGSBAohhBBCCCFEHfL/Ox/1oAfHNY0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 1000x500 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Get loss, val_loss, and the computed metric from history\n",
    "loss = [x['loss'] for x in history if 'loss' in x]\n",
    "val_loss = [x['eval_loss'] for x in history if 'eval_loss' in x]\n",
    "\n",
    "# Get spearman (for regression) or accuracy value (for classification)\n",
    "if [x['eval_spearmanr'] for x in history if 'eval_spearmanr' in x] != []:\n",
    "    metric = [x['eval_spearmanr'] for x in history if 'eval_spearmanr' in x]\n",
    "else:\n",
    "    metric = [x['eval_accuracy'] for x in history if 'eval_accuracy' in x]\n",
    "\n",
    "epochs = [x['epoch'] for x in history if 'loss' in x]\n",
    "\n",
    "# Create a figure with two y-axes\n",
    "fig, ax1 = plt.subplots(figsize=(10, 5))\n",
    "ax2 = ax1.twinx()\n",
    "\n",
    "# Plot loss and val_loss on the first y-axis\n",
    "line1 = ax1.plot(epochs, loss, label='train_loss')\n",
    "line2 = ax1.plot(epochs, val_loss, label='val_loss')\n",
    "ax1.set_xlabel('Epoch')\n",
    "ax1.set_ylabel('Loss')\n",
    "\n",
    "# Plot the computed metric on the second y-axis\n",
    "line3 = ax2.plot(epochs, metric, color='red', label='val_metric')\n",
    "ax2.set_ylabel('Metric')\n",
    "ax2.set_ylim([0, 1])\n",
    "\n",
    "# Combine the lines from both y-axes and create a single legend\n",
    "lines = line1 + line2 + line3\n",
    "labels = [line.get_label() for line in lines]\n",
    "ax1.legend(lines, labels, loc='lower left')\n",
    "\n",
    "# Show the plot\n",
    "plt.title(\"Training History\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7a4a53e",
   "metadata": {},
   "source": [
    "# Save and Load the finetuned model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "6ade4a9d",
   "metadata": {},
   "outputs": [],
   "source": [
    "def save_model(model,filepath):\n",
    "# Saves all parameters that were changed during finetuning\n",
    "\n",
    "    # Create a dictionary to hold the non-frozen parameters\n",
    "    non_frozen_params = {}\n",
    "\n",
    "    # Iterate through all the model parameters\n",
    "    for param_name, param in model.named_parameters():\n",
    "        # If the parameter has requires_grad=True, add it to the dictionary\n",
    "        if param.requires_grad:\n",
    "            non_frozen_params[param_name] = param\n",
    "\n",
    "    # Save only the finetuned parameters \n",
    "    torch.save(non_frozen_params, filepath)\n",
    "\n",
    "    \n",
    "def load_model(filepath, num_labels=1, mixed = False):\n",
    "# Creates a new PT5 model and loads the finetuned weights from a file\n",
    "\n",
    "    # load a new model\n",
    "    model, tokenizer = PT5_classification_model(num_labels=num_labels, half_precision=mixed)\n",
    "    \n",
    "    # Load the non-frozen parameters from the saved file\n",
    "    non_frozen_params = torch.load(filepath)\n",
    "\n",
    "    # Assign the non-frozen parameters to the corresponding parameters of the model\n",
    "    for param_name, param in model.named_parameters():\n",
    "        if param_name in non_frozen_params:\n",
    "            param.data = non_frozen_params[param_name].data\n",
    "\n",
    "    return tokenizer, model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be5ba621",
   "metadata": {},
   "source": [
    "This saves only the finetuned weights to a .pth file\n",
    "\n",
    "It is a 14 MB File, while the entire model would be around 4.8 GB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "d31dc7e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "save_model(model,\"./PT5_GB1_finetuned.pth\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f8e96aa",
   "metadata": {},
   "source": [
    "To load the weights again, we initialize a new PT5 model from the pretrained checkpoint and load the LoRA weights afterwards\n",
    "\n",
    "You need to specifiy the correct num_labels here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "edbd69f7",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "loading configuration file config.json from cache at /data2/cache/huggingface/hub/models--Rostlab--prot_t5_xl_uniref50/snapshots/973be27c52ee6474de9c945952a8008aeb2a1a73/config.json\n",
      "Model config T5Config {\n",
      "  \"architectures\": [\n",
      "    \"T5ForConditionalGeneration\"\n",
      "  ],\n",
      "  \"d_ff\": 16384,\n",
      "  \"d_kv\": 128,\n",
      "  \"d_model\": 1024,\n",
      "  \"decoder_start_token_id\": 0,\n",
      "  \"dense_act_fn\": \"relu\",\n",
      "  \"dropout_rate\": 0.1,\n",
      "  \"eos_token_id\": 1,\n",
      "  \"feed_forward_proj\": \"relu\",\n",
      "  \"initializer_factor\": 1.0,\n",
      "  \"is_encoder_decoder\": true,\n",
      "  \"is_gated_act\": false,\n",
      "  \"layer_norm_epsilon\": 1e-06,\n",
      "  \"model_type\": \"t5\",\n",
      "  \"n_positions\": 512,\n",
      "  \"num_decoder_layers\": 24,\n",
      "  \"num_heads\": 32,\n",
      "  \"num_layers\": 24,\n",
      "  \"output_past\": true,\n",
      "  \"pad_token_id\": 0,\n",
      "  \"relative_attention_max_distance\": 128,\n",
      "  \"relative_attention_num_buckets\": 32,\n",
      "  \"transformers_version\": \"4.26.1\",\n",
      "  \"use_cache\": true,\n",
      "  \"vocab_size\": 128\n",
      "}\n",
      "\n",
      "loading weights file pytorch_model.bin from cache at /data2/cache/huggingface/hub/models--Rostlab--prot_t5_xl_uniref50/snapshots/973be27c52ee6474de9c945952a8008aeb2a1a73/pytorch_model.bin\n",
      "Some weights of the model checkpoint at Rostlab/prot_t5_xl_uniref50 were not used when initializing T5EncoderModel: ['decoder.block.6.layer.0.SelfAttention.q.weight', 'decoder.block.10.layer.2.DenseReluDense.wo.weight', 'decoder.block.21.layer.1.layer_norm.weight', 'decoder.block.22.layer.1.EncDecAttention.q.weight', 'decoder.block.19.layer.0.SelfAttention.o.weight', 'decoder.block.10.layer.0.SelfAttention.k.weight', 'decoder.block.12.layer.0.SelfAttention.v.weight', 'decoder.block.8.layer.1.EncDecAttention.v.weight', 'decoder.block.19.layer.1.EncDecAttention.q.weight', 'decoder.block.8.layer.2.layer_norm.weight', 'decoder.block.15.layer.1.EncDecAttention.k.weight', 'decoder.block.17.layer.0.layer_norm.weight', 'decoder.block.21.layer.0.SelfAttention.v.weight', 'decoder.block.15.layer.2.DenseReluDense.wi.weight', 'decoder.block.8.layer.2.DenseReluDense.wi.weight', 'decoder.block.13.layer.2.layer_norm.weight', 'decoder.block.20.layer.0.SelfAttention.q.weight', 'decoder.block.17.layer.2.DenseReluDense.wi.weight', 'decoder.block.20.layer.1.EncDecAttention.v.weight', 'decoder.block.14.layer.1.layer_norm.weight', 'decoder.block.13.layer.1.EncDecAttention.q.weight', 'decoder.block.5.layer.2.DenseReluDense.wi.weight', 'decoder.block.1.layer.1.EncDecAttention.q.weight', 'decoder.block.23.layer.1.EncDecAttention.o.weight', 'decoder.block.21.layer.0.SelfAttention.o.weight', 'decoder.block.23.layer.2.layer_norm.weight', 'decoder.block.0.layer.1.EncDecAttention.q.weight', 'decoder.block.9.layer.0.SelfAttention.k.weight', 'decoder.block.20.layer.1.EncDecAttention.q.weight', 'decoder.block.6.layer.1.EncDecAttention.k.weight', 'decoder.block.5.layer.2.DenseReluDense.wo.weight', 'decoder.block.12.layer.2.DenseReluDense.wi.weight', 'decoder.block.3.layer.1.layer_norm.weight', 'decoder.block.2.layer.2.DenseReluDense.wo.weight', 'decoder.block.11.layer.2.layer_norm.weight', 'decoder.block.19.layer.2.layer_norm.weight', 'decoder.block.7.layer.2.DenseReluDense.wo.weight', 'decoder.block.5.layer.1.layer_norm.weight', 'decoder.block.13.layer.1.EncDecAttention.v.weight', 'decoder.block.19.layer.0.SelfAttention.k.weight', 'decoder.block.19.layer.1.EncDecAttention.o.weight', 'decoder.block.17.layer.0.SelfAttention.v.weight', 'decoder.block.4.layer.2.layer_norm.weight', 'decoder.block.12.layer.1.EncDecAttention.o.weight', 'decoder.block.1.layer.2.layer_norm.weight', 'decoder.block.3.layer.2.layer_norm.weight', 'decoder.block.17.layer.1.EncDecAttention.k.weight', 'decoder.block.14.layer.0.layer_norm.weight', 'decoder.block.9.layer.0.SelfAttention.o.weight', 'decoder.block.9.layer.0.layer_norm.weight', 'decoder.block.17.layer.2.layer_norm.weight', 'decoder.block.22.layer.1.EncDecAttention.o.weight', 'decoder.block.5.layer.1.EncDecAttention.q.weight', 'decoder.block.11.layer.0.SelfAttention.k.weight', 'decoder.block.22.layer.2.DenseReluDense.wi.weight', 'decoder.block.10.layer.0.layer_norm.weight', 'decoder.block.15.layer.0.SelfAttention.q.weight', 'decoder.block.14.layer.2.layer_norm.weight', 'decoder.block.4.layer.0.SelfAttention.q.weight', 'decoder.block.7.layer.0.layer_norm.weight', 'decoder.block.20.layer.2.DenseReluDense.wo.weight', 'decoder.block.11.layer.2.DenseReluDense.wi.weight', 'decoder.block.3.layer.1.EncDecAttention.v.weight', 'decoder.block.8.layer.1.EncDecAttention.o.weight', 'decoder.block.7.layer.1.EncDecAttention.o.weight', 'decoder.block.19.layer.2.DenseReluDense.wo.weight', 'decoder.block.1.layer.0.SelfAttention.o.weight', 'decoder.embed_tokens.weight', 'decoder.block.22.layer.0.SelfAttention.v.weight', 'decoder.block.13.layer.0.layer_norm.weight', 'decoder.block.9.layer.1.EncDecAttention.o.weight', 'decoder.block.13.layer.2.DenseReluDense.wi.weight', 'decoder.block.9.layer.2.layer_norm.weight', 'decoder.block.21.layer.1.EncDecAttention.o.weight', 'decoder.block.22.layer.0.SelfAttention.q.weight', 'decoder.block.9.layer.1.EncDecAttention.q.weight', 'decoder.block.16.layer.0.SelfAttention.v.weight', 'decoder.block.15.layer.1.layer_norm.weight', 'decoder.block.5.layer.1.EncDecAttention.o.weight', 'decoder.block.20.layer.0.layer_norm.weight', 'decoder.block.9.layer.1.EncDecAttention.k.weight', 'decoder.block.20.layer.1.EncDecAttention.k.weight', 'decoder.block.22.layer.0.SelfAttention.o.weight', 'decoder.block.15.layer.0.SelfAttention.v.weight', 'decoder.block.15.layer.0.SelfAttention.k.weight', 'decoder.block.4.layer.0.SelfAttention.o.weight', 'decoder.block.10.layer.2.layer_norm.weight', 'decoder.block.20.layer.0.SelfAttention.o.weight', 'decoder.block.3.layer.2.DenseReluDense.wo.weight', 'decoder.block.22.layer.0.layer_norm.weight', 'decoder.block.10.layer.0.SelfAttention.q.weight', 'decoder.block.16.layer.1.EncDecAttention.v.weight', 'decoder.block.23.layer.0.SelfAttention.k.weight', 'decoder.block.14.layer.0.SelfAttention.o.weight', 'decoder.block.9.layer.0.SelfAttention.v.weight', 'decoder.block.23.layer.0.layer_norm.weight', 'decoder.block.15.layer.1.EncDecAttention.v.weight', 'decoder.block.13.layer.1.EncDecAttention.o.weight', 'decoder.block.12.layer.0.layer_norm.weight', 'decoder.block.18.layer.1.EncDecAttention.q.weight', 'decoder.block.7.layer.1.EncDecAttention.v.weight', 'decoder.block.4.layer.1.EncDecAttention.v.weight', 'decoder.block.4.layer.0.SelfAttention.v.weight', 'decoder.block.18.layer.0.SelfAttention.v.weight', 'decoder.block.22.layer.1.EncDecAttention.v.weight', 'decoder.block.6.layer.2.DenseReluDense.wi.weight', 'decoder.block.21.layer.2.layer_norm.weight', 'decoder.block.12.layer.2.layer_norm.weight', 'decoder.block.16.layer.0.SelfAttention.k.weight', 'decoder.block.6.layer.1.layer_norm.weight', 'decoder.block.14.layer.2.DenseReluDense.wo.weight', 'decoder.block.12.layer.0.SelfAttention.k.weight', 'decoder.block.16.layer.2.DenseReluDense.wo.weight', 'decoder.block.17.layer.0.SelfAttention.q.weight', 'decoder.block.10.layer.0.SelfAttention.v.weight', 'decoder.block.13.layer.0.SelfAttention.o.weight', 'decoder.block.16.layer.1.EncDecAttention.q.weight', 'decoder.block.23.layer.1.EncDecAttention.k.weight', 'decoder.block.2.layer.0.SelfAttention.q.weight', 'decoder.block.1.layer.1.layer_norm.weight', 'decoder.block.7.layer.1.EncDecAttention.k.weight', 'decoder.block.16.layer.1.layer_norm.weight', 'decoder.block.12.layer.1.layer_norm.weight', 'decoder.block.14.layer.0.SelfAttention.q.weight', 'decoder.block.1.layer.2.DenseReluDense.wo.weight', 'decoder.block.7.layer.1.layer_norm.weight', 'decoder.block.11.layer.1.EncDecAttention.o.weight', 'decoder.block.12.layer.0.SelfAttention.q.weight', 'decoder.block.8.layer.2.DenseReluDense.wo.weight', 'decoder.block.2.layer.1.EncDecAttention.o.weight', 'decoder.block.21.layer.2.DenseReluDense.wi.weight', 'decoder.block.6.layer.1.EncDecAttention.q.weight', 'decoder.block.14.layer.2.DenseReluDense.wi.weight', 'decoder.block.3.layer.2.DenseReluDense.wi.weight', 'decoder.block.2.layer.0.SelfAttention.o.weight', 'decoder.block.5.layer.1.EncDecAttention.k.weight', 'decoder.block.14.layer.0.SelfAttention.k.weight', 'decoder.block.17.layer.1.EncDecAttention.q.weight', 'decoder.block.1.layer.0.SelfAttention.k.weight', 'decoder.block.11.layer.1.EncDecAttention.q.weight', 'decoder.block.17.layer.0.SelfAttention.k.weight', 'decoder.block.2.layer.0.layer_norm.weight', 'decoder.block.8.layer.1.layer_norm.weight', 'decoder.block.1.layer.1.EncDecAttention.v.weight', 'decoder.block.1.layer.0.SelfAttention.v.weight', 'decoder.block.11.layer.1.EncDecAttention.k.weight', 'decoder.block.18.layer.1.EncDecAttention.o.weight', 'decoder.block.14.layer.1.EncDecAttention.q.weight', 'decoder.block.7.layer.0.SelfAttention.o.weight', 'decoder.block.17.layer.1.layer_norm.weight', 'decoder.block.3.layer.0.SelfAttention.q.weight', 'decoder.block.15.layer.1.EncDecAttention.o.weight', 'decoder.block.10.layer.0.SelfAttention.o.weight', 'decoder.block.16.layer.2.DenseReluDense.wi.weight', 'decoder.block.11.layer.1.layer_norm.weight', 'decoder.block.0.layer.2.layer_norm.weight', 'decoder.block.3.layer.1.EncDecAttention.o.weight', 'decoder.block.0.layer.0.SelfAttention.o.weight', 'decoder.block.4.layer.1.EncDecAttention.k.weight', 'decoder.block.18.layer.2.DenseReluDense.wi.weight', 'decoder.block.4.layer.0.layer_norm.weight', 'decoder.block.15.layer.0.layer_norm.weight', 'decoder.block.0.layer.0.SelfAttention.k.weight', 'decoder.block.23.layer.0.SelfAttention.o.weight', 'decoder.block.4.layer.1.EncDecAttention.q.weight', 'decoder.block.4.layer.0.SelfAttention.k.weight', 'decoder.block.12.layer.1.EncDecAttention.q.weight', 'decoder.block.7.layer.0.SelfAttention.v.weight', 'decoder.block.3.layer.0.layer_norm.weight', 'decoder.block.21.layer.1.EncDecAttention.k.weight', 'decoder.block.2.layer.0.SelfAttention.k.weight', 'decoder.block.10.layer.1.EncDecAttention.k.weight', 'decoder.block.8.layer.0.layer_norm.weight', 'decoder.block.17.layer.1.EncDecAttention.v.weight', 'decoder.block.16.layer.0.SelfAttention.o.weight', 'decoder.block.9.layer.2.DenseReluDense.wi.weight', 'decoder.block.13.layer.2.DenseReluDense.wo.weight', 'decoder.block.19.layer.1.EncDecAttention.v.weight', 'decoder.block.19.layer.0.SelfAttention.q.weight', 'decoder.block.3.layer.0.SelfAttention.k.weight', 'decoder.block.10.layer.1.EncDecAttention.v.weight', 'decoder.block.10.layer.1.layer_norm.weight', 'decoder.block.18.layer.2.DenseReluDense.wo.weight', 'decoder.block.21.layer.1.EncDecAttention.q.weight', 'decoder.block.11.layer.0.SelfAttention.o.weight', 'decoder.block.2.layer.1.EncDecAttention.q.weight', 'decoder.block.2.layer.2.DenseReluDense.wi.weight', 'decoder.block.21.layer.0.SelfAttention.q.weight', 'decoder.block.9.layer.2.DenseReluDense.wo.weight', 'decoder.block.20.layer.2.layer_norm.weight', 'decoder.block.18.layer.1.EncDecAttention.v.weight', 'decoder.block.15.layer.2.layer_norm.weight', 'decoder.block.11.layer.0.layer_norm.weight', 'decoder.block.2.layer.1.EncDecAttention.v.weight', 'decoder.block.3.layer.0.SelfAttention.o.weight', 'decoder.block.19.layer.2.DenseReluDense.wi.weight', 'decoder.block.12.layer.0.SelfAttention.o.weight', 'decoder.block.16.layer.2.layer_norm.weight', 'decoder.block.10.layer.1.EncDecAttention.o.weight', 'decoder.block.4.layer.2.DenseReluDense.wi.weight', 'decoder.block.1.layer.0.layer_norm.weight', 'decoder.block.20.layer.0.SelfAttention.v.weight', 'decoder.block.16.layer.1.EncDecAttention.o.weight', 'decoder.block.2.layer.1.layer_norm.weight', 'decoder.block.4.layer.1.EncDecAttention.o.weight', 'decoder.block.3.layer.1.EncDecAttention.k.weight', 'decoder.block.6.layer.0.layer_norm.weight', 'decoder.block.18.layer.0.SelfAttention.q.weight', 'decoder.block.11.layer.2.DenseReluDense.wo.weight', 'decoder.block.18.layer.0.SelfAttention.k.weight', 'decoder.block.2.layer.1.EncDecAttention.k.weight', 'decoder.block.21.layer.1.EncDecAttention.v.weight', 'decoder.block.20.layer.1.layer_norm.weight', 'decoder.block.15.layer.2.DenseReluDense.wo.weight', 'decoder.block.13.layer.0.SelfAttention.k.weight', 'decoder.block.20.layer.1.EncDecAttention.o.weight', 'decoder.block.0.layer.2.DenseReluDense.wo.weight', 'decoder.block.14.layer.1.EncDecAttention.k.weight', 'decoder.block.16.layer.1.EncDecAttention.k.weight', 'decoder.final_layer_norm.weight', 'decoder.block.21.layer.0.layer_norm.weight', 'decoder.block.6.layer.2.layer_norm.weight', 'decoder.block.13.layer.1.layer_norm.weight', 'decoder.block.19.layer.1.EncDecAttention.k.weight', 'decoder.block.19.layer.0.SelfAttention.v.weight', 'decoder.block.18.layer.0.layer_norm.weight', 'decoder.block.16.layer.0.layer_norm.weight', 'decoder.block.16.layer.0.SelfAttention.q.weight', 'decoder.block.6.layer.2.DenseReluDense.wo.weight', 'decoder.block.5.layer.2.layer_norm.weight', 'decoder.block.12.layer.2.DenseReluDense.wo.weight', 'decoder.block.14.layer.0.SelfAttention.v.weight', 'decoder.block.20.layer.2.DenseReluDense.wi.weight', 'decoder.block.7.layer.2.layer_norm.weight', 'decoder.block.6.layer.0.SelfAttention.o.weight', 'decoder.block.5.layer.0.SelfAttention.q.weight', 'decoder.block.18.layer.1.EncDecAttention.k.weight', 'decoder.block.11.layer.0.SelfAttention.v.weight', 'decoder.block.0.layer.1.EncDecAttention.k.weight', 'decoder.block.2.layer.0.SelfAttention.v.weight', 'decoder.block.18.layer.2.layer_norm.weight', 'decoder.block.17.layer.0.SelfAttention.o.weight', 'decoder.block.10.layer.2.DenseReluDense.wi.weight', 'decoder.block.5.layer.0.layer_norm.weight', 'decoder.block.23.layer.0.SelfAttention.q.weight', 'decoder.block.19.layer.0.layer_norm.weight', 'lm_head.weight', 'decoder.block.0.layer.1.layer_norm.weight', 'decoder.block.6.layer.0.SelfAttention.k.weight', 'decoder.block.1.layer.1.EncDecAttention.o.weight', 'decoder.block.8.layer.0.SelfAttention.v.weight', 'decoder.block.11.layer.1.EncDecAttention.v.weight', 'decoder.block.23.layer.1.EncDecAttention.v.weight', 'decoder.block.5.layer.0.SelfAttention.k.weight', 'decoder.block.11.layer.0.SelfAttention.q.weight', 'decoder.block.8.layer.1.EncDecAttention.k.weight', 'decoder.block.22.layer.2.DenseReluDense.wo.weight', 'decoder.block.12.layer.1.EncDecAttention.k.weight', 'decoder.block.7.layer.2.DenseReluDense.wi.weight', 'decoder.block.9.layer.0.SelfAttention.q.weight', 'decoder.block.9.layer.1.EncDecAttention.v.weight', 'decoder.block.0.layer.0.SelfAttention.q.weight', 'decoder.block.22.layer.0.SelfAttention.k.weight', 'decoder.block.2.layer.2.layer_norm.weight', 'decoder.block.21.layer.0.SelfAttention.k.weight', 'decoder.block.6.layer.1.EncDecAttention.v.weight', 'decoder.block.1.layer.0.SelfAttention.q.weight', 'decoder.block.22.layer.1.EncDecAttention.k.weight', 'decoder.block.5.layer.0.SelfAttention.o.weight', 'decoder.block.0.layer.1.EncDecAttention.v.weight', 'decoder.block.15.layer.1.EncDecAttention.q.weight', 'decoder.block.17.layer.2.DenseReluDense.wo.weight', 'decoder.block.18.layer.0.SelfAttention.o.weight', 'decoder.block.0.layer.1.EncDecAttention.o.weight', 'decoder.block.23.layer.1.EncDecAttention.q.weight', 'decoder.block.9.layer.1.layer_norm.weight', 'decoder.block.8.layer.1.EncDecAttention.q.weight', 'decoder.block.17.layer.1.EncDecAttention.o.weight', 'decoder.block.3.layer.1.EncDecAttention.q.weight', 'decoder.block.5.layer.0.SelfAttention.v.weight', 'decoder.block.13.layer.0.SelfAttention.q.weight', 'decoder.block.13.layer.1.EncDecAttention.k.weight', 'decoder.block.15.layer.0.SelfAttention.o.weight', 'decoder.block.7.layer.0.SelfAttention.k.weight', 'decoder.block.23.layer.2.DenseReluDense.wo.weight', 'decoder.block.20.layer.0.SelfAttention.k.weight', 'decoder.block.23.layer.0.SelfAttention.v.weight', 'decoder.block.7.layer.1.EncDecAttention.q.weight', 'decoder.block.5.layer.1.EncDecAttention.v.weight', 'decoder.block.18.layer.1.layer_norm.weight', 'decoder.block.6.layer.1.EncDecAttention.o.weight', 'decoder.block.1.layer.2.DenseReluDense.wi.weight', 'decoder.block.7.layer.0.SelfAttention.q.weight', 'decoder.block.8.layer.0.SelfAttention.k.weight', 'decoder.block.3.layer.0.SelfAttention.v.weight', 'decoder.block.22.layer.2.layer_norm.weight', 'decoder.block.12.layer.1.EncDecAttention.v.weight', 'decoder.block.21.layer.2.DenseReluDense.wo.weight', 'decoder.block.6.layer.0.SelfAttention.v.weight', 'decoder.block.8.layer.0.SelfAttention.q.weight', 'decoder.block.0.layer.2.DenseReluDense.wi.weight', 'decoder.block.13.layer.0.SelfAttention.v.weight', 'decoder.block.0.layer.0.layer_norm.weight', 'decoder.block.0.layer.0.SelfAttention.relative_attention_bias.weight', 'decoder.block.8.layer.0.SelfAttention.o.weight', 'decoder.block.23.layer.1.layer_norm.weight', 'decoder.block.4.layer.1.layer_norm.weight', 'decoder.block.19.layer.1.layer_norm.weight', 'decoder.block.10.layer.1.EncDecAttention.q.weight', 'decoder.block.22.layer.1.layer_norm.weight', 'decoder.block.4.layer.2.DenseReluDense.wo.weight', 'decoder.block.0.layer.0.SelfAttention.v.weight', 'decoder.block.23.layer.2.DenseReluDense.wi.weight', 'decoder.block.14.layer.1.EncDecAttention.o.weight', 'decoder.block.14.layer.1.EncDecAttention.v.weight', 'decoder.block.1.layer.1.EncDecAttention.k.weight']\n",
      "- This IS expected if you are initializing T5EncoderModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing T5EncoderModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "All the weights of T5EncoderModel were initialized from the model checkpoint at Rostlab/prot_t5_xl_uniref50.\n",
      "If your task is similar to the task the model of the checkpoint was trained on, you can already use T5EncoderModel for predictions without further training.\n",
      "loading file spiece.model from cache at /data2/cache/huggingface/hub/models--Rostlab--prot_t5_xl_uniref50/snapshots/973be27c52ee6474de9c945952a8008aeb2a1a73/spiece.model\n",
      "loading file added_tokens.json from cache at None\n",
      "loading file special_tokens_map.json from cache at /data2/cache/huggingface/hub/models--Rostlab--prot_t5_xl_uniref50/snapshots/973be27c52ee6474de9c945952a8008aeb2a1a73/special_tokens_map.json\n",
      "loading file tokenizer_config.json from cache at /data2/cache/huggingface/hub/models--Rostlab--prot_t5_xl_uniref50/snapshots/973be27c52ee6474de9c945952a8008aeb2a1a73/tokenizer_config.json\n",
      "loading configuration file config.json from cache at /data2/cache/huggingface/hub/models--Rostlab--prot_t5_xl_uniref50/snapshots/973be27c52ee6474de9c945952a8008aeb2a1a73/config.json\n",
      "Model config T5Config {\n",
      "  \"_name_or_path\": \"Rostlab/prot_t5_xl_uniref50\",\n",
      "  \"architectures\": [\n",
      "    \"T5ForConditionalGeneration\"\n",
      "  ],\n",
      "  \"d_ff\": 16384,\n",
      "  \"d_kv\": 128,\n",
      "  \"d_model\": 1024,\n",
      "  \"decoder_start_token_id\": 0,\n",
      "  \"dense_act_fn\": \"relu\",\n",
      "  \"dropout_rate\": 0.1,\n",
      "  \"eos_token_id\": 1,\n",
      "  \"feed_forward_proj\": \"relu\",\n",
      "  \"initializer_factor\": 1.0,\n",
      "  \"is_encoder_decoder\": true,\n",
      "  \"is_gated_act\": false,\n",
      "  \"layer_norm_epsilon\": 1e-06,\n",
      "  \"model_type\": \"t5\",\n",
      "  \"n_positions\": 512,\n",
      "  \"num_decoder_layers\": 24,\n",
      "  \"num_heads\": 32,\n",
      "  \"num_layers\": 24,\n",
      "  \"output_past\": true,\n",
      "  \"pad_token_id\": 0,\n",
      "  \"relative_attention_max_distance\": 128,\n",
      "  \"relative_attention_num_buckets\": 32,\n",
      "  \"transformers_version\": \"4.26.1\",\n",
      "  \"use_cache\": true,\n",
      "  \"vocab_size\": 128\n",
      "}\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ProtT5_Classfier\n",
      "Trainable Parameter: 1209192449\n",
      "ProtT5_LoRA_Classfier\n",
      "Trainable Parameter: 3558401\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tokenizer, model_reload = load_model(\"./PT5_GB1_finetuned.pth\", num_labels=1, mixed=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5289780",
   "metadata": {},
   "source": [
    "To check if the original and the reloaded models are identical we can compare weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "e152714e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Models have identical weights\n"
     ]
    }
   ],
   "source": [
    "# Put both models to the same device\n",
    "model=model.to(\"cpu\")\n",
    "model_reload=model_reload.to(\"cpu\")\n",
    "\n",
    "# Iterate through the parameters of the two models and compare the data\n",
    "for param1, param2 in zip(model.parameters(), model_reload.parameters()):\n",
    "    if not torch.equal(param1.data, param2.data):\n",
    "        print(\"Models have different weights\")\n",
    "        break\n",
    "else:\n",
    "    print(\"Models have identical weights\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c478158",
   "metadata": {},
   "source": [
    "# Make predictions on a test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "8850480b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# For this we import the \"three_vs_rest\" GB1 dataset again \n",
    "# from https://github.com/J-SNACKKB/FLIP\n",
    "\n",
    "import requests\n",
    "import zipfile\n",
    "from io import BytesIO\n",
    "\n",
    "# Download the zip file from GitHub\n",
    "url = 'https://github.com/J-SNACKKB/FLIP/raw/main/splits/gb1/splits.zip'\n",
    "response = requests.get(url)\n",
    "zip_file = zipfile.ZipFile(BytesIO(response.content))\n",
    "\n",
    "# Load the `three_vs_rest.csv` file into a pandas dataframe\n",
    "with zip_file.open('splits/three_vs_rest.csv') as file:\n",
    "    df = pd.read_csv(file)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b896b40e",
   "metadata": {},
   "source": [
    "This time we take the test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "2466a7a5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                               sequence     label\n",
      "2990  MQYKLILNGKTLKGETTTEAVDAATAEKVFKQYANDNGAAAEWTYD...  1.611610\n",
      "2991  MQYKLILNGKTLKGETTTEAVDAATAEKVFKQYANDNGAACEWTYD...  3.741833\n",
      "2992  MQYKLILNGKTLKGETTTEAVDAATAEKVFKQYANDNGAAEEWTYD...  0.000000\n",
      "2993  MQYKLILNGKTLKGETTTEAVDAATAEKVFKQYANDNGAAFEWTYD...  1.081518\n",
      "2994  MQYKLILNGKTLKGETTTEAVDAATAEKVFKQYANDNGAAFEWTYD...  2.087250\n"
     ]
    }
   ],
   "source": [
    "# Select only test data\n",
    "my_test=df[df.set==\"test\"]\n",
    "\n",
    "# Set column names to \"sequence\" and \"label\"\n",
    "my_test.columns=[\"sequence\",\"label\"]+list(my_test.columns[2:])\n",
    "\n",
    "# Drop unneeded columns\n",
    "my_test=my_test[[\"sequence\",\"label\"]]\n",
    "print(my_test.head(5))\n",
    "\n",
    "# Preprocess sequences\n",
    "my_test[\"sequence\"]=my_test[\"sequence\"].str.replace('|'.join([\"O\",\"B\",\"U\",\"Z\"]),\"X\",regex=True)\n",
    "my_test['sequence']=my_test.apply(lambda row : \" \".join(row[\"sequence\"]), axis = 1)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d8b3625",
   "metadata": {},
   "source": [
    "Then we create predictions on our test data using the model we trained before"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "faf2feea",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█████████████████████████████████████████| 359/359 [04:41<00:00,  1.27it/s]\n"
     ]
    }
   ],
   "source": [
    "#Use reloaded model\n",
    "model = model_reload\n",
    "del model_reload\n",
    "\n",
    "# Set the device to use\n",
    "device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')\n",
    "model.to(device)\n",
    "\n",
    "# create Dataset\n",
    "test_set=create_dataset(tokenizer,list(my_test['sequence']),list(my_test['label']))\n",
    "# make compatible with torch DataLoader\n",
    "test_set = test_set.with_format(\"torch\", device=device)\n",
    "\n",
    "# Create a dataloader for the test dataset\n",
    "test_dataloader = DataLoader(test_set, batch_size=16, shuffle=False)\n",
    "\n",
    "# Put the model in evaluation mode\n",
    "model.eval()\n",
    "\n",
    "# Make predictions on the test dataset\n",
    "predictions = []\n",
    "with torch.no_grad():\n",
    "    for batch in tqdm(test_dataloader):\n",
    "        input_ids = batch['input_ids'].to(device)\n",
    "        attention_mask = batch['attention_mask'].to(device)\n",
    "        #add batch results(logits) to predictions\n",
    "        predictions += model(input_ids, attention_mask=attention_mask).logits.tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "520b90cd",
   "metadata": {},
   "source": [
    "Finally, we compute our desired performance metric for the test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "6246f740",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SpearmanrResult(correlation=0.85267829124194, pvalue=0.0)\n"
     ]
    }
   ],
   "source": [
    "# Regression\n",
    "print(stats.spearmanr(a=predictions, b=my_test.label, axis=0))\n",
    "\n",
    "# Classification\n",
    "# we need to determine the prediction class from the logit output\n",
    "# predictions= [item.argmax() for item in np.array(predictions)]\n",
    "# print(\"Accuracy: \", accuracy_score(my_test.label, predictions))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ad6ad00",
   "metadata": {},
   "source": [
    "Great, 0.85 Spearman is a decent test performance for this dataset (see results in [Table 4](https://openreview.net/pdf?id=p2dMLEwL8tF))\n",
    "\n",
    "More training epochs and / or hyperparameter optimization will most likely increase performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "993622d5",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "provenance": [
    {
     "file_id": "https://github.com/huggingface/notebooks/blob/main/examples/protein_language_modeling-tf.ipynb",
     "timestamp": 1670229986129
    }
   ]
  },
  "gpuClass": "standard",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "023504ef79df47cd9f5672b3537d781e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_f3f0b29c16df48f798dbdabc99207f2c",
       "IPY_MODEL_fb9ce840b4644d6eab03736688e57e23",
       "IPY_MODEL_dcb559ab088446f0a4cc9ae4720f8c29"
      ],
      "layout": "IPY_MODEL_cf11d66697d84676b9d8af11a2edd064"
     }
    },
    "07e6f8ff49d74ced96b87afaf99f52c0": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "0a7bab101d504c7cbe0c0e21222e4010": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_6e739bd554404481b4cfac9cd1710224",
      "max": 95,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_b66c522257d04820932c9c7014392136",
      "value": 95
     }
    },
    "0c9bd61e2e904fdaa4f578c19c0f6ea9": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "14ba9ccca4004a1e944767648e2b1253": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_d04e61e6abfe42b9afab583bcab66b92",
      "placeholder": "​",
      "style": "IPY_MODEL_279e487c47954c9abc0a57daab09f66a",
      "value": "Downloading: 100%"
     }
    },
    "14eaa8780fd543b38d5ea5da8e002b1e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_9a28a789e7814ee283b8ba922ecce252",
      "placeholder": "​",
      "style": "IPY_MODEL_3149e4afc57146ceac1fdd9a1902ccbc",
      "value": "Downloading: 100%"
     }
    },
    "15e24c603abb456984509d77813a0cf7": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_9ef0944bc4334b32a1a5e136b3fce9e6",
      "placeholder": "​",
      "style": "IPY_MODEL_c6407f5a50b34a0d9bfbd916ffb70d17",
      "value": " 134M/134M [00:01&lt;00:00, 79.0MB/s]"
     }
    },
    "1bdf17da40fb4d8687456e5379af5da9": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "1f5719ff48504d91968e1953107b060f": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "279e487c47954c9abc0a57daab09f66a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "2f1f8e5d1f5343978ccf1cc5dff59718": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "305e2c8af6194f4486b505ff25be86d7": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_611417b8118744c28ec2619c45b2a6ec",
      "max": 125,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_cd89abbac71740c5a013481dd9129f5b",
      "value": 125
     }
    },
    "3149e4afc57146ceac1fdd9a1902ccbc": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "334e77a8b6284dfb886fefadea6998ca": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "37dce533ad7648d0a09b3ef825c5aa31": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4d579e0c846b4e1bbc360d0ff2f2eccf": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "4db0117d9ac0453886964613627decc0": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "51b29673275343998567b43347eb591b": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "52a93c8ac4454ed1bf1a0d4711074bc2": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_d154e8e8a650447ab06425706d908f4e",
       "IPY_MODEL_0a7bab101d504c7cbe0c0e21222e4010",
       "IPY_MODEL_c8b35d958ecc4f3683b86278b979f300"
      ],
      "layout": "IPY_MODEL_4db0117d9ac0453886964613627decc0"
     }
    },
    "611417b8118744c28ec2619c45b2a6ec": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "6ae302430d7746edbc04b6786bdc53f5": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_14ba9ccca4004a1e944767648e2b1253",
       "IPY_MODEL_e499b443a3964dc2bca05692abde2dda",
       "IPY_MODEL_15e24c603abb456984509d77813a0cf7"
      ],
      "layout": "IPY_MODEL_1f5719ff48504d91968e1953107b060f"
     }
    },
    "6c3f28dcd36d434fa8e5fe38b1d34e3c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_8d4673cb688f4dacb1e96a0d80815667",
       "IPY_MODEL_305e2c8af6194f4486b505ff25be86d7",
       "IPY_MODEL_a6f664af44464825a9c8b73215b59318"
      ],
      "layout": "IPY_MODEL_2f1f8e5d1f5343978ccf1cc5dff59718"
     }
    },
    "6e739bd554404481b4cfac9cd1710224": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "728fbf87d2ff4bed925f67c9cf0e36b7": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "7a00dda470ef4f0893676c794e006a8f": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "8921dca0901741c5ab7723ca88803543": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_14eaa8780fd543b38d5ea5da8e002b1e",
       "IPY_MODEL_abee545f24074d33a2a171f140d84ca4",
       "IPY_MODEL_d2d78bbb74664973b1cde561152e9aad"
      ],
      "layout": "IPY_MODEL_fefc4de84c7847bf88635bfcad3a5f9e"
     }
    },
    "8d4673cb688f4dacb1e96a0d80815667": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_728fbf87d2ff4bed925f67c9cf0e36b7",
      "placeholder": "​",
      "style": "IPY_MODEL_97335231c9ab4668bc8cdaee57be202f",
      "value": "Downloading: 100%"
     }
    },
    "97335231c9ab4668bc8cdaee57be202f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "9a28a789e7814ee283b8ba922ecce252": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "9ef0944bc4334b32a1a5e136b3fce9e6": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "a6f664af44464825a9c8b73215b59318": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_fc1cf1de39dc45d69551ca2ca401a447",
      "placeholder": "​",
      "style": "IPY_MODEL_1bdf17da40fb4d8687456e5379af5da9",
      "value": " 125/125 [00:00&lt;00:00, 1.34kB/s]"
     }
    },
    "a7e27116c7b04f5e8980ea29c59bd9ce": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "a840d7d2dc0042c1955f064ef41e051d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "abee545f24074d33a2a171f140d84ca4": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_07e6f8ff49d74ced96b87afaf99f52c0",
      "max": 778,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_ba9fc00d9d144278bcf944a9328d5d5f",
      "value": 778
     }
    },
    "b66c522257d04820932c9c7014392136": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "ba9fc00d9d144278bcf944a9328d5d5f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "c555968966fb402e9ed39c024476f34b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "c6407f5a50b34a0d9bfbd916ffb70d17": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "c8b35d958ecc4f3683b86278b979f300": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_37dce533ad7648d0a09b3ef825c5aa31",
      "placeholder": "​",
      "style": "IPY_MODEL_e16a79b61af94733b44306e7b5444d29",
      "value": " 95.0/95.0 [00:00&lt;00:00, 3.19kB/s]"
     }
    },
    "cd89abbac71740c5a013481dd9129f5b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "cf11d66697d84676b9d8af11a2edd064": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d015260b083c40399f87d1a30be95f4f": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d04e61e6abfe42b9afab583bcab66b92": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d154e8e8a650447ab06425706d908f4e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_ef762a4251b6405f8065c3e4ce880688",
      "placeholder": "​",
      "style": "IPY_MODEL_4d579e0c846b4e1bbc360d0ff2f2eccf",
      "value": "Downloading: 100%"
     }
    },
    "d2d78bbb74664973b1cde561152e9aad": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_51b29673275343998567b43347eb591b",
      "placeholder": "​",
      "style": "IPY_MODEL_334e77a8b6284dfb886fefadea6998ca",
      "value": " 778/778 [00:00&lt;00:00, 26.3kB/s]"
     }
    },
    "dcb559ab088446f0a4cc9ae4720f8c29": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_eb8017bf12034a0297f17f0f3f5f1874",
      "placeholder": "​",
      "style": "IPY_MODEL_a840d7d2dc0042c1955f064ef41e051d",
      "value": " 93.0/93.0 [00:00&lt;00:00, 3.23kB/s]"
     }
    },
    "e16a79b61af94733b44306e7b5444d29": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "e499b443a3964dc2bca05692abde2dda": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_ebb2f370c71a470895096be1ab1eadbf",
      "max": 134360208,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_a7e27116c7b04f5e8980ea29c59bd9ce",
      "value": 134360208
     }
    },
    "eb8017bf12034a0297f17f0f3f5f1874": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ebb2f370c71a470895096be1ab1eadbf": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ef762a4251b6405f8065c3e4ce880688": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "f3f0b29c16df48f798dbdabc99207f2c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_7a00dda470ef4f0893676c794e006a8f",
      "placeholder": "​",
      "style": "IPY_MODEL_0c9bd61e2e904fdaa4f578c19c0f6ea9",
      "value": "Downloading: 100%"
     }
    },
    "fb9ce840b4644d6eab03736688e57e23": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_d015260b083c40399f87d1a30be95f4f",
      "max": 93,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_c555968966fb402e9ed39c024476f34b",
      "value": 93
     }
    },
    "fc1cf1de39dc45d69551ca2ca401a447": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "fefc4de84c7847bf88635bfcad3a5f9e": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}