notebooks/transfer_learning_hg/Fine_tunining_pretrained_LMs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Использование предобученных трансформеров"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# %pip install --upgrade keras\n",
    "# %pip install torch torchvision torchaudio"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Большинство SOTA моделей последних лет основаны на 1 нейросетевой архитектуре, которая называется Transformer. Эта архитектура была представлена в 2017 году в статье - [Attention is all you need](https://arxiv.org/abs/1706.03762). \n",
    "\n",
    "На тот момент в обработке последовательностей доминировали рекуррентные нейроные сети, но у них была проблема с \"забыванием\" информации при обработке длинных текстов. Для решения этой проблемы был придуман механизм внимания (attention), который позволял на каждом шаге учитывать агрегированную информацию от всех предыдущих и последующих элементов. LSTM+Attention некоторое время был sota подходом, но в attention is all you need было показано, что можно получить те же результаты, используя по сути только механизм внимания, без рекуррентности. \n",
    "\n",
    "Отсутствие рекуррентности позволяет гораздо эффективнее распараллеливать вычисления как на 1 gpu/tpu так и на кластере. На тот момент уже было несколько исследований показывающих, что при увеличении количества параметров и данных, модели начинают вести себя неожиданно хорошо (например, у OpenAI была статья в которой они рассказывали, что оставили LSTM на несколько месяцев и потом обнаружили, что она научилась достаточно точно определять тональность текста (без обучения на размеченном датасете) - https://openai.com/blog/unsupervised-sentiment-neuron/). Рекуррентные сети не позволяли серьезно увеличивать размер сетей даже при наличие ресурсов, а трансформеры позволили. \n",
    "\n",
    "Все последующие годы регулярно появлялись (и продолжают появляться) все более и более объемные трансформерные модели (BERT, GPT-1,2,3, Megatron, PaLM, Lambda etc). Есть попытки масштабировать RNN модели тоже, но успешных примеров сильно меньше доступных трансформеров (стоит упомянуть [RWKV](https://github.com/BlinkDL/RWKV-LM) как самый удачный). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Давайте попробуем чуть более подробно разобрать, как устроен трансформер. Это не простая тема и разбирать её мы будем постепенно.   \n",
    "**В этом семинаре посмотрим в целом на архитектуру и ее ключевые элементы, а также попробуем использовать предобученные трансформерные модели (а точнее BERT) для задачи классификации.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Seq2seq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Для начала вернемся к рекуррентным сетям. Их недостатки наиболее серьезно проявились в задаче seq2seq. В seq2seq по входной последовательности нужно сгенерировать целевую последовательность, а длины последовательностей при этом могут отличаться. Самый каноничный пример seq2seq задачи - машинный перевод.\n",
    "\n",
    "Использование LSTM* для машинного перевода можно схематически представить вот так:\n",
    "\n",
    "*_тут и далее когда я говорю LSTM я на самом деле подразумеваю рекуррентные нейроные сети в целом, просто LSTM самый популярный их представитель, но уточнять это каждый раз неудобно_\n",
    "\n",
    "![](https://i.ibb.co/XjP2Gmh/Lstm-seq2seq.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Для такой задачи нужно два LSTM слоя: один кодирует входную последовательность (encoder, или кодировщик), другой генерирует целевую последовательность (decoder или декодировщик). Такой подход работает, но есть серьезный недостаток - на вход декодеру подается только 1 вектор, в котором закодирована вся информация о целевой последовательности. Этот вектор - бутылочное горлышко (bottleneck), узкое место, в которое все упирается.\n",
    "\n",
    "Если приводить аналогии, это как пытаться первести текст сходу целиком, прочитав его только 1 раз (попробуйте, например, перевести это предложение, не перечитывая). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Attention"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Для устранения бутылочного горлышка стали использовать механизм внимания. Он дает декодирующей LSTM доступ ко всем состояниям энкодера, а не только к последнему. Это уже больше похоже на то, как человек переводит текст - текст прочитывается целиком, но при переводе можно возвращаться к каждому отдельному слову исходного текста.   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Схематично LSTM+Attention можно представить вот так. Такая архитектура уже в середине десятых годов позволила Google начать переход к нейронному переводу (до этого использовался статистический и правиловый):\n",
    "\n",
    "![](https://i.ibb.co/bzwNqwC/lstm-attention-seq2seq.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Давайте подробнее посмотрим на механизм внимания. Визуализировать его целиком достаточно сложно, поэтому рассмотрим только 1 шаг (генерацию первого слова в переводе). \n",
    "![](https://i.ibb.co/19xB94B/lstm-attention-1-step.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Само внимание (или attention) можно реализовать разными способами. Мы рассмотрим один возможный подход - через скалярное произведение. \n",
    "\n",
    "После прохода энкодера для каждого элемента входной последовательности существует вектор состояния (его можно считать контекстным эмбедингом слова). На каждом шаге декодера между текущим состоянием декодера и всеми состояними энкодера расчитывается скалярное произведение (можно сказать считается близость между переводным словом и всеми словами входного текста). Результатом такого скалярного произведения является набор весов, соответствующих каждому слову входной последовательности. Эти веса показывают, какие слова релеванты для генерации текущего слова в переводе. \n",
    "\n",
    "Далее каждый вес умножается на соотвутсвующий ему вектор состояния энкодера и эти вектора прибавляются к текущему состоянию декодера (то есть происходит взвешенная сумма векторов слов входной последовательности и итоговая сумма прибавляется или конкатенируется к состояние декодера). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Attention убрал существующий bottleneck, но обучать такие модели было очень тяжело, так как в основе был LSTM. В 2017 был предложен подход без использования рекуррентности - Transformer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transformer "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Вот как схематично решается задача машинного перевода с помощью трансформера. Самим трансформером называется блок с вниманием и полносвязными слоями, то есть и энкодер и декодер состоят из идентичных трансформерных блоков. Как и другие слои - трансформерные блоки можно накладывать друг на друга. Количество таких блоков в энкодере и декодере - настраиваемый гиперпараметр модели. \n",
    "![](https://i.ibb.co/XY0GK2R/transformer.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "В трансформерных блоках также есть heads, нормализация и skip-связи, но пока мы их пропустим. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Attention в трансформере конечно не такой простой как на примере выше. Давайте посмотрим на визуализацию из вот этого поста https://jalammar.github.io/illustrated-transformer/ (его имеет смысл почитать целиком)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Attention также вычисляется через скалаярное произведение, но оно считается не между векторами напрямую. \n",
    "\n",
    "Каждый эмбединг одтельного слова преобразуется в 3 отдельных вектора с помощью полносвязного слоя (умножения на матрицу). У них есть названия: query вектор, key вектор и value вектор.\n",
    "\n",
    "![](https://jalammar.github.io/images/t/transformer_self_attention_vectors.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Attention score вычисляется с помощью скалярного произведения query вектора текущего слова и key векторов всех слов в последовательности. Результат скалярного произведения пропускается через softmax, чтобы получить вероятности (это и есть оценки внимания, attention scores, высокая вероятность означает, что на это слово нужно обратить внимание, а низкий - что слово на текущем шаге не важно). \n",
    "\n",
    "Далее выполняется взвешенная сумма всех value векторов (веса = attention скоры). Полученный вектор уже передается дальше в полносвязный слой. При наслаивании транформерных блоков друг на друга на втором слое вместо эмбединга слова уже будет использоваться вектор полученный из предыдущего блока.\n",
    "\n",
    "![](https://jalammar.github.io/images/t/self-attention-output.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Но это еще не все. Это пример того, что называется self-attention - когда внимание расчитыватся между элементами одной последовательности. На схеме выше еще есть cross-attention. Он используется в декодере - query вектор тут получается из эмбедингов слов в целевой последовательности, а key и value вектора получаются из выходных векторов энкодера. Cross-attention связывает энкодер и декодер."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Self-attention также присутствует в декодере. При обучении seq2seq моделей на вход подаются сразу две последовательности (текст и его перевод, например), но целевая последовательность подается постепенно (по 1 слову) и задача декодера - по полной входной последовательности и имеющейся на данный момент целевой последовательности сгенерировать продолжение. При использовании обученной модели для перевода текста сначала в декодер передается пустая последовательсть (а точнее не пустая а с тегом SOS или start, можно тут вспомнить семинар по языковым моделям)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Positional embedding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Еще одна важная составляющая транформера - positinal encoding (на схеме вместо обычного emded - pos emded), позиционной кодирование. Механизм внимания ничего не знает о позиции слов, он рассматривает последовательность как мешок слов, а от рекурентности мы избавились. Поэтому нужно как-то закодировать информацию о последовательности в эмбединги слов. В Attention is all you need это было сделано простым добавлением к эмбедингу слова вектора, который зависит от индекса слова в последовательности. В статье они использовали периодичные функции (синус и косинус), которые не нужно обучать - они просто возвращают какое-то значение на каждый индекс последовательности. Количество уникальных векторов генерируемых таким образом конечно ограничено, но очень большое количество и не требуется, так как транформер сам по себе не очень хорошо масштабируется на длинные последовательности (обычно ограниченивают максимальную длину около 512)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Вот визуализация позиционных векторов из того же поста (изображено 20 векторов - по строкам, каждый размерности 512 - колонки; видно как вектора меняются при увеличении индекса):\n",
    "\n",
    "![](https://jalammar.github.io/images/t/transformer_positional_encoding_large_example.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "В трансформере еще много других технических деталей, но для использования пока будет достаточно интуитивного понимания."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Использование предобученных моделей"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Транформеры также можно обучать в self-supervised формате (как word2vec) и использовать готовую модель для transfer learning. В 2018 вышел BERT (https://arxiv.org/abs/1810.04805), который показал насколько хорошо эта идея работает. BERT - это трансформерная модель, которая обучалась восстанавливать пропуски в тексте и определять соседние предложения. Данные для обучения BERT генерируются синтетически из сырого текста (например, берется предложение из вики и в нем случайно удаляются 20% токенов и их нужно восстановить). Авторы показали, что если дообучение предобученной BERT модели позволяет получить state-of-the-art результаты в большом количестве задач. \n",
    "\n",
    "Самое замечательное в том, что BERT был выложен в открытый доступ и им можно легко воспользоваться.\n",
    "\n",
    "Давайте попробуем взять предобученные BERT и дообучить его решать задачу классификации текста."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# %pip install scikit-learn pandas matplotlib transformers -U"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.1.1\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "os.environ[\"KERAS_BACKEND\"] = \"torch\"\n",
    "\n",
    "import keras\n",
    "print(keras.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# стандартные библиотеки\n",
    "import os, re\n",
    "import numpy as np\n",
    "from time import time\n",
    "from sklearn.model_selection import train_test_split\n",
    "import pandas as pd\n",
    "from collections import Counter\n",
    "from string import punctuation\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "\n",
    "import keras, torch\n",
    "from transformers import AutoModel\n",
    "from transformers import AutoTokenizer\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Возьмем данные lenta.ru, но не целиком. Fine-tuning больших моделей лучше всего подходит, когда данных совсем мало и стандартным алгоритмам просто не хватает информации, чтобы обучиться. Поэтому возьмем только небольшой процент всех данных."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "# data_frac = pd.concat([data[data.topic=='Россия'].sample(frac=0.001),\n",
    "#                       data[data.topic=='Мир'].sample(frac=0.001),\n",
    "#                       data[data.topic=='Экономика'].sample(frac=0.002),\n",
    "#                       data[data.topic=='Спорт'].sample(frac=0.003),\n",
    "#                       data[data.topic=='Наука и техника'].sample(frac=0.005),\n",
    "#                       data[data.topic=='Бывший СССР'].sample(frac=0.005),\n",
    "#                       data[data.topic=='Культура'].sample(frac=0.005),\n",
    "#                       data[data.topic=='Интернет и СМИ'].sample(frac=0.005),\n",
    "#                       data[data.topic=='Из жизни'].sample(frac=0.01),\n",
    "#                       data[data.topic=='Дом'].sample(frac=0.01),\n",
    "#                       data[data.topic=='Силовые структуры'].sample(frac=0.01),\n",
    "#                       data[data.topic=='Ценности'].sample(frac=0.03),\n",
    "#                       data[data.topic=='Бизнес'].sample(frac=0.03),\n",
    "#                       data[data.topic=='69-я параллель'].sample(frac=0.05),\n",
    "#                       data[data.topic=='Крым'].sample(frac=0.05),\n",
    "#                       data[data.topic=='Культпросвет'].sample(frac=0.2),\n",
    "#                       data[data.topic=='Легпром'].sample(frac=0.8),\n",
    "#                       data[data.topic=='Библиотека']]).reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.read_csv('lenta_sample.csv')\n",
    "data.dropna(subset=['topic', 'text'], inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "topic\n",
       "Из жизни             55\n",
       "Наука и техника      54\n",
       "Бывший СССР          54\n",
       "Культура             53\n",
       "Ценности             45\n",
       "Дом                  45\n",
       "Бизнес               44\n",
       "Интернет и СМИ       44\n",
       "Силовые структуры    40\n",
       "Спорт                39\n",
       "Россия               32\n",
       "Экономика            32\n",
       "Мир                  27\n",
       "69-я параллель       13\n",
       "Легпром              13\n",
       "Библиотека           10\n",
       "Крым                  7\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.topic.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(607, 6)"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Загружаем предобученную модель из huggingface transformers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Список всех доступных моделей можно найти тут - https://huggingface.co/models  \n",
    "А вот тут основные с описанием - https://huggingface.co/transformers/pretrained_models.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "На HG предпочитают торч, поэтому многие модели не загрузятся в тф. Там есть специальный тэг, по которому можно фильтровать модели. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Другие виды моделей мы обсудим на следующем занятии, так как их очень много и они существенно отличаются друг от друга. Пока разберем 3 варианта BERT. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Можно сказать оригинальный берт\n",
    "# tokenizer = AutoTokenizer.from_pretrained('bert-base-cased')\n",
    "# model_bert = AutoModel.from_pretrained('bert-base-cased')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # # # Мультиязычный берт\n",
    "# tokenizer = AutoTokenizer.from_pretrained('bert-base-multilingual-uncased')\n",
    "# model_bert = AutoModel.from_pretrained('bert-base-multilingual-uncased')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # # # дистилированный (т.е. уменьшенный) берт \n",
    "tokenizer = AutoTokenizer.from_pretrained(\"distilbert-base-multilingual-cased\")\n",
    "model_bert = AutoModel.from_pretrained(\"distilbert-base-multilingual-cased\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Мы загружаем не только модель, а еще и токенайзер, т.е. свою предобработку нам писать не нужно"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'По факту пожара на\\xa0заводе в\\xa0Югре заведено дело'"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.loc[4, 'title']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Перевести токены в индексы очень просто"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[101,\n",
       " 11480,\n",
       " 61481,\n",
       " 10227,\n",
       " 10297,\n",
       " 62323,\n",
       " 10179,\n",
       " 10122,\n",
       " 69060,\n",
       " 543,\n",
       " 539,\n",
       " 63596,\n",
       " 10205,\n",
       " 10234,\n",
       " 48467,\n",
       " 29112,\n",
       " 102]"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.encode(data.loc[4, 'title'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "В этих моделях как правило используется BPE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['[CLS]',\n",
       " 'По',\n",
       " 'факт',\n",
       " '##у',\n",
       " 'по',\n",
       " '##жар',\n",
       " '##а',\n",
       " 'на',\n",
       " 'заводе',\n",
       " 'в',\n",
       " 'Ю',\n",
       " '##гр',\n",
       " '##е',\n",
       " 'за',\n",
       " '##ведено',\n",
       " 'дело',\n",
       " '[SEP]']"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# переводим индекс токена обратно в текст\n",
    "encoded = tokenizer.encode(data.loc[4, 'title'])\n",
    "[tokenizer.decode([x]) for x in encoded]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Индексы можно напрямую передавать в модель."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = data.loc[4, 'title']\n",
    "text_ids = tokenizer.encode(text, add_special_tokens=True, \n",
    "                            return_tensors='pt') # преобразуем в торч формат \n",
    "output = model_bert(text_ids, output_attentions=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "На выходе мы получим tuple из двух элементов. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Первый элемент - состояния енкодера для каждого из элементов последовательности"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 17, 768])"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output[0].size() # в пайторче вместо .shape используется size()\n",
    "output[0].shape # но shape тоже работает"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Второй - состояние енкодера на первом элементе, пропущенное через активацию (обычно этот элемент не используют)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "# в дистилберт его совсем нет, поэтому будет ошибка\n",
    "# output[1].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Обычно в задачах используют либо состояние первого элемента"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 768])"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output[0][:,0].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Либо усредненное состояние "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 768])"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output[0].mean(axis=1).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Полученные эмбеддинги уже можно использовать для какой-нибудь кластеризации или поиска похожих. А если есть разметка, то можно обучить на этих векторах стандартную модель из sklearn или даже дообучить всего Берта под конкретную задачу!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Давайте попробуем дообучить (fine-tune) модель на данных lenta ru"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Будем обучаться на заголовках, а не на самих текстах"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "# переводим тексты в последовательности индексов\n",
    "# У берта есть ограничение на максимальную длину последовательности - 512\n",
    "# Для заголовков это не очень актуально, но для других данных пригодится\n",
    "\n",
    "X = []\n",
    "\n",
    "for text in data.title:\n",
    "    ids = tokenizer.encode(text)\n",
    "    \n",
    "    X.append(ids[:512])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = keras.preprocessing.sequence.pad_sequences(X, padding='post', maxlen=512)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "id2label = {i:label for i, label in enumerate(set(data.topic.values))}\n",
    "label2id = {l:i for i, l in id2label.items()}\n",
    "\n",
    "y = keras.utils.to_categorical([label2id[label] for label in data.topic.values])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(607, 512)"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "В этот раз добавим стратификацию, т.к. в данных у нас дисбаланс классов."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_index, valid_index = train_test_split(list(range(len(X))), test_size=0.05, stratify=data.topic)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, y_train = X[train_index], y[train_index]\n",
    "X_valid, y_valid = X[valid_index], y[valid_index]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Загруженную модель в tf.keras можно использовать просто как слой в нейронной сети. Давайте сделаем новую модель, которая будет принимать на вход индексы символьных нграмм, пропускать их через берт, и затем брать первый вектор из энкодера и классифицировать его."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "У нас есть два варианты дообучения - обучать только последний классификатор, не трогая берт, либо обучать все целиком, используя предобученной состояние энкодера как отправную точку."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Дообучение с замораживанием предобученной модели"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "# если использовать tf как бекенд то заморозить веса можно специальным параметром\n",
    "# model_bert = AutoModel.from_pretrained('distilbert-base-multilingual-cased', trainable=False)\n",
    "\n",
    "model_bert = AutoModel.from_pretrained(\"distilbert-base-multilingual-cased\")\n",
    "# в торче веса замораживаются вот так\n",
    "# если нужно заморозить не все, то нужно просто добавить условия в цикл\n",
    "for param in model_bert.parameters():\n",
    "    param.requires_grad = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [],
   "source": [
    "# с huggingface немного сложно, потому что он возвращает тип BaseModelOutput \n",
    "# который является tuple как мы посмотрели выше\n",
    "# и это немного ломает расчет размерностей в керасе потому что он пытается вытащить эту размерность \n",
    "# по атрибуту .shape а BaseModelOutput нет такого\n",
    "# этот атрибут есть у элементов этого tuple поэтому нужно написать вот такую простую обертку\n",
    "# которая вызовет модель и вернет первый элемент tuple\n",
    "class bert_wrapper(keras.Model):\n",
    "    def __init__(self, model_bert, **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "        # модель нужно обернуть в специальный класс чтобы она была совместима с другими керас слоями\n",
    "        self.model_bert = keras.layers.TorchModuleWrapper(model_bert)\n",
    "        \n",
    "    def call(self, inputs):\n",
    "        x = self.model_bert(inputs)\n",
    "        return x[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "input_word_ids = keras.layers.Input(shape=(512,), dtype='int32', \n",
    "                                     name=\"input_ids\")\n",
    "\n",
    "\n",
    "output = bert_wrapper(model_bert)(input_word_ids)\n",
    "# добавим дропаут чтобы не переобучалось\n",
    "drop = keras.layers.Dropout(0.1)(output[:, 0]) # [0][:, 0] - означает что мы берем первое \n",
    "                                                     # состояние у всех текстов в батче\n",
    "dense = keras.layers.Dense(y.shape[1], activation='softmax')(drop)\n",
    "\n",
    "model_clf = keras.Model(inputs=[input_word_ids], outputs=dense)\n",
    "\n",
    "# обычно при дообучении трансформеров нужно ставить очень маленький лосс\n",
    "# но когда мы обучаем только итоговый классификатор то можно ставить побольше\n",
    "model_clf.compile(keras.optimizers.Adam(learning_rate=1e-3), \n",
    "                  loss='categorical_crossentropy', \n",
    "                  metrics=['accuracy', \n",
    "                           keras.metrics.RecallAtPrecision(0.80, name='rec_prec')])\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Model: \"functional_3\"</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1mModel: \"functional_3\"\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\"> Layer (type)                    </span>┃<span style=\"font-weight: bold\"> Output Shape           </span>┃<span style=\"font-weight: bold\">       Param # </span>┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n",
       "│ input_ids (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)          │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">512</span>)            │             <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ bert_wrapper_3 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">bert_wrapper</span>)   │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">512</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">768</span>)       │   <span style=\"color: #00af00; text-decoration-color: #00af00\">134,734,080</span> │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ get_item_3 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">GetItem</span>)            │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">768</span>)            │             <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ dropout_3 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dropout</span>)             │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">768</span>)            │             <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ dense_2 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dense</span>)                 │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">17</span>)             │        <span style=\"color: #00af00; text-decoration-color: #00af00\">13,073</span> │\n",
       "└─────────────────────────────────┴────────────────────────┴───────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1mLayer (type)                   \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mOutput Shape          \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m      Param #\u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n",
       "│ input_ids (\u001b[38;5;33mInputLayer\u001b[0m)          │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m512\u001b[0m)            │             \u001b[38;5;34m0\u001b[0m │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ bert_wrapper_3 (\u001b[38;5;33mbert_wrapper\u001b[0m)   │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m512\u001b[0m, \u001b[38;5;34m768\u001b[0m)       │   \u001b[38;5;34m134,734,080\u001b[0m │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ get_item_3 (\u001b[38;5;33mGetItem\u001b[0m)            │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m768\u001b[0m)            │             \u001b[38;5;34m0\u001b[0m │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ dropout_3 (\u001b[38;5;33mDropout\u001b[0m)             │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m768\u001b[0m)            │             \u001b[38;5;34m0\u001b[0m │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ dense_2 (\u001b[38;5;33mDense\u001b[0m)                 │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m17\u001b[0m)             │        \u001b[38;5;34m13,073\u001b[0m │\n",
       "└─────────────────────────────────┴────────────────────────┴───────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Total params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">134,747,153</span> (514.02 MB)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Total params: \u001b[0m\u001b[38;5;34m134,747,153\u001b[0m (514.02 MB)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">13,073</span> (51.07 KB)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m13,073\u001b[0m (51.07 KB)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Non-trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">134,734,080</span> (513.97 MB)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m134,734,080\u001b[0m (513.97 MB)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model_clf.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "if torch.cuda.is_available():\n",
    "    model_clf.to('cuda')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "С батчсайзом тут приходится уже быть осторожным, т.к. берт занимает очень много места на видеокарте. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "\u001b[1m29/29\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m3s\u001b[0m 97ms/step - accuracy: 0.0968 - loss: 2.8472 - rec_prec: 0.0000e+00 - val_accuracy: 0.0968 - val_loss: 2.6984 - val_rec_prec: 0.0000e+00\n",
      "Epoch 2/20\n",
      "\u001b[1m29/29\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m3s\u001b[0m 98ms/step - accuracy: 0.1082 - loss: 2.7129 - rec_prec: 0.0000e+00 - val_accuracy: 0.1290 - val_loss: 2.6888 - val_rec_prec: 0.0000e+00\n",
      "Epoch 3/20\n",
      "\u001b[1m29/29\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m3s\u001b[0m 98ms/step - accuracy: 0.0990 - loss: 2.7349 - rec_prec: 0.0017 - val_accuracy: 0.0968 - val_loss: 2.7035 - val_rec_prec: 0.0000e+00\n",
      "Epoch 4/20\n",
      "\u001b[1m29/29\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m3s\u001b[0m 98ms/step - accuracy: 0.0981 - loss: 2.7199 - rec_prec: 0.0000e+00 - val_accuracy: 0.0968 - val_loss: 2.6675 - val_rec_prec: 0.0000e+00\n",
      "Epoch 5/20\n",
      "\u001b[1m29/29\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m3s\u001b[0m 98ms/step - accuracy: 0.1048 - loss: 2.7227 - rec_prec: 0.0000e+00 - val_accuracy: 0.1290 - val_loss: 2.6889 - val_rec_prec: 0.0000e+00\n",
      "Epoch 6/20\n",
      "\u001b[1m29/29\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m3s\u001b[0m 98ms/step - accuracy: 0.0996 - loss: 2.6974 - rec_prec: 0.0000e+00 - val_accuracy: 0.0968 - val_loss: 2.6485 - val_rec_prec: 0.0645\n",
      "Epoch 7/20\n",
      "\u001b[1m29/29\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m3s\u001b[0m 98ms/step - accuracy: 0.0893 - loss: 2.6891 - rec_prec: 0.0018 - val_accuracy: 0.1290 - val_loss: 2.6485 - val_rec_prec: 0.0000e+00\n",
      "Epoch 8/20\n",
      "\u001b[1m29/29\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m3s\u001b[0m 98ms/step - accuracy: 0.1191 - loss: 2.6657 - rec_prec: 4.1667e-04 - val_accuracy: 0.0968 - val_loss: 2.6658 - val_rec_prec: 0.0000e+00\n",
      "Epoch 9/20\n",
      "\u001b[1m29/29\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m3s\u001b[0m 98ms/step - accuracy: 0.1231 - loss: 2.6861 - rec_prec: 2.0221e-04 - val_accuracy: 0.1290 - val_loss: 2.6154 - val_rec_prec: 0.0000e+00\n",
      "Epoch 10/20\n",
      "\u001b[1m29/29\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m3s\u001b[0m 98ms/step - accuracy: 0.1409 - loss: 2.6665 - rec_prec: 0.0060 - val_accuracy: 0.0968 - val_loss: 2.6257 - val_rec_prec: 0.0645\n",
      "Epoch 11/20\n",
      "\u001b[1m29/29\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m3s\u001b[0m 98ms/step - accuracy: 0.1211 - loss: 2.6381 - rec_prec: 0.0035 - val_accuracy: 0.0645 - val_loss: 2.6285 - val_rec_prec: 0.0000e+00\n",
      "Epoch 12/20\n",
      "\u001b[1m29/29\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m3s\u001b[0m 98ms/step - accuracy: 0.1891 - loss: 2.6274 - rec_prec: 0.0033 - val_accuracy: 0.0645 - val_loss: 2.6109 - val_rec_prec: 0.0645\n",
      "Epoch 13/20\n",
      "\u001b[1m29/29\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m3s\u001b[0m 98ms/step - accuracy: 0.1628 - loss: 2.6298 - rec_prec: 0.0014 - val_accuracy: 0.1613 - val_loss: 2.5901 - val_rec_prec: 0.0000e+00\n",
      "Epoch 14/20\n",
      "\u001b[1m29/29\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m3s\u001b[0m 99ms/step - accuracy: 0.1639 - loss: 2.6018 - rec_prec: 0.0056 - val_accuracy: 0.1613 - val_loss: 2.6105 - val_rec_prec: 0.0645\n",
      "Epoch 15/20\n",
      "\u001b[1m29/29\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m3s\u001b[0m 99ms/step - accuracy: 0.1681 - loss: 2.5765 - rec_prec: 0.0010 - val_accuracy: 0.2258 - val_loss: 2.6232 - val_rec_prec: 0.0000e+00\n",
      "Epoch 16/20\n",
      "\u001b[1m29/29\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m3s\u001b[0m 98ms/step - accuracy: 0.1813 - loss: 2.6262 - rec_prec: 0.0000e+00 - val_accuracy: 0.1290 - val_loss: 2.5773 - val_rec_prec: 0.0645\n",
      "Epoch 17/20\n",
      "\u001b[1m29/29\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m3s\u001b[0m 98ms/step - accuracy: 0.2050 - loss: 2.5835 - rec_prec: 0.0071 - val_accuracy: 0.1935 - val_loss: 2.6035 - val_rec_prec: 0.0000e+00\n",
      "Epoch 18/20\n",
      "\u001b[1m29/29\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m3s\u001b[0m 98ms/step - accuracy: 0.1846 - loss: 2.5720 - rec_prec: 0.0078 - val_accuracy: 0.1290 - val_loss: 2.5781 - val_rec_prec: 0.0000e+00\n",
      "Epoch 19/20\n",
      "\u001b[1m29/29\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m3s\u001b[0m 98ms/step - accuracy: 0.1686 - loss: 2.5773 - rec_prec: 0.0055 - val_accuracy: 0.1935 - val_loss: 2.5809 - val_rec_prec: 0.0000e+00\n",
      "Epoch 20/20\n",
      "\u001b[1m29/29\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m3s\u001b[0m 98ms/step - accuracy: 0.2159 - loss: 2.5216 - rec_prec: 0.0086 - val_accuracy: 0.1935 - val_loss: 2.5785 - val_rec_prec: 0.0000e+00\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.src.callbacks.history.History at 0x7f30c9d65180>"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# если есть ошибка с device meta то попробуйте запустить еще раз\n",
    "model_clf.fit(torch.LongTensor(X_train), torch.LongTensor(y_train), \n",
    "          validation_data=(torch.LongTensor(X_valid), torch.LongTensor(y_valid)),\n",
    "          batch_size=20,\n",
    "          epochs=20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['accuracy', 'loss', 'rec_prec', 'val_accuracy', 'val_loss', 'val_rec_prec'])\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkAAAAHHCAYAAABXx+fLAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/H5lhTAAAACXBIWXMAAA9hAAAPYQGoP6dpAACZOElEQVR4nOzdd3hUVfrA8e/MpHdCQkJCIPTemyAIKlJFQUBgUZpixYa6wu4K+1t3F1R0sSBYFsWCoCCuBVFEQAWk904oAdKB9D5zf3/czCQhdSYzcyfJ+3meeebmzp07584Q5s0573mPTlEUBSGEEEKIekSvdQOEEEIIIZxNAiAhhBBC1DsSAAkhhBCi3pEASAghhBD1jgRAQgghhKh3JAASQgghRL0jAZAQQggh6h0JgIQQQghR70gAJIQQQoh6RwIgIYRTXbhwAZ1Ox0cffWT1c7du3YpOp2Pr1q12b5cQon6RAEgIIYQQ9Y4EQEIIIYSodyQAEkIIjWVlZWndBCHqHQmAhKhn/v73v6PT6Th9+jT33XcfgYGBhIaG8uKLL6IoCpcuXeLuu+8mICCA8PBwXnvttTLnSEpK4oEHHiAsLAwvLy+6du3KypUryxyXmprK9OnTCQwMJCgoiGnTppGamlpuu06ePMn48eMJDg7Gy8uLXr168c0339h0jRcvXuSxxx6jbdu2eHt707BhQyZMmMCFCxfKbeMzzzxDdHQ0np6eNGnShKlTp5KSkmI5Jjc3l7///e+0adMGLy8vGjduzD333ENMTAxQcW5SeflO06dPx8/Pj5iYGEaOHIm/vz9TpkwB4LfffmPChAk0bdoUT09PoqKieOaZZ8jJySn3/br33nsJDQ3F29ubtm3b8te//hWALVu2oNPpWL9+fZnnrVq1Cp1Ox86dO619W4WoU9y0boAQQhsTJ06kffv2LFq0iO+//55//vOfBAcH8+6773Lbbbfx8ssv89lnn/Hcc8/Ru3dvbrnlFgBycnIYPHgwZ8+eZfbs2TRv3pwvv/yS6dOnk5qaylNPPQWAoijcfffd/P777zzyyCO0b9+e9evXM23atDJtOXbsGDfffDORkZHMnTsXX19fvvjiC8aMGcO6desYO3asVde2Z88eduzYwaRJk2jSpAkXLlxg2bJlDB48mOPHj+Pj4wNAZmYmAwcO5MSJE8ycOZMePXqQkpLCN998w+XLlwkJCcFoNHLnnXeyefNmJk2axFNPPUVGRgabNm3i6NGjtGzZ0ur3vrCwkGHDhjFgwAAWL15sac+XX35JdnY2jz76KA0bNmT37t289dZbXL58mS+//NLy/MOHDzNw4EDc3d156KGHiI6OJiYmhm+//ZZ//etfDB48mKioKD777LMy791nn31Gy5Yt6devn9XtFqJOUYQQ9cqCBQsUQHnooYcs+woLC5UmTZooOp1OWbRokWX/9evXFW9vb2XatGmWfUuWLFEA5dNPP7Xsy8/PV/r166f4+fkp6enpiqIoytdff60AyiuvvFLqdQYOHKgAyocffmjZf/vttyudO3dWcnNzLftMJpPSv39/pXXr1pZ9W7ZsUQBly5YtlV5jdnZ2mX07d+5UAOXjjz+27Js/f74CKF999VWZ400mk6IoirJixQoFUF5//fUKj6moXefPny9zrdOmTVMAZe7cudVq98KFCxWdTqdcvHjRsu+WW25R/P39S+0r2R5FUZR58+Ypnp6eSmpqqmVfUlKS4ubmpixYsKDM6whR38gQmBD11IMPPmjZNhgM9OrVC0VReOCBByz7g4KCaNu2LefOnbPs27BhA+Hh4UyePNmyz93dnSeffJLMzEy2bdtmOc7NzY1HH3201Os88cQTpdpx7do1fvnlF+69914yMjJISUkhJSWFq1evMmzYMM6cOcOVK1esujZvb2/LdkFBAVevXqVVq1YEBQWxf/9+y2Pr1q2ja9eu5fYw6XQ6yzEhISFl2l3yGFuUfF/Ka3dWVhYpKSn0798fRVE4cOAAAMnJyfz666/MnDmTpk2bVtieqVOnkpeXx9q1ay371qxZQ2FhIffdd5/N7RairpAASIh66sYvz8DAQLy8vAgJCSmz//r165afL168SOvWrdHrS//30b59e8vj5vvGjRvj5+dX6ri2bduW+vns2bMoisKLL75IaGhoqduCBQsANefIGjk5OcyfP5+oqCg8PT0JCQkhNDSU1NRU0tLSLMfFxMTQqVOnSs8VExND27ZtcXOzX8aAm5sbTZo0KbM/NjaW6dOnExwcjJ+fH6GhoQwaNAjA0m5zMFpVu9u1a0fv3r357LPPLPs+++wzbrrpJlq1amWvSxGi1pIcICHqKYPBUK19oObzOIrJZALgueeeY9iwYeUeY+0X9hNPPMGHH37I008/Tb9+/QgMDESn0zFp0iTL69lTRT1BRqOx3P2enp5lAkij0cgdd9zBtWvXeOGFF2jXrh2+vr5cuXKF6dOn29TuqVOn8tRTT3H58mXy8vL4448/ePvtt60+jxB1kQRAQgirNGvWjMOHD2MymUp9iZ88edLyuPl+8+bNZGZmluoFOnXqVKnztWjRAlCH0YYMGWKXNq5du5Zp06aVmsGWm5tbZgZay5YtOXr0aKXnatmyJbt27aKgoAB3d/dyj2nQoAFAmfObe8Oq48iRI5w+fZqVK1cydepUy/5NmzaVOs78flXVboBJkyYxZ84cPv/8c3JycnB3d2fixInVbpMQdZkMgQkhrDJy5EgSEhJYs2aNZV9hYSFvvfUWfn5+liGbkSNHUlhYyLJlyyzHGY1G3nrrrVLna9SoEYMHD+bdd98lPj6+zOslJydb3UaDwVCm1+qtt94q0yMzbtw4Dh06VO50cfPzx40bR0pKSrk9J+ZjmjVrhsFg4Ndffy31+DvvvGNVm0ue07z9xhtvlDouNDSUW265hRUrVhAbG1tue8xCQkIYMWIEn376KZ999hnDhw8vM8QpRH0lPUBCCKs89NBDvPvuu0yfPp19+/YRHR3N2rVr2b59O0uWLMHf3x+A0aNHc/PNNzN37lwuXLhAhw4d+Oqrr0rl4JgtXbqUAQMG0LlzZ2bNmkWLFi1ITExk586dXL58mUOHDlnVxjvvvJNPPvmEwMBAOnTowM6dO/n5559p2LBhqeOef/551q5dy4QJE5g5cyY9e/bk2rVrfPPNNyxfvpyuXbsydepUPv74Y+bMmcPu3bsZOHAgWVlZ/Pzzzzz22GPcfffdBAYGMmHCBN566y10Oh0tW7bku+++syp3qV27drRs2ZLnnnuOK1euEBAQwLp160rlX5m9+eabDBgwgB49evDQQw/RvHlzLly4wPfff8/BgwdLHTt16lTGjx8PwEsvvWTV+yhEnabV9DMhhDbM0+CTk5NL7Z82bZri6+tb5vhBgwYpHTt2LLUvMTFRmTFjhhISEqJ4eHgonTt3LjXV2+zq1avK/fffrwQEBCiBgYHK/fffrxw4cKDM1HBFUZSYmBhl6tSpSnh4uOLu7q5ERkYqd955p7J27VrLMdWdBn/9+nVL+/z8/JRhw4YpJ0+eVJo1a1ZqSr+5jbNnz1YiIyMVDw8PpUmTJsq0adOUlJQUyzHZ2dnKX//6V6V58+aKu7u7Eh4erowfP16JiYmxHJOcnKyMGzdO8fHxURo0aKA8/PDDytGjR8udBl/e+6woinL8+HFlyJAhip+fnxISEqLMmjVLOXToULnv19GjR5WxY8cqQUFBipeXl9K2bVvlxRdfLHPOvLw8pUGDBkpgYKCSk5NT6fsmRH2iUxQHZjcKIYTQVGFhIREREYwePZr//ve/WjdHCJchOUBCCFGHff311yQnJ5dKrBZCgPQACSFEHbRr1y4OHz7MSy+9REhISKkCkEII6QESQog6admyZTz66KM0atSIjz/+WOvmCOFypAdICCGEEPWO9AAJIYQQot6RAEgIIYQQ9Y4UQiyHyWQiLi4Of3//Gq32LIQQQgjnURSFjIwMIiIiyqy3dyMJgMoRFxdHVFSU1s0QQgghhA0uXbpEkyZNKj1GAqBymEv5X7p0iYCAAI1bI4QQQojqSE9PJyoqyvI9XhkJgMphHvYKCAiQAEgIIYSoZaqTviJJ0EIIIYSodyQAEkIIIUS9IwGQEEIIIeodyQGqAaPRSEFBgdbNqJXc3d0xGAxaN0MIIUQ9JQGQDRRFISEhgdTUVK2bUqsFBQURHh4utZaEEEI4nQRANjAHP40aNcLHx0e+wK2kKArZ2dkkJSUB0LhxY41bJIQQor6RAMhKRqPREvw0bNhQ6+bUWt7e3gAkJSXRqFEjGQ4TQgjhVJIEbSVzzo+Pj4/GLan9zO+h5FEJIYRwNgmAbCTDXjUn76EQQgitSAAkhBBCiHpHAiBhk+joaJYsWaJ1M4QQQgibSBJ0PTJ48GC6detml8Blz549+Pr61rxRQgghhAYkABIWiqJgNBpxc6v6n0VoaKgTWiSEqNcK80GnB4N8VQn7kyGwemL69Ols27aNN954A51Oh06n46OPPkKn0/HDDz/Qs2dPPD09+f3334mJieHuu+8mLCwMPz8/evfuzc8//1zqfDcOgel0Oj744APGjh2Lj48PrVu35ptvvnHyVQoh6oyCHHirB3xwOyiK1q0RdZAEQHagKArZ+YVOvylW/Kfwxhtv0K9fP2bNmkV8fDzx8fFERUUBMHfuXBYtWsSJEyfo0qULmZmZjBw5ks2bN3PgwAGGDx/O6NGjiY2NrfQ1/u///o97772Xw4cPM3LkSKZMmcK1a9dq9N4KIeqpxGOQdgniD0JumtatEXWQ9CvaQU6BkQ7zf3T66x7/xzB8PKr3EQYGBuLh4YGPjw/h4eEAnDx5EoB//OMf3HHHHZZjg4OD6dq1q+Xnl156ifXr1/PNN98we/bsCl9j+vTpTJ48GYB///vfvPnmm+zevZvhw4dbfW1CiHou8Vjxdtpl8A7SrCmibpIeIEGvXr1K/ZyZmclzzz1H+/btCQoKws/PjxMnTlTZA9SlSxfLtq+vLwEBAZblLoQQwio3BkBC2Jn0ANmBt7uB4/8Ypsnr2sONs7mee+45Nm3axOLFi2nVqhXe3t6MHz+e/Pz8Ss/j7u5e6medTofJZLJLG4UQ9UzS8eLttEvatUPUWRIA2YFOp6v2UJSWPDw8MBqNVR63fft2pk+fztixYwG1R+jChQsObp0QQhRRFOkBEg4nQ2D1SHR0NLt27eLChQukpKRU2DvTunVrvvrqKw4ePMihQ4f405/+JD05QgjnyUyEnBITKCQAEg4gAVA98txzz2EwGOjQoQOhoaEV5vS8/vrrNGjQgP79+zN69GiGDRtGjx49nNxaIUS9VbL3ByQAEg6hU6yZS11PpKenExgYSFpaGgEBAaUey83N5fz58zRv3hwvLy+NWlg3yHsphCjX9jdh04sQGKXm/wREwpzjVT9P1HuVfX/fyCV6gJYuXUp0dDReXl707duX3bt3V3js+++/z8CBA2nQoAENGjRgyJAhpY4vKCjghRdeoHPnzvj6+hIREcHUqVOJi4tzxqUIIYSoKXMCdOuh6n1GPBgLtGuPqJM0D4DWrFnDnDlzWLBgAfv376dr164MGzaswunTW7duZfLkyWzZsoWdO3cSFRXF0KFDuXLlCgDZ2dns37+fF198kf379/PVV19x6tQp7rrrLmdelhBCCFuZh8BaDAaDBygmNQgSwo40HwLr27cvvXv35u233wbAZDIRFRXFE088wdy5c6t8vtFopEGDBrz99ttMnTq13GP27NlDnz59uHjxIk2bNq3ynDIE5hzyXgohyjAWwr8jwJgHTx6AT+6B6+dhxg/QrL/WrRMurtYMgeXn57Nv3z6GDBli2afX6xkyZAg7d+6s1jmys7MpKCggODi4wmPS0tLQ6XQEBQWV+3heXh7p6emlbkIIITRw7Zwa/Lj7QlA0BKlL9kgitLA3TQOglJQUjEYjYWFhpfaHhYWRkJBQrXO88MILRERElAqiSsrNzeWFF15g8uTJFUaDCxcuJDAw0HIzr5ElhBDCyRKPqveN2oFeryZCgxRDFHaneQ5QTSxatIjVq1ezfv36codQCgoKuPfee1EUhWXLllV4nnnz5pGWlma5Xbokv2hCCKEJcwJ0ow7qfWAT9V56gISdaVq+OCQkBIPBQGJiYqn9iYmJlgU7K7J48WIWLVrEzz//XGoNKjNz8HPx4kV++eWXSscCPT098fT0tO0ihBBC2E9iUQAU1lG9NwdAqfKHqbAvTXuAPDw86NmzJ5s3b7bsM5lMbN68mX79+lX4vFdeeYWXXnqJjRs3llnIE4qDnzNnzvDzzz/TsGFDh7RfCCGEnSUVzQC7MQCSHiBhZ5ovYDVnzhymTZtGr1696NOnD0uWLCErK4sZM2YAMHXqVCIjI1m4cCEAL7/8MvPnz2fVqlVER0dbcoX8/Pzw8/OjoKCA8ePHs3//fr777juMRqPlmODgYDw8PLS5UCGEEJXLy4TrF9TtRuYAqEQOkKKATqdJ00Tdo3kO0MSJE1m8eDHz58+nW7duHDx4kI0bN1oSo2NjY4mPL67/sGzZMvLz8xk/fjyNGze23BYvXgzAlStX+Oabb7h8+TLdunUrdcyOHTs0uca6Ijo6miVLlmjdDCFEXZV0Qr33CwPfop77gEj1Pj8TctO0aZeokzTvAQKYPXs2s2fPLvexrVu3lvq5qlXJo6OjkdU9hBCiFjIPf5kToAE8fMAnBLJT1GEw7yBNmibqHs17gIQQQgigbAK0meQBCQeQAKieeO+994iIiMBkMpXaf/fddzNz5kxiYmK4++67CQsLw8/Pj969e/Pzzz9r1FohRL2UVFUAJDPBhP1IAGQPigL5Wc6/WTHUN2HCBK5evcqWLVss+65du8bGjRuZMmUKmZmZjBw5ks2bN3PgwAGGDx/O6NGjiY2NdcQ7JoQQpSlKiSKIHUo/JsUQhQO4RA5QrVeQra5d42x/iQMP32od2qBBA0aMGMGqVau4/fbbAVi7di0hISHceuut6PV6unbtajn+pZdeYv369XzzzTcV5mcJIYTdZCRAznXQ6SG0benHZAhMOID0ANUjU6ZMYd26deTl5QHw2WefMWnSJPR6PZmZmTz33HO0b9+eoKAg/Pz8OHHihPQACSGcw5wAHdwS3L1LPyYBkHAA6QGyB3cftTdGi9e1wujRo1EUhe+//57evXvz22+/8Z///AeA5557jk2bNrF48WJatWqFt7c348ePJz8/3xEtF0KI0iwJ0B3KPhYoC6IK+5MAyB50umoPRWnJy8uLe+65h88++4yzZ8/Stm1bevToAcD27duZPn06Y8eOBSAzM7PKkgNCCGE3lgToTmUfM/cAZcSDsQAM7s5rl6izJACqZ6ZMmcKdd97JsWPHuO+++yz7W7duzVdffcXo0aPR6XS8+OKLZWaMCSGEw1SUAA3gGwoGTzDmqUFQUFPntk3USZIDVM/cdtttBAcHc+rUKf70pz9Z9r/++us0aNCA/v37M3r0aIYNG2bpHRJCCIcyFkLyaXW7vCEwvR4CiypCyzBYnXDpWrbWTZAeoPpGr9cTF1c2Xyk6Oppffvml1L7HH3+81M8yJCaEcIhrMWrvjrsvBEWXf0xgE7h2Tl0VvplTWyfs7PczKUxdsYvJfZryzzGd0Gm0vpv0AAkhhNBWonkJjHZqb095pBZQnWAyKSz84QQmBdwNes2CH5AASAghhNYqqgBdkkyFrxO+ORTHsbh0/D3deOK2Vpq2RQIgIYQQ2rL0AEkAVJflFhh59cdTADwyuCUN/Tw1bY8EQEIIIbRlDoDKS4A2kwCo1vtk50WupOYQHuDFzJuba90cCYBspVixDpcon7yHQgjyMiD1orpdaQ9Q0dT3tEtWrYMoXENqdj5v/XIGgDlD2+DtYdC4RRIAWc3dXS3AlZ2t/RS+2s78HprfUyFEPZR0Ur33CwffhhUfZ54Gn58JuWmOb5ewq3e2xpCeW0jbMH/G9WiidXMAmQZvNYPBQFBQEElJSQD4+PhomsVeGymKQnZ2NklJSQQFBWEwaP+XgBBCI0nVGP4CdX0wnxDITlF7gbyDHN40YR+XrmXz0fYLAMwd2Q6D3jW+MyUAskF4eDiAJQgStgkKCrK8l0KIesqSAF1FAARqHlB2ipoHFN7Zse0SdvP6ptPkG030b9mQwW1CtW6OhQRANtDpdDRu3JhGjRpRUFCgdXNqJXd3d+n5EUKUWAS1kvwfs8AmEH9QEqFrkaNX0lh/4AoA80a0d6kREwmAasBgMMiXuBBC2EpRiofAqtUDJMUQaxNFUYseAtzdLYLOTQI1blFpkgQthBBCGxkJkHMddHoIbVf18TIVvlb59UwK289excOg57mhbbVuThkSAAkhhNCGufenYStw96r6eAmAag2jSWHhBrX3Z2q/ZkQF+2jcorIkABJCCKENaxKgAYLMQ2ASALm69QeucDIhgwAvN2ZrvORFRSQAEkIIoQ1rEqChOAcoIx6MMgHFVeUWGHntJ3XJi8dvbUWQj4fGLSqfBEBCCCG0YU0CNKh1gAyeoJggPc5x7RI18uH2C8Sn5RIZ5M20/tFaN6dCEgAJIYRwPmMhJKu9BNXuAdLriytCyzCYS7qWlc87W84C8OzQNni5u+5MaQmAhBBCON/Vs2DMB3dfCGpW/edJIrRLe/uXs2TkFdK+cQBjukVq3ZxKSQAkhBDC+SzDX+3Vnp3qklpALiv2ajaf/HEBgL+MbIfeRZa8qIgEQEIIIZzPkgBdzfwfM+kBclmv/nSKAqPCwNYhDGztOkteVEQCICGEEM6XVBQANapm/o9ZoEyFd0WHLqXy7aE4dDqYO6IaRS1dgEsEQEuXLiU6OhovLy/69u3L7t27Kzz2/fffZ+DAgTRo0IAGDRowZMiQMscrisL8+fNp3Lgx3t7eDBkyhDNnzjj6MoQQQlSXuQZQdROgzaQHyOUoisK/i4oeju0eSccI11ryoiKaB0Br1qxhzpw5LFiwgP3799O1a1eGDRtW4UrrW7duZfLkyWzZsoWdO3cSFRXF0KFDuXLliuWYV155hTfffJPly5eza9cufH19GTZsGLm5uc66LCGEEBXJy4DUi+q21QFQiRwgRbFvu4RNtpxKYtf5a3i46XnWBZe8qIhOUbT9F9S3b1969+7N22+/DYDJZCIqKoonnniCuXPnVvl8o9FIgwYNePvtt5k6dSqKohAREcGzzz7Lc889B0BaWhphYWF89NFHTJo0qcpzpqenExgYSFpaGgEBATW7QCGEEKVd2g3/vQP8wuG5U9Y9tyAH/hWubr9wAbwb2L15ovoKjSZGvPEbZ5IyefiWFswb2V7T9ljz/a1pD1B+fj779u1jyJAhln16vZ4hQ4awc+fOap0jOzubgoICgoODATh//jwJCQmlzhkYGEjfvn2rfU4hhBAOZBn+sjIBGsDdWy2ICDIM5gLW7b/MmaRMAr3deWyway55URFNA6CUlBSMRiNhYWGl9oeFhZGQkFCtc7zwwgtERERYAh7z86w5Z15eHunp6aVuQgghHMSSAG1DAASSB+QisvMLeX3TaQCeuK0VgT7uGrfIOprnANXEokWLWL16NevXr8fLqxorCVdg4cKFBAYGWm5RUVF2bKUQQohSLFPgO9n2fAmAXMKK38+TmJ5Hkwbe3N/PimKWLkLTACgkJASDwUBiYmKp/YmJiYSHh1f63MWLF7No0SJ++uknunTpYtlvfp4155w3bx5paWmW26VLUmBLCCEcQlEg8ai6bcsQGEgxRBeQkpnH8m3nAHh+WFs83Vx3yYuKaBoAeXh40LNnTzZv3mzZZzKZ2Lx5M/369avwea+88govvfQSGzdupFevXqUea968OeHh4aXOmZ6ezq5duyo8p6enJwEBAaVuQgghHCAjHnJTQWeAEBtnDAVJLSCtvbX5DJl5hXSKDGB0lwitm2MTN60bMGfOHKZNm0avXr3o06cPS5YsISsrixkzZgAwdepUIiMjWbhwIQAvv/wy8+fPZ9WqVURHR1vyevz8/PDz80On0/H000/zz3/+k9atW9O8eXNefPFFIiIiGDNmjFaXKYQQAoqHvxq2BHcbUxfMQ2Cp0gOkhfMpWXy2KxaAv4xo7/JLXlRE8wBo4sSJJCcnM3/+fBISEujWrRsbN260JDHHxsaiL7FOzLJly8jPz2f8+PGlzrNgwQL+/ve/A/DnP/+ZrKwsHnroIVJTUxkwYAAbN26sUZ6QEEIIO7CsAWbj8BdIDpDGXv3xJIUmhcFtQ+nfKkTr5thM8zpArkjqAAkhhIN89TAcXg23/g0GPW/bOTKTYHFrQAcvJoOhds0+qs32x17nnnd2oNPBD08NpF24a31H1po6QEIIIeqZmtQAMvMJAYMnoEB6nF2aVducTcrgl5OJFBpNTntNRVFYtOEkAON7NHG54Mdamg+BCSGEqCeMBZBSVPm5JkNgej0ERsK1c+owWIPaNwW7Jn45mchjn+0nt8BEZJA3DwxozsTeUfh6OvYr/ecTSey+cA1PNz1zhrZx6Gs5g/QACSGEcI6rMWDMB3dfCKph0FJP84DW7bvMrI/3kVtgwsOg50pqDv/47jj9F/3Cqz+eJCnDMWteFhpNLPpBXfD0gQHNaRzo7ZDXcSYJgIQQQjiHJQG6vdqLUxOBTdX7elQL6IPfzvHsl4cwmhTu6RHJ/vl38K+xnYhu6ENaTgFLt8Qw4OUtzF13mJjkTLu+9hd7LxOTnEUDH3ceGdzSrufWigyBCSGEcA5LBWgrV4Avj6UHqO4HQIqi8PLGUyzfFgPArIHNmVc0/XxK32ZM6t2UTccTePfXcxyITWX1nkus3nOJIe3DeGRQC3pFB9fo9bPyCvnPz+qSF0/e3poAr7qRdC4BkBBCCOewJEDbMwCq20NghUYTf1l/hC/2qtc5d0Q7Hr6lBTpdce0dg17H8E6NGdYxnL0Xr/PutnP8fCLRcuvRNIiHbmnJHR3CMNhQs+eD386TnJFH02AfpvStO/lWEgAJIYRwDnvUADJz9QDo9E+w5Z9wzwcQalvCcG6BkdmrDvDziUT0Olh0Txfu7V3xWpU6nY7e0cH0jg7mbFImH/x2jq/2X2F/bCqPfLqP5iG+PDiwOeN6NMHLvZKlKwpy4dN7IPkkJgXuz87nPk/wK3TD43U7Zs60Hw2j37Df+awkAZAQQgjHy8uAVLV6sH16gEosh6EooHOxasS734X4Q3DwU7jjH1Y/PS2ngFkr97L7wjU83PS8Pbk7QztWvkZmSa0a+bFoXBfmDG3Dyh0X+GTnRc6nZPHX9Ud5/afTTOsfzf03NaOBr0fZJ1/eDRe3A2qicLD5rc0vutlLnn3zlKwlAZAQQgjHS1JnEOEXDj41y0kB1GnwAPmZ6tpi3g1qfk57Muc7me+tkJSey9QVuzmZkIG/pxsfTOtF3xYNbWpGI38vnh/WjscGt2LNnkv89/fzXEnN4fVNp1m2NYZ7ezXhwYEtiAr2KdP27CYDGXvuLoyKwmsTutK1SZBNbaiQp799z2clCYCEEEI4nj3zfwDcvdWCiNkpai+QKwVA2dcgo6hAo/m6q+ni1Szu/+9uYq9lE+Lnyccz+9AhouYFB3093Zg5oDn392vGhiPxvLvtHMfj01m58yKf/HGREZ0b8/AtLejSJAgSjwKwNbMpp0yRDGnfiK49ete4Da5GAiAhhBCOZ48K0DcKbFIcAIV3tt95ayqpRK9PRhzkXK9WgHb0ShrTP9xDSmYezRr68MnMvjRt6FPl86zhbtBzd7dI7uoawfazV3n31xh+O5PC94fj+f5wPDe1COa93EMEAN8nNUSvgxeGt7NrG1yF1AESQgjheOagoJGdeoAAgorygFxtVfgbh72qMQy2M+Yqk9/7g5TMPDo0DuDLR/rZPfgpSafTMaB1CJ880JcNTw5kbPdI3PQ6dp1LwXBVrdZ9UoliYu8oWodpO1TlKBIACSGEcCxFcVAPkDkR2sUCoKQbhr2SKg+ANh5NYNqHu8nIK6Rv82BWP3wTjfy9HNjA0jpEBPCfid349c+38lwvT3x1eeQp7iS6RfL0kNq/5EVFJAASQgjhWBnxaqKyzgAhbe13XledCm/u8WkQXfRzxXlAq3fH8thn+8gvNDG0QxgrZ/bRrNBgRJA3j3dUp3llBbTkwwf6ERbgvEDM2SQAEkII4VjmAKBhK3C34xeqKwZAJlNxj0/ne9X7cgIgRVFYuuUsc786gkmBSb2jeGdKj8rr8zhDUVuDW3Sndw0rSLs6CYCEEEI4liOGv8A1A6C0WHVqvsFDLfQHagkARbEcYjIpvPTdCV79Uc21efzWliy8pzNuBhf4SrZnsUoXJ7PAhBBCOJYjEqChOAcoIx6MBWBwgTWqzMNfIW2LFn11h/yiIpANmlFgNPH8l4f4+qA6Tf7FOzvwwIDmGjb4Bpb12up+AOQC4aYQQog6zVFfqj4hYPAEFEiPs++5bZVUorfL4A4hRUnEScfJzi9k1sd7+fpgHG56HUsmdnOt4KcgB66pC67aPVh1QRIACSGEcBxjAaSoQz12H1bR611vVfgbV7wvus+5fJgpH+xi66lkvNz1vD+tF2O6R2rUyAoknwLFpNYs8q/+shu1lQRAQgghHOfqWTDmg4cfBDlgJXFXywMy5zuZe1CKer3+2PkbB2JTCfR257MHb+LWto00amAlLLlanVxvbTUHkABICCGE41gCgvZqj429uVItoMI8NeADS+AT59kCgMj8c4QHePHlI/3o2cyFlu0oyZKrVffzf0ACICGEEI7k6C9VV+oBSj4FihG8gsC/MQcvpTJzQxYALfXxrHuoJ21cuaqyo2bruSgJgIQQQjjOjTkx9uZKAVBS8bXuvXidP73/BydzAsjU+WLARGShC/RSVcZRs/VclARAQgghHCfJzqvA38iVAqCiHhSlUQf+/u0xsvONDGgVineTLkWPV70mmGayUiAzUd1uVDcXP72RBEBCCCEcIzddrX8DDhwCM+cAXS5VbFATRQHQRbdojl5Jx9NNz5uTu2MILwr+Eo9q2LgqmIe/GkSDpwsP09mRBEBCCCEcI+mEeu/fGHwctKxCYNFU8vxMdb0xLRUNIX0RGwDAPT0iCfb1KA7+qlgUVVP1bPgLJAASQgjhKM5YVsHdG3xD1e1UDXNssq+pFamBT875ADDj5qIih+bhP1ceAqtnCdAgAZAQQghHcdayCq6QB1TUg5Lq0ZgMxZuBrUOKZ3w1aq/eZ8RBznWNGliFejYFHiQAEkII4SiWWVGdHPs6rhAAFQV7B/LVIbmZN5dY4sIrEAKbljrOpZhMxcOVjkpWd0ESAAkhhLA/RSlO+nV0r4IrFEMsutajhU1oEeLLoDahpR8394KZh5pcyfXzUJCtrqsW3FLr1jiNBEBCCCHsLz0OctNAZ4DQto59LRfoAVKKenZOmaKYcXM0ev0NS0lYEqFdMAAy99SFtgWDm7ZtcSLNA6ClS5cSHR2Nl5cXffv2Zffu3RUee+zYMcaNG0d0dDQ6nY4lS5aUOcZoNPLiiy/SvHlzvL29admyJS+99BKK1tMjhRCiPjF/qTZsBW6ejn0trQMgkwljUQB02aM59/RoUvYYV06EdnSxShelaQC0Zs0a5syZw4IFC9i/fz9du3Zl2LBhJCUllXt8dnY2LVq0YNGiRYSHl79S7csvv8yyZct4++23OXHiBC+//DKvvPIKb731liMvRQghREnOnFWkdQCUFotbYRZ5ihv9evXB17OcXhRLD9AJ7esV3cgZs/VckKYB0Ouvv86sWbOYMWMGHTp0YPny5fj4+LBixYpyj+/duzevvvoqkyZNwtOz/L8oduzYwd13382oUaOIjo5m/PjxDB06tNKeJSGEEHaW5MReBXOCcUY8FOY7/vVucPnUXgDOKpHcN6BV+QeFtAa9O+RnFBeHdBXOmq3nYjQLgPLz89m3bx9Dhgwpboxez5AhQ9i5c6fN5+3fvz+bN2/m9OnTABw6dIjff/+dESNGVPicvLw80tPTS92EEELUgGUVeCcEQL4hagIvijrV3MmO7Ve/szID2xAZ5F3+QQb34lwoV0qELsiBazHqtqNn67kYzQKglJQUjEYjYWFhpfaHhYWRkJBg83nnzp3LpEmTaNeuHe7u7nTv3p2nn36aKVOmVPichQsXEhgYaLlFRUXZ/PpCCFHvGQvUldHBOb0KOp1mw2BXM/MwJqgzwCLb9qz8YFdMhE4+CYoJvIPBL6zq4+sQzZOg7e2LL77gs88+Y9WqVezfv5+VK1eyePFiVq5cWeFz5s2bR1pamuV26ZKLr9grhBCu7OpZMBWAh1/x8JSjaRQArdoVS2vU74zItr0qP9gyFd6FEqFLJkDrdJUfW8doNt8tJCQEg8FAYmJiqf2JiYkVJjhXx/PPP2/pBQLo3LkzFy9eZOHChUybNq3c53h6elaYUySEEMJKluGv9qB30t/ZGtQCyi808fnOszyqU5fA0FU1hGR+3JXWBKuHFaDNNOsB8vDwoGfPnmzevNmyz2QysXnzZvr162fzebOzs9Hf8AtnMBgwmUw2n1MIIYQVLDPAnDitWoMeoO+PxBGUdQ43nQnFuwH4V/HHuznISDkDhXmOb2B1aPFZuQhNKx7NmTOHadOm0atXL/r06cOSJUvIyspixowZAEydOpXIyEgWLlwIqInTx48ft2xfuXKFgwcP4ufnR6tWaub96NGj+de//kXTpk3p2LEjBw4c4PXXX2fmzJnaXKQQQtQ3Wqws7uQASFEU/vv7edrp1BldukbVGEIKiFCXxchNU3OkGndxQkur4MzZei5G0wBo4sSJJCcnM3/+fBISEujWrRsbN260JEbHxsaW6s2Ji4uje/fulp8XL17M4sWLGTRoEFu3bgXgrbfe4sUXX+Sxxx4jKSmJiIgIHn74YebPn+/UaxNCiHpLi2nVQUVDYE5aEX7vxescvZLOWI+igKs616rTqUFh7A418NA6AMpKgcyiNJTQdtq2RQOa17yePXs2s2fPLvcxc1BjFh0dXWVFZ39/f5YsWVJulWghhBAOlpsOaUV1bpyZV2LJAbqsFhp0cELvit/PAzAwIAkyqf61hnVQAyBXmApvbkODaPD007QpWqhzs8CEEEJoyLyquH9j8Al23usGRKj3BVmQc92hL3XpWjY/HlPLtbRQLqo7q1tDxzzU5AqJ0FoMVboQCYCEEELYj3kFeGfnlLh7g2/RCuwOzgP6eOcFTAqMaOGBW1bREFKjag4hmYMNV+oBqof5PyABkBBCCHvSclq1ExKhM/MKWb1HzTN6qG2uujOoGXj6V+8Ejdqr9xnxkH3NAS20QlL9XALDTAIgIYQQ9qPlyuJOCIDW7btMRm4hLUJ86ep5Rd1pzbV6BRQXh9RyGMxkKh6ulCEwIYQQogYURduVxR1cDNFkUvhwu5r8POPmaPS29na5QkXo6+ehIFtdQy24hXbt0JAEQEIIIewjPU6tcaMzFC/86UyWHiDHBEBbTiVx4Wo2AV5u3NOjie01dCyJ0BrmAZnbHtoWDJpPCNeEBEBCCCHsw5xUG9Ia3DRYXqjkVHgHWFHU+zO5T1N83fW2D/eZe4y0TIS2tL1+rQBfkgRAQggh7EPL4S9waA7QyYR0tp+9ikGvY2r/aEi9qE65N3hCcEvrTmbpATqh5uJowfxZ1dMEaJAASAghhL1oUQG6JHMPUEYCFObb9dQf/n4BgOEdw4kM8i4xhNTG+iGkhq1A7w75mcVFI50tUeNg1QVIACSEEMI+tC6s5xui9sigQEac3U57NTOP9QfVGV8zB0SrOxNrcK0G9+IcKS0SoQty4No5dbue1gACCYCEEELYg7FAXeATtOsB0ukcMgy2alcs+YUmujYJpEfTBurOpBoWEdQyETr5JCgm8A4GvzDnv76LkABICCFEzaWcAVMBePgX17nRgp0DoPxCEx//oS53MXNAc3TmNcYSa5hDo2UidMnkbQevmebKJAASQggNZecXYjJVvshzrWAZ/moPeg2/WgLtuyr890fiSM7IIyzAkxGdGqs7C3Lhaoy6betwn7kHSIshMFun79cxEgAJIYRGTidm0OufP3P769v4/UyK1s2pmZr2iNhLkP2KISqKwn+LVn2f2i8aD7eir8yUU6AYwbsB+IfbdnJzD9DVs1CYV+O2WsW8Xls9ToAGCYCEEEIzX+69RHa+kfMpWdz3313MXrWfxPRcrZtlG60ToM3sOAS29+J1jl5Jx9NNz+Q+JYb1SiZA2zqEFBABXoFqIGXOnXIWLZcrcSESAAkhhAYURWHDkQQABrQKQa+D7w7Hc/tr21jx+3kKjRrVh7GV1lPgzewYAK0o6v25p0ckwb4exQ/UNAEa1MDJXITQmWuCZaVAVpK6HVrNFezrKAmAhBBCA0eupHElNQcfDwMfTOvFt08MoHvTIDLzCvnHd8cZ/fZ29l28rnUzqyc3rbiejZ2GVRRF4X8Hr/DjsQTrnliyGrRie27VpWvZlteecXPz0g/aa7jPkgh9tGbnsYa57Q2ag6ef817XBUkAJIQQGjD3/tzarhFe7gY6RgSy7pH+LLqnM0E+7pyIT2fcsh3MXXeY61n2Lepnd+ZVxf0jwCe4xqczmhRe/N9Rnlp9kIc/2cdzXx4iJ99YvScHRKj3BVmQY3sA+fHOC5gUGNg6hDZh/qUfrEkNoJK0WBRVEqAtJAASQggnUxSFH47GAzDSPLMI0Ot1TOrTlF+eHcy9vdShnNV7LnHba1tZsyfWdWeL2TEBOq/QyJOfH+DTP2LR6UCvg7X7LjP2ne3EJGdWfQJ3b/ANVbdtHAbLzCtk9R41iXrmjb0/2dcgs6hXqlENh5DMAZQzh8AkAdpCAiAhhHCy4/HpXLyajZe7nsFtQ8s8HuzrwSvju7L2kX60C/fnenYBL6w7wvjlOzgel65Bi6tgSYCu2ZdqZl4hMz/aw/dH4nE36Hhrcnc+fbAvIX6enEzI4K63fue7w9Wo8FzDVeHX7btMRm4hLUJ8GdTmhs/HHOwFNQNP/7JPtkaj9up9RrwaWDmDq+RquQAJgIQQwsl+KBr+GtymEb6eFa8j1Ss6mO+eGMDfRrXH18PA/thU7nzrN/7x7XEycguc1dyqJdY8KfhqZh6T3/uD7Wev4uNh4MPpfbizSwT9W4aw4ckB9GkeTFa+kdmrDrDgf0fJK6xkSKwGidAmk8KHRau+z7g5Gr3+hlleSXZcRd0rAIKalj6vI5lMahVoqNerwJtJACSEEE6kzv5Sh79GdK66hoybQc+DA1uw+dnBjOrSGJMCK7af5/bXtvHtoTiUGiT62oWi1Hha9aVr2UxYvpMjV9II9vXg81k3MaB1iOXxRgFerHqwL48OVlddX7nzIvcu38nl69nln9BcidqGHqAtp5K4cDWbAC837unRpOwB5iEke/WgmIfBnFER+vp5KMgGNy8IbuH413NxEgAJIYQTnU7M5FxKFh5uem5r16jazwsP9GLpn3rw8cw+RDf0ISkjjyc+P8D9/93NuerkxjhK+hXISwOdAULaWP30UwkZjF++g3MpWUQGefPlI/3oGhVU5jg3g54Xhrfjv9N6EejtzqHLaYx683d+OZlY9qQ16AFaUdT7M7lP0/J75xLtM9xnEebEJTEsK9i3Bb3B8a/n4iQAEkIIJzL3/tzSOhR/L3ern39Lm1A2Pn0Lc+5og4ebnt/PpjB8yW+89tMpcguqOVPKnswBQUhrcPO06ql7L1xjwvIdJKbn0SbMj3WP9qdlaOVTs29vH8Z3Twyga5NA0nIKmPnRXl7eeLJ03SQbA6CTCelsP3sVg17H1P7RZQ8wmYpnvNlrFpU5kHLGEJg5yNK6WKWLkABICCGcyDL7qxrDXxXxcjfw5O2t2fTMLQxuG0q+0cRbv5zljv9sK79HxJHMRQGt7BH55WQi9/13F+m5hfRoGsQXD/cjPNCrWs+NCvbhi0f6Ma1fMwCWbY1hyge7SDJX0bYxAPpo+wUAhncMJzLIu+wBqRfV6fUGTwhuadW5K2RZFf6EGmA5kqssV+IiJAASQggnOZuUwenETNwNOm5vH1bj8zVr6MuH03uz/L6eNA704tK1HGZ+tJeHPt7LldQcO7S4Gmz4Ul237zKzPt5HboGJW9uG8tmDNxHk41H1E0vwdDPwf3d34q3J3fH1MLDr/DVGvvk7O2JSioshZiRAYfVqKF3NzOOrA1cAmDkguvyDSg4hGSpOXrdKw1Zg8ID8zOJiko4iNYBKkQBICCGc5IcSS18Eels//FUenU7H8E7h/DxnEA/f0gI3vY6fjicy5LVtLNsaQ36ho3sVrJsV9cFv53j2y0MYTQpju0fy3tReeHvYno8yumsE3zwxgLZh/qRk5nHfB7t4e9d1FIMnoEBGNabNA5/vjiW/0ETXJoH0aNqg/IPsMNutDIM7hLQtfX5HKMiBa+fUbRkCAyQAEkIIp9lwVA2ARnRuXMWR1vP1dGPeyPZ8/+RA+kQHk1Ng5OWNJxn55m9sOBKP0RFFFI0FkHJa3a5iCExRFF7eeJJ/fq/m0DwwoDmvTeiKu6HmX0MtQ/34+vGbGd+zCSYFFm86Q6KuaBZZatUzwfILTXy88yIAMwc0R1fRAqeJtg33VckZFaGTT4JiAp+G4Ff95Pu6TAIgIYRwgvMpWZyIT8dNr2Noh5oPf1Wkbbg/ax6+idcmdKWhrwdnkzJ57LP93PbaVj7ZeaH6S0pUR8oZMBWAh39xPZtyFBpNvLDuMMu2xgDwwvB2/G1U+7I1dmrA28PA4gldeWVcFzzd9JzJU3txLpyreqX1DUfiScrIIyzAkxGdKglOkxxURNCSCO3AHqCSwZutK9jXMRIACSGEE5iTn/u1bGh1vou1dDod43o24ZdnB/PEba0I8nHn4tVsXvzfMW5++Rf+s+k0VzPzav5ClgrQ7Sv8Us0tMPLoZ/v5Yu9l9Dp4eVxnHh3csuJelhq6t3cU6x+7mXRPNchcv/UPVvx+vsJ6SYqiWKa+T+0XjYdbBV+LBblwVQ3g7F5E0Hw+R/YA1bBWU10kAZAQQjiBOf9npAOGvyoS6OPOs0PbsmPubfx9dAeaNPDmWlY+b2w+w80v/8Lfvj7ChZQs21/AUhSw/C/V9NwCpq7YzabjiXi46Vl2X08m9q64p8heOkQEMOSmnmrTlBT+8d1xHl+1v9zq2fsuXufw5TQ83fRM7lNJ21JOgWIE72Dws3MPnrlH6epZNdByhCQH5C/VcnZKY7fd0qVLefXVV0lISKBr16689dZb9OnTp9xjjx07xvz589m3bx8XL17kP//5D08//XSZ465cucILL7zADz/8QHZ2Nq1ateLDDz+kV69eDr4aIYQo69K1bI5cSUOvw6HDX2Ukn4JfXsInP5vpwNQISPHP49L1bDJzC2EfXNwPWX6eRDXwIcDLyq+ESpKCkzJymbZiDyfi0/H3dOP9ab24qUXDGl9SdXk2VKfI3xKWh1u8jg1HEjgel847U3rSISLAcpy59+eeHpEE+1bSM1fyWu3de+XfGLyCIDdVDbQad7Xv+cF+K9jXIZoGQGvWrGHOnDksX76cvn37smTJEoYNG8apU6do1KhsklZ2djYtWrRgwoQJPPPMM+We8/r169x8883ceuut/PDDD4SGhnLmzBkaNKggq18IIRzMPPx1U4uGNPSzrlhgjex+D058a/lRDzQqulFy4lV20c1WTXqX+vHi1Szu/+9uYq9lE+LnycqZvekYEViDF7BBUS2gJrqrfPFIP2Z/tp8LV7MZ+852Xrq7E/f2juLy9Ww2FiWmz7hx1fcbOSoBGtSAKqwjXNyuBir2DoAykyErCdDVfAX7OkTTAOj1119n1qxZzJgxA4Dly5fz/fffs2LFCubOnVvm+N69e9O7t/qLVt7jAC+//DJRUVF8+OGHln3Nm1fxD1sIIRxowxHHzf6qVGpRXZke06BZ/3IPiU/LYcvJZPbHXrfMFAsL8OLWtqH0aNag6llagU0gopvlx2NxaUxbsYeUzDyaBvvwyQN9aNbQ1x5XYx1zLaC0y/SICuL7JwfyzBcH2XoqmT+vO8zuC9fwdjdgUmBg6xDahFWxsrujEqDNGnVQAyBHJEKbz9kgGjw0+CxclGYBUH5+Pvv27WPevHmWfXq9niFDhrBz506bz/vNN98wbNgwJkyYwLZt24iMjOSxxx5j1qxZFT4nLy+PvLzihMD09HSbX18IIUq6kprDwUup6HQwrKMTh7+guBJyh7uh1e3lHtIY+NMtcFtaLh9uP8+qXbFkpBayfBc0Ou7J9JujmdK3WbXqFv1x7iqzVu4lI6+Q9o0DWDmzN438q1fd2e4CItT7gizIuU4D32BWTOvNO1vP8vqm06zdV1wlemZVvT9gdb0jq5mHER2RCC0J0OXSLAk6JSUFo9FIWFjp/xDCwsJISEiw+bznzp1j2bJltG7dmh9//JFHH32UJ598kpUrV1b4nIULFxIYGGi5RUVF2fz6QghRknmIpXd0sPODAXMAFFj1/2nhgV7MG9me7fNu4y8j2xEe4EVSRh6vbDxF/4Wbeem745VWl/7xWAJTV+wmI6+QPs2DWfPwTdoFPwDu3uAbqm4XrQqv1+uYfVtrPn2wLyFFQ5EtQnwZ1Ca08nNlXYXMou+lUAcNIYU5cFV4SYAuV52bBWYymejRowf//ve/6d69Ow899BCzZs1i+fLlFT5n3rx5pKWlWW6XLlVdOEsIIarjh6LFT0d2sn3tL5vkpkFeUW92YGS1nxbg5c5Dt7Tk1z/fymsTutI2zJ+sfCP//f08g17ZwtOrD3A8rnQv+Zo9sTz66T7yC00M7RDGxzP7EGDDQq92V2IYrKT+LUPY8OQAnrytFe/c16PqekQlh5A8K1+s1WaN2qv3mQmQfc2+57b3CvZ1hE0B0JYtW2r8wiEhIRgMBhITSy/cl5iYSHi47f9RNG7cmA4dSn/I7du3Jza24jVWPD09CQgIKHUTQoiaSkjLZe/F6wAMr6zAniOYv/S9g23K+/Bw0zOuZxM2Pj2Qj2b0pn/LhhSaFL4+GMfIN3/j/v/u4rczybyz9SwvrDuCSYGJvaJ4Z0oPvNxtX9rCripZFLVRgBdzhralXXg1/r93xgwqzxLFJO3ZC2QyqlWgQXqAbmBTADR8+HBatmzJP//5T5t7Szw8POjZsyebN2+27DOZTGzevJl+/frZdE6Am2++mVOnSlf+PH36NM2aNbP5nEIIYYsfj6nDJj2bNaj2Sud2Yxn+alKj0+h0Oga3bcSqWTfx7ewBjO4agV4Hv51J4f7/7uaVjer/t48NbsmicZ1xs8PSFnZj6QGqYa9+kpNWUTcHWEl2zAO6fgEKssHNC4Jb2O+8dYBN/1KvXLnC7NmzWbt2LS1atGDYsGF88cUX5OdXb9Vdszlz5vD++++zcuVKTpw4waOPPkpWVpZlVtjUqVNLJUnn5+dz8OBBDh48SH5+PleuXOHgwYOcPXvWcswzzzzDH3/8wb///W/Onj3LqlWreO+993j88cdtuVQhhLDZhqLhrxHOHv6C4hlg1cj/qa7OTQJ5a3J3tj1/K9P7R+Nd1NPzt1Ht+fPwdg6r7myzSnqArOKsJGJH5AGZzxXaFvQu0jPnImwKgEJCQnjmmWc4ePAgu3btok2bNjz22GNERETw5JNPcujQoWqdZ+LEiSxevJj58+fTrVs3Dh48yMaNGy2J0bGxscTHx1uOj4uLo3v37nTv3p34+HgWL15M9+7defDBBy3H9O7dm/Xr1/P555/TqVMnXnrpJZYsWcKUKVNsuVQhhLBJckYeuy+ouRxOn/4OxV/6Qfaf1BEV7MPf7+rIH3+5nW3PD+bBgS7as2CPAMhkgiR1AVeHFxG0LIpqxwAoycGz12qxGk+D79GjB+Hh4TRs2JBFixaxYsUK3nnnHfr168fy5cvp2LHyfzCzZ89m9uzZ5T62devWUj9HR0dXuJ5LSXfeeSd33nlnta9BCCHs7cdjCSgKdI0KIjLI2/kNsNMQWGUCvd2rNT1eM+Zrr8aK8BVKvaBOpTd4On4IyTIEdkINvPR2GE50ZAHHWs7md7egoIC1a9cycuRImjVrxo8//sjbb79NYmIiZ8+epVmzZkyYMMGebRVCiFrDXP3Z6bO/zJwQALk88/BfZgIU2rj4q3n4K7QtGBxcOq9hSzB4qAFX6kX7nNPRBRxrMZs+zSeeeILPP/8cRVG4//77eeWVV+jUqbh7zdfXl8WLFxMREWG3hgohRG1xNTOPP84VDX85e/aXmRU1gOos3xA1+bcwF9LjINiGVQGSnFhE0OAOIW0h8Yj6ura0t6T87OIV7GUNsDJs6gE6fvw4b731FnFxcSxZsqRU8GMWEhJil+nyQghR22w6nojRpNApMoCmDX2c3wBjIWTEqdv1uQdIp6t5HlAlC746hD0rQiefBBTwaQh+ZdfXrO9s6gEqOXW9whO7uTFo0CBbTi+EELXahqLqz5r1/mTEg2ICvTv41vMvvsAmcPWs7QFQkpOLCFoSoY/W/Fwle69cbYaeC7CpB2jhwoWsWLGizP4VK1bw8ssv17hRQghRW6Vm57PjbAqg0fR3KK57Exhpn0Ta2qwmPUAFOWrwBM7rAbJnLSBnFHCsxWz6zXj33Xdp167seigdO3asdMkJIYSo6zYdT6TQpNAu3J8WoQ5aNqEqkv9TrCbFEJNPqT1p3sHg56SFbM09QFdjoCC3ZudyVgHHWsqmACghIYHGjct27YaGhpaq2yOEEPXND0XDXyO1qP1jZukBkgCouAfIhgBIiyEk/8bgFQSKEVJOVXl4pSxT4KUHqDw2BUBRUVFs3769zP7t27fLzC8hRL2VnlvAb2eSARjZWaPhL5Ap8CXVZAjM2QnQoAZa5qKFNUmEzkyGrGRAB40ctIJ9LWdTEvSsWbN4+umnKSgo4LbbbgPUxOg///nPPPvss3ZtoBBC1BabTyRSYFRo3ciPVo38tWuIBEDFSq4IryjW9eRoVUQwrANc/L1midDm4a/g5jYthlsf2BQAPf/881y9epXHHnvMsv6Xl5cXL7zwQqm1u4QQoj7ZcKRo9peWw18gAVBJAZHqfUE25FwHn+DqP9eZNYBKMgdcNUmETnTy7LVayKYASKfT8fLLL/Piiy9y4sQJvL29ad26NZ6envZunxBC1AqZeYVsO+0Cw18gSdAluXuppQCyktQ8oOoGQFlXITNR3Q518hCSPWoBJWkwfFfL1Kiut5+fH71797ZXW4QQotb65WQS+YUmWoT40jZMw+GvnFTIS1e3AyO1a4crCWxSFABdhsZdq/cccwDRIBo8nTybr1F79T4zAbKvWddrZSZrgFXJ5gBo7969fPHFF8TGxlqGwcy++uqrGjdMCCFqkx+OqDNgR3QOR6dl0Tlz7493sOR+mAU2gbj91iVCJ2q4irqnPwQ1U9cDSzwGzQda93yTEZJOqtuyCnyFbJoFtnr1avr378+JEydYv349BQUFHDt2jF9++YXAwEB7t1EIIVxadn4hW04lARpWfzYzf8kHyfCXhXkoMDW2+s8xJyBr1YNiGQY7Zv1zr1+Awhxw8675emJ1mE0B0L///W/+85//8O233+Lh4cEbb7zByZMnuffee2natKm92yiEEC5t66lkcgtMNA32oWNEgLaNkRpAZdkyFV7rVdQtidA2BEDmoCm0LegN9mtTHWNTABQTE8OoUaMA8PDwICsrC51OxzPPPMN7771n1wYKIYSr2+Aqw18gM8DKY20AZDIVDyFpVUTQsiaYDYnQWs1eq2VsCoAaNGhARkYGAJGRkRw9qnYVpqamkp2dbb/WCSGEi8stMPLLSXX4a6TWw18gAVB5gkrUAqqO1AtQkAUGTwhu4bBmVcqyJtgJNSCzhtbDd7WETQHQLbfcwqZNmwCYMGECTz31FLNmzWLy5Mncfvvtdm2gEEK4sm2nk8nONxIZ5E2XJi6QAykBUFnm4cDMBCjMq/p4Sw2ddmCo0WRp2zVsBQYPNRBLvWjdcxOlB6g6bPpk3377bXJz1UXa/vrXv+Lu7s6OHTsYN24cf/vb3+zaQCGEcGWW2V+dXGD4C6QGUHl8GoKbFxTmQnpc1YnBrrCGlsFNzeFJOKK2p7rJzPnZcO2cui0BUKWsDoAKCwv57rvvGDZsGAB6vZ65c+favWFCCOHq8gqN/HyiaPaX1tWfAYwFkBGnbksPUDGdTn0/rp5VA8SqgglXWUW9UUc1AEo6Du3vrN5zkk8CCviEgF8jhzavtrN6CMzNzY1HHnnE0gMkhBD11e9nUsjMKyQ8wIvuUUFaNwcy4kExqUMnvvLlV4o1q8K7yjISlkRoK2aCaT17rRaxKQeoT58+HDx40M5NEUKI2sW89tfwTuHo9S40/BUQCXqb/nuvu6o7E6wgB67FqNtaFxE0D2FZsyaYKwzf1RI25QA99thjzJkzh0uXLtGzZ098fUtXG+3SpYtdGieEEK4qv9DEpuNqADTSFYa/QBKgK2NZFb6KHqDkU2ovmk9D7YeQzEHM1bNqYObuXfVzEmUNsOqyKQCaNGkSAE8++aRln06nQ1EUdDodRqPRPq0TQggXtSMmhfTcQkL9PenZrIHWzVFJEcSKBVZzKnzJNbS0Tmr3DwfvBuoq9smnIKJb1c+RIbBqsykAOn/+vL3bIYQQtcoP5uGvjuEYXGH4C6QHqDLVHQJzpSKCOp3aC3Txd7VdVQVAmcmQlQzoILS9M1pYq9kUADVr1sze7RBCiFqjwGjix6LhrxGdwzVuTQkSAFWsZACkKBX37rjaKuphHdQAqDqJ0ObZa8HNwcPHse2qA2wKgD7++ONKH586dapNjRFCiNpg17lrpGYX0NDXgz7RwVo3p1iqeQhMAqAyAiLV+4JsdUjJp4LPLUnDVeDLY00itKsFby7OpgDoqaeeKvVzQUEB2dnZeHh44OPjIwGQEKJO23BULX44tGM4bgYXmW2lKMU5QEGyKHUZ7l5qaYCsJPV9Ki8AykqBzERAp1aBdgWNrFgVPtHFgjcXZ9Nv7vXr10vdMjMzOXXqFAMGDODzzz+3dxuFEMJlGE0KPx41z/5yoeGv3DTIz1S3zb0dojRzz1hqBTPBzEFGg2jw8C3/GGczB2KZiZB1tfJjXaWAYy1htz9dWrduzaJFi8r0DgkhRF2y+/w1rmblE+Tjzk0tGmrdnGLm/B+fhpL/UZGqEqFdKQHazNMfgorybpMq6QUyGbVfwb6WsWvfrZubG3FxcVY/b+nSpURHR+Pl5UXfvn3ZvXt3hcceO3aMcePGER0djU6nY8mSJZWee9GiReh0Op5++mmr2yWEEDf6wTz81SEMd1cZ/gJJgK6OqmoBuWoOjTkgS6wkD+j6BSjMATfv6q8bVs/ZlAP0zTfflPpZURTi4+N5++23ufnmm60615o1a5gzZw7Lly+nb9++LFmyhGHDhnHq1CkaNSpbhCo7O5sWLVowYcIEnnnmmUrPvWfPHt59910pzCiEsAuTSeGHo+bZXy5S/NBMagBVLaiKWkCu2AMEantObai8ByjxqHrfqB3oDc5pVy1nUwA0ZsyYUj/rdDpCQ0O57bbbeO2116w61+uvv86sWbOYMWMGAMuXL+f7779nxYoV5S6y2rt3b3r37g1Q6SKsmZmZTJkyhffff59//vOfVrVJCCHKsy/2OskZefh7uXFzyxCtm1Oa9ABVrbIhMJMJkk6o264WADWqxppglvXLXKztLsym/luTyVTqZjQaSUhIYNWqVTRuXP2/ivLz89m3bx9DhgwpbpBez5AhQ9i5c6ctTbN4/PHHGTVqVKlzCyFETWw4og5/3dEhDA83Fxr+ghI9QBIAVaiyAOj6eXWKvJsXBLdwbruqYpkKf1IN1MojCdBWs6kHyF5SUlIwGo2EhYWV2h8WFsbJkydtPu/q1avZv38/e/bsqdbxeXl55OXlWX5OT0+3+bWFEHWTyaSw0Tz7q5OLDX+B9ABVh3l4MDMBCvPAzbP4MfPwV2hb1xtCCm4JBk8oyILUC+UHaK6ygn0tYtOfMOPGjePll18us/+VV15hwoQJNW5UTVy6dImnnnqKzz77DC8vr2o9Z+HChQQGBlpuUVEyhi6EKO3g5VTi03Lx83RjQGsXG/6CEgGQ1ACqkE9DtYcHIP1K6cdceQjJ4AahbdTt8hKh87Ph2jl129WG71yYTQHQr7/+ysiRI8vsHzFiBL/++mu1zxMSEoLBYCAxMbHU/sTERMLDbauvsW/fPpKSkujRowdubm64ubmxbds23nzzTdzc3MpdqHXevHmkpaVZbpcuVbFasBCi3vmhaPjr9vaN8HJ3sR4CYwFkqO2THqBK6HQVD4Mlufgq6ubihuVVhE4+ASjgG6r9Cva1iE0BUGZmJh4eHmX2u7u7WzV85OHhQc+ePdm8ebNln8lkYvPmzfTr18+WpnH77bdz5MgRDh48aLn16tWLKVOmcPDgQQyGsv9xeXp6EhAQUOomhBBmiqKwoWjx0xGuOPyVEQ+KCQwe6pegqFhFAVCii6+ibkmEPlr2MRn+solNOUCdO3dmzZo1zJ8/v9T+1atX06GDdR/AnDlzmDZtGr169aJPnz4sWbKErKwsy6ywqVOnEhkZycKFCwE1cfr48eOW7StXrnDw4EH8/Pxo1aoV/v7+dOpUugy4r68vDRs2LLNfCCGq48iVNK6k5uDjYWBwWxcMMMxf5gGRoHex5GxXE1jOVPiCHLgWo2674hAYFAdm5Q2Buer0fRdnUwD04osvcs899xATE8Ntt90GwObNm/n888/58ssvrTrXxIkTSU5OZv78+SQkJNCtWzc2btxoSYyOjY1FX+IXOi4uju7du1t+Xrx4MYsXL2bQoEFs3brVlssRQohKmXt/bm3ngsNfIAnQ1iivGGLySbUHzaeh6w4hmQOzazFqwObuXfyYqxZwdHE2BUCjR4/m66+/5t///jdr167F29ubLl268PPPPzNo0CCrzzd79mxmz55d7mM3BjXR0dEoimLV+SUwEkLYSlEUS/Vnl5z9BZAaq95LEcSqlTcEVnIISadzfpuqwz8cvBuoK9knn4KIbsWPJbn48J2Lsnka/KhRoxg1apQ92yKEEC7neHw6F69m4+Wud83hL5AeIGuUFwAl1YJV1HU6tX0XflPbaw6AMpMgKxnQQWh7LVtY69g0WLxnzx527dpVZv+uXbvYu3dvjRslhBCu4oei4a/BbRrh66lp6bSKmb/Mg6QHqEolV4Q3jyYk1pIiguVVhDZvB7eQRXCtZFMA9Pjjj5c7VfzKlSs8/vjjNW6UEEK4grTsAr45pC7wPKKzbaU5nEJ6gKovIFK9L8yB7GvqtiWHxsWTiMPKCYBk+MtmNgVAx48fp0ePHmX2d+/e3TJDSwgharMjl9MY9dZvxF7LJtDbndvauWhyrKLIQqjWcPcC36LPMu0SZKVAVhKgUxcSdWXmAK1kLSBXLuDo4mwKgDw9PcsULwSIj4/Hzc1Fu4iFEKIaFEXhkz8uMm7ZDi5fzyEq2JvPHuyLv5e71k0rX24a5Geq2+beDVG5knlA5t6UBtHg4atZk6rFHKBlJkLWVXVb1gCzmU0B0NChQy3Vk81SU1P5y1/+wh133GG3xgkhhDNl5RXy1OqDvPj1UfKNJoZ2COO7JwbSKTJQ66ZVzDz85dNQckCqK6hELaDaVEPH018N1EANfExGdYFUcO0EbhdlU3fN4sWLueWWW2jWrJmlJs/BgwcJCwvjk08+sWsDhRDCGU4nZvDop/uISc7CoNcxd3g7HhzYHJ2rTos2k/wf65WsBZRb9Id8bQiAQB3qun5B7bnyj1Bzmdy8iwMjUW02BUCRkZEcPnyYzz77jEOHDuHt7c2MGTOYPHky7u4u2k0shBAV+Gr/Zf66/ig5BUbCA7x4+0/d6RUdrHWzqkfyf6xXcgjMXEOpthQRDOsAp75XA6CACHVfo3aut4J9LWBzwo6vry8DBgygadOm5OfnA/DDDz8AcNddd9mndUII4UC5BUb+79tjfL5bDSIGtg5hycRuNPTz1LhlVpAAyHqWqfAX1aKCUIt6gIoCtaTjxTlfkgBtE5sCoHPnzjF27FiOHDmCTqdDUZRS3cTlrbguhBCu5OLVLB79dD/H49PR6eCp21vzxG2tMehdfMjrRjIEZj3zexV/GBQjuHmpdXRqA3OglnQS/Isqk0sCtE1sSoJ+6qmnaN68OUlJSfj4+HD06FG2bdtGr169ZNkJIYTL23g0njvf/J3j8ekE+3rw8cw+PD2kTe0LfkACIFuYe8uUoj/WQ2vREFJwSzB4QkEWnNuq7qstvVcuxqYeoJ07d/LLL78QEhKCXq/HYDAwYMAAFi5cyJNPPsmBAwfs3U4hhKixAqOJRT+c5L+/nwegV7MGvP2nHoQHemncshqwBEAyBFZtPg3VxOHCHPXn2hRAGNwgtC0kHC4ufyBDYDaxqQfIaDTi7+8PQEhICHFxaqXUZs2acerUKfu1Tggh7CQuNYeJ7+60BD8P3dKCzx+6qXYHP8YCyFAXapUeICvodKXfr9qSAG1WMmDzDQU/F12jzsXZ1APUqVMnDh06RPPmzenbty+vvPIKHh4evPfee7RoUUvGUYUQ9ca208k8vfoA17ML8PdyY/GErgzr6MJLW1RXRjwoJjB4qF+EovoCm8DVM+p2bcuhKRmw1bbgzYXYFAD97W9/IysrC4B//OMf3HnnnQwcOJCGDRuyZs0auzZQCCFsZTQpvPHzad7achZFgY4RASyb0pOmDetIwcDUohlgAZGgt6lDv/4q2QNU24oIluwBqm1tdyE2BUDDhg2zbLdq1YqTJ09y7do1GjRo4PpFw+qyrBQ4+hX0uB/cvbVujWspzIcDH0ObERBYC5cLiP0Dsq9Cu1Fat8R6+dlw4BPoNB58GzrtZVMy83hq9QG2n1WXDPhT36bMv7MDXu61JNm1OiQB2nbmnCmfEPBz0XXeKlIqAJIeIFvZ7U+G4OBgCX60tuE5+OF5+PVVrVviev5YCt8/C98+qXVLrJeXAZ/cA6v/BAlHtG6N9bYtgh/+DBvnOu0l91y4xqg3f2P72at4uxtYMrEb/x7buW4FP1BcAyioqbbtqI3M094bd9G2HbbwCyueAt+4m6ZNqc1k5dK6IjcdTm5Qtw9/Cbf+TbrESzr8hXof8wtkJtWuv/hOblCnvIJ6HeGdtW2PNUwm9d8jwMnvIC8TPP0c9nKKovDer+d45cdTGE0KrRr5sWxKD1qH+TvsNTUlPUC2az8a7vgHtB5W9bGuRqeDCSvVQo7hMgRmK/mGrCtOfAvGPHU7LRYu7dK2Pa4k4WjxgoeKSR0mrE2OfFG8fXSdGlTUFhe3Q4Y6S5SCbDi1wWEvlZZdwKyP97Hwh5MYTQpjukXwv8dvrrvBD0gAVBPuXnDzU8UrrNc2TftCl3u1bkWtJgFQXXGk6K9sd5/SP4va/d5kJkPMFnXbzRvSr0DsDm3bZA0nvfdHLqdx59u/8fOJRDwMev41thP/mdgNX8863sktAZAQNpMAqC7ISITz29TtYf9S74+tV2uE1HcmExxZq27f8Q/Q6eHKXrgao227quvYerVabUQP6DxO3Xf4i8qf4yoK8+D41+q2+d/l2c1qsn4N5BYYOZmQzoYj8bz9yxmeWXOQcct2cOlaDlHB3nz1WH+m9G1W93MSFUXWAROiBur4n0f1xLGv1KGdJn2g+1TYshCyktR8lza1cHzbni79AemXwTMAut+vDsHE/KIOJQ36s9atq5q5x6TLvWq9jwOfwvH/wchXwc3FF+w8+zPkpoF/BPSYBvs/hrgDalDXZ1alT1UUhYT0XM4lZ3EuOZOY5CzOpajbV1JzUJSyzxnaIYxXJ3Ql0NvdQRfkYnJTiysBB9TCmY1CaEwCoLrA/CXZeYJaJr3TPbBrubq/vgdA5t6S9nepY/6dJ6gB0OEv4Jbn1WRCV3XtPFzerfZadRxbVPE1HDIT1J6UdiO1bmHlzP8uO92jrrPUeYIaAB350hIAZeUVcj4li5jkTDXYKQpyzqdkkZ1f8aLK/l5utAj1o2WILy1CfekYGcjgNqF1v9enJPPwl09D8KgjdY2EcCIJgGq7qzFwZR/oDOqXJKhfNLuWw8nvHT7rxqUV5hcPwXQer963uxPcnlErwMYfgohuWrWuakeLhu6a3wL+RVWLO41Tp/Qf+cK1A6DcdDj1g7rdeQIJabmc87qFfujQXdrFU8u+Ztd1fxLScys8hUGvo2mwDy2KgpwWoX5F236E+HnUr2CnPLIGmBA1IgFQbWfOb2l5a/F6MJE9oUFzuH5e/RLqMkG79mkp5hfIua7WzGh+i7rPKwDaDFcDoyNfum4ApCjF08c7l5jp0WWCGgCd+kGtD+TpojOcTn4PhbkQ0oZ3Tvny6k+bURT41L0DAwzHiLy8gQTj3QAE+3qUG+Q0DfbBw03SFCskCdBC1IgEQLWZohRPke5cIsjR6dSff31Ffby+BkDm96bTOHUIxqzzBDUAOrpOTYzWu2BxvIQjkHIKDJ7Q/s7i/Y27QcNWcPUsnPgOuk3WrImVKnrvDwQO4ZUfTwPQqpEfMZ4jGJB8jIeD93H7uEW0bORHkI+Hli2tvSQBWogakT+varP4g+oXoZt32SUSzAGRHWbd1Ep5mcWFITvfEAC2vgO8AtWFJC9ud37bqsOcP9N2uNpWM52uuEfIVafzZybBua0APHW8tXp/e2t+njOIaQ88CQZPAjPO0tMrToKfmpAeICFqRAKg2sw8/NV2RNmhkNA20LirOoXanAdTn5z8HgpzILglRHQv/ZibJ3RQh19cckq5yaT2TkHZ4A2K85nObVGDDVdzVJ2VeMDUilgljBk3R/P0EDUQwisQ2gxVt4+44Htfm0gAJESNSABUW5mMxQFQRdVAzT0Fh120p8CRSk4fLy9Z1vzeHP9GrVfjSmJ3qAUPPQOh9dCyjzdsqeZ5KSZ1SrmLydj7OQBfG29mfM8mvDiqQ+mEZUsPVi2rau1qUmUITIiakACotrrwuzod2isIWt5e/jGd7gF0ai2c6xed2TptZaWoCdCgrkBenmb91fo0eWlwZpPz2lYd5uCtw10V1/ox9wy52DDY0aMH8U85iFHRkd1qNIvu6Yxef0MA2nqoWpcp/bL6b1NYz1igDuGC9AAJYSMJgGor8/BBxzHgVkEeRUAERA9Qt81TqusDS/Xk7hDSqvxj9IaiABHXGoopzIdjX6vb5Q1/mXW8R60PdHkPXDvnlKZV5UR8OlvXLgXgmHcP/nn/bbgZyvkvxt1LrcsErjkEWRukxwGKmiTvG6p1a4SolVwiAFq6dCnR0dF4eXnRt29fdu/eXeGxx44dY9y4cURHR6PT6ViyZEmZYxYuXEjv3r3x9/enUaNGjBkzhlOnTjnwCpysIBeOf6tud65iMTzz8NiRehQAHSln+nh5zO/NqY1q3RpXcPZntcKvf+Pi4LU8/mHQfJC6fWSdU5pWmQspWdz/wS5GmH4DoO0dM/F0q2R2nXlm4vGv1aBPWMeS/xMJepf4b1yIWkfz35w1a9YwZ84cFixYwP79++natSvDhg0jKan85M7s7GxatGjBokWLCA8PL/eYbdu28fjjj/PHH3+wadMmCgoKGDp0KFlZWY68FOc5u0kdugloAk37VX5s+7vA4KGuhp54zDnt09L1C3Bpl9o7Yu7hqUh4FwhpA8Y8OPGtU5pXpYqm7pfHMgz2BeWuDeEk8Wk5TPlgF2HZp2ipj0dx88Kz412VPyl6oFrVOuc6xGx2TkPrEkmAFqLGNA+AXn/9dWbNmsWMGTPo0KEDy5cvx8fHhxUrVpR7fO/evXn11VeZNGkSnp7l50ds3LiR6dOn07FjR7p27cpHH31EbGws+/btc+SlOI952KDzuKr/+vMOKk6krQ/DDUfKqZ5cEVebUp6XUap6cpXaj1aHQFJOQ8Jhx7atAlcz87jvg11cSc1hqt8eAHRtR6gFJyujN6hBHrjGe1/bSA0gIWpM0wAoPz+fffv2MWTIEMs+vV7PkCFD2Llzp91eJy0tDYDg4OByH8/LyyM9Pb3UzWXlpsHpH9Xt6nxJljzuaB2fdaMopddFqw7zCuvnt0FGomPaVV3m6skNW6slDKriFaDWCQJNgoj03AKmfbibmOQsmgS4M96jKKG52u99UYL6yQ1q8CeqT3qAhKgxTQOglJQUjEYjYWFhpfaHhYWRkJBgl9cwmUw8/fTT3HzzzXTq1KncYxYuXEhgYKDlFhXlwn9VnfhWHbIJbQdh5V9PGW2GgYe/+ldjXZ51k3gUkk8WVU8eXb3nBLeAyF5FU8q/cmz7qnK4RFXv6q5zZRkGW6eWRnCSnHwjD360l6NX0mno68GaEQqGrAS1zk+rIVWfANQk9eCWar0mc9FKUT0SAAlRY5oPgTna448/ztGjR1m9enWFx8ybN4+0tDTL7dKlS05soZVK9nBU90vS3VudUl3y+XWR+draDCtdPbkqXVxgGKxE9WRLz0h1tB6q1gvKiIOLOxzStBvlF5p49LN97L5wDX8vN1bO7EPkpe/UBzuMqXjq/o10Otd472sjyxCYBEBC2ErTACgkJASDwUBiYumhh8TExAoTnK0xe/ZsvvvuO7Zs2UKTJhX/R+Hp6UlAQECpm0vKSIDzv6rb1nxJljz+2Pq6OevGZCqeDVXdIRizjmNBZ4Ar++BqjP3bVh3mqfuRPdVCh9Xl5lkiuHV8jpfRpPDMFwfZeioZL3c9H07vTacwLzj+P/UAa9978/Exv0Bmsn0bW1cpSokeoKbatkWIWkzTAMjDw4OePXuyeXPxLBCTycTmzZvp16+K2U2VUBSF2bNns379en755ReaN29uj+Zqr2iJAaL6QoNo657bfBD4NiqadfOLQ5qnqdidamG9iqonV8avEbQYrG5rVS6gulP3y2PuRTn+P4dWtVYUhb+uP8L3h+NxN+h49/5e9IoOVgtJ5qaphSWb3WzdSRu2hIge9XfJFlvkpkJ+prodGKlpU4SozTQfApszZw7vv/8+K1eu5MSJEzz66KNkZWUxY8YMAKZOncq8efMsx+fn53Pw4EEOHjxIfn4+V65c4eDBg5w9e9ZyzOOPP86nn37KqlWr8Pf3JyEhgYSEBHJycpx+fXZlbYJvSXV91o2596PDaLXQnrVKVlZ29pTya+fVgoY6vdobZa1mN6t1g3LT1DpCDqAoCv/ecILVey6h18Ebk7ozqE1RAT7Lv8tqzEosj/m9rw+zFO3B3PvjE6IObwshbKJ5ADRx4kQWL17M/Pnz6datGwcPHmTjxo2WxOjY2Fji4+Mtx8fFxdG9e3e6d+9OfHw8ixcvpnv37jz44IOWY5YtW0ZaWhqDBw+mcePGltuaNWucfn12czUG4varQzUdxth2DvMXzakN6mrpdUV1qydXpt0ocPOCq2cg/qC9WlY9lqn7g9QCh9YqGdw6KIhYuuUs7/92HoBF93RhZOfG6gO56XB6o7pt63vfyVzVercaDIrKSQK0EHbhpnUDQM3VmT17drmPbd26tdTP0dHRKFX8hV7V47WS+a/slreBn42l7yN7qLOerp1Tg6CKFlGtbWI2q8MCfuFqgT1beAVA2xFqLs6RtWVXkHcURSnuvarJ59F5Aux8Ww1GctOrrsNjhZU7LrD4p9MA/G1Ue+7tXWKW5Mnv1Kn7IW3VwpK28A9X6zad26qWarjluZo3ui6TAEgIu9C8B0hUgy31bcqj07nsIpo1ctiK6smVsbw3a503pTzhsFrI0M0L2t1p+3kad1XrBxXmqkGJnazbd5kF36gVxJ+8vTUPDmxR+gBbpu6XR8shyNpGiiAKYRcSANUGcQfg6llw81aHamrC/EVzdrO6anptV7J6cpcaBIcAre4AryDITIALv9e4adVimbo/vGa9Ng6YUv7jsQT+vE6tMD29fzTPDGld+oCMRLWAJFg/K/FG5qrWySfVek6iYqkyBV4Ie5AAqDYw54i0GwmefjU7V0hraNxNnXVzbH2Nm6a5k9+rhfQatlKvqybcPKDD3eq2M3rIajJ1vzzmPKBzW9W6QjWw/WwKT6w6gNGkMK5HE+bf2QHdjT08x9arsxKb9IbgGs609ApU6zeBJENXRYbAhLALCYBcncmo5kWAbVOky1OXhsFsKQxZGfN7c/wbKMit+fkqc3G7WsDQKxBa31Hz8zVsqdYRUkxqyQQb7Y+9zqyP95JvNDGsYxgvj+uMXl/Oe3ukxPCXPdSXJVtqyhwABckQmBA1IQGQq7vwmzok491ATYC2h07jAJ26avr1C/Y5pxYykyFmi7ptry/hZjer9Wzy0uDsJvucsyLm4K3D3dWvnlyVGi7uejIhnRkf7iE738jA1iG8Obk7boZy/pu4GqMWjtQZbJu6Xx5zVev0KxDrnKrWtY6xADKKZsVKDpAQNSIBkKs7bP6SHKMO0dhDQGNoXjRbyty7VBuZqydH9LCuenJl9PriBVIdORRTmFdc+M9ewRsUVbXWw5W9Vle1vpCSxf3/3U1aTgE9mgbx7v098XSrIKncPCzbYrBaSNIe3L3UOk5QN3onHSE9DlDUfCmfEK1bI0StJgGQKyvIhRPfqNv2nrJu7ik4XItn3Zi/JB313pz+US0u6Ahnf7a9enJl/MOKq1pbEdzGp+Uw5YNdJGfk0S7cnw+n98HHo4IqGSVnJTrqvT/2dd1csqWmLPk/kbYVnRRCWMhvkCs78xPkpUNAE4i6yb7nbj8aDB6QfAISj9n33M5w7bxaOM/W6smVCe+s1rUx5sGJb+17bjPL1P17ajZ1vzwlKytXI7hNTM/lvg92cSU1h+iGPnzyQF8CfdwrfkL8QbVgpJtXzWcl3ih6gFrPKTfVYVWtazVJgBbCbiQAcmWWJNPx9v9rzzuoeM0sJyyiaXdHzdWTb1EL6dmTo+sl2aN6cmXa3VmiqvWhSg89n5LFuGU7iEnOonGgF58+2JdQ/yrykczDX21HgKe/nRpdpK4v2VJTabHqveT/CFFjEgC5qpxUOP2Tuu2IL0koUTemls26UZTi3Ch7zYy7kbmuzflfISPBvuc++X1R9eQ2agFDe/MKUOsKQaVBxNEraYxftoPL19Weny8e7keTBj6Vn9sRsxJvZK7ndOoHtc6TKCY9QELYjQRArurEt+oQTGh7COvomNdoPQw8A9RV1C/94ZjXcISEI5BySk0EbV+D6smVCW6u1rep4ZTych2xU/XkypSaUl62qvWOmBQmvfcHV7Py6RgRwJeP9CcquIrgB9QCkRnxasHIVkPs22azxt3Uuk6FOWqwKIpZAiDpARKipiQAclWWJFMHfkm6e0H7u9Tt2lR8zvzetB2u1tBxlBpOKS9XZpJaqBBqXj25Mq3vUN+bjHi13lAJG4/GM33FHjLzCunXoiGrH7qp6mEvM/N70XGM/WYl3kinc8x7XxdID5AQdiMBkCvKSFCHXgA6OfBLEoq/hI9/XTtm3ZhMJYZgHDQ0aNZxjFrnJm6/1VPKK3T0K7VXKbKXujCto7h5Fle1LhHcrtoVy2Of7SffaGJ4x3A+nNEbf69KEp5LKshVC0SC499787/LmC1qvSehDv1KD5AQdiMBkCs6ug5Q1JlfDZo59rWa3wJ+YZBzXV1V3dXF7lAL5XkGqmt3OZJfo+Ip5fbqibDHorbVVaKqtVKQy9u/nOEv649gUmByn6YsndIDL3crZqCd3aQWiAyIhKb9HdNms4Yt1fpOdWXJFnvITYX8THU7MFLTpghRF0gA5IosX5IO7v2B2jfrxtyb0eEudQjP0UouMFrTeknXzqkFCnV6dfq7o5Woar3m8xUs/uk0AE/c1op/j+2EobzlLSpj/vfRaZxzatBY3vtaNDzrSObeH58QcPfWti1C1AESALmalLPq6u/2XGKgKuZA6+QGyMt0zmvaojAPjv9P3XZGDwqodW7cvOFq0edSE46onlwZvQFjRzXQ8j+j9qIsGN2BZ4e2LbuwaVVy0+CUA6ful6fjPWqweHmPWvepvpP8HyHsSgIgV2P+K7vV7eDrpFL3ET0guKXrz7o5u1kdBvBvrBbMcwZPf7XeDRQHMLZQlOLeK0dNH79Bdn4hf7+gziAcoj/A0nGtmHGzjau2n/iuaFZiO7VQpDP4h0HzQep2Td77uiL1knovAZAQdiEBkCspucSAs/7KBscX/rMX81BIp3H2r55cmSqmlFdL/CHHVU8ux/WsfP70/i4+uRBAjBKBp66AUW57bT9hyaKcjpqVWB7Lv8vqVbWu09KKAqCgptq2Q4g6QgIgVxK3H67FgLsPtB3p3Nc2f9HE/OKas27yMtTCeOCc3KiSWg1R695kJsCF32w7hzmwbDNcLVToQHGpOUx4dycHL6US5OOBd89JpdtgLWfOSrxR+zvVek8pp9X6T/WZDIEJYVcSALkSyxIDI8HTz7mvHdIKIrqrs27Mq5S7khPfqdWTG7ZWC+U5k5uHOiUebAsiSlZPtvfioTc4m5TBuGU7OJuUSeNAL9Y+0o+IAferD57fBhmJ1p/02Hp16n6TPmqBSGfyClTrPYEkQ0sAJIRdSQDkKkotMeDE4a+SXHkYrOTQoDOHYMxKTCmnINe6517cXlQ9OdBx1ZOBA7HXGb98J/FpubQM9WXto/1p1chfrTcU2UsNYo7ZUNX6cInK1Vqw/LusZUu22JsEQELYlQRAruL8r5CZCN7BagK0FjqNA3RwaRdcv6BNG8qTmQTntqjbzh7+MmvaX61/k5cOZ36y7rnm4K3DGLVAoQNsO53Mn97fRWp2AV2jgvjykf5EBpWYKt3FxsrKV2PUoVlnzkq8Ueuhat2njLgyVa3rDWOBGkSDFEEUwk4kAHIV5uGvjmPAUM3KvPbmH64WRizZHldgHoKJ7KkWyNOCXl+iXpIVQzFOmLr/v4NXeHDlHnIKjAxsHcKqB/sS7HvDMhUdx6pTyq/ss66qtTlgankr+IXar9HWcPNU6z6VbE99kx4HKGo+lI+TZocKUcdJAOQKCnLhhHmJAedMka5QyWEwV5l1o8XMuPKYX//0T5CTWr3nnNmk1tDxj1ALE9rZR9vP8/SagxQYFUZ3jeC/03rj6+lW9sBSVa2rGdxqNSuxPJYhyP+pQWV9Y54BFhjpnCKUQtQD8pvkZPmF5eQwnPlRHVoJjIKovs5vVEkd7lL/ykw+CYlHtW0LqNWTL+9Rey86OqF6cmXCO6t1cIx5cPK76j3HEkDYt3qyoii8/tMp/v7tcRQFpvVrxhsTu+HhVslrdLayqnX8QbUApJu3U6buVyp6gFr/KTcVzv6sbVu0IGuACWF3EgA5UVxqDiPf/I31By6XfuBwiRorWv915xUIbYaq264w3HCkKDG8+SC1MJ6WdLriHKTD1RgGy02H0/avnmw0Kfz166O8+ctZAJ69ow1/v6sj+qqWtmg3Sq1DdPWMGtxU5XDR5992hFoQUku1bckWe7P0AEkAJIS9SADkRGv3XeZsUibPfXmYn44lqDtzUouTarUeZjCz9BRoPOtGUYrzbRw8fbzazJ/R+V/V+jiVOVk0dT+kLYR3scvL5xUaeeLz/azaFYtOB/8a24knbm9dvaUtvAKqX9XaiVP3q8383p/6QQ0u6xOZASaE3UkA5ESzb23F+J5NMJoUZq86wPazKXDiWzDmQ6MOENZR6yaqWg8FzwBIvwyxO7VrR8JhtQCewRPa3aldO0pqEK3Ww0EpDhAqUnL6uB2m7mfmFTLjwz1sOJKAh0HP0j/1YErfZtadxJLjtbbyqtYXflMLP3oFQUuNZiXeqHFXtQ5UYa5rL9niCBIACWF3EgA5kV6vY9E9nRneMZx8o4lZH+8lY88q9UFX6f0BdZX19uZZNxoWnzMPdbS1T/Xk3AIbl7G4UXXqJWUkqoUHQc3/qaGUzDwmv/cHO2Ku4uth4KMZvRnZubH1J2o1RB3mzEyAC79XfJz52jqOUQtBuoJSS7bUs6KIEgAJYXcSADmZm0HPG5O7MbB1CH75KfjGF/WwaFXfpiJdir5ojn0NhfnOf32TsTj/p4Yz41Kz8/nL+iN0mL+RJz8/QF5hDQOhjmPVujhxByDlbPnHHPuqqHpyb7UQYQ1cupbNhOU7OXIljYa+Hqx+qB/9W9k4FdrNU61HBBUHEQW5asFH0H5W4o3Mvyfntqr1oeoDRZEkaCEcQAIgDXi6GXj3/p48FnIQPQoHdO25UNhQ62aVFj0Q/MLVWTcxm53/+hd3qIXvvAKh9R02ncJkUvhy7yVue20bq3bFYlLgm0NxPPDRXjLzCm1vm1+oWhcHKu4FstP08ZMJ6YxbtoPzKVlEBnnz5SP96NwksEbnLJ5S/m35Va3P/KTOSgxoAk371ey17K1hS7UelGJS60PVBznXIT9T3Q6M1LYtQtQhLhEALV26lOjoaLy8vOjbty+7d++u8Nhjx44xbtw4oqOj0el0LFmypMbn1IKPhxv3+altWpd/E1M+2EV8Wo7GrSqh5Kyb6sx4sjdz70SHu22qnnwyIZ2J7+3k+bWHuZaVT9swf/4+ugO+HgZ+P5vCn97/g6uZNagnU9mU8qsxasHBGlZP3nvhGvcu30lSRh5tw/z56rH+tAi1wxpxzW5W6xLlpcHZTWUfd9DUfbsxv/da/LvUgrn3xycE3L0rP1YIUW2a/++2Zs0a5syZw4IFC9i/fz9du3Zl2LBhJCWV372dnZ1NixYtWLRoEeHh4XY5pyZSzuCWcAhF78aRoNu4kprDfR/sqtmXsr2ZhxtO/aCuxu4sNaienJlXyL++P86oN39nz4Xr+HgY+OvI9nz35ACm39yczx+6iWBfDw5fTmPC8p1cupZtWxvbjVTr41wrWiqiJPMMqxaD1QKENth8IpEpH+wiPbeQXs0a8MXD/QgL8LKtrTfS64vzkm4MInLT4PSP6rYr5aWVZKlqvVetE1XXmQOgIBn+EsKeNA+AXn/9dWbNmsWMGTPo0KEDy5cvx8fHhxUrVpR7fO/evXn11VeZNGkSnp7l9wxYe05NFP2VrWt5O+/MuoOIQC9ikrOY9uFu0nMLNG5ckYju0LAVFOY4d9aNDdWTFUVhw5F4hry2jfd/O4/RpDCyczibnx3ErFta4G5Q/6l3aRLE2kf6ERnkzbmULMYv38GpBBuCO09/NQiC0lPKS1ZPtnH6+Lp9l3nok33kFZq4rV0jPnmgL4E+dl4exdyLcvpH9b02O/GtWugxtD2EdbLva9qLf5j1Va1rM0mAFsIhNA2A8vPz2bdvH0OGFK+QrdfrGTJkCDt32jb92pZz5uXlkZ6eXurmUDcsMRAZ5M0nD/aloa8HR6+k8+BHe8nJt9OMpZooNevGicXnzK/V6R51KK4KF1KymPbhHh77bD8J6bk0Dfbhwxm9eWdKTxoHlh0yaBHqx7pH+9MmzI/E9DwmLN/B3gvXrG+n+b05uq54Snn8QbXQoJuXTdWT3//1HM9+eQijSeGeHpG8e39PvD2qfg+sFt5ZrU9kzFODHrOSRTntMHXfYczv/eEvXGfJFkeRIohCOISmAVBKSgpGo5GwsNIVfsPCwkhIqKLInB3PuXDhQgIDAy23qCgH/0dzZb/ade/uY+lFaBnqx8qZffD3cmP3hWs8+tm+8pfNcDbzF03MFshMdvzrWVE9ObfAyH82nWbokl/59XQyHgY9T93emp+euYVb21Y+9BQe6MUXD/ejZ7MGpOcWct9/d/HLyUTr2trydvBuAJmJamFEKO6RsLJ6sqIoLPzhBP/acAKAWQObs3h8V0vPld2VF9xmJBRfh6vNSrxRuztLVLU+pHVrHEt6gIRwCM2HwFzBvHnzSEtLs9wuXbrk2Bc0f+G0GwUevpbdnSID+XB6b7zc9Ww9lcycLw5iNGn8123DlhDRAxSjc2bdWKont1EL31Vg66kkhi35lTc2nyG/0MTA1iH8+MwtPHNHG7zcq9djEuTjwacP9OW2do3ILTAx6+N9rNt3ueonmrl5lJhSvrZ09WQrpo8XGk38ee1h3t2m5rPMG9GOv47qUPXSFjVlDnLMVa2PfgUo6np0DaId+9o15RUAbYar23V9aQwJgIRwCE0DoJCQEAwGA4mJpf/yTkxMrDDB2RHn9PT0JCAgoNTNYUp9SZbt4egVHcy79/fC3aDju8Px/O3rIyhad/E7cxis5PTxcoZg4lJzePTTfUz/cA8Xr2YTHuDFO1N68PHMPjQP8S1zfFW8PdSSBPd0j8RoUnj2y0N88JsVibXm9+bEN3B2M2TEq9WTWw2p9GlmuQVGHvl0P1/uu4xeB6+M78LDg1pafR02CW6u1ilSTGrwc6RE5eraoLwhyLrIMgQmAZAQ9qRpAOTh4UHPnj3ZvLm4zozJZGLz5s3062db/RFHnNOuzm+DrCTwDoaWt5V7yKA2obwxqTt6HXy++xILfzipbRDU6R511s3l3XDtvONeJyNRLXAHZYZgCowm3vs1hiGvb+OHowkY9DpmDWzOz88OYmTnxtVbC6sC7gY9iyd05cEBzQH45/cnWFTd97xpP7VeTl46fD9H3VfN6slpOQVM/e9ufj6RiKebnnfv78W9vZyc52HuqfrjHbWwYw2n7jtV6zvUOlEZ8XBxu9atcYzC/OI15yQHSAi70nwIbM6cObz//vusXLmSEydO8Oijj5KVlcWMGTMAmDp1KvPmzbMcn5+fz8GDBzl48CD5+flcuXKFgwcPcvbs2WqfU1PmHJGOY8FQ8cyekZ0bs+gedQHN9349xztbY5zRuvL5h0PzWwBI2P4padkOmqV2bL3aGxHZq1T15N3nrzHqzd/494aTZOcb6dWsAd8/OYC/juqAn6ebXV5ar9fx11HtmTuiHQDLt8XwwrrDFBqryMMqOaXc/Jd6NXpQktJzmfjuTnZfuIa/lxsfz+zDHR00WO2+4xg16DG3veVt4GtjlWlnc/NU60RB3R0Gy4gDFHU9PN9QrVsjRJ1in2+PGpg4cSLJycnMnz+fhIQEunXrxsaNGy1JzLGxsehLFGOLi4uje/fulp8XL17M4sWLGTRoEFu3bq3WOTVTkFO8xEA1pkjf2zuK9NwC/vn9CV798RT+Xm5M7Rft2DaWIyY5kwP5NzGerWTv+ZQFOwvx93IjLMCLMH8vwgM91e0AL0L8PHGzNXflwCfqfVEAkZKZx8INJ1m3X82BaODjzryR7Rnfo4lD8mN0Oh2PDGpJsI8Hc786zBd7L3Mtq4C3/9S98ryizhNg+xvqdkAkNO1f6etcSMni/hW7uHQth1B/T1bO6EOHCAcOu1bGr5E6pdxc7bu2DH+ZdZ4A+z9W60aNXGxT0UyXVjL/x5Vn5QlRC+kUzRNMXE96ejqBgYGkpaXZNx/o2Nfw5TQIbApPHap2ld3XN53mzc1nAPjPxK6M7e6cXIC9F67x7q/n+PlEIn5KNns9H8VT5+AaRTo9xmdO8vnxXF7ZeJL0XHXJisl9mvLnYW1p4OuchTl/OpbA7M8PkF9ook90MO9P60WgdwU9dooC7/SD5BPQ/0kY+lKF5z16JY3pH+4mJTOfZg19+GRmX5o29HHQVVTTodWw/mG1sOPzZ8HTDtWmncVkhP90UntKJn4K7Udr3SL7Mn82zQfBtG+0bo0QLs+a72/Ne4DqFU9/tXegWT+rlhh4Zkhr0nMK+GjHBZ778jC+Hm4M7WhbknhVjCaFTccTee/XGPbHplr2920fTVzYAqLjvsdkMpFbYCS3wHxftF1oxFRJPO2m1+HlbsDLzYCnux5vdwNe7gY83fToi/66jW88hEc+OcuhS+prd2gcwD/HdqJH0wYOud6KDO0Yzicz+/Dgyr3svnCNie/u5OOZfWhUXjVmnQ5GvgL7VkL/Jyo8586Yq8z6WF2HrEPjAFbO7EOovwv0WHQcC5d2qUOPtSn4gaIlW+6BnW+rw8t1LQCSGkBCOIz0AJXDYT1AZiaT1WssmUwKz689zLr9l/Ew6PlwRm9utnVF8HLkFhhZt/8yH/x2nvMpWQB4GPSM7R7JrFua06pR1TVtTCaFhPRcziVncS4lk3PJWcQkq/dxaTkV1qvT66BJAx8aB3qx58I1TAr4e7rx7NA23HdTM9wcVQunGo7HpTN1xW5SMvNoGuzDJw/0oVlD62ebbTwaz5OfHyTfaOKmFsG8N7UXAV52ru5cX8UfgndvUesCPXdGnSJfV3z7FOz7CAbNhVvnVXm4EPWdNd/fEgCVw+EBkI0KjSYeX7WfH48l4uNh4LMH+9K9hj0j17Py+eSPi6zccYGrWfkABHi5cd9NzZjeP7r8Hg8b5BYYOZ+SpQZHyZmcSym6T84i44aV2e/uFsFfR7a322vXVOzVbO5fsYuLV7MJ8fPkoxm96RRZ/RXZP98dy1/XH8GkwLCOYbwxqYqcImEdRYGlfSDlNIxZBt3+pHWL7OfTcXD2Z7jrbehxv9atEcLlSQBUQ64aAAHkFRp5cOVefjuTQqC3O2sevol24da38dK1bD747Rxf7L1MToFaQyUyyJuZA5ozsXeU3WZXVUVRFJIz8ziXnMXFq1m0auRPz2bOHe6qjqSMXKav2MPx+HT8Pd14b2ov+rVsWOlzFEXhna0xvPrjKQAm94nin2M6Y3B0gcP6aNsrsOVf0OJWmPq11q2xn7f7QMopuP9raHmr1q0RwuVJAFRDrhwAAWTnF3LfB7vYH5tKiJ8nax/pR3Q1iwAevpzKu7+e44cj8ZiLTHdoHMDDg1owsnNjxy29UAek5xYwa+Vedp2/hoebnjcndWd4p/JzsUwmhZe+P86H2y8AMPvWVjw7tE2N6hWJSlw7B292V+tVzTmpLpha2ykK/DsSCrLgif1qVXYhRKWs+f6Wb7tayMfDjQ+n96FduD8pmXlM+WAX8Wk5FR6vKApbTiYx6b2d3PX2dr4/rAY/A1uH8OkDffn+yQHc3S1Sgp8qBHi5s3JmH4Z2CCO/0MRjn+1j9e7YMsflF5p45ouDluBn/p0deG5YWwl+HCm4hZrErZics2SLM+RcV4MfgIAIbdsiRB0k33i1VKCPO5880Jfohj5cSc3hvg92cTUzr9Qx+YUm1u67zPAlvzHjoz38ce4abnodY7tHsuHJgXzyQF8GtA6RL2YreLkbeGdKDyb1jsKkwNyvjrB0y1lL1ejs/EJmfbyX/x2Mw02vY8nEbswsqjAtHMxcW8u8pEdtZ64B5BsK7t7atkWIOkimwddiof6efPpgXyYs30lMchbTPtzNqlk3AfD5rlg+3H6BhPRcAHw9DEzu05SZA5oTEST/mdaEm0HPwns609DPg6Vb1Byfq5n5zL6tFTM/2sPBS6l4uxtYdl8PBlexKr2wo45jYeNcuLIPrsbU/iEjWQRVCIeSAKiWa9LAh08e6MvEd3dy9Eo697yzg4S0XDKLZlY18vdkxs3N+VPfphUX8hNW0+l0PD+sHcG+nrz03XFWbD/PF3svkZlXSJCPOyum93Z67aJ6z1LV+he1JtDgF7RuUc1IACSEQ8kQWB3QqpEfK2f2wd/TjbNJmWTmFdK6kR+vjO/Cby/cyqODW0rw4yAPDGjOkondcNPryMwrpHGgF18+3E+CH62YF3c98iUVFp6qLaQIohAOJT1AdUSnyEA+fbAvq3bFMqxTGIPbNHLIelmirDHdI2kU4MnGowk8PKglkTLEqJ12o9SCiFfPqAUSI7pp3SLbWQIg6QESwhEkAKpDukYF0TUqSOtm1Ev9W4bQv2UtWUW9LvMKgLYj1JlgR76s5QGQeQhMeoCEcAQZAhNC1C3mFe2PrlMXS62tJAdICIeSAEgIUbe0GgJegZARDxd+17o1tinMh4wEdVt6gIRwCAmAhBB1i5sndBijbh/5UtOm2CwjDlDA4Am+MrQqhCNIACSEqHvMw2DHv4HCvMqPdUUlh7+kUKkQDiEBkBCi7ml2M/hHQF4anPlJ69ZYT/J/hHA4CYCEEHWPXg+dx6nbtXEYLFVqAAnhaBIACSHqJvMw2KmNkJumbVusJTWAhHA4CYCEEHVTeBcIaQvGPDjxndatsY55CCxIeoCEcBQJgIQQdZNOV9wLVNuGwSQHSAiHkwBICFF3dR6v3p/fBhmJ2raluhRFqkAL4QQSAAkh6q7g5tCkNygmOPaV1q2pnpzrUJClbgdEaNsWIeowCYCEEHWbeRjs8BfatqO6zL0/vqHgLgvrCuEoEgAJIeq2jmNBZ4C4/XA1RuvWVE1mgAnhFBIACSHqNr9G0GKwun1kraZNqRZJgBbCKSQAEkLUfV3uVe+PfKEmGbsySw9QU23bIUQdJwGQEKLuazcK3Lzg6lmIP6h1ayonPUBCOIUEQEKIus/TH9qOULcPu3hNIAmAhHAKCYCEEPVD56JhsKPrwGTUti2VkQBICKdwiQBo6dKlREdH4+XlRd++fdm9e3elx3/55Ze0a9cOLy8vOnfuzIYNG0o9npmZyezZs2nSpAne3t506NCB5cuXO/IShBCurtUQ8AqCzAS48LvWrSlfYT5kJKjbUgRRCIfSPABas2YNc+bMYcGCBezfv5+uXbsybNgwkpKSyj1+x44dTJ48mQceeIADBw4wZswYxowZw9GjRy3HzJkzh40bN/Lpp59y4sQJnn76aWbPns0333zjrMsSQrgaNw/oOEbdPuKiNYEy4gAFDJ7gG6J1a4So0zQPgF5//XVmzZrFjBkzLD01Pj4+rFixotzj33jjDYYPH87zzz9P+/bteemll+jRowdvv/225ZgdO3Ywbdo0Bg8eTHR0NA899BBdu3atsmdJCFHHmYsiHv8WCnK1bUt5UkvUANLptG2LEHWcpgFQfn4++/btY8iQIZZ9er2eIUOGsHPnznKfs3PnzlLHAwwbNqzU8f379+ebb77hypUrKIrCli1bOH36NEOHDnXMhQghaoem/SEgEvLS4MxPWremLFkFXgin0TQASklJwWg0EhYWVmp/WFgYCQkJ5T4nISGhyuPfeustOnToQJMmTfDw8GD48OEsXbqUW265pdxz5uXlkZ6eXuomhKiD9HroNE7ddsUV4iUBWgin0XwIzBHeeust/vjjD7755hv27dvHa6+9xuOPP87PP/9c7vELFy4kMDDQcouKkr++hKizzMNgp3+E3DRt23IjSxFE+T9ICEfTNAAKCQnBYDCQmJhYan9iYiLh4eHlPic8PLzS43NycvjLX/7C66+/zujRo+nSpQuzZ89m4sSJLF68uNxzzps3j7S0NMvt0qVLdrg6IYRLCu8Moe3AmAcnvtW6NaVJD5AQTqNpAOTh4UHPnj3ZvHmzZZ/JZGLz5s3069ev3Of069ev1PEAmzZtshxfUFBAQUEBen3pSzMYDJhMpnLP6enpSUBAQKmbEKKO0umg83h129WGwSQAEsJp3LRuwJw5c5g2bRq9evWiT58+LFmyhKysLGbMmAHA1KlTiYyMZOHChQA89dRTDBo0iNdee41Ro0axevVq9u7dy3vvvQdAQEAAgwYN4vnnn8fb25tmzZqxbds2Pv74Y15//XXNrlMI4UI6jYdf/gnnf1Xr7viX3+PsVIpSIgCSITAhHE3zAGjixIkkJyczf/58EhIS6NatGxs3brQkOsfGxpbqzenfvz+rVq3ib3/7G3/5y19o3bo1X3/9NZ06dbIcs3r1aubNm8eUKVO4du0azZo141//+hePPPKI069PCOGCgptDkz5weTcc/Qr6PaZ1iyDnOhRkqdsBEdq2RYh6QKcorr40svOlp6cTGBhIWlqaDIcJUVfteg9+eB4iesBDW7RuDcQfgndvAd9QeP6s1q0Rolay5vu7Ts4CE0KIKnUcCzoDxO2HqzFat0aGv4RwMgmAhBD1k18otLxV3XaFZGhJgBbCqSQAEkLUX+aaQIe/UJOQtSQ1gIRwKgmAhBD1V7tR4OYN12Ig7oC2bZEeICGcSgIgIUT95ekPbUeo20fWatsWCYCEcCoJgIQQ9VuXe9X7o+vAZNSuHSVXghdCOJwEQEKI+q3l7eAVBJkJcOE3bdpQmKe+PkgOkBBOIgGQEKJ+c/OAjmPU7cMazQZLjytqixf4hmjTBiHqGQmAhBCic9Ew2IlvoCDX+a9fMv9Hp3P+6wtRD0kAJIQQTftBQBPIS4czPzn/9SUBWginkwBICCH0eug8Tt0+8oXzX18CICGcTgIgIYSA4qKIp3+CnFTnvrYUQRTC6SQAEkIIgLBOENoOjHlw4lvnvrb0AAnhdBIACSEEqMnH5l4gZ68NliY1gIRwNgmAhBDCrPN49f78r5CR4JzXVBRZCV4IDUgAJIQQZg2iIaovoKiVoZ0h5zoUZKvbAZHOeU0hhARAQghRirOHwczDX76NwN3LOa8phJAASAghSuk4FnQGdXX4lLOOfz1JgBZCExIACSFESb4h0PI2ddsZvUASAAmhCQmAhBDiRiWHwRTFsa8lNYCE0IQEQEIIcaN2o8DNG67FQNx+x75WqkyBF0ILEgAJIcSNPP2g3Uh1+8hax76WeQgsSHqAhHAmCYCEEKI85hXij64Dk9FxryM5QEJoQgIgIYQoT8vbwLsBZCaqhREdoTAPMosKLkoOkBBO5aZ1A4QQwiW5eUCHMbDvQzjwKTRsaf/XSLtS9Fpe4NPQ/ucXQlRIAiAhhKhIl3vVAOjoWvXmKIFN1LXIhBBOIwGQEEJUJOomaDUELvzuuNfQGaDrZMedXwhRLgmAhBCiIno93OekNcGEEE4lSdBCCCGEqHckABJCCCFEvSMBkBBCCCHqHZcIgJYuXUp0dDReXl707duX3bt3V3r8l19+Sbt27fDy8qJz585s2LChzDEnTpzgrrvuIjAwEF9fX3r37k1sbKyjLkEIIYQQtYjmAdCaNWuYM2cOCxYsYP/+/XTt2pVhw4aRlJRU7vE7duxg8uTJPPDAAxw4cIAxY8YwZswYjh49ajkmJiaGAQMG0K5dO7Zu3crhw4d58cUX8fLyctZlCSGEEMKF6RTF0UsdV65v37707t2bt99+GwCTyURUVBRPPPEEc+fOLXP8xIkTycrK4rvvvrPsu+mmm+jWrRvLly8HYNKkSbi7u/PJJ5/Y1Kb09HQCAwNJS0sjICDApnMIIYQQwrms+f7WtAcoPz+fffv2MWTIEMs+vV7PkCFD2LlzZ7nP2blzZ6njAYYNG2Y53mQy8f3339OmTRuGDRtGo0aN6Nu3L19//XWF7cjLyyM9Pb3UTQghhBB1l6YBUEpKCkajkbCwsFL7w8LCSEhIKPc5CQkJlR6flJREZmYmixYtYvjw4fz000+MHTuWe+65h23btpV7zoULFxIYGGi5RUXJmjxCCCFEXaZ5DpC9mUwmAO6++26eeeYZunXrxty5c7nzzjstQ2Q3mjdvHmlpaZbbpUuXnNlkIYQQQjiZppWgQ0JCMBgMJCYmltqfmJhIeHh4uc8JDw+v9PiQkBDc3Nzo0KFDqWPat2/P77+XX87e09MTT09PWy9DCCGEELWMpj1AHh4e9OzZk82bN1v2mUwmNm/eTL9+/cp9Tr9+/UodD7Bp0ybL8R4eHvTu3ZtTp06VOub06dM0a9bMzlcghBBCiNpI87XA5syZw7Rp0+jVqxd9+vRhyZIlZGVlMWPGDACmTp1KZGQkCxcuBOCpp55i0KBBvPbaa4waNYrVq1ezd+9e3nvvPcs5n3/+eSZOnMgtt9zCrbfeysaNG/n222/ZunWrFpcohBBCCBejeQA0ceJEkpOTmT9/PgkJCXTr1o2NGzdaEp1jY2PR64s7qvr378+qVav429/+xl/+8hdat27N119/TadOnSzHjB07luXLl7Nw4UKefPJJ2rZty7p16xgwYIDTr08IIYQQrkfzOkCuSOoACSGEELWPNd/fmvcAuSJzTCj1gIQQQojaw/y9XZ2+HQmAypGRkQEg9YCEEEKIWigjI4PAwMBKj5EhsHKYTCbi4uLw9/dHp9PZ9dzp6elERUVx6dKlOj+8Jtdad9Wn65Vrrbvq0/XWl2tVFIWMjAwiIiJK5Q+XR3qAyqHX62nSpIlDXyMgIKBO/yMsSa617qpP1yvXWnfVp+utD9daVc+PWZ2rBC2EEEIIURUJgIQQQghR70gA5GSenp4sWLCgXiy9Iddad9Wn65Vrrbvq0/XWp2utLkmCFkIIIUS9Iz1AQgghhKh3JAASQgghRL0jAZAQQggh6h0JgIQQQghR70gA5ABLly4lOjoaLy8v+vbty+7duys9/ssvv6Rdu3Z4eXnRuXNnNmzY4KSW2m7hwoX07t0bf39/GjVqxJgxYzh16lSlz/noo4/Q6XSlbl5eXk5qse3+/ve/l2l3u3btKn1ObfxMzaKjo8tcr06n4/HHHy/3+Nr0uf7666+MHj2aiIgIdDodX3/9danHFUVh/vz5NG7cGG9vb4YMGcKZM2eqPK+1v/POUtn1FhQU8MILL9C5c2d8fX2JiIhg6tSpxMXFVXpOW34fnKGqz3b69Oll2j18+PAqz+uKn21V11re769Op+PVV1+t8Jyu+rk6kgRAdrZmzRrmzJnDggUL2L9/P127dmXYsGEkJSWVe/yOHTuYPHkyDzzwAAcOHGDMmDGMGTOGo0ePOrnl1tm2bRuPP/44f/zxB5s2baKgoIChQ4eSlZVV6fMCAgKIj4+33C5evOikFtdMx44dS7X7999/r/DY2vqZmu3Zs6fUtW7atAmACRMmVPic2vK5ZmVl0bVrV5YuXVru46+88gpvvvkmy5cvZ9euXfj6+jJs2DByc3MrPKe1v/POVNn1Zmdns3//fl588UX279/PV199xalTp7jrrruqPK81vw/OUtVnCzB8+PBS7f78888rPaerfrZVXWvJa4yPj2fFihXodDrGjRtX6Xld8XN1KEXYVZ8+fZTHH3/c8rPRaFQiIiKUhQsXlnv8vffeq4waNarUvr59+yoPP/ywQ9tpb0lJSQqgbNu2rcJjPvzwQyUwMNB5jbKTBQsWKF27dq328XXlMzV76qmnlJYtWyomk6ncx2vr5woo69evt/xsMpmU8PBw5dVXX7XsS01NVTw9PZXPP/+8wvNY+zuvlRuvtzy7d+9WAOXixYsVHmPt74MWyrvWadOmKXfffbdV56kNn211Pte7775bue222yo9pjZ8rvYmPUB2lJ+fz759+xgyZIhln16vZ8iQIezcubPc5+zcubPU8QDDhg2r8HhXlZaWBkBwcHClx2VmZtKsWTOioqK4++67OXbsmDOaV2NnzpwhIiKCFi1aMGXKFGJjYys8tq58pqD+m/7000+ZOXNmpQsD19bPtaTz58+TkJBQ6rMLDAykb9++FX52tvzOu7K0tDR0Oh1BQUGVHmfN74Mr2bp1K40aNaJt27Y8+uijXL16tcJj68pnm5iYyPfff88DDzxQ5bG19XO1lQRAdpSSkoLRaCQsLKzU/rCwMBISEsp9TkJCglXHuyKTycTTTz/NzTffTKdOnSo8rm3btqxYsYL//e9/fPrpp5hMJvr378/ly5ed2Frr9e3bl48++oiNGzeybNkyzp8/z8CBA8nIyCj3+LrwmZp9/fXXpKamMn369AqPqa2f643Mn481n50tv/OuKjc3lxdeeIHJkydXulimtb8PrmL48OF8/PHHbN68mZdffplt27YxYsQIjEZjucfXlc925cqV+Pv7c88991R6XG39XGtCVoMXNfb4449z9OjRKseL+/XrR79+/Sw/9+/fn/bt2/Puu+/y0ksvObqZNhsxYoRlu0uXLvTt25dmzZrxxRdfVOuvqtrsv//9LyNGjCAiIqLCY2rr5yqKFRQUcO+996IoCsuWLav02Nr6+zBp0iTLdufOnenSpQstW7Zk69at3H777Rq2zLFWrFjBlClTqpyYUFs/15qQHiA7CgkJwWAwkJiYWGp/YmIi4eHh5T4nPDzcquNdzezZs/nuu+/YsmULTZo0seq57u7udO/enbNnzzqodY4RFBREmzZtKmx3bf9MzS5evMjPP//Mgw8+aNXzauvnav58rPnsbPmddzXm4OfixYts2rSp0t6f8lT1++CqWrRoQUhISIXtrguf7W+//capU6es/h2G2vu5WkMCIDvy8PCgZ8+ebN682bLPZDKxefPmUn8hl9SvX79SxwNs2rSpwuNdhaIozJ49m/Xr1/PLL7/QvHlzq89hNBo5cuQIjRs3dkALHSczM5OYmJgK211bP9MbffjhhzRq1IhRo0ZZ9bza+rk2b96c8PDwUp9deno6u3btqvCzs+V33pWYg58zZ87w888/07BhQ6vPUdXvg6u6fPkyV69erbDdtf2zBbUHt2fPnnTt2tXq59bWz9UqWmdh1zWrV69WPD09lY8++kg5fvy48tBDDylBQUFKQkKCoiiKcv/99ytz5861HL99+3bFzc1NWbx4sXLixAllwYIFiru7u3LkyBGtLqFaHn30USUwMFDZunWrEh8fb7llZ2dbjrnxWv/v//5P+fHHH5WYmBhl3759yqRJkxQvLy/l2LFjWlxCtT377LPK1q1blf9v7+5CmurjOIB/Rd1xYjHMMYa6KZhiksaoqAzEtC6KXq62oMiQ6qKbkixhYoW7mDfrQiPzIhJvCqmLiAXVQLsYCr1MKhm+1KjLRakYsxXu91xE52nZoz3P05zrfD9w4Gznv7//3/mfA1/OC4ZCIfH7/VJfXy95eXkSDodF5M+Z0+/Nz8+LxWKRlpaWBdtSeV5nZ2clEAhIIBAQAHLp0iUJBALqW08dHR1iMBjkzp078vz5c9m/f78UFxfL3Nyc2seOHTukq6tL/bzUOZ9Mi9X7+fNn2bdvnxQUFMjIyEjceRyNRtU+fqx3qfMhWRardXZ2Vpqbm2VoaEhCoZD4fD6x2Wyydu1a+fTpk9pHqsztUsexiMjMzIxkZ2dLd3f3T/tIlXlNJAagBOjq6hKLxSI6nU42b94sw8PD6raamhppaGiIa9/f3y+lpaWi0+mkoqJCvF7vMo/43wPw0+X69etqmx9rPX36tLpfTCaT7N69W549e7b8g/+XHA6HmM1m0el0kp+fLw6HQyYnJ9Xtf8qcfu/+/fsCQMbGxhZsS+V5HRgY+Olx+62eWCwmbW1tYjKZRFEUqaurW7APrFarXLhwIe67xc75ZFqs3lAo9I/n8cDAgNrHj/UudT4ky2K1RiIR2bVrlxiNRsnMzBSr1SrHjx9fEGRSZW6XOo5FRHp6ekSv18v09PRP+0iVeU2kNBGRhF5iIiIiIlph+AwQERERaQ4DEBEREWkOAxARERFpDgMQERERaQ4DEBEREWkOAxARERFpDgMQERERaQ4DEBHRLxgcHERaWhqmp6eTPRQi+g0YgIiIiEhzGICIiIhIcxiAiCglxGIxuN1uFBcXQ6/Xo6qqCrdu3QLw9+0pr9eLyspKZGVlYcuWLXj58mVcH7dv30ZFRQUURUFRURE8Hk/c9mg0ipaWFhQWFkJRFJSUlODatWtxbZ4+fYqNGzciOzsb27Ztw9jYWGILJ6KEYAAiopTgdrvR19eHq1evYnR0FE1NTTh8+DAePXqktjl79iw8Hg8eP34Mo9GIvXv34suXLwC+Bhe73Y6DBw/ixYsXuHjxItra2tDb26v+/siRI7hx4wY6OzsRDAbR09ODnJycuHG0trbC4/HgyZMnyMjIQGNj47LUT0S/F/8ZKhGteNFoFLm5ufD5fNi6dav6/bFjxxCJRHDixAnU1tbi5s2bcDgcAIAPHz6goKAAvb29sNvtOHToEN69e4cHDx6ovz937hy8Xi9GR0cxPj6OsrIyPHz4EPX19QvGMDg4iNraWvh8PtTV1QEA7t27hz179mBubg5ZWVkJ3gtE9DvxChARrXiTk5OIRCLYuXMncnJy1KWvrw+vXr1S230fjnJzc1FWVoZgMAgACAaDqK6ujuu3uroaExMTmJ+fx8jICNLT01FTU7PoWCorK9V1s9kMAAiHw/+7RiJaXhnJHgAR0VI+fvwIAPB6vcjPz4/bpihKXAj6r/R6/S+1y8zMVNfT0tIAfH0+iYhSC68AEdGKt27dOiiKgrdv36KkpCRuKSwsVNsNDw+r61NTUxgfH0d5eTkAoLy8HH6/P65fv9+P0tJSpKenY/369YjFYnHPFBHRn4tXgIhoxVu1ahWam5vR1NSEWCyG7du3Y2ZmBn6/H6tXr4bVagUAtLe3Y82aNTCZTGhtbUVeXh4OHDgAADhz5gw2bdoEl8sFh8OBoaEhXL58GVeuXAEAFBUVoaGhAY2Njejs7ERVVRXevHmDcDgMu92erNKJKEEYgIgoJbhcLhiNRrjdbrx+/RoGgwE2mw1Op1O9BdXR0YFTp05hYmICGzZswN27d6HT6QAANpsN/f39OH/+PFwuF8xmM9rb23H06FH1b3R3d8PpdOLkyZN4//49LBYLnE5nMsologTjW2BElPK+vaE1NTUFg8GQ7OEQUQrgM0BERESkOQxAREREpDm8BUZERESawytAREREpDkMQERERKQ5DEBERESkOQxAREREpDkMQERERKQ5DEBERESkOQxAREREpDkMQERERKQ5DEBERESkOX8Bvaoj4ef8kQ4AAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(model_clf.history.history.keys())\n",
    "plt.plot(model_clf.history.history['accuracy'])\n",
    "plt.plot(model_clf.history.history['val_accuracy'])\n",
    "plt.title('model accuracy')\n",
    "plt.ylabel('accuracy')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'val'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import classification_report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m4/4\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 44ms/step\n"
     ]
    }
   ],
   "source": [
    "pred = model_clf.predict(X_valid, batch_size=10).argmax(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                   precision    recall  f1-score   support\n",
      "\n",
      "        Экономика       0.00      0.00      0.00         2\n",
      "            Спорт       1.00      0.50      0.67         2\n",
      "         Ценности       0.00      0.00      0.00         2\n",
      "             Крым       0.00      0.00      0.00         0\n",
      "   69-я параллель       0.00      0.00      0.00         1\n",
      "           Бизнес       0.08      0.50      0.14         2\n",
      "       Библиотека       0.00      0.00      0.00         0\n",
      "Силовые структуры       0.40      1.00      0.57         2\n",
      "           Россия       0.00      0.00      0.00         2\n",
      "              Дом       0.00      0.00      0.00         2\n",
      "              Мир       0.00      0.00      0.00         1\n",
      "         Культура       1.00      0.33      0.50         3\n",
      "          Легпром       0.00      0.00      0.00         1\n",
      "  Наука и техника       0.25      0.33      0.29         3\n",
      "   Интернет и СМИ       0.00      0.00      0.00         2\n",
      "         Из жизни       0.00      0.00      0.00         3\n",
      "      Бывший СССР       0.00      0.00      0.00         3\n",
      "\n",
      "        micro avg       0.19      0.19      0.19        31\n",
      "        macro avg       0.16      0.16      0.13        31\n",
      "     weighted avg       0.22      0.19      0.17        31\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(y_valid.argmax(1), pred, labels=list(range(len(label2id))),\n",
    "                            target_names=list(label2id), zero_division=0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Дообучение с замораживанием может не работать, давайте попробуем обучать все сразу"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Дообучение без замораживания "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_bert = AutoModel.from_pretrained('distilbert-base-multilingual-cased')\n",
    "# по умолчанию веса обучаемые поэтому нужно просто загрузить модель"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_word_ids = keras.layers.Input(shape=(512,), dtype='int32',\n",
    "                                     name=\"input_ids\")\n",
    "\n",
    "output = bert_wrapper(model_bert)(input_word_ids)\n",
    "# берем только самый первый вектор (CLS токен)\n",
    "drop = keras.layers.Dropout(0.1)(output[:, 0]) \n",
    "dense = keras.layers.Dense(y.shape[1], activation='softmax')(drop)\n",
    "\n",
    "model_clf = keras.Model(inputs=input_word_ids, outputs=dense)\n",
    "\n",
    "\n",
    "model_clf.compile(keras.optimizers.Adam(learning_rate=2e-5,\n",
    "                                    ), \n",
    "                  loss='categorical_crossentropy', \n",
    "                  metrics=['accuracy', \n",
    "                           keras.metrics.RecallAtPrecision(0.80, name='rec_prec')])\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [],
   "source": [
    "if torch.cuda.is_available():\n",
    "    model_clf.to('cuda')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Батч сайз нужно поставить еще меньше"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "\u001b[1m144/144\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m13s\u001b[0m 90ms/step - accuracy: 0.0834 - loss: 2.8614 - rec_prec: 0.0000e+00 - val_accuracy: 0.0645 - val_loss: 2.8692 - val_rec_prec: 0.0323\n",
      "Epoch 2/10\n",
      "\u001b[1m144/144\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m13s\u001b[0m 91ms/step - accuracy: 0.1336 - loss: 2.6937 - rec_prec: 0.0074 - val_accuracy: 0.1613 - val_loss: 2.6319 - val_rec_prec: 0.0645\n",
      "Epoch 3/10\n",
      "\u001b[1m144/144\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m13s\u001b[0m 91ms/step - accuracy: 0.3963 - loss: 1.9457 - rec_prec: 0.1105 - val_accuracy: 0.4194 - val_loss: 1.8113 - val_rec_prec: 0.2258\n",
      "Epoch 4/10\n",
      "\u001b[1m144/144\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m13s\u001b[0m 91ms/step - accuracy: 0.8044 - loss: 0.7606 - rec_prec: 0.7777 - val_accuracy: 0.2903 - val_loss: 2.6512 - val_rec_prec: 0.0645\n",
      "Epoch 5/10\n",
      "\u001b[1m144/144\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m13s\u001b[0m 91ms/step - accuracy: 0.9398 - loss: 0.2551 - rec_prec: 0.9789 - val_accuracy: 0.4839 - val_loss: 2.2459 - val_rec_prec: 0.0323\n",
      "Epoch 6/10\n",
      "\u001b[1m144/144\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m13s\u001b[0m 91ms/step - accuracy: 1.0000 - loss: 0.0349 - rec_prec: 1.0000 - val_accuracy: 0.4516 - val_loss: 2.4018 - val_rec_prec: 0.0323\n",
      "Epoch 7/10\n",
      "\u001b[1m144/144\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m13s\u001b[0m 91ms/step - accuracy: 1.0000 - loss: 0.0082 - rec_prec: 1.0000 - val_accuracy: 0.4194 - val_loss: 2.4716 - val_rec_prec: 0.0645\n",
      "Epoch 8/10\n",
      "\u001b[1m144/144\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m13s\u001b[0m 91ms/step - accuracy: 1.0000 - loss: 0.0054 - rec_prec: 1.0000 - val_accuracy: 0.4194 - val_loss: 2.5473 - val_rec_prec: 0.0000e+00\n",
      "Epoch 9/10\n",
      "\u001b[1m144/144\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m13s\u001b[0m 91ms/step - accuracy: 1.0000 - loss: 0.0036 - rec_prec: 1.0000 - val_accuracy: 0.4194 - val_loss: 2.5890 - val_rec_prec: 0.0000e+00\n",
      "Epoch 10/10\n",
      "\u001b[1m144/144\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m13s\u001b[0m 91ms/step - accuracy: 1.0000 - loss: 0.0030 - rec_prec: 1.0000 - val_accuracy: 0.4194 - val_loss: 2.6227 - val_rec_prec: 0.0000e+00\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.src.callbacks.history.History at 0x7f30c6f15720>"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# если есть ошибка с device meta то попробуйте запустить еще раз\n",
    "model_clf.fit(torch.LongTensor(X_train), torch.LongTensor(y_train), \n",
    "          validation_data=(torch.LongTensor(X_valid), torch.LongTensor(y_valid)),\n",
    "          batch_size=4,\n",
    "          epochs=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['accuracy', 'loss', 'rec_prec', 'val_accuracy', 'val_loss', 'val_rec_prec'])\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAHHCAYAAABDUnkqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/H5lhTAAAACXBIWXMAAA9hAAAPYQGoP6dpAABcjUlEQVR4nO3dd3gUZcPF4d9uOiEVSCEEEnrvvSMooqJgARGlqKivWBBRQQUUFNRXfVFRQBTLpygqYAMLIr1Lkw6hxkASaippu/P9MRCIIEJIMsnm3Ne1l5vZ2d2TBJOTZ2aex2YYhoGIiIiIi7BbHUBERESkIKnciIiIiEtRuRERERGXonIjIiIiLkXlRkRERFyKyo2IiIi4FJUbERERcSkqNyIiIuJSVG5ERETEpajciEiBO3DgADabjY8//viKn7t48WJsNhuLFy8u8FwiUjqo3IiIiIhLUbkRERERl6JyIyIuIy0tzeoIVyw9Pd3qCCIuR+VGxAW98MIL2Gw2du/ezd13301AQAAVKlRg9OjRGIZBbGwst9xyC/7+/oSFhfHGG29c8BqJiYncd999hIaG4u3tTaNGjfjkk08u2O/UqVMMGjSIgIAAAgMDGThwIKdOnbporp07d3L77bcTHByMt7c3zZs35/vvv7+qz3H79u3cddddBAUF0b59+9zHP/vsM5o1a4aPjw/BwcHceeedxMbGXvA6a9as4YYbbiAoKAhfX18aNmzIW2+9ddk5zp4jNGvWLJ599lnCwsLw9fXl5ptvvuD9OnfuTP369Vm/fj0dO3akTJkyPPvsswBkZmYyduxYqlevjpeXF5GRkTz99NNkZmZe8J6fffYZLVu2pEyZMgQFBdGxY0d+/fXXy84s4urcrQ4gIoWnb9++1KlTh1deeYV58+bx0ksvERwczLRp07jmmmt49dVX+fzzzxkxYgQtWrSgY8eOAJw+fZrOnTsTExPDI488QnR0NF9//TWDBg3i1KlTPP744wAYhsEtt9zC8uXLeeihh6hTpw5z585l4MCBF2TZtm0b7dq1IyIigpEjR+Lr68tXX31Fr169mD17Nr17987X53jHHXdQo0YNJkyYgGEYALz88suMHj2aPn36cP/993P06FHeeecdOnbsyMaNGwkMDARgwYIF3HTTTYSHh/P4448TFhbGjh07+PHHH3M/x8v18ssvY7PZeOaZZ0hMTGTSpEl069aNTZs24ePjk7vf8ePH6dGjB3feeSd33303oaGhOJ1Obr75ZpYvX84DDzxAnTp12LJlC//73//YvXs33377be7zX3zxRV544QXatm3LuHHj8PT0ZM2aNfz+++9cd911+foairgcQ0RcztixYw3AeOCBB3K35eTkGJUqVTJsNpvxyiuv5G4/efKk4ePjYwwcODB326RJkwzA+Oyzz3K3ZWVlGW3atDHKli1rJCcnG4ZhGN9++60BGK+99lqe9+nQoYMBGB999FHu9q5duxoNGjQwMjIycrc5nU6jbdu2Ro0aNXK3LVq0yACMRYsWXdbn2K9fvzzbDxw4YLi5uRkvv/xynu1btmwx3N3dc7fn5OQY0dHRRpUqVYyTJ0/m2dfpdF7yvc93Nm9ERETu18UwDOOrr74yAOOtt97K3dapUycDMKZOnZrnNf7v//7PsNvtxrJly/Jsnzp1qgEYK1asMAzDMPbs2WPY7Xajd+/ehsPhyHdmEVenw1IiLuz+++/Pve/m5kbz5s0xDIP77rsvd3tgYCC1atVi3759udvmz59PWFgY/fr1y93m4eHBY489RmpqKkuWLMndz93dnf/85z953ufRRx/Nk+PEiRP8/vvv9OnTh5SUFI4dO8axY8c4fvw43bt3Z8+ePcTFxeXrc3zooYfyfDxnzhycTid9+vTJfZ9jx44RFhZGjRo1WLRoEQAbN25k//79DBs2LHck5yybzXbFOQYMGICfn1/ux7fffjvh4eHMnz8/z35eXl4MHjw4z7avv/6aOnXqULt27TyZr7nmGoDczN9++y1Op5MxY8Zgt+f98Z2fzCKuSoelRFxY5cqV83wcEBCAt7c35cuXv2D78ePHcz8+ePAgNWrUuOAXaJ06dXIfP/vf8PBwypYtm2e/WrVq5fk4JiYGwzAYPXo0o0ePvmjWxMREIiIiruCzM0VHR+f5eM+ePRiGQY0aNS66v4eHBwB79+4FoH79+lf8nhfz9/ez2WxUr16dAwcO5NkeERGBp6fnBZl37NhBhQoVLvraiYmJuZntdjt169YtkMwirkrlRsSFubm5XdY2IPd8lcLgdDoBGDFiBN27d7/oPtWrV8/Xa59/PsvZ97LZbPz0008X/Vz/XsSK2t/zgpm5QYMGvPnmmxd9TmRkZGHHEnEpKjcicoEqVarw559/4nQ684ze7Ny5M/fxs/9duHAhqampeUrDrl278rxe1apVAXPUpFu3boWavVq1ahiGQXR0NDVr1rzkfgBbt24tkEx79uzJ87FhGMTExNCwYcN/fW61atXYvHkzXbt2veThpWrVquF0Otm+fTuNGze+2sgiLkvn3IjIBW644Qbi4+OZNWtW7racnBzeeecdypYtS6dOnXL3y8nJYcqUKbn7ORwO3nnnnTyvFxISQufOnZk2bRpHjhy54P2OHj1aYNlvvfVW3NzcePHFFy8YjTIMI/fwW9OmTYmOjmbSpEkXXLqen1GsTz/9lJSUlNyPv/nmG44cOUKPHj3+9bl9+vQhLi6O6dOnX/DY6dOnc+fv6dWrF3a7nXHjxuWOhl1NZhFXpZEbEbnAAw88wLRp0xg0aBDr168nKiqKb775hhUrVjBp0qTcE2d79uxJu3btGDlyJAcOHKBu3brMmTOHpKSkC17z3XffpX379jRo0IAhQ4ZQtWpVEhISWLVqFX/99RebN28ukOzVqlXjpZdeYtSoURw4cIBevXrh5+fH/v37mTt3Lg888AAjRozAbrczZcoUevbsSePGjRk8eDDh4eHs3LmTbdu28csvv1zR+wYHB9O+fXsGDx5MQkICkyZNonr16gwZMuRfn3vPPffw1Vdf8dBDD7Fo0SLatWuHw+Fg586dfPXVV/zyyy80b96c6tWr89xzzzF+/Hg6dOjArbfeipeXF+vWraNixYpMnDgxv182Eddi1WVaIlJ4zl4mffTo0TzbBw4caPj6+l6wf6dOnYx69erl2ZaQkGAMHjzYKF++vOHp6Wk0aNAgz6XdZx0/fty45557DH9/fyMgIMC45557jI0bN15wKbhhGMbevXuNAQMGGGFhYYaHh4cRERFh3HTTTcY333yTu8+VXgr+98/xrNmzZxvt27c3fH19DV9fX6N27drG0KFDjV27duXZb/ny5ca1115r+Pn5Gb6+vkbDhg2Nd95555Lvfb6zeb/44gtj1KhRRkhIiOHj42PceOONxsGDB/Pse7Gv81lZWVnGq6++atSrV8/w8vIygoKCjGbNmhkvvviikZSUlGffGTNmGE2aNMndr1OnTsaCBQsuO7OIq7MZhsYyRUTya/HixXTp0oWvv/6a22+/3eo4IoLOuREREREXo3NuREQuIisrixMnTlxyn4CAgCJKIyJXQuVGROQiVq5cSZcuXS65z0cffURUVFTRBBKRy6ZzbkRELuLkyZOsX7/+kvvUq1eP8PDwIkokIpdL5UZERERcik4oFhEREZdS6s65cTqdHD58GD8/P62iKyIiUkIYhkFKSgoVK1a8YFHfvyt15ebw4cNahE5ERKSEio2NpVKlSpfcp9SVm7PTxsfGxuLv729xGhEREbkcycnJREZG5v4ev5RSV27OHory9/dXuRERESlhLueUEp1QLCIiIi5F5UZERERcisqNiIiIuJRSd87N5XI4HGRnZ1sdo0Ty8PDAzc3N6hgiIlJKqdz8jWEYxMfHc+rUKaujlGiBgYGEhYVpLiERESlyKjd/c7bYhISEUKZMGf1yvkKGYZCenk5iYiKA1t0REZEip3JzHofDkVtsypUrZ3WcEsvHxweAxMREQkJCdIhKRESKlE4oPs/Zc2zKlCljcZKS7+zXUOctiYhIUVO5uQgdirp6+hqKiIhVVG5ERETEpVhabpYuXUrPnj2pWLEiNpuNb7/99l+fs3jxYpo2bYqXlxfVq1fn448/LvScpU1UVBSTJk2yOoaIiEi+WFpu0tLSaNSoEe++++5l7b9//35uvPFGunTpwqZNmxg2bBj3338/v/zySyEnLf46d+7MsGHDCuS11q1bxwMPPFAgryUiIlLULL1aqkePHvTo0eOy9586dSrR0dG88cYbANSpU4fly5fzv//9j+7duxdWTJdgGAYOhwN393//lleoUKEIEomUTE6nQXxyBk7DsDqKSLHl6W4nxM/bsvcvUZeCr1q1im7duuXZ1r1790uOWGRmZpKZmZn7cXJycmHFs8ygQYNYsmQJS5Ys4a233gLgo48+YvDgwcyfP5/nn3+eLVu28OuvvxIZGcnw4cNZvXo1aWlp1KlTh4kTJ+b5ukZFRTFs2LDcr6vNZmP69OnMmzePX375hYiICN544w1uvvlmKz5dEUvsTkhh7sY4vtsYx+GkDKvjiBRrTSsHMufhdpa9f4kqN/Hx8YSGhubZFhoaSnJyMqdPn86dX+V8EydO5MUXX8z3exqGwelsR76ffzV8PNwu66qjt956i927d1O/fn3GjRsHwLZt2wAYOXIkr7/+OlWrViUoKIjY2FhuuOEGXn75Zby8vPj000/p2bMnu3btonLlyv/4Hi+++CKvvfYa//3vf3nnnXfo378/Bw8eJDg4uGA+WZFiKDE5g+83H2bOhji2Hzn3h5Gb3Ya7XVcEivwTDzdrr1cqUeUmP0aNGsXw4cNzP05OTiYyMvKyn38620HdMdac07N9XHfKeP77tyggIABPT0/KlClDWFgYADt37gRg3LhxXHvttbn7BgcH06hRo9yPx48fz9y5c/n+++955JFH/vE9Bg0aRL9+/QCYMGECb7/9NmvXruX666/P1+cmUlylZebwy7Z45m6MY0XMMZxnjj55uNnoXCuE3k0iuKZ2CN4empxSpLgqUeUmLCyMhISEPNsSEhLw9/e/6KgNgJeXF15eXkURr1hq3rx5no9TU1N54YUXmDdvHkeOHCEnJ4fTp09z6NChS75Ow4YNc+/7+vri7++fu8SCSEmX43CyPOYY326M45dtCXlGa5tWDqR300rc1CCcIF9PC1OKyOUqUeWmTZs2zJ8/P8+2BQsW0KZNm0J7Tx8PN7aPs+ZkZZ8C+MvQ19c3z8cjRoxgwYIFvP7661SvXh0fHx9uv/12srKyLvk6Hh4eeT622Ww4nc6rzidiFcMw2BqXzNyNcXy/+TDHUs+dmxdVrgy9m1SiV5OKVCnne4lXEZHiyNJyk5qaSkxMTO7H+/fvZ9OmTQQHB1O5cmVGjRpFXFwcn376KQAPPfQQkydP5umnn+bee+/l999/56uvvmLevHmFltFms13WoSGreXp64nD8+7lBK1asYNCgQfTu3RswvwcHDhwo5HQixcdfJ9P5btNh5m6MIyYxNXd7sK8nPRuG06tJBI0jAzXLtkgJZulv7T/++IMuXbrkfnz23JiBAwfy8ccfc+TIkTyHS6Kjo5k3bx5PPPEEb731FpUqVeKDDz7QZeCYVzitWbOGAwcOULZs2X8cValRowZz5syhZ8+e2Gw2Ro8erREYcXlJp7P5acsR5myMY+3+E7nbvdztdKsbyq1NIuhYs4LlJ0GKSMGwtNx07twZ4xJzRVxs9uHOnTuzcePGQkxVMo0YMYKBAwdSt25dTp8+zUcffXTR/d58803uvfde2rZtS/ny5XnmmWdc8vJ4kawcJ4t3JTJ3YxwLdyaSlWOWeJsNWkeXo3eTCK5vEIa/t8e/vJKIlDQ241LtwgUlJycTEBBAUlIS/v7+eR7LyMhg//79REdH4+1t3eRDrkBfS7GCYRhsOHSSuRvj+PHPI5xKP7cqfc3QsvRuUolbGlekYuDFL0AQkeLrUr+//674n0wiIvIv9h9LY+7GOL7dGMehE+m520P8vLilcUV6NYmgbri/zqMRKSVUbkSkRDqemsmPfx5h7sY4NsWeyt1extON6+uH0btJBG2rlcdNk+2JlDoqNyJSYmRkO1iwPYFvN8axZPdRcs7MsOdmt9GhRnl6N4ng2rqhJeIKRxEpPPoJICLFmtNpsHr/ceZuiOOnrfGkZubkPtYgIoDeTSLo2agiFfxK72SdIpKXyo2IFEu74s8sVLkpjiPnLVQZEehD7yYR9GpSkeohfhYmFJHiSuVGRIqNhOQMvt90mDkb49hx3kKVft7u3NQwnN5NKtG8ShB2nUcjIpegciMilkrNzOGXrWcWqtx7DOO8hSq71Arh1qYRdK6lhSpF5PKp3IhIkctxOFmWu1BlPBnZ52bJbl4liN5NI7ixQTiBZbRQpYhcOZUbESkShmGwJS6JuRvj+GHzYY6lnlustWp5X3o1iaBX4wgqlytjYUoRcQUqNwKYa1MNGzaMYcOGWR1FXJBhGAybtYnvNh3O3VbO15OejcwJ9hpVCtAEeyJSYFRuRKTQfbLyAN9tOoy73UaPBuH0blKRDjW0UKWIFA6VGxEpVNsOJzFh/k4Anr+xDoPaRVucSERcnf5scgHvv/8+FStWxOl05tl+yy23cO+997J3715uueUWQkNDKVu2LC1atOC3336zKK2UJulZOTz6xUayHE661QllYNsoqyOJSCmgcvNvDAOy0qy5XeaC7XfccQfHjx9n0aJFudtOnDjBzz//TP/+/UlNTeWGG25g4cKFbNy4keuvv56ePXty6NChwvqqiQAw9rtt7DuaRpi/N/+9vaHOqxGRIqHDUv8mOx0mVLTmvZ89DJ6+/7pbUFAQPXr0YObMmXTt2hWAb775hvLly9OlSxfsdjuNGjXK3X/8+PHMnTuX77//nkceeaTQ4kvp9t2mOL5e/xd2G0y6szFBvrqsW0SKhkZuXET//v2ZPXs2mZmZAHz++efceeed2O12UlNTGTFiBHXq1CEwMJCyZcuyY8cOjdxIoTl4PI3n5m4F4JFratC6ajmLE4lIaaKRm3/jUcYcQbHqvS9Tz549MQyDefPm0aJFC5YtW8b//vc/AEaMGMGCBQt4/fXXqV69Oj4+Ptx+++1kZWX9y6uKXLmsHCePfbGR1MwcWkYF89g11a2OJCKljMrNv7HZLuvQkNW8vb259dZb+fzzz4mJiaFWrVo0bdoUgBUrVjBo0CB69+4NQGpqKgcOHLAwrbiyN37dxea/kgjw8WDSnY1x1+XeIlLEVG5cSP/+/bnpppvYtm0bd999d+72GjVqMGfOHHr27InNZmP06NEXXFklUhCW7D7KtKX7AHjt9oZUDPSxOJGIlEb6k8qFXHPNNQQHB7Nr1y7uuuuu3O1vvvkmQUFBtG3blp49e9K9e/fcUR2RgpKYksGTX20C4J7WVeheL8zaQCJSamnkxoXY7XYOH77w/KCoqCh+//33PNuGDh2a52MdppKr4XQaPPnVZo6lZlE7zI/nbqxjdSQRKcU0ciMiV+39ZftYtucY3h52Jt/VBG8PN6sjiUgppnIjIldl46GTvP7LLgBe6FmP6iF+FicSkdJO5UZE8i05I5vHvtxIjtPgxobh9G0RaXUkERGVGxHJH8MweG7uVmJPnKZSkA8Tb22g5RVEpFhQubkI4zLXdJJ/pq+h6/v6j7/4YfNh3Ow23u7XBH9vD6sjiYgAKjd5eHiYP5zT09MtTlLynf0anv2aimuJSUxh7PfbAHjyupo0rRxkcSIRkXN0Kfh53NzcCAwMJDExEYAyZcpomP0KGYZBeno6iYmJBAYG4uamq2ZcTUa2g0dmbuR0toP21cvzUMdqVkcSEclD5eZvwsLMicfOFhzJn8DAwNyvpbiWifN3sDM+hXK+nrzZpxF2u/4AEJHiReXmb2w2G+Hh4YSEhJCdnW11nBLJw8NDIzYu6tdt8Xyy6iAAb/RpRIi/t8WJREQupHLzD9zc3PQLWuQ8R5JO8/TsPwEY0iGazrVCLE4kInJxOqFYRP6Vw2nw+JebOJWeTcNKATzVvbbVkURE/pHKjYj8q3d+38Pa/Sfw9XTj7Tub4OmuHx0iUnzpJ5SIXNKafcd5e+EeAF7u3YCo8r4WJxIRuTSVGxH5RyfTshg2axNOA25rWoleTSKsjiQi8q9UbkTkogzD4OnZf3IkKYOq5X0Zd0s9qyOJiFwWlRsRuaj/W32QBdsT8HSz83a/Jvh66eJKESkZVG5E5ALbDyfz0rwdAIzsUZv6EQEWJxIRuXwqNyKSR3pWDo9+sYGsHCdda4cwuF2U1ZFERK6Iyo2I5PHi99vZezSNUH8v/ntHI62vJiIljsqNiOT6YfNhZv0Ri80G/+vbmGBfT6sjiYhcMZUbEQEg9kQ6z87ZAsAjXarTtlp5ixOJiOSPyo2IkO1w8ugXG0nJzKF5lSAe71rD6kgiIvmmciMivPHrbjbFnsLf251JdzbG3U0/GkSk5NJPMJFSbtmeo0xdsheA125vSKWgMhYnEhG5Oio3IqXY0ZRMnpi1GYD+rSpzff1wixOJiFw9lRuRUsrpNHjy680cS82kVqgfo2+qa3UkEZECoXIjUkp9sHwfS3cfxdvDzjt3NcHbw83qSCIiBULlRqQU2hx7itd+3gXAmJvqUTPUz+JEIiIFR+VGpJRJycjm0S82kuM0uLFBOP1aRlodSUSkQKnciJQihmHw/LdbOXQinYhAHybc2kDLK4iIy1G5ESlFvln/F99tOoyb3cbb/ZoQ4ONhdSQRkQKnciNSSuw9msqY77YBMPzamjSrEmRxIhGRwqFyI1IKZOY4eHTmRk5nO2hbrRwPdapmdSQRkUKjciNSCkycv5PtR5IJ9vXkf30b42bXeTYi4rpUbkRc3G/bE/h45QEA3rijEaH+3tYGEhEpZCo3Ii4sPimDp74xl1e4r300XWqHWJxIRKTwqdyIuCiH0+DxLzdyMj2b+hH+PH19LasjiYgUCZUbERf17qIY1uw/ga+nG+/0a4qXu5ZXEJHSwfJy8+677xIVFYW3tzetWrVi7dq1l9x/0qRJ1KpVCx8fHyIjI3niiSfIyMgoorQiJcO6AyeY9NtuAMb3qk90eV+LE4mIFB1Ly82sWbMYPnw4Y8eOZcOGDTRq1Iju3buTmJh40f1nzpzJyJEjGTt2LDt27ODDDz9k1qxZPPvss0WcXKT4OpWexeNfbMRpwK1NIri1aSWrI4mIFClLy82bb77JkCFDGDx4MHXr1mXq1KmUKVOGGTNmXHT/lStX0q5dO+666y6ioqK47rrr6Nev37+O9oiUFoZh8MzsPzmclEFUuTKM61Xf6kgiIkXOsnKTlZXF+vXr6dat27kwdjvdunVj1apVF31O27ZtWb9+fW6Z2bdvH/Pnz+eGG24okswixd1naw7xy7YEPNxsvNOvKWW93K2OJCJS5Cz7yXfs2DEcDgehoaF5toeGhrJz586LPueuu+7i2LFjtG/fHsMwyMnJ4aGHHrrkYanMzEwyMzNzP05OTi6YT0CkmNkZn8z4H7cD8Mz1tWlQKcDiRCIi1rD8hOIrsXjxYiZMmMB7773Hhg0bmDNnDvPmzWP8+PH/+JyJEycSEBCQe4uMjCzCxCJF43SWg0dmbiQrx0mXWhW4r3201ZFERCxj2chN+fLlcXNzIyEhIc/2hIQEwsLCLvqc0aNHc88993D//fcD0KBBA9LS0njggQd47rnnsNsv7GqjRo1i+PDhuR8nJyer4IjLGffjNmISUwnx8+L1Oxphs2l5BREpvSwbufH09KRZs2YsXLgwd5vT6WThwoW0adPmos9JT0+/oMC4uZlzdxiGcdHneHl54e/vn+cm4krm/XmEL9bGYrPBpL6NKVfWy+pIIiKWsvRsw+HDhzNw4ECaN29Oy5YtmTRpEmlpaQwePBiAAQMGEBERwcSJEwHo2bMnb775Jk2aNKFVq1bExMQwevRoevbsmVtyREqT2BPpjJzzJwAPd65G2+rlLU4kImI9S8tN3759OXr0KGPGjCE+Pp7GjRvz888/555kfOjQoTwjNc8//zw2m43nn3+euLg4KlSoQM+ePXn55Zet+hRELJPtcPLYlxtJycihaeVAhnWraXUkEZFiwWb80/EcF5WcnExAQABJSUk6RCUl2ms/7+S9xXvx83Zn/mMdiAwuY3UkEZFCcyW/v0vU1VIiYloRc4wpS/YC8MqtDVVsRETOo3IjUsIcS81k2KxNGAb0axnJjQ3DrY4kIlKsqNyIlCBOp8GIrzdzNCWTGiFlGXNTPasjiYgUOyo3IiXIjBX7WbzrKF7udt65qwk+nrpKUETk71RuREqILX8l8erP5tIkz99Ul9phOiFeRORiVG5ESoDUzBwe/WID2Q6D7vVCubtVZasjiYgUWyo3IiXAmG+3cuB4OhUDvHn1toZaXkFE5BJUbkSKuTkb/mLOxjjsNnirXxMCy3haHUlEpFhTuREpxlIzc3jxh+0ADOtWkxZRwRYnEhEp/lRuRIqxL9ceIul0NlXL+zK0S3Wr44iIlAgqNyLFVFaOkw+W7QfggY5VcbPrPBsRkcuhciNSTH23KY745AxC/Lzo3TTC6jgiIiWGyo1IMeR0Gkxbug+Ae9tH4+WuyfpERC6Xyo1IMbRwZyIxian4eblzl+a0ERG5Iio3IsXQ1DMrfvdvXQV/bw+L04iIlCwqNyLFzLoDJ1h/8CSebnbubRdldRwRkRJH5UakmJl2ZtTm1qYRhPh7W5xGRKTkUbkRKUZ2J6Tw245EbDbz8m8REblyKjcixci0JeYVUt3rhlG1QlmL04iIlEwqNyLFxOFTp/luUxwAD3WuZnEaEZGSS+VGpJj4cPl+cpwGrasG0zgy0Oo4IiIllsqNSDFwKj2LL9YeAuChThq1ERG5Gio3IsXA/606SHqWgzrh/nSqWcHqOCIiJZrKjYjFMrIdfLzyAAAPdaqKzaYFMkVErobKjYjFvv4jluNpWUQE+nBjg3Cr44iIlHgqNyIWynE4eX+Zefn3kA7RuLvpf0kRkauln6QiFvppazyxJ04TVMaDPi0irY4jIuISVG5ELGIYRu4CmQPbRlHG093iRCIirkHlRsQiy2OOse1wMj4ebgxsE2V1HBERl6FyI2KRs6M2fVtEEuTraXEaERHXoXIjYoEtfyWxIuY4bnYb93eItjqOiIhLUbkRscDZUZubG1WkUlAZi9OIiLgWlRuRInbgWBo/bT0CwIOdqlqcRkTE9ajciBSx95ftw2lAl1oVqB3mb3UcERGXo3IjUoQSUzL4Zv1fADyoBTJFRAqFyo1IEfpk5QGycpw0jgykVXSw1XFERFySyo1IEUnNzOH/Vh0E4KFO1bRApohIIVG5ESkiX6w5RHJGDlUr+HJd3VCr44iIuCyVG5EikJXj5MPl+wF4sGNV7HaN2oiIFBaVG5Ei8O2mOOKTMwj196JXkwir44iIuDSVG5FC5nQaTDszad+97aLxcnezOJGIiGtTuREpZL/tSGDv0TT8vN25q1Vlq+OIiLg8lRuRQmQYRu5SC3e3roKft4fFiUREXJ/KjUghWnfgJBsOncLTzc7gtlFWxxERKRVUbkQK0dlzbW5rFkGIv7fFaURESgeVG5FCsis+hYU7E7HZYEgHLZApIlJUVG5ECsm0peaozfX1wqhaoazFaURESg+VG5FCEHfqNN9vOgyYSy2IiEjRUbkRKQQfLttPjtOgTdVyNIoMtDqOiEiponIjUsBOpWfx5bpDADzUWaM2IiJFTeVGpIB9uuog6VkO6ob707FGeavjiIiUOio3IgXodJaDj1ceAODBTlWx2bRApohIUVO5ESlAX6+P5URaFpWCfLixQbjVcURESiWVG5ECkuNwMn3ZPsCc18bdTf97iYhYQT99RQrI/K3xxJ44TbCvJ32aR1odR0Sk1FK5ESkAhmEwdbE5ad/ANlH4eLpZnEhEpPRSuREpAMv2HGP7kWR8PNwY0KaK1XFEREo1lRuRAjD1zAKZd7aMJMjX0+I0IiKlm8qNyFX6869TrNx7HHe7jfu1QKaIiOVUbkSu0tlRm5sbVSQi0MfiNCIionIjchX2H0vjp63xADzQSaM2IiLFgeXl5t133yUqKgpvb29atWrF2rVrL7n/qVOnGDp0KOHh4Xh5eVGzZk3mz59fRGlF8np/6T4MA7rUqkDtMH+r44iICOBu5ZvPmjWL4cOHM3XqVFq1asWkSZPo3r07u3btIiQk5IL9s7KyuPbaawkJCeGbb74hIiKCgwcPEhgYWPThpdRLTMlg9oa/AHiokxbIFBEpLiwtN2+++SZDhgxh8ODBAEydOpV58+YxY8YMRo4cecH+M2bM4MSJE6xcuRIPDw8AoqKiijKySK6PVxwgK8dJk8qBtIwOtjqOiIicka/DUvv372fPnj0XbN+zZw8HDhy4rNfIyspi/fr1dOvW7VwYu51u3bqxatWqiz7n+++/p02bNgwdOpTQ0FDq16/PhAkTcDgc+fk0RPItJSOb/1t9EDBHbbRApohI8ZGvcjNo0CBWrlx5wfY1a9YwaNCgy3qNY8eO4XA4CA0NzbM9NDSU+Pj4iz5n3759fPPNNzgcDubPn8/o0aN54403eOmll/7xfTIzM0lOTs5zE7laX6w9REpGDtUq+HJtndB/f4KIiBSZfJWbjRs30q5duwu2t27dmk2bNl1tpn/kdDoJCQnh/fffp1mzZvTt25fnnnuOqVOn/uNzJk6cSEBAQO4tMlJr/sjVycxx8OHy/QA82LEadrtGbUREipN8lRubzUZKSsoF25OSki77EFH58uVxc3MjISEhz/aEhATCwsIu+pzw8HBq1qyJm9u5dXvq1KlDfHw8WVlZF33OqFGjSEpKyr3FxsZeVj6Rf/LdxsMkJGcS6u/FLU0qWh1HRET+Jl/lpmPHjkycODFPkXE4HEycOJH27dtf1mt4enrSrFkzFi5cmLvN6XSycOFC2rRpc9HntGvXjpiYGJxOZ+623bt3Ex4ejqfnxae89/Lywt/fP89NJL+cToOpS81J++5rH42XuxbIFBEpbvJ1tdSrr75Kx44dqVWrFh06dABg2bJlJCcn8/vvv1/26wwfPpyBAwfSvHlzWrZsyaRJk0hLS8u9emrAgAFEREQwceJEAP7zn/8wefJkHn/8cR599FH27NnDhAkTeOyxx/LzaYhcsQU7Eth3NA0/b3f6taxsdRwREbmIfJWbunXr8ueffzJ58mQ2b96Mj48PAwYM4JFHHiE4+PIvie3bty9Hjx5lzJgxxMfH07hxY37++efck4wPHTqE3X5ucCkyMpJffvmFJ554goYNGxIREcHjjz/OM888k59PQ+SKGIaRu9TC3a2r4OftYXEiERG5GJthGIbVIYpScnIyAQEBJCUl6RCVXJG1+0/QZ9oqPN3tLH+mCyF+3lZHEhEpNa7k93e+l19YtmwZd999N23btiUuLg6A//u//2P58uX5fUmRYu3sqM1tTSup2IiIFGP5KjezZ8+me/fu+Pj4sGHDBjIzMwHzaqkJEyYUaECR4mBXfAq/70zEZoMHOmqBTBGR4ixf5eall15i6tSpTJ8+PXcZBDCvZtqwYUOBhRMpLqadGbXpUT+M6PK+FqcREZFLyVe52bVrFx07drxge0BAAKdOnbraTCLFStyp03y/+TCgBTJFREqCfJWbsLAwYmJiLti+fPlyqlbVkL24lg+W7SPHadC2WjkaVgq0Oo6IiPyLfJWbIUOG8Pjjj7NmzRpsNhuHDx/m888/Z8SIEfznP/8p6IwiljmZlsWXa81ZrTVqIyJSMuRrnpuRI0fidDrp2rUr6enpdOzYES8vL0aMGMGjjz5a0BlFLPPpqoOcznZQN9yfDjXKWx1HREQuwxWXG4fDwYoVKxg6dChPPfUUMTExpKamUrduXcqWLVsYGUUscTrLwSerDgDwYKeq2GxaIFNEpCS44nLj5ubGddddx44dOwgMDKRu3bqFkUvEcl/9EcuJtCwig324sUG41XFEROQy5eucm/r167Nv376CziJSbOQ4nExfZv4bH9KhKu5u+Z7vUkREili+57kZMWIEP/74I0eOHCE5OTnPTaSkm7flCH+dPE2wryd3NIu0Oo6IiFyBfJ1QfMMNNwBw88035zkPwTAMbDYbDoejYNKJWMBcINMctRnUNgofTzeLE4mIyJXIV7lZtGhRQecQKTaW7jnGjiPJlPF0Y0CbKlbHERGRK5SvctOpU6eCziFSbExdbC61cGeLygSW8bQ4jYiIXKl8lRuAkydP8uGHH7Jjxw4A6taty+DBgwkODi6wcCJFbXPsKVbtO4673cb9HaKtjiMiIvmQrxOKly5dSlRUFG+//TYnT57k5MmTvP3220RHR7N06dKCzihSZKaeWSDz5sYVqRjoY3EaERHJj3yN3AwdOpS+ffsyZcoU3NzMky0dDgcPP/wwQ4cOZcuWLQUaUqQo7Duays/b4gF4sKOWWhARKanyNXITExPDk08+mVtswJzcb/jw4RddUFOkJJi+bB+GAdfUDqFWmJ/VcUREJJ/yVW6aNm2ae67N+Xbs2EGjRo2uOpRIUUtMyWD2+jhAC2SKiJR0+Tos9dhjj/H4448TExND69atAVi9ejXvvvsur7zyCn/++Wfuvg0bNiyYpCKF6KMVB8hyOGlaOZAWUUFWxxERkatgMwzDuNIn2e2XHvCx2WzFdkK/5ORkAgICSEpKwt/f3+o4UgykZGTT9pXfScnI4f17mnFdvTCrI4mIyN9cye/vfI3c7N+/P1/BRIqjmWsOkZKRQ/WQsnSrE2p1HBERuUr5KjdVqlzerK033ngjH3zwAeHhWlFZiqfMHAcfLjfL+gMdq2K32/7lGSIiUtwV6lLHS5cu5fTp04X5FiJX5duNcSSmZBLm702vxhFWxxERkQJQqOVGpDhzOg2mLTUXyLyvfTSe7vrfQUTEFeinuZRav25PYN/RNPy83enXqrLVcUREpICo3EipZBhG7lIL97SuQlmvfC+zJiIixYzKjZRKa/efYFPsKTzd7QxupwUyRURcicqNlEpnR21ub1aJCn5eFqcREZGCVKjl5tlnnyU4OLgw30Lkiu2MT2bRrqPYbfBAh6pWxxERkQKWr3IzceJEZsyYccH2GTNm8Oqrr+Z+PGrUKAIDA/MdTqQwTFtiXiHVo344UeV9LU4jIiIFLV/lZtq0adSuXfuC7fXq1WPq1KlXHUqksPx1Mp3vNx8GtECmiIiryle5iY+Pv+iswxUqVODIkSNXHUqksHywbD8Op0G76uVoUCnA6jgiIlII8lVuIiMjWbFixQXbV6xYQcWKFa86lEhhOJmWxax1sYBGbUREXFm+JvcYMmQIw4YNIzs7m2uuuQaAhQsX8vTTT/Pkk08WaECRgvLJqgOcznZQr6I/7auXtzpO6bPpC9g1D8IbQ+XWULEpeJaxOpWIuKB8lZunnnqK48eP8/DDD5OVlQWAt7c3zzzzDKNGjSrQgCIFIT0rh09WHgDgwU7VsNm0QGaRil0L3w0FwwE7fjC32d0hvBFEtjbLTuXWUDbE2pwi4hJshmEY+X1yamoqO3bswMfHhxo1auDlVfznC0lOTiYgIICkpCT8/f2tjiNF5JOVBxj7/TYig31Y9GRn3N00xVORyUiCqe3h1CGI7gg+wXBoNaTGX7hvUPS5ohPZGsrXBLu+VyJyZb+/r2rO+fj4eE6cOEHHjh3x8vLCMAz9RSzFTo7DyfRl5uXfD3SoqmJTlAwD5j1pFpvAytD3M/AOMLefOgiH1kDsavO/idvh5H7ztvkL8/negRDZCiq3gsptoGIT8PCx9FMSkeIvX+Xm+PHj9OnTh0WLFmGz2dizZw9Vq1blvvvuIygoiDfeeKOgc4rk27wtR/jr5GnK+XpyR/NIq+OULn/Ogi1fg80NbvvQLDYANhsERZm3Rn3NbadPwV/rzFGd2DXw1x+QcQr2/GLeAOweULHxmcJzZnSnbIUi/7REpHjLV7l54okn8PDw4NChQ9SpUyd3e9++fRk+fLjKjRQb5gKZ5qjNoLZReHu4WZyoFDm+1xy1Aeg8CiJbXnp/n0Coca15A3Bkw5E/z4zsnCk8qQlmAfprHayabO4XXO1M0TlTeMrXNMuTiJRa+So3v/76K7/88guVKlXKs71GjRocPHiwQIKJFIQlu4+y40gyZTzduKdNFavjlB6ObJh9P2SlQpV20GH4lb+GmwdUambe2gw1D2WdPHCm6Jw5lHV0B5zYa942fW4+zyf43KGsyNZnDmV5F+inJyLFW77KTVpaGmXKXHgJ54kTJ0rEScVSepxdILNfy8oElvG0OE0psuhlOLzBPAx16/tgL4ARM5sNgqPNW+N+5rbTJyF2HRxaZY7sxK2H0ydg90/mDcDN0yw4uYeyWoGvpgIQcWX5KjcdOnTg008/Zfz48QDYbDacTievvfYaXbp0KdCAIvm1KfYUq/edwN1u47720VbHKT32LYHlk8z7N78DAZUuuftV8QmCmteZN4CcLIj/0xzdOVt40o6a/41dAyvfNvcrV/3cOTuVW5sf61CWiMvIV7n573//yzXXXMMff/xBVlYWTz/9NNu2bePEiRMXnblYxApTF5ujNrc0jqBioK6wKRJpx2Hug4ABTQdC3VuK9v3dPaFSc/PW9hHzUNaJfWaxOXTm3J1ju+B4jHnb+Jn5vDLl8p6kXLExuGsUWqSkuuJyk52dzWOPPcYPP/zAggUL8PPzIzU1lVtvvZWhQ4dedM0pkaK2JyGFX7ab86g82KmqxWlKCcOA7x+FlCPmSb3XT7Q6kTkaU66aeWt8l7kt/YQ5qeDZ83bi1kP6cdg137wBuHmZh7LOXoIe2QrKBFv3eYjIFbnicuPh4cGff/5JUFAQzz33XGFkErlq7/weg2FA93qh1Az1szpO6fDHh+byCm6ecNsH4OlrdaKLKxMMta43bwA5mXBk87krsg6thvRjZvmJXQ0r3jL3K1/z3OhO5TYQXFWHskSKqXzNUPzEE0/g5eXFK6+8UhiZCpVmKHZ9MYmpXPu/Jeb8cY+1p15Frf5d6BJ3wPudIScDuk8wr24qqc4eyjq06lzhObb7wv3KlM97CXp4Y/OwmIgUikKfoTgnJ4cZM2bw22+/0axZM3x98/6F9uabb+bnZUUKxOTf92AYcG3dUBWbopCdAd/cZxab6t2g1X+sTnR1zj+U1eRuc1vacfhr7ZnCs8a8Eiz9GOz80bwBuHubi4GevQQ9sqUOZYlYJF/lZuvWrTRt2hSA3bvz/kWj5RfESvuOpvL95sMAPN61hsVpSokFYyBxG/hWgF5TXHMtKN9yUKuHeQPzUNbhTecmGDy02rwE/dBK83ZWhdp5L0HXoSyRInFVC2eWRDos5dqGz9rEnI1xdKsTwgcDW1gdx/Xt+hm+OLN8Qv/ZUKObtXmsYhjm1Ve5EwyuNj/+O9+QcyM7lVtDWEMdyhK5TEW2cKZIcbL/WBrfbooD4PGuNS1OUwqkxMN3D5v3Wz9ceosNmKMx5WuYt6b3mNvSjp07QTl2DRzeCGmJsOMH8wbg7gMRzc47lNXCnLtHRK6Kyo24jMm/x+A04JraITSopHNtCpXTac5nk34cQhtAtxesTlT8+JaH2jeaNzDPTTq88dwl6LGrzRmWDy43b2dVqHPmiqwzh7KConQoS+QKqdyISzh4/PxRG51rU+hWTYZ9i82Rh9s/1IR3l8PDG6q0MW9gFsTje/Jegn5ir7le1tEdsP4jc7+yoefNptzKPJTl5mHd5yFSAqjciEuY/HsMDqdB51oVaBQZaHUc13Z4IywcZ97v8QpUqGVtnpLKbje/dhVqQbOB5rbUxL8dytpkroS+/TvzBuBR5syhrNbnDmV5a6RS5HwqN1LiHTqezpyNGrUpEpmp5mXfzmyo09NcYkEKTtkQ8+tap6f5cfZps0yevQQ9dg1knIIDy8wbADYIqZv3UFZgZR3KklJN5UZKvHcXmaM2HWtWoEllnYxZqH56xjx04h8BPd/WL9DC5uEDVdqaNzAPZR3bnfcS9JP7zUvxE7eZs0QD+IXnPZQV2gDc9ONeSg/9a5cSLfZEOrM3/AVo1KbQbZ0Nmz4DbHDr+5qgzgp2O4TUNm/NBpnbUhLOrXp+aJW5lETKEdg217wBePhCpWbn1smq1AK8NRWGuC6VGynR3lscQ47ToEON8jSr8rdRm92/QFYq1LtVIwxX6+RB+OEJ837HERDV3to8co5fKNS92bwBZKWbMyifPW8ndg1kJMH+peYNwGaHkHoQ0cQ8h0ekoAVWtnQZFpUbKbH+OpnO13+YozaP/X3UJn4LzOwLGHBwFfR4zTVnzi0KjhyYMwQyk8y/+Ds9Y3UiuRTPMmb5PFtAnU44ujPvJegnD0DCFvMmUhgqtVS5EcmP9xbvJcdp0LZaOVpE/e0QyW8vAGcm31433ZxPpNcUzQabH0v/a/717+lnrvaty5BLFrsdQuuat+b3mttS4s2RncTt4HRYm09cU0CEpW+vciMlUtyp03z9RyxwkXNt9i2BmN/A7g7XjIbfx8PWb8yh+T6fmn/ZyuU5uBKWvmbev+l/5oRyUvL5hUG9XuZNxAUVi3H6d999l6ioKLy9vWnVqhVr1669rOd9+eWX2Gw2evXqVbgBpdiZsjiGbIdB66rBtKpa7twDTif8Nta83/xeaD8M+s0yzyuIWQD/18scxZF/d/okzB4ChhMa9YOGd1idSETkslhebmbNmsXw4cMZO3YsGzZsoFGjRnTv3p3ExMRLPu/AgQOMGDGCDh06FFFSKS6OJJ3mq3Vnr5D62xpS27815wXxLAsdnza31egG93xrTnQWuwY+vskclpd/ZhjwwzBI/guCouGG/1qdSETksllebt58802GDBnC4MGDqVu3LlOnTqVMmTLMmDHjH5/jcDjo378/L774IlWrVi3CtFIcTFm8lyyHk5bRwbSpdt6ojSP73My5bR+DshXOPVa5FQz+yZzKPmErzOgOJ/YXbfCSZONnZlG0u8NtH4KXn9WJREQum6XlJisri/Xr19Ot27nVhO12O926dWPVqlX/+Lxx48YREhLCfffd96/vkZmZSXJycp6blFzxSRl8udY812bY38+1Wf+xOaGZb8jFz9IPrQf3/mKeN3LygFlwErYVduSS59ge+OnMqFeX58z5UUREShBLy82xY8dwOByEhobm2R4aGkp8/MUPGyxfvpwPP/yQ6dOnX9Z7TJw4kYCAgNxbZGTkVecW60xdYo7atIgKyjtqk5kCS14173d+BrzKXvwFgqPNghNSz1yz56Me5uWxYsrJhG/uhex0iO4I7YZZnUhE5IpZfljqSqSkpHDPPfcwffp0ypcvf1nPGTVqFElJSbm32NjYQk4phSUhOYOZaw8B5rk2tvMn5ls5GdKOQnC1f1/vyC8MBs8zp6bPSDJPMo75rfCClyQLx0H8n+ATDL2naW4gESmRLL0UvHz58ri5uZGQkJBne0JCAmFhYRfsv3fvXg4cOEDPnj1ztzmdTgDc3d3ZtWsX1apVy/McLy8vvLy8CiG9FLWpS/aSleOkWZUg2lU/b9QmNRFWvmPe7zrm8uZh8QmCe+bCVwPMq6hm3gm3ToP6txVO+JIgZiGsmmzev2Uy+Fe0No+ISD5Z+meZp6cnzZo1Y+HChbnbnE4nCxcupE2bNhfsX7t2bbZs2cKmTZtybzfffDNdunRh06ZNOuTkwhKTM5i55uyoTY28ozZLXoPsNIhoBnVvufwX9SwDd86E+rebq1x/cx+s+6CAk5cQqUdh7kPm/eb3Qe0brc0jInIVLJ/Eb/jw4QwcOJDmzZvTsmVLJk2aRFpaGoMHDwZgwIABREREMHHiRLy9valfv36e5wcGBgJcsF1cy7Sl+8jMcdKkciAdapx3SPL4Xlj/kXm/24tXvoaUuyfcOh18As1iM+9Jc36XDiNKz3pUhgHfPQxpiVChNnR/2epEIiJXxfJy07dvX44ePcqYMWOIj4+ncePG/Pzzz7knGR86dAi7jvuXakdTMvl8zUHgIqM2v48HZw7UuA6i8znnkd0ON7xunmey9DX4/SVIPwnXvVQ6zjlZMw32/ApuXnD7DPDwsTqRiMhVsRmGYVgdoiglJycTEBBAUlIS/v7+VseRyzBh/g7eX7qPRpGBfPtw23PlJm49TL8GsMFDyyGsAEbvVk+Bn0ea9xvdBTe/A26W/w1QeOK3wvQu4MiCHv+FVg9YnUhE5KKu5Pd3KfizVEqyY6mZ/N8qc9Rm2PmjNoYBC84ss9DozoIpNgCt/2NeJWRzg80z4at7IPt0wbx2cZOVbl727ciCGt2h5RCrE4mIFAiVGynWpi/bx+lsBw0rBdC51nkzDscshAPLzEMpXZ4r2DdtdCfc+Tm4e8Ou+fDZ7ZDhgpM//vocHNtlztrc673Sc46RiLg8lRsptk6kZeWO2uQ518bpOLc4ZsshEFgIV8nV6gF3zwEvfzi4HD65ybyiyFXs+AH+OLPESe+p4Ht580aJiJQEKjdSbE1fto/0LAcNIgK4pnbIuQe2fG2uD+UVAB2eLLwAUe1g0I9Qpjwc2QwfXQ+nDhXe+xWVpDj4/lHzfttHodo11uYRESlgKjdSLJ1My+LTlQcAeOz8UZvsDPNqJoAOT0CZ4MINEt7IXK4hIBKOx8CM6+HorsJ9z8LkdMDcB83L3cMbwTVjrE4kIlLgVG6kWPpg+T7SshzUq+hPtzrnjdqs+wCSYsGvIrR6qGjClK9uFpwKtSE5ziw4ceuL5r0L2opJ5rlKHr5w2wxznh8RERejciPFzqn0LD5ZaZ5rk2fU5vQpWPa6eb/Ls0U7H0tABAz+yZwF+fQJ+ORm2Le46N6/IPz1B/x+ZoK+G14zS5uIiAtSuZFi58Pl+0nNzKFOuD/X1T1vxfgVk8zDKRVqQ6N+RR+sTDAM+A6iO0FWKnx+B2z/vuhz5EdGMsy+DwwH1OsNjftbnUhEpNCo3EixkpSezccrDgDweNfq50Ztkg+bE+wBdB1r3cR6Xn7Q/2uoc7M5P8zXA2HDp9ZkuRLzR8DJA+a5QzdN0mXfIuLSVG6kWPlwxX5SMnOoHebHdXXPWxl+8UTIyYDKbczLtK3k7gV3fAxNB4DhNK88WvGWtZkuZfMs+HMW2Oxw2wfmOloiIi5M5UaKjaTT2Xy0Yj9gnmtjt58ZXUjcCRs/M+/nZ3HMwmB3g55vQ7th5scLxpgzJhe31UxO7DcXAwXo9AxUbm1tHhGRIqByI8XGRyv2k5KRQ61QP66vd96ozcJx5ghJ7ZugcivrAv6dzQbXvmgWLjDPCfrhMfNy6+LAkQ2z74esFIhsba50LiJSCqjcSLGQnJHNjOXmqM2jXaufG7U5tBp2zTMPqXQda2HCS2g/zFxg02Y3z7/5ehDkZFqdyjyUF/eHOdnhbdNdewFQEZHzqNxIsfDxigMkZ+RQI6QsN9QPNzcahnm4B6DJPVChpnUB/03TAXDHJ+DmCTu+h5l9IDPVujz7l8GyN837PSdBYGXrsoiIFDGVG7FcSkY2H+aO2px3rs2u+RC7Btx9oPMoCxNepro3w11fmRPk7VsMn94M6SeKPkf6CZjzAGBAk7uh/q1Fn0FExEIqN2K5T1YeIOl0NtUq+HJjgzOjNo4c+O0F836bh8E/3LJ8V6RaFxj4A/gEm7MYf9TDvIy9qBiGefVWymEoVx2uf7Xo3ltEpJhQuRFLpWbm8MHZUZtrauB2dtRm0+dwbLdZEto9bmHCfKjUDO792Vwi4uhO+LA7HIspmvde/xHs/BHsHuZl315li+Z9RUSKEZUbsdQnKw9wKj2bquV96dmoorkxK908GRag41PgHWBdwPyqUAvu+wWCq0HSIZjR3VxZvDAl7oSfnzXvdx0DFZsU7vuJiBRTKjdimbTMHD5Ytg+AR66pfm7UZs0USDlingTb4j4LE16lwMrmgpthDSH9GHx8ExxYUTjvlZ1hLq+QcxqqdoE2jxTO+4iIlAAqN2KZT1cd5GR6NlHlynDz2VGb9BOwfJJ5/5rR5mzAJVnZCjDoR6jSDjKT4bNbYddPBf8+v70ACVuhTDnoPRXs+l9bREov/QQUS6Rl5jA9d9SmBu5uZ/4pLn3dLAGhDaD+7RYmLEDeAXD3bKh1g7mExJf9YfOXBff6u381R7sAek0Bv7BL7y8i4uJUbsQSn60+yIm0LKqUK0OvxmdGbU4ehHXTzfvXvuBaow8ePtDn/8zVzA0HzH3w3EKgVyMlAb79j3m/5YNQs/vVv6aISAnnQr89pKRIz8rh/aXmqM3QLtXPjdosmmCutB3dCap1tTBhIXFzh1veg9YPmx//PNL8nPO7HpXTCd8+ZJ7PE1IPrh1XcFlFREowlRspcp+vPsTxtCwig33o3STC3Bi/xVy5GqDbC8VjcczCYLdD9wnQ5Xnz4yWvwvynzKJypVa/B3t/B3dvuH0GeHgXbFYRkRJK5UaK1OksB9OW7gXgkS7V8Tg7avPbC4AB9W6FiKaW5SsSNht0egpufAOwmYfi5gyBnKzLf43Dm85Ncth9AoTULoSgIiIlk8qNFKnP1xzkWGoWlYJ8uLVpJXPjviUQ8xvY3aHraGsDFqUW95sT7dndYes38OVd5hw//yYrzbzs25ltrpTe/N7CzyoiUoKo3EiRych2MO28c2083Ozm4Zjfzqz23fxeCK5qYUILNLgd+s0y18+KWQD/1xtOn7r0c356Bo7HgF/4mdXIXfQQnohIPqncSJH5Yu0hjqZkEhHow21nR222fwuHN4JnWej4tKX5LFOjGwz4zrxkPHY1fHwjpMRffN9tc2Hj/wE2uPV9KBNcpFFFREoClRspEhnZDqYuMc+1ebhLNTzd7eY5JgvPXOHT9jFzwrvSqnIrGDQfyoaak/HN6A4n9ufd51Qs/HBmna32T0B0x6LPKSJSAqjcSJGYtS6WhORMKgZ4c0ezSHPjhk/g5H7wDYE2Q60NWByE1TcX3AyKgpMHYMb1kLDNfMzpME86zkiCiGbQ5Vkrk4qIFGsqN1LoMnMcTFlsjtr8p0t1c9QmM8W8DBqg8zNavfqs4KrmelQh9SA1Hj7qAbFrzZmbD60yD9/d9gG4eVidVESk2HK3OoC4vq/WxRKfnEF4gDd9mp8512blZEg7av4ybzrQ2oDFjV8YDJ4HM/tC7Br45GZwZJqP3fhG6TvpWkTkCmnkRgpVZo6D986O2nSuhpe7G6Qmwsp3zB26jtEoxMX4BME9c6F6N3Olb8MJDfpAozutTiYiUuxp5EYK1dd//MWRpAxC/b3o0/zMuTZLXoXsNPPckbq9LM1XrHn6wp1fmJfKnzoEN75udSIRkRJB5UYKTVaO89y5Np2q4e3hBsf3wvqPzR26vag5Wv6NuydcP9HqFCIiJYoOS0mh+Wb9X8SdOk2Inxd3tqxsbvx9PDhzoMZ1EN3B2oAiIuKSVG6kUGTlOHl3UQwAD50dtYlbb05Chw26jrU2oIiIuCyVGykUczaYozYV/Ly4q1VlMAxYcKbQNLrTnNNFRESkEKjcSIHLdjiZfGbU5sGOVc1Rm5iFcGAZuHlBl+csTigiIq5M5UYK3NwNcfx18jTly3rRv1UVc3bds4tjthwCgZHWBhQREZemciMF6u+jNj6ebrDla3O9JK8A6PCkxQlFRMTVqdxIgfp2YxyHTqRTzteT/q0rQ3YG/P6S+WCHJ7SKtYiIFDqVGykwOeeN2jzQsSplPN1h3QeQFAt+FaHVQxYnFBGR0kDlRgrMd5sOc/B4OsG+ntzdugqcPgXLzsyq2+VZ8PCxNJ+IiJQOKjdSIM4ftbm/QzS+Xu6wYhKcPgkVakOjftYGFBGRUkPlRgrED38eZv+xNALLeDCgTRQkH4bVU8wHu44FN630ISIiRUPlRq6aw2nwzu/mqM2QDlUp6+UOiydCTgZUbgO1elicUEREShOVG7lqP/55mH1H0wjw8WBAmyqQuBM2fmY+qMUxRUSkiKncyFVxOA3eXrgHgPvbR+Pn7QELx4HhhNo3QeVWFicUEZHSRuVGrsq8LUfYezQNf293BraLgkOrYdc8sNm1OKaIiFhC5Ubyzek0eOfMqM197avi7+UOC8aYDza5ByrUtDCdiIiUVio3km/ztx5hT2Iqft7uDGoXBbvmQ+wacPeBzqOsjiciIqWUyo3ki/O8c23ubRdNgKcNfnvBfLDNw+Afbl04EREp1VRuJF9+3hbP7oRU/LzcubddNGz6HI7tBp9gaPe41fFERKQUU7mRK3b+qM3gdlEEuGeb89oAdHwKvAMsTCciIqWdyo1csV+3J7AzPoWyXu7c2z4a1kyBlCMQWBla3Gd1PBERKeVUbuSKGMa5UZtBbaMIJBWWTzIfvGY0uHtZF05ERASVG7lCC7YnsP1IMr6ebtzXPhqWvg6ZyRDaAOrfbnU8ERERlRu5fIZh8NaZUZuBbaMIyjoC66abD177Atj1z0lERKynpZrlsi3ckci2w8mU8XTj/g5V4ZdHwZEF0Z2gWler44mIiAAauZHLdP6ozYA2UQSn7II/Z5kPdntBi2OKiEixUSzKzbvvvktUVBTe3t60atWKtWvX/uO+06dPp0OHDgQFBREUFES3bt0uub8UjEW7EtkSl4SPhxtDOkSfmbDPgHq3QkRTq+OJiIjksrzczJo1i+HDhzN27Fg2bNhAo0aN6N69O4mJiRfdf/HixfTr149FixaxatUqIiMjue6664iLiyvi5KWHYRi89dvZUZsqlEtcDTG/gd0duo62OJ2IiEheNsMwDCsDtGrVihYtWjB58mQAnE4nkZGRPProo4wcOfJfn+9wOAgKCmLy5MkMGDDgX/dPTk4mICCApKQk/P39rzp/abBoVyKDP1qHj4cby57uRPmZ18ORTdDyAbjhv1bHExGRUuBKfn9bOnKTlZXF+vXr6datW+42u91Ot27dWLVq1WW9Rnp6OtnZ2QQHB1/08czMTJKTk/Pc5PKdP2pzd+vKlD/4k1lsPMtCx6etDSciInIRlpabY8eO4XA4CA0NzbM9NDSU+Pj4y3qNZ555hooVK+YpSOebOHEiAQEBubfIyMirzl2aLN1zjE2xp/D2sPNAu0hYOM58oO1jULaCteFEREQuwvJzbq7GK6+8wpdffsncuXPx9va+6D6jRo0iKSkp9xYbG1vEKUsuc9RmNwD9W1Whwu4v4eR+8A2BNkMtTiciInJxls5zU758edzc3EhISMizPSEhgbCwsEs+9/XXX+eVV17ht99+o2HDhv+4n5eXF15eWhIgP5bHHGPDoVN4udt5qE0IfPiK+UDnZ8CrrLXhRERE/oGlIzeenp40a9aMhQsX5m5zOp0sXLiQNm3a/OPzXnvtNcaPH8/PP/9M8+bNiyJqqXP+uTZ3tapMhT+nQ/oxCK4KTQdanE5EROSfWT5D8fDhwxk4cCDNmzenZcuWTJo0ibS0NAYPHgzAgAEDiIiIYOLEiQC8+uqrjBkzhpkzZxIVFZV7bk7ZsmUpW1ajCQVl5d7j/HHwJJ7udh5u7gcz3jEf6DoG3DysDSciInIJlpebvn37cvToUcaMGUN8fDyNGzfm559/zj3J+NChQ9jPW7NoypQpZGVlcfvteRdpHDt2LC+88EJRRndZeUZtWlamwoa3IDsNIppB3V7WhhMREfkXls9zU9Q0z82/W7n3GHdNX4Onm50VQ6pQ4dMO4MyBgT9CdAer44mISClUYua5keLH4TSYdGbU5s6WkVRY95pZbGpcp2IjIiIlgsqN5DqWmsnAGWtZu/8Enm52Hq2VDNvmAjboOtbqeCIiIpfF8nNupHhYu/8Ej36xgYTkTHw83HjttgZUWP2g+WCjOyGsvrUBRURELpPKTSnndBq8v2wf//1lFw6nQfWQskzp35QayavhwDJw84Iuz1kdU0RE5LKp3JRip9KzePKrzSzcaa7A3qtxRV7u3QBfDxvMPnMYquUQCNSSFSIiUnKo3JRSm2JPMfTzDcSdOo2nu50XetajX8tIbDYbbP4SEreBVwB0eNLqqCIiIldE5aaUMQyDT1Ye4OX5O8h2GFQpV4Z372pK/YgAc4eju88tjtnhCShz8dXWRUREiiuVm1IkJSObkbO3MG/LEQB61A/j1dsb4u/tAU4nrH0ffhsLORkQWBlaPWRxYhERkSunclNKbD+czNCZG9h/LA13u41nb6jD4HZR5mGopDj47mHYt9jcudo1cMu74OFjaWYREZH8ULlxcYZh8NUfsYz5bhuZOU4qBngzuX9TmlYOMnfY8g3MGw4ZSeDuA9eNhxb3g81mbXAREZF8UrlxYelZOTz/7VbmbIgDoEutCrzZpzFBvp6QfgLmj4Cts82dKzaFW9+H8jUsTCwiInL1VG5cVExiCg9/voHdCanYbfDkdbX4T6dq2O022Ps7fPswpBwBmxt0fAo6jtBq3yIi4hJUblzQd5viGDVnC+lZDir4efFOvya0rloOstLhtxdg7TRzx+BqcOt0qNTM0rwiIiIFSeXGhWRkOxj343ZmrjkEQNtq5XjrziZU8POCuA0w5wE4bi6KSYv74dpx4OlrYWIREZGCp3LjIg4eT+Phzzew7XAyNhs82qU6j3eriZvhgMWvwtIzq3uXDTOvhKrRzerIIiIihULlxgX8vDWep77ZTEpGDsG+nvyvb2M61awAx2Jg7oMQ94e5Y91ecNP/NDGfiIi4NJWbEiwrx8mrP+/kw+X7AWhWJYjJdzUh3N8b1n0Ivz4P2enmMgo3vg4N7tAl3iIi4vJUbkqouFOneWTmBjYeOgXAkA7RPH19bTzSE+HzeyBmgbljdEfoNQUCKlkXVkREpAip3JRAi3Yl8sSsTZxKz8bf253X72jEdfXCYPt38MMwOH0C3Lyg2wvmEgp2u9WRRUREiozKTQmS43Dyv9928+6ivQA0iAjgvf5NiSyTDXMehD+/NHcMa2hOyBdSx8K0IiIi1lC5KSESkzN47MuNrN53AoB7Wlfh+Zvq4BW7Ej75DyTFgs0O7Z+ATiPB3dPixCIiItZQuSkBVu49xmNfbOJYaia+nm5MvK0hN9cNhoVjYNW7gAFBUdD7fajcyuq4IiIillK5KcacToP3Fsfw5oLdOA2oFerHe3c3pVrOPnj/Vji6w9yx6UDoPgG8ylobWEREpBhQuSmmTqRlMWzWJpbuPgrA7c0qMb5nHXzWTYZFE8CZDb4V4ObJUOt6i9OKiIgUHyo3xdD6gyd4ZOZGjiRl4O1hZ9wt9elTNQc+7wmxq82dat8EPd8C3/LWhhURESlmVG6KEcMw+HD5fl75aSc5ToOq5X15r38Tah/5DqaOgqxU8PSDHq9C47s0IZ+IiMhFqNwUE0mns3nq6838uj0BgJsahvPq9WH4/vwg7P7J3KlyW+g9FYKqWJhURESkeFO5KQa2/JXEwzPXE3viNJ5udkbfVIe7A7di++AOSD8Gbp5wzfPQ5hGwu1kdV0REpFhTubGQYRh8tuYQ43/YTpbDSaUgH6beUZP6WybCL5+ZO4XUMyfkC6tvbVgREZESQuXGIqmZOTw7Zwvfbz4MQLc6oUxqc5qy398Apw4CNmj7qDli4+5lbVgREZESROXGArviU/jP5+vZdzQNN7uNZ6+ryr3ZX2Cb+RZgQEBl89yaqHZWRxURESlxVG6K2Dfr/+L5b7eQke0kzN+bD3qUof7q+yFhi7lD4/5w/Svg7W9tUBERkRJK5aaInM5yMPb7rXz1x18AdKwezJRqq/H9cQI4sqBMOXPemjo9LU4qIiJSsqncFIF9R1N5+PMN7IxPwWaDMe39GJQ4HtuSFeYONa+Hnm+DX6i1QUVERFyAyk0h+/HPwzzzzZ+kZTko7+vBzFYHqLl+PGQmg4cvXD/BXBtKE/KJiIgUCJWbQpKZ42DCvB18suogAF2ruPGu38d4r5xn7hDZyjxpOLiqhSlFRERcj8pNIYg9kc4jMzew+a8kAF5vFM9tf72CLSER7O7Q5VloN0wT8omIiBQClZsCtmB7Ak9+tYnkjBzCvHOYXW0+Ebu+NB+sUNuckC+8kbUhRUREXJjKTQHJdjh5/ZddTFu6D4A+YfG8bLyDx9795g6th0LXMeDhbWFKERER16dyU0C+/uMvpi3dhzs5zIj6nQ4Jn2IznOBfCXq9B1U7WR1RRESkVFC5KSB9W0Sya+s6hqe8TkD8dnNjw77Q4zXwCbQ0m4iISGmiclNA3HbN48UjD0NOBvgEwU3/g3q9rY4lIiJS6qjcFJSKTcwFLqu0g1veBf9wqxOJiIiUSio3BSUgAoYsMuet0YR8IiIillG5KUjlqlmdQEREpNSzWx1AREREpCCp3IiIiIhLUbkRERERl6JyIyIiIi5F5UZERERcisqNiIiIuBSVGxEREXEpKjciIiLiUlRuRERExKWo3IiIiIhLUbkRERERl6JyIyIiIi5F5UZERERcSqlbFdwwDACSk5MtTiIiIiKX6+zv7bO/xy+l1JWblJQUACIjIy1OIiIiIlcqJSWFgICAS+5jMy6nArkQp9PJ4cOH8fPzw2azFehrJycnExkZSWxsLP7+/gX62nLl9P0oXvT9KF70/Sh+9D25NMMwSElJoWLFitjtlz6rptSN3NjtdipVqlSo7+Hv769/mMWIvh/Fi74fxYu+H8WPvif/7N9GbM7SCcUiIiLiUlRuRERExKWo3BQgLy8vxo4di5eXl9VRBH0/iht9P4oXfT+KH31PCk6pO6FYREREXJtGbkRERMSlqNyIiIiIS1G5EREREZeiciMiIiIuReWmgLz77rtERUXh7e1Nq1atWLt2rdWRSq2JEyfSokUL/Pz8CAkJoVevXuzatcvqWHLGK6+8gs1mY9iwYVZHKbXi4uK4++67KVeuHD4+PjRo0IA//vjD6lilksPhYPTo0URHR+Pj40O1atUYP378Za2fJP9M5aYAzJo1i+HDhzN27Fg2bNhAo0aN6N69O4mJiVZHK5WWLFnC0KFDWb16NQsWLCA7O5vrrruOtLQ0q6OVeuvWrWPatGk0bNjQ6iil1smTJ2nXrh0eHh789NNPbN++nTfeeIOgoCCro5VKr776KlOmTGHy5Mns2LGDV199lddee4133nnH6mglmi4FLwCtWrWiRYsWTJ48GTDXr4qMjOTRRx9l5MiRFqeTo0ePEhISwpIlS+jYsaPVcUqt1NRUmjZtynvvvcdLL71E48aNmTRpktWxSp2RI0eyYsUKli1bZnUUAW666SZCQ0P58MMPc7fddttt+Pj48Nlnn1mYrGTTyM1VysrKYv369XTr1i13m91up1u3bqxatcrCZHJWUlISAMHBwRYnKd2GDh3KjTfemOf/FSl633//Pc2bN+eOO+4gJCSEJk2aMH36dKtjlVpt27Zl4cKF7N69G4DNmzezfPlyevToYXGykq3ULZxZ0I4dO4bD4SA0NDTP9tDQUHbu3GlRKjnL6XQybNgw2rVrR/369a2OU2p9+eWXbNiwgXXr1lkdpdTbt28fU6ZMYfjw4Tz77LOsW7eOxx57DE9PTwYOHGh1vFJn5MiRJCcnU7t2bdzc3HA4HLz88sv079/f6mglmsqNuLShQ4eydetWli9fbnWUUis2NpbHH3+cBQsW4O3tbXWcUs/pdNK8eXMmTJgAQJMmTdi6dStTp05VubHAV199xeeff87MmTOpV68emzZtYtiwYVSsWFHfj6ugcnOVypcvj5ubGwkJCXm2JyQkEBYWZlEqAXjkkUf48ccfWbp0KZUqVbI6Tqm1fv16EhMTadq0ae42h8PB0qVLmTx5MpmZmbi5uVmYsHQJDw+nbt26ebbVqVOH2bNnW5SodHvqqacYOXIkd955JwANGjTg4MGDTJw4UeXmKuicm6vk6elJs2bNWLhwYe42p9PJwoULadOmjYXJSi/DMHjkkUeYO3cuv//+O9HR0VZHKtW6du3Kli1b2LRpU+6tefPm9O/fn02bNqnYFLF27dpdMDXC7t27qVKlikWJSrf09HTs9ry/it3c3HA6nRYlcg0auSkAw4cPZ+DAgTRv3pyWLVsyadIk0tLSGDx4sNXRSqWhQ4cyc+ZMvvvuO/z8/IiPjwcgICAAHx8fi9OVPn5+fhec7+Tr60u5cuV0HpQFnnjiCdq2bcuECRPo06cPa9eu5f333+f999+3Olqp1LNnT15++WUqV65MvXr12LhxI2+++Sb33nuv1dFKNF0KXkAmT57Mf//7X+Lj42ncuDFvv/02rVq1sjpWqWSz2S66/aOPPmLQoEFFG0YuqnPnzroU3EI//vgjo0aNYs+ePURHRzN8+HCGDBlidaxSKSUlhdGjRzN37lwSExOpWLEi/fr1Y8yYMXh6elodr8RSuRERERGXonNuRERExKWo3IiIiIhLUbkRERERl6JyIyIiIi5F5UZERERcisqNiIiIuBSVGxEREXEpKjciUuotXrwYm83GqVOnrI4iIgVA5UZERERcisqNiIiIuBSVGxGxnNPpZOLEiURHR+Pj40OjRo345ptvgHOHjObNm0fDhg3x9vamdevWbN26Nc9rzJ49m3r16uHl5UVUVBRvvPFGnsczMzN55plniIyMxMvLi+rVq/Phhx/m2Wf9+vU0b96cMmXK0LZt2wtWzxaRkkHlRkQsN3HiRD799FOmTp3Ktm3beOKJJ7j77rtZsmRJ7j5PPfUUb7zxBuvWraNChQr07NmT7OxswCwlffr04c4772TLli288MILjB49mo8//jj3+QMGDOCLL77g7bffZseOHUybNo2yZcvmyfHcc8/xxhtv8Mcff+Du7q6VmUVKKC2cKSKWyszMJDg4mN9++402bdrkbr///vtJT0/ngQceoEuXLnz55Zf07dsXgBMnTlCpUiU+/vhj+vTpQ//+/Tl69Ci//vpr7vOffvpp5s2bx7Zt29i9eze1atViwYIFdOvW7YIMixcvpkuXLvz222907doVgPnz53PjjTdy+vRpvL29C/mrICIFSSM3ImKpmJgY0tPTufbaaylbtmzu7dNPP2Xv3r25+51ffIKDg6lVqxY7duwAYMeOHbRr1y7P67Zr1449e/bgcDjYtGkTbm5udOrU6ZJZGjZsmHs/PDwcgMTExKv+HEWkaLlbHUBESrfU1FQA5s2bR0RERJ7HvLy88hSc/PLx8bms/Tw8PHLv22w2wDwfSERKFo3ciIil6tati5eXF4cOHaJ69ep5bpGRkbn7rV69Ovf+yZMn2b17N3Xq1AGgTp06rFixIs/rrlixgpo1a+Lm5kaDBg1wOp15zuEREdelkRsRsZSfnx8jRozgiSeewOl00r59e5KSklixYgX+/v5UqVIFgHHjxlGuXDlCQ0N57rnnKF++PL169QLgySefpEWLFowfP56+ffuyatUqJk+ezHvvvQdAVFQUAwcO5N577+Xtt9+mUaNGHDx4kMTERPr06WPVpy4ihUTlRkQsN378eCpUqMDEiRPZt28fgYGBNG3alGeffTb3sNArr7zC448/zp49e2jcuDE//PADnp6eADRt2pSvvvqKMWPGMH78eMLDwxk3bhyDBg3KfY8pU6bw7LPP8vDDD3P8+HEqV67Ms88+a8WnKyKFTFdLiUixdvZKppMnTxIYGGh1HBEpAXTOjYiIiLgUlRsRERFxKTosJSIiIi5FIzciIiLiUlRuRERExKWo3IiIiIhLUbkRERERl6JyIyIiIi5F5UZERERcisqNiIiIuBSVGxEREXEpKjciIiLiUv4fF4UvmvsR7w0AAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(model_clf.history.history.keys())\n",
    "plt.plot(model_clf.history.history['accuracy'])\n",
    "plt.plot(model_clf.history.history['val_accuracy'])\n",
    "plt.title('model rec_prec')\n",
    "plt.ylabel('rec_prec')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'val'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import classification_report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m7/7\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 23ms/step\n"
     ]
    }
   ],
   "source": [
    "pred = model_clf.predict(X_valid, batch_size=5).argmax(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                   precision    recall  f1-score   support\n",
      "\n",
      "        Экономика       0.00      0.00      0.00         2\n",
      "            Спорт       0.40      1.00      0.57         2\n",
      "         Ценности       1.00      0.50      0.67         2\n",
      "             Крым       0.00      0.00      0.00         0\n",
      "   69-я параллель       0.00      0.00      0.00         1\n",
      "           Бизнес       0.50      0.50      0.50         2\n",
      "       Библиотека       0.00      0.00      0.00         0\n",
      "Силовые структуры       0.33      0.50      0.40         2\n",
      "           Россия       0.00      0.00      0.00         2\n",
      "              Дом       0.00      0.00      0.00         2\n",
      "              Мир       0.50      1.00      0.67         1\n",
      "         Культура       0.67      0.67      0.67         3\n",
      "          Легпром       0.00      0.00      0.00         1\n",
      "  Наука и техника       1.00      0.33      0.50         3\n",
      "   Интернет и СМИ       0.50      1.00      0.67         2\n",
      "         Из жизни       0.50      0.33      0.40         3\n",
      "      Бывший СССР       0.33      0.33      0.33         3\n",
      "\n",
      "        micro avg       0.42      0.42      0.42        31\n",
      "        macro avg       0.34      0.36      0.32        31\n",
      "     weighted avg       0.43      0.42      0.39        31\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(y_valid.argmax(1), pred, labels=list(range(len(label2id))),\n",
    "                            target_names=list(label2id), zero_division=0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ф-мера выглядит не очень, но нужно помнить, что мы обучились на 600 примерах. Для сравнения можно попробовать обучить лстм на тех же данных (честнее было конечно обучить трансформер, но это не так просто)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "def preprocess(text):\n",
    "    tokens = text.lower().split()\n",
    "    tokens = [token.strip(punctuation) for token in tokens]\n",
    "    return tokens\n",
    "\n",
    "vocab = Counter()\n",
    "\n",
    "for text in data.title:\n",
    "    vocab.update(preprocess(text))\n",
    "\n",
    "# индексируем слова\n",
    "word2id = {'PAD':0}\n",
    "\n",
    "for word in vocab:\n",
    "    word2id[word] = len(word2id)\n",
    "\n",
    "id2word = {i:word for word, i in word2id.items()}\n",
    "\n",
    "X = []\n",
    "\n",
    "for text in data.title:\n",
    "    tokens = preprocess(text)\n",
    "    ids = [word2id[token] for token in tokens]\n",
    "    X.append(ids)\n",
    "\n",
    "MAX_LEN = max(len(x) for x in X)\n",
    "\n",
    "\n",
    "# паддинг\n",
    "X = keras.preprocessing.sequence.pad_sequences(X, padding='post', maxlen=MAX_LEN)\n",
    "\n",
    "\n",
    "id2label = {i:label for i, label in enumerate(set(data.topic.values))}\n",
    "label2id = {l:i for i, l in id2label.items()}\n",
    "\n",
    "y = keras.utils.to_categorical([label2id[label] for label in data.topic.values])\n",
    "\n",
    "\n",
    "X_train, X_valid, y_train, y_valid = train_test_split(X, y, test_size=0.05, stratify=data.topic)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [],
   "source": [
    "# if no gpu memory left \n",
    "# del model_bert\n",
    "# torch.cuda.empty_cache()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs = keras.layers.Input(shape=(MAX_LEN,))\n",
    "embeddings = keras.layers.Embedding(input_dim=len(word2id), output_dim=30)(inputs, )\n",
    "\n",
    "lstm = keras.layers.Bidirectional(keras.layers.LSTM(128, return_sequences=False))(embeddings)\n",
    "drop = keras.layers.Dropout(0.3)(lstm)\n",
    "outputs = keras.layers.Dense(len(label2id), activation='softmax')(drop)\n",
    "\n",
    "model = keras.Model(inputs=inputs, outputs=outputs)\n",
    "optimizer = keras.optimizers.Adam(learning_rate=0.001)\n",
    "model.compile(optimizer=optimizer,\n",
    "              loss='categorical_crossentropy',\n",
    "              metrics=['accuracy'])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Model: \"functional_11\"</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1mModel: \"functional_11\"\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\"> Layer (type)                    </span>┃<span style=\"font-weight: bold\"> Output Shape           </span>┃<span style=\"font-weight: bold\">       Param # </span>┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n",
       "│ input_layer_2 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">InputLayer</span>)      │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">13</span>)             │             <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ embedding_2 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Embedding</span>)         │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">13</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">30</span>)         │        <span style=\"color: #00af00; text-decoration-color: #00af00\">93,060</span> │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ bidirectional_2 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Bidirectional</span>) │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">256</span>)            │       <span style=\"color: #00af00; text-decoration-color: #00af00\">162,816</span> │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ dropout_8 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dropout</span>)             │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">256</span>)            │             <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ dense_7 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dense</span>)                 │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">17</span>)             │         <span style=\"color: #00af00; text-decoration-color: #00af00\">4,369</span> │\n",
       "└─────────────────────────────────┴────────────────────────┴───────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1mLayer (type)                   \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mOutput Shape          \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m      Param #\u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n",
       "│ input_layer_2 (\u001b[38;5;33mInputLayer\u001b[0m)      │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m13\u001b[0m)             │             \u001b[38;5;34m0\u001b[0m │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ embedding_2 (\u001b[38;5;33mEmbedding\u001b[0m)         │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m13\u001b[0m, \u001b[38;5;34m30\u001b[0m)         │        \u001b[38;5;34m93,060\u001b[0m │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ bidirectional_2 (\u001b[38;5;33mBidirectional\u001b[0m) │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m256\u001b[0m)            │       \u001b[38;5;34m162,816\u001b[0m │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ dropout_8 (\u001b[38;5;33mDropout\u001b[0m)             │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m256\u001b[0m)            │             \u001b[38;5;34m0\u001b[0m │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ dense_7 (\u001b[38;5;33mDense\u001b[0m)                 │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m17\u001b[0m)             │         \u001b[38;5;34m4,369\u001b[0m │\n",
       "└─────────────────────────────────┴────────────────────────┴───────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Total params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">260,245</span> (1016.58 KB)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Total params: \u001b[0m\u001b[38;5;34m260,245\u001b[0m (1016.58 KB)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">260,245</span> (1016.58 KB)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m260,245\u001b[0m (1016.58 KB)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Non-trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> (0.00 B)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m0\u001b[0m (0.00 B)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 0.0636 - loss: 2.8246 - val_accuracy: 0.0968 - val_loss: 2.7631\n",
      "Epoch 2/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 0.0948 - loss: 2.7451 - val_accuracy: 0.0968 - val_loss: 2.6981\n",
      "Epoch 3/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 0.0960 - loss: 2.7234 - val_accuracy: 0.1613 - val_loss: 2.6713\n",
      "Epoch 4/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 0.1234 - loss: 2.6199 - val_accuracy: 0.1613 - val_loss: 2.6168\n",
      "Epoch 5/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 0.1432 - loss: 2.3254 - val_accuracy: 0.1290 - val_loss: 2.7214\n",
      "Epoch 6/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 0.2216 - loss: 2.0751 - val_accuracy: 0.0968 - val_loss: 2.7253\n",
      "Epoch 7/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 0.3673 - loss: 1.8050 - val_accuracy: 0.0968 - val_loss: 3.2464\n",
      "Epoch 8/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 0.4566 - loss: 1.4845 - val_accuracy: 0.1613 - val_loss: 3.5870\n",
      "Epoch 9/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 0.6686 - loss: 1.0658 - val_accuracy: 0.1290 - val_loss: 4.2642\n",
      "Epoch 10/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 0.7104 - loss: 0.7986 - val_accuracy: 0.0645 - val_loss: 4.1696\n",
      "Epoch 11/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 0.7653 - loss: 0.6648 - val_accuracy: 0.0968 - val_loss: 5.2872\n",
      "Epoch 12/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 0.8781 - loss: 0.3899 - val_accuracy: 0.1290 - val_loss: 5.4108\n",
      "Epoch 13/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 0.9436 - loss: 0.2654 - val_accuracy: 0.0645 - val_loss: 5.3915\n",
      "Epoch 14/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 0.9671 - loss: 0.1601 - val_accuracy: 0.1290 - val_loss: 5.9223\n",
      "Epoch 15/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 0.9715 - loss: 0.1268 - val_accuracy: 0.0968 - val_loss: 5.7120\n",
      "Epoch 16/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 0.9869 - loss: 0.0888 - val_accuracy: 0.1290 - val_loss: 6.6750\n",
      "Epoch 17/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 36ms/step - accuracy: 0.9810 - loss: 0.0737 - val_accuracy: 0.0968 - val_loss: 6.1753\n",
      "Epoch 18/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 36ms/step - accuracy: 0.9958 - loss: 0.0453 - val_accuracy: 0.0968 - val_loss: 6.6781\n",
      "Epoch 19/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 36ms/step - accuracy: 0.9929 - loss: 0.0540 - val_accuracy: 0.0968 - val_loss: 6.3819\n",
      "Epoch 20/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 0.9942 - loss: 0.0256 - val_accuracy: 0.1290 - val_loss: 6.7183\n",
      "Epoch 21/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 0.9993 - loss: 0.0252 - val_accuracy: 0.0968 - val_loss: 6.5521\n",
      "Epoch 22/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 1.0000 - loss: 0.0233 - val_accuracy: 0.1613 - val_loss: 6.6518\n",
      "Epoch 23/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 1.0000 - loss: 0.0153 - val_accuracy: 0.0645 - val_loss: 6.5761\n",
      "Epoch 24/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 1.0000 - loss: 0.0093 - val_accuracy: 0.0645 - val_loss: 6.6772\n",
      "Epoch 25/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 1.0000 - loss: 0.0072 - val_accuracy: 0.1290 - val_loss: 6.7944\n",
      "Epoch 26/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 1.0000 - loss: 0.0053 - val_accuracy: 0.0645 - val_loss: 6.8541\n",
      "Epoch 27/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 1.0000 - loss: 0.0048 - val_accuracy: 0.0645 - val_loss: 6.8202\n",
      "Epoch 28/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 1.0000 - loss: 0.0049 - val_accuracy: 0.0645 - val_loss: 6.8463\n",
      "Epoch 29/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 1.0000 - loss: 0.0043 - val_accuracy: 0.0645 - val_loss: 6.9164\n",
      "Epoch 30/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 1.0000 - loss: 0.0042 - val_accuracy: 0.0645 - val_loss: 6.9564\n",
      "Epoch 31/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 1.0000 - loss: 0.0032 - val_accuracy: 0.0645 - val_loss: 6.8720\n",
      "Epoch 32/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 1.0000 - loss: 0.0037 - val_accuracy: 0.0645 - val_loss: 7.0211\n",
      "Epoch 33/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 1.0000 - loss: 0.0032 - val_accuracy: 0.0645 - val_loss: 6.9831\n",
      "Epoch 34/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 1.0000 - loss: 0.0030 - val_accuracy: 0.0645 - val_loss: 7.1463\n",
      "Epoch 35/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 1.0000 - loss: 0.0044 - val_accuracy: 0.0645 - val_loss: 7.3191\n",
      "Epoch 36/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 1.0000 - loss: 0.0028 - val_accuracy: 0.0645 - val_loss: 6.8584\n",
      "Epoch 37/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 1.0000 - loss: 0.0025 - val_accuracy: 0.0645 - val_loss: 7.0234\n",
      "Epoch 38/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 1.0000 - loss: 0.0019 - val_accuracy: 0.0645 - val_loss: 7.0146\n",
      "Epoch 39/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 1.0000 - loss: 0.0021 - val_accuracy: 0.0968 - val_loss: 7.1876\n",
      "Epoch 40/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 1.0000 - loss: 0.0023 - val_accuracy: 0.0968 - val_loss: 7.4374\n",
      "Epoch 41/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 1.0000 - loss: 0.0026 - val_accuracy: 0.0645 - val_loss: 7.3234\n",
      "Epoch 42/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 1.0000 - loss: 0.0021 - val_accuracy: 0.0645 - val_loss: 7.1596\n",
      "Epoch 43/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 1.0000 - loss: 0.0020 - val_accuracy: 0.0645 - val_loss: 7.0531\n",
      "Epoch 44/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 1.0000 - loss: 0.0018 - val_accuracy: 0.0645 - val_loss: 7.2542\n",
      "Epoch 45/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 36ms/step - accuracy: 1.0000 - loss: 0.0014 - val_accuracy: 0.0645 - val_loss: 7.2964\n",
      "Epoch 46/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 1.0000 - loss: 0.0015 - val_accuracy: 0.0645 - val_loss: 7.2291\n",
      "Epoch 47/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 1.0000 - loss: 0.0014 - val_accuracy: 0.0645 - val_loss: 7.3167\n",
      "Epoch 48/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 1.0000 - loss: 0.0013 - val_accuracy: 0.0645 - val_loss: 7.2996\n",
      "Epoch 49/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 1.0000 - loss: 0.0012 - val_accuracy: 0.0645 - val_loss: 7.2824\n",
      "Epoch 50/50\n",
      "\u001b[1m18/18\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 37ms/step - accuracy: 1.0000 - loss: 0.0011 - val_accuracy: 0.0645 - val_loss: 7.3407\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.src.callbacks.history.History at 0x7f30c6dc3a00>"
      ]
     },
     "execution_count": 123,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train, y_train, \n",
    "          validation_data=(X_valid, y_valid),\n",
    "          batch_size=32,\n",
    "          epochs=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['accuracy', 'loss', 'val_accuracy', 'val_loss'])\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAHHCAYAAABDUnkqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/H5lhTAAAACXBIWXMAAA9hAAAPYQGoP6dpAABWSUlEQVR4nO3dd3xT5f4H8E+StuledFIKLZQNLVKGZY8qV7SCiiCCDBV+KiqCXAW8gIJaHCCiXPAqyPU6ABFwgCAyBZml7NUyC3TQ0r2bPL8/DkkJLR1pmpOkn/fr1ZfpyUnOt8dqPn3O93mOQgghQERERGQjlHIXQERERGRKDDdERERkUxhuiIiIyKYw3BAREZFNYbghIiIim8JwQ0RERDaF4YaIiIhsCsMNERER2RSGGyIiIrIpDDdEZHKXL1+GQqHAypUra/3anTt3QqFQYOfOnSavi4gaBoYbIiIisikMN0RERGRTGG6IyGbk5+fLXUKtFRQUyF0Ckc1huCGyQW+//TYUCgXOnz+P0aNHw8PDA76+vpg1axaEEEhKSsKQIUPg7u6OgIAALFiwoMJ7pKWl4bnnnoO/vz8cHR0RERGB//73vxX2y8rKwrhx4+Dh4QFPT0+MHTsWWVlZldZ19uxZDBs2DN7e3nB0dESXLl3wyy+/1OlnPH36NJ5++ml4eXmhV69e+ue//fZbREZGwsnJCd7e3njqqaeQlJRU4X0OHDiAwYMHw8vLCy4uLggPD8enn35a4zp0PUKrV6/GzJkzERAQABcXFzz66KMVjtevXz906NABcXFx6NOnD5ydnTFz5kwAQHFxMebMmYOwsDCo1WoEBwfjjTfeQHFxcYVjfvvtt+jWrRucnZ3h5eWFPn364I8//qhxzUS2zk7uAoio/owYMQJt27bF/PnzsXHjRrz77rvw9vbGF198gQEDBuCDDz7Ad999h2nTpqFr167o06cPAKCwsBD9+vVDYmIiXn75ZYSGhuLHH3/EuHHjkJWVhcmTJwMAhBAYMmQI9uzZgxdeeAFt27bF+vXrMXbs2Aq1nDp1Cj179kRQUBCmT58OFxcXrFmzBkOHDsVPP/2Exx57zKif8cknn0TLli3x/vvvQwgBAHjvvfcwa9YsDB8+HM8//zxu3ryJzz77DH369EF8fDw8PT0BAFu3bsUjjzyCwMBATJ48GQEBAThz5gx+++03/c9YU++99x4UCgXefPNNpKWlYdGiRYiOjsbRo0fh5OSk3y8jIwMPPfQQnnrqKYwePRr+/v7QarV49NFHsWfPHkycOBFt27bFiRMn8Mknn+D8+fPYsGGD/vXvvPMO3n77bfTo0QNz586Fg4MDDhw4gO3bt+PBBx806hwS2RxBRDZnzpw5AoCYOHGifltZWZlo0qSJUCgUYv78+frtmZmZwsnJSYwdO1a/bdGiRQKA+Pbbb/XbSkpKRFRUlHB1dRU5OTlCCCE2bNggAIgPP/zQ4Di9e/cWAMTXX3+t3z5w4EDRsWNHUVRUpN+m1WpFjx49RMuWLfXbduzYIQCIHTt21OhnHDlypMH2y5cvC5VKJd577z2D7SdOnBB2dnb67WVlZSI0NFQ0a9ZMZGZmGuyr1WqrPPaddPUGBQXpz4sQQqxZs0YAEJ9++ql+W9++fQUAsWzZMoP3+N///ieUSqX466+/DLYvW7ZMABB79+4VQgiRkJAglEqleOyxx4RGozG6ZiJbx8tSRDbs+eef1z9WqVTo0qULhBB47rnn9Ns9PT3RunVrXLx4Ub9t06ZNCAgIwMiRI/Xb7O3t8eqrryIvLw+7du3S72dnZ4cXX3zR4DivvPKKQR23bt3C9u3bMXz4cOTm5iI9PR3p6enIyMjAoEGDkJCQgOvXrxv1M77wwgsG369btw5arRbDhw/XHyc9PR0BAQFo2bIlduzYAQCIj4/HpUuX8Nprr+lHcnQUCkWt6xgzZgzc3Nz03w8bNgyBgYHYtGmTwX5qtRrjx4832Pbjjz+ibdu2aNOmjUHNAwYMAAB9zRs2bIBWq8Xs2bOhVBr+79uYmolsFS9LEdmwpk2bGnzv4eEBR0dH+Pj4VNiekZGh//7KlSto2bJlhQ/Qtm3b6p/X/TMwMBCurq4G+7Vu3drg+8TERAghMGvWLMyaNavSWtPS0hAUFFSLn04SGhpq8H1CQgKEEGjZsmWl+9vb2wMALly4AADo0KFDrY9ZmbuPp1AoEBYWhsuXLxtsDwoKgoODQ4Waz5w5A19f30rfOy0tTV+zUqlEu3btTFIzka1iuCGyYSqVqkbbAOj7VeqDVqsFAEybNg2DBg2qdJ+wsDCj3vvOfhbdsRQKBX7//fdKf9a7g5i53V0vINXcsWNHLFy4sNLXBAcH13dZRDaF4YaIKmjWrBmOHz8OrVZrMHpz9uxZ/fO6f27btg15eXkGoeHcuXMG79e8eXMA0qhJdHR0vdbeokULCCEQGhqKVq1aVbkfAJw8edIkNSUkJBh8L4RAYmIiwsPDq31tixYtcOzYMQwcOLDKy0stWrSAVqvF6dOn0alTp7qWTGSz2HNDRBUMHjwYKSkpWL16tX5bWVkZPvvsM7i6uqJv3776/crKyrB06VL9fhqNBp999pnB+/n5+aFfv3744osvkJycXOF4N2/eNFntjz/+OFQqFd55550Ko1FCCP3lt86dOyM0NBSLFi2qMHXdmFGsb775Brm5ufrv165di+TkZDz00EPVvnb48OG4fv06vvzyywrPFRYW6tfvGTp0KJRKJebOnasfDatLzUS2iiM3RFTBxIkT8cUXX2DcuHGIi4tDSEgI1q5di71792LRokX6xtmYmBj07NkT06dPx+XLl9GuXTusW7cO2dnZFd5zyZIl6NWrFzp27IgJEyagefPmSE1Nxb59+3Dt2jUcO3bMJLW3aNEC7777LmbMmIHLly9j6NChcHNzw6VLl7B+/XpMnDgR06ZNg1KpxNKlSxETE4NOnTph/PjxCAwMxNmzZ3Hq1Cls2bKlVsf19vZGr169MH78eKSmpmLRokUICwvDhAkTqn3tM888gzVr1uCFF17Ajh070LNnT2g0Gpw9exZr1qzBli1b0KVLF4SFheGtt97CvHnz0Lt3bzz++ONQq9U4dOgQGjdujNjYWGNPG5FtkWuaFhHVH9006Zs3bxpsHzt2rHBxcamwf9++fUX79u0NtqWmporx48cLHx8f4eDgIDp27GgwtVsnIyNDPPPMM8Ld3V14eHiIZ555RsTHx1eYCi6EEBcuXBBjxowRAQEBwt7eXgQFBYlHHnlErF27Vr9PbaeC3/0z6vz000+iV69ewsXFRbi4uIg2bdqISZMmiXPnzhnst2fPHvHAAw8INzc34eLiIsLDw8Vnn31W5bHvpKv3hx9+EDNmzBB+fn7CyclJPPzww+LKlSsG+1Z2nnVKSkrEBx98INq3by/UarXw8vISkZGR4p133hHZ2dkG+65YsULcd999+v369u0rtm7dWuOaiWydQgiOZRIRGWvnzp3o378/fvzxRwwbNkzucogI7LkhIiIiG8OeGyKiSpSUlODWrVtV7uPh4WGmaoioNhhuiIgq8ffff6N///5V7vP1118jJCTEPAURUY2x54aIqBKZmZmIi4urcp/27dsjMDDQTBURUU0x3BAREZFNYUMxERER2ZQG13Oj1Wpx48YNuLm58S66REREVkIIgdzcXDRu3LjCTX3v1uDCzY0bN3gTOiIiIiuVlJSEJk2aVLlPgws3umXjk5KS4O7uLnM1REREVBM5OTkIDg7Wf45XpcGFG92lKHd3d4YbIiIiK1OTlhI2FBMREZFNYbghIiIim8JwQ0RERDalwfXc1JRGo0FpaancZVgle3t7qFQqucsgIqIGiuHmLkIIpKSkICsrS+5SrJqnpycCAgK4lhAREZkdw81ddMHGz88Pzs7O/HCuJSEECgoKkJaWBgC87w4REZkdw80dNBqNPtg0atRI7nKslpOTEwAgLS0Nfn5+vERFRERmxYbiO+h6bJydnWWuxPrpziH7loiIyNwYbirBS1F1x3NIRERyYbghIiIimyJruNm9ezdiYmLQuHFjKBQKbNiwodrX7Ny5E507d4ZarUZYWBhWrlxZ73U2NCEhIVi0aJHcZRARERlF1nCTn5+PiIgILFmypEb7X7p0CQ8//DD69++Po0eP4rXXXsPzzz+PLVu21HOllq9fv3547bXXTPJehw4dwsSJE03yXkREROYm62yphx56CA899FCN91+2bBlCQ0OxYMECAEDbtm2xZ88efPLJJxg0aFB9lWkThBDQaDSws6v+X7mvr68ZKiJbU6rRIruwFEWlGrlLISKZOdgp4efmKNvxrWoq+L59+xAdHW2wbdCgQVWOWBQXF6O4uFj/fU5OTn2VJ5tx48Zh165d2LVrFz799FMAwNdff43x48dj06ZN+Ne//oUTJ07gjz/+QHBwMKZOnYr9+/cjPz8fbdu2RWxsrMF5DQkJwWuvvaY/rwqFAl9++SU2btyILVu2ICgoCAsWLMCjjz4qx49LZiaEQEJaHv5OTMeN7CJkF5Qiq7AE2YWlyCooRU5hKbIKS1FQwlBDRJLOTT2x7qWesh3fqsJNSkoK/P39Dbb5+/sjJycHhYWF+vVV7hQbG4t33nnH6GMKIVAo01+iTvaqGs06+vTTT3H+/Hl06NABc+fOBQCcOnUKADB9+nR8/PHHaN68Oby8vJCUlITBgwfjvffeg1qtxjfffIOYmBicO3cOTZs2vecx3nnnHXz44Yf46KOP8Nlnn2HUqFG4cuUKvL29TfPDkkVJzyvG3sR07D6fjj2JN5GaU1z9i25zsFOCc+WIGjZ7lbzzlawq3BhjxowZmDp1qv77nJwcBAcH1/j1haUatJstT0/P6bmD4OxQ/b8iDw8PODg4wNnZGQEBAQCAs2fPAgDmzp2LBx54QL+vt7c3IiIi9N/PmzcP69evxy+//IKXX375nscYN24cRo4cCQB4//33sXjxYhw8eBD/+Mc/jPrZyLIUlWpw+HIm/kq4ib8S0nE62XCEU22nRLdQb7QJcIOHkz08nB2kfzrZw/P2Pz2c7OHuZA+VktGGiORlVeEmICAAqampBttSU1Ph7u5e6agNAKjVaqjVanOUZ5G6dOli8H1eXh7efvttbNy4EcnJySgrK0NhYSGuXr1a5fuEh4frH7u4uMDd3V1/iwWSlxAC2YWluJJRgCu3CpB0qwBXMvJxJUN6nJxTBCFq/77tAt3Ru5UPeof5okuIFxztudI0EVkHqwo3UVFR2LRpk8G2rVu3Iioqqt6O6WSvwum58jQrO5ngw8TFxcXg+2nTpmHr1q34+OOPERYWBicnJwwbNgwlJSVVvo+9vb3B9wqFAlqtts71Ue1lF5Zi34V0/JWQjmPXsnAlowC5RWV1fl9/dzV6t/RF75Y+6BnmAx/XhvtHARFZN1nDTV5eHhITE/XfX7p0CUePHoW3tzeaNm2KGTNm4Pr16/jmm28AAC+88AI+//xzvPHGG3j22Wexfft2rFmzBhs3bqy3GhUKRY0uDcnNwcEBGk31vUF79+7FuHHj8NhjjwGQ/h1cvny5nqujuijVaHE0KQt/JaTjr4SbOJaUBW0lIzF+bmo0a+SMpt4uaOrtjGaNnBHs7YwmXk6wq+ZSkUKhgJezPVeWJiKbIOun9uHDh9G/f3/997remLFjx2LlypVITk42uFwSGhqKjRs3YsqUKfj000/RpEkTfPXVV5wGDmmG04EDB3D58mW4urrec1SlZcuWWLduHWJiYqBQKDBr1iyOwFiIkjJpKnX27ZlIp27kYPf5dOy/mIG8YsORmRa+Lujd0hf3N/dGc19XBHs5w8mBl42IiACZw02/fv0gqmgGqGz14X79+iE+Pr4eq7JO06ZNw9ixY9GuXTsUFhbi66+/rnS/hQsX4tlnn0WPHj3g4+ODN9980yanx1ui3KJS7LuQgb8vZOBGViGyCqVp1Lop1VXNyvNytkevlr7oHeaDXi190Niz8h4zIiICFKKqdGGDcnJy4OHhgezsbLi7uxs8V1RUhEuXLiE0NBSOjvItPmQLeC6BMo0Wx67pLiel42hSFjSVXU+6g0IBuDtKM4+CvZ3QM8wHfVr6ol2gO5SchUREDVhVn993s/xmEiIrciUjH7vPS9Op913IQO5dl5NCfVzQK8wHrQPc4Olsf8d0amlqtZujHUMMEVEdMdwQ1dGVjHz8djwZvx1Pxpm71ofxdLZHzxbSpaReYT4I9naWqUoiooaD4YbICNcyC7DxdqA5cT1bv91OqUCXEC/9lOr2jT24qB0RkZkx3BDVUHJ2oT7QHE3K0m9XKRXo0aIRHgkPxKD2AfB0dpCvSCIiYrghqk5ecRne23gGqw5d1a/0q1AA94c2wiMRgfhH+wA04oJ3REQWg+GGqAr7LmTgn2uP4VpmIQCgW4g3Hg4PxEMdA+Dn1jBngRERWTqGG6JKFJZo8MHms1j592UAQBMvJ3w0LAJRLRrJWxgREVWL4YboLnFXbmHaj8dxKT0fAPB096aYObgtXNX8z4WIyBrw/9ZEtxWVavDJn+fx5e6L0AogwN0RHwwLR99WvnKXRkREtaCUuwCyDCEhIVi0aJHcZcjmxLVsxHy2B1/skoLN452DsGVKHwYbIiIrxJEbavC+3X8Fc345BY1WwMfVAe8/1hEPtg+QuywiIjISww01aJfT8zH319PQaAUe7hiIeUM7wNuF69QQEVkzXpayAf/5z3/QuHFjaLVag+1DhgzBs88+iwsXLmDIkCHw9/eHq6srunbtij///FOmai3LvN9Oo0SjRe+WPvj86fsYbIiIbADDTXWEAEry5fmq4Q3bn3zySWRkZGDHjh36bbdu3cLmzZsxatQo5OXlYfDgwdi2bRvi4+Pxj3/8AzExMbh69Wp9nTWrsP1sKradTYOdUoE5Me2hUPA2CUREtoCXpapTWgC831ieY8+8ATi4VLubl5cXHnroIXz//fcYOHAgAGDt2rXw8fFB//79oVQqERERod9/3rx5WL9+PX755Re8/PLL9Va+JSsu02Dur6cBAM/2CkWYn6vMFRERkalw5MZGjBo1Cj/99BOKi4sBAN999x2eeuopKJVK5OXlYdq0aWjbti08PT3h6uqKM2fONOiRm6/+uoTLGQXwdVPjlQFhcpdDREQmxJGb6tg7SyMoch27hmJiYiCEwMaNG9G1a1f89ddf+OSTTwAA06ZNw9atW/Hxxx8jLCwMTk5OGDZsGEpKSuqrcouWnF2Iz7cnAgBmDm4DN0d7mSsiIiJTYripjkJRo0tDcnN0dMTjjz+O7777DomJiWjdujU6d+4MANi7dy/GjRuHxx57DACQl5eHy5cvy1itvN7fdBaFpRp0aeaFoZ2C5C6HiIhMjOHGhowaNQqPPPIITp06hdGjR+u3t2zZEuvWrUNMTAwUCgVmzZpVYWZVQ7H/YgZ+PXYDCgXw9qNsIiYiskXsubEhAwYMgLe3N86dO4enn35av33hwoXw8vJCjx49EBMTg0GDBulHdRqSMo0Wb/9yCgDwdLem6BDkIXNFRERUHzhyY0OUSiVu3KjYHxQSEoLt27cbbJs0aZLB9w3hMtV3B67ibEouPJ3tMe3B1nKXQ0RE9YQjN9QgZOQVY8Ef5wAArz/YGl5crI+IyGYx3FCD8PEf55BTVIZ2ge54ultTucshIqJ6xHBDNu/4tSysOpQEAJg7pD1USjYRExHZMoYbsmlarcCcX05BCOCx+4LQJcRb7pKIiKieMdxUQtTwnk50b5ZyDtfFX0f81Sy4OKgw46E2cpdDRERmwHBzB3t7aaXagoICmSuxfrpzqDunctBqBT7achYA8OrAlvBzd5StFiIiMh9OBb+DSqWCp6cn0tLSAADOzs5c5K2WhBAoKChAWloaPD09oVKpZKvlws08pOYUw9lBhfE9Q2Wrg4iIzIvh5i4BAQEAoA84ZBxPT0/9uZTLkauZAIDwJh5wsOMgJRFRQ8FwcxeFQoHAwED4+fmhtLRU7nKskr29vawjNjrxV7MAAPc19ZK3ECIiMiuGm3tQqVQW8QFNxtON3HRmuCEialA4Vk82KaeoFAlpeQCA+5p6ylsMERGZFcMN2aTjSdkQAmjq7QwfV7Xc5RARkRkx3JBN0l2S4qgNEVHDw3BDNileF26CPeUthIiIzI7hhmyOEALxSVkAgM7N2ExMRNTQMNyQzbmUno+sglKo7ZRoE+AudzlERGRmDDdkc3Tr23QM4uJ9REQNEf/PTzZHv74NL0kRETVIDDdkc/QrE7OZmIioQWK4IZtSUFKGsyk5ADhyQ0TUUDHckE05lpQNrQAaezjC391R7nKIiEgGDDdkU+KTdIv3cdSGiKihYrghm3LkShYArkxMRNSQMdyQzRBC4ChHboiIGjyGG7IZ1zILkZ5XAnuVAu0bc/E+IqKGiuGGbIZufZv2jT3gaK+SuRoiIpILww3ZDP36Nuy3ISJq0BhuyGboVyZmvw0RUYPGcEM2oahUg9M3pMX7OHJDRNSwMdyQTTh5PRtlWgFfNzWCPJ3kLoeIiGTEcEM2ofySlCcUCoXM1RARkZwYbsgmlDcTs9+GiKihY7ghm8A7gRMRkQ7DDVm9G1mFSMkpgkqpQHgTT7nLISIimTHckNXTjdq0DXSDkwMX7yMiaugYbsjq6ZqJ7wtmvw0RETHckA2I182UauYpbyFERGQRGG7IqhWXaXBSt3gfR26IiAgWEG6WLFmCkJAQODo6onv37jh48GCV+y9atAitW7eGk5MTgoODMWXKFBQVFZmpWrI0p2/koKRMC28XBzRr5Cx3OUREZAFkDTerV6/G1KlTMWfOHBw5cgQREREYNGgQ0tLSKt3/+++/x/Tp0zFnzhycOXMGy5cvx+rVqzFz5kwzV06W4s4p4Fy8j4iIAJnDzcKFCzFhwgSMHz8e7dq1w7Jly+Ds7IwVK1ZUuv/ff/+Nnj174umnn0ZISAgefPBBjBw5strRHrJd+mZi3k+KiIhuky3clJSUIC4uDtHR0eXFKJWIjo7Gvn37Kn1Njx49EBcXpw8zFy9exKZNmzB48GCz1EyWRzdywzuBExGRjp1cB05PT4dGo4G/v7/Bdn9/f5w9e7bS1zz99NNIT09Hr169IIRAWVkZXnjhhSovSxUXF6O4uFj/fU5Ojml+AJJdWk4RrmcVQqEAwrkyMRER3SZ7Q3Ft7Ny5E++//z7+/e9/48iRI1i3bh02btyIefPm3fM1sbGx8PDw0H8FBwebsWKqT0duj9q09neDq1q2nE5ERBZGtk8EHx8fqFQqpKamGmxPTU1FQEBApa+ZNWsWnnnmGTz//PMAgI4dOyI/Px8TJ07EW2+9BaWyYlabMWMGpk6dqv8+JyeHAcdGxCfp+m14SYqIiMrJNnLj4OCAyMhIbNu2Tb9Nq9Vi27ZtiIqKqvQ1BQUFFQKMSiUtty+EqPQ1arUa7u7uBl9kG8rvBO4pax1ERGRZZB3Lnzp1KsaOHYsuXbqgW7duWLRoEfLz8zF+/HgAwJgxYxAUFITY2FgAQExMDBYuXIj77rsP3bt3R2JiImbNmoWYmBh9yKGGoVSjxfFrWQDYTExERIZkDTcjRozAzZs3MXv2bKSkpKBTp07YvHmzvsn46tWrBiM1//rXv6BQKPCvf/0L169fh6+vL2JiYvDee+/J9SOQTOKvZqGoVAt3Rzs093GRuxwiIrIgCnGv6zk2KicnBx4eHsjOzuYlKis2dc1RrDtyHU9GNsFHT0bIXQ4REdWz2nx+W9VsKSIAyC4oxcbjyQCAp7s3lbkaIiKyNAw3ZHXWx19DcZkWbQLc0Inr2xAR0V0YbsiqCCHww8EkANKoDe8nRUREd2O4Iaty5GoWzqXmwtFeiSGdguQuh4iILBDDDVmVHw5eBQA8Et4YHk72MldDRESWiOGGrEZ2YSl+O34DADCyGxuJiYiocgw3ZDU2xF9HUakWrf3d0JmrEhMR0T0w3JBVkBqJpUtSbCQmIqKqMNyQVYhPysLZlFyo7ZQYeh8biYmI6N4Ybsgq/HCAjcRERFQzDDdk8XKKSvHr7Ubip7sHy1wNERFZOoYbsng/324kbuXvyjuAExFRtRhuyKIJIfDd7UtSI7uxkZiIiKrHcEMW7di1bH0j8WNsJCYiohpguCGL9v2BKwCAhzsGwtPZQeZqiIjIGjDckMXKKSrFr8eSAUhr2xAREdUEww1ZrJ+P3kBhqQYt/VwR2YyNxEREVDMMN2SRhBD4no3ERERkBIYbskjHr2XjTHIOHOyUeLwzG4mJiKjmGG7IIunuI8VGYiIiqi2GG7I4uUWl+OWYtCLxyG5sJCYiotphuCGL8+uxZBSUaNDC1wVdQ9hITEREtcNwQxZnzeEkAMBTXdlITEREtcdwQxYlMS0XR5OyoFIqMJQrEhMRkREYbsii/Bh3DQDQv7UvfN3UMldDRETWiOGGLEaZRot1R64DAIZFBstcDRERWSuGG7IYuxNu4mZuMbxdHDCgjZ/c5RARkZViuCGL8eNh6ZLU0E5BcLDjryYRERmHnyBkEW7ll+DPM6kAgCe7NJG5GiIismYMN2QRfj56HaUagQ5B7mgb6C53OUREZMUYbsgi6C5JPclGYiIiqiOGG5LdqRvZOJ2cAweVEo9GNJa7HCIisnIMNyS7tbfXtolu5wcvF94kk4iI6obhhmRVUqbFz0elm2TykhQREZkCww3JavvZVNzKL4Gfmxq9W/rIXQ4REdkAhhuSla6R+PHOTWCn4q8jERHVHT9NSDZpuUXYef4mAK5tQ0REpsNwQ7JZf+Q6NFqBzk090cLXVe5yiIjIRjDckCyEEPo7gPMmmUREZEoMNySLo0lZSEzLg6O9Eo9EBMpdDhER2RCGG5KFbtTmH+0D4O5oL3M1RERkSxhuyOyKSjX49djttW268JIUERGZFsMNmd2WUynILSpDkKcTopo3krscIiKyMQw3ZHa62y08EdkESqVC5mqIiMjWMNyQWV3PKsSexHQAwJORXNuGiIhMj+GGzGpD/HUIAdzf3BvB3s5yl0NERDaI4YbM6sS1bADAA+0CZK6EiIhsFcMNmdXVWwUAgFAfjtoQEVH9YLghsxFCIOl2uGnKS1JERFRPGG7IbLIKSpFbXAYAaOLFcENERPWD4YbMRndJyt9dDUd7lczVEBGRrWK4IbPRhZtm3i4yV0JERLaM4YbMRhduOAWciIjqE8MNmQ2biYmIyBwYbshsdCM3TRs5yVwJERHZMoYbMpurHLkhIiIzYLghsyjVaHEjqxAAe26IiKh+MdyQWdzIKoRWAI72Svi6quUuh4iIbBjDDZnFnZekFAqFzNUQEZEtY7ghs2C/DRERmQvDDZkF17ghIiJzkT3cLFmyBCEhIXB0dET37t1x8ODBKvfPysrCpEmTEBgYCLVajVatWmHTpk1mqpaMdTWDIzdERGQednIefPXq1Zg6dSqWLVuG7t27Y9GiRRg0aBDOnTsHPz+/CvuXlJTggQcegJ+fH9auXYugoCBcuXIFnp6e5i+eaoWXpYiIyFxkDTcLFy7EhAkTMH78eADAsmXLsHHjRqxYsQLTp0+vsP+KFStw69Yt/P3337C3twcAhISEmLNkMoIQgiM3RERkNkZdlrp06RISEhIqbE9ISMDly5dr9B4lJSWIi4tDdHR0eTFKJaKjo7Fv375KX/PLL78gKioKkyZNgr+/Pzp06ID3338fGo3GmB+DzCS7sBS5xWUAgCZeDDdERFS/jAo348aNw99//11h+4EDBzBu3LgavUd6ejo0Gg38/f0Ntvv7+yMlJaXS11y8eBFr166FRqPBpk2bMGvWLCxYsADvvvvuPY9TXFyMnJwcgy8yL90lKT83NZwcVDJXQ0REts6ocBMfH4+ePXtW2H7//ffj6NGjda3pnrRaLfz8/PCf//wHkZGRGDFiBN566y0sW7bsnq+JjY2Fh4eH/is4OLje6qPKsd+GiIjMyahwo1AokJubW2F7dnZ2jS8R+fj4QKVSITU11WB7amoqAgICKn1NYGAgWrVqBZWq/K//tm3bIiUlBSUlJZW+ZsaMGcjOztZ/JSUl1ag+Mh2GGyIiMiejwk2fPn0QGxtrEGQ0Gg1iY2PRq1evGr2Hg4MDIiMjsW3bNv02rVaLbdu2ISoqqtLX9OzZE4mJidBqtfpt58+fR2BgIBwcHCp9jVqthru7u8EXmVcS17ghIiIzMmq21AcffIA+ffqgdevW6N27NwDgr7/+Qk5ODrZv317j95k6dSrGjh2LLl26oFu3bli0aBHy8/P1s6fGjBmDoKAgxMbGAgBefPFFfP7555g8eTJeeeUVJCQk4P3338err75qzI9BZsKRGyIiMiejwk27du1w/PhxfP755zh27BicnJwwZswYvPzyy/D29q7x+4wYMQI3b97E7NmzkZKSgk6dOmHz5s36JuOrV69CqSwfXAoODsaWLVswZcoUhIeHIygoCJMnT8abb75pzI9BZqIPN40YboiIqP4phBBC7iLMKScnBx4eHsjOzuYlKjMo1WjRZtZmaLQCB2YOhL+7o9wlERGRFarN57fRt1/466+/MHr0aPTo0QPXr18HAPzvf//Dnj17jH1LskHJWUXQaAXUdkr4uqrlLoeIiBoAo8LNTz/9hEGDBsHJyQlHjhxBcXExAGm21Pvvv2/SAsm63XnDTKVSIXM1RETUEBgVbt59910sW7YMX375pf42CIA0m+nIkSMmK46sH5uJiYjI3IwKN+fOnUOfPn0qbPfw8EBWVlZdayIbwnBDRETmZlS4CQgIQGJiYoXte/bsQfPmzetcFNmOJIYbIiIyM6PCzYQJEzB58mQcOHAACoUCN27cwHfffYdp06bhxRdfNHWNZMU4ckNEROZm1Do306dPh1arxcCBA1FQUIA+ffpArVZj2rRpeOWVV0xdI1kxrnFDRETmVutwo9FosHfvXkyaNAn//Oc/kZiYiLy8PLRr1w6urq71USNZqeyCUmQXlgIAgr0YboiIyDxqHW5UKhUefPBBnDlzBp6enmjXrl191EU2IClTGrXxdVPDyUFVzd5ERESmYVTPTYcOHXDx4kVT10I2hv02REQkB6PXuZk2bRp+++03JCcnIycnx+CLCGC4ISIieRjVUDx48GAAwKOPPgqFonzVWSEEFAoFNBqNaaojq3Ylo3x1YiIiInMxKtzs2LHD1HWQDeIaN0REJAejwk3fvn1NXQfZIF6WIiIiORgVbgAgMzMTy5cvx5kzZwAA7dq1w/jx4+Ht7W2y4sh6lWm0uJ5VCIDhhoiIzMuohuLdu3cjJCQEixcvRmZmJjIzM7F48WKEhoZi9+7dpq6RrFBydhE0WgEHOyX83NRyl0NERA2IUSM3kyZNwogRI7B06VKoVNL6JRqNBi+99BImTZqEEydOmLRIsj66S1LBXk5QKhXV7E1ERGQ6Ro3cJCYm4vXXX9cHG0Ba3G/q1KmV3lCTGh722xARkVyMCjedO3fW99rc6cyZM4iIiKhzUWT9GG6IiEguRl2WevXVVzF58mQkJibi/vvvBwDs378fS5Yswfz583H8+HH9vuHh4aaplKyK/rIUww0REZmZUeFm5MiRAIA33nij0ucUCgUX9GvguMYNERHJxahwc+nSJVPXQTZGf1mqEcMNERGZl1HhplmzZjXa7+GHH8ZXX32FwMBAYw5DViq7sBRZBaUAgGAvhhsiIjIvoxqKa2r37t0oLCysz0OQBdJdkvJxdYCL2uh1IomIiIxSr+GGGqYkNhMTEZGMGG7I5DgNnIiI5MRwQybHcENERHJiuCGT4xo3REQkJ4YbMjldz00zhhsiIpJBvYabmTNnwtvbuz4PQRZGoxW4linNkOMaN0REJAejwk1sbCxWrFhRYfuKFSvwwQcf6L+fMWMGPD09jS6OrE9ydiHKtAIOKiX83RzlLoeIiBogo8LNF198gTZt2lTY3r59eyxbtqzORZH10vXbNPF2glKpkLkaIiJqiIwKNykpKZWuOuzr64vk5OQ6F0XW62oGZ0oREZG8jAo3wcHB2Lt3b4Xte/fuRePGjetcFFkvTgMnIiK5GbU2/oQJE/Daa6+htLQUAwYMAABs27YNb7zxBl5//XWTFkjWheGGiIjkZlS4+ec//4mMjAy89NJLKCkpAQA4OjrizTffxIwZM0xaIFkX3nqBiIjkZlS4USgU+OCDDzBr1iycOXMGTk5OaNmyJdRqtanrIyvDkRsiIpJbnda5SUlJwa1bt9CiRQuo1WoIIUxVF1mhnKJSZBaUAuDIDRERyceocJORkYGBAweiVatWGDx4sH6G1HPPPceemwZMd0mqkYsDXNVGDQoSERHVmVHhZsqUKbC3t8fVq1fh7Fz+F/qIESOwefNmkxVH1oX9NkREZAmM+vP6jz/+wJYtW9CkSROD7S1btsSVK1dMUhhZH/bbEBGRJTBq5CY/P99gxEbn1q1bbCpuwBhuiIjIEhgVbnr37o1vvvlG/71CoYBWq8WHH36I/v37m6w4si5Xb92+YSbDDRERycioy1IfffQRBgwYgMOHD6OkpARvvPEGTp06hVu3blW6cjHZPq1W4OT1bABAc18XmashIqKGrNbhprS0FK+++ip+/fVXbN26FW5ubsjLy8Pjjz+OSZMmVXrPKbJ9x69n41Z+CdzUdogI9pS7HCIiasBqHW7s7e1x/PhxeHl54a233qqPmsgK7Tp3EwDQM8wH9qo6LZ9ERERUJ0Z9Co0ePRrLly83dS1kxXaeTwMA9G3tK3MlRETU0BnVc1NWVoYVK1bgzz//RGRkJFxcDHssFi5caJLiyDpk5pfgaFIWAKAfww0REcnMqHBz8uRJdO7cGQBw/vx5g+cUCkXdqyKr8ldiOoQAWvu7IdDDSe5yiIiogTMq3OzYscPUdZAV23mOl6SIiMhysPOT6kSrFdh9Ph0A0K8Vww0REcmP4Ybq5HRyDtLziuHsoEKXEG+5yyEiImK4obrRXZLq0cIHDnb8dSIiIvnx04jqZNd5aX0bzpIiIiJLwXBDRssuLMWRq1kAgL7styEiIgvBcENG25OQDo1WoIWvC4J5s0wiIrIQDDdktF23VyXu19pP5kqIiIjKMdyQUYQQ7LchIiKLxHBDRjmTnIvUnGI42avQlVPAiYjIgjDckFF0ozZRLRrB0V4lczVERETlGG7IKLr1bXhJioiILI1FhJslS5YgJCQEjo6O6N69Ow4ePFij161atQoKhQJDhw6t3wLJQG5RKeKuZALgFHAiIrI8soeb1atXY+rUqZgzZw6OHDmCiIgIDBo0CGlpaVW+7vLly5g2bRp69+5tpkpJZ29iBsq0AqE+LmjWyEXucoiIiAzIHm4WLlyICRMmYPz48WjXrh2WLVsGZ2dnrFix4p6v0Wg0GDVqFN555x00b97cjNUSUD4FnKM2RERkiWQNNyUlJYiLi0N0dLR+m1KpRHR0NPbt23fP182dOxd+fn547rnnqj1GcXExcnJyDL7IeEII7DonNRP3Zb8NERFZIFnDTXp6OjQaDfz9/Q22+/v7IyUlpdLX7NmzB8uXL8eXX35Zo2PExsbCw8ND/xUcHFznuhuyhLQ83MgugtpOiajmjeQuh4iIqALZL0vVRm5uLp555hl8+eWX8PHxqdFrZsyYgezsbP1XUlJSPVdp23SzpO5vzingRERkmezkPLiPjw9UKhVSU1MNtqempiIgIKDC/hcuXMDly5cRExOj36bVagEAdnZ2OHfuHFq0aGHwGrVaDbVaXQ/VN0y69W3Yb0NERJZK1pEbBwcHREZGYtu2bfptWq0W27ZtQ1RUVIX927RpgxMnTuDo0aP6r0cffRT9+/fH0aNHecmpnuUXl+HQJWkKONe3ISIiSyXryA0ATJ06FWPHjkWXLl3QrVs3LFq0CPn5+Rg/fjwAYMyYMQgKCkJsbCwcHR3RoUMHg9d7enoCQIXtZHp/X8hAiUaLpt7OCPXhFHAiIrJMsoebESNG4ObNm5g9ezZSUlLQqVMnbN68Wd9kfPXqVSiVVtUaZLPunAKuUChkroaIiKhyCiGEkLsIc8rJyYGHhweys7Ph7u4udzlWQwiB3h/uwLXMQiwf2wUD2/pX/yIiIiITqc3nN4dEqEYu3MzHtcxCOKiUiGrBKeBERGS5GG6oRnSzpLqFesPZQfarmURERPfEcEM1wruAExGRtWC4oWqVarQ4cOkWAK5vQ0RElo/hhqp1JSMfJWVauDioEObnKnc5REREVWK4oWolpuUBAML8XDkFnIiILB7DDVUrIVUKNy04akNERFaA4YaqlXhTCjct/dxkroSIiKh6DDdULd3IDfttiIjIGjDcUJU0WoEL+pEbhhsiIrJ8DDdUpeuZhSgu08LBTolgb2e5yyEiIqoWww1VKfFmLgCguY8LVErOlCIiIsvHcENVYr8NERFZG4YbqlJCGmdKERGRdWG4oSrduYAfERGRNWC4oXsSQujDTUt/hhsiIrIODDd0T6k5xcgrLoNKqUBIIxe5yyEiIqoRhhu6p4Q0aaZUs0bOcLDjrwoREVkHfmLRPen7bXx5SYqIiKwHww3dUwL7bYiIyAox3NA9caYUERFZI4YbuqdErnFDRERWiOGGKpWRV4xb+SUAgOa+nClFRETWg+GGKqUbtWni5QRnBzuZqyEiIqo5hhuqVOJN9tsQEZF1YrihSulumNmS4YaIiKwMww1V6gJHboiIyEox3FCldCM3YZwpRUREVobhhirILSpFSk4RAI7cEBGR9WG4oQp0M6X83NTwcLKXuRoiIqLaYbihCrgyMRERWTOGG6qgfGVihhsiIrI+DDdUAUduiIjImjHcUAUJaZwpRURE1ovhhgwUlWqQlFkAgCM3RERknRhuyMCFm3kQAvB0toePq4Pc5RAREdUaww0Z0Pfb+LpCoVDIXA0REVHtMdyQAf1MKX9ekiIiIuvEcEMGdOGmhS/DDRERWSeGGzKQoB+54UwpIiKyTgw3pFeq0eJyej4AzpQiIiLrxXBDelcy8lGmFXBxUKGxh6Pc5RARERmF4Yb09P02fpwpRURE1ovhhvQSUnnbBSIisn4MN6SXeJPhhoiIrB/DDenpRm5a8p5SRERkxRhuCACg0QpcuKkLNxy5ISIi68VwQwCA65mFKC7TwsFOiWBvZ7nLISIiMhrDDQEAEm/mAgCa+7hApeRMKSIisl4MNwSAM6WIiMh2MNwQgDtumMlmYiIisnIMNwSg/J5SHLkhIiJrx3BDEELggv6GmQw3RERk3RhuCKk5xcgtLoNKqUBIIxe5yyEiIqoThhtCQpo0U6pZI2c42PFXgoiIrBs/yUjfTBzmy0tSRERk/RhuSN9MzH4bIiKyBQw3VD5yw5lSRERkAxhuiGvcEBGRTWG4aeAy8opxK78EANDclzOliIjI+jHcNHBbT6cCAEIaOcPZwU7maoiIiOrOIsLNkiVLEBISAkdHR3Tv3h0HDx68575ffvklevfuDS8vL3h5eSE6OrrK/eneSsq0+Gx7IgDgmagQeYshIiIyEdnDzerVqzF16lTMmTMHR44cQUREBAYNGoS0tLRK99+5cydGjhyJHTt2YN++fQgODsaDDz6I69evm7ly67fmcBKuZxXCz02NUd2byl0OERGRSSiEEELOArp3746uXbvi888/BwBotVoEBwfjlVdewfTp06t9vUajgZeXFz7//HOMGTOm2v1zcnLg4eGB7OxsuLu717l+a1VUqkG/j3YiJacI7zzaHmN7hMhdEhER0T3V5vNb1pGbkpISxMXFITo6Wr9NqVQiOjoa+/btq9F7FBQUoLS0FN7e3pU+X1xcjJycHIMvAlYdvIqUnCIEejjiqW7BcpdDRERkMrKGm/T0dGg0Gvj7+xts9/f3R0pKSo3e480330Tjxo0NAtKdYmNj4eHhof8KDuYHeVGpBkt2XgAAvDwgDGo7lcwVERERmY7sPTd1MX/+fKxatQrr16+Ho6NjpfvMmDED2dnZ+q+kpCQzV2l5vt1/BTdzixHk6YQnIxn2iIjItsg699fHxwcqlQqpqakG21NTUxEQEFDlaz/++GPMnz8ff/75J8LDw++5n1qthlqtNkm9tqCgpAzLdkmjNq8ODOONMomIyObI+snm4OCAyMhIbNu2Tb9Nq9Vi27ZtiIqKuufrPvzwQ8ybNw+bN29Gly5dzFGqzfhm3xWk55WgWSNnPN65idzlEBERmZzsq7ZNnToVY8eORZcuXdCtWzcsWrQI+fn5GD9+PABgzJgxCAoKQmxsLADggw8+wOzZs/H9998jJCRE35vj6uoKV1feG6kqecVl+EI3ajOgJexVHLUhIiLbI3u4GTFiBG7evInZs2cjJSUFnTp1wubNm/VNxlevXoVSWf4hvHTpUpSUlGDYsGEG7zNnzhy8/fbb5izd6qzcewmZBaVo7uOCIZ0ay10OERFRvZB9nRtza6jr3OQUlaLX/O3IKSrDp091wpBOQXKXREREVGNWs84Nmc+KPZeQU1SGln6ueCScozZERGS7GG4agOyCUiz/6xIA4LXoVlApFTJXREREVH8YbhqAL/+6iNziMrQJcMNDHaqeYk9ERGTtGG5s3K38Eny9Vxq1mfJAKyg5akNERDaO4cbG/Wf3ReSXaNC+sTsebOdf/QuIiIisHMONDUvPK8Z//74MAJj6QCsoFBy1ISIi2yf7OjdUib2fAge/AlDNLH0nL2DwR0DT+yt9ev7vZ1FYqkFEEw8MaONn+jrvtm8JcHYT8NhSwLNp/R+PLM/v04GCdGDoMkDF/70QkTz4fx9LU3AL2P4eoCmuft/sJGDVKOD/dgEehrdS2JOQjrVx16BQALNj2tf/qE1hFrBtHlBWKNX03B+AvVP9HpMsS/Ix4MBS6XGX54Bm976FChFRfWK4sTTH10jBxq8dMOTze+8nAPw2GUg5AaweDYzfDNhLd0YvKCnDjPXHAQBjo0IQ2cyr/us+8aMUbAAg5Tjw2xRg6FKAl8Iajrj/lj++uJPhhohkw54bSyIEcOT2B0SXZ4GgyHt/NYkERnwnXZq6EQ9sfF16PYBPtp5H0q1CNPZwxLRBrc1Td9xK6XH7xwGFCjj2A3DwP/V/bLIMJflSwNW5uFO2UoiIGG4sybVDQNppwM4J6Phk9ft7NQOGrQAUSuDot8Dh5Th+LQvL90hTv997rCNc1WYYnLt+BEg9CajUwMMLgAfmStu3zASu/F3/xyf5nVoPFOcAzj7S99cPA8W58tZERA0Ww40l0Y9+PAY4edbsNS0GAAPnAADE79Px39WroRXAkE6N0d8cTcQAEPe19M/2QwFnbyBqEtDhCUBbBqwZA2RfN08dJB/d726PlwGvEOnfPYMtEcmE4cZSFGUDJ9dJjyPH1e61PScD7YZCoS3Fmznvo5VTLmY/0s7kJVaqKKdi3QoF8OhngH8HIP+mFHDKatAgTdYp9ZQ06qi0AzqNApr3k7bz0hQRyYThxlIcXyM15Pq2BYK71e61CgUu9fwQ50UT+Cmy8IPnUjRyNFMj78m1QGk+4NMKaHpHA6mDCzDiW8DRU7pEsemf5qmHzE/XSNx6MODqx3BDRLJjuLEEQpR/QESOrfUMI61WYPpvFzGhZCryFS5olHkU2Dzd9HVWRnc5onMldXuHAsOWA1BIjdKHvzZPTWQ+JQXA8VXSY93IXUgfAAqpfyw3Va7KiKgBY7ixBDeOAKknpIbc8BG1fvmqQ0k4cOkW0uyCUPDIMgAK4PBy4Mj/TF/rnW7ES2ubqByAiJGV7xMWDQycJT3e9E8g6VD91kTmdfpn6ZKqZ1OgeX9pm0sjIKCj9PjSLvlqI6IGi+HGEugbiYdKDbm1kJpThNhNZwAA0wa1hm/ko0D/mdKTG6cC1+NMV+fddKNNbWOkD7R76TVV2kdbCqx5hn/N2xLd0gWdxwDKO/53or80xXBDRObHcCO34lzgxE/S485ja/3y2T+fRG5xGSKCPTGuR4i0sfc0qf9BUwKsfgbIu2m6enWK88rXNamuAVqhkBb082kN5CYDP44FykpMX1Nlcm4AmZfr/j5lJVJQFNXcEqMmMi9LjdiWJO0soCmt/Wuu7pPWNeo02vC5O/tuTHHO7iX1lHRpjIjoDgw3cjtxuyG3UUugWY9avXTzyWRsOZUKO6UCHzzRESrl7Z4XpRJ4bBnQKAzIuQ78OK72H1zVObUOKMkDvJsDIb2r31/tBjz1PaB2lz4QDywzbT2VSU8EltwPLO0J5KXV7b22zwO+HABsmla39zn/B7D4PmBJd8sZwdqzCPh3d2Dlw7ULnbpRm1b/ANwDDZ9rGiVdrsy5BmRcMFmpBs78BiztAfz8Uv28PxFZLYYbuekuSUWOq1UjcXZBKWb9fAoA8GK/FmgT4G64g6OHFCYcXIEre4Cts01Tr05VjcT34hNWvsBf3Nf1+xd9cS6w6mmgOFsKYUe/M/69SgrKP8gPfWV8L1PGBeCn5wGhBXJvmHcE614S/wT+fFt6nHQA2PxmzV5XWiStQg1UPnLn4AwEd5ceX9xR1yorpwvIp3/mWkpEZIDhRk43jgLJR6tuyK2EEAKzfj6Jm7nFaO7rgkn9wyrf0be1NIIDAPv/LU03N4WUE9IlGqW9tK5JbYQPBxzcgFsXgct/maaeuwkBbHgRSD8nrb0CSP1BWq1x76drmtW9lzG9TMV50j3AirOBwIjyEawtM42ryRRuXQLWPgdAAM16QWpEXwEc+ab61575FSjMBNybAGEDK9+neV/pn/XRVJyeWP77I7RA/LemPwYRWS2GGzkdqWFD7l2+2XcFvxy7AZVSgY+GhcPRXnXvndvGSD04APDLq0Dy8ToUfJuukbjNw4Crb+1e6+AChN++tYRu9MfU9iyUPnyV9sDon6QwlXnJ+DCl+/fU903jepmEAH55WZoa7eoPjFwNPH77vluHvgSOfm9cXXVRUiD9DEVZQFAX4Jl1QP+3pOc2vg5cqya86UfungGU9/j9082eurQb0GpMUXU53b8TR0/pn/H/M/0xiMhqMdzIpTgPOF7Dhtw7xF3JxLsbTwMAZjzUBpHNajC7qv9MaUp2WSGwehRQcMuIgm8rKQCOr5Ye13YlZR3d6878CuRnGF9LZRL/BLbNkx4P/khqbK1LmLqzafa+Z4zrZfp7sXTvJaUdMPwbqT+l9UNA39trEf36mjSt3lyEAH59VVp+wMVXqslODfR+HWj9sBTe1lQR3tITpUudCiVw3+jK9wGAwE6A2kMa9Uo+arr6y4rLLzM+8okUcLKTgAvbTXcMIrJqDDdyObUOKMmteUMugPS8Ykz67ghKNQKDOwbguV6hNTuWUgU88ZV0z5+sq8DaZ43/K1d3g0TPZkBoX+PeIzACaHyf9CGq69swhTsvs3QeA3QZL203CFPptXvPu5tma9vLdGFHeU/LP+YDTe8vf67vm9L7aoqBVaNrX5ux9v9bmummtAOe/C/gESRt1zeit6w6vB1ZKf0z7AHAo8m9j6OyA0Jv/26bcrXisxuBggzALRBo+2j5Jd36GgkkIqvDcCMX3aWdGjbklmm0ePWHeKTkFKG5rws+HBYBRW1WMnbyAkZ8B9g7Sw2e2+YaV7fuwz5yrOG6JrWlm/Yet9I0jcUl+VJPi+4yy+CPy5/ThSltae3C1L2aZn1bS1Pbgdu9TD9W/vrMK1KQFFppqnTX5w2fVyqBx74AvFtIs4rWjgc0ZTWvzxiXdgN/3F5UcdD7QEhPw+cd3W+HNzcpvOn21SkrLr+MVpORO10ANmW40f0O3jdaClCRt3+Xzv0O5KaY7jhEZLUYbuSQckK631ItGnIXbD2Pvy9kwNlBhS9GR8JVbVf74wZ0kG5oCQB7FwGnNtTu9amnpRk1SruK65rUVsdhgL0LkJFQ97tHCyH1E6WeNLzMcidjwlRVTbPtHpUu4wDAL69U7GUqLZTCVuEtKVg9vKDyEOvkCTz1nXQuLu0G/pxTs9qMkZUE/DgeEBog/Cmg28TK9/NtVd6IfmApcGx1+XN3jpq0fLD6Y+rWu7l6QDondXXr4u2gpJAuEwKAX1sg+H7p52JjMRGB4UYe+obcwTVqyP3jVAqW7pTWCvngiXC09Hcz/tgdhwFRL0uPN7wEpJ2p+WvvvETj5m98DYC07k3HJwzf11j7/y3dwPPuyyx30oepRODK3pq9b3VNs/3fAloMrNjLJITUR5NyHHBuBAz/H2DveO/j+LUFhv5berzvc2ntI1MrLZL6aArSgYBwIGZR1SOGbR8pb0T/dbJ0mw2g/JzoRk2q49MScGssXXq7ur8uP4FEN5OrxQDAq1n5dt3ozZFvjJ8VR0Q2g+HG3EoKyqdk12BY/1J6Pl5fI32wjO8ZgpiIxnWvIfodILSPtHjgqlFAYVb1ryktBI7pbpA4vu41AOU//6kNxjc5V3eZRUftJgUcoDxcVqUmTbN39zL99JzUy3TwP9LNJBUq4MmVgGdw9cdrPxToNUV6/PPL0uieqQghTV+/EQ84eUt3a7d3qv51/WdKfTVlt0ehrsXdntZ9x6hJdRQK090lXFMKxN9uJL77v512Q6Xm5awrwKU6HoeIrB7Djbmd3iCtdeLZDAjtV+WuBSVlePHbOOQWl6FLMy/MHNzWNDWo7IBhKwGPYODWBWD9/1X/1+7pX6R+Fo+mQIv+pqmjcWfpBoua4vIZWLVR08ssOrq/7k//XH2YqmnTrLN3eS/The3Swny6tWsenCeFyJoaMEsakSgrlEJnXWa13enQV9LsIoUSGLbCcMSjKkoV8MSXgFeoFN7++4i0PWxgzd8DMF24Ofc7kJ8GuPhJs83u5OAMRNy+6Swbi4kaPCMaN6hSN88Df7xV/X66v8jvvtHgXYQQeGv9SZxNyYWPqxpLRnWGvcqEWdSlETDif8DyQcD5zcCKQVL/xz3rPnm77irWNakthULqhdk0TRpN6f5CzVc7ru1lFqA8TKWckEahou6xbH9tm2Z1vUw/PSf16QBAxyeB+2t5WwClCnhiOfCfvtIIxIpB0qhQXQhRvkJw9Nu1D6ZOXlJP0FfRQOntezjV9h5ouoCXfEwKbLW8Oaye/pLYKEBlX/H5zmOlUbOzG6Xbbbj6GXccsh43zwG7PpRmcFZFaQ90fe7eC06SzWG4MZWibCDhj5rtq1JXvT4IgG8PXMX6+OtQKRX4/On74O9eRc+GsRrfB8R8Cmx4Abh2sPr9VQ7V1l1r4cOly0o3zwBJB4Gm3at/jRDSQnO1vcyiUEhhZePrUp/P/S9WHohq2zQLSJe8bsRLPTP+HYCYxbW6nYaebiRo+YNA+nnpyxTaPwb0eNW41/q3B4Z8Ls38cg+qOGpSHfdAwLcNcPOstJBiuyG1ryHzSvk6Np3HVL5PQAdpptz1w1I47fVa7Y9D1iM/Hfj2CWmNo5pI/BN4djMQ1Ll+6yKLwHBjKl4hwJAlNdvXvwPgFgBAGqHJLS5DdkEpsgulr2uZBZj7q3TfqDcGtcb9zWu+enGtdRoJ+LSSwkW1dbcH3E3Q83MnRw+gwxPA0W+lv8xrEm4OL5f2r+1lFkAaUfljlvRBm3TAcN0Zndo2zeo8+K70wR/YSbpMYqzAcGDSfuDyHuPf406OnlITuDFhS6fDE9IlSRefykdNqtO8n3TOL+40LtzE/w+AkN7Hu/m994scJ4WbI/+Vwlxdlisgy6Upk5ZOyE6Sfh96Ta369/vkOuDCNmlV7ok7a7+yOlkdhRD1efdCy5OTkwMPDw9kZ2fD3d29+heYUEmZFgu3nse+ixnILijRhxntPf4N/KN9AJaO7ly79WysUdJBYPkDgJ0T8PrZqi+PXd0v3b1aWybdhLPn5Nofb8MkKRxFjCyf8qxz66J0124ogMnHahec6N7O/Q788JT0QfRqLVdj1pQBizoAucnAsK+BDo/fe9+SfODj1tICmWN+Kb+/FdmWLW9Jo6T2LsCEbdKMw6oUZQNfDpBmS4b0Bp7ZULs/XMgi1Obzm3/WmElOUSnGrzyIZbsu4FhSFi5nFCCzoDzYqO2U8HdXo5W/K7qGeGFMVDN89GS47QcbAGjSFfBrJzXSnrjHgngAkJMMrBkjBZu6XGbRz9JaL61jcyfdVOPaNs1S1Zr1lGaP3booNSfXRsIfUrBx9gHaPFL1vnfeu6yuSwyQZTqxVgo2gLSEQnXBBpBGiEd8J60sfvmv6lcWJ6vH6GoGydmFGP/1IZxNyYWzgwpzYtqhua8rPJzs9V9V3vzS1ul6YX5/Q7ok1PX5ikPMZSVSsMlLlYLQkCXGX2Zp0kV6j7TT0urC3SeWH0O3CFxtm2apao7uQFCk1Nt1cZfUmF5TusuEnZ4G7Byq3z9ynHR3c93tNlx8jKmYLFHKSWmpBADo+Zq0hEJN+bWRVhZf8wywf4nUc6gLwmRzOHJTz86m5OCxJX/jbEoufN3UWPN/URjRtSm6hnijlb8b/N0dG3aw0QkfDtg5SqsMXz9S8fnNb0ofjI4eUgOxg4vxx9KFKcBwxeLzvwP5Nyufakx1Z8yU8OxrQOJW6XFNA2dghNT3ZOp7l5G8Cm5Ji2WWFUp3nB9oxOhLu0el/hxAWlnclOtJkUVhuKlHfyem48ml+5CSU4QwP1ese7EHOgR5yF2WZXLykhZiA4C4rw2fO/KN9Jc4FMDjXwGNWtT9eLowlXYKuB53+7h33rPIiKZZqtqd4aamqwjHfyvdmyukN+ATVvNj6cPrf01z7zKSl1YDrJsAZF4GPJtKEwmMXZJiwL/KVxY35XpSZFEYburJ+vhrGPv1QeQWl6FbiDfWvhCFYO86zKBpCHQfSCfXAUW31624FidN3Qak2x20quHU7OrcHaYyL98x1bgWl0yo5pp0lRY7LEiXLglWR6sp74GqyXpDdzLlvctIfjvek6Zy2zlJvTPGrpUE3LWy+JXylcXJpjDcmJgQAkt2JGLK6mMo1Qg8HB6Ib57rBk/nGvQKNHRN7wd8Wku3hTi5VlqIbfVo6fJC64fLb1RpKneGqX3/Ro2mGpPx7ByAZj2kx5d2Vb9/4p9AznUpiFbXSHy3O+9dxhWLrduZX4G/FkiPH10sLZVQV86318eyc5L+qNn+bt3fkywKw40JlWm0+NeGk/hoyzkAwITeofjsqfvYU1NTCkX5LRIOrwB+HAfk3gAatZSmbJt6zRJ9mCoADn4hbavtCAHVTm36bnSXCSOervrGo/ei+3dZk9ttkGW6eQ5Y/4L0+P6XpMvJphLQUVqcEgD2LJR+T8hmcLaUiRSUlOGV7+Ox7WwaFApg9iPtML5nqNxlWZ/wp4A/3y5v9HNwA576XpptY2q6MKW7F5SzjzRCRPVHF24u7wVuXbp330RhpnRbEKA88NZW486Af0cg9QRwaHn5vafIOpSVAKueBkrygGa9pHWtTO3OlcXXvwi4+pt+odKGSqUG3PxlOzzDjYn8euwGtp1Ng9pOiU+f6oR/dAiUuyTr5NIIaPuodFkKkEZsfFvV3/EiRkphSlNS86nGZDy/9lKILEgHFneqfv+mUYBva+OOpQuvm6YBO96Vvsj6uAcBT66svyb/6HeAlOPApd3S/dzINJp0A57fKtvhGW5MZHiXYFy8mY8H2vmjS0gdmt0I6DVFWrW42wSgbS17LWrL2RvoPU26W3v3/6vfY5F0aTHqJWD3Aulu7lWxdwb6TKvb8cJHSIv5pSfU7X1IHh5NgMe/rN/bJajspJWvV4+WRnHINFTy/qHI2y8QERGRxePtF4iIiKjBYrghIiIim8JwQ0RERDaF4YaIiIhsCsMNERER2RSGGyIiIrIpDDdERERkUxhuiIiIyKYw3BAREZFNYbghIiIim8JwQ0RERDaF4YaIiIhsCsMNERER2RSGGyIiIrIpdnIXYG5CCADSrdOJiIjIOug+t3Wf41VpcOEmNzcXABAcHCxzJURERFRbubm58PDwqHIfhahJBLIhWq0WN27cgJubGxQKhUnfOycnB8HBwUhKSoK7u7tJ35sq4vk2L55v8+L5Ni+eb/My5nwLIZCbm4vGjRtDqay6q6bBjdwolUo0adKkXo/h7u7O/zjMiOfbvHi+zYvn27x4vs2rtue7uhEbHTYUExERkU1huCEiIiKbwnBjQmq1GnPmzIFarZa7lAaB59u8eL7Ni+fbvHi+zau+z3eDaygmIiIi28aRGyIiIrIpDDdERERkUxhuiIiIyKYw3BAREZFNYbgxkSVLliAkJASOjo7o3r07Dh48KHdJNmP37t2IiYlB48aNoVAosGHDBoPnhRCYPXs2AgMD4eTkhOjoaCQkJMhTrJWLjY1F165d4ebmBj8/PwwdOhTnzp0z2KeoqAiTJk1Co0aN4OrqiieeeAKpqakyVWzdli5divDwcP1CZlFRUfj999/1z/Nc16/58+dDoVDgtdde02/jOTedt99+GwqFwuCrTZs2+ufr81wz3JjA6tWrMXXqVMyZMwdHjhxBREQEBg0ahLS0NLlLswn5+fmIiIjAkiVLKn3+ww8/xOLFi7Fs2TIcOHAALi4uGDRoEIqKisxcqfXbtWsXJk2ahP3792Pr1q0oLS3Fgw8+iPz8fP0+U6ZMwa+//ooff/wRu3btwo0bN/D444/LWLX1atKkCebPn4+4uDgcPnwYAwYMwJAhQ3Dq1CkAPNf16dChQ/jiiy8QHh5usJ3n3LTat2+P5ORk/deePXv0z9XruRZUZ926dROTJk3Sf6/RaETjxo1FbGysjFXZJgBi/fr1+u+1Wq0ICAgQH330kX5bVlaWUKvV4ocffpChQtuSlpYmAIhdu3YJIaRza29vL3788Uf9PmfOnBEAxL59++Qq06Z4eXmJr776iue6HuXm5oqWLVuKrVu3ir59+4rJkycLIfj7bWpz5swRERERlT5X3+eaIzd1VFJSgri4OERHR+u3KZVKREdHY9++fTJW1jBcunQJKSkpBuffw8MD3bt35/k3gezsbACAt7c3ACAuLg6lpaUG57tNmzZo2rQpz3cdaTQarFq1Cvn5+YiKiuK5rkeTJk3Cww8/bHBuAf5+14eEhAQ0btwYzZs3x6hRo3D16lUA9X+uG9yNM00tPT0dGo0G/v7+Btv9/f1x9uxZmapqOFJSUgCg0vOve46Mo9Vq8dprr6Fnz57o0KEDAOl8Ozg4wNPT02Bfnm/jnThxAlFRUSgqKoKrqyvWr1+Pdu3a4ejRozzX9WDVqlU4cuQIDh06VOE5/n6bVvfu3bFy5Uq0bt0aycnJeOedd9C7d2+cPHmy3s81ww0RVWrSpEk4efKkwTVyMr3WrVvj6NGjyM7Oxtq1azF27Fjs2rVL7rJsUlJSEiZPnoytW7fC0dFR7nJs3kMPPaR/HB4eju7du6NZs2ZYs2YNnJyc6vXYvCxVRz4+PlCpVBU6vFNTUxEQECBTVQ2H7hzz/JvWyy+/jN9++w07duxAkyZN9NsDAgJQUlKCrKwsg/15vo3n4OCAsLAwREZGIjY2FhEREfj00095rutBXFwc0tLS0LlzZ9jZ2cHOzg67du3C4sWLYWdnB39/f57zeuTp6YlWrVohMTGx3n+/GW7qyMHBAZGRkdi2bZt+m1arxbZt2xAVFSVjZQ1DaGgoAgICDM5/Tk4ODhw4wPNvBCEEXn75Zaxfvx7bt29HaGiowfORkZGwt7c3ON/nzp3D1atXeb5NRKvVori4mOe6HgwcOBAnTpzA0aNH9V9dunTBqFGj9I95zutPXl4eLly4gMDAwPr//a5zSzKJVatWCbVaLVauXClOnz4tJk6cKDw9PUVKSorcpdmE3NxcER8fL+Lj4wUAsXDhQhEfHy+uXLkihBBi/vz5wtPTU/z888/i+PHjYsiQISI0NFQUFhbKXLn1efHFF4WHh4fYuXOnSE5O1n8VFBTo93nhhRdE06ZNxfbt28Xhw4dFVFSUiIqKkrFq6zV9+nSxa9cucenSJXH8+HExffp0oVAoxB9//CGE4Lk2hztnSwnBc25Kr7/+uti5c6e4dOmS2Lt3r4iOjhY+Pj4iLS1NCFG/55rhxkQ+++wz0bRpU+Hg4CC6desm9u/fL3dJNmPHjh0CQIWvsWPHCiGk6eCzZs0S/v7+Qq1Wi4EDB4pz587JW7SVquw8AxBff/21fp/CwkLx0ksvCS8vL+Hs7Cwee+wxkZycLF/RVuzZZ58VzZo1Ew4ODsLX11cMHDhQH2yE4Lk2h7vDDc+56YwYMUIEBgYKBwcHERQUJEaMGCESExP1z9fnuVYIIUTdx3+IiIiILAN7boiIiMimMNwQERGRTWG4ISIiIpvCcENEREQ2heGGiIiIbArDDREREdkUhhsiIiKyKQw3RNTg7dy5EwqFosJ9bojIOjHcEBERkU1huCEiIiKbwnBDRLLTarWIjY1FaGgonJycEBERgbVr1wIov2S0ceNGhIeHw9HREffffz9Onjxp8B4//fQT2rdvD7VajZCQECxYsMDg+eLiYrz55psIDg6GWq1GWFgYli9fbrBPXFwcunTpAmdnZ/To0QPnzp2r3x+ciOoFww0RyS42NhbffPMNli1bhlOnTmHKlCkYPXo0du3apd/nn//8JxYsWIBDhw7B19cXMTExKC0tBSCFkuHDh+Opp57CiRMn8Pbbb2PWrFlYuXKl/vVjxozBDz/8gMWLF+PMmTP44osv4OrqalDHW2+9hQULFuDw4cOws7PDs88+a5afn4hMizfOJCJZFRcXw9vbG3/++SeioqL0259//nkUFBRg4sSJ6N+/P1atWoURI0YAAG7duoUmTZpg5cqVGD58OEaNGoWbN2/ijz/+0L/+jTfewMaNG3Hq1CmcP38erVu3xtatWxEdHV2hhp07d6J///74888/MXDgQADApk2b8PDDD6OwsBCOjo71fBaIyJQ4ckNEskpMTERBQQEeeOABuLq66r+++eYbXLhwQb/fncHH29sbrVu3xpkzZwAAZ86cQc+ePQ3et2fPnkhISIBGo8HRo0ehUqnQt2/fKmsJDw/XPw4MDAQApKWl1flnJCLzspO7ACJq2PLy8gAAGzduRFBQkMFzarXaIOAYy8nJqUb72dvb6x8rFAoAUj8QEVkXjtwQkazatWsHtVqNq1evIiwszOArODhYv9/+/fv1jzMzM3H+/Hm0bdsWANC2bVvs3bvX4H337t2LVq1aQaVSoWPHjtBqtQY9PERkuzhyQ0SycnNzw7Rp0zBlyhRotVr06tUL2dnZ2Lt3L9zd3dGsWTMAwNy5c9GoUSP4+/vjrbfego+PD4YOHQoAeP3119G1a1fMmzcPI0aMwL59+/D555/j3//+NwAgJCQEY8eOxbPPPovFixcjIiICV65cQVpaGoYPHy7Xj05E9YThhohkN2/ePPj6+iI2NhYXL16Ep6cnOnfujJkzZ+ovC82fPx+TJ09GQkICOnXqhF9//RUODg4AgM6dO2PNmjWYPXs25s2bh8DAQMydOxfjxo3TH2Pp0qWYOXMmXnrpJWRkZKBp06aYOXOmHD8uEdUzzpYiIoumm8mUmZkJT09PucshIivAnhsiIiKyKQw3REREZFN4WYqIiIhsCkduiIiIyKYw3BAREZFNYbghIiIim8JwQ0RERDaF4YaIiIhsCsMNERER2RSGGyIiIrIpDDdERERkUxhuiIiIyKb8P0kd/XpaUvlDAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(model.history.history.keys())\n",
    "plt.plot(model.history.history['accuracy'])\n",
    "plt.plot(model.history.history['val_accuracy'])\n",
    "plt.title('model rec_prec')\n",
    "plt.ylabel('rec_prec')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'val'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m7/7\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 18ms/step\n",
      "                   precision    recall  f1-score   support\n",
      "\n",
      "        Экономика       0.00      0.00      0.00         2\n",
      "            Спорт       0.00      0.00      0.00         2\n",
      "         Ценности       1.00      0.50      0.67         2\n",
      "             Крым       0.00      0.00      0.00         0\n",
      "   69-я параллель       0.00      0.00      0.00         1\n",
      "           Бизнес       0.00      0.00      0.00         2\n",
      "       Библиотека       0.00      0.00      0.00         0\n",
      "Силовые структуры       0.00      0.00      0.00         2\n",
      "           Россия       0.33      0.50      0.40         2\n",
      "              Дом       0.00      0.00      0.00         2\n",
      "              Мир       0.00      0.00      0.00         1\n",
      "         Культура       0.00      0.00      0.00         3\n",
      "          Легпром       0.00      0.00      0.00         1\n",
      "  Наука и техника       0.00      0.00      0.00         3\n",
      "   Интернет и СМИ       0.00      0.00      0.00         2\n",
      "         Из жизни       0.00      0.00      0.00         3\n",
      "      Бывший СССР       0.00      0.00      0.00         3\n",
      "\n",
      "        micro avg       0.06      0.06      0.06        31\n",
      "        macro avg       0.08      0.06      0.06        31\n",
      "     weighted avg       0.09      0.06      0.07        31\n",
      "\n"
     ]
    }
   ],
   "source": [
    "pred = model.predict(X_valid, batch_size=5).argmax(1)\n",
    "\n",
    "print(classification_report(y_valid.argmax(1), pred, labels=list(range(len(label2id))),\n",
    "                            target_names=list(label2id), zero_division=0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Модель очень сильно переобучается. Она полностью выучивает обучающую выборку, но с предсказанием новых данных все очень плохо (разрыв в качестве с предобученной моделью огромный). Увеличение количества параметров не поможет - будет точно такая же картина. Для того, чтобы увеличить качество необходимо больше данных. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}