notebooks/word_embeddings/nn_intro_tf.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Дисклеймер\n",
    "Эту тетрадку лучше запускать в колабе или в vast.ai. Не мучатесь с установкой библиотек и с обучением на cpu."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# !pip install pandas scikit-learn gensim matplotlib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Введение в нейронные сети "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Все задачи, которые мы решали до этого, можно решать с помощью нейронных сетей. Часто нейронные сети будут наилучшим выбором, особенно когда у нас текстовые или картиночные данные и их достаточно много (примерно от 50к). Но в любом случае, перед тем как доставать нейронки лучше прогнать несколько стандартных моделей. Часто они решат задачу на приемлимом уровне и время, затраченное на обучение нейронки, не окупит себя. Также с табличными данными лучше просто использовать бустинги."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Для написания нейронок есть два основных фреймворка -  tensorflow и pytorch. Сказать, что лучше, тяжело. В целом, тензорфлоу с встроенным keras (раньше был отдельным фреймворком) сильно проще для новичков, но многое в нём происходит \"под капотом\". Когда все работает это даже хорошо, но когда что-то ломается, очень сложно понять, что именно.  Pytorch сложнее в освоении и многие вещи там приходится делать вручную, но это заставляет разобраться в технических деталях и дает большую свободу. \n",
    "\n",
    "В наших занятиях мы попробуем tensorflow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Для наглядности возьмем две разные задачи - классификации и регресии. В обоих задачах мы будем работать с текстом и по большей части весь пайплайн у нас будет одинаков. Отличаться будут только последние штрихи в самих моделях."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Регрессия на данных Avito"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Данные из соревнования - https://www.kaggle.com/c/avito-demand-prediction/overview  \n",
    "Я взял только поле description, так как наша задача разобраться с нейронками, а не добиться самого большого качества."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from string import punctuation\n",
    "from sklearn.model_selection import train_test_split\n",
    "from collections import Counter\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Во всех предыдущих семинарах мы обрезали данные Авито, так как модели не могли столько переварить. В случае нейронок такой проблемы нет, модель обучается на батчах (кусках данных) и можно использовать неограниченное количество данных. Но для семинара будет слишком долго обучаться, поэтому пока тоже возьмем только 10 %."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "avito = pd.read_csv('avito_data.csv').dropna().sample(frac=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>description</th>\n",
       "      <th>deal_probability</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1056015</th>\n",
       "      <td>очень игривая ласковая девочка кличка Рада про...</td>\n",
       "      <td>0.09346</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>675100</th>\n",
       "      <td>Тетрадь \"Проверим себя \" 4 класс,окружающий ми...</td>\n",
       "      <td>0.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>729638</th>\n",
       "      <td>Если у ваших близких или друзей намечается важ...</td>\n",
       "      <td>0.35468</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>286811</th>\n",
       "      <td>Страна вывоза Объединённые Арабские Эмираты,Ст...</td>\n",
       "      <td>0.73760</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>263617</th>\n",
       "      <td>Очки в отличном состоянии!/\\nОправа итальянска...</td>\n",
       "      <td>0.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1497885</th>\n",
       "      <td>развивающие игрушки и погремушки. в отличном с...</td>\n",
       "      <td>0.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>770848</th>\n",
       "      <td>Продам утепленный комбинезон на мальчика. Разм...</td>\n",
       "      <td>0.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1414269</th>\n",
       "      <td>Monnalisa 30размер. Абсолютно новые. Пересыл</td>\n",
       "      <td>0.00000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>367424</th>\n",
       "      <td>В идеальном состоянии. Не бита не крашена. Сиг...</td>\n",
       "      <td>0.73760</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>690356</th>\n",
       "      <td>нарядный сарафан на 2-3 г.</td>\n",
       "      <td>0.00000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>138715 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               description  deal_probability\n",
       "1056015  очень игривая ласковая девочка кличка Рада про...           0.09346\n",
       "675100   Тетрадь \"Проверим себя \" 4 класс,окружающий ми...           0.00000\n",
       "729638   Если у ваших близких или друзей намечается важ...           0.35468\n",
       "286811   Страна вывоза Объединённые Арабские Эмираты,Ст...           0.73760\n",
       "263617   Очки в отличном состоянии!/\\nОправа итальянска...           0.00000\n",
       "...                                                    ...               ...\n",
       "1497885  развивающие игрушки и погремушки. в отличном с...           0.00000\n",
       "770848   Продам утепленный комбинезон на мальчика. Разм...           0.00000\n",
       "1414269       Monnalisa 30размер. Абсолютно новые. Пересыл           0.00000\n",
       "367424   В идеальном состоянии. Не бита не крашена. Сиг...           0.73760\n",
       "690356                          нарядный сарафан на 2-3 г.           0.00000\n",
       "\n",
       "[138715 rows x 2 columns]"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "avito"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Предобработка нам не так важна. Главное - получить из текстов токены."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess(text):\n",
    "    tokens = text.lower().split()\n",
    "    tokens = [token.strip(punctuation) for token in tokens]\n",
    "    return tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Теперь нам нужно собрать все уникальные слова в словарь. Лучше сразу посчитать количество упоминаний, чтобы отсеять самые редкие."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Теперь нам нужно заменить в каждом тексте слова на числа (индексы в словаре). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Создадим для этого специальный словарь с индексами. В этот словарь нам нужно положить дополнительный токен - PAD (чуть ниже станет понятнее)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "word2id = { 'PAD':0}\n",
    "\n",
    "for word in filtered_vocab:\n",
    "    word2id[word] = len(word2id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab = Counter()\n",
    "\n",
    "for text in avito.description:\n",
    "    vocab.update(preprocess(text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "почти 300к слов - это много, учитывая, что это только 10% данных"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "284191"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Возьмем только те, что встретились больше 30 раз."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "filtered_vocab = set()\n",
    "\n",
    "for word in vocab:\n",
    "    if vocab[word] > 30:\n",
    "        filtered_vocab.add(word)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "11к - уже нормальная цифра"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "В нашей задаче нам не понадобится раскодировать индексы, но если бы мы делали языковую модель, нам бы понадобился ещё и обратный словарь."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "id2word = {i:word for word, i in word2id.items()}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Заменяем слова на индексы."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "X = []\n",
    "\n",
    "for text in avito.description:\n",
    "    tokens = preprocess(text)\n",
    "    ids = [word2id.get(token, 1) for token in tokens]\n",
    "    X.append(ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "А теперь про паддинг. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Все тексты в нашем трейне разной длины и это проблема, так как модель ожидает последовательности одной длины. Для этого есть паддинг - т.е. забивание последовательности нулями до определенной длинны (мы не просто так присвоили PAD нулевой индекс - это важно)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Лучше всего сделать паддиг до самого длинного текста, но это зачастую не оптимально или даже невозможно из-за нехватки ресурсов. Поэтому значение паддинга приходится подбирать на глаз. \n",
    "\n",
    "Пояснение: если значение паддинга меньше длины текста, то этот текст обрежется."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_LEN = max(len(x) for x in X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "700 - довольно мого, довайте попробуем поменьше."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "564"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "MAX_LEN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_LEN = 100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = tf.keras.preprocessing.sequence.pad_sequences(X, maxlen=MAX_LEN)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(138715, 100)"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Мы почти готовы к обучению. Осталось достать таргет и разбить выборку на трейн и тест."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = avito.deal_probability.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_valid, y_train, y_valid = train_test_split(X, y, test_size=0.05)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Теперь сама модель. В tf.keras два способа построения моделей: Sequential и Functional. В первом мы как будто создаем список и добавляем в него слои, а во втором вручную применяем каждый слой к результату предыдущего слоя."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sequential"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = tf.keras.Sequential()\n",
    "# первым добавляем эмбединг слой\n",
    "# он каждому слову сопоставит вектор заданой размерности\n",
    "# также нам нужно передать параметры нашей выборки - размер словаря (input_dim) и длинну текста (input_length)\n",
    "model.add(tf.keras.layers.Embedding(input_dim=len(word2id), \n",
    "                                    input_length=MAX_LEN, output_dim=100))\n",
    "\n",
    "# вообще для работы с последовательностями нужно использовать рекурентные или сверточные слои\n",
    "# но с ними мы познакомимся в следующий раз\n",
    "# Пока, чтобы получить один вектор из последовательности векторов,\n",
    "# просто склеим их в один \n",
    "model.add(tf.keras.layers.Flatten())\n",
    "\n",
    "# теперь можно добавить любое количество полносвязных слоев, но не обязательно\n",
    "# model.add(tf.keras.layers.Dense(100, activation='relu'))\n",
    "\n",
    "# теперь последний слой, выход которого и будет предсказанием\n",
    "# тут нужно указать количество классов и функцию активации\n",
    "# в задаче регрессии мы предсказываем одно число == один класс, \n",
    "# а активация в задаче регрессии должна быть linear или sigmoid, если число от 0 до 1 (как у нас в задаче)\n",
    "# model.add(tf.keras.layers.Dense(1, activation='sigmoid'))\n",
    "model.add(tf.keras.layers.Dense(1, activation='linear'))\n",
    "\n",
    "# компилируем модель\n",
    "# выбираем лосс - функция, по которой модель будет оценивать качество и обновлять веса\n",
    "# для регрессии стандартно используется mean_squared_error\n",
    "# выбираем оптимизатор - он влияет на то, как будет обновляться модель \n",
    "# (подробнее вот тут - \n",
    "# https://medium.com/datadriveninvestor/overview-of-different-optimizers-for-neural-networks-e0ed119440c3 ).\n",
    "# По умолчанию можно ставить adam и менять только learning_rate\n",
    "\n",
    "# также можно указать метрики, по которым уже мы будет оценивать качество модели и принимать решения о смене параметров\n",
    "# мы используем метрику соревнования RMSE\n",
    "model.compile(optimizer='adam', loss='mse', metrics=[tf.keras.metrics.RootMeanSquaredError()])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Обучаем"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "1030/1030 [==============================] - 9s 9ms/step - loss: 0.0672 - root_mean_squared_error: 0.2591 - val_loss: 0.0631 - val_root_mean_squared_error: 0.2513\n",
      "Epoch 2/10\n",
      "1030/1030 [==============================] - 9s 8ms/step - loss: 0.0487 - root_mean_squared_error: 0.2207 - val_loss: 0.0674 - val_root_mean_squared_error: 0.2596\n",
      "Epoch 3/10\n",
      "1030/1030 [==============================] - 8s 8ms/step - loss: 0.0378 - root_mean_squared_error: 0.1944 - val_loss: 0.0737 - val_root_mean_squared_error: 0.2714\n",
      "Epoch 4/10\n",
      "1030/1030 [==============================] - 8s 8ms/step - loss: 0.0317 - root_mean_squared_error: 0.1781 - val_loss: 0.0769 - val_root_mean_squared_error: 0.2773\n",
      "Epoch 5/10\n",
      "1030/1030 [==============================] - 8s 8ms/step - loss: 0.0299 - root_mean_squared_error: 0.1729 - val_loss: 0.0791 - val_root_mean_squared_error: 0.2813\n",
      "Epoch 6/10\n",
      "1030/1030 [==============================] - 8s 8ms/step - loss: 0.0278 - root_mean_squared_error: 0.1667 - val_loss: 0.0806 - val_root_mean_squared_error: 0.2839\n",
      "Epoch 7/10\n",
      "1030/1030 [==============================] - 8s 8ms/step - loss: 0.0267 - root_mean_squared_error: 0.1634 - val_loss: 0.0847 - val_root_mean_squared_error: 0.2910\n",
      "Epoch 8/10\n",
      "1030/1030 [==============================] - 8s 8ms/step - loss: 0.0258 - root_mean_squared_error: 0.1607 - val_loss: 0.0859 - val_root_mean_squared_error: 0.2931\n",
      "Epoch 9/10\n",
      "1030/1030 [==============================] - 8s 8ms/step - loss: 0.0245 - root_mean_squared_error: 0.1566 - val_loss: 0.0875 - val_root_mean_squared_error: 0.2959\n",
      "Epoch 10/10\n",
      "1030/1030 [==============================] - 8s 8ms/step - loss: 0.0247 - root_mean_squared_error: 0.1573 - val_loss: 0.0898 - val_root_mean_squared_error: 0.2997\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x1adc58978>"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "model.fit(X_train, y_train, \n",
    "          validation_data=(X_valid, y_valid),\n",
    "          batch_size=128,\n",
    "         epochs=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Обучение можно визуализировать"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['loss', 'root_mean_squared_error', 'val_loss', 'val_root_mean_squared_error'])\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Z1A+gAAAACXBIWXMAAAsTAAALEwEAmpwYAAAzEElEQVR4nO3deXhV5bn+8e+TeU5ICFNCICDK4AAagYrziFpFaysO2NpqPbXa2lp7fvZ0PB7b41GPrW1tKw6ttg5V60CPWkUL1hEBRZFBmSHMhCkMIYQ8vz/WCtkJW0IkOzvZuT/XtS/WXkPy7ADrznrftd7X3B0REZHmkuJdgIiIdEwKCBERiUoBISIiUSkgREQkKgWEiIhEpYAQEZGoFBAirWBmfzKzWw9w36VmdnqsaxKJFQWESByEQVNrZtvMbKOZTTazwRHbrzQzN7NfNjtuXLj+TxHrrjKz+WZWbWZrzewFM8uN8n0aXh+02weVTk0BIRI/t7t7DlACrAQeaLZ9EXCxmaVErPsK8EnDGzM7CfgFcKm75wJDgL9G+z4Rr6Pa+oNIYlJASMIJm3a+b2Yfmtl2M3vAzHqa2Yvhb9mvmFm3iP3PN7M5ZrbZzKaa2ZCIbSPM7L3wuL8CGc2+1+fNbFZ47FtmdmRr63X3ncATwPBmm9YAs4Gzwu9VCBwHTIrY51jgbXd/P/xaG939IXevbm0dIs0pICRRXQScARwKnAe8CPwHUEzw7/7bAGZ2KPAY8J1w2wvA380szczSgGeBPwOFwJPh1yU8dgTwIPBvQBFwLzDJzNJbU6iZZQOXAgujbH4Y+HK4fAnwHLArYvs04Cwz+08zG9Pa7y2yPwoISVS/cfe17r4SeB2Y5u7vu3sN8AwwItxvPPC8u092993AnUAmwW/qo4FU4FfuvtvdnwKmR3yPa4B73X2au+9x94cITt6jD7DGm8xsM1ANHA9cEWWfZ4CTzSyfICgejtzo7q8DXwCOBp4HqszsLjNLbv59Il4PHWB90sUpICRRrY1Y3hnlfU643AdY1rDB3euBFQT9An2Ald50RMtlEcv9gO9FnnyBvuFxB+JOdy8A+oc1HdZ8h7D56XngR0CRu78ZZZ8X3f08gqucccCVwNXNv0/E6ysHWJ90cQoI6epWEZzoATAzIzjJrwRWAyXhugZlEcsrgJ83O/lmuftjrSnA3ZcDNwB3m1lmlF0eBr4H/KWFr1Pv7q8C/wQOb00NItEoIKSrewI418xOM7NUghPxLuAt4G2gDvi2maWa2ReAkRHH3gd8w8xGWSDbzM5tuMW0Ndx9MkFYXRNl82sE/Sm/ab4hvO31EjPrFtYwEjgJeKe1NYg0p4CQLs3dPwYmEJx8NxB0aJ/n7rXuXkvQvn8lsJGgv+LpiGNnAF8HfgtsIuhkvvIgyrkD+PfmHc0eeNXdN0Y5ZlNYwwJgK8FVxh3u/kjEPv/e7DmIDQdRo3QhpgmDREQkGl1BiIhIVAoIERGJSgEhIiJRKSBERCSqlJZ36Ry6d+/u/fv3j3cZIiKdysyZMze4e3G0bQkTEP3792fGjBnxLkNEpFMxs2Wftk1NTCIiEpUCQkREolJAiIhIVAnTBxHN7t27qayspKamJt6lxFxGRgalpaWkpqbGuxQRSRAJHRCVlZXk5ubSv39/mg7ImVjcnaqqKiorKykvL493OSKSIGLaxGRmY83sYzNbaGY3R9n+DTObHU7Z+IaZDY3Y9oPwuI/N7KzP8v1ramooKipK6HAAMDOKioq6xJWSiLSfmAVEOKPVPcDZwFDg0sgACD3q7ke4+3DgduCu8NihBNMrDgPGAr9rNkNWa+r4bB+gk+kqn1NE2k8sm5hGAgvdfTGAmT1OMNvV3IYd3H1rxP7ZQMPQsuOAx919F7DEzBaGX+/tGNYrItI51NfD1krY8AlsWAApGVDx1Tb/NrEMiBKCGbcaVAKjmu9kZtcBNwJpwKkRx0ZOeFIZrmt+7DWEE6yUlZU139whbN68mUcffZRvfvObrTrunHPO4dFHH6WgoCA2hYlIx7d7J1QtCoMg8rUQ6nY27lc6stMFxAFx93uAe8zsMoJ5dw94vlx3nwhMBKioqOiQE1ts3ryZ3/3ud/sERF1dHSkpn/7jf+GFF2Jdmoh0BO6wowrWf9x4RdAQBJuX09iwYlBQBt0Phf4nQvdBUHxY8D6rKCalxTIgVhLM7dugNFz3aR4Hfv8Zj+2wbr75ZhYtWsTw4cNJTU0lIyODbt26MX/+fD755BMuuOACVqxYQU1NDTfccAPXXBPMONkwdMi2bds4++yzOf7443nrrbcoKSnhueeeIzMz2tTFItJh7amDzcuaXQmEYbBzU+N+KZnQ/RAoPRaGXx4EQfdDoWggpLbv//tYBsR0YJCZlROc3C8BLovcwcwGufuC8O25BNMmAkwCHjWzu4A+wCDg3YMp5j//Poe5q7a2vGMrDO2Tx0/PG7bffW677TY++ugjZs2axdSpUzn33HP56KOP9t6O+uCDD1JYWMjOnTs59thjueiiiygqavrbwIIFC3jssce47777uPjii/nb3/7GhAkT2vSziEgb2VUdnvgXNA2CjYtgT23jfjk9gxP/sAuDPxuCIK8UkjrGM8wxCwh3rzOz64GXgGTgQXefY2a3ADPcfRJwvZmdDuwmmFv3K+Gxc8zsCYIO7TrgOnffE6ta29PIkSObPKvw61//mmeeeQaAFStWsGDBgn0Cory8nOHDhwNwzDHHsHTp0vYqV0SiqauFrSvDK4IwCNZ/HCxXr2rcz5KhcEBw4j/0rDAIDg2uEDK7xa/+AxTTPgh3fwF4odm6n0Qs37CfY38O/LytamnpN/32kp2dvXd56tSpvPLKK7z99ttkZWVx8sknR32WIT29cQ775ORkdu7cuc8+ItJG6uth+zrYUtn42rqy6fK2dTT2DQDpecEVwICTwiuBsG+gW39ISYvXJzloce+kTnS5ublUV1dH3bZlyxa6detGVlYW8+fP55133om6n4i0EXeo2QxbwhP+1sqI5ZWwZQVsXQ31u5sel5oN+aWQXwI9h0F+32C5odM4pyck4LNICogYKyoqYsyYMRx++OFkZmbSs2fPvdvGjh3LH/7wB4YMGcJhhx3G6NGj41ipSALYvTM44W9t+O2/2fKWSti9vekxSSmQ1yc46fcdHZz480uDvoCG5YyChAyAlph7h7w7tNUqKiq8+YRB8+bNY8iQIXGqqP11tc8rXUj9HtixEbavhx0bgiaevc0+Db/5rwxuF20upyfklYQn+77hcmkYAiWQ0wOSPtNADQnBzGa6e0W0bbqCEJH2V18f3NrZcMLfvh62bwhee99XNW7fsZEmbf4N0vMbf8svOabpiT+/NLgySEnf9zg5IAoIETl49fVB2/6OqoiT/fpmJ/yIANhRBV4f/WtldoPsYsjqHjwIlj2m8X12w6s4CIGMvHb9mF2NAkJEWlZfD2s+gMWvQfXqxgBoCIQdVVBfF/3YjPzGE3zRQCgbFf2En9U9eCI4WaeljkJ/EyIS3a5qWDQFFrwECybDtrXB+vS84KSe1R0K+kHJ0REn/GLILmp6wu/Et3l2dQoIEWlUtQg+eSkIhaVvBrd7ZuTDwNOCB70OOT0IB+kSFBAiXVldLSx7Exa8HATDxkXB+uLBMPpaOHQs9B2lZp8uSn/rHUxOTg7btm2LdxmSyKrXBoGw4CVYNBVqqyE5HcpPDEJh0JnQrV+8q5QOQAEhkujq62HV+0EgfPISrJ4VrM8rgSO+GDQdlZ8Iadn7/TLS9SggYuzmm2+mb9++XHfddQD87Gc/IyUlhSlTprBp0yZ2797Nrbfeyrhx4+JcqSSUmi1hB/PLwWv7erCkYGKZ034Cg84Khozogk8Hy4HrOgHx4s2wZnbbfs1eR8DZt+13l/Hjx/Od73xnb0A88cQTvPTSS3z7298mLy+PDRs2MHr0aM4//3zNKy2fnXswkmjDVcLyt4PbTjMKgo7lhg7mrMJ4VyqdSNcJiDgZMWIE69atY9WqVaxfv55u3brRq1cvvvvd7/Kvf/2LpKQkVq5cydq1a+nVq1e8y5XOpG4XLH2j8a6jTUuD9T2GwXHfCq4SSo9VB7N8Zl3nX04Lv+nH0pe+9CWeeuop1qxZw/jx43nkkUdYv349M2fOJDU1lf79+0cd5ltkH1tXhXccvQyLpwYDz6VkQPlJjaFQ0LfFLyNyILpOQMTR+PHj+frXv86GDRt47bXXeOKJJ+jRowepqalMmTKFZcuWxbtE6Ujq94SD0a0KJp/ZuiqYm3jJa43NpPl9YfilQSCUn9DuU1FK16CAaAfDhg2jurqakpISevfuzeWXX855553HEUccQUVFBYMHD453idJedtcEQ1VUrw5O/FtXBvMPbF3ZuK56DTSfQDE5DUoq4PSfBaHQY4g6mCXmFBDtZPbsxg7y7t278/bbb0fdT89AdGI1W5v+1t9w4o9cF2046rScYNTRvD5BU1HDcsMrt08wZEUHmadYuo6YBoSZjQXuJpiT+n53v63Z9huBqwnmnV4PfM3dl4XbbgfOBZKAycANniiTV0jnUl8fnNj3nvgjXpFhUBtl5sCsosaTfElF8OxBXu/GdXl9NCKpdFgxCwgzSwbuAc4AKoHpZjbJ3edG7PY+UOHuO8zsWuB2YLyZHQeMAY4M93sDOAmYGqt6Rajd0Tj5/Pr54ST0Hwft/3tqm+5rSZDbO3gVD4aBp4a/8ZcE6/L6BH+mZsTns4i0gVheQYwEFrr7YgAzexwYB+wNCHefErH/O8CEhk1ABpAGGJAKrP0sRbh7l3i+QBdXrVCzNQyC+Y1BsH4+bF7B3klpklKgcCD0GAqHnRNMPpPbu/EKILuHbh+VhBfLf+ElwIqI95XAqP3sfxXwIoC7v21mU4DVBAHxW3ef19oCMjIyqKqqoqioKKFDwt2pqqoiI0O/rTaxY2NEAERcFVSvatwnOR26DwqeFxhxRTABffFgKBygYaqly+sQvwKZ2QSggqAZCTM7BBgClIa7TDazE9z99WbHXQNcA1BWVrbP1y0tLaWyspL169fHsPqOISMjg9LS0pZ3TDTuwS2h6+dHXBWEYbA94u89NSs4+ZefGMxSVjw4+LOgn64ERD5FLP9nrAQin9gpDdc1YWanAz8ETnL3XeHqC4F33H1buM+LwOeAJgHh7hOBiQAVFRX7tLGkpqZSXl5+8J9E4s89uCOoyRVBGAQ1mxv3S88LTvyHnhWGQBgEeaW6C0iklWIZENOBQWZWThAMlwCXRe5gZiOAe4Gx7r4uYtNy4Otm9t8ETUwnAb+KYa3SkdTugMrpwQikkVcFtRG3AGcWBif/YRc2hkDxYMjtpecDRNpIzALC3evM7HrgJYLbXB909zlmdgsww90nAXcAOcCTYR/Bcnc/H3gKOBWYTdBr+A93/3usapU421UNy6cFE9csexNWvhfMZAaQ0ys4+Q+/LKJpaLBmNRNpB5Yod79UVFT4jBkz4l2GHIidm2D5O8FAc8vegtUfBE8OJ6VAnxHQ7zjodzyUVmj0UZEYM7OZ7l4RbZt65yT2tm8Irw7eCuY5XvsR4I3DR5xwYxAKpSMhPSfe1YpISAEhbW/r6sbmoqVvBg+bAaRkQt+RcPIPoP+YIBz0IJlIh6WAkIO3eXkQBA2hsHFxsD4tF8pGwVGXQL8xQfORni0Q6TQUENI67kEANFwdLHsLtiwPtmUUBE1FFVcFf/Y6Us8YiHRi+t8r++ce3GK67I3GPoRta4JtWd2DIDju+uAKocdQPWsgkkAUENKUe9CJvPTNxlBoGKI6tzf0Pz7oP+g3JngyWc8ciCQsBYQE6uth3iR4/c7GWcsKyoLJafodF4RCt3IFgkgXooDo6vbUwUd/g9f/N7jbqHAgfP6XcMgZmttYpItTQHRVdbXw4ePw+l2waUnQf3DRA8HQFUnJ8a5ORDoABURXs3snvPdnePNu2FoJvYfD+EeCOQ/UwSwiERQQXcWubTDzj/DWb2DbWug7Gs67Gw45Tf0KIhKVAiLR7dwM0++Dt38HOzdC+UlBU1L/4xUMIrJfCohEtb0Kpv0ept0Lu7YGdyOdeFMw1IWIyAFQQCSa6rXw9m9g+oOwezsMOT8Iht5HxbsyEelkFBCJYvMKeOvXMPOhYC6Fw78YjJLaY0i8KxORTkoB0dltXAxv/BJmPQY4HHUpHP9dKBoY78pEpJNTQHRW6+bDG3fB7CchKRWOuRLG3KCH20SkzSggOpvVHwbDYcydBKmZMPqbcNy3grmYRUTakAKis1gxPQiGT/4B6XlwwveCcMguindlIpKgYhoQZjYWuBtIBu5399uabb8RuBqoA9YDX3P3ZeG2MuB+oC/gwDnuvjSW9XY47sG8C/+6AxZPhcxucMqPYOTXIbMg3tWJSIKLWUCYWTJwD3AGUAlMN7NJ7j43Yrf3gQp332Fm1wK3A+PDbQ8DP3f3yWaWA9THqtYOxx0WvQr/uhOWvw3ZPeCM/4KKr2nOZhFpN7G8ghgJLHT3xQBm9jgwDtgbEO4+JWL/d4AJ4b5DgRR3nxzuty2GdXYc9fXwyYvBFcOq9yGvBM6+A46+IuhvEBFpR7EMiBJgRcT7SmDUfva/CngxXD4U2GxmTwPlwCvAze6+JxaFxt3W1TDnGXj/z7BuLnTrD+f9OrhlVXM4i0icdIhOajObAFQAJ4WrUoATgBHAcuCvwJXAA82Ouwa4BqCsrKydqm0j26tg3nPw0dOw9A3AgzmcL5wIh1+kuZxFJO5ieRZaSdDB3KA0XNeEmZ0O/BA4yd13hasrgVkRzVPPAqNpFhDuPhGYCFBRUeFtXH/bq9kK858PJuhZPAXq66BoEJx8cxAK3QfFu0IRkb1iGRDTgUFmVk4QDJcAl0XuYGYjgHuBse6+rtmxBWZW7O7rgVOBGTGsNXZqd8CCl2D2U7BgMuzZBfll8Lnr4YgvQs/DNaqqiHRIMQsId68zs+uBlwhuc33Q3eeY2S3ADHefBNwB5ABPWnCSXO7u57v7HjO7CXjVgg0zgftiVWubq6uFRf+Ej56C+S8Eg+bl9ISKrwZjJJVWKBREpMMz947fMnMgKioqfMaMOF5k1O+Bpa8HVwrz/g41m4PnFoacH1wp9BujqTxFpMMxs5nuXhFtm3pCD0Z9PVROD64U5jwL29dBWg4MPje4Uhhwsu5CEpFOSwHRWu6w5sPgSmHOM7BlBaRkwKFnBR3Ng87UMwsikhAUEEDN7j1kpLbQ/LP+k+BK4aO/QdVCSEqBgafBqT+Gw86GjLz2KVZEpJ10+YBYXrWDy+5/hx+eM4Szj+jddOOmZTDnaZj9N1g7GzAoPyEYPXXI+ZBVGJeaRUTaQ5cPiJ756XTPSeemJz9gUM8cDsncHvQnfPRU0L8AUHosjP0fGHaBhtUWkS6jywdEekoyf7ioP/dPvJvqe2/F93yE4dDzCDj9ZzDswmDoCxGRLqbLBwQbF9Nr4rH8qL6Oxd6b5wuv4NxLr8N6DI53ZSIicaWA6FYOp/wHHHI6/1yQw60vzGfF3FSu7RHvwkRE4ksBYRbMzgZc1cuZVbmFO16az5Gl+Yw5pHucixMRiZ+keBfQkZgZ/3PRkRzSI4dvPfY+KzfvjHdJIiJxo4BoJjs9hT9MOIbddfVc+5eZ1OxOzCkoRERaooCIYkBxDv978VF8WLmFn02aE+9yRETiQgHxKc4c1ovrThnI49NX8Ni7y+NdjohIu1NA7MeNZxzGCYO689Pn5jBrxeZ4lyMi0q4UEPuRnGT8+pIRFOem882/zKRq266WDxIRSRAKiBZ0y07j3iuOoWp7Ld967H3q9tTHuyQRkXahgDgAh5fkc+sFh/PWoiruePnjeJcjItIuFBAH6EsVfbl8VBn3vraYF2evjnc5IiIxp4BohZ+cN5ThfQu46ckPWLiuOt7liIjEVEwDwszGmtnHZrbQzG6Osv1GM5trZh+a2atm1q/Z9jwzqzSz38ayzgOVnpLM7yccTWZaMv/255lU1+yOd0kiIjETs4Aws2TgHuBsYChwqZkNbbbb+0CFux8JPAXc3mz7fwH/ilWNn0Xv/Ex+c+nRLK3awfef/BB3j3dJIiIxEcsriJHAQndf7O61wOPAuMgd3H2Ku+8I374DlDZsM7NjgJ7AyzGs8TP53MAibh47mH/MWcMfXlsc73JERGIilgFRAqyIeF8Zrvs0VwEvAphZEvC/wE37+wZmdo2ZzTCzGevXrz/Iclvn6hPKOffI3tzx0nzeXLihXb+3iEh76BCd1GY2AagA7ghXfRN4wd0r93ecu0909wp3ryguLo51mU2YGbdr5FcRSWCxDIiVQN+I96XhuibM7HTgh8D57t7wqPLngOvNbClwJ/BlM7sthrV+Jhr5VUQSWSwDYjowyMzKzSwNuASYFLmDmY0A7iUIh3UN6939cncvc/f+BM1MD7v7PndBdQQa+VVEElXMAsLd64DrgZeAecAT7j7HzG4xs/PD3e4AcoAnzWyWmU36lC/XoWnkVxFJRLa/2zTN7FR3/2e4XO7uSyK2fcHdn26HGg9IRUWFz5gxI27ff0+9c+Uf32Xa4o088Y3PMbxvQdxqERE5UGY2090rom1r6QrizojlvzXb9qODqirBaORXEUk0LQWEfcpytPddnkZ+FZFE0lJA+KcsR3svaORXEUkcKS1sHxB2HFvEMuH78phW1ol9qaIvs1Zs5t7XFjO8tICzj+gd75JERFqtpYCIHBrjzmbbmr+XCD85byhzVm3lpic/YFDPHA7pkRvvkkREWmW/TUzu/lrkC3gL2ArMC9/Lp9DIryLS2e03IMzsD2Y2LFzOBz4AHgbeN7NL26G+Tk0jv4pIZ9ZSJ/UJ7t7wePBXgU/c/QjgGODfY1pZgtDIryLSWbUUELURy2cAzwK4+5pYFZSINPKriHRGLQXEZjP7fDhm0hjgHwBmlgJkxrq4RKGRX0WkM2opIP6NYDylPwLfibhyOA14PpaFJRqN/CoinU1LdzF94u5j3X24u/8pYv1L7v69mFeXYDTyq4h0Jvt9DsLMfr2/7e7+7bYtJ/E1jPx6z5RFHNW3gEtHlsW7JBGRqFp6UO4bwEfAE8AqNP5Sm7jxjMP4sHILP31uDkN652nkVxHpkFrqg+gNTATOAq4AUoHn3P0hd38o1sUlKo38KiKdQUt9EFXu/gd3P4XgOYgCYK6ZXdEexSUyjfwqIh3dAc0oZ2ZHAzcAE4AXgZmxLKqr0MivItKRtdRJfQtwLsGUoY8DPwinEpU2opFfRaSjaqmT+kfAEuCo8PULM4Ogs9rd/cjYltc1RI782q8om6F98uJdkohIi01M5cCpwOfD13nhq2F5v8xsrJl9bGYLzezmKNtvNLO5Zvahmb1qZv3C9cPN7G0zmxNuG9/aD9aZNIz8mp+ZypcfnMai9dviXZKISIud1MuivYAVwPH7O9bMkoF7gLOBocClZja02W7vAxXhlchTwO3h+h3Al919GDAW+JWZFbTys3UqvfMz+fPVo3CHCfdPo3LTjniXJCJdXEvDfeeZ2Q/M7LdmdqYFvgUsBi5u4WuPBBa6+2J3ryXow4icgAh3n+LuDWfCd4DScP0n7r4gXF4FrAOKW/vhOpuBxTn8+apRbN9Vx+X3T2Pd1pp4lyQiXVhLTUx/Bg4DZgNXA1OALwIXuPu4/R0IlBBcaTSoDNd9mqsI7pBqwsxGAmnAoijbrjGzGWY2Y/369S2U0zkM7ZPHH786kvXVu7jigXfZtL225YNERGKgpYAY4O5Xuvu9wKUETUVnufustizCzCYAFcAdzdb3Jgipr7r7Pg8KuPtEd69w94ri4sS5wDimXzfu+3IFS6q2c+Uf39VsdCISFy0FxN4zk7vvASrd/UDbPVYCfSPel4brmjCz04EfAue7+66I9XkEI8b+0N3fOcDvmTDGHNKd3112NB+t2spVD83Q6K8i0u5aCoijzGxr+KoGjmxYNrOtLRw7HRhkZuVmlgZcAkyK3CGcZ+JegnBYF7E+DXgGeNjdn2rth0oUpw/tyV0XH8X0pRu59i8zqa3T09Yi0n5auosp2d3zwleuu6dELO/3Zv3wgbrrgZcIHrR7wt3nmNktZnZ+uNsdQA7wpJnNMrOGALkYOBG4Mlw/y8yGH8Tn7LTGDS/h5xccwZSP1/Pdv85iT73mtRaR9tHSg3IHxd1fAF5otu4nEcunf8pxfwH+EsvaOpPLRpWxfVcdP39hHllpyfzPRUeSlKSBdUUktmIaENJ2vn7iAKp31fHrVxeQnZ7CT88bSvhUu4hITCggOpHvnj6IbTV1PPjmEvIyUrjxzMPiXZKIJDAFRCdiZvz480PYtms3v/7nQnIyUrjmxIHxLktEEpQCopMxM/77C0eyfdcefvHCfHLSU7lslKYtFZG2p4DohJKTjF+OH86O2jp++OxsstOTGTd8fw+pi4i03gFNGCQdT1pKEr+fcAwj+xdy4xMf8PKcNfEuSUQSjAKiE8tITeaBK4/l8JJ8rn/0fd5YsCHeJYlIAlFAdHI56Sk89NVjKe+ezdcfnsHMZZviXZKIJAgFRAIoyErjz1ePpGdeOl/947vMWbUl3iWJSAJQQCSIHrkZ/OXqUWSnp/DlB97VrHQictAUEAmktFsWj1w9CrNgVroVGzUrnYh8dgqIBDOgOIeHvxbMSjfhAc1KJyKfnQIiAQ3tk8efvhbMSjfhgWmalU5EPhMFRII6uqwb93+5gqVVO/iKZqUTkc9AAZHAjgtnpZsTzkq3s1az0onIgVNAJLgms9I9olnpROTAKSC6gHHDS/jFhUcw9eP1fOev71O3RyEhIi3TYH1dxKUjg1npbn1+Hllps7lds9KJSAsUEF3I1ScMYGtNMCtdjmalE5EWxLSJyczGmtnHZrbQzG6Osv1GM5trZh+a2atm1i9i21fMbEH4+kos6+xKvnv6IL42ppw/vbWUuyZ/Eu9yRKQDi9kVhJklA/cAZwCVwHQzm+TucyN2ex+ocPcdZnYtcDsw3swKgZ8CFYADM8NjNRLdQWqYlW77rjp+88+F5KSn8G8naVY6EdlXLK8gRgIL3X2xu9cCjwPjIndw9ynu3jAexDtAabh8FjDZ3TeGoTAZGBvDWrsUM+MXXziCc4/szX+/OJ9Hpi2Ld0ki0gHFsg+iBFgR8b4SGLWf/a8CXtzPsftMmWZm1wDXAJSVadrN1khOMn558XB21u7hR89+RHZaCheM0Kx0ItKoQ9zmamYTCJqT7mjNce4+0d0r3L2iuLg4NsUlsLSUJH53+dGMKi/ke09qVjoRaSqWAbES6BvxvjRc14SZnQ78EDjf3Xe15lg5eBmpydz/Fc1KJyL7imVATAcGmVm5maUBlwCTIncwsxHAvQThsC5i00vAmWbWzcy6AWeG6yQG9p2VbmO8SxKRDiBmAeHudcD1BCf2ecAT7j7HzG4xs/PD3e4AcoAnzWyWmU0Kj90I/BdByEwHbgnXSYxEzkr35Qfe5fdTF1GzW2M3iXRl5u7xrqFNVFRU+IwZM+JdRqe3estOfvTMR7w6fx19CzO5eewQzjmilx6oE0lQZjbT3SuibesQndTScfTOz+SBK4/lL1eNIjsthesefY+L732bD1ZsjndpItLOFBAS1fGDuvP8t0/gFxcewZIN2xl3z5vc+NdZrN6yM96liUg7UUDIp0pOMi4bVcaUm07m2pMH8n+zV3PKnVO5a/In7Kiti3d5IhJjCghpUW5GKv9v7GBevfEkThvSk1+/uoBT7pzKUzMrqa9PjD4sEdmXAkIOWN/CLO657Gj+du3n6JWfyU1PfsD597zBtMVV8S5NRGJAASGtdky/Qp659jh+NX44VdtqGT/xHb7x55ksq9oe79JEpA1pPgj5TJKSjAtGlHDWsF7c9/pifj91Ef+cv44rx/TnulMOIT8zNd4lishB0hWEHJTMtGS+fdogpn7/ZMYN78N9ry/mlDun8ue3l2pqU5FOTgEhbaJnXgZ3fOko/n798QzqkcOPn5vD2Xe/ztSP17V8sIh0SAoIaVOHl+Tz+DWj+cOEY6jdU8+Vf5zOVx58l0/WVse7NBFpJQWEtDkzY+zhvZj83ZP40blDeG/5Js6++3V+9OxsqrbtavkLiEiHoICQmElLSeLqEwbw2vdP4fJRZTz27gpOvnMqE/+1iF11GghQpKNTQEjMFWanccu4w/nHDSdQ0a8bv3hhPmfc9S9enL2aRBksUiQRKSCk3QzqmcsfvzqSh782kozUJK595D3GT3yH2ZVb4l2aiEShgJB2d+Khxbzw7RO49YLDWbRuG+ff8wbfe+ID1mypiXdpIhJBASFxkZKcxITR/Zjy/ZO55oQB/P2DVZxy51R+9con7KxV/4RIR6CAkLjKy0jlB+cM4ZUbT+KUwcX86pVgIMCn39NAgCLxphnlpEN5d8lGbn1+Lh9WbuGQHjmcc3gvzhzWi2F98jSrnUgM7G9GOQWEdDj19c5zH6zksWkrmLFsI/UOffIzOGNoT84c1ouR5YWkJuviV6QtxC0gzGwscDeQDNzv7rc1234i8CvgSOASd38qYtvtwLkEzWCTgRt8P8UqIBJT1bZdvDp/HS/PWcvrC9azq66evIwUTh3cgzOH9eKkQ4vJTteYkyKf1f4CImb/s8wsGbgHOAOoBKab2SR3nxux23LgSuCmZsceB4whCA6AN4CTgKmxqlc6pqKcdC6u6MvFFX3ZUVvH6ws28PKctfxz/lqenbWKtJQkjj+kO2cM7cnpQ3pSnJse75JFEkYsf/UaCSx098UAZvY4MA7YGxDuvjTc1nzYTwcygDTAgFRgbQxrlU4gKy2Fs4b14qxhvajbU8+MZZt4ec5aXp67hn/OX8d/2GyOLusWNEUN7cmA4px4lyzSqcUyIEqAFRHvK4FRB3Kgu79tZlOA1QQB8Vt3n9d8PzO7BrgGoKys7KALls4jJTmJ0QOKGD2giB9/fgjz11Tz8py1TJ63httenM9tL87nkB45e8PiqNICkpLUyS3SGh2y8dbMDgGGAKXhqslmdoK7vx65n7tPBCZC0AfRvlVKR2FmDOmdx5Deedxw+iBWbt7J5DlrmDxvLRP/FUxm1CM3ndPDsPjcwCLSU5LjXbZIhxfLgFgJ9I14XxquOxAXAu+4+zYAM3sR+Bzw+n6PEgFKCjK5ckw5V44pZ/OOWqZ8vI7Jc9fy7PsreXTacnLSUzj5sGLOGNqTUwb3IC9Ds9+JRBPLgJgODDKzcoJguAS47ACPXQ583cz+m6CJ6SSCu51EWqUgK40LR5Ry4YhSanbv4a1FG5g8dy2T567l/z5cTWqyMXpAEWcO7ckZQ3vRKz8j3iWLdBixvs31HIITezLwoLv/3MxuAWa4+yQzOxZ4BugG1ABr3H1YeAfU74ATCTqs/+HuN+7ve+k2V2mNPfXOrBWbeHnuWibPWcviDdsBOKo0f+/zFoN65OjhPEl4elBOpAUL123j5blreHnOWmat2AxA/6Iszhjak5MP68HRZd3ITFO/hSQeBYRIK6zdWsMr89by8py1vLVoA7v3OKnJxpGlBYwsL2RUeSHH9OtGrvouJAEoIEQ+o+qa3cxYtolpizfy7pIqPqzcQl29k2TB/Nsj+xcyakARx/bvRkFWWrzLFWk1BYRIG9lRW8f7yzczbXEV05Zs5P0Vm6mtq8cMDuuZy6jyhsAo1FPd0ikoIERipGb3Hj6s3MK0xVW8u3QjM5ZuYufuYD6LgcXZjCwvYvSAQkaWF9I7PzPO1YrsKy5jMYl0BRmpyYwsDwIAYPeeej5auYVpSzYybXEV//fBKh57dzkAZYVZe/swRg8oorRbpu6Skg5NVxAiMbSn3pm3eivTlgR9GO8u2cimHbsB6J2fwajyQkaWFzGyvJCBxdkKDGl3amIS6SDq652F67cxbXEV7yzZyLtLNrK+ehcA3XPSwiuMIkYNKOTQHrkaP0piTgEh0kG5O0s2bOfdJRv3Nkut2lIDQEFWKsf2LwyvMgoZ3CuPtBRNlCRtS30QIh2UmTGgOIcBxTlcMjIYkXjFxh1hYARNUpPnBiPdpyYbA4tzOKxXLof1ymVwr1wO65VHn/wMNU1JTCggRDqYvoVZ9C3M4qJjgsGM12ypYfrSjcxdvZWP11QzY+kmnpu1au/+uRkpHNazMTQG987j0J655GfqQT45OGpiEumEtuzczSdrq5m/ppqP1wTBMX9NNdU1dXv36ZOfEV5t5IVXG7kMLM5RM5U0oSYmkQSTnxn0Txzbv3DvOndn9ZaavWHx8ZqtzF9TzRsLg+FCAFKSjAHF2Y2h0TOXwb1zKSnQLbeyLwWESIIwM/oUZNKnIJNTBvfYu373nnqWbNjOvLCJ6uM11by3bBN//yCimSo9hUMj+zZ65jK4Vx75WWqm6soUECIJLjU5iUN75nJoz9wm66trIpupgj+f/3A1j05bvnefXnkZER3iwau8ezZZaTp1dAX6WxbponIzUjmmXyHH9GvaTLV26y7mr2m82pi3ppq3F1VRu6d+737dc9LpV5RFv7BDvV9R8CorzKZ7TpqaqxKEAkJE9jIzeuVn0Cs/g5MPa9pMtXTDdj5eW82yqh0sr9rBso3beWdxFc/MWknkvS5ZacmUFWZRFgZHWVE2/cL3Jd0ySU1WJ3lnoYAQkRalJicxqGcug5o1UwHsqttD5aadQWhUbWfZxh2s2LiDJRu289on69lV13jlkZxk9CnIoF9hNmVFYYgUZlFWlEW/omxy0nVK6kj0tyEiByU9JZmBxTkMLM7ZZ1t9vbOuehfLqrazfOMOlm/cwbKqHSzbuIMXZ6/eOy5Vg6LstMYmq8Lw6iMMkh656Wq6amcKCBGJmaSkxiarUQOK9tm+tWY3y6sag2P5xu0sq9rBzPAuq/qIpquM1KSw6SqbssIs+hRkUBLetdW7IIPu2ekau6qNxTQgzGwscDeQDNzv7rc1234i8CvgSOASd38qYlsZcD/QF3DgHHdfGst6RaR95WWkcnhJPoeX5O+zrbaunpWbdzZefYRXHsurdvDmwg17591okJacRO+CDHrnZ9CnIJOSgkx652fuDZLeBZlqwmqlmP20zCwZuAc4A6gEppvZJHefG7HbcuBK4KYoX+Jh4OfuPtnMcoD6KPuISIJKS0mivHs25d2z99nm7mzZuZuVm3eyanMNq7fsbFzevJN3FlWxZmtNkysQgLyMlL3PivQpCIKkT37j+555GepEjxDLOB0JLHT3xQBm9jgwDtgbEA1XBGbW5ORvZkOBFHefHO63LYZ1ikgnY2YUZKVRkJXGsD77Xn0A1O2pZ131LlZtDsJj9ZYaVm3eGb5qeG/5JjY36wMxg565GfQuiLwKabpcmN11buONZUCUACsi3lcCow7w2EOBzWb2NFAOvALc7O5NrinN7BrgGoCysrKDLlhEEkdKctLeq4WoAw0RzDG+anNEcESEyNxVW5k8dy21dU0bL9JTksImqwx65maQk5FCTnoK2ekp5IbLe18ZTZczU5M7Vbh01Aa5FOAEYARBM9RfCZqiHojcyd0nAhMhGKyvfUsUkc4uKy2FQ3rkcEiPfe/AgqApq2p7Las314RNWDtZvSW4Alm1ZSfTlmxke20d22rqqGvenhVFkhEESRgY2WF45EaGTJRt2WnButz01HBbMukpyW3949hHLANiJUEHc4PScN2BqARmRTRPPQuMpllAiIjEkpnRPSed7jnpHFEavSkLgiDZVVfPtl1BWGzbVddkuXpXHds/bVtNHau31Ox9v722jgMZZDstOWlvWBxVWsBvLzu6DT95IJYBMR0YZGblBMFwCXBZK44tMLNid18PnApoLG8R6ZDMjIzUZDJSk+mek35QX6u+3tmxe0/UMAmWd4fLe9i2azfbauroU5DZRp+kqZgFhLvXmdn1wEsEt7k+6O5zzOwWYIa7TzKzY4FngG7AeWb2n+4+zN33mNlNwKsWNNjNBO6LVa0iIh1FUpLt7beIN00YJCLShe1vwiDd8CsiIlEpIEREJCoFhIiIRKWAEBGRqBQQIiISlQJCRESiUkCIiEhUCfMchJmtB5YdxJfoDmxoo3I6O/0smtLPoyn9PBolws+in7sXR9uQMAFxsMxsxqc9LNLV6GfRlH4eTenn0SjRfxZqYhIRkagUECIiEpUCotHEeBfQgehn0ZR+Hk3p59EooX8W6oMQEZGodAUhIiJRKSBERCSqLh8QZjbWzD42s4VmdnO864knM+trZlPMbK6ZzTGzG+JdU7yZWbKZvW9m/xfvWuLNzArM7Ckzm29m88zsc/GuKZ7M7Lvh/5OPzOwxM8uId01trUsHhJklA/cAZwNDgUvNbGh8q4qrOuB77j6UYA7w67r4zwPgBmBevIvoIO4G/uHug4Gj6MI/FzMrAb4NVLj74QSzZl4S36raXpcOCGAksNDdF7t7LfA4MC7ONcWNu6929/fC5WqCE0BJfKuKHzMrBc4F7o93LfFmZvnAicADAO5e6+6b41pU/KUAmWaWAmQBq+JcT5vr6gFRAqyIeF9JFz4hRjKz/sAIYFqcS4mnXwH/DtTHuY6OoBxYD/wxbHK738yy411UvLj7SuBOYDmwGtji7i/Ht6q219UDQqIwsxzgb8B33H1rvOuJBzP7PLDO3WfGu5YOIgU4Gvi9u48AtgNdts/OzLoRtDaUA32AbDObEN+q2l5XD4iVQN+I96Xhui7LzFIJwuERd3863vXE0RjgfDNbStD0eKqZ/SW+JcVVJVDp7g1XlE8RBEZXdTqwxN3Xu/tu4GnguDjX1Oa6ekBMBwaZWbmZpRF0Mk2Kc01xY2ZG0MY8z93vinc98eTuP3D3UnfvT/Dv4p/unnC/IR4od18DrDCzw8JVpwFz41hSvC0HRptZVvj/5jQSsNM+Jd4FxJO715nZ9cBLBHchPOjuc+JcVjyNAa4AZpvZrHDdf7j7C/ErSTqQbwGPhL9MLQa+Gud64sbdp5nZU8B7BHf/vU8CDruhoTZERCSqrt7EJCIin0IBISIiUSkgREQkKgWEiIhEpYAQEZGoFBAiHYCZnawRY6WjUUCIiEhUCgiRVjCzCWb2rpnNMrN7w/kitpnZL8O5AV41s+Jw3+Fm9o6ZfWhmz4Tj92Bmh5jZK2b2gZm9Z2YDwy+fEzHfwiPhE7oicaOAEDlAZjYEGA+McffhwB7gciAbmOHuw4DXgJ+GhzwM/D93PxKYHbH+EeAedz+KYPye1eH6EcB3COYmGUDwZLtI3HTpoTZEWuk04BhgevjLfSawjmA48L+G+/wFeDqcP6HA3V8L1z8EPGlmuUCJuz8D4O41AOHXe9fdK8P3s4D+wBsx/1Qin0IBIXLgDHjI3X/QZKXZj5vt91nHr9kVsbwH/f+UOFMTk8iBexX4opn1ADCzQjPrR/D/6IvhPpcBb7j7FmCTmZ0Qrr8CeC2cqa/SzC4Iv0a6mWW154cQOVD6DUXkALn7XDP7EfCymSUBu4HrCCbPGRluW0fQTwHwFeAPYQBEjn56BXCvmd0Sfo0vtePHEDlgGs1V5CCZ2TZ3z4l3HSJtTU1MIiISla4gREQkKl1BiIhIVAoIERGJSgEhIiJRKSBERCQqBYSIiET1/wGAPZUcTtqIpAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(model.history.history.keys())\n",
    "# summarize history for accuracy\n",
    "plt.plot(model.history.history['root_mean_squared_error'])\n",
    "plt.plot(model.history.history['val_root_mean_squared_error'])\n",
    "plt.title('model RMSE')\n",
    "plt.ylabel('RMSE')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'val'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Это график плохой модели. Качество на валидационной выборке никак не растет, а это значит мы ничему хорошему не обучаемся. Но не всегда это проблема самой модели. В самих данных может быть недостаточно информации для предсказания (в этом конкретном случае так и есть, это сложно сореванование и текст описания в итоговых моделях не был основным признаком)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Теперь почти тоже самое только в функциональном виде.\n",
    "Вместо склеивания векторов в один попробуем просто их усреднять."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "# input можно не задавать и указывать параметры в эмбединг слое,\n",
    "# но это может пригодиться, когда мы хотим сделать больше входных слоев\n",
    "inputs = tf.keras.layers.Input(shape=(100,))\n",
    "\n",
    "\n",
    "embeddings = tf.keras.layers.Embedding(input_dim=len(word2id), output_dim=10)(inputs)\n",
    "mean = tf.keras.layers.Lambda(lambda x: tf.keras.backend.mean(x,  axis=1))(embeddings)\n",
    "dense = tf.keras.layers.Dense(64, activation='relu')(mean)\n",
    "dense = tf.keras.layers.Dense(64, activation='relu')(mean)\n",
    "outputs = tf.keras.layers.Dense(1, activation='sigmoid')(dense)\n",
    "\n",
    "model = tf.keras.Model(inputs=inputs, outputs=outputs)\n",
    "optimizer = tf.keras.optimizers.Adam(learning_rate=0.0001)\n",
    "model.compile(optimizer=optimizer,\n",
    "              loss='mse',\n",
    "              metrics=[tf.keras.metrics.RootMeanSquaredError()])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "258/258 [==============================] - 1s 2ms/step - loss: 0.1872 - root_mean_squared_error: 0.4326 - val_loss: 0.1521 - val_root_mean_squared_error: 0.3900\n",
      "Epoch 2/10\n",
      "258/258 [==============================] - 0s 2ms/step - loss: 0.1391 - root_mean_squared_error: 0.3728 - val_loss: 0.1010 - val_root_mean_squared_error: 0.3179\n",
      "Epoch 3/10\n",
      "258/258 [==============================] - 0s 2ms/step - loss: 0.0925 - root_mean_squared_error: 0.3040 - val_loss: 0.0756 - val_root_mean_squared_error: 0.2749\n",
      "Epoch 4/10\n",
      "258/258 [==============================] - 0s 2ms/step - loss: 0.0734 - root_mean_squared_error: 0.2710 - val_loss: 0.0702 - val_root_mean_squared_error: 0.2650\n",
      "Epoch 5/10\n",
      "258/258 [==============================] - 1s 3ms/step - loss: 0.0699 - root_mean_squared_error: 0.2644 - val_loss: 0.0694 - val_root_mean_squared_error: 0.2635\n",
      "Epoch 6/10\n",
      "258/258 [==============================] - 1s 2ms/step - loss: 0.0696 - root_mean_squared_error: 0.2638 - val_loss: 0.0692 - val_root_mean_squared_error: 0.2630\n",
      "Epoch 7/10\n",
      "258/258 [==============================] - 1s 2ms/step - loss: 0.0697 - root_mean_squared_error: 0.2640 - val_loss: 0.0690 - val_root_mean_squared_error: 0.2627\n",
      "Epoch 8/10\n",
      "258/258 [==============================] - 0s 2ms/step - loss: 0.0685 - root_mean_squared_error: 0.2618 - val_loss: 0.0689 - val_root_mean_squared_error: 0.2624\n",
      "Epoch 9/10\n",
      "258/258 [==============================] - 0s 2ms/step - loss: 0.0689 - root_mean_squared_error: 0.2626 - val_loss: 0.0687 - val_root_mean_squared_error: 0.2621\n",
      "Epoch 10/10\n",
      "258/258 [==============================] - 0s 2ms/step - loss: 0.0689 - root_mean_squared_error: 0.2625 - val_loss: 0.0686 - val_root_mean_squared_error: 0.2618\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x18d6c2748>"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train, y_train, \n",
    "          validation_data=(X_valid, y_valid),\n",
    "          batch_size=512,\n",
    "         epochs=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['loss', 'root_mean_squared_error', 'val_loss', 'val_root_mean_squared_error'])\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Z1A+gAAAACXBIWXMAAAsTAAALEwEAmpwYAAA0QUlEQVR4nO3dd3hc1Zn48e87MypWcZPlKtuSwb3JWG4YFkI1kBiH5kInCZCFBRLY4CT8sgmbZEMJ2bDrBEhCcTAYYyBrqhPAhgC2sdxxwwUXucq9S5qZ9/fHvZLHYmxJo7keafR+nmce3XvuOWdezWPrnVvOOaKqGGOMMdX5Eh2AMcaYhskShDHGmKgsQRhjjInKEoQxxpioLEEYY4yJyhKEMcaYqCxBGFMHIvK8iPyylnU3iMhFXsdkjFcsQRiTAG6iKReRQyKyR0T+ISK9Io7fIiIqIr+r1u5Kt/z5iLLviMgqETkoIjtE5B0RyY7yPpWvJaftFzWNmiUIYxLnUVXNAjoBW4C/VDu+DrhORAIRZTcDX1buiMh5wK+B8aqaDfQGXon2PhGvgfH+RUxysgRhko57aeffRWSpiBwWkb+ISDsRedf9lv2+iLSKqD9aRJaLyD4RmS0ivSOODRKRhW67V4D0au/1TRFZ7Lb9TEQG1DVeVT0KTAMKqx3aDiwDLnXfqzVwNjAjos4QYI6qLnL72qOqL6jqwbrGYUx1liBMsroauBjoAXwLeBf4CZCL8+/+HgAR6QG8DNznHnsHeFNEUkUkFfgb8FegNfCq2y9u20HAs8AdQA7wNDBDRNLqEqiIZALjgbVRDk8GbnK3xwH/B5RFHJ8HXCoivxCRkXV9b2NOxRKESVb/o6o7VHUL8E9gnqouUtVjwBvAILfeWOBtVf2HqlYAjwPNcL6pDwdSgP9W1QpVnQ7Mj3iP24GnVXWeqoZU9QWcP97DaxnjAyKyDzgInAPcGKXOG8D5ItICJ1FMjjyoqv8ErgLOAt4GdovIEyLir/4+Ea8XahmfaeIsQZhktSNi+2iU/Sx3uyOwsfKAqoaBzTj3BToCW/TEGS03Rmx3Be6P/OMLdHbb1cbjqtoSyHdj6lm9gnv56W3gISBHVT+NUuddVf0WzlnOlcAtwHerv0/E6+ZaxmeaOEsQpqnbivOHHgAREZw/8luAbUAnt6xSl4jtzcCvqv3xzVDVl+sSgKpuAu4Ffi8izaJUmQzcD7xYQz9hVf0A+BDoV5cYjInGEoRp6qYBV4jIhSKSgvOHuAz4DJgDBIF7RCRFRK4Chka0/RNwp4gME0emiFxR+YhpXajqP3CS1e1RDn+Ecz/lf6ofcB97HScirdwYhgLnAXPrGoMx1VmCME2aqq4GbsD547sL54b2t1S1XFXLca7v3wLswblf8XpE22Lge8D/AntxbjLfUo9wHgN+VP1Gszo+UNU9UdrsdWNYAxzAOct4TFWnRNT5UbVxELvqEaNpQsQWDDLGGBONnUEYY4yJyhKEMcaYqCxBGGOMicoShDHGmKgCNVdpHNq0aaP5+fmJDsMYYxqVBQsW7FLV3GjHkiZB5OfnU1xcnOgwjDGmURGRjSc7ZpeYjDHGRGUJwhhjTFSWIIwxxkSVNPcgoqmoqKCkpIRjx44lOhTPpaenk5eXR0pKSqJDMcYkiaROECUlJWRnZ5Ofn8+JE3ImF1Vl9+7dlJSUUFBQkOhwjDFJwtNLTCIySkRWi8haEZl4inpXuwuxF7n7F4vIAhFZ5v68IJb3P3bsGDk5OUmdHABEhJycnCZxpmSMOX08O4NwV7SahDNNcQkwX0RmqOqKavWycebCnxdRvAtnRs2tItIPmImzgEssccTSrNFpKr+nMeb08fIMYiiwVlXXu9MmT8VZ7aq6/wQeAaq+/rpLQ251d5cDzbxaazcYCrP9wDGOVYS86N4YYxotLxNEJ5wVtyqVUO0sQETOAjqr6tun6OdqYKGqllU/ICK3i0ixiBSXlpbGHGjpwTJ2H/pa93Gxb98+/vCHP9S53eWXX86+ffviH5AxxtRSwh5zFREf8ATOCl4nq9MX5+zijmjHVfUZVS1S1aLc3KgjxWsU8Pto2SyFvUcqCIXDMfVxKidLEMFg8JTt3nnnHVq2bBn3eIwxpra8TBBbcNb2rZTnllXKxlk3d7aIbACGAzMiblTnAW8AN6nqOg/jJCcrlbAqe49UxL3viRMnsm7dOgoLCxkyZAjnnnsuo0ePpk+fPgCMGTOGwYMH07dvX5555pmqdvn5+ezatYsNGzbQu3dvvve979G3b18uueQSjh49Gvc4jTGmOi8fc50PdBeRApzEMA6YUHlQVfcDbSr3RWQ28ICqFotIS+BtYKKqfhqPYH7x5nJWbD1w0uNHK0Kg0CzVX+s++3Rszn98q+8p6/zmN7/hiy++YPHixcyePZsrrriCL774oupx1GeffZbWrVtz9OhRhgwZwtVXX01OTs4JfaxZs4aXX36ZP/3pT1x33XW89tpr3HDDDbWO0xhjYuHZGYSqBoG7cZ5AWglMU9XlIvKwiIyuofndwJnAz0Rksftq61WsACl+H2FVQmFvl2AdOnToCWMVnnzySQYOHMjw4cPZvHkza9as+VqbgoICCgsLARg8eDAbNmzwNEZjjAGPB8qp6jvAO9XKfnaSuudHbP8S+GU8Y6npm344rKzafoDMtABdczLj+dYnyMw83vfs2bN5//33mTNnDhkZGZx//vlRxzKkpR1/gMvv99slJmPMaWFzMbl8PqFVZioHjgYpD8bvZnV2djYHDx6Memz//v20atWKjIwMVq1axdy5c+P2vsYYU19JPdVGXeVkplJ6sIw9h8tp3yI9Pn3m5DBy5Ej69etHs2bNaNeuXdWxUaNG8dRTT9G7d2969uzJ8OHD4/KexhgTD6Lq7TX306WoqEirLxi0cuVKevfuXad+Nuw6zJHyEL06ZONrZKOTY/l9jTFNm4gsUNWiaMfsElM1rbNSCYbDHDga/0dejTGmMbEEUU12WoDUgI/dh8oTHYoxxiSUJYhqRISczDQOlwc5Wm7zMxljmi5LEFG0ykjBJ8Luw97Mz2SMMY2BJYgoKudn2nekgqAH8zMZY0xjYAniJCrnZ9rnwfxMxhjTGFiCOIlmqQEyUgPsPlTO6XwUOCsr67S9lzHGnIoliFPIyUqlLBjiUNmpp+Y2xphkZCOpT6FFegrbfM4jr9npKTH1MXHiRDp37sxdd90FwM9//nMCgQCzZs1i7969VFRU8Mtf/pIrr4y22J4xxiRO00kQ706E7cvq1MQHnBEKURFUwqn+r4+sbt8fLvvNKfsYO3Ys9913X1WCmDZtGjNnzuSee+6hefPm7Nq1i+HDhzN69GhbV9oY06A0nQQRoxSfjwpCBMNhUv21Xyui0qBBg9i5cydbt26ltLSUVq1a0b59e37wgx/w8ccf4/P52LJlCzt27KB9+/Ye/AbGGBObppMgavimfzI+YFc952e69tprmT59Otu3b2fs2LFMmTKF0tJSFixYQEpKCvn5+VGn+TbGmESym9S1kOPOz7Q/xvmZxo4dy9SpU5k+fTrXXnst+/fvp23btqSkpDBr1iw2btwY54iNMab+PE0QIjJKRFaLyFoRmXiKeleLiFauR+2W/dhtt1pELvUyzppkpQVIC/hjnp+pb9++HDx4kE6dOtGhQweuv/56iouL6d+/P5MnT6ZXr15xjtgYY+rPs0tMIuIHJgEXAyXAfBGZoaorqtXLBu4F5kWU9cFZw7ov0BF4X0R6qGpCJkcSEVpnprJt/1GOlgdpllr3j23ZsuM3yNu0acOcOXOi1jt06FDMcRpjTDx5eQYxFFirqutVtRyYCkR7lvM/gUeAyIvwVwJTVbVMVb8C1rr9Jczx+ZlslldjTNPgZYLoBGyO2C9xy6qIyFlAZ1V9u65t3fa3i0ixiBSXlpbGJ+qTCPh9tMxw52cK2fxMxpjkl7Cb1CLiA54A7o+1D1V9RlWLVLUoNzf3ZHVi7f5rcjKd+Zn2NsD5mZJlZUBjTMPhZYLYAnSO2M9zyyplA/2A2SKyARgOzHBvVNfUtlbS09PZvXt33P54Vs7PtOdwWYP6g6yq7N69m/T0+KyjbYwx4O04iPlAdxEpwPnjPg6YUHlQVfcDbSr3RWQ28ICqFovIUeAlEXkC5yZ1d+DzugaQl5dHSUkJ8bz8dKQ8yJ7DFRzZmUp6St0HznklPT2dvLy8RIdhjEkiniUIVQ2KyN3ATMAPPKuqy0XkYaBYVWecou1yEZkGrACCwF2xPMGUkpJCQUFBjL9BdGXBECN/8yGFnVvy55uHxLVvY4xpSDwdSa2q7wDvVCv72Unqnl9t/1fArzwLLkZpAT/jhnRh0uy1bN5zhM6tMxIdkjHGeMJGUsdgwrAuCPDS55sSHYoxxnjGEkQMOrZsxkW92/HK/M0cq0jI2D1jjPGcJYgY3TQinz2Hy3ln2bZEh2KMMZ6wBBGjkWfm0C03k8lzbKI9Y0xysgQRIxHhxuFdWbx5H8tK9ic6HGOMiTtLEABH9kD54To3u3pwHhmpfibP2RD/mIwxJsEsQexZD7/tCcum17lp8/QUxgzqxIwlW9lrk/gZY5KMJYhWBdC6GyyeElPzm0Z0pSwYZvqCkjgHZowxiWUJQgQKJ8DmebBrbZ2b92rfnKH5rXlx3kbC4YYzP5MxxtSXJQiAAWNBfLDkpZia3zCiKxt3H+GjNd5OOW6MMaeTJQiA7PZw5kWwZCqE6z7wbVTf9rTJSuOv9sirMSaJWIKoVDgBDmyBrz6qc9PUgI8JQzsza/VONu854kFwxhhz+lmCqNTjMkhvCYtju8w0YVhXfCK8ONfOIowxycESRKWUdOh/Dax8E47VfeBb+xbpXNKnHa8U2/xMxpjkYAkiUuEECB6D5W/E1PzGEV3Zd6SCN5dsjXNgxhhz+lmCiNTxLMjtFfNlphHdcjizbZZdZjLGJAVPE4SIjBKR1SKyVkQmRjl+p4gsE5HFIvKJiPRxy1NE5AX32EoR+bGXcUYEBIXXxzwmonJ+piUl+1myeV/84zPGmNPIswQhIn5gEnAZ0AcYX5kAIrykqv1VtRB4FHjCLb8WSFPV/sBg4A4Ryfcq1hMMuA7EH/OYiKvO6kRmqt9meTXGNHpenkEMBdaq6npVLQemAldGVlDVAxG7mUDlUGQFMkUkADQDyoHIut6p55iI7PQUvn1WJ95cupU9Nj+TMaYR8zJBdAI2R+yXuGUnEJG7RGQdzhnEPW7xdOAwsA3YBDyuqnuitL1dRIpFpLi0NI6jmOsxJgKcxYTKg2GmFW+uubIxxjRQCb9JraqTVPUM4EHgIbd4KBACOgIFwP0i0i1K22dUtUhVi3Jzc+MXVE93TMSi2Cbw69Eum2EFrXlx7kZCNj+TMaaR8jJBbAE6R+znuWUnMxUY425PAN5T1QpV3Ql8ChR5EWRUgTTofy2seguO7oupi5tG5FOy9yizV++Mb2zGGHOaeJkg5gPdRaRARFKBccCMyAoi0j1i9wpgjbu9CbjArZMJDAdWeRjr19VzTMQlfdvRrnma3aw2xjRaniUIVQ0CdwMzgZXANFVdLiIPi8hot9rdIrJcRBYDPwRudssnAVkishwn0Tynqku9ijWqjoMgt3fMYyJS/D7GD+3CR1+WsnF33VerM8aYRBPV5LhGXlRUpMXFxfHt9NMn4R//D+4uhjbda65fzY4Dxxj5mw+5dWQ+P72i+hO+xhiTeCKyQFWjXsJP+E3qBq1yTESMZxHtmqdzad/2TCsu4Wi5zc9kjGlcLEGcSj3HRIAzP9P+ozY/kzGm8bEEUZPCCXBwK6yfHVPzYQWt6dEui8lzN5Asl/OMMU2DJYia9LwMmrWK+TKTiHDjiHy+2HKARTY/kzGmEbEEUZM4jIn49qBOZKUFbElSY0yjYgmiNuo5JiIrLcDVZ3Xi7aXb2HWoLM7BGWOMNyxB1EaHQmjbJ+bLTODcrC4P2fxMxpjGwxJEbYg4ZxEln0PplzF1cWbbbEZ0y2HK3E02P5MxplGwBFFb/eu3TgTATSO6smXfUT5cZfMzGWMaPksQtZXdDrpfXK8xERf3aUf75ulMnrMhvrEZY4wHLEHUReEEOLgN1s+KqXnA72PCsC78c80u1pceinNwxhgTX5Yg6qLHqHqNiQAYN7QzKX7hxbmb4hiYMcbEnyWIuqgcE7Ey9jERbbPTGdWvA68u2MyR8mB84zPGmDiyBFFXhRMgVAbLX4+5i5tGdOXgsSAzFtv8TMaYhssSRF3FYUxEUddW9GqfzeQ5G21+JmNMg2UJoq6qxkTMj3lMhDM/U1dWbDvAwk174xygMcbEh6cJQkRGichqEVkrIhOjHL9TRJaJyGIR+URE+kQcGyAic9wV55aJSLqXsdbJgLH1HhMxprAT2WkBW5LUGNNgeZYgRMSPs3ToZUAfYHxkAnC9pKr9VbUQeBR4wm0bAF4E7lTVvsD5QIVXsdZZVlvofkm9xkRkpgW4enAe7yzbRulBm5/JGNPweHkGMRRYq6rrVbUcmApcGVlBVQ9E7GYClRfkLwGWquoSt95uVW1YS7LVc0wEOPMzVYSUV+bbI6/GmIbHywTRCYicma7ELTuBiNwlIutwziDucYt7ACoiM0VkoYj8KNobiMjtIlIsIsWlpaVxDr8GPUZBs9b1ull9Rm4W55zZhinzNhEMheMYnDHG1F/Cb1Kr6iRVPQN4EHjILQ4A5wDXuz+/LSIXRmn7jKoWqWpRbm7uaYvZiTA1YkxE7DeabxzRlW37j/H+SpufyRjTsHiZILYAnSP289yyk5kKjHG3S4CPVXWXqh4B3gHO8iLIeqkcE/FF7GMiLuzVlo4t0nlxrt2sNsY0LF4miPlAdxEpEJFUYBwwI7KCiHSP2L0CWONuzwT6i0iGe8P6PGCFh7HGpsNAaNu3XpeZKudn+mTtLtbZ/EzGmAbEswShqkHgbpw/9iuBaaq6XEQeFpHRbrW73cdYFwM/BG522+7FeaJpPrAYWKiqb3sVa8wqx0RsKYbS1TF3M3ZIF1L8YkuSGmMaFEmWkbxFRUVaXFx8+t/40E74bS84+9/g4l/E3M29Uxfx4cqdzP3JhWSmBeIYoDHGnJyILFDVomjHEn6TutGrHBOx9JWYx0QA3DQin4NlQV5fdKrbNMYYc/pYgoiHyjER62IfE3FWl5YMzGvBc59+RdiWJDXGNACWIOKhakzElJi7EBFuHVnA+tLDfLzmNI/pMMaYKCxBxEMgFQZcB6verteYiMv7d6BtdhrPfrohfrEZY0yMLEHESxzGRKQGfNw4vCsff1nK2p0H4xicMcbUnSWIeGk/ANr1q9eYCIAJw7qQGvDxnJ1FGGMSzBJEvMRpTEROVhpjCjvy+sIt7DtSHscAjTGmbk6ZIETkgojtgmrHrvIqqEar/3XgC9T7LOLWkQUcrQgxdf7mmisbY4xHajqDeDxi+7Vqxx7CnCgr9/g6EaFgzN307tCcEd1ymPzZBpvl1RiTMDUlCDnJdrR9A85lpkPb67VOBMBt5xSwdf8xZi7fEafAjDGmbmpKEHqS7Wj7BqD7pfUeEwFwQa+2dGmdwXOffhWnwIwxpm5qShDdRGSGiLwZsV25X1BD26YpTmMi/D7hlrPzKd64l6Ul++IXnzHG1FJNCeJK4Lc49yIqtyv3x3gaWWNWOAFC5fBF9ds2dXNtUR5ZaQF75NUYkxCnTBCq+lHkC/gMOACsdPdNNHEaE5GdnsK1RXm8tXQrOw8ci1NwxhhTOzU95vqUiPR1t1sAS4DJwCIRGX8a4mucqsZELICdq+rV1S1n5xMMq604Z4w57Wq6xHSuqi53t28FvlTV/sBg4EeeRtbYVY6JWFK/s4iuOZlc2KsdU+Zt4lhF7NOJG2NMXdWUICKH8l4M/A1AVbfXpnMRGSUiq0VkrYhMjHL8ThFZJiKLReQTEelT7XgXETkkIg/U5v0alKxc54mmJa/Ua0wEwG0j89l9uJwZS7bGKThjjKlZTQlin4h8U0QGASOB9wDcdaKbnaqhiPiBScBlQB9gfPUEALykqv1VtRB4FGeZ0UhPAO/W5hdpkOI0JmLEGTn0ap/Ns598RbKsAGiMafhqShB34Kwr/RxwX8SZw4VATWtEDwXWqup6VS0HpuI8CVVFVQ9E7GYSMbZCRMYAXwHLaay6XwIZOfUeE+GsFZHPqu0HmbN+d5yCM8aYU6vpKaYvVXWUqhaq6vMR5TNV9f4a+u4ERE4mVOKWnUBE7hKRdThnEPe4ZVnAg8ApF3kWkdtFpFhEiktLG+AiO4FU515EPcdEAFxZ2InWman2yKsx5rQJnOqgiDx5quOqek99A1DVScAkEZmAM7/TzcDPgd+p6iGRk8/ooarPAM8AFBUVNcxrL4UTYN4fYdl0GPq9mLtJT/EzYWgXJs1ey8bdh+makxnHII0x5utqusR0J3AOsBUoBhZUe53KFqBzxH6eW3YyUzk++G4Y8KiIbADuA34iInfX8H4NU4cB0K5/vcdEANw4oit+EV74zB55NcZ4r6YE0QHnG/qlwI1ACvB/qvqCqr5QQ9v5QHcRKRCRVGAcMCOygoh0j9i9AlgDoKrnqmq+quYD/w38WlX/t3a/UgNUOAG2LoSdK+vVTbvm6VwxoAPTijdz8FhFnIIzxpjoaroHsVtVn1LVb+CMg2gJrBCRG2vqWFWDODe4ZwIrgWmqulxEHhaR0W61u0VkuYgsBn6Ic3kp+fS/Ni7rRICzVsShsiDTF5TEITBjjDk5qc1jkyJyFjAeZyzEAuC3qrrC49jqpKioSIuLixMdxsm97K4294MV4D/lrZ8aXfWHT9l9uJwP7z8fv89mXTfGxE5EFqhqUbRjNU218bCILMD5dv8RUKSq32loyaFRKJwAh3bAug/r3dVt5xSwcfcRZq3aGYfAjDEmupruQTyEc1lpIPBfwEIRWeqOfl7qdXBJJU5jIgBG9W1PxxbpPGtrRRhjPFTTtQ5b8yFeKsdEFP8FjuyBjNaxd+X3ceOIfB55bxWrth+gV/vmcQzUGGMcNd2k3hjthTMA7pzTE2ISGXR9XNaJABg/tDPpKT6e+2RD/eMyxpgoaroH0VxEfiwi/ysil4jj34D1wHWnJ8Qk0r6/84rD00wtM1K56qw83li8hd2HyuIQnDHGnKimexB/BXoCy4DvArOAa4AxqnrlqRqakyi8Pi5jIgBuPTuf8mCYlz/fFIfAjDHmRDWuSa2qt6jq0ziPufYBLlXVxZ5HlqziOCaie7tszu3ehr/O3Uh5MByH4Iwx5riaEkTVcF1VDQElqmprX9ZHZhvoMQqW1n+dCHAeed1xoIx3v9gWh+CMMea4mhLEQBE54L4OAgMqt0XkQA1tzclUjYn4oN5dndc9l25tMm2tCGNM3NX0FJNfVZu7r2xVDURs27OVsep+CWS0icuYCJ/PWStiScl+Fm7aV//YjDHGVdMZhPGCPwUGXAer33XGRNTTVWflkZ0e4DkbOGeMiSNLEIlSOCFuYyIy0wKMH9qFd7/YztZ9R+MQnDHGWIJInKoxEfW/zARw04iuqCp/nWtrRRhj4sMSRCIVXg9bF8GO+s99mNcqg0v7tueleZs4Wh6KQ3DGmKbOEkQiVY6JWFL/MRHgrBWx/2gFbyw61cJ9xhhTO5YgEqlyTMSS+IyJGJLfin6dmvPcp/bIqzGm/jxNECIySkRWi8haEZkY5fid7tThi0XkExHp45ZfLCIL3GMLROQCL+NMqMLr4fDOuIyJEBFuPbuANTsP8cnaXXEIzhjTlHmWIETED0wCLsOZomN8ZQKI8JKq9lfVQuBR4Am3fBfwLVXtj7MM6V+9ijPhul8ctzERAN8c2IE2WWk8+4k98mqMqR8vzyCGAmtVdb2qlgNTgRMm+FPVyNHYmYC65YtUdatbvhxoJiJpHsaaOP4UGDA2bmMi0gJ+bhjehVmrS1lfeigOARpjmiovE0QnnHUjKpW4ZScQkbtEZB3OGcQ9Ufq5Glioql+b01pEbheRYhEpLi0tjVPYCRDHMREA1w/rSqrfx/OfbYhLf8aYpinhN6lVdZKqngE8iLPEaRUR6Qs8AtxxkrbPqGqRqhbl5uZ6H6xX2veD9gPidpkpNzuNbw3syPQFJew/WlFzA2OMicLLBLEF6Byxn+eWncxUYEzljojkAW8AN6nqOi8CbFAqx0RsWRCX7m4dmc+R8hDT5m+uubIxxkThZYKYD3QXkQIRSQXGATMiK4hI94jdK4A1bnlL4G1goqp+6mGMDUfheMhqB2/9MC6PvPbr1IKhBa15/rMNBEO2VoQxpu48SxCqGgTuBmYCK4FpqrpcRB4WkdFutbtFZLmILAZ+iPPEEm67M4GfuY/ALhaRtl7F2iCkt4BRv4Fti+HzZ+LS5W0jC9iy7yjvr9wRl/6MMU2LJMuAqqKiIi0uLk50GPWjCi+NhQ2fwF3zoGXnmtucQiisnPfYLDq2bMa0O0bEKUhjTDIRkQWqWhTtWMJvUpsIInDF44DCOw84CaMe/D7hlrPz+fyrPXyxZX98YjTGNBmWIBqall3gGz+FL9+DFf9X7+6uG9KZzFQ/z326of6xGWOaFEsQDdGwO6HDQHj3QTi6r15dNU9P4ZrBeby5ZCulB782lMQYY07KEkRD5A/At37vzNH0wS/q3d0tIwsoD4WZMs/WijDG1J4liIaq4yAY9n0ofhY2zatXVwVtMrmgV1tenLuRsqCtFWGMqR1LEA3ZN34CLTrDm/dCsLxeXd02soBdh8p5a8m2OAVnjEl2liAasrQsuPxxKF0Jnz1Zr65GnplDj3ZZPGtrRRhjaskSREPXcxT0GQMfPQq7Y59xRES4dWQBy7ceYP6GvfGLzxiTtCxBNAaXPQKBdHjrvnqNjRhT2ImWGSm2VoQxplYsQTQG2e3hov+Arz6GJVNj7qZZqp8JQ7vw9xXb2bznSBwDNMYkI0sQjcXgW6HzMJj5Ezi8O+ZubhzRFRFh8pwN8YvNGJOULEE0Fj6fMzai7CD8/acxd9OhRTMu79+BqfM3c7is/rPGGmOSlyWIxqRtbxh5Lyx5GdbPjrmbW0fmc/BYkNcWlsQvNmNM0rEE0dj8ywPQuhu89QOoOBpTF2d1aUVh55Y89+kGwmF75NUYE50liMYmpRl883ewZz18/HjM3dw6Mp+vdh3moy8b8VrexhhPWYJojLqdDwMnwKf/DTtWxNTF5f070K55Gs9+ao+8GmOi8zRBiMgoEVktImtFZGKU43eKyDJ3xbhPRKRPxLEfu+1Wi8ilXsbZKF3yS0hr7oyNCNd9SdEUv4+bRuTzzzW7+HLHwfjHZ4xp9DxLECLiByYBlwF9gPGRCcD1kqr2V9VC4FHgCbdtH5w1rPsCo4A/uP2ZSpk5cOmvYfM8WPBcTF2MH9qFtIDP1oowxkTl5RnEUGCtqq5X1XJgKnBlZAVVPRCxmwlU3jG9EpiqqmWq+hWw1u3PRBo4DgrOg/d/DgfqPglf68xUvj2oE28sKmHv4fpNBmiMST5eJohOwOaI/RK37AQicpeIrMM5g7injm1vF5FiESkuLW2CN1tFnBvWoXJ478GYurh1ZAHHKsK8PH9TnIMzxjR2Cb9JraqTVPUM4EHgoTq2fUZVi1S1KDc315sAG7qcM+Bf/t1ZnnT1e3Vu3rN9NiPPzOGvczZSEar7vQxjTPLyMkFsATpH7Oe5ZSczFRgTY9um7ex7ILc3vPMAlB2qc/PbRhawbf8x3vtiuwfBGWMaKy8TxHygu4gUiEgqzk3nGZEVRKR7xO4VwBp3ewYwTkTSRKQA6A587mGsjVsg1ZmGY/9mmPXrOjf/Rs+25Odk8Jw98mqMieBZglDVIHA3MBNYCUxT1eUi8rCIjHar3S0iy0VkMfBD4Ga37XJgGrACeA+4S1VtrcxT6TIMir4D8/4IWxfVqanPJ9xydj4LN+1j8eZ93sRnjGl0JFlWFysqKtLi4uJEh5FYx/bD/w6FrLbwvVngD9S66aGyICN+/QEX9G7L78cN8jBIY0xDIiILVLUo2rGE36Q2cZTewllcaPtSmPdUnZpmpQW4bkhn3l66je37j3kUoDGmMbEEkWz6XAk9RsGsX8G+uj26esvZ+YRVeXHuRo+CM8Y0JpYgko0IXP44IPD2A3VaorRz6wwu6t2OF+dtZOPuw97FaIxpFCxBJKOWneGCh2DNTFj+Rp2a3n9JTwS4+o+f8cWW/d7EZ4xpFCxBJKthd0CHQnhvIhzdV+tmPdtnM/37Z5MW8DP26Tl8smaXZyEaYxo2SxDJyud3xkYcLnXmaqqDM3KzeP1fz6Zz6wxuff5zZizZ6k2MxpgGzRJEMutYCMP/1ZntdeOcOjVt1zydV+4YwaAurbjn5UU2iM6YJsgSRLL7xk+gRRdn3Yhg3WZsbdEshcm3DeXSvu34xZsreOS9VSTLuBljTM0sQSS71Ey44rdQugo+/X2dm6en+PnD9YOZMKwLf5y9jn+fvtQm9TOmibAE0RT0uAT6fhs+fgx2ra1zc79P+NWYfvzgoh5MX1DC7ZOLOVIe9CBQY0xDYgmiqRj1CATSnUtNMVwmEhHuvag7v/p2Pz76spQJf5pniwwZk+QsQTQV2e3g4l/Ahn/C4pdi7ub6YV354w2DWbHtANc89Rlb9h2NY5DGmIbEEkRTctbN0Hk4/P2ncDj28Q2X9m3PX28bys6DZVz1h09Ztf1AzY2MMY2OJYimxOdzxkaUHYKZP61XV8O65fDqnSMAuPapOXz+1Z54RGiMaUAsQTQ1bXvBOT+ApVNh3ax6ddWrfXNe+/7Z5GanccNf5jFzua1IZ0wysQTRFJ17P+ScCW/9ACrqdw8hr1UGr915Nn07Nuf7Ly5gyjybCdaYZOFpghCRUSKyWkTWisjEKMd/KCIrRGSpiHwgIl0jjj3qrja3UkSeFBHxMtYmJSUdvvk72PsVfPRovbtrlZnKlO8O4/yebfnpG1/w+/fX2IA6Y5KAZwlCRPzAJOAyoA8wXkT6VKu2CChS1QHAdOBRt+3ZwEhgANAPGAKc51WsTVLBv0Dh9fDZk7Bjeb27y0gN8PSNg7n6rDx+9/6XPPS3LwiFLUkY05h5eQYxFFirqutVtRyYClwZWUFVZ6nqEXd3LpBXeQhIB1KBNCAF2OFhrE3TJb90VqF7814I1390dIrfx+PXDuD755/BlHmbuGvKQo5V2FLixjRWXiaITsDmiP0St+xkvgO8C6Cqc4BZwDb3NVNVV1ZvICK3i0ixiBSXlpbGLfAmI6M1XPpfUDIfiv8Sly5FhAdH9eJn3+zDe8u3c9Ozn7P/aEVc+jbGnF4N4ia1iNwAFAGPuftnAr1xzig6AReIyLnV26nqM6papKpFubm5pzPk5DHgOuj2DfjgYTiwLW7d3nZOAU+OH8SiTXsZ+/Qcdhywda6NaWy8TBBbgM4R+3lu2QlE5CLgp8BoVS1zi78NzFXVQ6p6COfMYoSHsTZdIvDNJyBUDu/+KK5djx7YkeduGcrmPUe46g+fsa70UFz7N8Z4y8sEMR/oLiIFIpIKjANmRFYQkUHA0zjJYWfEoU3AeSISEJEUnBvUX7vEZOKkdTc470FYOQNWvRPXrs/p3oZX7hhBWTDENX/8jEWb9sa1f2OMdzxLEKoaBO4GZuL8cZ+mqstF5GERGe1WewzIAl4VkcUiUplApgPrgGXAEmCJqr7pVawGOPvfoG1feOcBKDsY1677dWrBa98/m+bNUpjwp3nMWr2z5kbGmISTZHlevaioSIuLixMdRuO2eT785WIY+j247FHn8lMclR4s49bnP2fltoM8cvUArhmcV3MjY4ynRGSBqhZFO9YgblKbBqLzEBjyXfj8Gfj9QJj1a9i9Lm7d52anMfX2EYzolsMDry7hqY/W2YA6YxowO4MwJwpVwLLpzlxN6z8CFPKGwsCx0Pcq59HYeioPhrn/1SW8uWQrt40s4KEreuPz2UB5YxLhVGcQliDMyR3YCstehSVTYecK8KVAj0th4DjofgkE0mLuOhxW/vPtFTz36QZGD+zI49cOJDVgJ7TGnG6WIEz9qML2ZbD0FVg6DQ7vhPSW0O9qJ1nkDYnpfoWq8tRH63nkvVWcc2YbnrpxMFlpgfjHb4w5KUsQJn5CQVg/27kEtfItCB51HpMdMM4ZdNe6oM5dTl9QwoOvLaVPh+Y8e8sQcrNjPzMxxtSNJQjjjWMHYOWbTrL46p+AOivWDRwHfcdAs1a17mrWqp3865SFtG2exuTbhtI1J9OzsI0xx1mCMN7bX+JcfloyFXatBn8q9LzMObM48yIIpNbYxcJNe7nt+fkEfMLztw6lX6cWpyFwY5o2SxDm9FGFbYthySvODe4ju6BZa+h/jZMsOp11yvsVa3ce4mZ3gr//uqo/w7vl0CYrFVsOxBhvWIIwiRGqgHUfOmcVq96GUJmzkt3AcTBgLLTsErXZ9v3HuPnZz1m9wxnR3TozlR7tsujZLpvu7bLp2T6bHm2zaZGRcjp/G2OSkiUIk3jH9sOK/3POLDZ+4pR1Hekkiz5XOutSRFavCLFg415Wbz/IlzsqX4c4VBasqtO+eTo92mfTs10WPdzEcWbbLDJS7UkoY2rLEoRpWPZuhGXTnGSxew0E0p37FQPHwxkXgD/6mYGqsnX/Mb7cfpDVOw5W/Vyz8xDlQWfBIxHo3CrDTRjHE0e3Nlk2zsKYKCxBmIZJFbYsdJ6CWjYdju6BjDbO/YqB46BDYa3GV4TCyqY9R6rONiqTx/pdh6uWPQ34hII2mfRol31C8uiak4nfRnGbJswShGn4guWw9n0nWax+11mfIqMNNGsJac0hvbnzM3L7hJ/ZkNbihLIyUvhq95GIy1SH+HLHQTbtOULlP/u0gI8z2zr3N3q0z6aHe7mqU8tmdmPcNAmWIEzjcnQvLP+b8zTUsQNQduD4z7KDznZ5LaYk96U4iaMqabSAtGyCKVnsCzdjZ3ka246lsPFICuv3+yg5GuCAZnCQDMKp2bTLzaVr+1xyW2SQFvA5rxT/8e2An7SUiO2Aj/SU49upEeU215RpqE6VIOxunml4mrWColtPXScccpJFVfKI3N7/9YRSeWzfJgLHDtCmzHn10fDxPqsP1dgFoV0+yjSFID5C+AnhI4SPIH5C6v7EX3X8ED7246923EdY/KgEUJ/zE58f9QVA/OBLQfx+8AUQfwCJ+OkLBPD5AvgCKYjPj4gffH7E5/z0+Y7vV758fj+Ic8znd+v5A4g4+z6fH/H78fsDbv0APp8Pnz+A31+5H8Dn91ft+936x+scj4Wqn5YEk40lCNM4+fzO5admLWPvQxXKD1c7QzlwQkLxlx2gWcUxQqEKwsEgoVAFoWAF4VCIcChIOFSBhoKEQ0E0VIGGgxAKoeEKJ4mFgxCuQMLHQINIOAQawqchJBzE52771E09GnJTThg/4Zp/hwYkrEJIfChC2E2k4aqXHxU5vi9OWVh8KM4rLD5UKus624oPleP7YfxQeUwqt53jVP30OQlYIpPX8fLIsspEi/gRn6/qmM/ncxNvwD3mR/w+RAKIz+fuO8fFTdROYve5CdbZ91UmZ78fn3vc73MSrQT87raTiCvLq+I9IdbEJGFPE4SIjAJ+D/iBP6vqb6od/yHwXSAIlAK3qepG91gX4M8461orcLmqbvAyXtPEiEBalvNq3vHk1UjQN6lwGDSEhiqoqKggFAoRDFWgwRDBUJBwOEg4FCYUCqKhkJPEwqHjycvd1nDI3XbqhcNBtyyMhoJoOFxVpm5Sc8pC4JZp5bZW/gwjET9xEx8adl+hqmQoVWVhREPuy9l2ysIITpmvqk444hU8nmY0DDj1fJVpyN0WtGrbf0KK0qoyP8eP+6XxXV4PR02+Pkoy+tD7R+/H/f08+3cvIn5gEnAxUALMF5EZqroiotoioEhVj4jI94FHgbHuscnAr1T1HyKSBY3s65Qx9eXzAT7En0JqzTOVmJNQVcLqPO0WVqUirBxTdZJr2EmwYTdxhoNOIg1pCA2GCIWdY05idV56QlmwqkxDTqLFrUdEXSfRh1ENViXdryff8PFE6345OJ54nW2JqCOViVhD0CKP3h58dl5+MRoKrFXV9QAiMhW4EqhKEKo6K6L+XOAGt24fIKCq/3DrHfIwTmNMEhMR/II9zhwDL0cOdQI2R+yXuGUn8x3gXXe7B7BPRF4XkUUi8ph7RnICEbldRIpFpLi0tDRugRtjjGkga1KLyA1AEfCYWxQAzgUeAIYA3YBbqrdT1WdUtUhVi3Jzc09TtMYY0zR4mSC24NxgrpTnlp1ARC4CfgqMVtUyt7gEWKyq61U1CPwNOMvDWI0xxlTjZYKYD3QXkQIRSQXGATMiK4jIIOBpnOSws1rbliJSeVpwARH3LowxxnjPswThfvO/G5gJrASmqepyEXlYREa71R4DsoBXRWSxiMxw24ZwLi99ICLLcJ40/JNXsRpjjPk6m2rDGGOasFNNtdEgblIbY4xpeCxBGGOMiSppLjGJSCmwsR5dtAF2xSmcxs4+ixPZ53Ei+zyOS4bPoquqRh0nkDQJor5EpPhk1+GaGvssTmSfx4ns8zgu2T8Lu8RkjDEmKksQxhhjorIEcdwziQ6gAbHP4kT2eZzIPo/jkvqzsHsQxhhjorIzCGOMMVFZgjDGGBNVk08QIjJKRFaLyFoRmZjoeBJJRDqLyCwRWSEiy0Xk3kTHlGgi4nfXJHkr0bEkmoi0FJHpIrJKRFaKyIhEx5RIIvID9//JFyLysoikJzqmeGvSCSJiWdTLgD7AeHc1u6YqCNyvqn2A4cBdTfzzALgXZ7JJ46wv/56q9gIG0oQ/FxHpBNyDs2RyP8CPM2N1UmnSCYKIZVFVtRyoXBa1SVLVbaq60N0+iPMH4FSrACY1EckDrgD+nOhYEk1EWgD/AvwFQFXLVXVfQoNKvADQTEQCQAawNcHxxF1TTxB1XRa1yRCRfGAQMC/BoSTSfwM/AsIJjqMhKABKgefcS25/FpHMRAeVKKq6BXgc2ARsA/ar6t8TG1X8NfUEYaIQkSzgNeA+VT2Q6HgSQUS+CexU1QWJjqWBCOCs6vhHVR0EHAaa7D07EWmFc7WhAOgIZLpLJyeVpp4garUsalMiIik4yWGKqr6e6HgSaCQwWkQ24Fx6vEBEXkxsSAlVApSoauUZ5XSa9jLAFwFfqWqpqlYArwNnJzimuGvqCaLGZVGbEhERnGvMK1X1iUTHk0iq+mNVzVPVfJx/Fx+qatJ9Q6wtVd0ObBaRnm7RhTTtZYA3AcNFJMP9f3MhSXjTPpDoABJJVYMiUrksqh94VlWXJzisRBoJ3AgsE5HFbtlPVPWdxIVkGpB/A6a4X6bWA7cmOJ6EUdV5IjIdWIjz9N8iknDaDZtqwxhjTFRN/RKTMcaYk7AEYYwxJipLEMYYY6KyBGGMMSYqSxDGGGOisgRhTAMgIufbjLGmobEEYYwxJipLEMbUgYjcICKfi8hiEXnaXS/ikIj8zl0b4AMRyXXrForIXBFZKiJvuPP3ICJnisj7IrJERBaKyBlu91kR6y1McUfoGpMwliCMqSUR6Q2MBUaqaiEQAq4HMoFiVe0LfAT8h9tkMvCgqg4AlkWUTwEmqepAnPl7trnlg4D7cNYm6YYzst2YhGnSU20YU0cXAoOB+e6X+2bATpzpwF9x67wIvO6un9BSVT9yy18AXhWRbKCTqr4BoKrHANz+PlfVEnd/MZAPfOL5b2XMSViCMKb2BHhBVX98QqHI/6tWL9b5a8oitkPY/0+TYHaJyZja+wC4RkTaAohIaxHpivP/6Bq3zgTgE1XdD+wVkXPd8huBj9yV+kpEZIzbR5qIZJzOX8KY2rJvKMbUkqquEJGHgL+LiA+oAO7CWTxnqHtsJ859CoCbgafcBBA5++mNwNMi8rDbx7Wn8dcwptZsNldj6klEDqlqVqLjMCbe7BKTMcaYqOwMwhhjTFR2BmGMMSYqSxDGGGOisgRhjDEmKksQxhhjorIEYYwxJqr/D8S2B7pz5JajAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(model.history.history.keys())\n",
    "# summarize history for accuracy\n",
    "plt.plot(model.history.history['root_mean_squared_error'])\n",
    "plt.plot(model.history.history['val_root_mean_squared_error'])\n",
    "plt.title('model RMSE')\n",
    "plt.ylabel('RMSE')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'val'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "График должен выглядеть как-то так, но разница между начальными и конечными числами должна быть побольше."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Обучать эмбединги под задачу в теории правильнее всего, но на практике бывает лучше использовать предобученные эмбединги (word2vec, fastext, glove). Отдельно эмбединги обучаются быстрее и можно использовать гораздо больше данных, так как нам не нужна разметка."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Обучим фастекст модель."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gensim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "texts = avito.description.apply(preprocess).tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2min 3s, sys: 621 ms, total: 2min 4s\n",
      "Wall time: 46.9 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "ft = gensim.models.FastText(texts, vector_size=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('➡️коляска', 0.9847369194030762),\n",
       " ('коляска-люлька', 0.9759660363197327),\n",
       " ('санки-коляска', 0.9555612802505493),\n",
       " ('кресло-коляска', 0.9402867555618286),\n",
       " ('колясочка', 0.933530330657959),\n",
       " ('коляска-трансформер', 0.908164918422699),\n",
       " ('коляски', 0.8965704441070557),\n",
       " ('колясок', 0.8908169269561768),\n",
       " ('коляску-люльку', 0.8881371021270752),\n",
       " ('коляске', 0.884643018245697)]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ft.wv.most_similar('коляска')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Теперь нам нужно сделать матрицу с векторами. Индексы строчек в этой матрице должны совпадать с индексами слов в словаре."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "weights = np.zeros((len(word2id), 100))\n",
    "\n",
    "for word, i in word2id.items():\n",
    "    # вектор паддинга оставим нулевым\n",
    "    if word == 'PAD':\n",
    "        continue\n",
    "\n",
    "    try:\n",
    "        weights[i] = ft.wv[word]\n",
    "    \n",
    "    \n",
    "    except KeyError:\n",
    "        # для слов, которых нет в модели тоже возьмем какой-то  рандомный вектор\n",
    "        weights[i] = ft.wv['опрагпллирао']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs = tf.keras.layers.Input(shape=(100,))\n",
    "\n",
    "# передаем матрицу в эмбединг слой\n",
    "# указываем параметр trainable=False, чтобы вектора не обучались\n",
    "# Пояснение: кажется, что обучение векторов отключать не стоит, так как модель должна взять готовые \n",
    "# эмбединги и подогнать их к нашей задаче. На практитке это однако не работает - эмбединги просто \n",
    "# обучатся заново\n",
    "embeddings = tf.keras.layers.Embedding(input_dim=len(word2id), output_dim=100, trainable=False,\n",
    "                                      weights=[weights])(inputs, )\n",
    "mean = tf.keras.layers.Lambda(lambda x: tf.keras.backend.mean(x,  axis=1))(embeddings)\n",
    "dense = tf.keras.layers.Dense(64, activation='relu')(mean)\n",
    "dense = tf.keras.layers.Dense(64, activation='relu')(mean)\n",
    "outputs = tf.keras.layers.Dense(1, activation='linear')(dense)\n",
    "\n",
    "model = tf.keras.Model(inputs=inputs, outputs=outputs)\n",
    "optimizer = tf.keras.optimizers.Adam(learning_rate=0.0001)\n",
    "model.compile(optimizer=optimizer,\n",
    "              loss='mse',\n",
    "              metrics=[tf.keras.metrics.RootMeanSquaredError()])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/30\n",
      "258/258 [==============================] - 1s 2ms/step - loss: 0.0683 - root_mean_squared_error: 0.2614 - val_loss: 0.0684 - val_root_mean_squared_error: 0.2615\n",
      "Epoch 2/30\n",
      "258/258 [==============================] - 0s 2ms/step - loss: 0.0682 - root_mean_squared_error: 0.2611 - val_loss: 0.0682 - val_root_mean_squared_error: 0.2611\n",
      "Epoch 3/30\n",
      "258/258 [==============================] - 0s 2ms/step - loss: 0.0680 - root_mean_squared_error: 0.2607 - val_loss: 0.0680 - val_root_mean_squared_error: 0.2608\n",
      "Epoch 4/30\n",
      "258/258 [==============================] - 0s 2ms/step - loss: 0.0677 - root_mean_squared_error: 0.2603 - val_loss: 0.0678 - val_root_mean_squared_error: 0.2603\n",
      "Epoch 5/30\n",
      "258/258 [==============================] - 0s 2ms/step - loss: 0.0675 - root_mean_squared_error: 0.2598 - val_loss: 0.0675 - val_root_mean_squared_error: 0.2599\n",
      "Epoch 6/30\n",
      "258/258 [==============================] - 0s 2ms/step - loss: 0.0672 - root_mean_squared_error: 0.2593 - val_loss: 0.0673 - val_root_mean_squared_error: 0.2594\n",
      "Epoch 7/30\n",
      "258/258 [==============================] - 0s 2ms/step - loss: 0.0669 - root_mean_squared_error: 0.2587 - val_loss: 0.0670 - val_root_mean_squared_error: 0.2588\n",
      "Epoch 8/30\n",
      "258/258 [==============================] - 0s 2ms/step - loss: 0.0666 - root_mean_squared_error: 0.2580 - val_loss: 0.0665 - val_root_mean_squared_error: 0.2580\n",
      "Epoch 9/30\n",
      "258/258 [==============================] - 0s 2ms/step - loss: 0.0661 - root_mean_squared_error: 0.2570 - val_loss: 0.0660 - val_root_mean_squared_error: 0.2569\n",
      "Epoch 10/30\n",
      "258/258 [==============================] - 0s 2ms/step - loss: 0.0655 - root_mean_squared_error: 0.2559 - val_loss: 0.0654 - val_root_mean_squared_error: 0.2558\n",
      "Epoch 11/30\n",
      "258/258 [==============================] - 0s 2ms/step - loss: 0.0649 - root_mean_squared_error: 0.2547 - val_loss: 0.0649 - val_root_mean_squared_error: 0.2547\n",
      "Epoch 12/30\n",
      "258/258 [==============================] - 0s 2ms/step - loss: 0.0643 - root_mean_squared_error: 0.2536 - val_loss: 0.0643 - val_root_mean_squared_error: 0.2536\n",
      "Epoch 13/30\n",
      "258/258 [==============================] - 0s 2ms/step - loss: 0.0637 - root_mean_squared_error: 0.2525 - val_loss: 0.0638 - val_root_mean_squared_error: 0.2526\n",
      "Epoch 14/30\n",
      "258/258 [==============================] - 0s 2ms/step - loss: 0.0632 - root_mean_squared_error: 0.2514 - val_loss: 0.0634 - val_root_mean_squared_error: 0.2517\n",
      "Epoch 15/30\n",
      "258/258 [==============================] - 0s 2ms/step - loss: 0.0627 - root_mean_squared_error: 0.2505 - val_loss: 0.0629 - val_root_mean_squared_error: 0.2509\n",
      "Epoch 16/30\n",
      "258/258 [==============================] - 0s 2ms/step - loss: 0.0623 - root_mean_squared_error: 0.2496 - val_loss: 0.0626 - val_root_mean_squared_error: 0.2502\n",
      "Epoch 17/30\n",
      "258/258 [==============================] - 0s 2ms/step - loss: 0.0619 - root_mean_squared_error: 0.2488 - val_loss: 0.0623 - val_root_mean_squared_error: 0.2496\n",
      "Epoch 18/30\n",
      "258/258 [==============================] - 0s 2ms/step - loss: 0.0616 - root_mean_squared_error: 0.2481 - val_loss: 0.0620 - val_root_mean_squared_error: 0.2491\n",
      "Epoch 19/30\n",
      "258/258 [==============================] - 1s 2ms/step - loss: 0.0613 - root_mean_squared_error: 0.2475 - val_loss: 0.0618 - val_root_mean_squared_error: 0.2485\n",
      "Epoch 20/30\n",
      "258/258 [==============================] - 0s 2ms/step - loss: 0.0610 - root_mean_squared_error: 0.2469 - val_loss: 0.0616 - val_root_mean_squared_error: 0.2481\n",
      "Epoch 21/30\n",
      "258/258 [==============================] - 0s 2ms/step - loss: 0.0607 - root_mean_squared_error: 0.2464 - val_loss: 0.0614 - val_root_mean_squared_error: 0.2478\n",
      "Epoch 22/30\n",
      "258/258 [==============================] - 0s 2ms/step - loss: 0.0605 - root_mean_squared_error: 0.2459 - val_loss: 0.0612 - val_root_mean_squared_error: 0.2474\n",
      "Epoch 23/30\n",
      "258/258 [==============================] - 0s 2ms/step - loss: 0.0603 - root_mean_squared_error: 0.2455 - val_loss: 0.0611 - val_root_mean_squared_error: 0.2471\n",
      "Epoch 24/30\n",
      "258/258 [==============================] - 0s 2ms/step - loss: 0.0601 - root_mean_squared_error: 0.2451 - val_loss: 0.0610 - val_root_mean_squared_error: 0.2469\n",
      "Epoch 25/30\n",
      "258/258 [==============================] - 1s 2ms/step - loss: 0.0599 - root_mean_squared_error: 0.2447 - val_loss: 0.0608 - val_root_mean_squared_error: 0.2467\n",
      "Epoch 26/30\n",
      "258/258 [==============================] - 0s 2ms/step - loss: 0.0597 - root_mean_squared_error: 0.2443 - val_loss: 0.0608 - val_root_mean_squared_error: 0.2465\n",
      "Epoch 27/30\n",
      "258/258 [==============================] - 0s 2ms/step - loss: 0.0595 - root_mean_squared_error: 0.2440 - val_loss: 0.0607 - val_root_mean_squared_error: 0.2463\n",
      "Epoch 28/30\n",
      "258/258 [==============================] - 0s 2ms/step - loss: 0.0593 - root_mean_squared_error: 0.2436 - val_loss: 0.0606 - val_root_mean_squared_error: 0.2461\n",
      "Epoch 29/30\n",
      "258/258 [==============================] - 1s 2ms/step - loss: 0.0592 - root_mean_squared_error: 0.2433 - val_loss: 0.0605 - val_root_mean_squared_error: 0.2460\n",
      "Epoch 30/30\n",
      "258/258 [==============================] - 0s 2ms/step - loss: 0.0590 - root_mean_squared_error: 0.2430 - val_loss: 0.0604 - val_root_mean_squared_error: 0.2458\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x18e6f0198>"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train, y_train, \n",
    "          validation_data=(X_valid, y_valid),\n",
    "          batch_size=512,\n",
    "         epochs=30)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['loss', 'root_mean_squared_error', 'val_loss', 'val_root_mean_squared_error'])\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZYAAAEWCAYAAABFSLFOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Z1A+gAAAACXBIWXMAAAsTAAALEwEAmpwYAAA9TklEQVR4nO3dd3xV9f3H8dcnN4skjEACgYQlArJBIoKjUgeCVsAJKE4UR6labetstf60WrEVbVVwoKKIIopilSEUhcqQgCB7yExYGRASspPP749zoteYQSCXm5t8no/HfeSc7/mec77HK3nnrO9XVBVjjDGmpgT5uwHGGGPqFgsWY4wxNcqCxRhjTI2yYDHGGFOjLFiMMcbUKAsWY4wxNcqCxRgfE5G3ROTJY6y7U0Qu9HWbjPElCxZjAoQbUAUiki0iGSLypYic5rX8JhFREXm+zHrD3PK3vMrGiMgmEckSkQMi8oWINCxnP6WfNSftQE3As2AxJrA8q6pRQDyQArxRZvkPwDUiEuxVdiOwpXRGRM4D/gaMUtWGQBfgg/L24/XpVdMHYuouCxZj+PES1B9F5HsROSoib4hICxGZ7f5VP19Eor3qDxWR9SJyWES+EpEuXsv6iMgqd70PgPAy+/qNiKx2110iIj2r215VzQWmA73LLNoPrAUudvfVFDgLmOVV5wxgqap+524rQ1XfVtWs6rbDmPJYsBjzkyuBi4BOwGXAbOBhIBbn38rdACLSCZgG3Osu+wL4TERCRSQU+AR4B2gKfOhuF3fdPsBk4HagGTAJmCUiYdVpqIhEAqOAbeUsngLc4E6PBD4F8r2WLwcuFpG/isjZ1d23MVWxYDHmJ/9S1QOqmgIsBpar6neqmgfMBPq49UYAn6vql6paCDwHNMA5M+gPhAATVLVQVWcAK7z2MRaYpKrLVbVYVd/G+aXf/xjb+AcROQxkAecA15dTZyYwUEQa4wTMFO+FqroYuAI4HfgcSBeRf4qIp+x+vD5vH2P7jLFgMcbLAa/p3HLmo9zpVsCu0gWqWgLswbnv0QpI0Z/37rrLa7otcL/3L22gtbvesXhOVZsA7dw2dS5bwb1M9jnwKNBMVb8pp85sVb0M56xqGHATcGvZ/Xh9bjzG9hljwWLMcdiLExAAiIjghEMKsA+Id8tKtfGa3gM8VeaXdoSqTqtOA1R1N3AP8IKINCinyhTgfuDdKrZToqoLgP8C3avTBmMqYsFiTPVNBy4VkQtEJATnF3g+sARYChQBd4tIiIhcAfTzWvc14A4ROVMckSJyaemjvtWhql/ihNzYchZ/jXO/6F9lF7iPH48UkWi3Df2A84Bl1W2DMeWxYDGmmlR1MzAa55d2Gs6N/stUtUBVC3DuX9wEZODcj/nYa90k4Dbg38AhnJvvN51Ac8YDfyp7A14dC1Q1o5x1Drlt2AocwTmrGa+qU73q/KnMeyxpJ9BGU8+IDfRljDGmJtkZizHGmBplwWKMMaZGWbAYY4ypURYsxhhjalRw1VXqrpiYGG3Xrp2/m2GMMQFl5cqVaaoaW9Hyeh0s7dq1Iykpyd/NMMaYgCIiuypbbpfCjDHG1CgLFmOMMTXKgsUYY0yNqtf3WMpTWFhIcnIyeXl5/m6Kz4WHh5OQkEBISIi/m2KMqUMsWMpITk6mYcOGtGvXjp93UFu3qCrp6ekkJyfTvn17fzfHGFOH2KWwMvLy8mjWrFmdDhUAEaFZs2b14szMGHNyWbCUo66HSqn6cpzGmJPLguV4FBXAkb1QmOvvlhhjTK1jwXIcCnKz0OwDkLoJPbgJsg9AcUGNbPvw4cO8/PLL1V7vkksu4fDhwzXSBmOMOREWLMchL7gR26Qde7UZuYUlcGQvemA9mrYVjqZDSdFxb7uiYCkqqnybX3zxBU2aNDnu/RpjTE2xp8KOQ6MGITQMb8LR/CgycgpJyc2hEVlEFxwltGA3milIeGNoEA3hjUCOPb8ffPBBfvjhB3r37k1ISAjh4eFER0ezadMmtmzZwvDhw9mzZw95eXncc889jB3rjEpb2j1NdnY2Q4YM4ZxzzmHJkiXEx8fz6aef0qBBecOiG2NMzbNgqcRfP1vPhr1HjqluUYlSVFyCajHBFBNMGoICAkHB4AkBCaJrq0Y8dlm3CrfzzDPPsG7dOlavXs1XX33FpZdeyrp16358JHjy5Mk0bdqU3NxczjjjDK688kqaNWv2s21s3bqVadOm8dprr3HNNdfw0UcfMXr06OP+72CMMdVhwVJDgoOE4CAPioei4hLyShQpKSZYigkuKYSSQufMpTDEuVQWdGz/6fv16/ez90xefPFFZs6cCcCePXvYunXrL4Klffv29O7dG4C+ffuyc+fOGjlGY4w5FhYslajszOJY5BcWcyinkCM5eUSWHKGpZNGAAnT/OqRBE4hoBqFRUMljv5GRkT9Of/XVV8yfP5+lS5cSERHBwIEDy30PJSws7Mdpj8dDbq49vWaMOXksWHwoLMRDXGMPLRqFkZ0fycGjzSjMO0o0R2iSm4kn9xDqCUUimjkh4wmhYcOGZGVllbu9zMxMoqOjiYiIYNOmTSxbtuwkH5ExxlTNguUkEBEahofQMDyEouIGHM5pzPajeYQXZdFUs4jM2odm7YOwxjSLjOHss8+me/fuNGjQgBYtWvy4ncGDBzNx4kS6dOlC586d6d+/vx+Pyhhjyieq6u82+E1iYqKWHehr48aNdOnSxef7VlVyC4vJOFpAbk4OjTlCU8kmmGLUE4ZExjhnMUEen7bjZB2vMabuEJGVqppY0XI7Y/ETESEiNJiI0GCKGzcgM7cRu47mE1KQSYweIeJICpq1D2nQFCJjIMQeFzbGBAYLllrAEyQ0jQylaWQouQUNSM9uyt7cbJpqJk1y0gnKSUNDo5DIWAhvXOnNfmOM8TcLllqmQWgwCU2DKSoO51BOY37IziWqJJNmBUcILdiBBoW4l8liwGNfnzGm9rHfTLVUsCeI2IZhxESFkpUXScrRZkjeEWL0CFFZ+9Cs/UhkM4hsDsFhVW/QGGNOEp/2FSYig0Vks4hsE5EHy1l+n4hsEJHvRWSBiLT1WtZGROaJyEa3Tju3vL2ILHe3+YGIhLrlYe78Nnd5O18e28kiIjRqEEL7mChatmjBkci2bCOBQxqJHk1HD25AD+2Ewhx/N9UYYwAfBouIeICXgCFAV2CUiHQtU+07IFFVewIzgGe9lk0BxqtqF6AfcNAt/zvwvKqeChwCxrjlY4BDbvnzbr06JSzEQ6smDWgfF0NJozb8IG1I08aU5GZC6mY0fRvkZ0E9ftLPGON/vjxj6QdsU9XtqloAvA8M866gqgtVtfRP7WVAAoAbQMGq+qVbL1tVc8QZmep8nBACeBsY7k4Pc+dxl18gdXQkK0+QENMwjFPiomnfuSc7gtqyX6Mpzs+B9G1o2hbIPWQBY4zxC18GSzywx2s+2S2ryBhgtjvdCTgsIh+LyHciMt49A2oGHFbV0j7kvbf54/7c5Zlu/Z8RkbEikiQiSampqcd5aLVDkJubHVo0JqJpPLuD25OsMRQUFsKhnejBDXA0DbTEzy01xtQnteLmvYiMBhKB89yiYOBcoA+wG/gAuAn49ET3paqvAq+C84LkiW6vpj344IO0bt2a3/72twA8/vjjBAcHs3DhQg4dOkRhYSFPPvkkw4b9dPJXeh+mUYMQjuY3YF9WNJKXSaxmEpG5B80+gDRs6XTjXzdP4owxtYgvgyUFaO01n+CW/YyIXAg8ApynqvlucTKwWlW3u3U+AfoDk4EmIhLsnpV4b7N0f8kiEgw0BtJP6AhmPwj7157QJn4hrgcMeabCxSNGjODee+/9MVimT5/O3Llzufvuu2nUqBFpaWn079+foUOHljtmfWRYMJFhUeQVNiA1K5qDuZm0KMqgweFdTsA0ioewhhYwxhif8WWwrAA6ikh7nF/6I4FrvSuISB9gEjBYVQ+WWbeJiMSqairOfZUkVVURWQhchXPP5kZ+OouZ5c4vdZf/VwOwv5o+ffpw8OBB9u7dS2pqKtHR0cTFxfH73/+eRYsWERQUREpKCgcOHCAuLq7C7YSHeGjdNIL8ojAOHmmM5mbQovAQYRk/oCGRSON4CI2scH1jjDlePgsWVS0SkXHAXMADTFbV9SLyBE5IzALGA1HAh+5f37tVdaiqFovIH4AF7g34lcBr7qYfAN4XkSdxnip7wy1/A3hHRLYBGThBdmIqObPwpauvvpoZM2awf/9+RowYwdSpU0lNTWXlypWEhITQrl27crvLL09YsBMweYVhHDjSBE9eBs0LDxOStgUNawTFhT4+GmNMfePTeyyq+gXwRZmyv3hNX1jJul8CPcsp347zxFnZ8jzg6hNpb20xYsQIbrvtNtLS0vj666+ZPn06zZs3JyQkhIULF7Jr165qbzM8xEObZlHkFoSz90g0YfnpxOZnQtZ+ij9+Gc/5D0OTNj44GmNMfePTFyTN8enWrRtZWVnEx8fTsmVLrrvuOpKSkujRowdTpkzhtNNOO+5tNwgNpm1MQxrGJJAc0p4sIij6fgbFL5xO8dw/Q8HRGjwSY0x9VCueCjO/tHbtTw8NxMTEsHTp0nLrZWdnH9f2I8OCiYxtxKF90fwu5g0u2v86Vy99kbzvPyJ82AToNOi4tmuMMXbGUs+FhXiY9NuhNBv9Or8Lf4rdWcB7V5MzdTQc2efv5hljApAFi0FEOP+0Foy//07m/2oGz5eMIGjLHPJf6EvhsklQUuzvJhpjAogFSzkC8Cnl41L2OMNDPNx1YVeu/v0Enmz7BssLTiFkzp848tKvYd/3fmqlMSbQWLCUER4eTnp6ep0PF1UlPT2d8PDwXyxLiI7gyVuGUXzdRzwRdh/5aTsonnQe2Z89aDf3jTFVsjHvy4x5X1hYSHJy8jG/JxLIwsPDSUhIICQkpMI6eYXFvPPf1TT65ilGBC0gKyyO8OHPE9LlkpPYUmNMbVLVmPcWLGWCxZRvT0YO782YzvDk8XQOSia1+xhiL3/WRrE0ph6qKljsUpg5Jq2bRvDA2JvYN3IuH3guJXbdG+z918WUZAV2D9HGmJpnwWKqZWDXBC6+/y0mxz5A00NrODRhAIe3fevvZhljahELFlNtTSJCufmuh5g/YAp5RUqDdy9h67xX/d0sY0wtYcFijouI8JvBl5B9w3w2eE6j45I/smribRQV5Fe9sjGmTrNgMSekc4f2dP7jfL5udg2n75/OlvHnsy9lt7+bZYzxIwsWc8IiwsM573evkdT3WdoXbCHotfNYumiev5tljPETCxZTYxIvu51DI/8DQSGcvmAUM994hrxC6w7GmPrGgsXUqFZdziT63iWkNO7D5XueZs7zY8nKLfB3s4wxJ5EFi6lxoY1iOOWeOexsP4LhOTOY9e8/kJVnI1UaU19YsBjf8ATT7vqJpLQZynVH32bqS49xxMLFmHrBp8EiIoNFZLOIbBORB8tZfp+IbBCR70VkgYi09VpWLCKr3c8sr/LFXuV7ReQTt3ygiGR6LftL2f2ZkywoiPgbJ3Ow5UDGHnmZSS+NJzPXwsWYus5nwSIiHuAlYAjQFRglIl3LVPsOSFTVnsAM4FmvZbmq2tv9DC0tVNVzS8uBpcDHXuss9lrnCR8clqkuTwjNb55GZuzp3HPkOf75ystk5li4GFOX+fKMpR+wTVW3q2oB8D4wzLuCqi5U1Rx3dhmQcKwbF5FGwPnAJzXTXOMzoRFEj/mY/OiOPJD5FE9OfJPDOXZD35i6ypfBEg/s8ZpPdssqMgaY7TUfLiJJIrJMRIaXU384sEBVj3iVDRCRNSIyW0S6lbcTERnrbjcpNdU6UDxpGjSh4a2fQcM4Hs18nIcnfmDhYkwdVStu3ovIaCARGO9V3NbtlvlaYIKIdCiz2ihgmtf8KnedXsC/qOBMRlVfVdVEVU2MjY2tqUMwxyKqORFjPiM8IorHMx/lvomfcuiohYsxdY0vgyUFaO01n+CW/YyIXAg8AgxV1R87mlLVFPfnduAroI/XOjE4l9o+96p/RFWz3ekvgBC3nqlNotsSdvOnRIfB45kP89tXZ5Nh4WJMneLLYFkBdBSR9iISCowEZnlXEJE+wCScUDnoVR4tImHudAxwNrDBa9WrgP+oap7XOnEiIu50P5xjS/fJkZkT07wLITd8RHxINo8dfpSxk+aTnm2dVxpTV/gsWFS1CBgHzAU2AtNVdb2IPCEipU95jQeigA/LPFbcBUgSkTXAQuAZVfUOlpH8/DIYOGGzzl3nRWCk1ufhMWu7hEQ8o6bS0bOPhzIf5+ZXv7Z7LsbUETY0sQ1N7F/rP0E/vIlFJb2Y2OpJ3rr1LMKCPf5ulTGmEjY0sandug1HLpvAeUGruTLlWR6a8T31+Y8dY+oCCxbjf31vgoEPcZVnEW3XvcCE+Vv93SJjzAmwYDG1w3kPoH2u557gmRz8aiIfrUz2d4uMMcfJgsXUDiLIb56npMOFPBXyJnNnvsXSH+yhPmMCkQWLqT08IQRd8zYlcT14IfhFXnznfbYdzPZ3q4wx1WTBYmqXsCiCR88gpFELXuIZHpn8qb3jYkyAsWAxtU9Uc4Jv+JhGYR7+nvsE9721wIY4NiaAWLCY2immI8Gjp9Pac4h7D/6ZB95fRkmJPYZsTCCwYDG1V+t+eK6eTO+gH/jNlj8zfs56f7fIGHMMLFhM7dblNzDkWS7yrKTVkseYtnyXv1tkjKmCBYup9eTMsZScdQ/XB88n+bOn+HqLjaNjTG1mwWICQtCFj1PY7Sr+GPwBc6ZOYPP+LH83yRhTAQsWExiCggi5/BXyW5/D/8krTHzjVQ5m5VW9njHmpLNgMYEjOJSw66ZR2LQzTxb8nb+/Po3cAnsM2ZjaxoLFBJbwRjS4eSZBkTE8dPgvPP3u5/YYsjG1jAWLCTwN42hwy6dEhQZxy84/8O//LPV3i4wxXixYTGCK6UjYDTNo5TnMwKS7mLFkk79bZIxx+TRYRGSwiGwWkW0i8mA5y+8TkQ0i8r2ILBCRtl7Lit3hir2HLEZE3hKRHV7LervlIiIvuvv6XkRO9+WxGf+T1mfgueYtugXtosWc21iyea+/m2SMwYfBIiIe4CVgCNAVGCUiXctU+w5IVNWewAzgWa9luara2/0MLbPeH72WrXbLhgAd3c9Y4JWaPSJTG3m6XELBkOc5N+h70qbdztb9R/zdJGPqPV+esfQDtqnqdlUtAN4HhnlXUNWFqprjzi4DEk5gf8OAKepYBjQRkZYnsD0TIBqceROZA/7EUBax/PW7SbPekI3xK18GSzywx2s+2S2ryBhgttd8uIgkicgyERlepu5T7uWu50Uk7Dj3Z+qQxoMeJu200YwumsmnEx+13pCN8aNacfNeREYDicB4r+K2qpoIXAtMEJEObvlDwGnAGUBT4IFq7musG1hJqanWNUidIULMNS+yv9VF3Jz1GlMnT7DHkI3xE18GSwrQ2ms+wS37GRG5EHgEGKqqP17DUNUU9+d24Cugjzu/z73clQ+8iXPJ7Zj3p6qvqmqiqibGxsYe/9GZ2ifIQ9zN73CgSS9G7/0bH854z98tMqZe8mWwrAA6ikh7EQkFRgKzvCuISB9gEk6oHPQqjy69xCUiMcDZwAZ3vqX7U4DhwDp3tVnADe7TYf2BTFXd58PjM7VRSAPibv+EQ+GtGbL+fuYtmOfvFhlT7/gsWFS1CBgHzAU2AtNVdb2IPCEipU95jQeigA/LPFbcBUgSkTXAQuAZVd3gLpsqImuBtUAM8KRb/gWwHdgGvAbc5atjM7WbRETT7I7PKAhuyOmLxrBq1Qp/N8mYekVU6+916MTERE1KSvJ3M4yPZKdsoOj1izmqYeRf/wWndOjk7yYZUyeIyEr3Hni5asXNe2N8ISq+K/kjptOELHj3CtIO7vd3k4ypFyxYTJ3W4rQB7BvyJvEl+0l/dSi52Zn+bpIxdZ4Fi6nzTj3zEjacPYFTC7ew/aUrKCmwcVyM8SULFlMv9Bk0mv91fYxuuUlsmngtlNgLlMb4igWLqTd+dc29zGk1jq4ZC9jy5u1Qjx9cMcaXLFhMvSEiXDjm//i80Ug67fmQXTMe8neTjKmTLFhMvRLsCWLgXf9mdujFtF3/Cgfm/sPfTTKmzrFgMfVOZHgIfe58k/9Kf1osfYLMZVP83SRj6hQLFlMvxUVH0vKWd1miPYiacw95a2dVvZIx5phYsJh6q0vrWIqufoe1Je0J/uhmijfN8XeTjKkTLFhMvfar7u3ZOmgKG0sSKPngenTrfH83yZiAZ8Fi6r2rz+nO4v6vs6W4JcXTroXtX/m7ScYENAsWY4A7hyQyo+u/2VrUnKJ3R8COxf5ukjEBq9JgEZHzvabbl1l2ha8aZczJJiI8cvU5TGz7T7YXNaPo3Wtg1xJ/N8uYgFTVGctzXtMflVn2aA23xRi/CvYE8fT15/NU7N/ZXdSE4neugt3L/d0sYwJOVcEiFUyXN29MwIsIDeafNw/iT5FPklzUiJJ3roBkG7PHmOqoKli0guny5o2pE5pFhfH8rUO40/NX9hVFUvLO5ZCyyt/NMiZgVBUsp4jILBH5zGu6dL59FesaE7BaN43g2VsGc2PxXzhYGI6+cznsW+PvZhkTEKoKlmHAP3DutZROl84Pr2rjIjJYRDaLyDYRebCc5feJyAYR+V5EFohIW69lxSKy2v3M8iqf6m5znYhMFpEQt3ygiGR6rfOXYzh+YyrUPb4xj11/MSPyHiWtMBSdMhz2r/N3s4yp9SoNFlX92vsDLAGOABvd+QqJiAd4CRgCdAVGiUjXMtW+AxJVtScwA3jWa1muqvZ2P0O9yqcCpwE9gAbArV7LFnut80Rl7TPmWJzbMZZ7r76AK3Me4nBhEDplKOxd7e9mGVOrVfW48UQR6eZONwbWAFOA70RkVBXb7gdsU9XtqloAvI9z1vMjVV2oqjnu7DIgoaoGq+oX6gK+PZZ1jDkRl/dJ4NrB53H50YfILApB374Mdn7j72YZU2tVdSnsXFVd707fDGxR1R5AX+BPVawbD+zxmk92yyoyBpjtNR8uIkkiskxEhpet7F4Cux7w7uBpgIisEZHZpYFYznpj3e0mpaamVnEIxjhu/9UpDBwwgCFHHuGwpym8ewVstr7FjClPVcFS4DV9EfAJgKrur8lGiMhoIBEY71XcVlUTgWuBCSLSocxqLwOLVLX0FelV7jq9gH+VtrUsVX1VVRNVNTE2NrYmD8PUYSLCX37TlTN69eD8jAdJj+wA718Laz7wd9OMqXWqCpbDIvIbEekDnI17diAiwTj3NyqTArT2mk9wy35GRC4EHgGGqmp+abmqprg/twNfAX281nkMiAXu86p/RFWz3ekvgBARiamijcYcs6Ag4bmre9GjUwcGHryP9JgzYOZYWDbR300zplapKlhuB8YBbwL3ep2pXAB8XsW6K4COItJeREKBkcDPBr1wA2sSTqgc9CqPFpEwdzoGJ9Q2uPO3AhcDo1S1xGudOBERd7qfe2zpVbTRmGoJDQ5i4ujTObV1S87bexfprS+COQ/AwqdB7dUuYwBEffiPQUQuASYAHmCyqj4lIk8ASao6S0Tm4zzdtc9dZbeqDhWRs3ACpwQnICao6hvuNouAXUCWu87HqvqEiIwD7gSKgFzgPlWttLOnxMRETUqyt6pN9R3OKeCaSUs5cPgoX3WeSfSW6dBvLAz+OwRZ366mbhORle6tivKXVxYsIvJiZRtX1btPoG1+Z8FiTsT+zDyufGUJeQVFLOg5nyarJ0GPa2D4y+AJ8XfzjPGZqoKlqj+t7gDOAfYCScDKMh9j6q24xuG8M6YfiHDpxovJOvthWDsd3r8OCnP93Txj/KaqYGkJvIpzT+N6IAT4VFXfVtW3fd04Y2q7U2KjeOvmfmTmFXHluv7kDHoOts6Dd66A3MP+bp4xflHVm/fpqjpRVX+N8x5LE2CDiFx/MhpnTCDokdCYV2/oy860HK5b3ZX84a9B8gp4cwgc3lP1BoypY47pLqOInA7cA4zGeYnRLoMZ4+WsDjG8OKoPa/YcZuyqthSOnA6ZyfD6hdYFjKl3qurS5QkRWYnzvsjXOP16jVHVDSeldcYEkMHd4/jb5T34eksq96+MpuTmOc5N/Dcvsbf0Tb1S1RnLoziXv3oBTwOr3J6I14rI975unDGBZmS/Njww+DRmrdnLo0tLKLnlS4jpCO+Pgm9f83fzjDkpgqtYbmOuGFNNd5x3Cll5hbz81Q8UFyt/u/FzPB/fCl/8ATJ2wKD/gyCPv5tpjM9UGiyququ8chEJAkbhvKhojPEiIvzx4s6EeIJ4YcFWCotLePbqdwie/ygsewkO74IrXoPQCH831RifqDRYRKQR8FucXolnAV/idPFyP04X+lN93UBjApGI8PuLOhHiEZ6bt4XCEuWf1zxNSHQ7mPMQvP0bGPU+RDX3d1ONqXFVXQp7BzgELMUZUOthQIDhqrrat00zJvCNO78jocFB/O2LTRQWlfDiqNsJbdIGZoyB1y+A62ZAbGd/N9OYGlXlmPeqepOqTsK59NUVuNhCxZhjN/ZXHXjssq7MWb+fu6auJP/UwXDz51CYB29cBDsW+buJxtSoqoKlsHRCVYuBZFXN822TjKl7bj67PU9d3p35Gw9y25SV5DXvDbfOh4Yt4Z3L4evxUFzk72YaUyOqCpZeInLE/WQBPUunReTIyWigMXXFdWe25dkre7J4ayq3vLWCnMh4uGUudB0OC5+ENwdD+g/+bqYxJ6yqLl08qtrI/TRU1WCv6UYnq5HG1BXXnNGaf17Ti2Xb07lp8gqyg6LgqjfgqsmQthUmngMrXrexXUxAs4EjjDnJLu+TwAsj+7By9yFueGM5R/IKofuVcNdSaNMfPr8fpl4FR/ZVvTFjaiELFmP84LJerXjp2tNZm5LJ6NeXk56dD41aweiP4ZLnYOc38MoAWD/T3001ptosWIzxk8Hd45g4ui+b92dxxStL2JF2FESg321wx2Joegp8eBN8dJt1wW8CigWLMX50QZcWTBvbn6y8Iq54+RtW7spwFsR0hFvmwcCHYd1H8MpZsP0rv7bVmGPl02ARkcEisllEtonIg+Usv09ENrgdWy4QkbZey4pFZLX7meVV3l5Elrvb/EBEQt3yMHd+m7u8nS+PzZiacnqbaD6+8yyaRIQy6rXlzF7r3lvxBMPAB+DWLyEkAqYMg8//APlZ/m2wMVXwWbCIiAd4CRiC82LlKBHpWqbadzhd8fcEZgDPei3LVdXe7meoV/nfgedV9VScXgHGuOVjgENu+fNuPWMCQruYSD668yx6xDfmrvdW8fri7Wjpk2HxfeH2RXDmnc4TYy8PgG0L/NtgYyrhyzOWfsA2Vd2uqgXA+8Aw7wqqulBVc9zZZUBCZRsUEQHOxwkhgLeB4e70MHced/kFbn1jAkLTyFCm3nomg7vF8eTnG/nrZxsoLnHDJTQChjwDt8yBkAbw7hXwyV2Qk+HfRhtTDl8GSzzgPS5rsltWkTE4o1OWCheRJBFZJiLD3bJmwGFVLX1F2XubP+7PXZ7p1v8ZERnrbjcpNTW1modkjG+Fh3h46drTue3c9ry1ZCd3vLuS3ILinyq06Q+3L4Zz74c178NLZ8KGWRVv0Bg/qBU370VkNJAIjPcqbquqicC1wAQR6VAT+1LVV1U1UVUTY2Nja2KTxtSooCDhkUu78vhlXZm/8QAjX1tGWnb+TxVCwuGCv8DYr6BhHEy/Hj64HrIO+K3NxnjzZbCkAK295hPcsp8RkQuBR4Chqvrjvx5VTXF/bge+AvoA6UATESntldl7mz/uz13e2K1vTEC66ez2TBrdl837j3DFy0v4ITX75xVa9oTb/gsXPAZb5sJL/WD1NHtr3/idL4NlBdDRfYorFBiJM6bLj0SkDzAJJ1QOepVHi0iYOx0DnA1sUOdu5kLgKrfqjcCn7vQsdx53+X9V7V+YCWyDusXx/tgBHM0v4spXlrB8e5m/lTwhcO59cMf/IPY0+OQO5639w3vK36AxJ4HPgsW9zzEOmAtsBKar6noReUJESp/yGg9EAR+Weay4C5AkImtwguQZVd3gLnsAuE9EtuHcQ3nDLX8DaOaW3wf84vFmYwJR79ZNmHnX2TSNDOW615cz+X87+MXfTLGd4ObZMGQ87FrqnL3M/6vd3Dd+IfX5j/rExERNSkrydzOMOSZH8gq5f/oavtxwgKG9WvHMlT2ICC1nrL5Du2DBX2HdxxDWEPrfBQPugvDGJ7/Rpk4SkZXuPfDyl1uwWLCYwFFSokxc9APPzd1Mx+YNeWX06ZwSG1V+5QMb4Ku/wcbPILwJnPU7OPMOCKugvjHHqKpgqRVPhRljjk1QkHDXwFOZcsuZHMzKY9i/v2He+v3lV27RFUa867xc2aY//Pf/4IWe8M2LUJBT/jrG1AALFmMC0DkdY/jP3efSPjaSse+s5Nk5m356mbKslr3g2g/g1gUQ1xO+/DO82BuWTXSGRzamhlmwGBOg4ps0YPrtAxjVrzUvf/UDN07+loyjBRWvkJAIN3zi3ORv1hHmPAD/Oh2S3oTiworXM6aaLFiMCWDhIR6evqInf7+yB9/uzOA3Ly5mzZ7Dla/U9iy46T9ww6fOGDD/udd5g3/dR1BScjKabeo4CxZj6oARZ7ThozvOQkS4euJS3lu++5ePJHsTgVMGwpgvYeQ0CA6DGbfAq+fB1vn2kqU5IRYsxtQRPRIa85/fncOZpzTl4ZlrGTftOzJzqrjEJQKnXeK8YHn5q5B3GKZeCW/9BvZ8e1LabeoeCxZj6pDoyFDeurkff7y4M3PX7WfIC4tYVvZt/fIEeaDXCBi30hkaOW0LvHERTBvlPLZsTDVYsBhTx3iChN/++lQ+uvMsQoODGPXaMsbP3URh8THcPwkOdYZGvmc1nP9n2PmNM3rlx2MhY4fP227qBntB0l6QNHXY0fwi/vrZeqYnJdMroTEvjOxDu5jIY99ATgZ8MwGWT3KeHDvtEkgc49yfseGO6i17874SFiymvvhi7T4e+ngthcUlPD60G1f3TaBa4+Ad2QvLJ8KqdyA3A5qd6gRM71HQINp3DTe1kgVLJSxYTH2y93Au901fzbLtGVzaoyV/u7wHjSNCqreRwjzY8KkzRHLytxDcAHpcCWfcCq36+KbhptaxYKmEBYupb4pLlEmLfuCf87YQ2zCMf17TmwEdfjHQ6rHZ9z0kvQHfT4fCHGh1uhMw3a9whk82dZYFSyUsWEx99X3yYe55fzU7048y5uz2/P6iTkSGldNT8rHIy4Q1HzhnMWmbnQ4ve42E3tc5g5GZOseCpRIWLKY+O5pfxFNfbOS95btp1Ticx4d2Y1C3uOPfoCrs/J9zFrPpcygugLge0Hs09LgaIo/zzMjUOhYslbBgMQZW7srg4Y/XsflAFhd1bcHjQ7sR3+QEL2XlZDhdxHz3LuxbDUEh0HmwEzKnXgie4zw7MrWCBUslLFiMcRQWl/DG/3YwYf4WgkT4/YWduPnsdgR7auBVtwPrYfV78P0HcDQVIps7L2P2Hg3NTzvx7ZuTzq/BIiKDgRcAD/C6qj5TZvl9wK1AEZAK3KKqu7yWNwI2AJ+o6jgRaQgs9tpEAvCuqt4rIjfhDHWc4i77t6q+Xln7LFiM+bk9GTk8Pms9CzYdpEvLRjx1eXdOb1NDjxMXF8LWL2H1VNgyB0qKnCfJug6HrkOh6Sk1sx/jc34LFhHxAFuAi4BkYAUwymvsekTk18ByVc0RkTuBgao6wmv5C0AskKGq48rZx0rg96q6yA2WxPLqVcSCxZhfUlXmrt/P47M2cCArj2v7teFPg0+jcYNqPppcmexUWPuhcxazb7VTFtcDugxzQia2c83ty9Q4f44g2Q/YpqrbVbUAeB8Y5l1BVReqaulQdstwzkAAEJG+QAtgXnkbF5FOQHN+fgZjjDlBIsLg7i2Zf/953HxWe6Z9u5sL/vE1n65OqbzH5OqIioUBd8HtX8M938OgpyAkAhY+CS/1g3/3g/8+CfvXWk/LAciXwRIP7PGaT3bLKjIGmA0gIkHAP4A/VFJ/JPCB/vz/9CtF5HsRmSEirY+v2cYYgKiwYP5yWVdmjTuHVk3Cuef91Yx4dRnr92bW7I6i28JZ42DMPLhvIwwZD1HNYfE/YOI58GIf+PIvTm/LJcU1u2/jE768FHYVMFhVb3XnrwfOrOCS1mhgHHCequaLyDggQlWfregSl4hsAK5X1ZXufDMg213/dmCEqp5fzr7GAmMB2rRp03fXrl1lqxhjyiguUT5YsYfn5m3mcE4BI/u14Q+DOtM0MtR3O81Ohc2fO2/671jk3JOJaAYdB0GnwdDhfAhv5Lv9mwr58x7LAOBxVb3YnX8IQFWfLlPvQuBfOKFy0C2bCpwLlABRQCjwsqo+6C7vBXyoqp0q2LcH575M48raaPdYjKmezJxCJizYwpSlu4gM9fD7izoxun9bQmri6bHK5B6CbQtgy1zYOs8ZNyYoGNqe7YRMp4uhWQfftsH8yJ/BEoxz8/4CnCe1VgDXqup6rzp9gBk4ZzZbK9jOTZQ5YxGRZ4B8VX3Mq6ylqu5zpy8HHlDV/pW10YLFmOOz9UAWT/xnA4u3ptGxeRSPXdaNczrGnJydFxc5/ZRtmeMETeomp7xZRydgOl0MCf0gJPzktKce8vfjxpcAE3AeN56sqk+JyBNAkqrOEpH5QA9gn7vKblUdWmYbN/HLYNkOXKKqm7zKngaG4jy6nAHc6b28PBYsxhw/VeXLDQd48vON7M7IYVDXFjx6aVfaNIs4uQ3J2OGcxWyZ47z5X1wAweHQuh+0+xW0Pxfi+4KnBp9qq+fsBclKWLAYc+LyCot54387eGnhNopKlNvObc9dA089/r7HTkR+lhMuOxY792UOrHXKQyKhTX8nZNr/CuJ62dv/J8CCpRIWLMbUnP2Zefx9ziZmfpdCTFQo4359KqPObENYsMd/jTqaDrvcoNm5+KfLZmGNoO1Z0O4c56cFTbVYsFTCgsWYmrdq9yGenbOJZdszSIhuwO8v7MTwPvF4gmrBiJNZB5yA2bnYCZuMH5zykEjn0lnbs6HtAOfSmXX9XyELlkpYsBjjG6rK4q1pjJ+7mbUpmXRqEcX9gzozqGuL6o1c6WtZ+2HXkp8+B91nizyhzvgybc9ywqb1GRBe6UOm9YoFSyUsWIzxLVVl9rr9PDd3M9vTjtK7dRP+NLgzZ3U4SU+QVVdOBuxZDru+gV1LYe93oO5LmVFxzpDMzTq4n1OdT3Q7CA7za7NPNguWSliwGHNyFBWX8NGqZCbM38q+zDzO7RjDHy/uTM+EJv5uWuXysyF5BaSshIztkL4N0n+AnLSf6kgQNG79U9DEdYf4RKe/syA/3l/yIQuWSliwGHNy5RUW8+6yXby0cBuHcgoZ0j2Ouy/oSJeWAfYGfe4hSHeDJuMHN3Dc0CnIduqERDq9Nyf0de7ZxPeFRvFQmy4FHicLlkpYsBjjH1l5hby+eAdv/G8H2flFXNytBb87vyPd4wP8PkZJiXNmk5LknOWkrHQ60iwucJZHtXDOZuJPdz4te0NEU782+XhYsFTCgsUY/8rMKWTyNzuY/M0OsvKKuLBLc353fkd6tW7i76bVnKJ82L/ODRo3cNK3/bS8cRto2dMJmZa9nE/DFn5r7rGwYKmEBYsxtcORvELe/mYnr/9vB5m5hQzsHMvdF3SsuUHGapucDGccmn3fw741zqf00WdwHhQoDZmWvaBFV2jSttbcs7FgqYQFizG1S1ZeIVOW7uL1xds5lFPIuR1juOeCjiS2C7zLRdWWl+mc2ZQGzb7VkLYFtMRZ7glzHg6I6QgxnZyHA2I6OmWhkSe1qRYslbBgMaZ2OppfxLvLdvHqou2kHy3grA7NGPfrUxnQoVnteg/G1wqOwoH1To8BaVsgbSukbobDu34KHHAup5UGTtP2EN3eeQw6uq1PHoW2YKmEBYsxtVtOQRHvLd/NpEXbSc3Kp3t8I27/VQeGdI8j2Ndd9ddmhXnOQwJpm52wSdvifrZB4VGviuI8idbUDZrS0GnaHpp2OO7xbCxYKmHBYkxgyCssZuZ3Kby2aDvb046SEN2AW89pzzVntCYi1Pr4+pEqHE11enw+tOOnn4d2OtNHD/5Ud8A4uPip49qNBUslLFiMCSwlJcqXGw/w6qLtrNx1iCYRIdzQvy03nNWOmKj69fb7ccnPdkLm0A7nYYCWPY9rMxYslbBgMSZwrdyVwaSvt/PlxgOEeoK4qm8Ct517Cu1iTu6N7PrIgqUSFizGBL4fUrN5ffF2PlqZQmFJCRd3jePGs9rR/5Sm9etG/0lkwVIJCxZj6o6DWXm8vWQnU5fv5nBOIZ1aRHH9gHZc0SfeP4OO1WEWLJWwYDGm7skrLGbWmr1MWbqTdSlHaBgWzJV9E7h+QFs6xEb5u3l1gr/HvB8MvIAz5v3rqvpMmeX3AbfijFOfCtyiqru8ljcCNgCflI55LyJfAS2BXLfaIFU9KCJhwBSgL5AOjFDVnZW1z4LFmLpLVfluz2GmLNnJ52v3UVisnNsxhhsGtOP805rXjoHHApTfgkVEPMAW4CIgGVgBjFLVDV51fg0sV9UcEbkTGKiqI7yWvwDEAhllguUPqvqzRBCRu4CeqnqHiIwELvfeVnksWIypH1Kz8vlgxW6mLt/Nvsw84ps04Lr+bRiR2Jpm9jRZtVUVLL58w6gfsE1Vt6tqAfA+MMy7gqouVNUcd3YZkFC6TET6Ai2Aece4v2HA2+70DOACsTt3xhggtmEY487vyOI//ZqJo0+nbbMInp2zmf5PL2Dce6v4ZlsaJSX197ZATfPlHa14YI/XfDJwZiX1xwCzAUQkCPgHMBq4sJy6b4pIMfAR8KQ6p10/7k9Vi0QkE2gGpHmvKCJjgbEAbdq0qf5RGWMCVrAniMHdWzK4e0u2Hshi2rd7+GhVMv/5fh9tm0Uw8ow2XNU3gdiGdhZzImpFnwgiMhpIBMa7RXcBX6hqcjnVr1PVHsC57uf66uxLVV9V1URVTYyNjT2RZhtjAljHFg35y2VdWf7wBUwY0Zu4RuH8fc4mBjy9gDvfXcmiLal2FnOcfHnGkgK09ppPcMt+RkQuBB4BzlPVfLd4AHCue98kCggVkWxVfVBVUwBUNUtE3sO55DbFa3/JIhIMNMa5iW+MMRUKD/EwvE88w/vEs+1gNh+s2M2MlcnMXref1k0bMPKMNlzdN4HmjcL93dSA4cub98E4N+8vwPmlvwK4VlXXe9Xpg3M/ZLCqbq1gOzcBiao6zt1mE1VNE5EQYBowX1UnishvgR5eN++vUNVrKmuj3bw3xpQnv6iYuesPMG35bpZuT8cTJPy6c3NGnNGaX3eOrd8dYFL1zXufnbG49znGAXNxHjeerKrrReQJIElVZ+Fc+ooCPnTvs+9W1aGVbDYMmOuGigeYD7zmLnsDeEdEtgEZwEhfHJcxpu4LC/YwtFcrhvZqxY60o7y/YjcfrUxh/sYDxDYM48rTExhxRmvaW/cx5bIXJO2MxRhzDAqLS1i46SDTk/awcHMqxSVKv/ZNGZHYmkt6tKRBaO0Y3fFksDfvK2HBYow5HgeP5DFjVTLTV+xhZ3oODcOCuax3K0ae0Zoe8Y3rfB9lFiyVsGAxxpwIVeXbHRl8kLSHL9buI6+whE4topyHAXrH06pJA3830ScsWCphwWKMqSlH8gr5bM1eZq5KIWnXIUTgzPZNubxPPEN6tKRReIi/m1hjLFgqYcFijPGF3ek5fLI6hU++S2F72lFCg4O4qEsLLu8Tz686xRIaHNhPlVmwVMKCxRjjS6rKmuRMPvkuhc/W7CX9aAHRESFc1qsVw/vE06d1k4C8H2PBUgkLFmPMyVJYXMLiranM/G4v89bvJ7+ohDZNI7isV0uG9Y6nU4uG/m7iMbNgqYQFizHGH7LyCpm7/gCz1uzlm21pFJcop8U1ZGjvVlzWsxWtm0b4u4mVsmCphAWLMcbfUrPy+WLtPmat2cvKXYcA6Ns2mmG9W3FJj5bE1MJu/S1YKmHBYoypTfZk5PDZ93uZtXovm/Zn4QkSzj41hkt7xHFhlxa1ZuwYC5ZKWLAYY2qrzfuzmLUmhVlr9rInI5cggcS2TRnUrQWDusbRppn/LpdZsFTCgsUYU9upKhv2HWHe+gPM23CAjfuOAHBaXEMGdYtjUNcWdGvV6KQ+XWbBUgkLFmNMoNmdnsO8DfuZt+EASTszKFGIb9KAi7q2YFC3FvRr19TnvS9bsFTCgsUYE8jSs/NZsOkg89bvZ9HWNAqKSmgWGcrF3eO4pHtL+p/im5CxYKmEBYsxpq44ml/EV5tTmb1uH//ddJCcgmKiI0IY1DWOIT3iOKtDTI298W/BUgkLFmNMXZRXWMzXW1KZvXYf8zceJDu/iEbhwVzUNY5LesRxTscYwoKPv5t/C5ZKWLAYY+q6vMJivtmWxhdr9/Plhv0cySuiYVgw91zYkVvPPeW4tum3ESSNMcb4X3iIhwu6tOCCLi0oKOrBkh/SmL12P3GNw322TwsWY4ypJ0KDgxjYuTkDOzf36X58+kyaiAwWkc0isk1EHixn+X0iskFEvheRBSLStszyRiKSLCL/ducjRORzEdkkIutF5BmvujeJSKqIrHY/t/ry2IwxxpTPZ8EiIh7gJWAI0BUYJSJdy1T7DkhU1Z7ADODZMsv/D1hUpuw5VT0N6AOcLSJDvJZ9oKq93c/rNXUsxhhjjp0vz1j6AdtUdbuqFgDvA8O8K6jqQlXNcWeXAQmly0SkL9ACmOdVP0dVF7rTBcAq73WMMcb4ny+DJR7Y4zWf7JZVZAwwG0BEgoB/AH+oqLKINAEuAxZ4FV/pXlabISKtK1hvrIgkiUhSamrqMR2IMcaYY1crxscUkdFAIjDeLboL+EJVkyuoHwxMA15U1e1u8WdAO/ey2pfA2+Wtq6qvqmqiqibGxsbW5GEYY4zBt0+FpQDeZw0JbtnPiMiFwCPAeaqa7xYPAM4VkbuAKCBURLJVtfQBgFeBrao6oXQ7qprutdnX+eX9GmOMMSeBL4NlBdBRRNrjBMpI4FrvCiLSB5gEDFbVg6XlqnqdV52bcG7wP+jOPwk0Bm4ts62WqrrPnR0KbKzpAzLGGFM1nwWLqhaJyDhgLuABJqvqehF5AkhS1Vk4l76igA/dLp93q+rQirYpIgk4ZzebgFXuOv92nwC7W0SGAkVABnCTr47NGGNMxep1ly4ikgrsOs7VY4C0GmxObVDXjqmuHQ/UvWOqa8cDde+Yyjuetqpa4U3qeh0sJ0JEkirrKycQ1bVjqmvHA3XvmOra8UDdO6bjOZ5a8VSYMcaYusOCxRhjTI2yYDl+r/q7AT5Q146prh0P1L1jqmvHA3XvmKp9PHaPxRhjTI2yMxZjjDE1yoLFGGNMjbJgOQ5VjTMTiERkp4isdceyCbjxmkVksogcFJF1XmVNReRLEdnq/oz2Zxurq4JjelxEUrzGHbrEn22sDhFpLSIL3TGY1ovIPW55QH5PlRxPIH9H4SLyrYiscY/pr255exFZ7v7O+0BEQivdjt1jqR53nJktwEU4PTavAEap6ga/NuwEichOnK5zAvLFLhH5FZANTFHV7m7Zs0CGqj7j/gEQraoP+LOd1VHBMT0OZKvqc/5s2/EQkZZAS1VdJSINgZXAcJxeMgLue6rkeK4hcL8jASJVNVtEQoD/AfcA9wEfq+r7IjIRWKOqr1S0HTtjqb4qx5kxJ5+qLsLpysfbMH7q5fptnH/0AaOCYwpYqrpPVVe501k4/fnFE6DfUyXHE7DUke3OhrgfBc7HGYwRjuE7smCpvuqOMxMoFJgnIitFZKy/G1NDWnh1TLofZ+C4umCcO+7Q5EC5bFSWiLTDGQV2OXXgeypzPBDA35GIeERkNXAQZwiSH4DDqlrkVqnyd54Fiyl1jqqejjOU9G/dyzB1hjrXfOvCdd9XgA5Ab2AfzoB4AUVEooCPgHtV9Yj3skD8nso5noD+jlS1WFV74wx10g84rbrbsGCpvmMaZybQqGqK+/MgMBPnf6hAd8C9Dl56PfxgFfVrPVU94P7DLwFeI8C+J/e6/UfAVFX92C0O2O+pvOMJ9O+olKoeBhbijI/VxB1gEY7hd54FS/X9OM6M+2TESGCWn9t0QkQk0r35iIhEAoOAdZWvFRBmATe60zcCn/qxLTWi9Bew63IC6Htybwy/AWxU1X96LQrI76mi4wnw7yhWnGHfEZEGOA8pbcQJmKvcalV+R/ZU2HFwHx+cwE/jzDzl3xadGBE5BecsBZwxet4LtGMSkWnAQJwuvg8AjwGfANOBNjjDI1yjqgFzM7yCYxqIc4lFgZ3A7V73J2o1ETkHWAysBUrc4odx7ksE3PdUyfGMInC/o544N+c9OCce01X1Cfd3xPtAU+A7YLTXiL+/3I4FizHGmJpkl8KMMcbUKAsWY4wxNcqCxRhjTI2yYDHGGFOjLFiMMcbUKAsWYwKUiAwUkf/4ux3GlGXBYowxpkZZsBjjYyIy2h3jYrWITHI7+csWkefdMS8WiEisW7e3iCxzOzCcWdqBoYicKiLz3XEyVolIB3fzUSIyQ0Q2ichU921wY/zKgsUYHxKRLsAI4Gy3Y79i4DogEkhS1W7A1zhv1QNMAR5Q1Z44b3SXlk8FXlLVXsBZOJ0bgtOj7r1AV+AU4GwfH5IxVQquuoox5gRcAPQFVrgnEw1wOlksAT5w67wLfCwijYEmqvq1W/428KHbj1u8qs4EUNU8AHd736pqsju/GmiHMziTMX5jwWKMbwnwtqo+9LNCkT+XqXe8fSt599dUjP2bNrWAXQozxrcWAFeJSHP4cXz3tjj/9kp7i70W+J+qZgKHRORct/x64Gt3dMJkERnubiNMRCJO5kEYUx32140xPqSqG0TkUZzROYOAQuC3wFGgn7vsIM59GHC6JJ/oBsd24Ga3/Hpgkog84W7j6pN4GMZUi/VubIwfiEi2qkb5ux3G+IJdCjPGGFOj7IzFGGNMjbIzFmOMMTXKgsUYY0yNsmAxxhhToyxYjDHG1CgLFmOMMTXq/wG8Pwc2BLZe3gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(model.history.history.keys())\n",
    "# summarize history for accuracy\n",
    "plt.plot(model.history.history['root_mean_squared_error'])\n",
    "plt.plot(model.history.history['val_root_mean_squared_error'])\n",
    "plt.title('model RMSE')\n",
    "plt.ylabel('RMSE')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'val'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Классификация"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Теперь проделаем тоже самое с классификацией. \n",
    "\n",
    "Данные возьмем из соревнования - https://www.kaggle.com/c/quora-insincere-questions-classification/overview  \n",
    "В этом соревновании нужно было предсказывать токсичность вопроса."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "quora = pd.read_csv('quora.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>qid</th>\n",
       "      <th>question_text</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>00002165364db923c7e6</td>\n",
       "      <td>How did Quebec nationalists see their province...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>000032939017120e6e44</td>\n",
       "      <td>Do you have an adopted dog, how would you enco...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0000412ca6e4628ce2cf</td>\n",
       "      <td>Why does velocity affect time? Does velocity a...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>000042bf85aa498cd78e</td>\n",
       "      <td>How did Otto von Guericke used the Magdeburg h...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0000455dfa3e01eae3af</td>\n",
       "      <td>Can I convert montra helicon D to a mountain b...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1306117</th>\n",
       "      <td>ffffcc4e2331aaf1e41e</td>\n",
       "      <td>What other technical skills do you need as a c...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1306118</th>\n",
       "      <td>ffffd431801e5a2f4861</td>\n",
       "      <td>Does MS in ECE have good job prospects in USA ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1306119</th>\n",
       "      <td>ffffd48fb36b63db010c</td>\n",
       "      <td>Is foam insulation toxic?</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1306120</th>\n",
       "      <td>ffffec519fa37cf60c78</td>\n",
       "      <td>How can one start a research project based on ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1306121</th>\n",
       "      <td>ffffed09fedb5088744a</td>\n",
       "      <td>Who wins in a battle between a Wolverine and a...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1306122 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                          qid  \\\n",
       "0        00002165364db923c7e6   \n",
       "1        000032939017120e6e44   \n",
       "2        0000412ca6e4628ce2cf   \n",
       "3        000042bf85aa498cd78e   \n",
       "4        0000455dfa3e01eae3af   \n",
       "...                       ...   \n",
       "1306117  ffffcc4e2331aaf1e41e   \n",
       "1306118  ffffd431801e5a2f4861   \n",
       "1306119  ffffd48fb36b63db010c   \n",
       "1306120  ffffec519fa37cf60c78   \n",
       "1306121  ffffed09fedb5088744a   \n",
       "\n",
       "                                             question_text  target  \n",
       "0        How did Quebec nationalists see their province...       0  \n",
       "1        Do you have an adopted dog, how would you enco...       0  \n",
       "2        Why does velocity affect time? Does velocity a...       0  \n",
       "3        How did Otto von Guericke used the Magdeburg h...       0  \n",
       "4        Can I convert montra helicon D to a mountain b...       0  \n",
       "...                                                    ...     ...  \n",
       "1306117  What other technical skills do you need as a c...       0  \n",
       "1306118  Does MS in ECE have good job prospects in USA ...       0  \n",
       "1306119                          Is foam insulation toxic?       0  \n",
       "1306120  How can one start a research project based on ...       0  \n",
       "1306121  Who wins in a battle between a Wolverine and a...       0  \n",
       "\n",
       "[1306122 rows x 3 columns]"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "quora"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Предобработка точно такая же"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "# словарь\n",
    "vocab = Counter()\n",
    "\n",
    "for text in quora.question_text:\n",
    "    vocab.update(preprocess(text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "273056"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "# отфильтрованный словарь\n",
    "filtered_vocab = set()\n",
    "\n",
    "for word in vocab:\n",
    "    if vocab[word] > 50:\n",
    "        filtered_vocab.add(word)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "14083"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(filtered_vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "# индексируем слова\n",
    "word2id = {'PAD':0}\n",
    "\n",
    "for word in filtered_vocab:\n",
    "    word2id[word] = len(word2id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "id2word = {i:word for word, i in word2id.items()}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "# переводим тексты в последовательности индексов\n",
    "X = []\n",
    "\n",
    "for text in quora.question_text:\n",
    "    tokens = preprocess(text)\n",
    "    ids = [word2id.get(token, 1) for token in tokens]\n",
    "    X.append(ids)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_LEN = max(len(x) for x in X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "134"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "MAX_LEN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Тут уже максимальная длина не такая большая"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "# паддинг\n",
    "X = tf.keras.preprocessing.sequence.pad_sequences(X, maxlen=MAX_LEN)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1306122, 134)"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = quora.target.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_valid, y_train, y_valid = train_test_split(X, y, test_size=0.05)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Разлчия в самой модели тоже минимальные - нужно поменять активацию в последнем слое, лосс и метрику."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs = tf.keras.layers.Input(shape=(MAX_LEN,))\n",
    "\n",
    "# передаем матрицу в эмбединг слой\n",
    "# указываем параметр trainable=False, чтобы вектора не обучались\n",
    "# Пояснение: кажется, что обучение векторов отключать не стоит, так как модель должна взять готовые \n",
    "# эмбединги и подогнать их к нашей задаче. На практитке это однако не работает - эмбединги просто \n",
    "# обучатся заново\n",
    "embeddings = tf.keras.layers.Embedding(input_dim=len(word2id), output_dim=100)(inputs, )\n",
    "mean = tf.keras.layers.Flatten()(embeddings)\n",
    "dense = tf.keras.layers.Dense(64, activation='relu')(mean)\n",
    "dense = tf.keras.layers.Dense(64, activation='relu')(mean)\n",
    "outputs = tf.keras.layers.Dense(1, activation='sigmoid')(dense)\n",
    "\n",
    "model = tf.keras.Model(inputs=inputs, outputs=outputs)\n",
    "optimizer = tf.keras.optimizers.Adam(learning_rate=0.0001)\n",
    "model.compile(optimizer=optimizer,\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=['accuracy'])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Как выбирать batch_size? Размер батча может влиять на качество модели и точно влияет на скорость обучения. Обычно, чем больше батч, тем быстрее будет обучаться модель. Размер батча ограничен памятью видеокарты, поэтому чем лучше gpu, тем больше может быть batch_size. \n",
    "\n",
    "Примерное максимальное значение можно найти быстрым перебором. Если появляется ошибка, то слишком большой batch_size, а если обучается, то можно попробовать поставить побольше."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "63/63 [==============================] - 40s 589ms/step - loss: 0.3748 - accuracy: 0.8755 - val_loss: 0.2258 - val_accuracy: 0.9378\n",
      "Epoch 2/10\n",
      "63/63 [==============================] - 39s 613ms/step - loss: 0.2237 - accuracy: 0.9380 - val_loss: 0.2198 - val_accuracy: 0.9378\n",
      "Epoch 3/10\n",
      "63/63 [==============================] - 39s 616ms/step - loss: 0.2160 - accuracy: 0.9381 - val_loss: 0.2057 - val_accuracy: 0.9378\n",
      "Epoch 4/10\n",
      "63/63 [==============================] - 44s 696ms/step - loss: 0.2001 - accuracy: 0.9380 - val_loss: 0.1826 - val_accuracy: 0.9382\n",
      "Epoch 5/10\n",
      "63/63 [==============================] - 44s 698ms/step - loss: 0.1766 - accuracy: 0.9390 - val_loss: 0.1600 - val_accuracy: 0.9414\n",
      "Epoch 6/10\n",
      "63/63 [==============================] - 43s 675ms/step - loss: 0.1565 - accuracy: 0.9422 - val_loss: 0.1463 - val_accuracy: 0.9443\n",
      "Epoch 7/10\n",
      "63/63 [==============================] - 43s 676ms/step - loss: 0.1438 - accuracy: 0.9453 - val_loss: 0.1380 - val_accuracy: 0.9465\n",
      "Epoch 8/10\n",
      "63/63 [==============================] - 44s 702ms/step - loss: 0.1362 - accuracy: 0.9475 - val_loss: 0.1322 - val_accuracy: 0.9485\n",
      "Epoch 9/10\n",
      "63/63 [==============================] - 42s 671ms/step - loss: 0.1303 - accuracy: 0.9493 - val_loss: 0.1282 - val_accuracy: 0.9498\n",
      "Epoch 10/10\n",
      "63/63 [==============================] - 42s 673ms/step - loss: 0.1267 - accuracy: 0.9502 - val_loss: 0.1253 - val_accuracy: 0.9506\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x1ae02cd68>"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# как выглядит ошибка при слишком большом батче\n",
    "model.fit(X_train, y_train, \n",
    "          validation_data=(X_valid, y_valid),\n",
    "          batch_size=20000,\n",
    "         epochs=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# model.fit(X_train, y_train, \n",
    "#           validation_data=(X_valid, y_valid),\n",
    "#           batch_size=512,\n",
    "#          epochs=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['loss', 'accuracy', 'val_loss', 'val_accuracy'])\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEWCAYAAABxMXBSAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Z1A+gAAAACXBIWXMAAAsTAAALEwEAmpwYAAA150lEQVR4nO3deXwV5dn/8c+VPZAEQsKasIR9lyUCKiouKKCCYutStbWL1qpVa22Lffq0Pmp/2uexm7vWYrVVqcUNlUVBwA2RsCbsi0B2spAEAlnP9ftjBjjEACeYk8lyvV+vvJgz27nmAOebue+Ze0RVMcYYYwIV4nUBxhhjWhYLDmOMMQ1iwWGMMaZBLDiMMcY0iAWHMcaYBrHgMMYY0yAWHMachIj8Q0QeDnDd3SJycbBrMsZrFhzGGGMaxILDmDZARMK8rsG0HhYcpsVzm4h+ISIbRKRcRP4uIl1FZIGIHBCRxSIS77f+dBHZKCIlIrJMRIb4LRstImvc7f4NRNV5r8tFZJ277eciMjLAGi8TkbUiUiYimSLyQJ3lE939lbjLb3bnR4vIH0Vkj4iUisin7rxJIpJVz+dwsTv9gIjMFZF/iUgZcLOIjBORFe575IrIkyIS4bf9MBH5UESKRSRfRH4tIt1E5JCIJPitN0ZECkQkPJBjN62PBYdpLa4GJgMDgSuABcCvgc44/87vAhCRgcBrwD3usvnAuyIS4X6Jvg38E+gE/MfdL+62o4HZwI+BBOA5YJ6IRAZQXznwXaAjcBnwExG50t1vb7feJ9yaRgHr3O0eA8YCZ7s1/RLwBfiZzADmuu/5ClAL/AxIBM4CLgJud2uIBRYDC4EeQH9giarmAcuAa/z2exMwR1WrA6zDtDIWHKa1eEJV81U1G/gEWKmqa1W1AngLGO2udy3wvqp+6H7xPQZE43wxTwDCgb+oarWqzgVW+b3HrcBzqrpSVWtV9SWg0t3upFR1maqmq6pPVTfghNf57uLvAItV9TX3fYtUdZ2IhAA/AO5W1Wz3PT9X1coAP5MVqvq2+56HVXW1qn6hqjWquhsn+I7UcDmQp6p/VNUKVT2gqivdZS8BNwKISChwPU64mjbKgsO0Fvl+04freR3jTvcA9hxZoKo+IBNIcpdl6/Ejf+7xm+4N/Nxt6ikRkRKgp7vdSYnIeBFZ6jbxlAK34fzmj7uPnfVslojTVFbfskBk1qlhoIi8JyJ5bvPV/wugBoB3gKEikoJzVleqql+eZk2mFbDgMG1NDk4AACAigvOlmQ3kAknuvCN6+U1nAr9X1Y5+P+1U9bUA3vdVYB7QU1U7AM8CR94nE+hXzzaFQMUJlpUD7fyOIxSnmctf3aGvnwG2AANUNQ6nKc+/hr71Fe6etb2Oc9ZxE3a20eZZcJi25nXgMhG5yO3c/TlOc9PnwAqgBrhLRMJFZCYwzm/bvwG3uWcPIiLt3U7v2ADeNxYoVtUKERmH0zx1xCvAxSJyjYiEiUiCiIxyz4ZmA38SkR4iEioiZ7l9KtuAKPf9w4HfAKfqa4kFyoCDIjIY+InfsveA7iJyj4hEikisiIz3W/4ycDMwHQuONs+Cw7QpqroV5zfnJ3B+o78CuEJVq1S1CpiJ8wVZjNMf8qbftmnALcCTwH5gh7tuIG4HHhSRA8BvcQLsyH73AtNwQqwYp2P8DHfxfUA6Tl9LMfAHIERVS919voBztlQOHHeVVT3uwwmsAzgh+G+/Gg7gNENdAeQB24EL/JZ/htMpv0ZV/ZvvTBsk9iAnY0wgROQj4FVVfcHrWoy3LDiMMackImcCH+L00Rzwuh7jLWuqMsaclIi8hHOPxz0WGgbsjMMYY0wD2RmHMcaYBmkTA58lJiZqnz59vC7DGGNalNWrVxeqat37g9pGcPTp04e0tDSvyzDGmBZFROq99NqaqowxxjSIBYcxxpgGseAwxhjTIG2ij6M+1dXVZGVlUVFR4XUpQRUVFUVycjLh4fbMHWNM42izwZGVlUVsbCx9+vTh+MFQWw9VpaioiKysLFJSUrwuxxjTSrTZpqqKigoSEhJabWgAiAgJCQmt/qzKGNO02mxwAK06NI5oC8dojGlabbapyhhjWqXaaijcDvkZkJcOk2ZBRPtGfQsLDo+UlJTw6quvcvvttzdou2nTpvHqq6/SsWPH4BRmjGk5Du+HvAw3JDIgPx3dtwWpdR5LX004lYOvJqbXqEZ9WwsOj5SUlPD0009/LThqamoICzvxX8v8+fODXZoxprnx+WD/V84ZxNGQyIDSY4+VPxAWzzZ6k1Y5mU2+XmylN+FdBvHnqH70b+RyghocIjIF+CsQCrygqo/WWd4b59GYnXGebnajqma5y2pxnnwGsFdVp7vzU4A5QAKwGrjJfXJbizJr1ix27tzJqFGjCA8PJyoqivj4eLZs2cK2bdu48soryczMpKKigrvvvptbb70VODZ8ysGDB5k6dSoTJ07k888/JykpiXfeeYfo6GiPj8wY841UlUP+JsjbcCwk9m2CqoMAqISyP7o326Q/X8j5rKlMYrOvN5VRiYzpHc/YXvFc2zuekT07EhMZnK/4oA2rLiKhOM9FnozzSMtVwPWquslvnf8A76nqSyJyIfB9Vb3JXXZQVWPq2e/rwJuqOkdEngXWq+ozJ6slNTVV645VtXnzZoYMGQLA/7y7kU05Zd/gaL9uaI84fnfFsBMu3717N5dffjkZGRksW7aMyy67jIyMjKOXzRYXF9OpUycOHz7MmWeeyfLly0lISDguOPr3709aWhqjRo3immuuYfr06dx4441fey//YzXGNBOqUJbtBENeOuSnO9PFuwDne9kXEUdx7EB2hvRh5aEeLCnpypbaJCqJoF/n9oztHc/Y3vGM6RVPv84xhIQ07sUwIrJaVVPrzg/mGcc4YIeq7nILmAPMADb5rTMUuNedXgq8fbIdinOJ0IU4z00GeAl4ADhpcLQE48aNO+5ei8cff5y33noLgMzMTLZv305CQsJx26SkpDBq1CgAxo4dy+7du5uqXGNMQ9RUQsEWv5Bw/6woObqKxqdQEjeIXR0ms6oiiQWFiawvi4MyITo8lFE9O3LuyHju6R3P6F4d6dguwrPDCWZwJAGZfq+zgPF11lkPzMRpzroKiBWRBFUtAqJEJA2oAR5V1bdxmqdKVLXGb59J9b25iNwK3ArQq1evkxZ6sjODptK+/bGrHpYtW8bixYtZsWIF7dq1Y9KkSfXeixEZGXl0OjQ0lMOHDzdJrcaYk6iugMyVkLv+WEgUbgOf+7UVFg1dh3F44BXsCk1hdUUSHxYmsDK3hqpcHwDJ8dGM7RfP1e7ZxOBusYSFNp+7J7zuHL8PeFJEbgY+BrKBWndZb1XNFpG+wEcikg6UBrpjVX0eeB6cpqpGrboRxMbGcuBA/U/hLC0tJT4+nnbt2rFlyxa++OKLJq7OGNMgZTmwbRFs/wB2LYPqQ8782B7QbQS+AVPIjurHmspklhfGkra3jL07nXUiQkMYntSO7511rNmpS1yUd8cSgGAGRzbQ0+91sjvvKFXNwTnjQERigKtVtcRdlu3+uUtElgGjgTeAjiIS5p51fG2fLUVCQgLnnHMOw4cPJzo6mq5dux5dNmXKFJ599lmGDBnCoEGDmDBhgoeVGmO+xlcL2avdsFjknFkAdOgFo75Dea8LWaP9+TIP1uzdz7pPSiivqgXK6Rxbw9he8dw0oTdjesczPCmOyLBQTw+noYLZOR6G0zl+Ec6X+yrgO6q60W+dRKBYVX0i8nugVlV/KyLxwCFVrXTXWQHMUNVNbof6G36d4xtU9emT1XKqzvHWri0dqzFBc3g/7FjinFXsWAyHikBCodcEGHAJB3tfxIK8DszbkMtnOwrxKYSGCEO6xzKm17GzieT46BYzokOTd46rao2I3Akswrkcd7aqbhSRB4E0VZ0HTAIeERHFaaq6w918CPCciPhwhkV51O9qrF8Bc0TkYWAt8PdgHYMxpg1TdTq0jzRB7f0CtBaiO8GAyTDgEip6X8BHe6qYty6HjxZmUVWzl56dornt/H5MHJDIGckdaR+kS2K9FNQjUtX5wPw6837rNz0XmFvPdp8DI06wz104V2wZY0zjqj4Muz+FbQth2wdQuteZ33UETPwZDLyUmm6j+XTXfuatz+GDuWkcrKwhMSaSG8b3YvoZPRjVs2OLOaM4Xa0vCo0xpiFKs/w6tpdDzWEIbwd9J8G598KAS/DF9mDN3v28szqH+enLKCqvIjYqjGkjujFjVBIT+iYQ2sj3UDRnFhzGmLbFVwtZq5yw2LYI9rndrh17w5ibYMCl0GciGhbJ5twDzPs8h3fXLyW75DCRYSFcPLQr08/owaRBnVtcp3ZjseAwxrR+h4rdju1FTsf24f1ux/ZZMPkhGHgpJA4EEfYUlTPv40zmrc9h+76DhIYI5w1I5L5LBzJ5aLegDePRktgnYIxpfVSd8Z2OnFVkfQnqg3aJMHAKDLgE+l0I0R0B2FdWwbuf7Wbe+hzWZ5YAMK5PJx6+cjjTRnSnU3vv7tJujiw4WoiYmBgOHjzodRnGNF8+H+xaClveczq2y7Kc+d1Gwrk/dwKjx2gIcZqXSg9Vs3DVXt5Zl8MXu4rwKQzrEcf9Uwdz+Rk9SOpoA4aeiAWHMaZlqzoE61+DL56Bou0Q3h76XQDn/9I5s4jrfnTVw1W1LN6cw7z1OSzbuo/qWiUlsT13XjiA6Wf0oH+Xr42rauphweGRWbNm0bNnT+64w7l15YEHHiAsLIylS5eyf/9+qqurefjhh5kxY4bHlRrTTB3Igy//Bml/d/osuo+CmS/A0OkQdmwct+paH59sL2Deuhw+2JTPoapausZF8r2z+jB9VA9GJHVo9ZfPNjYLDoAFs44NGdBYuo2AqY+ecPG1117LPffcczQ4Xn/9dRYtWsRdd91FXFwchYWFTJgwgenTp9s/amP85W6AL56G9LnOwIGDL4Oz7nA6ut3/Kz6fsmp3Me+sz2FBei77D1XTITqcGaN6MP2MJMaldGpTl882NgsOj4wePZp9+/aRk5NDQUEB8fHxdOvWjZ/97Gd8/PHHhISEkJ2dTX5+Pt26dfO6XGO85fM591mseBJ2f+I0R6X+AMb/GBL6HV1tU04Zb6/L5t31OeSWVhAdHspk9/LZ8wZ2JiKs+Yww25JZcMBJzwyC6dvf/jZz584lLy+Pa6+9lldeeYWCggJWr15NeHg4ffr0qXc4dWPajKpDsP5Vt/9iB8QlweQHYcx3IToeAFXlsx1FPLt8J5/uKCQ8VDh/YGdmTR3M5KFdaRdhX3ONzT5RD1177bXccsstFBYWsnz5cl5//XW6dOlCeHg4S5cuZc+ePV6XaIw3ynJh1d8gbbbTf9FjNFz9dxg6A0LDAaj1KYs25vHMsp2kZ5fSOTaSWVMHc21qT+Lt8tmgsuDw0LBhwzhw4ABJSUl0796dG264gSuuuIIRI0aQmprK4MGDvS7RmKaVux5WPA0Zb/j1X9zpjEDr9l9U1tTy5ppsnv94F18VltMnoR2PzBzBVaOTiApvm3dyNzULDo+lpx/rlE9MTGTFihX1rmf3cJhWy+dz7uhe8dSx/oszf+j0X3Tqe3S1AxXVvLpyL3//9Cv2HahkRFIHnr5hDJcO62Yd3U3MgsMY442qcljn9l8U74S4ZGf4jzHfPXpHN0DBgUpe/Owr/vnFHg5U1DCxfyJ/vnYUZ/dLsCsOPWLBYYxpWmU58OXzkPYiVJRAjzFf678A2FNUzvMf7+I/q7OorvUxbXh3bju/HyOSO3hXuwHaeHCoaqv/jSVYT3g0psFy1jn3X2S84YwbdaT/ouf4o/0XABnZpTy7fCfz03MJCwnh6rHJ3HpeX1IS23tXuzlOmw2OqKgoioqKSEhovae7qkpRURFRUc37wfemFfP5nIcirXgK9nwKETFw5i1u/0XK0dVUlRW7inhm2U4+2V5ITGQYt5zXlx+ek0KXOPv329y02eBITk4mKyuLgoICr0sJqqioKJKTk70uw7Q1R/svnobiXSfsv/D5lA82OZfUrs8qJTEmkl9OGcQN43vTITr8xPs3nmqzwREeHk5KSsqpVzTGBK402+m/WP0Pp/8iaSx8azYMmQGhx75uKmtqeXttNs99vItdBeX0TmjH768aztVjku2S2hagzQaHMaYR5a6Hz5+EjW+6/ReXu/0X447rvzhYWcNrK/fywqe7yC+rZFiPOJ78zmimDu9ul9S2IBYcxpjTl7selv0Btr7v9F+Mu9Xpv4jvc9xqhQcr+cdnu3l5xW7KKmo4u18Cj337DCb2T2y1fYytmQWHMabhcjfA8j84D02K7ACTfu0Ehl//BUBm8SGe/3gXr6dlUlXrY8qwbtx2fj/O6Nmx3t2alsGCwxgTuLx0WPaoX2DcD+Nv+1pgbMop49nlO3k/PZcQgavHJHPLeX3p19kelNQaWHAYY04tLwOWPwqb34XIODh/Fkz4yXGBoaqs/KqYZ5fvZNnWAtpHhPKjiSn8YGIKXe2S2lbFgsMYc2L5G50zjM3z3MD4lRsY8UdXUVWWbS3g8Y+2s3ZvCYkxEfzi0kHcOL43HdrZJbWtkQWHMebr8jc5Zxib3oGIWDjvl3DW7ccFBsCugoP8z7ubWL6tgJ6donnoyuF8e6xdUtvaWXAYY47Zt9k5w9j0thsYv4AJt0O7TsetdrCyhic+2s7sT78iMiyU31w2hO+d3YfwUHvCXltgwWGMcQJj+R9g49sQ0R7Ovc95jnedwFBV3lmXwyMLNpNfVsnVY5L51dRBdIm1Poy2xILDmLZs3xY3MN5yA+Ne58a9OoEBsDGnlAfmbWTV7v3uszDGMrZ3fD07Na1dUINDRKYAfwVCgRdU9dE6y3sDs4HOQDFwo6pm+S2PAzYBb6vqne68ZUB34LC72iWqui+Yx2FMq1Ow1QmMjDedwJj4Mzj7p/UGRsmhKh77YCuvrtxLx3YRPDJzBNek9rQ7vduwoAWHiIQCTwGTgSxglYjMU9VNfqs9Brysqi+JyIXAI8BNfssfAj6uZ/c3qGpakEo3pvUq2OYGxhsQ3g4m3gNn/RTaJ3xt1Vqf8tqXe3nsg62UHa7mpgm9uXfyILtSygT1jGMcsENVdwGIyBxgBs4ZxBFDgXvd6aXA20cWiMhYoCuwEEgNYp3GtH6F253ASJ/rBMY5d8PZd9UbGABpu4v53byNbMwpY1xKJ/5n+jCGdI9r4qJNcxXM4EgCMv1eZwHj66yzHpiJ05x1FRArIgnAfuCPwI3AxfXs+0URqQXeAB7Wep5WJCK3ArcC9OrV65sdiTEtVeEO9wxjLoRFwTl3uYGRWO/q+8oqeGTBFt5am023uCgev340V4zsbuNJmeN43Tl+H/CkiNyM0ySVDdQCtwPzVTWrnn+wN6hqtojE4gTHTcDLdVdS1eeB5wFSU1PtMXimbSncAR//H6S/7gTGWXc6ZxknCIyqGh8vfvYVjy/ZTnWtcvukftxxQX/aR3r9FWGao2D+q8gGevq9TnbnHaWqOThnHIhIDHC1qpaIyFnAuSJyOxADRIjIQVWdparZ7rYHRORVnCaxrwWHMW1S0U4nMDb8G0IjnUtqz74bYjqfcJOPtxXwwLsb2VVQzoWDu/Dby4fSxx7Tak4imMGxChggIik4gXEd8B3/FUQkEShWVR9wP84VVqjqDX7r3AykquosEQkDOqpqoYiEA5cDi4N4DMa0DEU74ePH3MCIcG7aO+duiOlywk0yiw/x0Hub+GBTPn0S2jH75lQuHNy1CYs2LVXQgkNVa0TkTmARzuW4s1V1o4g8CKSp6jxgEvCIiChOU9Udp9htJLDIDY1QnND4W7COwZhmb/8epw9j/Rw3MH5yysA4XFXLM8t38tzynYSI8ItLB/Gjc1OIDLNhQkxgpJ5+5VYnNTVV09Ls6l3TivhqYeWz8NHDzhP3Un/oBEbsic8YVJWFGXk8/P5msksOc8UZPfj1tMF07xDdhIWblkREVqvq165qtZ4vY1qa/E0w76eQnQYDLoXL/wQdkk+6yfb8Azzw7kY+21HE4G6xzLl1AhP61n8prjGnYsFhTEtRUwmf/BE++RNExcHMF2DEt457pnddZRXV/HXxdl76fDftIkL5n+nDuGF8L8JsMELzDVhwGNMSZH7pnGUUbIER18CUR0948x6Az6e8sSaLPyzcSlF5Jded2ZP7LhlEQkxkExZtWisLDmOas8qD8NFDsPI5iEuCG+bCgMkn3WRDVgm/m7eRtXtLGN2rI7NvTmVkcsemqde0CRYcxjRXOxbDuz+D0kw480dw8e8gMvaEqxcdrOT/Fm3l32mZJLSP5LFvn8HM0UmE2GCEppFZcBjT3BwqhoX3w4Y5kDgQfrAQek044eo1tT7+9cUe/vThNg5V1fLDc1K46+IBxEXZYIQmOCw4jGkuVGHjmzD/l1BR4jx979z7IPzED0lavWc///VWOlvyDjCxfyIPTB9K/y4nPisxpjFYcBjTHJRmw/s/h20LoMdomP4OdBt+wtVrfcpTS3fwl8Xb6N4hmmdvHMOlw7rZYISmSVhwGOMlnw/W/AM+/B3UVsMlD8P4n0Doif9r5pQc5p5/r+PLr4q5clQPHrpyOLHWLGWakAWHMV4p3AHv3gV7PoOU8+CKv0KnvifdZGFGHr96YwM1tT7+dM0ZzBxz8hv/jAkGCw5jmlptNXz+BCx71BnyfPoTMPqmk97IV1Fdy0PvbeKVlXsZkdSBx68fTYqNYGs8YsFhTFPKWefcyJe3AYZcAdMeg9huJ91ka94BfvraGrblH+TH5/Xl55cMIiLM7vw23rHgMKYpVB92zjA+f8J5mNI1L8PQGSfdRFX51xd7ePj9zcRGhfPyD8Zx3sATP1fDmKZiwWFMsO3+FObdBcU7YfSNTgd4dPxJN9lfXsUv39jAh5vymTSoM499+wwSbbgQ00xYcBgTLBWlztVSq1+E+D7w3Xeg76RTbrZiZxE/+/c6isor+e/Lh/L9s/vY3d+mWbHgMCYYtsyH9++Fg/nO874v+DVEnLwzu7rWx18Xb+epZTtISWjPC987h+FJHZqoYGMCZ8FhTGM6uA8W/BI2vgVdhsF1r0DS2FNulll8iLvnrGXN3hKuSU3md1cMo32k/fc0zZP9yzSmMag6j29ddD9UlcMFv3GeyBcWccpN312fw6/fTAfgietHc8UZPYJdrTHfiAWHMd/U/j3w3j2w8yPoOd65L6PzoFNudqiqhgfmbeT1tCxG9+rI49eNpmendsGv15hvyILDmNPlq4Uvn4clDzk37039P2f485BT32ORkV3KXa+t5auicu68oD93XzyAcHsqn2khLDiMOR0F2+Cd2yFrFfSfDJf/GTr2POVmPp8y+7Ov+N+FW4lvH84rPxrP2f0Sm6BgYxqPBYcxDaHqXF678NfOcOdXPQ8jrznpcCFHFB6s5L7/rGfZ1gImD+3K/149kvj2p+4DMaa5seAwJlDlRc6ghFvec+7HuPJZiOse0KYfbyvg3tfXU1ZRzUMzhnHjhN42BLppsSw4jAnEzqXw1m1wqMi583vCHQH1ZVTV+PjjB1t57uNdDOwaw79+NI7B3eKaoGBjgseCw5iTqamEJQ/Ciiedx7je8B/oPjKgTb8qLOfuOWvZkFXKDeN78ZvLhhIdERrkgo0JPgsOY06kYBu88UNnJNvUH8Alv4eIwC6XfXNNFv/9dgZhoSE8e+NYpgw/+Qi4xrQkFhzG1HVcB3g0XPcqDL4soE0PVFTz329n8Pa6HMaldOIv146iR8foIBdsTNOy4DDGX3mR87yMre9D3wvgqmdP+byMI9ZllnDXa2vJ2n+IeycP5I4L+hNqgxOaViiodxyJyBQR2SoiO0RkVj3Le4vIEhHZICLLRCS5zvI4EckSkSf95o0VkXR3n4+LXZpiGsvOj+CZs2HHh06z1I1vBhQaPp/y9LIdfOuZz6n1Ka//+CzuumiAhYZptYIWHCISCjwFTAWGAteLyNA6qz0GvKyqI4EHgUfqLH8I+LjOvGeAW4AB7s+URi7dtDU1lbDov+CfV0FUB/jREjj7zoCumsovq+Cm2Sv534VbuXRYN+bfdS6pfTo1QdHGeCeg4BCRN0XkMhFpSNCMA3ao6i5VrQLmAHUfeTYU+MidXuq/XETGAl2BD/zmdQfiVPULVVXgZeDKBtRkzPEKtsILFzlXTaX+EG5dFvBVU0s25zP1r5+wZk8Jj84cwZPfGU2HduHBrdeYZiDQPo6nge8Dj4vIf4AXVXXrKbZJAjL9XmcB4+ussx6YCfwVuAqIFZEEYD/wR+BG4OI6+8yqs8+kAI/BmGNUIW22c6YR0Q6uew0GTzvlZtW1PpZszue1LzNZvq2AId3jeOL60fTvEtMERRvTPAQUHKq6GFgsIh2A693pTOBvwL9Utfo03/8+4EkRuRmnSSobqAVuB+aratbpdmGIyK3ArQC9evU6zfJMq1Re6HaAz4d+F8KVz5yyL2N3YTlzVmUyd3UWhQcr6d4hinsnD+TW8/oSFW73Zpi2JeCrqtwzgRuBm4C1wCvAROB7wKR6NskG/Ed9S3bnHaWqOThnHIhIDHC1qpaIyFnAuSJyOxADRIjIQZwzk+ST7dNv388DzwOkpqZqoMdpWrmdHzl3gB/eD5c+AuNvO2FfRmVNLYs25jPny718vrOI0BDhwsFduH5cT84f2MU6v02bFVBwiMhbwCDgn8AVqprrLvq3iKSdYLNVwAARScH5cr8O+E6d/SYCxarqA+4HZgOo6g1+69wMpKrqLPd1mYhMAFYC3wWeCOQYTBvnfwd458Fw4xvQbUS9q+7Yd4DXvszkzTVZ7D9UTXJ8NPddMpBvp/aka1xUExduTPMT6BnH46q6tL4Fqpp6gvk1InInsAgIBWar6kYReRBIU9V5OGcqj4iI4jRV3RFALbcD/wCigQXujzEntm8LvPEjyE93npcx+aGv3QF+uKqW+em5zFm1l1W79xMeKlwytBvXjevJOf0SCbGzC2OOEufipFOsJHIH8Iqqlriv44HrVfXp4JbXOFJTUzUt7UQnRqbVUoW0v7sd4O1hxtMw6PirtzfllDFn1V7eWpvNgYoaUhLbc92ZPbl6bDKJMZEeFW5M8yAiq+s7OQj0jOMWVX3qyAtV3S8it+BcbWVM81NeCO/cCdsWQL+L3A7wrs6iyhreXZ/Da6syWZ9ZQkRYCNOGd+O6cb0Yn9LJhjs35hQCDY5QERH33okjN/fZE2hM87RjCbz9k+M6wFWEDZklzFm1l3nrciivqmVg1xh+e/lQZo5JomM7++dsTKACDY6FOB3hz7mvf+zOM6b5qKmExQ/AF0+7HeBvUtZxEO+s3MurX2ayObeM6PBQLh/ZnevG9WJMr452dmHMaQg0OH6FExY/cV9/CLwQlIqMOR37tjhDoOdnoGfewppB9/LqxwW8n76Yimofw3rE8dCVw5kxqgdxUXZ3tzHfRKA3APpwxoh6JrjlGNNAqrDqBfjgN/giYlh8xuP839Y+bP9kLTGRYcwck8z1Z/ZiRHIHrys1ptUI9D6OATgDEA4Fjl7Irqp9g1SXMadWXoi+czuybRGb2o/jRyU/IGdlHKN6hvGHq0dw+cgetI+0JwcY09gC/V/1IvA74M/ABTjjVgV1SHZjTqYkfQER795BWFUpj1TfxJvll3HV+F5cN66nPdPbmCALNDiiVXWJe2XVHuABEVkN/DaItRnXvrIKskoOe10GqqCq1PqUWlV8Ptw//ec5f9b6FJ8qtT7qmVdn+dfm+e2rVgmrPURUVTHtqotpV72fpP1fcFHZO2z1JfO3zg9x9jnn86sR3W3MKGOaSKDBUekOqb7dvRs8G2cMKRNktT5l+pOfkVdW4XUpjSaCajpRRoIcIEFKSaCMBCkjUcpIkFIS3elOHKATpURL1df2sarLt+h05aM81qOzB0dgTNsWaHDcDbQD7sJ5uNIFOIMbmiBbtbuYvLIK7p08kJHNoIM3RITQEDn6Z2gIhKiPiOpSIiqKiKgsJryyiPDDRYRVFBF2uJCwiiJCDxcRcqiQkMOFhFSW1btvDY1A2neGdgnQPgXad4b2ie6fR6YTIS6JMwN8nKsxpvGdMjjcm/2uVdX7gIM4/Rttw4b/QPFOT0s4vGUf94aXcpv2IyLX424l9cHhEigvcH8KnT8PFzvL6pIQNwTcMIgf9fUQ8HstkXFg91UY0+ydMjhUtVZEJjZFMc1O+uuw/YNTrxdEFwAXhAKfelrGMVEd3CBIhIR+0GvCicMgOh5CrN/BmNYm0KaqtSIyD/gPUH5kpqq+GZSqmovvvO7p26/eU8zVz67gr9eOYsaoZvKgQzsjMKbNCzQ4ooAi4EK/eQq07uDw+EtyfkY+EaGhXDikq+e1GGPMEYHeOd52+jWaCVVlQXou5w5IJNaGyDDGNCOB3jn+Is4ZxnFU9QeNXpEBYH1WKTmlFdx7ySCvSzHGmOME2lT1nt90FHAVkNP45ZgjFmTkEhYiTB7S1etSjDHmOIE2Vb3h/1pEXqP5XOfT6jjNVHmc0z+RDu2smcoY07yc7o0BA4AujVmIOWZjThl7iw8xdbjd5GaMaX4C7eM4wPF9HHk4z+gwQbAwI4/QEOGSYRYcxpjmJ9CmqthgF2Icqsr89Fwm9O1Ep/b2OFNjTPMTUFOViFwlIh38XncUkSuDVlUbti3/ILsKy5k6vLvXpRhjTL0C7eP4naqWHnmhqiU4z+cwjWx+ei4icMkwu5rKGNM8BRoc9a1nj1YLgoUZeZzZpxNdYqNOvbIxxngg0OBIE5E/iUg/9+dPwOpgFtYW7dh3kK35B5hmV1MZY5qxQIPjp0AV8G9gDlAB3BGsotqqhRm5AEyx/g1jTDMW6FVV5cCsINfS5s1Pz2NMr45062DNVMaY5ivQq6o+FJGOfq/jRWRR0Kpqg/YUlbMpt4xpI+xswxjTvAXaVJXoXkkFgKrux+4cb1QLMvIAmGL9G8aYZi7Q4PCJSK8jL0SkD/WMlluXiEwRka0iskNEvtbUJSK9RWSJiGwQkWUikuw3f42IrBORjSJym982y9x9rnN/WkWALUjPZWRyB5Lj23ldijHGnFSgl9T+F/CpiCwHBDgXuPVkG7jPKn8KmAxkAatEZJ6qbvJb7THgZVV9SUQuBB4BbgJygbNUtVJEYoAMd9sjI/LeoKppAdbe7GXtP8T6rFJ+NWWw16UYY8wpBXTGoaoLgVRgK/Aa8HPg8Ck2GwfsUNVdqlqFczXWjDrrDAU+cqeXHlmuqlWqWunOjwy0zpZqodtMZYMaGmNagkA7x38ELMEJjPuAfwIPnGKzJCDT73WWO8/femCmO30VECsiCe579hSRDe4+/uB3tgHwottM9d8i9T9TVURuFZE0EUkrKCg45TF6aUFGHkO7x9Ensb3XpRhjzCkF+pv83cCZwB5VvQAYDZQ0wvvfB5wvImuB84FsoBZAVTNVdSTQH/ieiBwZg+MGVR2B01x2Lk7T1teo6vOqmqqqqZ07d26EUoMjr7SC1Xv229mGMabFCDQ4KlS1AkBEIlV1C3CqZ5pmAz39Xie7845S1RxVnamqo3H6UfC/euvIOkAGTkigqtnunweAV3GaxFqsIzf9TbXLcI0xLUSgwZHl3sfxNvChiLwD7DnFNquAASKSIiIRwHXAPP8VRCRRRI7UcD8w252fLCLR7nQ8MBHYKiJhIpLozg8HLscJlRZrQUYeA7vG0L9LjNelGGNMQAK9c/wqd/IBEVkKdAAWnmKbGhG5E1gEhAKzVXWjiDwIpKnqPGAS8IiIKPAxx4YxGQL80Z0vwGOqmi4i7YFFbmiEAouBvwV+uM1LwYFKvtxdzF0XDvC6FGOMCViDR7hV1eUNWHc+ML/OvN/6Tc8F5taz3YfAyHrmlwNjG1Jvc7ZoYx6qMHWE9W8YY1qOVn2Za3O3ICOXvontGdTVHrBojGk5LDg8UlxexRe7ipk6ohsnuKLYGGOaJQsOj3y4KY9an9ojYo0xLY4Fh0fmp+fRs1M0w3rEeV2KMcY0iAWHB0oPVfPZjkKmDe9uzVTGmBbHgsMDizfnU+NTu+nPGNMiWXB4YEFGLj06RHFGcgevSzHGmAaz4GhiByqq+XhbIVOsmcoY00JZcDSxj7bso6rWxzS76c8Y00JZcDSxBel5dImNZEyveK9LMcaY02LB0YTKK2tYunUfU4d3IyTEmqmMMS2TBUcTWra1gMoaH1Pspj9jTAtmwdGEFmTkktA+gnEpnbwuxRhjTpsFRxOpqK7loy37uHR4N0KtmcoY04JZcDSR5dsKOFRVyzRrpjLGtHAWHE1kQXouHduFM76vNVMZY1o2C44mUFlTy5LN+7hkaFfCQ+0jN8a0bPYt1gQ+21HIgcoaG5vKGNMqWHA0gfnpecRGhXFOv0SvSzHGmG/MgiPIqmt9fLgpn8lDuhIRZh+3Mabls2+yIFuxs4jSw9XWTGWMaTUsOIJsQUYu7SNCOXeANVMZY1oHC44gqqn1sWhjPhcO6UpUeKjX5RhjTKOw4AiiL78qpri8imnDbQh1Y0zrYcERRAsy8ogOD2XSoC5el2KMMY3GgiNIan3Kwo15XDC4M9ER1kxljGk9LDiCZPWe/RQcqLQh1I0xrY4FR5DMT88lIiyECwdbM5UxpnWx4AgCn09ZtDGP8wd2JiYyzOtyjDGmUQU1OERkiohsFZEdIjKrnuW9RWSJiGwQkWUikuw3f42IrBORjSJym982Y0Uk3d3n4yLS7B5usS6rhNzSCqaNsKupjDGtT9CCQ0RCgaeAqcBQ4HoRGVpntceAl1V1JPAg8Ig7Pxc4S1VHAeOBWSLSw132DHALMMD9mRKsYzhdC9JzCQ8VLhzc1etSjDGm0QXzjGMcsENVd6lqFTAHmFFnnaHAR+700iPLVbVKVSvd+ZFH6hSR7kCcqn6hqgq8DFwZxGNoMFVlfnoeE/sn0iE63OtyjDGm0QUzOJKATL/XWe48f+uBme70VUCsiCQAiEhPEdng7uMPqprjbp91in3ibn+riKSJSFpBQcE3PphAZWSXkV1y2MamMsa0Wl53jt8HnC8ia4HzgWygFkBVM90mrP7A90SkQe0+qvq8qqaqamrnzp0bu+4Tmp+RS1iIcMlQa6YyxrROwbzkJxvo6fc62Z13lHsWMRNARGKAq1W1pO46IpIBnAt85u7nhPv0kqqyID2Xs/ol0LFdhNflGGNMUATzjGMVMEBEUkQkArgOmOe/gogkisiRGu4HZrvzk0Uk2p2OByYCW1U1FygTkQnu1VTfBd4J4jE0yObcA+wuOsRUu+nPGNOKBS04VLUGuBNYBGwGXlfVjSLyoIhMd1ebBGwVkW1AV+D37vwhwEoRWQ8sBx5T1XR32e3AC8AOYCewIFjH0FALM3IJEbhkmDVTGWNar6Denaaq84H5deb91m96LjC3nu0+BEaeYJ9pwPDGrbRxzM/IY3xKAokxkV6XYowxQeN153irsT3/ADv2HWSq3fRnjGnlLDgayYKMPETg0mEWHMaY1s2Co5HMT88ltXc8XeOivC7FGGOCyoKjEewqOMiWvAN2NZUxpk2w4GgECzLyAJhij4g1xrQBFhyNYGFGHqN6dqRHx2ivSzHGmKCz4PiGMosPkZ5dakOoG2PaDAuOb2hBRi6A9W8YY9oMC45vaH56HsOT4ujZqZ3XpRhjTJOw4PgGckoOsy6zxM42jDFtigXHN7DQvZpqql1NZYxpQyw4voEFGbkM7hZL384xXpdijDFNxoLjNO0rqyBtz35rpjLGtDkWHKdp0cY8VLHLcI0xbY4Fx2man55H/y4xDOga63UpxhjTpCw4TkPhwUpWflVkneLGmDbJguM0fLAxH5/aTX/GmLbJguM0LMjIpU9CO4Z0t2YqY0zbY8HRQPvLq/h8ZxFTR3RHRLwuxxhjmpwFRwN9uDmfWp9a/4Yxps2y4GigBem5JHWMZkRSB69LMcYYT1hwNEBZRTWf7ihk2ohu1kxljGmzLDgaYMnmfKprlakj7GoqY0zbZcHRAPPT8+gWF8Wo5I5el2KMMZ6x4AjQwcoalm8rYMrwboSEWDOVMabtsuAI0NIt+6iq8THNmqmMMW2cBUeAFmTk0jk2krG9470uxRhjPGXBEYBDVTUs3VLApcO6EmrNVMaYNs6CIwDLtxZwuLqWaTY2lTHGBDc4RGSKiGwVkR0iMque5b1FZImIbBCRZSKS7M4fJSIrRGSju+xav23+ISJficg692dUMI8BYEFGHp3aRzAupVOw38oYY5q9oAWHiIQCTwFTgaHA9SIytM5qjwEvq+pI4EHgEXf+IeC7qjoMmAL8RUQ6+m33C1Ud5f6sC9YxAFRU17Jkcz6XDutKWKidoBljTDC/CccBO1R1l6pWAXOAGXXWGQp85E4vPbJcVbep6nZ3OgfYB3QOYq0n9Mn2QsqraplizVTGGAMENziSgEy/11nuPH/rgZnu9FVArIgk+K8gIuOACGCn3+zfu01YfxaRyPreXERuFZE0EUkrKCg47YNYkJFLh+hwzu6XcOqVjTGmDfC67eU+4HwRWQucD2QDtUcWikh34J/A91XV586+HxgMnAl0An5V345V9XlVTVXV1M6dT+9kparGx4eb8pk8tCvh1kxljDEAhAVx39lAT7/Xye68o9xmqJkAIhIDXK2qJe7rOOB94L9U9Qu/bXLdyUoReREnfILis52FHKiosSHUjTHGTzB/jV4FDBCRFBGJAK4D5vmvICKJInKkhvuB2e78COAtnI7zuXW26e7+KcCVQEawDmBBei4xkWFMHJAYrLcwxpgWJ2jBoao1wJ3AImAz8LqqbhSRB0VkurvaJGCriGwDugK/d+dfA5wH3FzPZbeviEg6kA4kAg8H6xhSEmO46azeRIaFBustjDGmxRFV9bqGoEtNTdW0tDSvyzDGmBZFRFaramrd+dbja4wxpkEsOIwxxjSIBYcxxpgGseAwxhjTIBYcxhhjGsSCwxhjTINYcBhjjGkQCw5jjDEN0iZuABSRAmDPaW6eCBQ2YjktnX0ex9hncTz7PI7XGj6P3qr6tVFi20RwfBMiklbfnZNtlX0ex9hncTz7PI7Xmj8Pa6oyxhjTIBYcxhhjGsSC49Se97qAZsY+j2PssziefR7Ha7Wfh/VxGGOMaRA74zDGGNMgFhzGGGMaxILjJERkiohsFZEdIjLL63q8IiI9RWSpiGwSkY0icrfXNTUHIhIqImtF5D2va/GaiHQUkbkiskVENovIWV7X5BUR+Zn7/yRDRF4TkSiva2psFhwnICKhwFPAVGAocL2IDPW2Ks/UAD9X1aHABOCONvxZ+Lsb57HIBv4KLFTVwcAZtNHPRUSSgLuAVFUdDoQC13lbVeOz4DixccAOVd2lqlXAHGCGxzV5QlVzVXWNO30A50shyduqvCUiycBlwAte1+I1EekAnAf8HUBVq1S1xNOivBUGRItIGNAOyPG4nkZnwXFiSUCm3+ss2viXJYCI9AFGAys9LsVrfwF+Cfg8rqM5SAEKgBfdprsXRKS910V5QVWzgceAvUAuUKqqH3hbVeOz4DABE5EY4A3gHlUt87oer4jI5cA+VV3tdS3NRBgwBnhGVUcD5UCb7BMUkXiclokUoAfQXkRu9LaqxmfBcWLZQE+/18nuvDZJRMJxQuMVVX3T63o8dg4wXUR24zRhXigi//K2JE9lAVmqeuQsdC5OkLRFFwNfqWqBqlYDbwJne1xTo7PgOLFVwAARSRGRCJwOrnke1+QJERGc9uvNqvonr+vxmqrer6rJqtoH59/FR6ra6n6rDJSq5gGZIjLInXURsMnDkry0F5ggIu3c/zcX0QovFAjzuoDmSlVrROROYBHOlRGzVXWjx2V55RzgJiBdRNa5836tqvO9K8k0Mz8FXnF/ydoFfN/jejyhqitFZC6wBudqxLW0wqFHbMgRY4wxDWJNVcYYYxrEgsMYY0yDWHAYY4xpEAsOY4wxDWLBYYwxpkEsOIxp5kRkko3Aa5oTCw5jjDENYsFhTCMRkRtF5EsRWSciz7nP6zgoIn92n8+wREQ6u+uOEpEvRGSDiLzljnGEiPQXkcUisl5E1ohIP3f3MX7Pu3jFvSvZGE9YcBjTCERkCHAtcI6qjgJqgRuA9kCaqg4DlgO/czd5GfiVqo4E0v3mvwI8papn4IxxlOvOHw3cg/NsmL44d/Mb4wkbcsSYxnERMBZY5Z4MRAP7cIZd/7e7zr+AN93nV3RU1eXu/JeA/4hILJCkqm8BqGoFgLu/L1U1y329DugDfBr0ozKmHhYcxjQOAV5S1fuPmyny33XWO90xfir9pmux/7vGQ9ZUZUzjWAJ8S0S6AIhIJxHpjfN/7FvuOt8BPlXVUmC/iJzrzr8JWO4+XTFLRK509xEpIu2a8iCMCYT91mJMI1DVTSLyG+ADEQkBqoE7cB5qNM5dtg+nHwTge8CzbjD4jyZ7E/CciDzo7uPbTXgYxgTERsc1JohE5KCqxnhdhzGNyZqqjDHGNIidcRhjjGkQO+MwxhjTIBYcxhhjGsSCwxhjTINYcBhjjGkQCw5jjDEN8v8B4UQrW/md1LMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(model.history.history.keys())\n",
    "# summarize history for accuracy\n",
    "plt.plot(model.history.history['accuracy'])\n",
    "plt.plot(model.history.history['val_accuracy'])\n",
    "plt.title('model accuracy')\n",
    "plt.ylabel('accuracy')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'val'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Из-за перекоса в классах accuracy не очень честно оценивает модель. F1 меры в tf.keras нет, но можно просто сделать предикт и посчитать через sklearn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import classification_report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "preds = model.predict(X_valid).reshape(-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      0.99      0.97     61244\n",
      "           1       0.68      0.39      0.50      4063\n",
      "\n",
      "    accuracy                           0.95     65307\n",
      "   macro avg       0.82      0.69      0.74     65307\n",
      "weighted avg       0.94      0.95      0.94     65307\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# можно подбирать порог для отнесения к токсичному классу\n",
    "print(classification_report(y_valid, (preds > 0.5).astype(int)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Домашка\n",
    "\n",
    "1) На одной из задач сравните, что лучше: а) конкатенировать эмбединги в один большой эмбединг, б) усреднять эмбединги, в) складывать эмбединги. Зафиксируйте разбиение на train_test  через random_seed, чтобы обучаться и оцениваться на одних и тех же данных. Параметры нейронок не нужно делать одинаковыми. Просто попробуйте добиться максимума от каждого из подходов. \n",
    "\n",
    "2) В одной из задач используйте сразу две предобученные эмбединг модели (можно фастекст и ворд2век, а можно просто 2 фастекста с разными параметрами) в одной нейронке (имеется в виду предобученные до обучения нейронки, т.е вы можете сами обучить фастекст/ворд2век и не искать готовые модели в интернете). Попробуйте усреднять эмбединги и конкатенировать (общие эмбединги потом в один вектор собирайте любым способом - например, тем который оказался лучше в первой задаче).\n",
    "Подсказка: используйте functional api, сделайте два Embedding слоя и к их выходам примените слой Concatenate([emb1, emb2)) или Average([emb1, emb2))\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Бонусный балл можно получить за добавление к пайплайну subword токенизации (через библиотеку tokenizers например)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Лучше делать задание на второй задаче, так как в Авито в любом случае разница будет в несколько десятых процентов."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}