notebooks/cnn/cnn_tf.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Дисклеймер\n",
    "Эту тетрадку нужно запускать в колабе или в vast.ai. Не мучатесь с установкой библиотек и с обучением на cpu."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Archive:  lenta-ru-news.csv.zip\n",
      "  inflating: lenta-ru-news.csv       \n",
      "  inflating: __MACOSX/._lenta-ru-news.csv  \n"
     ]
    }
   ],
   "source": [
    "!unzip lenta-ru-news.csv.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting pandas\n",
      "  Downloading pandas-1.1.5-cp36-cp36m-manylinux1_x86_64.whl (9.5 MB)\n",
      "\u001b[K     |████████████████████████████████| 9.5 MB 11.9 MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting scikit-learn\n",
      "  Downloading scikit_learn-0.24.1-cp36-cp36m-manylinux2010_x86_64.whl (22.2 MB)\n",
      "\u001b[K     |████████████████████████████████| 22.2 MB 2.3 MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting matplotlib\n",
      "  Downloading matplotlib-3.3.4-cp36-cp36m-manylinux1_x86_64.whl (11.5 MB)\n",
      "\u001b[K     |████████████████████████████████| 11.5 MB 1.1 MB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: numpy>=1.15.4 in /usr/local/lib/python3.6/dist-packages (from pandas) (1.19.5)\n",
      "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas) (2021.1)\n",
      "Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.6/dist-packages (from pandas) (2.8.1)\n",
      "Collecting joblib>=0.11\n",
      "  Downloading joblib-1.0.1-py3-none-any.whl (303 kB)\n",
      "\u001b[K     |████████████████████████████████| 303 kB 52.8 MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting scipy>=0.19.1\n",
      "  Downloading scipy-1.5.4-cp36-cp36m-manylinux1_x86_64.whl (25.9 MB)\n",
      "\u001b[K     |████████████████████████████████| 25.9 MB 25.3 MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting threadpoolctl>=2.0.0\n",
      "  Downloading threadpoolctl-2.1.0-py3-none-any.whl (12 kB)\n",
      "Collecting cycler>=0.10\n",
      "  Downloading cycler-0.10.0-py2.py3-none-any.whl (6.5 kB)\n",
      "Collecting kiwisolver>=1.0.1\n",
      "  Downloading kiwisolver-1.3.1-cp36-cp36m-manylinux1_x86_64.whl (1.1 MB)\n",
      "\u001b[K     |████████████████████████████████| 1.1 MB 50.4 MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting pillow>=6.2.0\n",
      "  Downloading Pillow-8.2.0-cp36-cp36m-manylinux1_x86_64.whl (3.0 MB)\n",
      "\u001b[K     |████████████████████████████████| 3.0 MB 37.4 MB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3 in /usr/local/lib/python3.6/dist-packages (from matplotlib) (2.4.7)\n",
      "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.6/dist-packages (from python-dateutil>=2.7.3->pandas) (1.15.0)\n",
      "Installing collected packages: pandas, joblib, scipy, threadpoolctl, scikit-learn, cycler, kiwisolver, pillow, matplotlib\n",
      "Successfully installed cycler-0.10.0 joblib-1.0.1 kiwisolver-1.3.1 matplotlib-3.3.4 pandas-1.1.5 pillow-8.2.0 scikit-learn-0.24.1 scipy-1.5.4 threadpoolctl-2.1.0\n",
      "\u001b[33mWARNING: You are using pip version 20.2.4; however, version 21.0.1 is available.\n",
      "You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip install pandas scikit-learn matplotlib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Сверточные нейронные сети"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Несколько полносвязных слоев (с нелинейностями посередине) - это уже полноценная нейронная сеть. Однако это скорее какая-то базовая вещь. Действительно нужен полносвязный слой только в конце сети, а в качестве основных используются сверточные или рекуррентные слои. Главное их отличие (=преимущество) - учитывание структукры данных. В этом семинаре мы попробуем использовать сверточные сети. Лучше всего они работают в картинках, но для текстов их тоже можно применять (они быстрее рекуррентных и могут лучше работать на длинных последовательностях)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Про то как работают сверточные слои посмотрите в этом видео - https://www.youtube.com/watch?v=bNb2fEVKeEo\n",
    "\n",
    "Основная идея - использовать много небольших (сильно меньше размера текста или картинки) матриц, которые применяются к кусочкам векторизованного текста/картинки. Между кусочком и матрицей считается близость, и эта близость передается дальше по сети. Т.е. кусочек как-бы **сворачивается** в одно число. \n",
    "\n",
    "Эти небольшие матрицы обучаются на данных и, например, для картинок в них собираются различные паттерны, важные для задачи. Например, для определения рукописных цифр, такие матрицы могут представлять горизонтальные черточки, волнистые линии, кружки и т.д. Есть даже целое направление исследований, которое занимается визуализацией и интерпретацией обученных сверток.\n",
    "\n",
    "Из-за того, что матрицы применяются к кусочкам, а не отедльным элементам, гораздо лучше учитывается структура. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Классификация с помощью CNN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Давайте посмотрим, как в тензорфлоу использовать сверточные сети и как их соединять с Embedding и Dense слоями."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from string import punctuation\n",
    "from sklearn.model_selection import train_test_split\n",
    "from collections import Counter\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py:3072: DtypeWarning: Columns (3) have mixed types.Specify dtype option on import or set low_memory=False.\n",
      "  interactivity=interactivity, compiler=compiler, result=result)\n"
     ]
    }
   ],
   "source": [
    "# data = pd.read_csv('lenta-ru-news.csv').sample(frac=0.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# data.dropna(subset=['topic', 'text'], inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# data.to_csv('lenta_sample.csv', index=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.read_csv('lenta_sample.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Предобработка нам тут особо не важна."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess(text):\n",
    "    tokens = text.lower().split()\n",
    "    tokens = [token.strip(punctuation) for token in tokens]\n",
    "    return tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "В предыдущей домашке не хватало фмеры. Ее нет в керасе, но можно взять вот такую функцию (взятую со стаковерфлоу)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras import backend as K\n",
    "def f1(y_true, y_pred):\n",
    "    def recall(y_true, y_pred):\n",
    "        \"\"\"Recall metric.\n",
    "\n",
    "        Only computes a batch-wise average of recall.\n",
    "\n",
    "        Computes the recall, a metric for multi-label classification of\n",
    "        how many relevant items are selected.\n",
    "        \"\"\"\n",
    "        true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\n",
    "        possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))\n",
    "        recall = true_positives / (possible_positives + K.epsilon())\n",
    "        return recall\n",
    "\n",
    "    def precision(y_true, y_pred):\n",
    "        \"\"\"Precision metric.\n",
    "\n",
    "        Only computes a batch-wise average of precision.\n",
    "\n",
    "        Computes the precision, a metric for multi-label classification of\n",
    "        how many selected items are relevant.\n",
    "        \"\"\"\n",
    "        true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))\n",
    "        predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))\n",
    "        precision = true_positives / (predicted_positives + K.epsilon())\n",
    "        return precision\n",
    "    precision = precision(y_true, y_pred)\n",
    "    recall = recall(y_true, y_pred)\n",
    "    return 2*((precision*recall)/(precision+recall+K.epsilon()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Предобработка точно такая же"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# словарь\n",
    "vocab = Counter()\n",
    "\n",
    "for text in data.text:\n",
    "    vocab.update(preprocess(text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "670782"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# отфильтрованный словарь\n",
    "filtered_vocab = set()\n",
    "\n",
    "for word in vocab:\n",
    "    if vocab[word] > 30:\n",
    "        filtered_vocab.add(word)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "55293"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(filtered_vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# индексируем слова\n",
    "word2id = {'PAD':0}\n",
    "\n",
    "for word in filtered_vocab:\n",
    "    word2id[word] = len(word2id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "id2word = {i:word for word, i in word2id.items()}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# переводим тексты в последовательности индексов\n",
    "X = []\n",
    "\n",
    "for text in data.text:\n",
    "    tokens = preprocess(text)\n",
    "    ids = [word2id.get(token, 1) for token in tokens]\n",
    "    X.append(ids)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_LEN = max(len(x) for x in X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Можно посчитать среднюю или медианную длину, если максимальная кажется слишком большой."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "MEAN_LEN = np.median([len(x) for x in X])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3578, 170.0)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "MAX_LEN, MEAN_LEN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_LEN = int(MEAN_LEN + 30)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# паддинг\n",
    "X = tf.keras.preprocessing.sequence.pad_sequences(X, maxlen=MAX_LEN)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(147852, 200)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "id2label = {i:label for i, label in enumerate(set(data.topic.values))}\n",
    "label2id = {l:i for i, l in id2label.items()}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = tf.keras.utils.to_categorical([label2id[label] for label in data.topic.values])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "19"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(label2id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "В этот раз добавим стратификацию, т.к. в данных у нас дисбаланс классов."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_valid, y_train, y_valid = train_test_split(X, y, test_size=0.05)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Для сравнения сначала обучим сеть только с помощью Dense слоев."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs = tf.keras.layers.Input(shape=(MAX_LEN,))\n",
    "embeddings = tf.keras.layers.Embedding(input_dim=len(word2id), output_dim=30)(inputs, )\n",
    "mean = tf.keras.layers.Flatten()(embeddings)\n",
    "dense = tf.keras.layers.Dense(30, activation='relu')(mean)\n",
    "outputs = tf.keras.layers.Dense(len(label2id), activation='softmax')(dense)\n",
    "\n",
    "model = tf.keras.Model(inputs=inputs, outputs=outputs)\n",
    "optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)\n",
    "model.compile(optimizer=optimizer,\n",
    "              loss='categorical_crossentropy',\n",
    "              # фмеры тут нет, но есть точность и полнота по отдельности, \n",
    "              # и есть полнота при заданном пороге точности\n",
    "              # name тут нужен для удобства,\n",
    "              # иначе метрика будет называться типа \"Precision_i\"\n",
    "              # и придется менять i в дальнейших визуализациях \n",
    "              metrics=[tf.keras.metrics.RecallAtPrecision(0.8, name='rec@prec')])\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "71/71 [==============================] - 5s 54ms/step - loss: 2.5193 - rec@prec: 0.0019 - val_loss: 1.5970 - val_rec@prec: 0.1969\n",
      "Epoch 2/5\n",
      "71/71 [==============================] - 3s 42ms/step - loss: 1.3394 - rec@prec: 0.4452 - val_loss: 0.8981 - val_rec@prec: 0.6848\n",
      "Epoch 3/5\n",
      "71/71 [==============================] - 3s 42ms/step - loss: 0.7545 - rec@prec: 0.7627 - val_loss: 0.7005 - val_rec@prec: 0.7841\n",
      "Epoch 4/5\n",
      "71/71 [==============================] - 3s 42ms/step - loss: 0.5140 - rec@prec: 0.8828 - val_loss: 0.6365 - val_rec@prec: 0.8173\n",
      "Epoch 5/5\n",
      "71/71 [==============================] - 3s 42ms/step - loss: 0.3629 - rec@prec: 0.9446 - val_loss: 0.6273 - val_rec@prec: 0.8156\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7fea82039208>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train, y_train, \n",
    "          validation_data=(X_valid, y_valid),\n",
    "          batch_size=2000,\n",
    "         epochs=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['loss', 'rec@prec', 'val_loss', 'val_rec@prec'])\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAxYklEQVR4nO3dd3gVZfbA8e9Jr5CQQguQEHoXIyAooliwYUHEunZcXV0Luta1u7Z1i7vuz76661qoCipio7mICyihl9BDSyGE9Hbf3x9zgRADJDCTubn3fJ6Hh3tnJu97MnDnzLwz97xijEEppVTgCnI7AKWUUu7SRKCUUgFOE4FSSgU4TQRKKRXgNBEopVSA00SglFIBThOBUsdJRN4VkWcauO1mETnzMOsiRWSGiBSKyCR7o1Tq8DQRKOU7LgNaAwnGmLEi0lZEpovIDhExIpLqcnzKT2kiUMp3dALWGWOqve89wJfAGPdCUoFAE4EKCN4hmftFZJmIlIjI2yLSWkRmikiRiHwjIvG1th8tIitFZK+IzBGRnrXWnSAiP3l/7mMgok5fF4jIUu/PLhCRfg2I70ngMWCciBSLyE3GmN3GmH8Ai+zbE0r9kiYCFUjGAGcB3YALgZnAw0AS1mfhtwAi0g34ELjbu+4LYIaIhIlIGPAJ8G+gFTCJWmfsInIC8A5wK5AAvA5MF5HwIwVmjHkc+APwsTEmxhjzti2/sVINoIlABZK/ec+ytwPzgR+NMT8bY8qBacAJ3u3GAZ8bY742xlQBfwQigaHAECAU+IsxpsoYM5lDz9jHA68bY340xtQYY94DKrw/p5RPCnE7AKWa0O5ar8vqeR/jfd0O2LJ/hTHGIyLbgPZADbDdHFqtcUut152A60TkzlrLwrxtKuWTNBEo9Us7gL7734iIAB2A7YAB2ouI1EoGHYEN3tfbgGeNMc82YbxKHRcdGlLqlyYC54vISBEJBSZgDe8sAH4AqoHfikioiFwKDKr1s28CvxaRwWKJFpHzRST2WAIRkQhg//2FcO97pWyliUCpOowxa4FrgL8BeVg3li80xlQaYyqBS4HrgT1Y9xOm1vrZxcAtwN+BAiDLu+2xKgOKva/XeN8rZSvRiWmUUiqw6RWBUkoFOE0ESikV4DQRKKVUgNNEoJRSAa7ZfY8gMTHRpKamuh2GUko1K0uWLMkzxiTVt67ZJYLU1FQWL17sdhhKKdWsiMiWw63ToSGllApwmgiUUirAaSJQSqkA1+zuEdSnqqqK7OxsysvL3Q7FUREREaSkpBAaGup2KEopP+IXiSA7O5vY2FhSU1OxCkX6H2MM+fn5ZGdnk5aW5nY4Sik/4hdDQ+Xl5SQkJPhtEgAQERISEvz+qkcp1fT8IhEAfp0E9guE31Ep1fT8YmhIKaX8UXlVDZvzS8jKKWZDTgkjeybTp31L2/vRRGCDvXv38sEHH3D77bc36ufOO+88PvjgA+Li4pwJTCnVLOwtrWRDbrF1wM/1Hvhzi9m2pxSPd6YAEWgVE6aJwFft3buXf/zjH79IBNXV1YSEHH4Xf/HFF06HppTyER6PYUdh2SEH+qycYjbmFpNXXHlgu7CQIDonRtOnfUsuHtCe9OQYuiTFkJYYTWRYsCOxaSKwwYMPPsiGDRsYMGAAoaGhREREEB8fz5o1a1i3bh0XX3wx27Zto7y8nLvuuovx48cDB8tlFBcXc+6553LKKaewYMEC2rdvz6effkpkZKTLv5lSqrEqqmvYnFda6wx//wG/hLKqmgPbxUWF0iUphpE9WtMlOYb05Gi6JMXSPj6S4KCmvR/od4ngyRkrWbVjn61t9mrXgscv7H3Y9c8//zwrVqxg6dKlzJkzh/PPP58VK1YceMzznXfeoVWrVpSVlXHSSScxZswYEhISDmlj/fr1fPjhh7z55ptcfvnlTJkyhWuuucbW30MpZZ/CsqoDB/oNtQ74W2sN5wC0j4skPTmGwWkJ3oN9DF2SY2gVHeYzD4D4XSLwBYMGDTrkWf9XXnmFadOmAbBt2zbWr1//i0SQlpbGgAEDADjxxBPZvHlzU4WrlDoMYww7C8sPObO3/i4hr7jiwHZhwUGkJUbTq10LRvdvR3pyDOlJMXROiiYqzPcPs74fYSMd6cy9qURHRx94PWfOHL755ht++OEHoqKiGDFiRL3fBQgPDz/wOjg4mLIynaNcqaZSWe1hS35JnQN+CRtyiymtPDic0yIihC7JMZzePckazvGe3afERxIS3Hyfxve7ROCG2NhYioqK6l1XWFhIfHw8UVFRrFmzhoULFzZxdEqp/faVV7Eh59CnczbmFrNlTyk1tcZz2rWMID05hsszOhxywE+M8Z3hHDtpIrBBQkICw4YNo0+fPkRGRtK6desD60aNGsVrr71Gz5496d69O0OGDHExUqX8nzGG3fsqDjm73/86p+jgcE5osJCaEE33NrGc17ftgQN+56RoosMD69Aoxpijb+VDMjIyTN2JaVavXk3Pnj1diqhpBdLvqtSRVNV42JJf+osbthtySyiuqD6wXWx4yIEx+y7JMd4DfjQdW0U16+GcxhKRJcaYjPrWBVbaU0o1O0XlVWzMrTt+X8yW/FKqaw3ntGkRQZfkGMYMbH/IcE5SbLhfDufYSROBUsp1xhhyiiqs8Xvv2b31dwm79h18uCIkSOiUEEWX5BjO6d3mwAE/PTmGmAAbzrGT7jmlVJPyeAyrdu7jv1l5rNttHfA35hRTVGs4JyY8hPSkaIZ2SThwZp+eFEOnhChCA2g4p6loIlBKOa6gpJJ563OZuy6XeevyDjyD37pFOOlJMVwysP0hB/zWLXQ4pylpIlBK2a7GY1iWvZe563KZszaXzOy9GGOVVRjeNYnTuiUxvFsSSbHhR29MOU4TgVLKFrlFFcxfbx3456/PpaC0ChHonxLHXSO7clq3JPqlxDV5HR11dJoIXBATE0NxcbHbYSh1XKprPCzdtpc5a60hn+XbCwFIjAnj9B7J1ll/1yTio8NcjlQdjSYCpVSD7d5XzlzvgX/++lz2lVcTHCQM7BjHfWd3Y0T3ZHq1bUGQnvU3K5oIbPDggw/SoUMHfvOb3wDwxBNPEBISwuzZsykoKKCqqopnnnmGiy66yOVIlWqcqhoPS7YUMGdtLnPW5rBml1VKpXWLcEb1acNp3ZI5pUsiLaNCXY5UHQ//SwQzH4Rdy+1ts01fOPf5w64eN24cd99994FEMHHiRGbNmsVvf/tbWrRoQV5eHkOGDGH06NH6JITyedv3lnnP+nP4b1Y+xRXVhAQJGanxPDCqByO6J9GjTaz+X/Yj/pcIXHDCCSeQk5PDjh07yM3NJT4+njZt2nDPPfcwb948goKC2L59O7t376ZNmzZuh6vUISqqa1i0qYC563KYszaX9TnW/at2LSO4sH87RnRPYmh6ArERetbvr/wvERzhzN1JY8eOZfLkyezatYtx48bxn//8h9zcXJYsWUJoaCipqan1lp9Wyg1b80sPHPgXbMinrKqGsOAgBqW1YtxJHTitm1VmWc/6A4P/JQKXjBs3jltuuYW8vDzmzp3LxIkTSU5OJjQ0lNmzZ7Nlyxa3Q1QBrLyqhoUb85mzNpd563LZmFcCQMdWUYzNSOG0bkmcnJ7QLCZRUfbTf3Wb9O7dm6KiItq3b0/btm25+uqrufDCC+nbty8ZGRn06NHD7RBVADHGsCmv5MCjnQs35lNR7SE8JIiT0xO49uROjOieTGpClJ71K00Edlq+/OBN6sTERH744Yd6t9PvECgnlFZWsyAr3/o277octu2xZrnrnBTNVYM7MqJ7MoPTWhERGuxypMrXaCJQqpkyxrA+p5i5a60D/6JNBVTWeIgKC2ZoegLjh6czolsSHVpFuR2q8nGaCJRqRorKq/hvVj5z1+Uwd20uOwqtBxC6tY7h+mGpnNYtiYzUeMJD9KxfNZzfJAJjjN+PdTa32eTU8TPGsHpnkbd4Ww5LthRQ7THEhocwrEsid460Cri1i4t0O9SGMQYqi6E4B0rywNQAAhIEIt7X+//GWn7IMjnC9t7l0Mjt6y6rZ7uGttFMj0F+kQgiIiLIz88nISHBb5OBMYb8/HwiIiLcDkU5rLC0iu+z8pizNoe563IPzLPbq20LbhnemRHdkhjYKd636vJXlUNJjnWAL97t/VP7vffvklyoKnU72ibQyORz2CRUp40zn4D+V9geraOJQERGAX8FgoG3jDHP11nfEXgPiPNu86Ax5ovG9pOSkkJ2dja5ubnHH7QPi4iIICUlxe0wlM08HsPKHfsOHPh/2lqAx0CLiBBO7ZbEiG7WWX9yiyY+Caiptg7c+w/kJXUO6sW11lUU1t9GZCuIaQ0xSdBhkPd1MkQnQ3QiBAVbVwkYMIDxeF+bOn976lnmXQ5HWNeQtjjK9kdb19jtDxfzkX5H7+uWznz+HUsEIhIMvAqcBWQDi0RkujFmVa3NHgUmGmP+T0R6AV8AqY3tKzQ0lLS0NBuiVqpp7CmpZP76XOauzWXe+lzyiisB6JfSkjtO78Jp3ZPonxJn/+TqHg+UFdQ5a99d52ze+7o0H+voXEdYrHUwj2kNrXtD+ukH38e0huikg3+HaOXR5sDJK4JBQJYxZiOAiHwEXATUTgQGaOF93RLY4WA8SrmmxmPIzN7rfcInl2XeiVpaRYcxvGsip3VP4tSuSSTGHMNELcZAxb76h2LqnsmX5IKn+pdtBIdDbGvrTD0+DToMPngmv/8Av/9MPkyfQvI3TiaC9sC2Wu+zgcF1tnkC+EpE7gSigTPra0hExgPjATp27Gh7oEo5IbeognnrrAP//PW57C2tIkhgQIc47h7ZjRHdk+jbvuXhSzZXltZ/pl7fmXx1PeVLJNh7pr7/7L1vrTP32n8nQ3iLZnujUx0/t28WXwm8a4x5WUROBv4tIn2M2T+IZjHGvAG8AZCRkaGPziifVF3j4edtew+M9a/Yvg+AxJhwRvZozYjuSZyS1oJ4U+g9gG+Epbt/Od6+/+/Kovo7iko8eLaekH7woB6dfOgQTWQ8BPnQDWXls5xMBNuBDrXep3iX1XYTMArAGPODiEQAiUCOg3EpZStjDB8s3MSUWbOJrswhOWgf4+IreKZLOanhJbSs2YPk5sKm3VC2p/5GwlsePIi37X/omfwh4+6JEKxVQJW9nEwEi4CuIpKGlQCuAK6qs81WYCTwroj0BCIA/370R/mPfTspXPElq7//hPNKFnO1FMP+e6NFQFnkwXH3hHToNPTQ4Zja4+6h+liwco9jicAYUy0idwCzsB4NfccYs1JEngIWG2OmAxOAN0XkHqwbx9cb/daU8lXVFbBlAWz4FrK+g5yVtAQ6mzj2tD+duEHnIfFpBw/0YTE67q6aBWlux92MjAyzePFit8NQgcAYyM+CrG+tg//m76GqFBMUyrqIvkwt7E5O8jB+e9UlpCXFuB2tUkckIkuMMRn1rXP7ZrFSvqW8EDbNg6xvrLP+wq3W8lbpcMI1LI84kTsWRLN9bzD3nNWN+4d3tv9Zf6WamCYCFdg8Hti59OBZ/7b/WfVvwmIhbTicchekj6Q0pgPPfr6a/8zbSrfWMXxywwD6tG/pdvRK2UITgQo8Rbtgw3fWwX/jbO83aLGe1jnlbkgfaZVD8D6ds2TLHu59ez5b95Qyfnhn7j2rm9b0V35FE4Hyf9UVsHXhwZu8u70TCEUnQZezoMtI6Hy69Vx+LZXVHv7yzTpem7uBdnGRfHTLEAZ3TnDhF1DKWZoIlP8xBvZstM74s77x3uQtgaBQ6DgERj5uHfxb9z3sF67W7NrHPR9nsnrnPq44qQOPXtCLmHD9uCj/pP+zlX8o32fd5N3wrZUA9m6xlsenwYArreGetFMhPPaIzdR4DG/O38ifvlpHi8hQ3r4ug5E9WzfBL6CUezQRqObJ44Fdmd6bvN/Bth+tYmqh0dZN3qF3Wmf9rTo3uMmt+aVMmLSURZsLOLdPG569pC+torV6pvJ/mghU81Gcc/Am74bvoDTPWt6mn3XgTx9pVc1sZOljYwwfLdrG05+tIjhI+PO4/lw8oL3fTnKkVF2aCJTvqq60zvSzvrGGfHZ5b/JGJUL6GdYZf/oZ1rd4j1HOvnIemLKM2WtzGdYlgZcu6998pn1UyiaaCJRvOXCT91vYPN+a3zYoxDrTH/mYddbfpp8tVTU/X7aTRz5ZTnlVDU+O7s21QzodviS0Un5ME4FyV0URbJp/8CZvwSZreVwn6DfOOutPPRUiWhy5nUYoLK3i8ekr+GTpDvp3iONPl/cnXUtEqACmiUA1LY/Heo5//zj/1oXgqfLe5D0Vhtx+8CavA2P089fncv+kZeQVV3DvWd24fUS6lohQAU8TgXJeca510N/gPfiXeCuNt+4LJ99uDfd0HAIhxzBNYwOVVlbz/Mw1/OuHLXRJjuHNX2XQN0VLRCgFmgiUE2qqvDd5vfV7dmZay6MSrJu76d6bvLFN83z+T1sLmDAxk015Jdx0Shr3n9NdS0QoVYsmAmWPPZsOlnDYNM+aZlGCrZu8ZzxqHfzbDmjSqRMrqz387bv1vDo7i7YtI/nglsEMTU9ssv6Vai40EahjU1FslW7Y4C3jsGejtTyuI/S9zBrnTxsOEe4Mv6zbXcQ9Hy9l5Y59XHZiCo9d2IsWETrFo1L10USgGsYY2L3CW6f/21o3eaMg9RQYdCt0OdOaktHFL2LVeAzvfL+Jl75aS2x4CK9feyLn9G7jWjxKNQeaCNThleTBhtkHb/IW77aWJ/eGIbdZZ/0dT3b0Jm9jbNtTyoRJmfxv0x7O6tWa5y7tS2KMb8SmlC/TRKDqt2wiTPu1NUlLZCtIP/3gTd4Wbd2O7hDGGCYtzubJGSsREf44tj9jBmqJCKUaShOB+qW89TDjbkg5Cc75A7QbAEG++ZRNTlE5D09dzjerczi5cwIvje1HSnyU22Ep1axoIlCHqq6AyTdYwz1j/wkt2rkd0WHNXL6Th6ctp6Syht9f0IsbhqZqiQiljoEmAnWorx+zirtd+ZHPJoHCsiqenL6SqT9vp2/7lvzp8v50bX3keQaUUoeniUAdtOYL+PE1GHwbdD/X7Wjq9f36PO6fnElOUQV3jezKHWd0IVRLRCh1XDQRKEvhdvj0dquy51lPuh3NL5RV1vDCl2t4d8FmOidFM/W2ofTvEOd2WEr5BU0ECjw1MHW8Vf//sn/6zOOg+y3dtpd7Jy5lY24J1w9N5YFRPYgM882b10o1R5oIFMz7I2z5Hi5+DRK7uB3NAVU1Hv72XRavzs4iOTac/9w8mGFdtESEUnbTRBDotiyAuc9btf8HXOl2NAes313EPROXsmL7Pi4d2J7HL+xNy0gtEaGUEzQRBLLSPTDlZohPhfNfdjsaADwewzv/3cSLs9YSEx7Ca9cMZFQf3/oCm1L+RhNBoDIGPr3DmhD+5q8h3P3HL7MLSrlvUiYLN+7hzJ7J/OHSviTHRrgdllJ+TxNBoFr0Fqz93PvN4RNcDcUYw+Ql2Tw5YxXGGF4c04+xGSlaIkKpJqKJIBDtWg6zHoGuZ1tTQ7oor7iCh6Yu5+tVuxmU1oqXx/anQystEaFUU9JEEGgqS2DSDRAZDxf/n6slo2et3MXDU5dTVF7NI+f15KZT0rREhFIu0EQQaGb+DvKz4FefQrQ7j2LuK6/iyemrmPJTNr3bteCDWwbQvY379yiUClSaCALJ8snw8/tw6n3Q+TRXQliwIY/7Jy1jZ2EZd57RhTvP6EpYiJaIUMpNjn4CRWSUiKwVkSwRefAw21wuIqtEZKWIfOBkPAFtz0artHSHwTDioSbvvryqhqdmrOKqN38kLCSIybcNZcLZ3TUJKOUDHLsiEJFg4FXgLCAbWCQi040xq2pt0xV4CBhmjCkQkWSn4glo1ZUw+SZr4vgxb0Fw014ILsveyz0fL2VDbgm/OrkTD57bg6gwvRhVylc4+WkcBGQZYzYCiMhHwEXAqlrb3AK8aowpADDG5DgYT+D67mnY8RNc/i9rcvkmUlXj4dXZWfztuyySYsL5142DGN4tqcn6V0o1jJOJoD2wrdb7bGBwnW26AYjIf4Fg4AljzJd1GxKR8cB4gI4dm+5A5heyvoEFr0DGjdDroqbrNqeYCROXkpldyMUD2vHk6D60jNISEUr5Irevz0OArsAIIAWYJyJ9jTF7a29kjHkDeAMgIyPDNHGMzVfRbmve4eRe1hfHmoDHY3jvh808P3MNkWHBvHrVQM7vpyUilPJlTiaC7UCHWu9TvMtqywZ+NMZUAZtEZB1WYljkYFyBweOBaeOhohiumwGhkY53uX1vGfdPymTBhnxO757EC2P6kdxCS0Qo5eucTASLgK4ikoaVAK4ArqqzzSfAlcA/RSQRa6hoo4MxBY4Ff4WNc+DCv0JyT0e7MsYw7eftPP7pSmqM4blL+3LFSR20RIRSzYRjicAYUy0idwCzsMb/3zHGrBSRp4DFxpjp3nVni8gqoAa43xiT71RMAWPbIvj2aeh1MQy8ztGu8osreGTaCr5cuYuTUuN5eewAOiZoiQilmhMxpnkNuWdkZJjFixe7HYbvKtsLr58KBvj1fIiMc6yrr1ft5qGpy9hXVs2Es7tx86mdCdYSEUr5JBFZYozJqG+d2zeLlZ2Mgc/utuYfvnGWY0mgqLyKpz9bxcTF2fRs24L3b+5PjzYtHOlLKeU8TQT+5Kd/wcppMPJx6HCSI10s3JjPfZMy2bG3jNtHpHPXmV0JD9H5g5VqzjQR+IucNTDzAeg8AobdbXvz5VU1vPzVWt76fhMdW0Ux6dcnc2KnVrb3o5RqepoI/EFVGUy+AcKi4ZLXrVISNlqxvZB7Pl7K+pxirhnSkYfO7Ul0uP7XUcpf6KfZH8x6BHJWwdVTILaNbc1W13j4vzkb+Ou362kVHca7N5zEiO5aDkopf6OJoLlb9SksfhuG3gldz7St2Zx95Yz/9xKWbtvLhf3b8fRFvYmLCrOtfaWU79BE0Jzt3QrT74R2A+GMx2xr1hjDhEmZrN1VxCtXnsDo/u1sa1sp5Xu0GHxzVVMNU26xSklc9jaE2He2/v6PW5m/Po9Hzu+pSUCpAKBXBM3V3Odh20IY8za06mxbs1vyS/jD56s5tWsiVw/WSq9KBQK9ImiONs6FeX+EAddA38tsa7bGY5gwMZOQYOHFy/pprSClAoReETQ3JXkwdTwkdIHzXrS16be/38jiLQX86fL+tG3pfLVSpZRv0ETQnBgDn9wGZQVw9STrewM2Wbe7iD/OWsc5vVtzyQntbWtXKeX7NBE0Jwv/Aeu/gnNfgrb9bGu2qsbDvROXEhMRwrOX9NUhIaUCjCaC5mLHz/D149D9fBh0i61Nvzo7ixXb9/HaNQNJjAm3tW2llO/Tm8XNQUURTL4RYpLhor+DjWfsy7ML+ft3WVxyQntG9dEpJZUKRHpF0Bx8fh8UbIbrPoMo+wq9lVfVcO/EpSTEhPHEhb1ta1cp1bzoFYGvW/ohLPsITnsAUofZ2vSfv17H+pxiXhjTj5ZRoba2rZRqPo4pEYhIjN2BqHrkZcHnE6DTMBh+v61NL9q8hzfmb+SqwR21kJxSAe5YrwhW2RqF+qXqCqu0dEgYXPomBNk3+UtJRTUTJmaSEh/Jw+c5O7G9Usr3HfYegYjce7hVgF4ROO2bJ2DXMrjiQ2hp73P9z81czbaCUj66ZQgxOq+AUgHvSFcEfwDigdg6f2KO8nPqeK390vrOwKBbocd5tjY9b10u7y/cyk3D0hjcOcHWtpVSzdORTgd/Aj4xxiypu0JEbnYupAC3b4f17eHWfeGsp2xturCsigemLKNLcgz3ndPd1raVUs3Xkc7stwNbROSuetZlOBRPYPPUWHWEqitg7D8hNMLW5p+csZKcogpeHtufiFCdcF4pZTlSIugFhAE3iki8iLTa/weoaprwAsz8l2HzfDjvJUjsamvTs1buYupP2/nNiHT6d4iztW2lVPN2pKGh14Fvgc7AEqybxPsZ73Jlly0/wJznoO9YGHCVrU3nF1fwyLTl9G7XgjvOsDfBKKWav8MmAmPMK8ArIvJ/xpjbmjCmwFO6B6bcDHGd4Pw/2VpCwhjDo5+sYF9ZNe/f3J+wEL3Pr5Q61FGfHdQk4DBjrHmHi3fDTV9BRAtbm5+euYOZK3bxwKge9Ghjb9tKKf+gD5G7bdFbsOYzOPsZaD/Q1qZ37yvn95+sYGDHOMYP15E8pVT9dJzATbtWwKxHoMuZMOQ3tjZtjOF3k5dRWePh5csHEBykcwwopeqnicAtlSVWaenIOLj4NQiy95/io0XbmLsul4fO7Ulaon0zmSml/I8ODbll5gOQtw5+9QnEJNna9LY9pTzz2SqGdUng2iGdbG1bKeV/9IrADSumwM//hlPugc4jbG3a4zFMmJSJiPDiZf0J0iEhpdRRaCJoagWbYcbdkHISnP6w7c3/c8Fm/rdpD49d2Iv2cZG2t6+U8j+OJgIRGSUia0UkS0QePMJ2Y0TEiIh/l66oqbLuCyAw5m0ItncymKycYl78cg1n9kxm7IkptratlPJfjiUCEQkGXgXOxSpXcaWI9Kpnu1jgLuBHp2LxGd89DduXwOhXIN7esfvqGg8TJi4lMiyYP1zaF7HxS2lKKf/m5BXBICDLGLPRGFMJfARcVM92TwMvAOUOxuK+rG/hv3+FE6+H3hfb3vxrczeQmV3IMxf3ITnW3mJ1Sin/5mQiaA9sq/U+27vsABEZCHQwxnx+pIZEZLyILBaRxbm5ufZH6rSi3TDtVkjqCec8Z3vzK3cU8tdv13Nh/3Zc0K+d7e0rpfybazeLRSQI+BMw4WjbGmPeMMZkGGMykpLsfdTScR6PlQQqiqzS0mFRtjZfUV3DhImZxEWF8dTo3ra2rZQKDE4mgu1Ah1rvU7zL9osF+gBzRGQzMASY7nc3jBe8Ahtnw6jnINn++YH/8s161uwq4oUxfYmPDrO9faWU/3MyESwCuopImoiEAVcA0/evNMYUGmMSjTGpxphUYCEw2hiz2MGYmlb2YusGcc/RcOINtje/ZEsBr8/dwLiMDpzRo7Xt7SulAoNjicAYUw3cAcwCVgMTjTErReQpERntVL8+o7zQelQ0tp31lJDNT/GUVdZw36RM2raM5NEL7L/SUEoFDkdLTBhjvgC+qLPsscNsO8LJWJqUMdaXxgqz4cYvITLe9i5e+HINm/JK+OCWwcRG2Pt9BKVUYNFvFjvh53/DyqnWN4c7DLK9+QVZeby7YDPXD01laHqi7e0rpQKLJgK75a6FL34HaadZtYRsVlRexf2Tl9E5MZoHRvWwvX2lVODR6qN2qiqDSTdAWDRc+gYEBdvexdOfrWJnYRmTbxtKZJj97SulAo8mAjt99SjkrISrJ0NsG9ub/3b1biYuzub2EekM7Gj/fQelVGDSoSG7rJ5hTTt58h3Q9Szbmy8oqeTBqcvp0SaWu87sanv7SqnApVcEdti7DT69A9oOgJGPO9LF7z9dwd7SSt67YRDhITokpJSyj14RHK+aaphyM3iq4bJ3IMT+b/fOyNzBZ8t2ctfIrvRq18L29pVSgU2vCI7X3Bdg20K49E1ISLe9+Zx95fz+0xX07xDHr0+zv32llNIrguOxaT7Mewn6XwX9Lre9eWMMD01dTlllDS+P7U9IsP5zKaXsp0eWY1WSD1Nvsa4CznvJkS4mLcnm2zU5/G5UD7okxzjSh1JK6dDQsTAGPr0dSvPhqokQbv9BOruglKdmrGJwWituGJpqe/tKKbWfJoJj8eNrsO5LGPUCtO1ne/Mej+F3k5dhjOGPY/sTFKTTTiqlnKNDQ421Yyl8/Rh0OxcG3+pIF/9euIUFG/J59IJedGhl70Q2SilVlyaCxqgoskpLRyXCxf+wvbQ0wMbcYp6buZoR3ZO44qQOR/8BpZQ6Tjo01Bhf3A8Fm+C6GRDVyvbmazyGCZMyCQ8J5oUx/RAHEo1SStWlVwQNlfkxZH4Iw++H1FMc6eL1eRv4eetenrqoN61bRDjSh1JK1aWJoCHyN8Dn90LHoTD8d450sWbXPv789TrO69uG0f3bOdKHUkrVRxPB0VRXwOQbICgExrwJwfaPplVWe7j340xaRoby9EV9dEhIKdWk9B7B0XzzJOzMhCs+gJYpjnTxt+/Ws2rnPt649kQSYsId6UMppQ5HrwiOZN0sWPgqDBoPPc53pIul2/byjzkbGDMwhbN72z+HgVJKHY0mgsPZtxM+uQ1a94Wznnaki/KqGiZMXEpybDiPXdjLkT6UUupodGioPp4aq45QVZlVWjrUmSd4Xpq1lg25Jbx/02BaRoY60odSSh2NJoL6fP8n2DwfRv8dkro50sXCjfm8899NXDukE6d0TXSkD6WUaggdGqpr60KY/Rz0uQxOuMaRLoorqrlvUiYdW0Xx0Hk9HOlDKaUaSq8IaisrsGYbi+sAF/zZkRISAM9+vprte8uYdOvJRIXpP4FSyl16FNrPGJh+JxTthBu/gghnpoScszaHD/+3lVuHdyYj1f4yFUop1Vg6NLTf4ndg9QwY+RiknOhIF4WlVTwwZRndWsdwz1nO3HtQSqnG0isCgN0r4cuHIH0knHynY908Pn0F+cWVvPWrk4gIDXasH6WUagy9IqgshUk3QERLuOQ1CHJml8xcvpNPlu7gjjO60DelpSN9KKXUsdArgi8fhLy1cO00iEl2pIvcogoe+WQFfdu35Dend3GkD6WUOlaBfUWwYir89B6ccg+kn+FIF8YYHpm2nOKKal6+vD+hwYG9y5VSvidwj0oFm2HGXZByEpz+iGPdTPt5O1+t2s19Z3ejW+tYx/pRSqljFZiJoKYKJt9kvR7zFgQ7U95hx94yHp++kpNS47nplM6O9KGUUscrMO8RzH4Wti+Gy/4J8amOdGGM4YEpy6iuMfxxbH+Cg3SOAaWUb3L0ikBERonIWhHJEpEH61l/r4isEpFlIvKtiHRyMh4ANnwH3/8FBl4HfS51rJv3f9zK/PV5PHx+TzolRDvWj1JKHS/HEoGIBAOvAucCvYArRaRureWfgQxjTD9gMvCiU/EAUJwDU2+FpO4w6nnHutmSX8IfPl/NqV0TuWZwR8f6UUopOzh5RTAIyDLGbDTGVAIfARfV3sAYM9sYU+p9uxBwZgowAI8Hpv0aKvZZpaXDohzppsZjuG9SJiHBwgtj+um0k0opn+dkImgPbKv1Ptu77HBuAmbWt0JExovIYhFZnJube2zRLPwHbPgWzvkDtO59bG00wNvfb2TR5gKeuLA37eIiHetHKaXs4hM3i0XkGiADOK2+9caYN4A3ADIyMswxddJtFJTmQ8aNxxrmUa3bXcQfZ63j7F6tuXTgkXKeUkr5DicTwXagQ633Kd5lhxCRM4FHgNOMMRWORZPYBc583LHmq2o8TJiYSUxECM9e0leHhJRSzYaTQ0OLgK4ikiYiYcAVwPTaG4jICcDrwGhjTI6DsTju1dlZLN9eyLMX9yEpNtztcJRSqsEcSwTGmGrgDmAWsBqYaIxZKSJPicho72YvATHAJBFZKiLTD9OcT1ueXcjfv8vi4gHtOLdvW7fDUUqpRnH0HoEx5gvgizrLHqv1+kwn+28K5VU13DtxKQkxYTw5uo/b4SilVKP5xM3i5uzPX69jfU4x795wEi2jnClVoZRSTgrMWkM2Wbx5D2/M38iVgzoyorszJayVUsppmgiOUUlFNRMmZZISH8kj5/d0OxyllDpmOjR0jJ6fuYate0r58JYhxITrblRKNV96RXAM5q/P5d8Lt3DjsDSGdE5wOxyllDoumggaqbCsit9NXkZ6UjT3n9Pd7XCUUuq46ZhGIz01YxU5RRVMuW0oEaHBboejlFLHTa8IGuGrlbuY8lM2t49IZ0CHOLfDUUopW2giaKD84goenracXm1bcOcZXd0ORymlbKNDQw1gjOHRT1ZQWFbF+zcPJixE86dSyn/oEa0BpmfuYOaKXdxzVjd6tGnhdjhKKWUrTQRHsXtfOb//ZAUndIzj1uHpboejlFK200RwBMYYHpiyjMoaDy+P7U9wkM4xoJTyP5oIjuCjRduYszaXB0f1oHNSjNvhKKWUIzQRHMa2PaU889kqhqYn8KuTU90ORymlHKOJoB4ej+G+SZmICC9e1o8gHRJSSvkxTQT1+OeCzfy4aQ+PXdCLlPgot8NRSilHaSKoIyunmBe/XMPIHsmMzUhxOxyllHKcJoJaqms8TJiUSWRYMM9d2hcRHRJSSvk//WZxLa/N3UDmtr38/aoTSG4R4XY4SinVJPSKwGvljkL++u16LujXlgv6tXM7HKWUajKaCICK6homTMwkLiqMpy/q43Y4SinVpHRoCPjrN+tZs6uIt6/LID46zO1wlFKqSQX8FcFPWwt4be4GLs9IYWTP1m6Ho5RSTS6gE0FZZQ33TcykbctIfn9BL7fDUUopVwT00NALX65hY14JH9w8mNiIULfDUUopVwTsFcGCrDzeXbCZ64emMrRLotvhKKWUawIyERSVV3H/5GWkJUbzwKgeboejlFKuCsihoWc+W83OwjIm/XookWHBboejlFKuCrgrgu/W7Objxdu49bR0TuwU73Y4SinluoBKBAUllTwwZTk92sRy95ld3Q5HKaV8QkANDT02fSUFJZW8e8NJhIfokJBSSkEAXRF8tmwHMzJ3cNfIrvRu19LtcJRSymcETCJoERHKWb1ac9uIdLdDUUopn+JoIhCRUSKyVkSyROTBetaHi8jH3vU/ikiqU7EM75bEm7/KICQ4YHKfUko1iGNHRREJBl4FzgV6AVeKSN06DjcBBcaYLsCfgRecikcppVT9nDw9HgRkGWM2GmMqgY+Ai+pscxHwnvf1ZGCk6LRgSinVpJxMBO2BbbXeZ3uX1buNMaYaKAQS6jYkIuNFZLGILM7NzXUoXKWUCkzNYsDcGPOGMSbDGJORlJTkdjhKKeVXnEwE24EOtd6neJfVu42IhAAtgXwHY1JKKVWHk4lgEdBVRNJEJAy4ApheZ5vpwHXe15cB3xljjIMxKaWUqsOxbxYbY6pF5A5gFhAMvGOMWSkiTwGLjTHTgbeBf4tIFrAHK1kopZRqQo6WmDDGfAF8UWfZY7VelwNjnYxBKaXUkUlzG4kRkVxgyzH+eCKQZ2M4dtG4GkfjajxfjU3japzjiauTMabep22aXSI4HiKy2BiT4XYcdWlcjaNxNZ6vxqZxNY5TcTWLx0eVUko5RxOBUkoFuEBLBG+4HcBhaFyNo3E1nq/GpnE1jiNxBdQ9AqWUUr8UaFcESiml6tBEoJRSAc4vE4EvTYjTyLiuF5FcEVnq/XNzE8X1jojkiMiKw6wXEXnFG/cyERnoI3GNEJHCWvvrsfq2szmmDiIyW0RWichKEbmrnm2afH81MC439leEiPxPRDK9cT1ZzzZN/nlsYFyufB69fQeLyM8i8lk96+zfX8YYv/qDVc5iA9AZCAMygV51trkdeM37+grgYx+J63rg7y7ss+HAQGDFYdafB8wEBBgC/OgjcY0APmvifdUWGOh9HQusq+ffscn3VwPjcmN/CRDjfR0K/AgMqbONG5/HhsTlyufR2/e9wAf1/Xs5sb/88YrAVyfEaUhcrjDGzMOq9XQ4FwH/MpaFQJyItPWBuJqcMWanMeYn7+siYDW/nGejyfdXA+Nqct59UOx9G+r9U/cJlSb/PDYwLleISApwPvDWYTaxfX/5YyKwbUIcF+ICGOMdTpgsIh3qWe+GhsbuhpO9l/czRaR3U3bsvSQ/AetssjZX99cR4gIX9pd3mGMpkAN8bYw57P5qws9jQ+ICdz6PfwF+B3gOs972/eWPiaA5mwGkGmP6AV9zMOur+v2EVT+lP/A34JOm6lhEYoApwN3GmH1N1e/RHCUuV/aXMabGGDMAa06SQSLSpyn6PZoGxNXkn0cRuQDIMcYscbqv2vwxEfjqhDhHjcsYk2+MqfC+fQs40eGYGqoh+7TJGWP27b+8N1al21ARSXS6XxEJxTrY/scYM7WeTVzZX0eLy639Vav/vcBsYFSdVa5OUHW4uFz6PA4DRovIZqzh4zNE5P0629i+v/wxEfjqhDhHjavOOPJorHFeXzAd+JX3aZghQKExZqfbQYlIm/1joyIyCOv/s6MHEG9/bwOrjTF/OsxmTb6/GhKXS/srSUTivK8jgbOANXU2a/LPY0PicuPzaIx5yBiTYoxJxTpGfGeMuabOZrbvL0fnI3CD8dEJcRoY129FZDRQ7Y3reqfjAhCRD7GeKEkUkWzgcaybZxhjXsOaU+I8IAsoBW7wkbguA24TkWqgDLiiCRL6MOBaYLl3fBngYaBjrbjc2F8NicuN/dUWeE9EgrESz0RjzGdufx4bGJcrn8f6OL2/tMSEUkoFOH8cGlJKKdUImgiUUirAaSJQSqkAp4lAKaUCnCYCpZQKcJoIlGpCYlUA/UVFSaXcpIlAKaUCnCYCpeohItd469UvFZHXvQXKikXkz9769d+KSJJ32wEistBbnGyaiMR7l3cRkW+8Rd5+EpF0b/Mx3iJma0TkP05X2lTqaDQRKFWHiPQExgHDvEXJaoCrgWisb3f2BuZifdMZ4F/AA97iZMtrLf8P8Kq3yNtQYH+ZiROAu4FeWPNTDHP4V1LqiPyuxIRSNhiJVWBskfdkPRKrVLEH+Ni7zfvAVBFpCcQZY+Z6l78HTBKRWKC9MWYagDGmHMDb3v+MMdne90uBVOB7x38rpQ5DE4FSvyTAe8aYhw5ZKPL7Otsda32Wilqva9DPoXKZDg0p9UvfApeJSDKAiLQSkU5Yn5fLvNtcBXxvjCkECkTkVO/ya4G53lnCskXkYm8b4SIS1ZS/hFINpWciStVhjFklIo8CX4lIEFAF/AYowZrA5FGsoaJx3h+5DnjNe6DfyMFqo9cCr3srR1YBY5vw11CqwbT6qFINJCLFxpgYt+NQym46NKSUUgFOrwiUUirA6RWBUkoFOE0ESikV4DQRKKVUgNNEoJRSAU4TgVJKBbj/BxRcKuSQ2I+yAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(model.history.history.keys())\n",
    "# summarize history for accuracy\n",
    "plt.plot(model.history.history['rec@prec'])\n",
    "plt.plot(model.history.history['val_rec@prec'])\n",
    "plt.title('model f1')\n",
    "plt.ylabel('f1')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'val'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "А теперь попробуем со сверточным слоем. В керасе они называются Conv1D (2D тоже есть, они используются для картинок). Их нужно ставить сразу после Embedding слоя."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs = tf.keras.layers.Input(shape=(MAX_LEN,))\n",
    "embeddings = tf.keras.layers.Embedding(input_dim=len(word2id), output_dim=30)(inputs, )\n",
    "\n",
    "# все остальное точно такое же\n",
    "# возвращает сверточный слой тоже последовательность векторов слегка измененной или такой же длины\n",
    "conv1 = tf.keras.layers.Conv1D(kernel_size=10, filters=128, strides=5)(embeddings)\n",
    "\n",
    "# выход со сверточного слоя также можно просто конкатенировать\n",
    "concat = tf.keras.layers.Flatten()(conv1)\n",
    "dense = tf.keras.layers.Dense(64, activation='relu')(concat)\n",
    "outputs = tf.keras.layers.Dense(len(label2id), activation='softmax')(dense)\n",
    "\n",
    "model = tf.keras.Model(inputs=inputs, outputs=outputs)\n",
    "optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)\n",
    "model.compile(optimizer=optimizer,\n",
    "              loss='categorical_crossentropy',\n",
    "              metrics=[f1, tf.keras.metrics.RecallAtPrecision(0.8, name='rec@prec')])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_7\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_11 (InputLayer)        [(None, 201)]             0         \n",
      "_________________________________________________________________\n",
      "embedding_9 (Embedding)      (None, 201, 30)           1659180   \n",
      "_________________________________________________________________\n",
      "conv1d_5 (Conv1D)            (None, 39, 128)           38528     \n",
      "_________________________________________________________________\n",
      "flatten_7 (Flatten)          (None, 4992)              0         \n",
      "_________________________________________________________________\n",
      "dense_14 (Dense)             (None, 64)                319552    \n",
      "_________________________________________________________________\n",
      "dense_15 (Dense)             (None, 19)                1235      \n",
      "=================================================================\n",
      "Total params: 2,018,495\n",
      "Trainable params: 2,018,495\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "141/141 [==============================] - 8s 44ms/step - loss: 2.1177 - f1: 0.1492 - rec@prec: 0.1014 - val_loss: 0.9621 - val_f1: 0.6896 - val_rec@prec: 0.6129\n",
      "Epoch 2/5\n",
      "141/141 [==============================] - 5s 38ms/step - loss: 0.7510 - f1: 0.7637 - rec@prec: 0.7556 - val_loss: 0.7174 - val_f1: 0.7744 - val_rec@prec: 0.7629\n",
      "Epoch 3/5\n",
      "141/141 [==============================] - 5s 38ms/step - loss: 0.4071 - f1: 0.8754 - rec@prec: 0.9221 - val_loss: 0.7081 - val_f1: 0.7906 - val_rec@prec: 0.7851\n",
      "Epoch 4/5\n",
      "141/141 [==============================] - 5s 38ms/step - loss: 0.2123 - f1: 0.9410 - rec@prec: 0.9791 - val_loss: 0.8861 - val_f1: 0.7800 - val_rec@prec: 0.7626\n",
      "Epoch 5/5\n",
      "141/141 [==============================] - 5s 38ms/step - loss: 0.0864 - f1: 0.9800 - rec@prec: 0.9965 - val_loss: 1.1194 - val_f1: 0.7797 - val_rec@prec: 0.7587\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f70a0682588>"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# как выглядит ошибка при слишком большом батче\n",
    "model.fit(X_train, y_train, \n",
    "          validation_data=(X_valid, y_valid),\n",
    "          batch_size=1000,\n",
    "         epochs=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['loss', 'f1', 'rec@prec', 'val_loss', 'val_f1', 'val_rec@prec'])\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAudElEQVR4nO3deXwVhbn/8c9DVgIhgRC2JCwCCggCkiIVbd1oUVvsrbXg1tqNX1vtYpd79dZu3lZt676L1attXavVixaLoChaN0BBRSAsigkghCUhLNmf3x/nJBxCwprJnOR8369XXpw5M5nzZPTMd7ZnxtwdERFJXJ3CLkBERMKlIBARSXAKAhGRBKcgEBFJcAoCEZEEpyAQEUlwCgKRI2RmD5jZ7w5y2o/M7IwWxnU2s2fMrNzM/t66VYq0TEEgEj++AvQGctz9PDPra2YzzWy9mbmZDQy5PumgFAQi8WMAUOTutdHheuBfwLnhlSSJQEEgCSF6SObnZvaume00s/vMrLeZPWdmFWY218y6x0w/xcyWmlmZmb1kZsNjxo01s7ejv/cYkN7ks75gZoujv/uamR13EPX9FvgVMNXMdpjZt9x9o7vfCSxovSUhsi8FgSSSc4FJwNHAF4HngP8Gcol8F34IYGZHA48AP46OmwU8Y2apZpYKPA38FegB/J2YLXYzGwvcD/w/IAe4B5hpZmn7K8zdfw1cAzzm7l3d/b5W+YtFDoKCQBLJbdGt7HXAK8Cb7v6Ou1cCTwFjo9NNBf7p7nPcvQa4HugMnAhMAFKAm929xt2fYO8t9unAPe7+prvXufuDQFX090TiUnLYBYi0oY0xr3c3M9w1+rofsLZhhLvXm1kxkAfUAet877s1ro15PQD4upn9IOa91Og8ReKSgkBkX+uBUQ0DZmZAAbAOcCDPzCwmDPoDq6Ovi4Hfu/vv27BekSOiQ0Mi+3ocONvMTjezFOCnRA7vvAa8DtQCPzSzFDP7MjA+5nfvBb5rZidYRBczO9vMMg+nEDNLBxrOL6RFh0ValYJApAl3XwFcBNwGbCZyYvmL7l7t7tXAl4FLgK1Ezif8I+Z3FwLfAW4HtgGrotMert3Ajujr5dFhkVZlejCNiEhi0x6BiEiCUxCIiCQ4BYGISIJTEIiIJLh210fQs2dPHzhwYNhliIi0K4sWLdrs7rnNjWt3QTBw4EAWLlwYdhkiIu2Kma1taZwODYmIJDgFgYhIglMQiIgkuHZ3jqA5NTU1lJSUUFlZGXYpgUpPTyc/P5+UlJSwSxGRDiSwIDCz+4EvAJvcfWQz4w24BTgL2AVc4u5vH85nlZSUkJmZycCBA4nMtuNxd7Zs2UJJSQmDBg0KuxwR6UCCPDT0ADB5P+PPBIZGf6YDdx3uB1VWVpKTk9NhQwDAzMjJyenwez0i0vYCCwJ3n0/k7owtOQf4i0e8AWSbWd/D/byOHAINEuFvFJG2F+Y5gjwiD/FoUBJ9b0PTCc1sOpG9Bvr3798mxYmItLWaunrKd9dQvruGsl01lO+ubnxdtquG04f34rj87Fb/3HZxstjdZwAzAAoLC+PuvtllZWU8/PDDfP/73z+k3zvrrLN4+OGHyc7ODqYwEWlz7s6u6jrKdtdQtiuyIi/fVRMdromu6KsbX+95r4YdVbX7nXduZlqHC4J1RB7/1yA/+l67U1ZWxp133rlPENTW1pKc3PIinjVrVtClichhqo1unZdFV9KRlXn1Xiv17btr9l7hR9+vrW95ezU1qRNZGSlkdU4hu3MK/bLTGdY3k+zOqWQ3vB/9N/I6lezOKWSmJ5OcFMzR/DCDYCZwmZk9CpwAlLv7PoeF2oMrrriC1atXM2bMGFJSUkhPT6d79+4sX76coqIivvSlL1FcXExlZSU/+tGPmD59OrDndhk7duzgzDPP5KSTTuK1114jLy+P//u//6Nz584h/2Ui7Vvs1nnsirx894G30A+0dZ6Znty40s7unErfrM5kZURW7ntW5ql7pomu3DunJMXd+b4gLx99BDgF6GlmJcCvgRQAd78bmEXk0tFVRC4f/UZrfO5vn1nKB+u3t8asGo3o141ff/HYFsdfd911vP/++yxevJiXXnqJs88+m/fff7/xMs/777+fHj16sHv3bj71qU9x7rnnkpOTs9c8Vq5cySOPPMK9997LV7/6VZ588kkuuuiiVv07RNqr2rp6tlfWUrareu8t9KbDjVvle7bQa+pa3jpPSTKyolvi2Z1T6NMtnWP6RLbOY1fg3aJb79kZkfe7Bbh1HobAgsDdzz/AeAcuDerzwzR+/Pi9rvW/9dZbeeqppwAoLi5m5cqV+wTBoEGDGDNmDADjxo3jo48+aqtyRdpcRWUNyzZUsHVn1Z4t8b0Ot+zZQi/fVUPFgbbO05L3HG7JSGFYn26RlXdGwwq84VDL3odf4nHrPAzt4mTxodjflntb6dKlS+Prl156iblz5/L666+TkZHBKaec0mwvQFpaWuPrpKQkdu/WM8qlY6ipq2f5hgoWl5Sx+OMylpSUsbp0B00fl96wdZ7VOZnsjNTI1nnvzOjhlj3vxx5fz85I7XBb52HocEEQhszMTCoqKpodV15eTvfu3cnIyGD58uW88cYbbVydSNtxdz7euovFxWUsKS5ncfE2lq7fTlVtPQA9uqQypiCbKaP7MSovi97d0hu30DNStXUeFgVBK8jJyWHixImMHDmSzp0707t378ZxkydP5u6772b48OEcc8wxTJgwIcRKRVrX1p3VLInZ0l9SXMa2XTUApCV3YlReFhdPGMDogmzGFGST372zVvZxyLzp/lmcKyws9KYPplm2bBnDhw8PqaK2lUh/q8SXypo6lq7fHt3aj6z4127ZBYAZDO3VlTEF2YwuyGZ0fjbH9MkkRYds4oaZLXL3wubGaY9ARPZRX++s2byDdxq39MtZtmF74/XxfbqlM6Ygm2mf6s+YgmxG5WfRNU2rk/ZK/+VEhE3bK3knZkv/3eLyxit1uqYlc1x+Ft/5zFGRLf78bPpkpYdcsbQmBYFIgtlZVcu7JeWNx/QXF5exoTxyJVtyJ2NY30ymjOnHmOhx/aNyu5LUScf1OzIFgUgHVltXz4qNFSwpLm9c6a/cVEHDHRD698igcGAPRudnMbZ/Nsf2yyI9JSncoqXNKQhEOgh3p2Tb7r229N9bV05lTeTSzeyMFEbnZzN5ZB/GFGRzXH4WOV3TDjBXSQQKApF2qnxXzV4r/SUlZWzeUQ1AanInju3XjfPH9288rj8gJ0OXbkqzFAQh6Nq1Kzt27Ai7DGlHqmrrWLahYs9Kv7iMNZt3No4fnNuFzx7dizEFWYwuyGZYn26kJuvSTTk4CgKROFNf73y0ZWdjo9biknKWrd9OdV3kEE9uZhpjCrI5d1w+o/OzOa4gi27pKSFXLe2ZgqAVXHHFFRQUFHDppZF76P3mN78hOTmZefPmsW3bNmpqavjd737HOeecE3KlEo8276hq7Mxt2NrfXhm5dDMjNYlReVl8Y+LAxu7cvlnpOsQjrarjBcFzV8An77XuPPuMgjOva3H01KlT+fGPf9wYBI8//jizZ8/mhz/8Id26dWPz5s1MmDCBKVOm6Auc4HZX1/HeuugVPNEt/nVlkRsMdjI4pk83zj6uL6PzsxnTP5shuV11QzUJXMcLghCMHTuWTZs2sX79ekpLS+nevTt9+vTh8ssvZ/78+XTq1Il169axceNG+vTpE3a50kbq6p2VmxqO65ezuLiMoo0V1EWv3czL7syYgmy+fuIAxhR0Z2ReNzJS9ZWUttfx/q/bz5Z7kM477zyeeOIJPvnkE6ZOncpDDz1EaWkpixYtIiUlhYEDBzZ7+2npOD4pr+Sdj7c1bum/t66cXdV1QORpVmMKsjl92ODIpZsFWfTKVHeuxIeOFwQhmTp1Kt/5znfYvHkzL7/8Mo8//ji9evUiJSWFefPmsXbt2rBLlIAs/2Q7N80pYvbSjUDkvvoj+nbjK+PyG2/CNiinC53UnStxSkHQSo499lgqKirIy8ujb9++XHjhhXzxi19k1KhRFBYWMmzYsLBLlFa2unQHN89dybPvrqdrajI/OG0Ipw3rxYh+3UhLVneutB8Kglb03nt7TlL37NmT119/vdnp1EPQvhVv3cUtL6zkH2+XkJacxHc/O5jpJx9F9y6pYZcmclgCDQIzmwzcAiQBf3b365qMHwDcD+QCW4GL3L0kyJpEDteG8t3c/uIqHltQTKdOxjcmDuK7nx1MbqZu0yDtW2BBYGZJwB3AJKAEWGBmM939g5jJrgf+4u4PmtlpwLXAxUHVJHI4SiuquPOlVTz05se4O+eP78+lpw7RrZilwwhyj2A8sMrd1wCY2aPAOUBsEIwAfhJ9PQ94+nA/zN07/DX67e1pcu3dtp3V3DN/DQ++9hHVdfWce3wePzhtKAU9MsIuTaRVBRkEeUBxzHAJcEKTaZYAXyZy+Og/gEwzy3H3LYfyQenp6WzZsoWcnJwOGwbuzpYtW0hP11Zo0LZX1nDfKx9y36sfsrO6limj+/Gj04dyVG7XsEsTCUTYJ4t/BtxuZpcA84F1QF3TicxsOjAdoH///vvMJD8/n5KSEkpLSwMtNmzp6enk5+eHXUaHtbOqlgde+4gZ89dQvruGycf24fJJR3NMn8ywSxMJVJBBsA4oiBnOj77XyN3XE9kjwMy6Aue6e1nTGbn7DGAGRB5e33R8SkoKgwYNarXCJbFU1tTxtzfWctdLq9mys5rThvXiJ5OOZmReVtilibSJIINgATDUzAYRCYBpwAWxE5hZT2Cru9cDVxK5gkikTVTX1vPYgo+5fd4qNm6vYuKQHH4y6RjGDegedmkibSqwIHD3WjO7DJhN5PLR+919qZldDSx095nAKcC1ZuZEDg1dGlQ9Ig1q6+r5x9vruOWFlawr203hgO7cPHUsnx6cE3ZpIqGw9nYlSmFhoS9cuDDsMqQdqqt3nn13PTfPXcmHm3dyXH4WP/3cMXxmaM8Oe5GBSAMzW+Tuhc2NC/tksUjg3J3ZSz/hxjlFFG3cwbA+mcy4eByTRvRWAIigIJAOzN2Zt2ITNzxfxNL12zkqtwu3nT+Ws0f11Q3gRGIoCKTDcXf+vWoLN8xZwTsfl1HQozM3nDeac8b000NeRJqhIJAOZcFHW7l+9gre/HArfbPSueY/RnFeYT4pCgCRFikIpENYUlzGDXOKmF9USs+uafzmiyOYNr4/6Sm6HbTIgSgIpF1btmE7N84pYs4HG+mekcKVZw7ja58eSOdUBYDIwVIQSLu0atMObp5bxLPvbiAzLZmfTDqab0wcSGZ6StilibQ7CgJpVz7esoubXyji6XfWkZ6SxGWnDuE7Jx9FVoYCQORwKQikXVhftpvbXlzF3xcWk9TJ+NZJkYfC5HTVQ2FEjpSCQOLapu2V3PnSah5+82Mc54ITIg+F6d1Nt+MWaS0KAolLW3dWc8/Lq3nw9Y+oqXPOG5fPZacNIb+7Hgoj0toUBBJXynfXcN8ra7jv1Q/ZVVPHl8bk8aPThzKwZ5ewSxPpsBQEEhd2VNXywL8/ZMb8NWyvrOWsUX24/IyjGdpbD4URCZqCQEJVWVPHX19fy10vr2brzmrOGN6LyycdzbH99FAYkbaiIJBQVNXW8diCYm5/cRWbKqo4eWhPfjLpaMb210NhRNqagkDaVE1dPU8uKuG2F1exrmw34wf24NbzxzLhKD0URiQsCgJpE3X1zswl67h57krWbtnF6IJsrjt3FCcN0UNhRMKmIJBA1dc7z73/CTfNLWLVph0M79uNP3+tkNOH91IAiMQJBYEEwt15YdkmbphTxLIN2xnSqyt3XHA8Z47so4fCiMQZBYG0Knfn1VWbuf75IpYUlzEgJ4Obpo5myug8khQAInEp0CAws8nALUAS8Gd3v67J+P7Ag0B2dJor3H1WkDVJcN5cs4Ub5hTx1odb6ZeVznVfHsW54/RQGJF4F1gQmFkScAcwCSgBFpjZTHf/IGayq4DH3f0uMxsBzAIGBlWTBGNxcRk3PL+CV1ZuJjczjd9OOZZp4wtIS9YzAUTagyD3CMYDq9x9DYCZPQqcA8QGgQPdoq+zgPUB1iOtbOn6cm6aU8TcZZvo0SWVX5w1nIsmDNBDYUTamSCDIA8ojhkuAU5oMs1vgOfN7AdAF+CM5mZkZtOB6QD9+/dv9ULl0KzcWMFNc4uY9d4ndEtP5mefO5pLJg6ia5pOOYm0R2F/c88HHnD3G8zs08BfzWyku9fHTuTuM4AZAIWFhR5CnQJ8tHknt7ywkqcXryMjJYkfnjaEb518FFmd9VAYkfYsyCBYBxTEDOdH34v1LWAygLu/bmbpQE9gU4B1ySFaV7ab215Yyd8XlZCSZEw/+Sj+32cH06NLatiliUgrCDIIFgBDzWwQkQCYBlzQZJqPgdOBB8xsOJAOlAZYkxyCTdsruWPeKh55K3KE7+IJA/j+KYPplYgPhamvg11bYddm2Lk55t8tsLM08rpqO6RnQ5eekNEz8m+XntAld89wejZ00lVUEl8CCwJ3rzWzy4DZRC4Nvd/dl5rZ1cBCd58J/BS418wuJ3Li+BJ316GfOPDa6s18438XUFfvnFdYwGWnDSEvu3PYZbWeutrISnzX5j0r8l1bmlnJR8fv3kbkf9FmNKz807rBtrWR368sb35aS4KMnEg4dMmJBkRuNDxy9rzukhsZVnBIG7D2tt4tLCz0hQsXhl1Gh1ZbV8+Zt7xCVW09f/3WeAbktIOHwtRWN9laP8BKvrKshRkZZPTYswWfkdNkqz4nZlzPyLRJzZwjqa1uEjRN6omtZedmqNpPcDTuYeTE1NFCkHTuDrp1hzTDzBa5e2Fz48I+WSxx6PGFJazctIO7Ljw+vBCo2d3MSr2FrfVdWyKHZZrTuAUeXVH2GbWflXzPyIq0Uytc/pqcCt36Rn4ORkNw7CzdOyBig2RnKax/J/K6peDolLxnz6K5EGsaJOnZCg5REMjedlTVcuOcIgoHdGfyyD6tN+PqnQe/tb5rC1TvaH4+nZL3XpHnjdt7izl2a709HZM/5OCo2jcMG5dnTJCsf2fP+YvmNF2ejYemYs9zxASJgqNDUhDIXma8vJrNO6q492vjWr47qDtUVRxga73JSr52d/PzSkrbe+u8x+B9V0qxK6r0LK2IAJLToFu/yM/BqK3aNyQaX8cEybpF+9/D6pTSwjmOnL33rBrPcei/V3ugIJBGn5RXMuOVNXzhuL6M7bYDXn0Admxq/pBMXVXzM0nuvPfVMr2Gxxya6ck+x7tTu2pF0RaS0yArL/JzMGoqmxyqauGw1baPIuOqK5qfT6eUfa+iig325LTI4btOSZG9k07JMa+j/1rTcU2G9ze+cVw72CsMkYJAGt3w/Ao61+/imm7/gNvvhdrKyIq6YUWe2TdyjL25lXrDcGo7OLEsB5aSfhjB0cxJ8KZBsu3DyPstHfoLjB16yFinmHEthUwLv39I4RUNqtjhveYd82/uMMjKb/WloyAQAD4o2Ura4v/l1c5P02XhNjhuKpx2FWTrlh5yEFLSIyuog11JNexx1NdEejTqa2P+jb722OFaqK/fe9jr954+9l9vMq/9zrvJeG9m+oZ/a6tifvdA822m5pYuQT5YX7gJCr95ZPNohoIg0bnjRf8i6+8/53cpxdT2+TSceQ3kHR92ZdKRNexxJJr6+iZBcojhlT0gkLIUBIlsw7vw/C+wD+dTWd+XOWNuYtJ/fEPH7EWC0qkT0Kn53pMQKQgS0fb18ML/wJJH8M7duTN9Ok92+hz/mnK6QkAkASkIEklVBfz7Vnjttshu5ok/4MkuU/nTM2u588KRpCbrygqRRKQgSAR1tbD4b/Di72HnJhh5Lpz+a3Zm5HHdn15i3IDunNmazWMi0q4oCDq6lXNhzi9h0wdQMAHOfwTyI7cbuWdOEZt3VDFjf81jItLhKQg6qo1L4fmrYPWL0H0QfPUvMHxK4zmAT8ormTF/NWcf15fj+3cPuVgRCZOCoKOp+ARe/B0sfihyW+TPXwuf+nbkXjYxbpyzgrp6578+PyykQkUkXigIOorqnZGTwP++Feqq4YTvwWd+FrlNchPLNmzn74tK+NbEQfTPyQihWBGJJwqC9q6+DpY8EtkLqNgAI86BM34DPY5q8VeumbWMbukpXHbakLarU0TiloKgPVs9D57/JWx8D/IK4bwHoP+E/f7Ky0WlvLJyM1edPZzsDD1zWEQUBO3TpmWRAFg1J3IvoK/cD8d++YDNYHX1zjX/XEb/Hhlc/OlgWtVFpP1RELQnOzbBvGvg7QchNRMm/Q+Mnx65b8tBeGJRMSs2VnDHBceTltwKT+ESkQ4h0CAws8nALUQeXv9nd7+uyfibgFOjgxlAL3fPDrKmdql6F7xxB7x6c+TW0OOnw2f+M3L754O0s6qW658v4vj+2Zw1Ss1jIrJHYEFgZknAHcAkoARYYGYz3f2Dhmnc/fKY6X8AjA2qnnapvh7eexxeuBq2r4NhX4Azfgs9D/0k74z5ayitqOLui9Q8JiJ7C3KPYDywyt3XAJjZo8A5wActTH8+8OsA62lfPnwFnv8FbFgC/cbCl++FgRMPa1Ybt1cyY/4azh7Vl3ED1DwmInsLMgjygOKY4RLghOYmNLMBwCDgxRbGTwemA/Tv38EflFJaBHN+BUXPQbf8SACM/MoRPWrvxueLqK2v5z8nH9OKhYpIRxEvJ4unAU+4e11zI919BjADoLCw8Agf8ROndm6Gl66DhfdDSgac/muY8D1I6XxEs122YTuPLyrmmxMHMSBHj5EUkX0FGQTrgIKY4fzoe82ZBlwaYC3xq6YS3rwLXrkx0h1c+A347BXQNbdVZn/tc8vJTEvmB2oeE5EWBBkEC4ChZjaISABMAy5oOpGZDQO6A68HWEv8qa+H95+EF34L5cVw9Jkw6beQ23qHb14uKmV+Uamax0RkvwILAnevNbPLgNlELh+9392XmtnVwEJ3nxmddBrwqLt3zEM+zVn7Gsz+Bax/G/ocB+fcAUd9tlU/oq7euXbWMgp6dFbzmIjsV6DnCNx9FjCryXu/ajL8myBriCtbVkdOBC9/FjL7wZfuhuOmHtGJ4JY8uaiE5Z9UcPsFY9U8JiL7FS8nizu2XVvh5T/CgnshKQ1OvQo+fSmkBnPnz0jz2ArG9s/m7FF9A/kMEek4FARBqq2Ct2bA/D9Fnhd8/NfglP+GzN6Bfuy9r6xhU0UVd110vJrHROSAFARBcIelT8Hc30DZWhgyCSZdDb1HBP7Rm7ZXcs/LazhrVB/GDdj3WQQiIk0pCFpb8VuRE8Elb0GvY+Hip2DwaW328TfOiTaP6cljInKQFAStZeuHkT2AD56Grn1gyu0w5gLo1HYnapd/sp3HFxZzyYmDGNhTzWMicnAUBEdq9zaYf33kXECn5Egz2Ik/gLSubV7KtbOW01XNYyJyiBQEh6u2GhbeBy//AXaXwdgLI1cDdQvnKp35RaW8XFTKL84aTvcuah4TkYN3WEFgZl3dfUdrF9MuuMOyZ2Dur2HrGjjqVPjc76DPyNBKqqt3rpm1jPzunfnaiWoeE5FDc7h7BB8AHfw2oM0oWQTPXwUfvwa5w+DCJ2DIGQd8RGTQnnw70jx22/lqHhORQ9diEJjZT1oaBbT9AfAwlX0Mc38L7z8BXXLhCzfD2IshKfwja7uqa7l+9grGFGTzhePUPCYih25/a7JrgD8Btc2Ma/17IsSjynJ45QZ44+7IVv/JP4OTfgxpmWFX1uje+R+yqaKKOy9U85iIHJ79BcHbwNPuvqjpCDP7dnAlxYG6Glj0ALx0beT2EKOnwWm/hKy8sCvby6btldwzfzVnjuxD4UA1j4nI4dlfEKwD1prZj9z9libjCgOsKTzusOK5yI3htqyEgSdHTgT3GxN2Zc26aW4R1bX1/NdkNY+JyOHbXxCMAFKBb5rZX4icG2hQE2hVYVj/Djz/S/joFcgZCuc/CkdPDv1EcEtWfFLBYwuK+fqJA9U8JiJHZH9BcA/wAnAUsIi9g8Cj77d/5SXwwv/Au49CRg6cdT2MuwSSUsKubL+ufW4ZXdKS+eFpQ8MuRUTauRaDwN1vBW41s7vc/XttWFPbqKqAV2+C1++IHBI66fLIT3pW2JUd0CsrS3lpRSn/fdYwNY+JyBE74PWPHS4E6mrh7QcjJ4J3lsKor8Lpv4Ts9tEWUVfv/P6f0eaxTw8MuxwR6QDCvxC+rbjDyjkw55dQuhz6nwgXPAZ548Ku7JD8I9o8duv5Y0lPUfOYiBy5xAmC+dfDvN9Bj8Ew9SEYdnbcnghuye7qOq5/fgWjC7L5oprHRKSVBBoEZjYZuIXIw+v/7O7XNTPNV4HfEDkBvcTdLwikmJFfjjSCFX4TktvncfV7X1nDxu1V3H6BmsdEpPUEFgRmlgTcAUwCSoAFZjbT3T+ImWYocCUw0d23mVmvoOohZ3Dkp53aVFHJ3S+vZvKxffiUmsdEpBUFeauI8cAqd1/j7tXAo8A5Tab5DnCHu28DcPdNAdbTrt00Z2WkeexMNY+JSOsKMgjygOKY4ZLoe7GOBo42s3+b2RvRQ0n7MLPpZrbQzBaWlpYGVG78KtpYwWMLPuaiCQMYpOYxEWllYd88LhkYCpwCnA/ca2bZTSdy9xnuXujuhbm5uW1bYRy4dla0eex0NY+JSOsLMgjWAQUxw/nR92KVADPdvcbdPwSKiASDRL26cjPzVpRy2alD6KHmMREJQJBBsAAYamaDzCwVmAbMbDLN00T2BjCznkQOFa0JsKZ2pa7e+f2sZeRld+brJw4MuxwR6aACCwJ3rwUuA2YDy4DH3X2pmV1tZlOik80GtpjZB8A84OfuviWomtqbp95Zx7IN2/nPyceoeUxEAmPuHnYNh6SwsNAXLlwYdhmB211dx6nXv0Tvbmk89f2JdOqkvgEROXxmtsjdm32EQNgni6UFf35lDZ9sr+QXZ49QCIhIoBQEcWhTRSV3vbyazx/bm/GD1DwmIsFSEMShm+eu1JPHRKTNKAjizMqNFTz6VqR57KjcrmGXIyIJQEEQZ659bjldUtU8JiJtR0EQR/69ajMvLt/EpaepeUxE2o6CIE7UR588lpfdmUvUPCYibUhBECeeemcdH6h5TERCoCCIAw1PHjsuP4svHtcv7HJEJMEoCOLAfa+uYUN5Jb84a7iax0SkzSkIQlZaUcVdL63mcyN6c8JROWGXIyIJSEEQspvnFlFVW88VevKYiIREQRCilRsreHRBMRee0F/NYyISGgVBiK57bjkZKUlqHhORUCkIQvLaqs28sHwT3z91CDld08IuR0QSmIIgBPUxTx77xsSBYZcjIglOQRCCpxevY+n67fz882oeE5HwKQjaWGVNHX+avYJReVlMGa3mMREJn4Kgjd336oeR5rGz1TwmIvEh0CAws8lmtsLMVpnZFc2Mv8TMSs1scfTn20HWE7bSiirunLeKSSN6M0HNYyISJ5KDmrGZJQF3AJOAEmCBmc109w+aTPqYu18WVB3x5JYXiqhU85iIxJkg9wjGA6vcfY27VwOPAucE+HlxbdWmCh55K9I8NljNYyISR4IMgjygOGa4JPpeU+ea2btm9oSZFQRYT6iue245nVOS+JGax0QkzoR9svgZYKC7HwfMAR5sbiIzm25mC81sYWlpaZsW2BpeW72Zucs28f1TB6t5TETiTpBBsA6I3cLPj77XyN23uHtVdPDPwLjmZuTuM9y90N0Lc3NzAyk2KPX1zjWzltEvK51vThwUdjkiIvsIMggWAEPNbJCZpQLTgJmxE5hZ35jBKcCyAOsJxf8tWcf767bzcz15TETiVGBXDbl7rZldBswGkoD73X2pmV0NLHT3mcAPzWwKUAtsBS4Jqp4wVNbU8ad/rWBkXjfOGd3c6RERkfAFFgQA7j4LmNXkvV/FvL4SuDLIGsJ0/78/ZH15JTd8dYyax0QkboV9srjD2ryjijvnreaM4b359GA1j4lI/FIQBOSWuSvZXVOn5jERiXsKggCs2rSDh9/6mAvG92dILzWPiUh8UxAEoLF57Aw1j4lI/FMQtLLXV29h7rKNfO+UwfRU85iItAMKglbU0DzWNyudb52k5jERaR8UBK1o5pL1vLeuXE8eE5F2RUHQShqePHZsv258aYyax0Sk/VAQtJL//fdHrCvbrSePiUi7oyBoBVt2RJ48dsbwXpw4uGfY5YiIHBIFQSu45YWV7FLzmIi0UwqCI7S6dAcPvfkx548vYEivzLDLERE5ZAqCI9TQPPbjM44OuxQRkcOiIDgCb6zZwpwP1DwmIu2bguAwxTaP6cljItKeKQgO0zPvrufdknJ+9rlj6Jyq5jERab8UBIehsqaOP/5rBSP6duM/xqp5TETaNwXBYXjgtUjz2FVqHhORDkBBcIi27KjijhdXcfqwXpw4RM1jItL+KQgO0a3R5rErz1LzmIh0DIEGgZlNNrMVZrbKzK7Yz3TnmpmbWWGQ9RyphuaxaZ9S85iIdByBBYGZJQF3AGcCI4DzzWxEM9NlAj8C3gyqltbyh+eWk5bcSc1jItKhBLlHMB5Y5e5r3L0aeBQ4p5np/gf4A1AZYC1H7M01W3g+2jyWm6nmMRHpOIIMgjygOGa4JPpeIzM7Hihw93/ub0ZmNt3MFprZwtLS0tav9AAamsf6dEvnWycd1eafLyISpNBOFptZJ+BG4KcHmtbdZ7h7obsX5ubmBl9cE8+8u54lJeX87PNqHhORjifIIFgHFMQM50ffa5AJjAReMrOPgAnAzHg7YazmMRHp6IIMggXAUDMbZGapwDRgZsNIdy93957uPtDdBwJvAFPcfWGANR2yB1/b8+SxJDWPiUgHFFgQuHstcBkwG1gGPO7uS83sajObEtTntqatO6u5fd4qThvWi4lqHhORDio5yJm7+yxgVpP3ftXCtKcEWcvhuPWFleysquVKPXlMRDowdRa3YE3pDv72xlqmje/P0N5qHhORjktB0II//KuheWxo2KWIiARKQdCMtz7cyuylG/nuZwfTKzM97HJERAKlIGiivt75/T8/oHe3NL59sprHRKTjUxA08ex7GyLNY3rymIgkCAVBjMqaOv7w3HKG9+3Gl4/PD7scEZE2oSCI8ZfXo81jZ6l5TEQSh4IgatvOam57cRWnHpPLSUPVPCYiiUNBEHVLQ/PYWcPDLkVEpE0pCIAPN+/kb2+sZeqn+nO0msdEJMEoCIg8eSw1uROXT1LzmIgknoQPggUfbeVfSz9R85iIJKyEDgJ353f/XBZtHhsUdjkiIqFI6CB49t0NLCku46efO4aM1EBvxCoiErcSNgiqauv4w7+WM6xPJueqeUxEEljCBsFfXltLyTY9eUxEJCGDINI8tpJTjsnl5KG5YZcjIhKqhAyCW19cyY6qWq48U81jIiIJFwQfbt7JX19fy9RPFXBMHzWPiYgEGgRmNtnMVpjZKjO7opnx3zWz98xssZm9amYjgqwH4I//ijaPnXF00B8lItIuBBYEZpYE3AGcCYwAzm9mRf+wu49y9zHAH4Ebg6oHYOFHW3nu/U/4f58ZTK9uah4TEYFg9wjGA6vcfY27VwOPAufETuDu22MGuwAeVDENzWO9MtP4zmfUPCYi0iDILqo8oDhmuAQ4oelEZnYp8BMgFTituRmZ2XRgOkD//v0Pq5h/vreBxcVl/PHc49Q8JiISI/STxe5+h7sPBv4LuKqFaWa4e6G7F+bmHt7lnl3Skpk0ojfnjlPzmIhIrCA3jdcBBTHD+dH3WvIocFdQxZx6TC9OPaZXULMXEWm3gtwjWAAMNbNBZpYKTANmxk5gZrH3fT4bWBlgPSIi0ozA9gjcvdbMLgNmA0nA/e6+1MyuBha6+0zgMjM7A6gBtgFfD6oeERFpXqBnTd19FjCryXu/inn9oyA/X0REDiz0k8UiIhIuBYGISIJTEIiIJDgFgYhIglMQiIgkOHMP7PY+gTCzUmDtYf56T2BzK5bTWlTXoVFdhy5ea1Ndh+ZI6hrg7s3emqHdBcGRMLOF7l4Ydh1Nqa5Do7oOXbzWproOTVB16dCQiEiCUxCIiCS4RAuCGWEX0ALVdWhU16GL19pU16EJpK6EOkcgIiL7SrQ9AhERaUJBICKS4DpkEJjZZDNbYWarzOyKZsanmdlj0fFvmtnAOKnrEjMrNbPF0Z9vt1Fd95vZJjN7v4XxZma3Rut+18yOj5O6TjGz8pjl9avmpmvlmgrMbJ6ZfWBmS81snzvohrG8DrKuMJZXupm9ZWZLonX9tplp2vz7eJB1hfJ9jH52kpm9Y2bPNjOu9ZeXu3eoHyLPPlgNHEXkOchLgBFNpvk+cHf09TTgsTip6xLg9hCW2WeA44H3Wxh/FvAcYMAE4M04qesU4Nk2XlZ9geOjrzOBomb+O7b58jrIusJYXgZ0jb5OAd4EJjSZJozv48HUFcr3MfrZPwEebu6/VxDLqyPuEYwHVrn7GnevJvIIzHOaTHMO8GD09RPA6WZmcVBXKNx9PrB1P5OcA/zFI94Ass2sbxzU1ebcfYO7vx19XQEsA/KaTNbmy+sg62pz0WWwIzqYEv1peoVKm38fD7KuUJhZPpEnNv65hUlafXl1xCDIA4pjhkvY9wvROI271wLlQE4c1AVwbvRwwhNmVtDM+DAcbO1h+HR09/45Mzu2LT84uks+lsjWZKxQl9d+6oIQllf0MMdiYBMwx91bXF5t+H08mLognO/jzcB/AvUtjG/15dURg6A9ewYY6O7HAXPYk/rSvLeJ3D9lNHAb8HRbfbCZdQWeBH7s7tvb6nMP5AB1hbK83L3O3ccA+cB4MxvZFp97IAdRV5t/H83sC8Amd18U9GfF6ohBsA6ITe786HvNTmNmyUAWsCXsutx9i7tXRQf/DIwLuKaDdTDLtM25+/aG3XuPPBY1xcx6Bv25ZpZCZGX7kLv/o5lJQlleB6orrOUV8/llwDxgcpNRYXwfD1hXSN/HicAUM/uIyOHj08zsb02mafXl1RGDYAEw1MwGmVkqkZMpM5tMMxP4evT1V4AXPXrmJcy6mhxHnkLkOG88mAl8LXo1zASg3N03hF2UmfVpODZqZuOJ/P8c6Aok+nn3Acvc/cYWJmvz5XUwdYW0vHLNLDv6ujMwCVjeZLI2/z4eTF1hfB/d/Up3z3f3gUTWES+6+0VNJmv15RXow+vD4O61ZnYZMJvIlTr3u/tSM7saWOjuM4l8Yf5qZquInIycFid1/dDMpgC10bouCbouADN7hMgVJT3NrAT4NZGTZ7j73cAsIlfCrAJ2Ad+Ik7q+AnzPzGqB3cC0Ngj0icDFwHvR48sA/w30j6krjOV1MHWFsbz6Ag+aWRKR4Hnc3Z8N+/t4kHWF8n1sTtDLS7eYEBFJcB3x0JCIiBwCBYGISIJTEIiIJDgFgYhIglMQiIgkOAWBSBuyyB1A97mjpEiYFAQiIglOQSDSDDO7KHq/+sVmdk/0BmU7zOym6P3rXzCz3Oi0Y8zsjejNyZ4ys+7R94eY2dzoTd7eNrPB0dl3jd7EbLmZPRT0nTZFDkRBINKEmQ0HpgITozclqwMuBLoQ6e48FniZSKczwF+A/4renOy9mPcfAu6I3uTtRKDhNhNjgR8DI4g8n2JiwH+SyH51uFtMiLSC04ncYGxBdGO9M5FbFdcDj0Wn+RvwDzPLArLd/eXo+w8CfzezTCDP3Z8CcPdKgOj83nL3kujwYmAg8Grgf5VICxQEIvsy4EF3v3KvN81+2WS6w70/S1XM6zr0PZSQ6dCQyL5eAL5iZr0AzKyHmQ0g8n35SnSaC4BX3b0c2GZmJ0ffvxh4OfqUsBIz+1J0HmlmltGWf4TIwdKWiEgT7v6BmV0FPG9mnYAa4FJgJ5EHmFxF5FDR1OivfB24O7qiX8Oeu41eDNwTvXNkDXBeG/4ZIgdNdx8VOUhmtsPdu4Zdh0hr06EhEZEEpz0CEZEEpz0CEZEEpyAQEUlwCgIRkQSnIBARSXAKAhGRBPf/AYDRWgNuE8XlAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(model.history.history.keys())\n",
    "# summarize history for accuracy\n",
    "plt.plot(model.history.history['rec@prec'])\n",
    "plt.plot(model.history.history['val_rec@prec'])\n",
    "plt.title('model f1')\n",
    "plt.ylabel('f1')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'val'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['loss', 'f1', 'rec@prec', 'val_loss', 'val_f1', 'val_rec@prec'])\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAtUklEQVR4nO3deXxU9b3/8dcnGyEkYUkCiQkhLAGCKxpxxWrViorSVito7a2/Lt7b1la73WtvW9va5Xrv7e1ub2tbf11+4lK7oWKp1n1DglURwhKQJaxJICQhZP/8/pgBQgxLICdnJvN+Ph7zyJw538x8cmDO+8w53/l+zd0REZHElRR2ASIiEi4FgYhIglMQiIgkOAWBiEiCUxCIiCQ4BYGISIJTEIgcJzP7tZl96yjbrjezSw6xbqiZPWJmu83s9/1bpcihKQhEYse1wBggx90/YGYFZrbAzLaYmZtZScj1ySClIBCJHeOA1e7eEV3uAv4KXBNeSZIIFASSEKKnZL5oZm+a2R4z+5WZjTGzx82s0cyeNLOR3dpfbWbLzazezJ4xs7Ju66ab2WvR33sQSO/xWrPN7PXo775kZqccRX3fAO4A5ppZk5l91N23u/tPgSX9tyVE3klBIInkGuBSYDJwFfA48O9AHpH3wmcAzGwycD9wW3TdQuARM0szszTgz8DvgFHA7+l2xG5m04F7gX8GcoCfAwvMbMjhCnP3rwHfAR5090x3/1W//MUiR0FBIInkx9Gj7M3A88Bid/+Hu7cAfwKmR9vNBR5z9yfcvR34LjAUOBc4G0gFfuDu7e7+MAcfsd8M/NzdF7t7p7v/BmiN/p5ITEoJuwCRAbS92/29vSxnRu+fAGzYt8Ldu8xsE1AIdAKb/eDRGjd0uz8O+LCZfbrbY2nR5xSJSQoCkXfaApy8b8HMDBgLbAYcKDQz6xYGxcDa6P1NwLfd/dsDWK/IcdGpIZF3egi40swuNrNU4PNETu+8BLwMdACfMbNUM3s/MKPb7/4C+BczO8sihpnZlWaWdSyFmFk6sO/6wpDoski/UhCI9ODuq4AbgR8DtUQuLF/l7m3u3ga8H7gJ2EnkesIfu/1uBfBx4CfALqAq2vZY7QWaovdXRpdF+pVpYhoRkcSmTwQiIglOQSAikuAUBCIiCU5BICKS4AL7HoGZ3QvMBna4+0m9rDfgh8AVQDNwk7u/dqTnzc3N9ZKSkn6uVkRkcFu6dGmtu+f1ti7IL5T9mkgXut8eYv3lQGn0dhbwv9Gfh1VSUkJFRUU/lSgikhjMbMOh1gV2asjdnyPSz/pQ5gC/9YhXgBFmVhBUPSIi0rswrxEUEvk6/j7V0cfewcxuNrMKM6uoqakZkOJERBJFXFwsdvd73L3c3cvz8no9xSUiIscozEHnNhMZyGufouhjfdbe3k51dTUtLS39UlisSk9Pp6ioiNTU1LBLEZFBJMwgWADcYmYPELlIvNvdtx7LE1VXV5OVlUVJSQmRzkiDj7tTV1dHdXU148ePD7scERlEguw+ej9wIZBrZtXA14hM6IG7/4zIrE9XEBmUqxn4P8f6Wi0tLYM6BADMjJycHHSNRET6W2BB4O7XH2G9A5/qr9cbzCGwTyL8jSIy8DQxjYhIjOrqcqp37aVyWwMrtzZycdloTioc3u+voyDoB/X19cyfP59PfvKTffq9K664gvnz5zNixIhgChORuNHU2sGqbQ1Ubm2kcmsDK7c1smpbI02tHQCYQU5mmoIgVtXX1/PTn/70HUHQ0dFBSsqhN/HChQuDLk1EYkxXl7NxZzMre+z0N+5s3t8mKz2Fsvxsrjm9kKkF2UzNz2JKfhYZacHsshUE/eD2229n7dq1nHbaaaSmppKens7IkSNZuXIlq1ev5r3vfS+bNm2ipaWFW2+9lZtvvhk4MFxGU1MTl19+Oeeffz4vvfQShYWF/OUvf2Ho0KEh/2UicjwaWtpZta2RlVsbWLG1kZXbGli1rZHmtk4AkgxKcodxctFwrisvYmp+NmUnZHPC8PQBvSY46ILgG48sZ8WWhn59zmknZPO1q0485Pq77rqLt956i9dff51nnnmGK6+8krfeemt/N897772XUaNGsXfvXs4880yuueYacnJyDnqONWvWcP/99/OLX/yC6667jj/84Q/ceOON/fp3iEgwOrucDXV7WLktcoRfGd3pV+86MLPo8KGpTM3P4rrysZQVZDE1P5vJY7IYmpYcYuURgy4IYsGMGTMO6uv/ox/9iD/96U8AbNq0iTVr1rwjCMaPH89pp50GwBlnnMH69esHqlwR6YPdze2s3NZwYKe/rZHV2xrZ237gKH9CXibTi0dy/YxiygqyKCvIJj97YI/y+2LQBcHhjtwHyrBhw/bff+aZZ3jyySd5+eWXycjI4MILL+z1G9BDhgzZfz85OZm9ezVHuUiYOruct2v3RM/lR3rtrNzWyOb6A+/NkRmplBVkc/2MYqYWZDGtIJtJozNJTw3/KL8vBl0QhCErK4vGxsZe1+3evZuRI0eSkZHBypUreeWVVwa4OhE5kvrmtm4Xbg/02Gnt6AIgOcmYmDeM8pKR3Jg/bv9Of3TWkJg9yu8LBUE/yMnJ4bzzzuOkk05i6NChjBkzZv+6WbNm8bOf/YyysjKmTJnC2WefHWKlIomto7OLt2v3UBk9rbMyej5/W8OBT+k5w9IoK8jmQ2ePY2pBNmUFWUwancmQlPg6yu8Li3zBN36Ul5d7z4lpKisrKSsrC6migZVIf6vI8di5py3aWydyhL9yWwOrtzfRFj3KT0kyJo3OpCzaPbOsIJupBVnkZQ6Oo/yezGypu5f3tk6fCEQkrrV3drGuJnIuf0X0XH7l1gZ2NLbub5ObOYSygixuOrdk/05/Yl4maSlxMRJ/4BQEIhI3apta9+/o9w27ULWjibbOyFF+WnISk0Zncn5pLmX52ZQVZDMlP4u8rCFHeObEpiAQkZjT1tHF2pqmg759W7m1kdqmA0f5o7OGUFaQzczJB3b6E/KGkZqso/y+UhCISGjcnZpuR/n7+uavrWmivTNy/TItJYnJYzK5cEoeU/MjvXWm5GeRk6mj/P6iIBCRAdPV5Szf0sDzVTW8vLaOFVsaqNvTtn99wfB0puZncdHU0ZQVZFOWn8X43GGk6Cg/UAoCEQnUlvq9vLCmlufW1PDS2jp2Rnf8U/OzuLhsdGR8nWjPnZHD0kKuNjEpCEKQmZlJU1NT2GWIBKKptYPF6+p4fk0tz6+pYW3NHgDysoZw4ZQ8Zpbmct6kXEZnpYdcqeyjIBCR49LZ5bxZXc8La2p5vqqW1zbsoqPLSU9N4qzxOVw/o5iZpXlMHpM5KPvnDwYKgn5w++23M3bsWD71qcjMm1//+tdJSUnh6aefZteuXbS3t/Otb32LOXPmhFypSP/YtLN5/xH/S2vr2L23HYCTCrP5+AUTmDkpl9PHjYy7MXcS1eALgsdvh23L+vc580+Gy+865Oq5c+dy22237Q+Chx56iEWLFvGZz3yG7OxsamtrOfvss7n66qt1RCRxqaGlnZfX1vH8mhpeWFPL+rrIJCoFw9O57MQxnF+ax3kTc9STJ04NviAIwfTp09mxYwdbtmyhpqaGkSNHkp+fz2c/+1mee+45kpKS2Lx5M9u3byc/Pz/sckWOqKOzizeq63ludS0vVNXy+qZ6OrucjLRkzpmQw4fPLWFmaR4T84bp4GYQGHxBcJgj9yB94AMf4OGHH2bbtm3MnTuX++67j5qaGpYuXUpqaiolJSW9Dj8tEgvcnQ11zTy/pobn19Ty8to6Gls7MINTikbwiXdNZGZpLtOLR2pYhkFo8AVBSObOncvHP/5xamtrefbZZ3nooYcYPXo0qampPP3002zYsCHsEkUOUt/cxktrD/Tu2TebVuGIocw+tYCZpXmcOzGHERnq0jnYKQj6yYknnkhjYyOFhYUUFBTwwQ9+kKuuuoqTTz6Z8vJypk6dGnaJkuDaOrr4x8ZdvFBVy3NrallWXU+XQ9aQFM6emMM/XzCB80vzKMnJ0OmeBBNoEJjZLOCHQDLwS3e/q8f6ccC9QB6wE7jR3auDrClIy5YduEidm5vLyy+/3Gs7fYdABoK7s7Zmz/4LvK+sq2NPWyfJScapRcP59LtLmVmay6ljR2h8ngQXWBCYWTJwN3ApUA0sMbMF7r6iW7PvAr9199+Y2buB/wA+FFRNIoPdzj1tvFBVywvRc/1bd0euS43LyeB9pxdy/qQ8zpmYw/ChqSFXKrEkyE8EM4Aqd18HYGYPAHOA7kEwDfhc9P7TwJ8DrEdk0Gnt6GTp+l08XxU5z798SwPukJ2ewnmTcrnl3bnMnJRHcU5G2KVKDAsyCAqBTd2Wq4GzerR5A3g/kdNH7wOyzCzH3eu6NzKzm4GbAYqLi3t9MXcf9Oc14202Oel/7s7q7U37e/e8+vZO9rZ3kpJknF48ks9eMpmZpbmcUjSC5KTB/X6Q/hP2xeIvAD8xs5uA54DNQGfPRu5+D3APRKaq7Lk+PT2duro6cnJyBm0YuDt1dXWkp2t8lkRT09jKi1WRQdteWFO7f+atCXnDuK68iJmleZw9MYfMIWG/nSVeBfk/ZzMwtttyUfSx/dx9C5FPBJhZJnCNu9f39YWKioqorq6mpqbm2KuNA+np6RQVFYVdhgSspb2TJet3Rrt11lK5tQGAkRmpnDcpl5mluZxfmkfhiKEhVyqDRZBBsAQoNbPxRAJgHnBD9wZmlgvsdPcu4EtEehD1WWpqKuPHjz/OckXC0dXlrNzWGOndU1XL4rd30tbRRWqyUT5uFF+8bAoXlOZx4gnZJOl0jwQgsCBw9w4zuwVYRKT76L3uvtzM7gQq3H0BcCHwH2bmRE4NfSqoekRiyfaGlv1f5HqxqpbapsgY/ZPHZHLjWeOYOTmXs8aPIiNNp3skeBZvFyDLy8u9oqIi7DJE+qS5rYPFb+/k+dW1vFBVw+rtke+S5GamRU/35HH+pFzyh+sakATDzJa6e3lv63S4IRKAfVMy7rvAu3TDLto6u0hLSWJGySiuOb2I80sjk67rdI+ETUEg0k+21O/d363zxapadjVHxuifmp/FTeeVcP6kXGaMH6Ux+iXmKAhEjkNbRxeLlm/jvsUbeGXdTiAyJeNFU0drSkaJGwoCkWOwsa6Z+a9u5PcVm6jb00bRyKF84T2TuXRavqZklLijIBA5Su2dXfy9cgf3Ld7A82tqSTK4uGwMHzyrmAtK83SuX+KWgkDkCDbX7+XBVzfywJJN7GhsJT87ndsuKWXumWMpGK4vdUn8UxCI9KKzy3l29Q7ue2UjT6/agQPvmpzHt88ax0VT8kjRsM0yiCgIRLrZ0dDCg0s28cCSTWyu30tu5hA+ceFE5p1ZzNhRGsFTBicFgSS8ri7nxbW1zF+8kSdWbKejyzlvUg5fvrKMS8rGaI5eGfQUBJKw6ppa+f3Sau5/dSMb6poZmZHKR84fz/UzihmfOyzs8kQGjIJAEoq7s/jtncxfvJG/vrWNts4uZpSM4nOXTuayE/P1ZS9JSAoCSQj1zW384bXNzF+8gbU1e8hKT+GGs4r54FnFlI7JCrs8kVApCGTQcnde21jP/MUbefTNLbR2dHHa2BH897WnMPuUExiapqN/EVAQyCDU2NLOn1/fwn2vbGDltkaGpSVz7RlF3HBWMSeeMDzs8kRijoJABo1l1buZ/+oG/vL6FprbOjnxhGy+876Tufq0EzSNo8hh6N0hca25rYNH3tjCfYs38mb1btJTk7j61BO44axxnFo0XGP+iBwFBYHEpZXbGpi/eCN/em0zja0dTB6Tydevmsb7Ti9i+NDUsMsTiSsKAokbLe2dLFy2lfsWb2Tphl2kpSRx5ckF3HBWMeXjRuroX+QYKQgk5q2taWL+4o08vLSa3XvbGZ87jC9fUcY1ZxQxalha2OWJxD0FgcSknhO+pCQZl52UzwdnFHPOxBwd/Yv0IwWBxJR9E748vHQTtU2RCV++eNkUPlBepJm+RAKiIJDQdXR28WSPCV8uKRvDDZrwRWRAKAgkNPsmfHmwYhPbG1opGK4JX0TCoCCQAbVvwpf5izfy1MoDE758672a8EUkLIEGgZnNAn4IJAO/dPe7eqwvBn4DjIi2ud3dFwZZk4RDE76IxK7AgsDMkoG7gUuBamCJmS1w9xXdmn0FeMjd/9fMpgELgZKgapKBdbgJXy6dNoZUHf2LxIQgPxHMAKrcfR2AmT0AzAG6B4ED2dH7w4EtAdYjA0QTvojElyCDoBDY1G25GjirR5uvA38zs08Dw4BLensiM7sZuBmguLi43wuV46cJX0TiV9gXi68Hfu3u/2Nm5wC/M7OT3L2reyN3vwe4B6C8vNxDqFMOoeeEL9ma8EUk7gQZBJuBsd2Wi6KPdfdRYBaAu79sZulALrAjwLrkOCX0hC9dndCyG/bugpb6yM+90Z+tDZAyFNKG9XLLhNSMA/dTNDSGxI4gg2AJUGpm44kEwDzghh5tNgIXA782szIgHagJsCY5DoNqwpeOth4780Pdeqxv2U3k0tZxSkqFtIxIKPQaFtHltG7hkTYsuj6z9zapwyA57A/5Eo8C+1/j7h1mdguwiEjX0HvdfbmZ3QlUuPsC4PPAL8zss0TeXTe5u079xJiYnfDFHdqb+7Aj73a/fc+hn9eSIH04DB0ZuWWMgpyJB5aHjoT0EQcvDx0JQzKhowXamqFtD7Q1RX62Nx+43+utKdpmDzRtiz7W7Tm88+i3SUp6L2HRYzm1l08rhwuc1AxIUg+vwczibb9bXl7uFRUVYZcx6HV2OQ8v3TQwE750dUHr7kPswOsPv6Pvaj/08yanvXNnfdCOfETv64Zkx86Ozx06WnsJk6a+B057j/Dpi56fVN7xyeVIgdNL2KSkgwYPHDBmttTdy3tbp8+R0qsfP7WGHzy5hsljMvnG1Sfy3umFR57wpaOtl1MtPZd73FrqI20Od7olLTO6kx4R+Tl66qGPyrvfUofG/47GDFLTI7eMUf33vF1d0LE3GiZNhw6LnmFzUOA0QdP2g8OmY29f/rjIp6/uf2v3dWE+ftB/m5Br6f74xXfAqfPobwoCeYct9Xu559k1fGxyC1++KAvb+yasePbwO/K9u45wlGkHH4HvO91ypJ15+nBdWA1CUtKBo3Xy+u95uzoPnOY6bJhEf+4/I9HtQOCgsxT99TiHeLy39kHXchyPZxcSBAWBHGxHJcsf+CFPJS0if+POyAAg3SWlRnbi+3bUw8fCmJO77bxH9NiZR5eHDI+d0y0SnKRkGJIVuUncUBAINNXAWw/DG/fD1je4yJPYMOpceNeNMLzo4J17akb8n24RkYMoCBJVewusfhzeeADWPAHeiRecyi+H3cwf2s7mD/8yB8LsESQiA0bv9ETiDhtfiRz5L/9zpKdO1glw7qfh1Hn8YVMW3/79G3x/7qkMUwiIJAy92xPBznXwxoPw5gOwa33k9E7Z1ZHeB+MvgKRk9rR28F+/eIbTxo5gzqnBXJASkdikIBis9u6C5X+KnPrZtBgwmPAuuPBLMHV25MtP3fz0mSp2NLby8w+doakhRRKMgmAw6WyHqicjp35WPQ6dbZA3FS75Opx8HQzv/Uh/085mfvH827xveiHTi0cObM0iEjoFQbxzhy3/iBz5v/UwNNdBRi6UfxROnQsFpx2xl89/PF5Jshn/NmvqwNQsIjFFQRCvdlfDmw9FAqB2FSQPgSmXw6nXw6SLIfkI3wKOemVdHQuXbeNzl04mf3h6wEWLSCxSEMST1kaofCRy6uft5wGH4nNg9g/gxPdG+vn3QWeXc+cjKygcMZSbL5gQRMUiEgcUBLGuqxPefjZy5F/5SOTr+yNL4MLb4ZTrYNSx78AfqtjEiq0N/Pj66ZpBTCSBKQhi1fYVkSP/Zb+Hxq2RMXdOmRvp8jn2rOP+dm9DSzvfXbSKM0tGMvuUgn4qWkTikYIgljTtgGXRoR62vQlJKTDpUph1F0yeFRmBsp/85Kkqdja38evZM/p3SGkRiTsKgrC174VVCyNf+Kp6MjIJyQnTYdZ/wknXQGY/jgwZ9XbtHv7vi29z7elFnFwUZzOLiUi/UxCEoasLNnUf6qEhMrzseZ+BU+ZFxtsP0LcfqyQtOYkvzpoS6OuISHxQEAykurWRi75vPgj1GyIzOE2LDvVQMjMyhG/AXlhTy5OV2/nXWVMYnaXuoiKiIAhe884DQz1Uv0pkqIcL4aIvQ9ns6MQgA6Ojs4s7H11O8agMPnLe+AF7XRGJbQqCIHS0QdUTkZ3/6r9Gh3oog0u+EenymX1CKGXd/+pGVm9v4mc3nqHuoiKyn4Kgv7jDltciO/9lD8PenTAsD878WOTUT/4poU7osru5ne89sZpzJuRw2YljQqtDRGKPguB41W+CZfuGelgdGeph6hWRoR4mvvuoh3oI2g/+vprde9u546pp6i4qIgdREByL1kZYsSDS62f9C0SGejgXrroFps2JTO0YQ6p2NPG7lzcwb0YxZQXZYZcjIjFGQXC0ujph3dOR/v6Vj0DHXhg5PjK+/ynXwajYvfj6rcdWMDQtmc9fOjnsUkQkBgUaBGY2C/ghkAz80t3v6rH++8BF0cUMYLS7jwiypj7bvjxy5P/m76FpW2Soh9Ouj/T3Hzsj5idyf3rlDp5ZVcNXriwjJ3NI2OWISAwKLAjMLBm4G7gUqAaWmNkCd1+xr427f7Zb+08D04Oqp08at0fG9n/jfti2LDLUQ+l7Ihd9Sy/r16EegtTe2cU3H1vBhNxh/NM5JWGXIyIxKshPBDOAKndfB2BmDwBzgBWHaH898LUA6zm89r2w8rHIRd+1T0WHejgdLv9vOOn9MCw3tNKO1W9f3sC6mj386sPlpKUkhV2OiMSoIIOgENjUbbkaOKu3hmY2DhgPPHWI9TcDNwMUFxf3X4VdXbDxpcjOf8Vfug31cGvk6D8vfodg2LmnjR8+uZqZpbm8e+rosMsRkRgWKxeL5wEPu3tnbyvd/R7gHoDy8nI/7lerrYI3H4hc+N29EdIyI719TpkbHeoh/o+ev/fEKva0dXLHbHUXFZHDCzIINgNjuy0XRR/rzTzgUwHWEh3q4Y/RoR6WgCVFhnq4+Ksw9coBHeohaCu3NTB/8UY+dPY4SsdkhV2OiMS4IINgCVBqZuOJBMA84IaejcxsKjASeDnAWuDVe+CZ/4DR0+DSO+HkD4Q21EOQ3J1vPrqCrPRUbrtE3UVF5MgCCwJ37zCzW4BFRLqP3uvuy83sTqDC3RdEm84DHnD34z/lczhn3ARTroD8k2O+y+fxeGLFdl6squPrV01j5LC0sMsRkThgx7L/NbNMd28KoJ4jKi8v94qKijBeOua1dnTynu8/R1pyEgtvnUlqcvxf6xCR/mFmS929vLd1x7qnOFQXUAnRr19cz4a6Zr46e5pCQESO2iFPDZnZ5w61CsgMphw5VjWNrfz4qSounjqaCyb3//SWIjJ4He6w8TtELuJm9bhlHuH3JAT/87dVtHZ08uUry8IuRUTizOEuFr8G/Nndl/ZcYWYfC64k6au3Nu/mwYpNfPS88UzI04c1Eembwx3ZbwY2mNmtvazr9YKDDDx3585HVjAyI41PX1wadjkiEocOFwTTgDTgI2Y20sxG7bsB7QNTnhzJwmXbeHX9Tj7/nskMHxobk+CISHw53KmhnwN/ByYAS4lcJN7Ho49LiFraO/nOwkqm5mcx78x+HINJRBLKIT8RuPuP3L2MyBfBJrj7+G43hUAM+OXz69hcv5c7rppGctLg/ZKciATriL1/3P0TA1GI9M32hhZ++sxaLjtxDOdOjL8hskUkdqgbaJz6z7+upKPT+fIV08IuRUTinIIgDr2+qZ4/vraZj5w/nuKcjLDLEZE4pyCIM5HuosvJzRzCLe+eFHY5IjIIKAjizII3tvDaxnr+ddYUMofEyrxCIhLPFARxpLmtg7seX8nJhcO59vSisMsRkUFCQRBHfv7sOrbubuGOq6aRpO6iItJPFARxYnP9Xn7+3Fpmn1LAmSWjwi5HRAYRBUGcuOvxlbjDl67Q6KIi0r8UBHGgYv1OHnljC/98wQQKRwwNuxwRGWQUBDGuq8v5xiMryM9O518unBh2OSIyCCkIYtwfXqtm2ebd/NvlU8hIU3dREel/CoIY1tTawX8tWsVpY0cw59TCsMsRkUFKQRDDfvp0FTWNrXxN3UVFJEAKghi1aWczv3zhbd43vZDpxSPDLkdEBjEFQYz6zsJKks34t1lTwy5FRAa5QIPAzGaZ2SozqzKz2w/R5jozW2Fmy81sfpD1xItX1tXx+Fvb+MSFE8kfnh52OSIyyAXWDcXMkoG7gUuBamCJmS1w9xXd2pQCXwLOc/ddZjY6qHriRWe0u2jhiKHcfIEmghOR4AX5iWAGUOXu69y9DXgAmNOjzceBu919F4C77wiwnrjwUMUmKrc28KUrppKemhx2OSKSAIIMgkJgU7fl6uhj3U0GJpvZi2b2ipnN6u2JzOxmM6sws4qampqAyg1fQ0s73120ihklo7jy5IKwyxGRBBH2xeIUoBS4ELge+IWZjejZyN3vcfdydy/Py8sb2AoH0I//voadzW3ccdU0zNRdVEQGRpBBsBkY2225KPpYd9XAAndvd/e3gdVEgiHhvF27h1+/tJ4PnFHESYXDwy5HRBJIkEGwBCg1s/FmlgbMAxb0aPNnIp8GMLNcIqeK1gVYU8z69mMrGJKSzBcumxJ2KSKSYAILAnfvAG4BFgGVwEPuvtzM7jSzq6PNFgF1ZrYCeBr4orvXBVVTrHp+TQ1PVu7gUxdNYnSWuouKyMAydw+7hj4pLy/3ioqKsMvoNx2dXVzxo+dpae/iic9dwJAU9RQSkf5nZkvdvby3dWFfLE5481/dyOrtTfz7FWUKAREJhYIgRPXNbXzvidWcMyGHy04cE3Y5IpKgFAQh+sGTa2jY267uoiISKgVBSKp2NPK7VzYwb0YxZQXZYZcjIglMQRCSbz5aSUZaMp+/dHLYpYhIglMQhODplTt4dnUNt15cSk7mkLDLEZEEpyAYYO2dXXzzsRVMyB3GP51TEnY5IiIKgoH225c3sK5mD1+ZXUZaija/iIRPe6IBVNfUyg+eXM0Fk/O4aErCT70gIjFCQTCAvvfEaprbOvnqlWXqLioiMUNBMEAqtzZw/6sb+dDZ4ygdkxV2OSIi+ykIBoC7881HV5A9NJXbLknIUbZFJIYpCAbA31Zs56W1dXz2ksmMyEgLuxwRkYMoCALW2tHJdxZWUjo6kw+eVRx2OSIi76AgCNj/fXE9G+qa+ersaaQka3OLSOzRnilANY2t/OSpKi6eOpoLJg/euZZFJL4pCAL03UWraO3o5MtXloVdiojIISkIAvLW5t08tHQTHz6nhAl5mWGXIyJySAqCALg7dz6ygpEZaXz6YnUXFZHYpiAIwMJl23h1/U6+8J4pDB+aGnY5IiKHpSDoZy3tke6iZQXZzD1zbNjliIgckYKgn/3iuXVsrt/LHbOnkZyk8YREJPYpCPrRtt0t/PSZtcw6MZ9zJuaEXY6IyFFREPSj//rrSjq7nH+/Qt1FRSR+BBoEZjbLzFaZWZWZ3d7L+pvMrMbMXo/ePhZkPUH6x8Zd/PEfm/nozPEU52SEXY6IyFFLCeqJzSwZuBu4FKgGlpjZAndf0aPpg+5+S1B1DAR3585HV5CXNYRPXTQp7HJERPokyE8EM4Aqd1/n7m3AA8CcAF8vNH95fQv/2FjPFy+bQuaQwLJVRCQQQQZBIbCp23J19LGerjGzN83sYTPrtb+lmd1sZhVmVlFTUxNErcesua2Dux5fycmFw7n29KKwyxER6bOwLxY/ApS4+ynAE8Bvemvk7ve4e7m7l+flxdbgbT97dh3bGlq446ppJKm7qIjEoSCDYDPQ/Qi/KPrYfu5e5+6t0cVfAmcEWE+/21y/l58/u5bZpxRwZsmosMsRETkmQQbBEqDUzMabWRowD1jQvYGZFXRbvBqoDLCefnfX4ysB+JK6i4pIHAvsyqa7d5jZLcAiIBm4192Xm9mdQIW7LwA+Y2ZXAx3ATuCmoOrpb0vW7+SRN7bwmXdPonDE0LDLERE5ZubuYdfQJ+Xl5V5RURFqDV1dzpy7X6SmsZWnvvAuMtLUU0hEYpuZLXX38t7WhX2xOC49/Fo1yzbv5vbLpyoERCTuKQj6qKm1g/9etIrpxSOYc9oJYZcjInLcFAR9dPfTVdQ0tvK1q07ETN1FRST+KQj6YGNdM796/m3eP72Q08aOCLscEZF+oSDog+8srCQ5yfjXWVPDLkVEpN8oCI7Sy2vr+OvybXzywonkD08PuxwRkX6jIDgKnV2R0UULRwzl4xdMCLscEZF+pSA4Cg8u2UTl1ga+dMVU0lOTwy5HRKRfKQiOoKGlnf/52ypmlIziypMLjvwLIiJxRt+GOoIf/30NO5vb+M1V09RdVEQGJX0iOIy3a/fw65fW84EzijipcHjY5YiIBEJBcBjffmwFQ1KS+cJlU8IuRUQkMAqCQ3hudQ1PVu7gUxdNYnSWuouKyOClIOhFR2cX33x0BeNyMvjI+SVhlyMiEigFQS/uW7yRNTua+PcryhiSou6iIjK4KQh6qG9u4/tPrubciTm8Z9qYsMsREQmcgqCHHzy5hoa97dyh7qIikiAUBN2s2d7I717ZwPUzipmanx12OSIiA0JBEOXufPOxSjLSkvncpZPDLkdEZMAoCKKeXrWD51bXcOvFpeRkDgm7HBGRAaMgANo6uvjWo5VMyB3GP51TEnY5IiIDSkEA/Pbl9ayr3cNXZpeRlqJNIiKJJeH3enVNrfzw72u4YHIeF00ZHXY5IiIDLtAgMLNZZrbKzKrM7PbDtLvGzNzMyoOspzffe2I1zW2dfPXKMnUXFZGEFFgQmFkycDdwOTANuN7MpvXSLgu4FVgcVC2HUrm1gftf3ciHzh5H6ZisgX55EZGYEOQnghlAlbuvc/c24AFgTi/tvgn8J9ASYC3v4O7c+cgKsoemctslpQP50iIiMSXIICgENnVbro4+tp+ZnQ6MdffHAqyjV4uWb+fldXV87tLJjMhIG+iXFxGJGaFdLDazJOB7wOePou3NZlZhZhU1NTXH/dqtHZ18Z2Elk8dkcsOM4uN+PhGReBZkEGwGxnZbLoo+tk8WcBLwjJmtB84GFvR2wdjd73H3cncvz8vLO+7C7n1hPRt3NvPV2dNISU74jlMikuCC3AsuAUrNbLyZpQHzgAX7Vrr7bnfPdfcSdy8BXgGudveKAGtiR2MLP3lqDZeUjWZm6fGHiohIvAssCNy9A7gFWARUAg+5+3Izu9PMrg7qdY/ku4tW0dbZxZevfEcHJhGRhJQS5JO7+0JgYY/H7jhE2wuDrAXgrc27+f3Saj52/njG5w4L+uVEROJCwpwgd3e+8chyRmWk8emL1V1URGSfhAmCx5ZtZcn6XXz+PVPITk8NuxwRkZiRMEEwbEgKl04bw9wzxx65sYhIAgn0GkEsuWjKaA0qJyLSi4T5RCAiIr1TEIiIJDgFgYhIglMQiIgkOAWBiEiCUxCIiCQ4BYGISIJTEIiIJDhz97Br6BMzqwE2HOOv5wK1/VhOf1FdfaO6+i5Wa1NdfXM8dY1z917H3o+7IDgeZlbh7u+Y+CZsqqtvVFffxWptqqtvgqpLp4ZERBKcgkBEJMElWhDcE3YBh6C6+kZ19V2s1qa6+iaQuhLqGoGIiLxTon0iEBGRHhQEIiIJblAGgZnNMrNVZlZlZrf3sn6ImT0YXb/YzEpipK6bzKzGzF6P3j42QHXda2Y7zOytQ6w3M/tRtO43zez0GKnrQjPb3W173TEANY01s6fNbIWZLTezW3tpM+Db6yjrCmN7pZvZq2b2RrSub/TSZsDfj0dZVyjvx+hrJ5vZP8zs0V7W9f/2cvdBdQOSgbXABCANeAOY1qPNJ4GfRe/PAx6MkbpuAn4Swja7ADgdeOsQ668AHgcMOBtYHCN1XQg8OsDbqgA4PXo/C1jdy7/jgG+vo6wrjO1lQGb0fiqwGDi7R5sw3o9HU1co78foa38OmN/bv1cQ22swfiKYAVS5+zp3bwMeAOb0aDMH+E30/sPAxWZmMVBXKNz9OWDnYZrMAX7rEa8AI8ysIAbqGnDuvtXdX4vebwQqgcIezQZ8ex1lXQMuug2aooup0VvPHioD/n48yrpCYWZFwJXALw/RpN+312AMgkJgU7flat75htjfxt07gN1ATgzUBXBN9HTCw2Y2NuCajtbR1h6Gc6If7x83sxMH8oWjH8mnEzma7C7U7XWYuiCE7RU9zfE6sAN4wt0Pub0G8P14NHVBOO/HHwD/CnQdYn2/b6/BGATx7BGgxN1PAZ7gQOpL714jMn7KqcCPgT8P1AubWSbwB+A2d28YqNc9kiPUFcr2cvdOdz8NKAJmmNlJA/G6R3IUdQ34+9HMZgM73H1p0K/V3WAMgs1A9+Quij7WaxszSwGGA3Vh1+Xude7eGl38JXBGwDUdraPZpgPO3Rv2fbx394VAqpnlBv26ZpZKZGd7n7v/sZcmoWyvI9UV1vbq9vr1wNPArB6rwng/HrGukN6P5wFXm9l6IqeP321m/69Hm37fXoMxCJYApWY23szSiFxMWdCjzQLgw9H71wJPefTKS5h19TiPfDWR87yxYAHwT9HeMGcDu919a9hFmVn+vnOjZjaDyP/nQHcg0df7FVDp7t87RLMB315HU1dI2yvPzEZE7w8FLgVW9mg24O/Ho6krjPeju3/J3YvcvYTIPuIpd7+xR7N+314px/PLscjdO8zsFmARkZ4697r7cjO7E6hw9wVE3jC/M7MqIhcj58VIXZ8xs6uBjmhdNwVdF4CZ3U+kR0mumVUDXyNy8Qx3/xmwkEhPmCqgGfg/MVLXtcAnzKwD2AvMG4BAPw/4ELAsen4Z4N+B4m51hbG9jqauMLZXAfAbM0smEjwPufujYb8fj7KuUN6PvQl6e2mICRGRBDcYTw2JiEgfKAhERBKcgkBEJMEpCEREEpyCQEQkwSkIRAaQRUYAfceIkiJhUhCIiCQ4BYFIL8zsxuh49a+b2c+jA5Q1mdn3o+PX/93M8qJtTzOzV6KDk/3JzEZGH59kZk9GB3l7zcwmRp8+MzqI2Uozuy/okTZFjkRBINKDmZUBc4HzooOSdQIfBIYR+XbnicCzRL7pDPBb4N+ig5Mt6/b4fcDd0UHezgX2DTMxHbgNmEZkforzAv6TRA5r0A0xIdIPLiYywNiS6MH6UCJDFXcBD0bb/D/gj2Y2HBjh7s9GH/8N8HszywIK3f1PAO7eAhB9vlfdvTq6/DpQArwQ+F8lcggKApF3MuA37v6lgx40+2qPdsc6Pktrt/ud6H0oIdOpIZF3+jtwrZmNBjCzUWY2jsj75dpomxuAF9x9N7DLzGZGH/8Q8Gx0lrBqM3tv9DmGmFnGQP4RIkdLRyIiPbj7CjP7CvA3M0sC2oFPAXuITGDyFSKniuZGf+XDwM+iO/p1HBht9EPAz6MjR7YDHxjAP0PkqGn0UZGjZGZN7p4Zdh0i/U2nhkREEpw+EYiIJDh9IhARSXAKAhGRBKcgEBFJcAoCEZEEpyAQEUlw/x9+DI85eJEYFgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(model.history.history.keys())\n",
    "# summarize history for accuracy\n",
    "plt.plot(model.history.history['f1'])\n",
    "plt.plot(model.history.history['val_f1'])\n",
    "plt.title('model f1')\n",
    "plt.ylabel('f1')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'val'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Сверточные слои можно накладывать друг на друга."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Давайте еще разберем параметры:  \n",
    "    **kernel_size** - это размер кусочков, к которым мы применяем сверточные матрицы (похоже на размер нграмма)   \n",
    "    **filters** - количество матриц свертки  (это как бы размерность до которой будет сжат или развернут вектор)    \n",
    "    **stride** - шаг по которому выбираются кусочки из последовательности (это как параметр шаг в функции range в питоне)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs = tf.keras.layers.Input(shape=(MAX_LEN,))\n",
    "embeddings = tf.keras.layers.Embedding(input_dim=len(word2id), output_dim=100)(inputs, )\n",
    "conv1 = tf.keras.layers.Conv1D(kernel_size=5, filters=32, strides=1)(embeddings)\n",
    "conv2 = tf.keras.layers.Conv1D(kernel_size=5, filters=32, strides=1, activation='relu')(conv1)\n",
    "concat = tf.keras.layers.Flatten()(conv2)\n",
    "dense = tf.keras.layers.Dense(64, activation='relu')(concat)\n",
    "outputs = tf.keras.layers.Dense(len(label2id), activation='softmax')(dense)\n",
    "\n",
    "model = tf.keras.Model(inputs=inputs, outputs=outputs)\n",
    "optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)\n",
    "model.compile(optimizer=optimizer,\n",
    "              loss='categorical_crossentropy',\n",
    "              metrics=[f1])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_8\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_12 (InputLayer)        [(None, 201)]             0         \n",
      "_________________________________________________________________\n",
      "embedding_10 (Embedding)     (None, 201, 100)          5530600   \n",
      "_________________________________________________________________\n",
      "conv1d_6 (Conv1D)            (None, 197, 32)           16032     \n",
      "_________________________________________________________________\n",
      "conv1d_7 (Conv1D)            (None, 193, 32)           5152      \n",
      "_________________________________________________________________\n",
      "flatten_8 (Flatten)          (None, 6176)              0         \n",
      "_________________________________________________________________\n",
      "dense_16 (Dense)             (None, 64)                395328    \n",
      "_________________________________________________________________\n",
      "dense_17 (Dense)             (None, 19)                1235      \n",
      "=================================================================\n",
      "Total params: 5,948,347\n",
      "Trainable params: 5,948,347\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "71/71 [==============================] - 14s 174ms/step - loss: 2.2801 - f1: 0.0609 - val_loss: 1.1753 - val_f1: 0.6315\n",
      "Epoch 2/5\n",
      "71/71 [==============================] - 12s 167ms/step - loss: 0.9642 - f1: 0.6986 - val_loss: 0.7664 - val_f1: 0.7607\n",
      "Epoch 3/5\n",
      "71/71 [==============================] - 12s 167ms/step - loss: 0.5140 - f1: 0.8425 - val_loss: 0.7034 - val_f1: 0.7917\n",
      "Epoch 4/5\n",
      "71/71 [==============================] - 12s 166ms/step - loss: 0.2918 - f1: 0.9170 - val_loss: 0.8021 - val_f1: 0.7897\n",
      "Epoch 5/5\n",
      "71/71 [==============================] - 12s 170ms/step - loss: 0.1633 - f1: 0.9564 - val_loss: 1.0091 - val_f1: 0.7790\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f7028d3a358>"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train, y_train, \n",
    "          validation_data=(X_valid, y_valid),\n",
    "          batch_size=2000,\n",
    "          epochs=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['loss', 'f1', 'val_loss', 'val_f1'])\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAtp0lEQVR4nO3deXyV9Zn//9dFCARCgEDCloQkKCK4gUQE6eLaolaxLgWtWmc6dWaqo3aZ72gXp9NfndplWrW1Y23LFK1WrVWLFfetVUAJiAoCgpiQhC1A2EJCtuv3x30SDiGBBHLnnJzzfj4eeXju+/6c+1znlnNf93Z9PubuiIhI8uoV6wBERCS2lAhERJKcEoGISJJTIhARSXJKBCIiSU6JQEQkySkRiBwlM/u9mf2gg21LzOzcdpb1M7OnzWynmf2pa6MUaZ8SgUj8uBwYDgx19yvMbKSZzTOzDWbmZlYQ4/gkQSkRiMSPfOBDd2+ITDcBzwGXxS4kSQZKBJIUIpdk/t3M3jOzajP7nZkNN7NnzWy3mb1kZplR7S82sxVmtsPMXjOz8VHLJpnZ0sj7HgXSWn3W58xsWeS9C8zs5A7E91/A7cAsM9tjZl92983u/itgcddtCZGDKRFIMrkMOA84DrgIeBb4FpBN8Fu4CcDMjgP+CNwSWTYfeNrM+phZH+Ap4EFgCPAnoo7YzWwSMAf4Z2Ao8Gtgnpn1PVRg7v6fwH8Dj7r7AHf/XZd8Y5EOUCKQZPKLyFF2BfB34C13f8fda4EngUmRdrOAZ9z9RXevB34K9APOAKYCqcBd7l7v7o9z4BH79cCv3f0td29097nAvsj7ROJS71gHINKNNke9rmljekDk9SigtHmBuzeZWRmQAzQCFX5gb42lUa/zgS+Z2b9FzesTWadIXFIiEDnYBuCk5gkzMyAPqAAcyDEzi0oGo4GPIq/LgDvc/Y5ujFfkqOjSkMjBHgMuNLNzzCwV+AbB5Z0FwEKgAbjJzFLN7FJgStR7fwP8i5mdboF0M7vQzDKOJBAzSwOa7y/0jUyLdCklApFW3H01cDXwC2ArwY3li9y9zt3rgEuB64DtBPcTnoh6bzHwFeCXQBWwNtL2SNUAeyKvV0WmRbqUaWAaEZHkpjMCEZEkp0QgIpLklAhERJKcEoGISJLrcXUEWVlZXlBQEOswRER6lCVLlmx19+y2lvW4RFBQUEBxcXGswxAR6VHMrLS9Zbo0JCKS5JQIRESSnBKBiEiS63H3CNpSX19PeXk5tbW1sQ4lVGlpaeTm5pKamhrrUEQkgSREIigvLycjI4OCggKCjiITj7uzbds2ysvLKSwsjHU4IpJAEuLSUG1tLUOHDk3YJABgZgwdOjThz3pEpPslRCIAEjoJNEuG7ygi3S8hLg2JiCSiXbX1lGyt5uOt1ZRs3cvZxw/jpNxBXf45SgRdYMeOHTz88MN89atf7dT7LrjgAh5++GEGDx4cTmAiEvf21jVQsnUvJduCHX6w06+mZFs1W/fUtbQzg6ED+igRxKsdO3bwq1/96qBE0NDQQO/e7W/i+fPnhx2aiMSBfQ2NrN+2N9jJH7DD38umXQfe9xuW0ZeCrHTOOX44hdnpFAxNpzArnfyh/UlLTQklPiWCLnDrrbfy0UcfMXHiRFJTU0lLSyMzM5NVq1bx4Ycfcskll1BWVkZtbS0333wz119/PbC/u4w9e/Zw/vnn84lPfIIFCxaQk5PDX/7yF/r16xfjbyYiHVXf2ER5VQ0lW6tZF3VU//HWaip21BA9BtiQ9D4UDO3PGccOZUxWOgVZwQ6/ICudAX27f7eccIngv55ewQcbdnXpOieMGsh/XnRCu8vvvPNOli9fzrJly3jttde48MILWb58ectjnnPmzGHIkCHU1NRw2mmncdlllzF06NAD1rFmzRr++Mc/8pvf/IYvfOEL/PnPf+bqq6/u0u8hIkenscnZsKOGkm3VrXb4eynbvpeGpv17+4y03hRmpXPq6EwuOzWXwsgOv3BoOoP6x1ctUMIlgngwZcqUA571v+eee3jyyScBKCsrY82aNQclgsLCQiZOnAjA5MmTKSkp6a5wRSSKu7N51742LuNUU7p9L3UNTS1t+6WmUJCVzviRGVxw0oiWyziFWekMSe/TY570S7hEcKgj9+6Snp7e8vq1117jpZdeYuHChfTv358zzzyzzVqAvn37trxOSUmhpkZjlIuExd3ZVl3X8kTO/p3+Xkq2VlNT39jStk/vXuQP6U9BVjpnHz+s5TJOYVY6wwf27TE7+0NJuEQQCxkZGezevbvNZTt37iQzM5P+/fuzatUqFi1a1M3RiSSvnXvr+figyzjVfFxZze59DS3tevcy8ob0p2Bof6aNGUphVrDjL8xKZ+SgfqT06vk7+0NRIugCQ4cOZfr06Zx44on069eP4cOHtyybMWMG9913H+PHj2fcuHFMnTo1hpGKJJ7qfQ37j+grq1t2/B9vraZqb31LOzPIGdyPwqx0Pn9qTstRfUFWOrmZ/UhNSZj62k4zj76V3QMUFRV564FpVq5cyfjx42MUUfdKpu8q0qy2vpHSyOOXzdfrP45cv6/cve+AtiMGplGQ1Z/CrAHBkX1kh583JLzHL3sCM1vi7kVtLdMZgYjEhbqGJsqq9h583b6ymo27ag94/DJrQB8KhqZz5nHZLZdwgscv+9O/j3ZrnaUtJiLdqqGxiQ827mJZ2Q7WVe7f6VfsqKEx6vHLQf1SKchK5/QxQ1t28s2Xcgamxdfjlz1dqInAzGYAdwMpwG/d/c5Wy/OBOUA2sB242t3Lw4xJRLrXzpp6lq6vYmlpFcUlVSwr29HyVE56n+Dxy5NzBzFz4qiWoqoxWelkpveJceTJI7REYGYpwL3AeUA5sNjM5rn7B1HNfgo84O5zzexs4IfANWHFJCLhcnfWb9/LktIqikurWFJSxYdbduMOKb2M8SMzmHVaHqfmZzI5P5NRg9IS4vHLni7MM4IpwFp3XwdgZo8AM4HoRDAB+Hrk9avAUyHGIyJdrK6hieUbdrYc7ReXVrF1T3DzNqNvbyblZ3LhySMpys/klLzBpMeg+wQ5vDD/r+QAZVHT5cDprdq8C1xKcPno80CGmQ11920hxiUiR6iquo4lpVUsWR8c7b9bvoN9kUrbvCH9+OTYLCZHjvaPG56R8M/fJ4pYp+dvAr80s+uAvwEVQGPrRmZ2PXA9wOjRo7szvlAMGDCAPXv2xDoMkUNyd9ZtrWZJSVXkUs92PqqsBoICrBNyBnH11HyKIjv+YQPTYhyxHKkwE0EFkBc1nRuZ18LdNxCcEWBmA4DL3H1H6xW5+/3A/RDUEYQUr0hSq61v5P2KncFOv6SKpeur2F4d9Ic/qF8qk/MzufTUXCbnZ3JK7mD69UneZ/ITTZiJYDEw1swKCRLAbOCq6AZmlgVsd/cm4DaCJ4h6nFtvvZW8vDxuuOEGAL73ve/Ru3dvXn31Vaqqqqivr+cHP/gBM2fOjHGkIvtt3bOvZYdfXLKd5RW7qGsMLvMURvrVaT7aPyZ7AL10mSdhhZYI3L3BzG4Enid4fHSOu68ws+8Dxe4+DzgT+KGZOcGloRuO+oOfvRU2vX/UqznAiJPg/DvbXTxr1ixuueWWlkTw2GOP8fzzz3PTTTcxcOBAtm7dytSpU7n44ov1hITERFOTs7ZyD8WRyzxLSrdTsm0vAH1SenFS7iD+YXoBk/MzOTU/k6wBfQ+zRkkkod4jcPf5wPxW826Pev048HiYMXSHSZMmsWXLFjZs2EBlZSWZmZmMGDGCr33ta/ztb3+jV69eVFRUsHnzZkaMGBHrcCUJ1NQ1sqxsB0tKt7OktIql63ewsybod2dIeh8m52dy5ZTRFBVkcsKoQUnd9YLE/mZx1zvEkXuYrrjiCh5//HE2bdrErFmzeOihh6isrGTJkiWkpqZSUFDQZvfTIl1h867aA472V2zY1TJIyrHDBnD+iSNanuYpzErXmakcIPESQYzMmjWLr3zlK2zdupXXX3+dxx57jGHDhpGamsqrr75KaWlprEOUBNHY5KzetLvlaL+4tIryqmD8ir69e3FK3mCu/9QYigoymZSXqQpdOSwlgi5ywgknsHv3bnJychg5ciRf/OIXueiiizjppJMoKiri+OOPj3WI0kPt2dfAsvU7KI7s+Jet39HSl352Rl+K8jO57owCigqGMGHkQPr0Tt7ulOXIKBF0offf33+TOisri4ULF7bZTjUEcigVO2ooLtkeucxTxcqNu2jyoD/9ccMzuHjiKIoKMinKH0JuZj9d5pGjpkQgEkMNjU2s3Li75Wh/SWkVG3cG95L690lhYt5gbjzrWCYXDGFi3mAG9VOvm9L1lAhEutGu2nqWlla17PSXle1gb11QTD9yUBqT8zMpys+kqGAIx4/IoHcSj5ol3SdhEoG7J/wpck8bTS7ZuTtl22soLt1OcWnQDfPqzUFPnL0Mxo8cyBWTc5lcMITJ+ZnkDO4X65AlSSVEIkhLS2Pbtm0MHTo0YZOBu7Nt2zbS0tSfS7yqa2hixYb9XTQsWV/VMozigL69mTR6MOefOJKigqAnzgHqiVPiREL8S8zNzaW8vJzKyspYhxKqtLQ0cnNzYx2GtLK8YicPLixl3rsbWgZcyc3sx/RjhjK5YAhF6olT4lxCJILU1FQKCwtjHYYkkbqGJp5bsYkHFpRQXFpFv9QULj5lFJ8el83k/EyGqydO6UESIhGIdJfNu2p5+K31PPz2eip37yN/aH++c+F4rpicx6D+eqJHeiYlApHDcHeKS6uYu6CE55ZvotGds8YN49pp+XxqbLZ65ZQeT4lApB01dY38ZVkFcxeWsnLjLgam9eYfphdw9dR88oemxzo8kS6jRCDSSum2ah5cWMpjxWXsqm1g/MiB3HnpScycmKPBWCQhKRGIEPTX//qaSh5YUMJrH1aSYsaME0fwpTMKKMrPTNjHkkVAiUCS3M699fxpSRkPLiqldNtesjP6ctPZY7nq9NF68keShhKBJKWVG3fxwMJSnnqngpr6RoryM/nGZ8Yx44QR6r1Tko4SgSSN+sYmXlixmbkLS3j74+307d2LSybmcM20fE7MGRTr8ERiRolAEt6W3bU88nYZD71VyuZd+8gb0o9vXzCeK4pyGdxfg7aIhJoIzGwGcDfB4PW/dfc7Wy0fDcwFBkfa3BoZ51jkqLg7S9fv4IGFJcx/fyP1jc6nj8vmh5fm8+njhqm7B5EooSUCM0sB7gXOA8qBxWY2z90/iGr2HeAxd/9fM5tAMNB9QVgxSeKrrW9k3rINzF1YwooNu8hI6801Uwu4Zlo+hVl69l+kLWGeEUwB1rr7OgAzewSYCUQnAgcGRl4PAjaEGI8ksLLte/nDolIeLS5jx956xg3P4I7Pn8glE3NIVy+fIocU5i8kByiLmi4HTm/V5nvAC2b2b0A6cG5bKzKz64HrAUaPHt3lgUrP1NTkvLF2Kw8sLOHlVVvoZcZnTxjOtdMKOL1wiJ79F+mgWB8qXQn83t3/x8ymAQ+a2Ynu3hTdyN3vB+4HKCoq0ugsSW5XbT1/XlLOgwtLWbe1mqwBfbjxrGO56vTRjBzUTYO7uEP1VthVEfnbAHs2B/OtV6s/a/XfNv6g/WWHWwet5x1uOmreQe9t3eZQMR/qe7UzT+JSmImgAsiLms6NzIv2ZWAGgLsvNLM0IAvYEmJc0kN9uHk3Dyws4YmlFeyta2TS6MHcNWsi5580gr69u7DrB3fYux12lQc7+J3l+3f2O6N2/I37Wr0xsvPzJoKrnnKQ6CTSKwV694XU/pDaL+q/kde90w6e1/Lfdpb1bt22X/A5ckhhJoLFwFgzKyRIALOBq1q1WQ+cA/zezMYDaUBijy4jndLQ2MRLKzczd0EpC9dto0/vXsw8ZRTXTivgpNwjePbfHWqqgp15y0691etdG6Ch9sD39eoNGaNgUA7knArjL4KBOcH0wMhfejb06rX/c9yDpND8R6tpb2rVpq3lze+l/WUt7z3U+w/1OYeJq824O/CZLe9tp01TIzTsg/q9UF8T/DVE/lu7cf+86OVHkmBT+kYljDYSzgEJpIMJp61E1YMTTmiJwN0bzOxG4HmCR0PnuPsKM/s+UOzu84BvAL8xs68R/B++zjUwrwBb9+zjkbfX89Bb69m4s5acwf249fzj+UJRHkPS23n23x1qd0R26huijugrDnzdUHPg+ywFMkYGO/WRE2HcBTAod/8OflDzTr4TP/TmyyqoSrnLuB+cOOr3Bkm7ZV6rZfWHWNZQ207C2Xtk8bUknI6csbROOP1oO1m1cabTq+v/TVlP2+8WFRV5cXFxrMOQELg7y8p28MDCUp55byN1jU18cmwW104r4Oxx2aTU7Yrs4Ns4im/e+ddXH7hS6xXs5AfmwMBRkR38qMgOPvJ6wPAefTQnXay9hBN9xtLWsjanW7/vKBPOBT+FKV85oq9lZkvcvaitZbG+WSxCbX0jzy1Zw4uLlrB7SymFqTu4P6+eUwfvZWDdZnhlAzxVAXV7Dnyj9YIBI4Kd+fAJMPa8gy/XDBgOKfpnLp1gFjmiD7nTQffI2Uw7SaKtBJLX+sHLrqFfiIRv3+6om66RI/qd5dRuL2PPllL61mziEmq4BKD5qs8mgz3Dg5169nFwzNmRHfwoGBg5ks8YASkaHlJ6KLP9l4JiTIlAjk5d9aFvuu6sgH07D3rbzpQhlNYPZqNnkTpkImPGHEf+mOOw5iP6jJHayYt0EyUCaV/d3kPfdN1VEdycbS09O7gsk1kIBZ+AgTnU9h/Ba5v68OCKet7elkZGen9mn5HHF6fmkzM49kdEIslMiUAOtv4tePMuWP0sBz2u1z8ruCyTmQ/5Zxx8A3bgqODZ8Ii1W3bzwMJSnnixgj37Gjglbzh3np3PhSePJC1VN2hF4oESgQSammDNC0ECWL8Q0gbDGf8Gw0/YfwM2Y1SHbqA1NDbx8qotPLCwhDfXbqNPSi8+d8pIrp1WwMS8wWF/ExHpJCWCZNdYD+8/Dm/eDZUrgxuxM+6ESddA3wGdWtX26joeWbyehxatp2JHDaMGpfHvnx3H7NPyGDqg7+FXICIxoUSQrPbtgaUPwMJ7g+v+wybA538NJ17W6Zu075XvYO6CUp5+bwN1DU2cccxQvvu5CZw7fhi9U1RQJRLvlAiSTfVWeOs+ePs3wY3e0WfA534GYz/TqU7B9jU0Mv/9jcxdUMqysh3075PCrKI8rp2Wz9jhGeHFLyJdTokgWWz/GBb+Et75Q1DEMu5C+MQtkDelU6vZsKOGh99azx/fXs+26jrGZKXzvYsmcOnkXAam6XFPkZ5IiSDRbXwvuAG84smgT51TZsEZN0H2uA6vwt1ZtG47Dyws4YUPNuPunDN+ONdOy2f6MVn00rCPIj2aEkEicoeP/xYkgI9egT4ZMO0GmPrV4PHODqre18AT71Tw4MISPty8h8H9U/nKJ8fwxdNHkzekf3jxi0i3UiJIJE2NsPLpIAFseAfSh8E5t0PRl6Hf4E6tanHJdv7x94vZXdvAiTkD+cnlJ3PRKaP07L9IAlIiSAT1tfDuw7DgF7B9HQwZA5/7OZxy1RF1nNXY5Hz3qeUMTEtl7j9OYVLeYA37KJLAlAh6spodUPw7WHQfVG8J+tK/4vcw/uKj6lb5z0vLWbVpN7+4chKnjs7sqmhFJE4pEfREuzbAol9B8e+hbnfQM+f0W6DwU0c9Luzeugb+54XVTMwbzOdOHtkl4YpIfFMi6EkqP4QFd8O7j4I3wgmfh+k3w8hTuuwjfvv3j9m8ax+/vOpUXQ4SSRJKBD1B2eLgBvCqZ4IO3SZ/CabdCEMKu/Rjtuyq5b7XP2LGCSM4rWBIl65bROJXqInAzGYAdxOMWfxbd7+z1fKfA2dFJvsDw9x9cJgx9RjusObFIAGUvhl0Avepb8KUf4YB2aF85M9f+pC6hiZuPf/4UNYvIvEptERgZinAvcB5QDmw2MzmufsHzW3c/WtR7f8NmBRWPD1GYz0sfyLoBG7LiqATuM/+EE69ttOdwHXG6k27eXRxGV86o4CCrPTQPkdE4k+YZwRTgLXuvg7AzB4BZgIftNP+SuA/Q4wnvtVV7+8EbmcZZI+HS+6Dky7vlpG6fvjsStL79uams8eG/lkiEl/CTAQ5QFnUdDnQ5sjLZpYPFAKvtLP8euB6gNGjR3dtlLFWvQ3e/jW8fT/UVMHoaXDBT4NO4Hp1T8+df19TyWurK/nWBceTmd7n8G8QkYQSLzeLZwOPu3tjWwvd/X7gfoCioiJvq02PU1UadAK39EFoqIFxFwSPgI5uM1eGprHJueOZleRm9uPaaQXd+tkiEh/CTAQVQF7UdG5kXltmAzeEGEv82PR+cP1/+RNgveDkWTC9c53AdaUnIsVj91w5Sd1HiCSpMBPBYmCsmRUSJIDZwFWtG5nZ8UAmsDDEWGLLHUr+Dm/cBR+9DH0GwNR/DTqBG5QTs7D21jXw0xdWc0reYC5S8ZhI0gotEbh7g5ndCDxP8PjoHHdfYWbfB4rdfV6k6WzgEXdPjEs+0ZoaYdVfgwSwYSmkZ8PZ34XTvgz9Yt91g4rHRARCvkfg7vOB+a3m3d5q+nthxhATDfvg3T8GncBtWwuZhXDhz2DiVZDaL9bRAbBlt4rHRCQQLzeLE0PtTiieA4v+F/ZsDrp+uPz/YMLMo+oELgw/f3ENdQ1N/IeKx0SSnhJBV9i9KdIJ3P/Bvl0w5qxgIPgxZx51J3Bh+HDzbh5dvJ5rpxVQqOIxkaSnRHA0tq6BBffAu49AUwNMuCToBG7UxFhHdkg/nB8pHjtHxWMiokRwZMqL4Y2f7+8EbtI1cMaNwYAwce6NNVt5dXUlt51/PENUPCYiKBF0nDusfSl4Aqj0DUgbBJ/8Bpz+L6F1AtfVGpucO+avJGdwP750RkGswxGROKFEcDiNDbAi0gnc5uWQMQo+c0fQFXTfjFhH1ylPLC1n5cZdKh4TkQMoEbSnrhre+QMs+CXsXA9Z42Dmr+CkK6B3z7ukUlPXqOIxEWmTEkFre7cHHcC99Wuo2Q55U+GCH8PYz3ZbJ3Bh+O3f17F51z5+caWKx0TkQEoEzXasD7qAXvoA1O+F486HT9wCo6fGOrKjtmV3Lf/7+kd89oThTClU8ZiIHEiJYNPySCdwfw6e+T/pC0EncMPGxzqyLnPXS5HisRkqHhORgyVnInAPhn984y5Y+yKkpgdP/0z7KgzKjXV0XWrN5t088nZQPDYmO7wRzkSk50quRNDUBKufCRJARTH0z4KzvhN0Atc/MS+Z/LeKx0TkMJInEax+Fl74LmxbA5kFcOH/wMQvxk0ncGFQ8ZiIdETyJIL6vcFO//I5MH4mpCT2V1fxmIh0VGLvDaNN+DyccGlcdgIXhiffqWDlxl3cPXuiisdE5JCSJxH04BqAzqqpa+Snz6/mlNxBXHTyqFiHIyJxLnn2jknkd2+sY9OuWr51wXh69UqOMyAROXJKBAmmcvc+/ve1j/jMhOGcPmZorMMRkR5AiSDB/PylD9nX0MStGnlMRDoo1ERgZjPMbLWZrTWzW9tp8wUz+8DMVpjZw2HGk+iai8eunpqv4jER6bDQbhabWQpwL3AeUA4sNrN57v5BVJuxwG3AdHevMrNhYcWTDH747CrS+6h4TEQ6J8wzginAWndf5+51wCPAzFZtvgLc6+5VAO6+JcR4Etqba7fyyqot3HD2sSoeE5FOOaJEYGYdue6QA5RFTZdH5kU7DjjOzN40s0VmNqOdz7vezIrNrLiysvJIQk5oTU3OHc8ExWPXqXhMRDrpSM8IPjh8kw7pDYwFzgSuBH5jZoNbN3L3+929yN2LsrN7xrCQ3enJdyr4YOMu/t+McSoeE5FOa/cegZl9vb1FQEfOCCqAvKjp3Mi8aOXAW+5eD3xsZh8SJIbFHVi/sH/ksZNVPCYiR+hQZwT/DWQCGa3+Bhzmfc0WA2PNrNDM+gCzgXmt2jxFcDaAmWURXCpa1/Hw5XdvrGPjzlq+reIxETlCh3pqaCnwlLsvab3AzP7pcCt29wYzuxF4HkgB5rj7CjP7PlDs7vMiyz5jZh8AjcC/u/u2I/kiyUjFYyLSFQ6VCCqAUjO72d3vbrWsqCMrd/f5wPxW826Peu3A1yN/0kl3qXhMRLrAoS7xTAD6AP9oZplmNqT5D6jvnvCkPWs27+aRxWV88fTRKh4TkaNyqDOCXwMvA2OAJQQ3iZt5ZL7EyJ3PrqJ/aoqKx0TkqLV7RuDu97j7eIJr+2PcvTDqT0kghhas3crLq7bw1bOOZeiAvrEOR0R6uMM+/ePu/9odgUjHNEWNPPYP0wtiHY6IJAD1PtrDPLWsghUbdvHvn1XxmIh0DSWCHqSmrpGfPB8Uj118iorHRKRrKBH0IHPe/JiNOzXymIh0LSWCHqJy9z5+9epazpswnKkqHhORLqRE0EPc/fKH1Kp4TERCoETQA6zdsps/vh0Ujx2j4jER6WJKBD3Anc+uol9qCjereExEQqBEEOcWfLSVl1Zu4atnHaPiMREJhRJBHGtqcv47Ujz2j9MLYx2OiCQoJYI49tSyCpZXqHhMRMKlRBCnauuD4rGTclQ8JiLhUiKIU797Q8VjItI9lAji0NY9wchj544fzrRjVDwmIuFSIohDd7+0hpr6RhWPiUi3CDURmNkMM1ttZmvN7NY2ll9nZpVmtizyd9ixkBPd2i17ePjt9Vw1ZTTHDlPxmIiE71AjlB0VM0sB7gXOA8qBxWY2z90/aNX0UXe/Maw4epo7n10ZFI+dq+IxEekeYZ4RTAHWuvs6d68DHgFmhvh5PV508ViWisdEpJuEmQhygLKo6fLIvNYuM7P3zOxxM8tra0Vmdr2ZFZtZcWVlZRixxlxz8dioQWkqHhORbhXrm8VPAwXufjLwIjC3rUbufr+7F7l7UXZ2drcG2F3+8m6keGyGisdEpHuFmQgqgOgj/NzIvBbuvs3d90UmfwtMDjGeuFVb38hPnlvNiTkDmXlKWydNIiLhCTMRLAbGmlmhmfUBZgPzohuY2cioyYuBlSHGE7fmvPkxG1Q8JiIxEtpTQ+7eYGY3As8DKcAcd19hZt8Hit19HnCTmV0MNADbgevCiidebduzj1+9+hHnjh/GGcdkxTocEUlCoSUCAHefD8xvNe/2qNe3AbeFGUO8u6uleGx8rEMRkSQV65vFSU3FYyISD5QIYqhl5DEVj4lIDCkRxMjCj7bx0srN/OuZKh4TkdhSIoiB5uKxkYPS+PInVDwmIrGlRBAD897dwPsVOzXymIjEBSWCbtY88tgJowZyyUQVj4lI7CkRdLM5b35MxY4avn2hisdEJD4oEXQjFY+JSDxSIuhGd7+skcdEJP4oEXSTjyr38NBb67lySh7HDsuIdTgiIi2UCLpJc/HYLeceF+tQREQOoETQDRat28aLH6h4TETikxJByKKLxzTymIjEIyWCkM17dwPvle/km58ZR78+Kh4TkfijRBCi6OKxz09S8ZiIxCclghD935slQfGYRh4TkTimRBCSoHhsLeccP4wzjlXxmIjELyWCkNzz8hr21jdy2wUqHhOR+BZqIjCzGWa22szWmtmth2h3mZm5mRWFGU93aS4em32aisdEJP6FlgjMLAW4FzgfmABcaWYT2miXAdwMvBVWLN3tR8+uom/vXioeE5EeIcwzginAWndf5+51wCPAzDba/X/Aj4DaEGPpNm+t28YLkeKx7AwVj4lI/AszEeQAZVHT5ZF5LczsVCDP3Z851IrM7HozKzaz4srKyq6PtIs0NTl3tIw8NibW4YiIdEjMbhabWS/gZ8A3DtfW3e939yJ3L8rOzg4/uCP09HsqHhORnifMRFAB5EVN50bmNcsATgReM7MSYCowr6feMK6tb+THz61mwkgVj4lIzxJmIlgMjDWzQjPrA8wG5jUvdPed7p7l7gXuXgAsAi529+IQYwrN7xcExWPf0chjItLDhJYI3L0BuBF4HlgJPObuK8zs+2Z2cVifGwvbq+u495W1nK3iMRHpgXqHuXJ3nw/MbzXv9nbanhlmLGG65+U1VNc1cJtGHhORHkiVxUdpXeUe/rColNlTRjN2uIrHRKTnUSI4SndGise+puIxEemhlAiOgorHRCQRKBEcoeaRx0YMVPGYiPRsSgRH6On3NvBu+U6++VkVj4lIz6ZEcARUPCYiiUSJ4AjMjRSPffvC8aSoeExEejglgk7aXl3HL19dy1njspmu4jERSQBKBJ10z8trqN7XwLcuGB/rUEREuoQSQSeoeExEEpESQSf86LnmkcfGxjoUEZEuo0TQQW9/vJ3nV2zmXz59DMMy0mIdjohIl1Ei6ICmJueOZz5g+MC+/NMnVTwmIolFiaAD/vr+xqB4TCOPiUgCUiI4jNr6Rn707CrGjxzIpafmxjocEZEup0RwGHOjRh5T8ZiIJCIlgkNQ8ZiIJAMlgkNoLh67TcVjIpLAlAja8fHWav6wqJRZp43mOBWPiUgCCzURmNkMM1ttZmvN7NY2lv+Lmb1vZsvM7A0zmxBmPJ3xo2dX0ad3L752norHRCSxhZYIzCwFuBc4H5gAXNnGjv5hdz/J3ScCPwZ+FlY8nbG4ZDvPrdik4jERSQphnhFMAda6+zp3rwMeAWZGN3D3XVGT6YCHGE+HuDs/eGZlpHisMNbhiIiErneI684ByqKmy4HTWzcysxuArwN9gLPbWpGZXQ9cDzB69OguDzTa0+9t5N2yHfzk8pPp3yfMzSMiEh9ifrPY3e9192OA/wC+006b+929yN2LsrOzQ4tFxWMikozCTAQVQF7UdG5kXnseAS4JMZ7DemBhZOSxC1Q8JiLJI8xEsBgYa2aFZtYHmA3Mi25gZtGP5FwIrAkxnkOqqq7jF6+s5cxx2XxirIrHRCR5hHYR3N0bzOxG4HkgBZjj7ivM7PtAsbvPA240s3OBeqAK+FJY8RzOPa9EisfOV/GYiCSXUO+Guvt8YH6rebdHvb45zM/vqI+3VvPgwlJmnZbHuBEqHhOR5BLzm8Xx4MfPNRePHRfrUEREul3SJ4LFJdt5drmKx0QkeSV1IlDxmIhIkieCv0aKx77xmXEqHhORpJW0iWBfQyM/em4Vx4/I4DIVj4lIEkvaRPDAglLKq2r4tkYeE5Ekl5SJICgeW8Onj8vmk2PD67JCRKQnSMpE8ItX1rJnXwPf0shjIiLJlwhKtlbz4KISFY+JiEQkXSL40XOrSE3pxdfOVfGYiAgkWSIojhSP/fOnjmHYQBWPiYhAEiWC5uKxYRl9+cqnVDwmItIsaRLBM+9vZFnZDr6p4jERkQMkTSJI79ub8yYM57LJKh4TEYmWNIfGZ40bxlnjhsU6DBGRuJM0ZwQiItI2JQIRkSSnRCAikuRCTQRmNsPMVpvZWjO7tY3lXzezD8zsPTN72czyw4xHREQOFloiMLMU4F7gfGACcKWZTWjV7B2gyN1PBh4HfhxWPCIi0rYwzwimAGvdfZ271wGPADOjG7j7q+6+NzK5CNCznSIi3SzMRJADlEVNl0fmtefLwLNtLTCz682s2MyKKysruzBEERGJi5vFZnY1UAT8pK3l7n6/uxe5e1F2tsYPEBHpSmEWlFUAeVHTuZF5BzCzc4FvA592932HW+mSJUu2mlnpEcaUBWw9wveGSXF1juLqvHiNTXF1ztHE1e7DOObuR7jOQzOz3sCHwDkECWAxcJW7r4hqM4ngJvEMd18TSiAHxlTs7kVhf05nKa7OUVydF6+xKa7OCSuu0C4NuXsDcCPwPLASeMzdV5jZ983s4kiznwADgD+Z2TIzmxdWPCIi0rZQ+xpy9/nA/Fbzbo96fW6Yny8iIocXFzeLu9H9sQ6gHYqrcxRX58VrbIqrc0KJK7R7BCIi0jMk2xmBiIi0okQgIpLkEjIRdKCzu75m9mhk+VtmVhAncV1nZpWRJ6iWmdk/dVNcc8xsi5ktb2e5mdk9kbjfM7NT4ySuM81sZ9T2ur2tdl0cU56ZvRrpLHGFmd3cRptu314djCsW2yvNzN42s3cjcf1XG226/ffYwbhi8nuMfHaKmb1jZn9tY1nXby93T6g/IAX4CBgD9AHeBSa0avNV4L7I69nAo3ES13XAL2OwzT4FnAosb2f5BQTdfxgwFXgrTuI6E/hrN2+rkcCpkdcZBLUyrf8/dvv26mBcsdheBgyIvE4F3gKmtmoTi99jR+KKye8x8tlfBx5u6/9XGNsrEc8IDtvZXWR6buT148A5ZmZxEFdMuPvfgO2HaDITeMADi4DBZjYyDuLqdu6+0d2XRl7vJqiRad2HVrdvrw7G1e0i22BPZDI18tf6CZVu/z12MK6YMLNc4ELgt+006fLtlYiJoCOd3bW08aDwbScwNA7iArgscjnhcTPLa2N5LHS2A8HuNC1yev+smZ3QnR8cOSWfRHA0GS2m2+sQcUEMtlfkMscyYAvworu3u7268ffYkbggNr/Hu4D/BzS1s7zLt1ciJoKe7GmgwIPxGV5kf9aXti0F8t39FOAXwFPd9cFmNgD4M3CLu+/qrs89nMPEFZPt5e6N7j6RoL+xKWZ2Ynd87uF0IK5u/z2a2eeALe6+JOzPipaIiaAjnd21tLGgT6RBwLZYx+Xu23x/x3u/BSaHHFNHdagDwe7m7ruaT+89qGJPNbOssD/XzFIJdrYPufsTbTSJyfY6XFyx2l5Rn78DeBWY0WpRLH6Ph40rRr/H6cDFZlZCcPn4bDP7Q6s2Xb69EjERLAbGmlmhmfUhuJnSug+jecCXIq8vB17xyJ2XWMbV6jryxQTXeePBPODayNMwU4Gd7r4x1kGZ2Yjma6NmNoXg33OoO5DI5/0OWOnuP2unWbdvr47EFaPtlW1mgyOv+wHnAataNev232NH4orF79Hdb3P3XHcvINhHvOLuV7dq1uXbK9S+hmLB3RvMrLmzuxRgjkc6uwOK3X0ewQ/mQTNbS3AzcnacxHWTBR3yNUTiui7suADM7I8ET5RkmVk58J8EN89w9/sI+ou6AFgL7AX+IU7iuhz4VzNrAGqA2d2Q0KcD1wDvR64vA3wLGB0VVyy2V0fiisX2GgnMtWDo2l4EnU/+Nda/xw7GFZPfY1vC3l7qYkJEJMkl4qUhERHpBCUCEZEkp0QgIpLklAhERJKcEoGISJJTIhDpRhb0AHpQj5IisaREICKS5JQIRNpgZldH+qtfZma/jnRQtsfMfh7pv/5lM8uOtJ1oZosinZM9aWaZkfnHmtlLkU7elprZMZHVD4h0YrbKzB4Ku6dNkcNRIhBpxczGA7OA6ZFOyRqBLwLpBNWdJwCvE1Q6AzwA/Eekc7L3o+Y/BNwb6eTtDKC5m4lJwC3ABILxKaaH/JVEDinhupgQ6QLnEHQwtjhysN6PoKviJuDRSJs/AE+Y2SBgsLu/Hpk/F/iTmWUAOe7+JIC71wJE1ve2u5dHppcBBcAboX8rkXYoEYgczIC57n7bATPNvtuq3ZH2z7Iv6nUj+h1KjOnSkMjBXgYuN7NhAGY2xMzyCX4vl0faXAW84e47gSoz+2Rk/jXA65FRwsrN7JLIOvqaWf/u/BIiHaUjEZFW3P0DM/sO8IKZ9QLqgRuAaoIBTL5DcKloVuQtXwLui+zo17G/t9FrgF9Heo6sB67oxq8h0mHqfVSkg8xsj7sPiHUcIl1Nl4ZERJKczghERJKczghERJKcEoGISJJTIhARSXJKBCIiSU6JQEQkyf3/z0BtE98xZm0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(model.history.history.keys())\n",
    "# summarize history for accuracy\n",
    "plt.plot(model.history.history['f1'])\n",
    "plt.plot(model.history.history['val_f1'])\n",
    "plt.title('model f1')\n",
    "plt.ylabel('f1')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'val'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Со сверточными слоями часто используется Pooling. Это тоже своего рода свертка, но без обучаемых параметров. В пулингах используется какое-то правило, по которому из чисел в каком-то окне выбирается 1 число (т.е. происходит свертка). В качестве правила используется максимум или среднее. Pooling слой ставят после сверточного. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs = tf.keras.layers.Input(shape=(MAX_LEN,))\n",
    "embeddings = tf.keras.layers.Embedding(input_dim=len(word2id), output_dim=100)(inputs, )\n",
    "\n",
    "conv1 = tf.keras.layers.Conv1D(kernel_size=10, filters=32, strides=5)(embeddings)\n",
    "conv2 = tf.keras.layers.Conv1D(kernel_size=5, filters=32, strides=1, activation='relu')(conv1)\n",
    "pool = tf.keras.layers.AveragePooling1D()(conv2)\n",
    "\n",
    "concat = tf.keras.layers.Flatten()(pool)\n",
    "dense = tf.keras.layers.Dense(64, activation='relu')(concat)\n",
    "outputs = tf.keras.layers.Dense(len(label2id), activation='softmax')(dense)\n",
    "\n",
    "model = tf.keras.Model(inputs=inputs, outputs=outputs)\n",
    "optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)\n",
    "model.compile(optimizer=optimizer,\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=[f1])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "71/71 [==============================] - 14s 170ms/step - loss: 0.4321 - f1: 0.0148 - val_loss: 0.1698 - val_f1: 0.0000e+00\n",
      "Epoch 2/5\n",
      "71/71 [==============================] - 11s 157ms/step - loss: 0.1631 - f1: 0.0547 - val_loss: 0.1431 - val_f1: 0.3513\n",
      "Epoch 3/5\n",
      "71/71 [==============================] - 11s 156ms/step - loss: 0.1292 - f1: 0.4426 - val_loss: 0.0919 - val_f1: 0.6686\n",
      "Epoch 4/5\n",
      "71/71 [==============================] - 11s 156ms/step - loss: 0.0762 - f1: 0.7320 - val_loss: 0.0744 - val_f1: 0.7375\n",
      "Epoch 5/5\n",
      "71/71 [==============================] - 11s 156ms/step - loss: 0.0522 - f1: 0.8315 - val_loss: 0.0717 - val_f1: 0.7595\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f70289fb470>"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# как выглядит ошибка при слишком большом батче\n",
    "model.fit(X_train, y_train, \n",
    "          validation_data=(X_valid, y_valid),\n",
    "          batch_size=2000,\n",
    "         epochs=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['loss', 'f1', 'val_loss', 'val_f1'])\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA0x0lEQVR4nO3dd3wVVf7/8deHdCBASEJLgIQSpLcQQARZ24IoIEUQUJroWta67uJ+3bXu/tx1V9euqAjSi4iIKFaKSgsISCeEFmoSSCCQfs/vj7liSIEAuZlbPs/Hg4f33pnMfWfMnc+dc2bOEWMMSimlfFcVuwMopZSylxYCpZTycVoIlFLKx2khUEopH6eFQCmlfJwWAqWU8nFaCJS6QiIyRUReKOe6+0TkhjKWhYjIZyKSKSLzKjalUmXTQqCU+xgC1AXCjTFDRaS+iCwSkcMiYkQkxuZ8yktpIVDKfTQGdhljCpzPHcCXwGD7IilfoIVA+QRnk8wTIrJZRM6IyAciUldEvhCR0yLyjYiEFVm/v4hsFZEMEVkmIi2LLOsoIhucPzcHCC72XreIyEbnz/4kIu3Kke9Z4O/AMBHJEpHxxphjxpi3gHUVtyeUKkkLgfIlg4EbgTjgVuAL4K9AJNZn4SEAEYkDZgGPOJctAT4TkUARCQQWAtOA2sA8inxjF5GOwGTgXiAceBdYJCJBFwpmjHka+CcwxxhT3RjzQYX8xkqVgxYC5Uted37LPgSsBNYYY342xuQAnwAdnesNAz43xnxtjMkH/gOEAFcD3YAA4H/GmHxjzHzO/8Z+D/CuMWaNMabQGDMVyHX+nFJuyd/uAEpVomNFHmeX8ry683EDYP+vC4wxDhE5CEQBhcAhc/5ojfuLPG4MjBaRPxZ5LdC5TaXckhYCpUo6DLT99YmICNAQOAQYIEpEpEgxaATscT4+CPzDGPOPSsyr1BXRpiGlSpoL9BOR60UkAHgcq3nnJ2AVUAA8JCIBIjIISCjys+8BfxCRrmKpJiL9RCT0coKISDDwa/9CkPO5UhVKC4FSxRhjdgKjgNeBNKyO5VuNMXnGmDxgEDAGOIHVn7CgyM8mAhOAN4CTQJJz3cuVDWQ5H+9wPleqQolOTKOUUr5NzwiUUsrHaSFQSikfp4VAKaV8nBYCpZTycR53H0FERISJiYmxO4ZSSnmU9evXpxljIktb5nGFICYmhsTERLtjKKWURxGR/WUt06YhpZTycVoIlFLKx2khUEopH+dxfQSlyc/PJyUlhZycHLujuFRwcDDR0dEEBATYHUUp5UW8ohCkpKQQGhpKTEwM1kCR3scYQ3p6OikpKcTGxtodRynlRbyiaSgnJ4fw8HCvLQIAIkJ4eLjXn/UopSqfVxQCwKuLwK984XdUSlU+rykESinljQodhg0HTvK/b3ax7fApl7yHV/QR2C0jI4OZM2dy//33X9LP3XzzzcycOZNatWq5JphSyiMdycxmxa5UVuxK44ekNDKz8xGBiOpBtGpQo8LfTwtBBcjIyOCtt94qUQgKCgrw9y97Fy9ZssTV0ZRSHiAnv5A1e084D/6p7D5uzUVUt0YQN7WqS6+4SK5pFkFYtUCXvL8WggowceJE9uzZQ4cOHQgICCA4OJiwsDB27NjBrl27GDhwIAcPHiQnJ4eHH36Ye+65B/htuIysrCz69u3LNddcw08//URUVBSffvopISEhNv9mSilXMMaw+3gWK3alsnxXKmv3niC3wEGgfxW6xtbm9viG9IqLJK5u9UrpG/S6QvDsZ1srvB2tVYMaPH1r6zKXv/jii2zZsoWNGzeybNky+vXrx5YtW85d5jl58mRq165NdnY2Xbp0YfDgwYSHh5+3jd27dzNr1izee+89br/9dj7++GNGjRpVob+HUso+GWfz+CEpjRW7Ulm5O40jmdYVgM3qVGdk18b0iouga2w4IYF+lZ7N6wqBO0hISDjvWv/XXnuNTz75BICDBw+ye/fuEoUgNjaWDh06ANC5c2f27dtXWXGVUi5QUOhgU0oGy3dZB//NKRk4DNQI9uea5hE83DySnnGRRNWy/8zfpYVARPoArwJ+wPvGmBeLLW8ETAVqOdeZaIy5oobzC31zryzVqlU793jZsmV88803rFq1iqpVq9K7d+9S7wUICgo699jPz4/sbJ2jXClPcygj+1w7/49JaZzKKaCKQPuGtfjjdc3pFRdJ++ia+Pu51wWbLisEIuIHvAncCKQA60RkkTFmW5HVngLmGmPeFpFWwBIgxlWZXCU0NJTTp0+XuiwzM5OwsDCqVq3Kjh07WL16dSWnU0q5SnZeIav3pp87+O9JPQNA/ZrB9G1Tn15xkfRoFk6tqq7p5K0orjwjSACSjDHJACIyGxgAFC0EBvj1WqiawGEX5nGZ8PBwevToQZs2bQgJCaFu3brnlvXp04d33nmHli1b0qJFC7p162ZjUqXUlTDGsPPY6XOXdq7dd4K8AgdB/lXo2iScOxIacW1cJM3qVE4nb0URY4xrNiwyBOhjjLnb+fxOoKsx5sEi69QHvgLCgGrADcaY9Rfabnx8vCk+Mc327dtp2bJlBf8G7smXflel3MHJM3msPNfJm8qxU7kAxNWtTq/mkfSKiyQhtjbBAZXfyXspRGS9MSa+tGV2dxbfAUwxxvxXRLoD00SkjTHGUXQlEbkHuAegUaNGNsRUSvmK/EIHGw9mnGvu2XwoE2OgZkgA1zSP4NrmkfSMi6B+Tfs7eSuKKwvBIaBhkefRzteKGg/0ATDGrBKRYCACOF50JWPMJGASWGcErgqslPJNB0+cZcVu68D/U1I6p3OtTt6OjcJ45Po4esVF0C66Fn5VPKe551K4shCsA5qLSCxWARgOjCi2zgHgemCKiLQEgoFUF2ZSSinO5hWwOjmdFc5LO5PTrE7eqFoh3NK+Pr2aR3J1swhqhvjG3B8uKwTGmAIReRBYinVp6GRjzFYReQ5INMYsAh4H3hORR7E6jscYV3VaKKV8ljGG7UdOn/vWn7jvJHmFDoIDqtCtSTijujWmV1wkTSOreVQnb0VxaR+B856AJcVe+3uRx9uAHq7MoJTyTelZufyQlMZy5528qaetTt6r6oUypkcMvZpHEh8T5vadvJXB7s5ipZSqEPmFDjbsP+n81p/GlsNWJ29Y1QCuaR5Jr+YR9IqLpG6NYLujuh0tBDaoXr06WVlZdsdQyuMdSD/Lcmdzz6o96WTlFuBXRejUqBaP3RBHr7hI2kTV9NpO3oqihUAp5TGycgtYvSf9XFv/vvSzAESHhdC/QwNnJ284NYJ9o5O3omghqAATJ06kYcOGPPDAAwA888wz+Pv78/3333Py5Eny8/N54YUXGDBggM1JlfIsDodh25FTLHde07/hwEnyCw0hAX50bxrOmKtj6BUXSWyEF3TyFhZATiZkn4ScDOu/2RnnP281EBp1rfC39r5C8MVEOPpLxW6zXlvo+2KZi4cNG8YjjzxyrhDMnTuXpUuX8tBDD1GjRg3S0tLo1q0b/fv39/w/VqVcLPV0Liud3/h/SEojLSsPgJb1azDumliubR5J55gwgvzdsJPXGMg9XfaBvNTnGdbz3IsMnx8YCnVaaSFwVx07duT48eMcPnyY1NRUwsLCqFevHo8++igrVqygSpUqHDp0iGPHjlGvXj274yrlVvIKHCTuP3Humv5tR6wDYu1qgfRsHkGv5pH0bB5Bncrs5C3IvYQDebHnprDs7foFQnAtCAmDkFpQo4F1cP/1eUjY+ctDwpyv1QQ/1zV3eV8huMA3d1caOnQo8+fP5+jRowwbNowZM2aQmprK+vXrCQgIICYmptThp5XyRXkFDuatP8j3O46zak86Z/IK8a8idGocxhO/b0Gv5pG0blCDKlfSyesovEhTS0bZB/aCCw0DL9aBueiBu1ajsg/kRZ8HVAU3bBXwvkJgk2HDhjFhwgTS0tJYvnw5c+fOpU6dOgQEBPD999+zf/9+uyMq5RZ2HD3Fo3M2sf3IKRrVrsptnaLo1TyS7k3DCS3eyWsM5J25xKYW5/OcU1j3qZYhoOr5B+raTayD9YUO5CFhEFQTqrjXfAJXSgtBBWndujWnT58mKiqK+vXrM3LkSG699Vbatm1LfHw8V111ld0RlbJVocPw/spkXv5qB22CU1nUu5B2tfIhZw0cOAk7yziwO/LL3mgV//MP1NXrQETchQ/kwbWsx/5BZW/Xx2ghqEC//PJbJ3VERASrVq0qdT29h0D5lDNppO74keXffUnLU7+wIWgv1QqzoOgcTUE1IaTmbwfqGg3K19QSWN0tm1o8jRYCpVTFyc+Bo5shJREOJWJSEpGM/UQCtxnhdFgcVZsNgeh4aNDROuAH1QA/PRTZSfe+UuryOBxwYs+5gz6H1sPRLeeacgpDo/i5sClf5V9NXr1OTBg2iKg6ETaHVqXxmkJgjPH6a/R1YFZlqzNp1sG+6IE/J9NaFljd+oZ/9YMQFc/3WY14/MtjZOUW8Oc+LRjXI/bKrgBSLuUVhSA4OJj09HTCw8O9thgYY0hPTyc4WAfMUpWgWBMPKYmQ4bzyTapAndbQ+jaI6gxR8RDZAqr4cSonn2cXbePjDQdpE1WDV27vQPO6ofb+LuqivKIQREdHk5KSQmqqd89pExwcTHR0tN0xlLdxOOBE8m8H/EOJ5zXxUCPKOuB3GW8d9Bt0gMBqJTbz0540npi3maOncnjoumY8eF1zAv296zJLb+UVhSAgIIDY2Fi7YyjlGc6kn3/QL7OJx/ltv0b9C24uJ7+Ql5bu5IMf9hIbUY35f+hOx0ZhlfCLqIriFYVAKVWG/Bxr7K2iB/6T+6xlUsUa3qDVQOsqniJNPOW15VAmj87ZyO7jWdzVvTET+15F1UA9rHga/T+mlLcwBtL3XLyJJ36c9d/6HSCo+mW9VUGhg7eX7eHVb3cTXj2Qj8Yl0CsusuJ+F1WptBAo5anOa+JZ72ziybCWBVSDqE7Q/YHfvu1fpImnvJJTs3hs7iY2Hsygf/sGPD+gDTWr6vj/nkwLgVKeoFxNPP2tA350PERedUlNPOVhjGH66v38Y8l2gvz9eP2OjtzavkGFvoeyhxYCpdxNiSae9VYR+LWJJ7QBRHeGzmOtg/4VNPGU19HMHJ6Yv4mVu9PoFRfJS0Pa6dy/XkQLgVJ2O5PubNopcuAv0cRz/2/f9mtU7rfwRZsO87eFW8grcPD8wDaM6trIa+/X8VVaCJSqTAW5cGTz+Qf9k3utZVIFIlu6vImnvDLO5vHUwi0s3nyEjo1q8fLtHYiNKHn/gPJ8WgiUcpVzTTxFvu2X2sQz2nmjVkeXN/GU17Kdx/nz/M2cOJPHn26K4w/XNsXfT28O81ZaCJSqKGdPnD8kQ/EmngYdbW3iKY+zeQX8c8l2pq8+QPM61Zk8pgttomraHUu5mBYCpSrC+imw+DFrvtrzmnicd+fWaWlbE095bThwksfmbGT/ibPcfU0sf/p9C4ID3DuzqhhaCJS6Uuveh88fh6bXwzWPWmPxBHnOQGt5BQ5e+3Y3by1Lon7NEGbe3Y3uTcPtjqUqkRYCpa7EmknwxRMQ1xdun+px0x/uOnaaR+dsZOvhUwzpHM3Tt7YqOW+w8npaCJS6XKvegqVPQot+MHQK+AfanajcHA7D5B/38u+lOwkN8ufdOzvz+9b17I6lbKKFQKnL8dPr8NVT0PJWGPIh+HnOt+iUk2d5fO4m1uw9wQ0t6/Li4LZEVPesMxlVsbQQKHWpfvgffPO0NWrn4Pc9pggYY5i/PoVnP9sGwL+HtGNo52i9OUxpIVDqkqz4D3z3PLQZDLdN8phJ19Oycvnrgl/4atsxEmJr89+h7WlYu6rdsZSb8Iy/YqXcwfJ/w/f/gLZDYeA7HlMEvt52jCcXbOZUdgH/d3NLxl+j8wer83nGX7JSdjIGlr0Iy1+EdsNh4Ftuf08AwOmcfJ5fvI25iSm0ql+DGXd3oEU9z7msVVUeLQRKXYgx1lnAipegw0jo/7pHFIE1yek8Pm8ThzOyeeB3TXn4+jidP1iVSQuBUmUxBr59Fn54BTrdBbe8ClXc+2Cak1/If7/ayfs/7KVR7arM+0N3OjeubXcs5ea0EChVGmPg67/DT69Z4/73e9nti8DWw5k8NmcTO4+dZmTXRvz15pZUC9KPuLo4/StRqjhjYOn/weo3ocvd0Pclty4CBYUO3l2RzP++2UVY1UA+HNuF37WoY3cs5UG0EChVlDHw5URY8w4k3At9/wVufJ39vrQzPDZ3IxsOZNCvXX1eGNCGsGqec4ezcg8u/ZojIn1EZKeIJInIxDLWuV1EtonIVhGZ6co8Sl2QMbDkCasIdLvfrYvAr/MH9311JUnHs3h1eAfeuKOjFgF1WVx2RiAifsCbwI1ACrBORBYZY7YVWac58CTQwxhzUkT0fFbZw+GAJY9D4mS4+o9w4/NuWwSOn8rhzx9vZtnOVHo2j+DfQ9pRv2aI3bGUB3Nl01ACkGSMSQYQkdnAAGBbkXUmAG8aY04CGGOOuzCPUqVzOGDxw7DhI+jxCNzwjNsWgcWbD/PUwi3k5Bfy3IDWjOraWG8OU1fMlYUgCjhY5HkK0LXYOnEAIvIj4Ac8Y4z5sviGROQe4B6ARo0auSSs8lEOB3z2R/h5OvT8E1z3lFsWgcyz+fx90RY+3XiY9g1r8fLt7Wka6R7TWirPZ3dnsT/QHOgNRAMrRKStMSaj6ErGmEnAJID4+HhTyRmVt3IUwqcPwqaZcO1foPeTblkEVu5O5Yl5m0nLyuWxG+O4v7fOH6wqlisLwSGgYZHn0c7XikoB1hhj8oG9IrILqzCsc2EupawisPA+2DwHev8Vev/F7kQlZOcV8uIX25m6aj9NI6sx6a6raRddy+5Yygu5shCsA5qLSCxWARgOjCi2zkLgDuBDEYnAaipKdmEmpaCwAD65F7bMt5qCej1hd6ISNh7M4LE5G0lOO8O4HrH8uY/OH6xcx2WFwBhTICIPAkux2v8nG2O2ishzQKIxZpFz2U0isg0oBJ4wxqS7KpNSFObDggmw9ROrU/iaR+1OdJ78Qgevf5fEm98nUTc0iJl3d+XqZhF2x1JeTozxrCb3+Ph4k5iYaHcM5YkK82H+ONi+yLo8tMdDdic6T9Lx0zw6ZxO/HMpkUKconr61NTVDPGPSG+X+RGS9MSa+tGV2dxYrVTkK8mD+WNixGH7/T+j+gN2JznE4DFN+2se/vtxB1UA/3h7Zib5t69sdS/kQLQTK+xXkwtzRsOsL6PMv6PYHuxOdcygjmyfmbeKnPelcf1Ud/t/gttQJDbY7lvIxWgiUd8vPgbl3we6lcPN/IGGC3YkAa4iIT34+xNOfbsVhDC8OasuwLg11/mBlCy0Eynvl58CckZD0DdzyCsSPszsRACfO5PHXBb/w5dajdIkJ479DO9AoXOcPVvbRQqC8U342zLoDkpfBra9B59F2JwLg2+3H+MvHv3AqO5+Jfa9iQs8m+OkQEcpmWgiU98k7C7OGw94VMOAN6DjK7kRk5RbwwuJtzF53kKvqhTJtfAIt69ewO5ZSgBYC5W3yzsDMYbDvBxj4NnS4w+5ErNt3gsfmbiTlZDZ/uLYpj97YnCB/vTlMuQ8tBMp75GbBzNvhwCoYNAna3W5vnIJCXv56F5NWJNMwrCpz7+1OlxidP1i5Hy0EyjvknobpQyBlHQx6D9oOsTXO9iOneHTORnYcPc0dCQ35v36tqK7zBys3pX+ZyvPlnILpg+HQehjyAbS+zbYohQ7DpBXJvPz1TmqGBDJ5TDzXXVXXtjxKlYcWAuXZcjJh2iA4shGGfgitBtgW5UD6WR6bu5HE/Sfp26Ye/7itLbV16kjlAbQQKM+VfdIqAkd/gaFToeUttkVZnZzO+CnrqFJFeGVYewZ2iNKbw5TH0EKgPNPZEzBtIBzfDsOmQYu+tkXZefQ0Ez5KpH6tED4al0CDWjp/sPIsWgiU5zl7Aj7qD6k7YdgMiLvJtihHM3MY8+FaQgL8mDK2ixYB5ZG0ECjPciYNPhoAabth+CxofoNtUU7l5DPmw7Wczilgzr3diA7TYSKUZ9JCoDxHVqp1JnAiGUbMhqbX2RYlr8DBfdPXk3Q8iw/HdqF1g5q2ZVHqSmkhUJ4h6zhMvRVO7ocRc6BJb9uiGGP4y8eb+TEpnf8ObU/P5pG2ZVGqImghUO7v9FGrCGSmwMh5ENvT1jgvLd3JJz8f4k83xTG4c7StWZSqCFoIlHs7ddgqAqeOwMj5ENPD1jjTVu/nrWV7GNG1EQ/8rpmtWZSqKFoIlPvKPARTb7GahUZ9DI272xrnq61HefrTLVx/VR2e699a7xNQXkMLgXJPGQetInAmHUYtgEZdbY2z4cBJHpr9M22javL6iI74+1WxNY9SFUkLgXI/GQdgyi3WncN3LYToeFvj7E07w91TE6lbI5gPxnShaqB+bJR30b9o5V5O7oMpt0JuplUEojrbGictK5cxH64FYOrYBCKqB9maRylX0EKg3MeJZKsI5GXBXZ9Cg462xjmbV8D4Kes4diqHWRO6ERNRzdY8SrmKFgLlHtL3WFcH5Z+F0Yugfntb4xQUOvjjzJ/55VAm794ZT8dGYbbmUcqVtBAo+6XttopAYR6M/gzqtbU1jjGGv326lW93HOeFgW24sZXOJ6C8mxYCZa/UXdbVQY5CGL0Y6rayOxFvfp/ErLUHuL93U0Z1a2x3HKVcTguBss/xHdaZAMCYz6HOVfbmAeavT+E/X+3ito5RPPH7FnbHUapS6MXQyh7HtsGUfiDiNkVg5e5UJn68mR7NwvnX4HZ6w5jyGVoIVOU7+ovVHOQXYBWByDi7E7H1cCb3Td9AszrVeXtUZwL99aOhfIf+tavKdWST1RzkF2QVgYjmdici5eRZxn64jtBgf6aMTaBGcIDdkZSqVFoIVOU5/DNM7Q8B1WDs5xDe1O5EZJ7NZ8yH68jOL2TquATq1Qy2O5JSlU4Lgaoch9ZbM4sF1bCKQO0mdiciJ7+QCdMSOZB+lkl3xhNXN9TuSErZQguBcr2URPhoIATXgjGLISzG5kDgcBgen7eJtXtP8J/b29O9abjdkZSyjRYC5VoH1lhFoGptq08gzD2uy//nku18vvkIf735Kvq3b2B3HKVsdVmFQESqV3QQ5YX2r4Lpg6B6JIxZArUa2p0IgA9+2Mv7P+xlzNUxTOhpfxOVUna73DOCbRWaQnmffT/C9MEQWs86E6gZZXciAJb8coQXPt9Gn9b1+NstrfReAaW4wJ3FIvJYWYsAPSNQZdu7EmbeDjWjrbGDQuvZnQiAtXtP8MicjXRqFMb/hnfAr4oWAaXgwmcE/wTCgNBi/6pf5OfOEZE+IrJTRJJEZOIF1hssIkZE7J2BRF255GUwYyjUamSNHeQmRSDp+GkmfJRIdFgI798VT3CAn92RlHIbFxpraAOw0BizvvgCEbn7YhsWET/gTeBGIAVYJyKLjDHbiq0XCjwMrLmU4MoNJX0Ls0dYl4betcjqG3ADx0/lMHryOgL8qjB1bAJh1QLtjqSUW7nQN/tDwH4RebiUZeX55p4AJBljko0xecBsYEAp6z0P/AvIKcc2lbva/Q3MugPCm1nNQW5SBLJyCxg7ZR0nz+bx4ZguNKxd1e5ISrmdCxWCVkAgME5EwkSk9q//gPxybDsKOFjkeYrztXNEpBPQ0Bjz+YU2JCL3iEiiiCSmpqaW461Vpdr1Fcy+wxozaPRnUC3C7kQA5Bc6uG/6enYcPc2bIzvRNrqm3ZGUcksXahp6F/gWaAKsx+ok/pVxvn7ZRKQK8DIw5mLrGmMmAZMA4uPjzZW8r6pgO7+AOXda8wjcudC6X8ANGGN4csEvrNydxr8Ht+N3LerYHUkpt1XmGYEx5jVjTEtgsjGmiTEmtsi/8hSBQ0DRC8ejna/9KhRoAywTkX1AN2CRdhh7kO2LrSJQr601x7CbFAGAV77exfz1KTxyQ3Nu7+Ie9y8o5a4uevWPMea+y9z2OqC5iMSKSCAwHFhUZLuZxpgIY0yMMSYGWA30N8YkXub7qcq07VOYN9qaW/jOTyDEfeb0nbX2AK99l8Sw+IY8fL39o5sq5e5cNsSEMaYAeBBYCmwH5hpjtorIcyLS31XvqyrB1k9g3lho0MlZBGrZneic73Yc46mFW+jdIpIXbmujN4wpVQ4unarSGLMEWFLstb+XsW5vV2ZRFeSX+bDgHmiYACPnQZD7jNi5OSWDB2b8TMv6obw5ohMBfjqUllLloZ8UVX6b58KCCdCoG4yc71ZF4ED6WcZNWUd49UAmj+lCtSCdjlup8tJCoMpn4yzrTKBxD+eZgPuMMnLiTB6jP1xLgcMwdVwCdUJ1chmlLoUWAnVxP0+HhfdBbC8YMRcCq9md6JzsvELGT13H4YxsPhgdT9NI9ylQSnkKLQTqwtZPhU8fgKa/gxFzINB97swtdBgenv0zGw9m8OrwDnRu7D6XryrlSbQQqLIlTobPHoJmN8DwWRAQYneic4wxPLNoK19tO8bTt7SiT5v6dkdSymNpIVCl2/ARLH4Umt8Ew2ZAgHu1u7+7Iplpq/dzb68mjOkRa3ccpTyaXlqhStr+GXz2MDS9HoZNB/8guxOd59ONh3jxix3c2r4Bf+lzld1xlPJ4ekagzrd3Jcwfb90sNmya2xWBn5LS+NO8TXSNrc1/hrajik4uo9QV00KgfnNkkzWUdFiMdYmoG10dBLDj6Cnunbae2IhqTLorniB/nVxGqYqghUBZ0vdYcwwH14Q7F7jVAHIARzKzGTN5HVWD/JgyNoGaIQF2R1LKa2ghUHD6KEy7DRyF1thBNaPtTnSezOx8xkxeR1ZuAVPGJtCglvtcvaSUN9DOYl+XnWGdCZxJsyaViYyzO9F5cgsKuXdaIslpWUwZm0DL+jXsjqSU19FC4Mvys60+gdSdMHIuRHe2O9F5HA7DE/M2szr5BP8b1oEezdxj5jOlvI0WAl9VWADzx8GBVTDkA2h6nd2JSvj30p0s2nSYP/dpwcCOURf/AaXUZdFC4IuMse4Y3rkEbv4PtBlsd6ISPlq1j3eW72FUt0bcd21Tu+Mo5dW0s9gXffM0bJwB106EhAl2pynhyy1HeXrRVm5sVZdn++vkMkq5mhYCX/Pja/DjqxA/HnpPtDtNCev3n+Th2T/TProWrw3viJ/eMKaUy2kh8CUbZ8LXf4PWt8HNL4GbfdNOTs3i7qnrqF8zmA9GxxMSqDeMKVUZtBD4ip1fwKcPQpPecNu7UMW9DrKpp3MZ/eFaqogwdVwC4dXda2gLpbyZdhb7gv2rYN4YqN/OLQeRO5NbwLgp60g7ncfse7rRONy9hrZQytvpGYG3O7oFZg6z7hZ2s3mGAQoKHTw4cwNbD2fyxoiOtG9Yy+5ISvkcPSPwZif3wfRB1uBxd34C1dzrhixjDE8t3ML3O1P5521tub5lXbsjKeWTtBB4q6zj1vhBBbkw7kuo1cjuRCW8/l0Ss9cd5I/XNWNEV/fLp5Sv0ELgjXJOWeMHnToCoxdBnZZ2JyphXuJBXv56F4M6RfHYje41vpFSvkYLgbfJz4HZI+D4NrhjNjRMsDtRCct3pfLkgl/o2TyCFwe10xvGlLKZFgJv4iiEj8fDvpUw6D1ofqPdiUrYciiT+6evJ65uKG+N7ESgv16voJTd9FPoLYyxJpvfsRj6vAjtbrc7UQkHT5xl7JR11KoayIdjuxAarJPLKOUO9IzAW3z3PGyYCj0fh2732Z2mhIyzeYz5cC25+YXMvLsrdWsE2x1JKeWkhcAbrHoLVv4XOo2G6/5md5oScvILuXtqIgdPZDNtfALN67rXvQxK+TotBJ5u81xY+iS0vBVuecXtxg9yOAyPztlI4v6TvDmiE12bhNsdSSlVjPYReLLdX8PC+yCmJwx63+3GDwJ44fPtfLHlKE/1a0m/dvXtjqOUKoUWAk91cC3MuRPqtILhMyHA/drc31+ZzOQf9zKuRyx392xidxylVBm0EHii49thxlCoUR9GfQzB7jeh+2ebDvPC59vp17Y+T/VzvxvalFK/0ULgaTIOwLRB1giid34C1evYnaiENcnpPD53E11iwvjv7e2popPLKOXWtLPYk5xJs8YPyjsDY5dAWIzdiUrYdew0Ez5KpGHtEN67K57gAPfrt1BKnU8LgafIPQ0zhkBmCty5EOq1sTtRCcdO5TBm8lqCAvyYMjaBWlUD7Y6klCoHLQSeoCAXZo+EI5th+Axo3N3uRCWczslnzIfryMzOZ8693WlYu6rdkZRS5aSFwN05CmHBPbB3OQx8G1r0tTtRCXkFDu6bvoHdx04zeUwX2kTVtDuSUuoSuLSzWET6iMhOEUkSkYmlLH9MRLaJyGYR+VZEGrsyj8cxBpY8AdsWwo3PQ4cRdicqwRjDxAWb+SEpjf83qC294iLtjqSUukQuKwQi4ge8CfQFWgF3iEirYqv9DMQbY9oB84F/uyqPR1r2IiR+AD0ehh4P2Z2mVP/9ahcLNhzi8RvjGBrf0O44SqnL4MozggQgyRiTbIzJA2YDA4quYIz53hhz1vl0NRDtwjyeZe17sPxF6DAKbnjW7jSlmrFmP298n8QdCQ158LpmdsdRSl0mVxaCKOBgkecpztfKMh74woV5PMeWj60moRY3w62vut34QQDfbDvG3xZu4XctInl+QBudXEYpD+YWncUiMgqIB64tY/k9wD0AjRp5+dy2Sd/CgnuhUXcYMhn83OJ/0Xk2HszgwVkbaBNVkzdGdMLfT+9LVMqTufITfAgo2mgc7XztPCJyA/B/QH9jTG5pGzLGTDLGxBtj4iMjvbgzMmW9NX5QZAu4YxYEhNidqIR9aWcYP2UddUKD+WB0F6oFuV+hUkpdGlcWgnVAcxGJFZFAYDiwqOgKItIReBerCBx3YRb3l7rLumGseqQ1flBILbsTlZCelcuYD9fiMIYpY7sQGRpkdySlVAVwWSEwxhQADwJLge3AXGPMVhF5TkT6O1d7CagOzBORjSKyqIzNebfMFGvoiCr+1vhBofXsTlRCdl4h46cmciQzh/dHd6FJZHW7IymlKohLz+uNMUuAJcVe+3uRxze48v09wtkT1iByuadgzGKo7X7DNRcUOvjjrA1sTsngnVGd6dw4zO5ISqkKpA28dso7Yw0nfXKf1RxUv73diUowxvDMZ1v5ZvtxnhvQmptau9/ZilLqymghsEtBntUxfHgD3D4NYnvanahUby3bw/TVB/jDtU25q3uM3XGUUi6ghcAODoc1xeSeb6H/69DyFrsTlWrBhhReWrqTgR0a8Offt7A7jlLKRbQQVDZj4MuJsGU+XP80dLrL7kQlZJzN46NV+3nt291c3TScfw/RyWWU8mZaCCrbyv/A2neh2wNwzaN2pznPkcxs3l+5l1lrD3A2r5AbWtbl5WHtCfTXG8aU8mZaCCpT4mT47gVoNxxuesFtho5IOn6ad5cns3DjIRwG+rdvwL3XNuGqeu43F7JSquJpIagsWxfC4seg+U0w4A2oYv+37A0HTvLOsj18te0YwQFVGNm1MeOvidVJZZTyMVoIKkPyclgwARomwNCp4BdgWxRjDMt3pfL2sj2s2XuCmiEBPHRdM0ZfHUN4db1TWClfpIXA1Q7/DLNHQHgzGDEHAu35tl1Q6ODzX47wzvJkth85Rb0awTzVryV3JDTS8YKU8nF6BHCltCSYPgRCajvHD6r8O3Jz8guZtz6FSSv2cPBENk0jq/HSkHYM6BClncBKKUALgeucOmKNH4Sxxg+q0aBS3z4zO5/pq/fz4Y97ScvKo0PDWjzVrxU3tqyrl4Iqpc6jhcAVsk/C9EGQfcIaPyii8mbvOnYqhw9+2MvMNQfIyi3g2rhI7uvdlK6xtXXyGKVUqbQQVLS8szBzOKQnwcj50KBjpbxtcmoWk1Yks2DDIQocDm5pZ10C2rpBzUp5f6WU59JCUJEK82HeGDi4BoZOgSalTrhWoTYdzOCd5Xv4cutRAv2qMKxLQyb0bEKjcL0EVClVPloIKorDAZ8+CLuXwi2vQOuBLnsrYww/JKXxzvI9/JiUTmiwP/f3bsqYq2N1shil1CXTQlARjIGv/wabZ8PvnoL4cS55m0KH4YstR3hn+R62HDpFndAg/nrzVdyR0IjQYPvuTVBKeTYtBBXhx//Bqjcg4V7o9acK33xOfiELNhxi0oo97Es/S5OIavxrcFsGdowiyN+vwt9PKeVbtBBcqQ0fwTfPQJsh0OfFCh0/6FROPjNWH2Dyj3tJPZ1Lu+iavD2yEze1roefXgKqlKogWgiuxPbF8NnD0PR6GPh2hY0fdPx0DpN/2MeM1fs5nVtAz+YRvDqsA92bhusloEqpCqeF4HLt+wHmj4MGnWDYNPAPvPJNpp1h0spk5q9PoaDQQd+29bnv2qa0idJLQJVSrqOF4HIc2Qyz7oCwGBg5DwKrXdHmthzK5O3le/jilyP4V6nCkPho7unZhJiIK9uuUkqVhxaCS3UiGaYPhqAacOcCqFr7sjZjjGHVnnTeXr6HlbvTCA3y555eTRnXI4Y6NYIrOLRSSpVNC8GlOH3MGj/IUQBjPoea0Ze8iUKH4ettR3l72R42pWQSUT2Iv/S5ipHdGlFDLwFVStlAC0F5ZWdYZwJZqTD6M4iMu6Qfzy0oZOHPh3h3eTLJaWdoHF6Vf97WlkGdoggO0EtAlVL20UJQHvnZ1pwCqTusOQWiO5f7R0/n5DNr7QE++GEvx07l0iaqBm+M6EjfNvX1ElCllFvQQnAxhQUwfzzs/wmGfADNri/Xj6Vl5fLhj3uZtmo/p3IKuLppOP8Z2p5rmkXoJaBKKbeiheBCjIHFD8POz6HvS9Bm8EV/5ED6Wd5bmczcxIPkFTro07oef7i2Ke0b1nJ9XqWUugxaCC7km2fg5+lw7V+g6z0XXHXb4VO8s3wPizcfxq+KMLhTNBN6NaFpZPXKyaqUUpdJC0FZfnrdGkMofjz0frLUVYwxrNl7gneW72HZzlSqBfpxd88mjOsRS72aegmoUsozaCEozcZZ8NVT0Gog3PxSifGDHA7D19uP8c7yPfx8IIPwaoE88fsWjOramJpV9RJQpZRn0UJQ3M4v4dMHIPZaGDQJqvx2aWdegYNPNx7ineV72JN6hoa1Q3h+YBuGdo7WS0CVUh5LC0FR+1fBvNFQvx0MnwH+1iQvZ3ILzl0CeiQzh5b1a/Dq8A70a1sff7+KGWhOKaXsooXgV8e2wqxh1t3CI+dDUCjpWblM/WkfU1ftJzM7n66xtfl/g9pybVykXgKqlPIaWggATu6HaYMgoCrc+QkpeVV5/9utzF53gJx8Bze1qssfejelU6Mwu5MqpVSF00KQlQrTBkJBDsn95/P60hMs2rQFAW7rGMW91zahWZ1Qu1MqpZTL+HYhyDkFMwZTmHmYFyP/zXsfHadqoB9jro5h/DWxNKgVYndCpZRyOZ8tBI68bDImD6XG8S3cnfc4m1Mb8NiNMdzVvTG1ql75JDNKKeUpfK4Q5Bc6+GzjQSK/uJeeBat5NuARfnfTSN6Ob0hIoF4CqpTyPT5TCM7mFTBn3UHeX5HMA2feoKf/Kja3mchfb/sLAXoJqFLKh7n0CCgifURkp4gkicjEUpYHicgc5/I1IhLjqixvL9vDs59t41H/eYzw/w5zzeO0G/KkFgGllM9z2VFQRPyAN4G+QCvgDhFpVWy18cBJY0wz4BXgX67Kc2f3xqzotYMhZ2ZBp7uQ6//mqrdSSimP4sqvwwlAkjEm2RiTB8wGBhRbZwAw1fl4PnC9uOhOrTp7P6PR2ufgqlug3yslxg9SSilf5cpCEAUcLPI8xflaqesYYwqATCC8+IZE5B4RSRSRxNTU1MtLU6M+tOgHgz8AP5/pGlFKqYvyiCOiMWYSMAkgPj7eXNZGYq6x/imllDqPK88IDgENizyPdr5W6joi4g/UBNJdmEkppVQxriwE64DmIhIrIoHAcGBRsXUWAaOdj4cA3xljLu8bv1JKqcvisqYhY0yBiDwILAX8gMnGmK0i8hyQaIxZBHwATBORJOAEVrFQSilViVzaR2CMWQIsKfba34s8zgGGujKDUkqpC9O7qZRSysdpIVBKKR+nhUAppXycFgKllPJx4mlXa4pIKrD/Mn88AkirwDgVRXNdGs116dw1m+a6NFeSq7ExJrK0BR5XCK6EiCQaY+LtzlGc5ro0muvSuWs2zXVpXJVLm4aUUsrHaSFQSikf52uFYJLdAcqguS6N5rp07ppNc10al+TyqT4CpZRSJfnaGYFSSqlitBAopZSP88pCICJ9RGSniCSJyMRSlgeJyBzn8jUiEuMmucaISKqIbHT+u7uSck0WkeMisqWM5SIirzlzbxaRTm6Sq7eIZBbZX38vbb0KztRQRL4XkW0islVEHi5lnUrfX+XMZcf+ChaRtSKyyZnr2VLWqfTPYzlz2fJ5dL63n4j8LCKLS1lW8fvLGONV/7CGvN4DNAECgU1Aq2Lr3A+843w8HJjjJrnGAG/YsM96AZ2ALWUsvxn4AhCgG7DGTXL1BhZX8r6qD3RyPg4FdpXy/7HS91c5c9mxvwSo7nwcAKwBuhVbx47PY3ly2fJ5dL73Y8DM0v5/uWJ/eeMZQQKQZIxJNsbkAbOBAcXWGQBMdT6eD1wv4vLZ7MuTyxbGmBVY80GUZQDwkbGsBmqJSH03yFXpjDFHjDEbnI9PA9spORd3pe+vcuaqdM59kOV8GuD8V/wKlUr/PJYzly1EJBroB7xfxioVvr+8sRBEAQeLPE+h5Afi3DrGmAIgEwh3g1wAg53NCfNFpGEpy+1Q3ux26O48vf9CRFpX5hs7T8k7Yn2bLMrW/XWBXGDD/nI2c2wEjgNfG2PK3F+V+HksTy6w5/P4P+DPgKOM5RW+v7yxEHiyz4AYY0w74Gt+q/qqdBuwxk9pD7wOLKysNxaR6sDHwCPGmFOV9b4Xc5FctuwvY0yhMaYD1rzlCSLSpjLe92LKkavSP48icgtw3Biz3tXvVZQ3FoJDQNHKHe18rdR1RMQfqAmk253LGJNujMl1Pn0f6OziTOVVnn1a6Ywxp349vTfWbHgBIhLh6vcVkQCsg+0MY8yCUlaxZX9dLJdd+6vI+2cA3wN9ii2y4/N40Vw2fR57AP1FZB9W8/F1IjK92DoVvr+8sRCsA5qLSKyIBGJ1piwqts4iYLTz8RDgO+PsebEzV7F25P5Y7bzuYBFwl/NqmG5ApjHmiN2hRKTer22jIpKA9ffs0gOI8/0+ALYbY14uY7VK31/lyWXT/ooUkVrOxyHAjcCOYqtV+uexPLns+DwaY540xkQbY2KwjhHfGWNGFVutwveXS+cstoMxpkBEHgSWYl2pM9kYs1VEngMSjTGLsD4w00QkCaszcrib5HpIRPoDBc5cY1ydC0BEZmFdURIhIinA01idZxhj3sGad/pmIAk4C4x1k1xDgPtEpADIBoZXQkHvAdwJ/OJsXwb4K9CoSC479ld5ctmxv+oDU0XED6vwzDXGLLb781jOXLZ8Hkvj6v2lQ0wopZSP88amIaWUUpdAC4FSSvk4LQRKKeXjtBAopZSP00KglFI+TguBUpVIrBFAS4woqZSdtBAopZSP00KgVClEZJRzvPqNIvKuc4CyLBF5xTl+/bciEulct4OIrHYOTvaJiIQ5X28mIt84B3nbICJNnZuv7hzEbIeIzHD1SJtKXYwWAqWKEZGWwDCgh3NQskJgJFAN6+7O1sByrDudAT4C/uIcnOyXIq/PAN50DvJ2NfDrMBMdgUeAVljzU/Rw8a+k1AV53RATSlWA67EGGFvn/LIegjVUsQOY41xnOrBARGoCtYwxy52vTwXmiUgoEGWM+QTAGJMD4NzeWmNMivP5RiAG+MHlv5VSZdBCoFRJAkw1xjx53osifyu23uWOz5Jb5HEh+jlUNtOmIaVK+hYYIiJ1AESktog0xvq8DHGuMwL4wRiTCZwUkZ7O1+8EljtnCUsRkYHObQSJSNXK/CWUKi/9JqJUMcaYbSLyFPCViFQB8oEHgDNYE5g8hdVUNMz5I6OBd5wH+mR+G230TuBd58iR+cDQSvw1lCo3HX1UqXISkSxjTHW7cyhV0bRpSCmlfJyeESillI/TMwKllPJxWgiUUsrHaSFQSikfp4VAKaV8nBYCpZTycf8fHpgXk6D9a4sAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(model.history.history.keys())\n",
    "# summarize history for accuracy\n",
    "plt.plot(model.history.history['f1'])\n",
    "plt.plot(model.history.history['val_f1'])\n",
    "plt.title('model f1')\n",
    "plt.ylabel('f1')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'val'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "До этого мы разбирали сами слои и как-то не обращали внимание на результаты. А они получаются так себе. Видно, что есть переобучение."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Для борьбы с переобучением в нейронках есть регуляризация и Dropout. \n",
    "\n",
    "    Базовая регуляризация тут как в классических алгоритмах - l1 и l2. В большинстве слоев есть параметры smth_regularizer (можно строкой указать l1 или l2), где вместо smth какая-то специфичная часть слоя, которая обучается (kernel_regularizer в свертках  это регуляризация весов в матрицах).\n",
    "    \n",
    "    А Dropout - это отдельный слой, в котором часть подаваемых значений просто случайно зануляется. Это застявляет нейронку смотреть  на большее количество паттернов, а не переобучаться только на какой-то один.\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs = tf.keras.layers.Input(shape=(MAX_LEN,))\n",
    "\n",
    "embeddings = tf.keras.layers.Embedding(input_dim=len(word2id), output_dim=100)(inputs)\n",
    "drop1 = tf.keras.layers.Dropout(0.1)(embeddings)\n",
    "conv1 = tf.keras.layers.Conv1D(kernel_size=5, filters=32, strides=1)(embeddings)\n",
    "conv2 = tf.keras.layers.Conv1D(kernel_size=5, filters=32, strides=1, \n",
    "                                               kernel_regularizer='l2',\n",
    "                                              activation='relu')(conv1)\n",
    "\n",
    "drop2 = tf.keras.layers.Dropout(0.1)(conv2) # это значит что случайно заменяется половина всех значений\n",
    "\n",
    "concat = tf.keras.layers.Flatten()(drop2)\n",
    "dense = tf.keras.layers.Dense(50, activation='relu', kernel_regularizer='l2')(concat)\n",
    "outputs = tf.keras.layers.Dense(len(label2id), activation='softmax')(dense)\n",
    "\n",
    "model = tf.keras.Model(inputs=inputs, outputs=outputs)\n",
    "optimizer = tf.keras.optimizers.Adam(learning_rate=0.01)\n",
    "model.compile(optimizer=optimizer,\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=[f1])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_15\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_20 (InputLayer)        [(None, 201)]             0         \n",
      "_________________________________________________________________\n",
      "embedding_18 (Embedding)     (None, 201, 100)          5530600   \n",
      "_________________________________________________________________\n",
      "conv1d_22 (Conv1D)           (None, 197, 32)           16032     \n",
      "_________________________________________________________________\n",
      "conv1d_23 (Conv1D)           (None, 193, 32)           5152      \n",
      "_________________________________________________________________\n",
      "dropout_11 (Dropout)         (None, 193, 32)           0         \n",
      "_________________________________________________________________\n",
      "flatten_15 (Flatten)         (None, 6176)              0         \n",
      "_________________________________________________________________\n",
      "dense_30 (Dense)             (None, 50)                308850    \n",
      "_________________________________________________________________\n",
      "dense_31 (Dense)             (None, 19)                969       \n",
      "=================================================================\n",
      "Total params: 5,861,603\n",
      "Trainable params: 5,861,603\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "71/71 [==============================] - 14s 178ms/step - loss: 0.5320 - f1: 0.0476 - val_loss: 0.1560 - val_f1: 0.2912\n",
      "Epoch 2/10\n",
      "71/71 [==============================] - 12s 168ms/step - loss: 0.1432 - f1: 0.4668 - val_loss: 0.2562 - val_f1: 0.4885\n",
      "Epoch 3/10\n",
      "71/71 [==============================] - 12s 170ms/step - loss: 0.1639 - f1: 0.6210 - val_loss: 0.1199 - val_f1: 0.6566\n",
      "Epoch 4/10\n",
      "71/71 [==============================] - 12s 170ms/step - loss: 0.1014 - f1: 0.7104 - val_loss: 0.2223 - val_f1: 0.6487\n",
      "Epoch 5/10\n",
      "71/71 [==============================] - 12s 169ms/step - loss: 0.1570 - f1: 0.6927 - val_loss: 0.1100 - val_f1: 0.6638\n",
      "Epoch 6/10\n",
      "71/71 [==============================] - 12s 169ms/step - loss: 0.0919 - f1: 0.7388 - val_loss: 0.0952 - val_f1: 0.6980\n",
      "Epoch 7/10\n",
      "71/71 [==============================] - 12s 168ms/step - loss: 0.0781 - f1: 0.7756 - val_loss: 0.1007 - val_f1: 0.6958\n",
      "Epoch 8/10\n",
      "71/71 [==============================] - 12s 169ms/step - loss: 0.0906 - f1: 0.7638 - val_loss: 0.1093 - val_f1: 0.7012\n",
      "Epoch 9/10\n",
      "71/71 [==============================] - 12s 171ms/step - loss: 0.0774 - f1: 0.7947 - val_loss: 0.0991 - val_f1: 0.7018\n",
      "Epoch 10/10\n",
      "71/71 [==============================] - 12s 170ms/step - loss: 0.0668 - f1: 0.8180 - val_loss: 0.0962 - val_f1: 0.7048\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f700dc8b710>"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train, y_train, \n",
    "          validation_data=(X_valid, y_valid),\n",
    "          batch_size=2000,\n",
    "         epochs=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Переобучаться стало сильно меньше."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['loss', 'f1', 'val_loss', 'val_f1'])\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAv5UlEQVR4nO3deXxU9bn48c+Tyb6QhGxAEkjCjoCgEag7Wy+4ti51v7VVaetubW/1tr/a2uXazd7Wq3WrVVutC9UWK3WBAC6AEpAiS4CALAlLQiAbIdvM8/vjTMIQE9YMJ5l53q/XvHLmnO/MPBnI9znne875PqKqGGOMCV8RbgdgjDHGXZYIjDEmzFkiMMaYMGeJwBhjwpwlAmOMCXOWCIwxJsxZIjDmBInIsyLy06Nsu0VEpnWxLU5E3hCRGhF5tXujNKZrlgiM6TmuALKANFW9UkT6i8gcEdkhIioieS7HZ0KUJQJjeo5BwAZVbfU/9wFvAZe7F5IJB5YITFjwD8l8V0RWich+EfmjiGSJyL9EpE5E5olIakD7S0RkjYhUi8hCERkZsG28iKzwv+5lILbDZ10kIiv9r10sImOPIr4fAz8ErhKRehG5SVV3q+pjwLLu+yaM+TxLBCacXA5MB4YBFwP/Av4byMD5W7gTQESGAX8F7vZvmwu8ISLRIhIN/B34M9AXeJWAPXYRGQ88A3wDSAOeAOaISMzhAlPVB4CfAy+raqKq/rFbfmNjjoIlAhNOHvHvZZcD7wMfqeonqtoIvA6M97e7CnhTVd9V1Rbg10AccCYwCYgC/ldVW1R1Nofusc8CnlDVj1TVq6rPAU3+1xnTI0W6HYAxJ9HugOUDnTxP9C8PALa2bVBVn4hsB7IBL1Cuh87WuDVgeRDwVRG5I2BdtP89jemRLBEY83k7gDFtT0REgFygHFAgW0QkIBkMBDb5l7cDP1PVn53EeI05ITY0ZMznvQJcKCJTRSQKuBdneGcxsARoBe4UkSgRuQyYEPDap4BvishEcSSIyIUiknQ8gYhILNB2fiHG/9yYbmWJwJgOVHU9cD3wCLAH58TyxararKrNwGXAjcBenPMJrwW8thi4Bfg/YB9Q6m97vA4A9f7lEv9zY7qVWGEaY4wJb3ZEYIwxYc4SgTHGhDlLBMYYE+YsERhjTJjrdfcRpKena15entthGGNMr7J8+fI9qprR2bZelwjy8vIoLi52OwxjjOlVRGRrV9tsaMgYY8KcJQJjjAlzlgiMMSbM9bpzBJ1paWmhrKyMxsZGt0MJqtjYWHJycoiKinI7FGNMCAmJRFBWVkZSUhJ5eXk4E0WGHlWlqqqKsrIy8vPz3Q7HGBNCQmJoqLGxkbS0tJBNAgAiQlpaWsgf9RhjTr6QSARASCeBNuHwOxpjTr6QGBoyxphQdKDZy+Y99Wyu3M/myv1MHZnJ6Ozkbv+coCYCEZkB/A7wAE+r6kMdtg8EngNS/G3uU9W5wYwpGKqrq3nxxRe59dZbj+l1F1xwAS+++CIpKSnBCcwY0+P5fMrO2kY2VdSzubKezXv2+zv+enbUHBwKFoG0xOjelQhExAM8CkwHyoBlIjJHVdcGNPsB8Iqq/kFERgFzgbxgxRQs1dXVPPbYY59LBK2trURGdv0Vz53b63KeMeY41Te1Oh29v5Pf5O/wP9tTT2OLr71dUkwkBRkJTCxIoyA9gYKMRAZnJpCXlkBslCcosQXziGACUKqqmwFE5CXgUiAwESjQx7+cjFMrtte577772LRpE+PGjSMqKorY2FhSU1MpKSlhw4YNfOlLX2L79u00NjZy1113MWvWLODgdBn19fXMnDmTs88+m8WLF5Odnc0//vEP4uLiXP7NjDHHwutTyvcdYJN/OGdTZX17519R19TeLkIgt288BekJnDk4jcEZiRRkJFCQkUBGYsxJPx8YzESQjVPIu00ZMLFDmx8B74jIHUACMK2zNxKRWcAsgIEDBx72Q3/8xhrW7qg9voi7MGpAHx64+JQutz/00EOsXr2alStXsnDhQi688EJWr17dfpnnM888Q9++fTlw4ABnnHEGl19+OWlpaYe8x8aNG/nrX//KU089xVe+8hX+9re/cf3113fr72GM6R41DS3tnX37Xv6eerZUNdDcenDvPjkuisEZCZw7LMPp6NMTGZyRwMC0eGIig7N3fzzcPll8DfCsqv5GRL4A/FlERquqL7CRqj4JPAlQWFjY42trTpgw4ZBr/X//+9/z+uuvA7B9+3Y2btz4uUSQn5/PuHHjADj99NPZsmXLyQrXGNMFVWXZln2s3L6v/YTtpsp6qvY3t7eJjBAGpsVTkJ7I5OGZ/j37RArSE+ibEN0rrvYLZiIoB3IDnuf41wW6CZgBoKpLRCQWSAcqjvdDD7fnfrIkJCS0Ly9cuJB58+axZMkS4uPjOf/88zu9FyAmJqZ92ePxcOCA1Sg3xi2qysINlfx+/kY+2VYNQFpCNAUZCUwbmcXgTGfvviAjgdy+8UR5eveV+MFMBMuAoSKSj5MArgau7dBmGzAVeFZERgKxQGUQYwqKpKQk6urqOt1WU1NDamoq8fHxlJSUsHTp0pMcnTHmaKkq89ZV8EjRRlaV1ZCdEsdPvjSai8f2JyU+2u3wgiZoiUBVW0XkduBtnEtDn1HVNSLyIFCsqnOAe4GnROQenBPHN6pqjx/66SgtLY2zzjqL0aNHExcXR1ZWVvu2GTNm8PjjjzNy5EiGDx/OpEmTXIzUGNMZn095a80uHikqZd3OWgb2jecXl4/hy+NziI7s3Xv7R0N6W79bWFioHQvTrFu3jpEjR7oU0ckVTr+rMcHm9Sn/XLWDRxeUsmF3PQXpCdw2eQiXjhtAZC8f7ulIRJaramFn29w+WWyMMSddq9fHP1bu4NGFpWyu3M/QzER+d/U4Lho7AE9Ezz+5290sERhjwkZzq4/XPynj0QWb2La3gRH9knjsutOYcUo/IsIwAbSxRGBMmGtq9bJhVz2ryqv5tKyGVWU1NDS3cu6wDKaOzGJSQd8edc378Whq9fJqcRl/WLiJ8uoDjMlO5skbTmfayKywTgBtLBEYE0ZavD427K5zOvzyGj4tq6FkVy0tXudcYUp8FGOyk+nvieWV4u08v2QrCdEezhmawdSRmUwZkUlaYswRPqXnaGzx8tLH23h80WZ21TYyLjeFn35pNOcPz+gV1/efLJYIjAlRrV4fpZX1rCqrae/41+2sbb/zNSk2krE5ydx0dgFjspMZm5NMTmpcewfZ2OJlyaYq5q3bzfx1Fby1ZhciMD43hakjs5g2MothWYk9skNtaG7lxY+28cR7m6msa2JCXl9+deVYzh6S3iPjdZtdNdTLhNPvao6e16dsbuv0y53Hmh017ZOZJcZEMjq7D2OykxmTk8LY7GQGpcUfdaeoqqzZUcv8dRXML9nNqrIaAHJS45g2MospIzKZ2AOGkOqbWvnzkq08/f5mqvY3c+bgNO6cOpRJBWlHfnGIs6uGepjExETq6+vdDsP0Uj6f8lnVfj5t6/TLali9o4aGZi8A8dEeRg9I5toJgxibk8yYnGTy0xJOaCxcRBidnczo7GTumjaU3bWNFJVUMH/dbl5ato1nF28hIdrTfl5h8vCMkzqEVNvYwnMfbuGPH35GdUML5w7L4M4pQyjM63vSYujNLBEY04OpKlurGtr38leVVbO6vJb6plYAYiIjOGVAH75SmNs+vFOQkRj0SyCz+sRyzYSBXDNhII0tXj4s3cO8dRUUlezmX6udIaTTBqYydWQm00ZmMTQzOENI1Q3NPPPBZ/xp8RbqGluZOiKTO6YOZVxuSrd/ViizRNAN7rvvPnJzc7ntttsA+NGPfkRkZCQLFixg3759tLS08NOf/pRLL73U5UhNT1e2r4FV/it3PvVfxVPb6HT60ZERjOzfhy+Pz2ZMjtPpD8lIdP3Gp9goD1NHZjF1ZBaqo1mzo7b9vMIv31rPL99aT27fOKaOyGLqyEwm5qed8N26VfVNPP3BZzy/eAv7m738xylZ3DFlaFCKtoSD0DtH8K/7YNen3fuh/cbAzIe63PzJJ59w9913s2jRIgBGjRrF22+/TXJyMn369GHPnj1MmjSJjRs3IiInNDRk5whC0/Kt+/jtuxv4oHQPAFEeYUS/Pk6Hn+0M7wzLSup1k5vtrm10zius280HpXtoavWRGBPJucPSmToii8kjMumbcPRz+FTUNfLUe5v5y9JtNLZ6uXBMf26fMoQR/foc+cVhzs4RBNn48eOpqKhgx44dVFZWkpqaSr9+/bjnnnt47733iIiIoLy8nN27d9OvXz+3wzU9yMrt1fz23Q0s2lBJWkI0/zVjOGcPSWd4vyTXT7x2h6w+sVw7cSDXThzIgWZnCGl+iXO0MPfTXUS0DyFlMW1kJkO6GELaVdPI44s28dePt9Hi9XHpuGxumzyYIZlJLvxWoSf0EsFh9tyD6corr2T27Nns2rWLq666ihdeeIHKykqWL19OVFQUeXl5nU4/bcLT6vIafvvuBuaXVJAaH8V9M0fwn18YRHx06P1JtomL9jBtVBbTRmXh8+nBIaSS3fzirRJ+8VYJA/vGM2WEc15hQn5fKuqcBPDKsjK8qlw2PptbJw8hPz3hyB9ojlro/q87ya666ipuueUW9uzZw6JFi3jllVfIzMwkKiqKBQsWsHXrVrdDND3Aup21/PbdDbyzdjfJcVF89z+G89Uz80iMCa8/xYgIYYz/iqZ7pg9jV01j+5HCXz92rkJKiomksdW5EuqK03O59fzB5PaNdzny0BRe//uC6JRTTqGuro7s7Gz69+/Pddddx8UXX8yYMWMoLCxkxIgRbodoXLRhdx2/m7eRNz/dSVJsJPdMG8bXzs6jT2yU26H1CP2SY7lu4iCumziIA81ePijdQ1FJBfHRHm46O58BKVa/O5gsEXSjTz89eJI6PT2dJUuWdNrO7iEIH6UV9fx+/kbeWLWDhOhI7pwyhJvOLiA53hJAV+KiPUwflcX0UVlHbmy6hSUCY4Jgy579/H7+Rv6+spzYKA/fPG8ws84pIPUYrpAx5mSxRGBMN9q+t4Hfz9/Ia5+UE+URbj6ngFnnFpDeiyZqM+EnZBKBqob8ZFK97Z6PcFJefYD/Kyrl1eLtREQIX/1CHt88v4DMpFi3QzPmiEIiEcTGxlJVVUVaWlrIJgNVpaqqithY61h6kp01B3hswSZeWrYNQbhu4kC+df4Q+iXbv5PpPUIiEeTk5FBWVkZlZaXboQRVbGwsOTk5bodhgIraRh5buIkXP96GqvKVwlxumzzErm4xvVJQE4GIzAB+B3iAp1X1oQ7bfwtM9j+NBzJVNeVYPycqKor8/PwTjNaYI6usa+KJRZv489KttPqUK0/P4bbJQ+z6dnPifF5obQJvM3hbwBuw3Nrk/EzJhcTMbv/ooCUCEfEAjwLTgTJgmYjMUdW1bW1U9Z6A9ncA44MVjzEnYu/+Zp54bxPPL95KU6uXL4/P4c6pQxiUZne4dhtVaG2E5v0HH95mUB+gznZV/7LP/zxwm+/QbYGv6bRdZ6+h8/f2tQR0yIfpqL3N/vX+5UPWBzxamz//Puo78nd04cNwxk3d/tUH84hgAlCqqpsBROQl4FJgbRftrwEeCGI8xhyz6oZmnnp/M89+uIWGFi+XnjqAO6cOpSAjsfs/rGEv7FwJCHiiICIKPJEQEelfjvIvR3bYHnVw3ck4R9beYTdAcz20NPg77nr/uv3HsX4/tOw/us6wpxAPeKIhMtr52fERGfAzJsm/PgoiY5yfnmjwBCwfsr6T9/FEQ+aooPwqwUwE2cD2gOdlwMTOGorIICAfKOpi+yxgFsDAgQO7N0pjOlFzoIU/fvAZf/rgM+qaWrlobH/umjqUoVndOMmZzwe7/g0b58HGd6C8+MQ7QvF0njAiIg8mDU8URHgOTS7tr/EnF9WADjvg0eLv/I8lTk8MRCcc+oiKhz45/ufxEJ3orItOcJaj/cueaJAI54E4iU7Evxzx+eVD2nXXa8T5Xjp21BG9f1LANj3lZPHVwGxV9Xa2UVWfBJ4EZxrqkxmYCa499U0caPYSF+0hNspDbGSEq/Pr1zW28OyHW3jq/c3UNrYy45R+3D19aPdNc3xgH2wqcjr/0nmwv8JZP+A0OPe7MOgsp0P2tfqHI/w/fa0Hl73+575W/3Jn270By/7nbcve1oD397+2tfHQ94WDHXOf7IMdc1cddlTC5zv7tvWentLNmK4E81+oHMgNeJ7jX9eZq4HbghiL6UEONHt5a81OZi8vY/GmKjreHhHlEWIjPcRGe4iNiiAuyp8k/I+4qAh/0vAQF+0hJqCN8zMioG3n69vety3p7G9q5bklW3jyvc1UN7QwbWQWd0/rhkInqrBrFWx813mUfezsTcemwJCpMPSLMHgqJGac2OcYcwKCmQiWAUNFJB8nAVwNXNuxkYiMAFKBzifmMSFBVVm+dR+zl5fxz1U7qW9qJbdvHHdOGUp2ahxNLV4OtHhpbPH5f7Y9fBxo9tLY6jyvOdBCRa23vY2zzUdz6/ENqURGCHFRHlp8PhpbfEwensE904cxNifl+H/ZxhrYtMDp+EvnQf0uZ33/cXDOvTBkOmSfbnvKpscI2v9EVW0VkduBt3EuH31GVdeIyINAsarO8Te9GnhJ7bbZkLSj+gCvf1LO7OVlfLZnP/HRHi4Y058rTs9hQl7fEyqoHsjrU5paDyaGA80dkklb4mjx0tTJOlW4ZNwAThuYeuwfrgq71zjj/KXzYNtSUC/EJsPgKQf3+pNsEjXTM4VEqUrTszS2eHl7zS5mLy/jg9I9qMLE/L5ccXoOF4zpT0IozL3fWAubF/o7//lQt8NZ32+M0/EPmQ45Z9hev+kxrFSlCTpV5ZPt1cxeXsYb/95BXWMr2Slx3DFlKJeflt37r7dXhYp1AXv9S5yTqjF9YPBkp+MfMg369Hc7UmOOmSUCc0J21zby2opyZi/fzqbK/cRGRXDBaGfoZ1JBWrcN/biiqQ42L4JS/4neWv+1Dlmj4cw7nM4/d4JzSaExvZglAnPMGlu8zFu3m1eLy3h/YyU+hTPyUpl1bgEXjOlPUm+tuqUKlev9e/3vwtYlziWW0Ukw+Hw473vOXn9yttuRGtOtLBGYo6KqrCqr4dXl25mzcge1ja30T47l1vOHcPnpOb2rmLiqcz1/9Tao3ur83LPBudKnxn8PZOYomPQtZ7w/d6Jzd6cxIcoSgTmsirpGXl/hXPWzsaKemMgIZozux5Wn5/KFwWl4eurQT2ON08Hv83f0gZ3+vq3QXHdo+9hkyDvHubxz6HRItlleTfiwRBCuSudD2TLnxqa4FOdnbDLEpdAUlcSCrS28snIPizZU4vUppw1M4X8uG8OFY/v3jILrTfWHdu7V22DfloPLjdWHto9KgNRBkDLIuXs3dRCkDPQ/BjnfgTFhyhJBOCpbDi9c6Vzr3okYYAYwmShaEvoQndiX6Ji+sDEFticfTBxd/YxNdqYXOJEJ0JobnGGafVsDOvuATr+h6tD2kXFOp546yDmBmxLQ0afmQVzqyZmQzZheyBJBuGmqh9duhqT+8I1F7Klvpmjlej74dBPVeytI8xxgYn8PE/pHkBffQkxTNRyodvaw63ZCZYmz3FgLHOYelIio9iOMLpNFnP9nQ9Xnh3Ha5uBp44lx5mJPGeTcodu+R+/fy09It47emONkiSDcvH0/7P2MNV98kd/O3sLC9RW0+pRxuSO54pLpXDx2AMnxRzH04/NCU+3BJHGg2hmXb1/u8LNhL+zdfLBdx6ORiChnXD51EAyf4e/k8w7u5SdkQoR7k9EZE8osEYSTdW/AiufZMuIWLv0npCZUc9M5+VxxWs6xT68c4XGGW+KOc0qGpjr/kUUNxPWFpH4hNa2vMb2JJYJwUbsT5txBfd9TuHD1eYzs34cXbpnozolfEYjt4zyMMa6zRBAOfD74+7fwNR/gqv03kZ3Wh+e/PqFnXP1jjHGdJYJw8NHjsHkBP+Nm6pMKePWmiaQm2A1SxhiHJYJQt2s1+u4DvEchc6Nn8urNE8nsE+t2VMaYHsQSQShraaT51Zuo88XzE8+3eOGWSeSkxrsdlTGmh7Hr8ULY/rk/ILqqhP/Ht3jk5i9SkJHodkjGmB7IEkGIqlv9FgmfPMVfdAa3fP0bjOxvV+gYYzpniSAE1e3dScvfvskGzWHotQ8z/njKLxpjwoYlghDT0NTCuie+RqKvjuoZjzFxuM2db4w5vKAmAhGZISLrRaRURO7ros1XRGStiKwRkReDGU+oa2zx8tfHf8qEpiWUjr2XCV84z+2QjDG9QNCuGhIRD/AoMB0oA5aJyBxVXRvQZihwP3CWqu4TkcxgxRPqWrw+fvrcHP577x/YnTGJUV/uNO8aY8znBPOIYAJQqqqbVbUZeAm4tEObW4BHVXUfgKp2mHLSHA2vT/mvV5Zz5bYHiYiOJes//2QTtBljjlowe4tsYHvA8zL/ukDDgGEi8qGILBWRGZ29kYjMEpFiESmurKwMUri9k6ryg7+vZvCaRzg1YjOxX34E+gxwOyxjTC/i9m5jJDAUOB+4BnhKRFI6NlLVJ1W1UFULMzIyTm6EPZiq8vO569i07G1ui5wD46+HUR0Puowx5vCCeWdxOZAb8DzHvy5QGfCRqrYAn4nIBpzEsCyIcYWM383fyMvvr+a9pCchMQ9m/MLtkIwxvVAwjwiWAUNFJF9EooGrgTkd2vwd52gAEUnHGSraHMSYQsbT72/mf+dt5Nmsl0lu3YNc/jTE2J3DxphjF7REoKqtwO3A28A64BVVXSMiD4rIJf5mbwNVIrIWWAB8V1WrOn9H0+bFj7bx0zfX8cCg1ZxWMw85/z7IKXQ7LGNMLyWqh6k72wMVFhZqcXGx22G45h8ry7n75ZVcXuDlV3tuRTJHwY1zwWPzBxpjuiYiy1W10z1Gt08Wm2PwzppdfPuVfzNxUDK/jHgMUeCyJy0JGGNOiCWCXuKDjXu4/cVPGJ2dzHPDFxOxfQlc+GtIzXM7NGNML2eJoBco3rKXW54vpiAjgb/MiCTm/V/AKZfB2KvcDs0YEwJsTKGHW11ew9f+tIx+ybH8+YZTSHphOiT2g4sedorAG2PMCbJE0INt3F3HDX/8iD5xUfzl5olkvPc92LsZvvoGxNnU0saY7mFDQz3UtqoGrnv6IyI9Ebxw80Syd86HFc/BWXdB/jluh2eMCSGWCHqgnTUHuPbppTR7ffzlponkRdfCnDug/6kw+ftuh2eMCTGWCHqYPfVNXPf0R1Q3tPD81ycwPDMB/nErtByAy56GyGi3QzTGhBg7R9CD1DS0cMMfP2ZH9QGe//pExuakwNI/wKYiuPBhyBjmdojGmBBkRwQ9xP6mVm589mNKK+p44oZCJuT3hd1r4N0HYNhMKPy62yEaY0KUHRH0AI0tXm55vphVZTU8eu14zhuWAS2N8LebITYZLnnELhU1xgSNJQKXtXh93PbCChZvquLhr5zKjNH9nQ3zfwwVa+G62ZBoNRiMMcFjQ0Mu8vqUe15eyfySCn7ypdFcdlqOs6F0Pix9DCbMgqHT3Q3SGBPyLBG4xOdT7n9tFf9ctZP7Z47ghkmDnA37q+Dvt0LGCJj+oLtBGmPCgg0NuUBV+cmba3mluIw7pgzhG+cNbtsAb9wJB/bC9bMhKs7dQI0xYcGOCFzw8Lsb+NOHW/jaWXl8e3rAJaErnoeSf8LUH0K/Me4FaIwJK5YITrIXPtrKI0WlXFWYyw8vGoW0XQ20pxTeug/yz4NJt7kbpDEmrFgiOMmeX7yVU3NT+PllYw4mAW8LvHYLRMbAlx+HCPtnMcacPNbjnERl+xpYv7uOi8b0xxMRcF/Awodgxwq4+HfQZ4B7ARpjwpIlgpNoQUkFAJNHZB5cuXUJfPAwjLseRl3qUmTGmHAW1EQgIjNEZL2IlIrIfZ1sv1FEKkVkpf9xczDjcVtRSQUD+8YzOCPBWdFYA6/NgpRBMPMhd4MzxoStoF0+KiIe4FFgOlAGLBOROaq6tkPTl1X19mDF0VMcaPayeFMV10wYePDcwJvfgdpyuOkdiElyN0BjTNgK5hHBBKBUVTerajPwEhC2Yx+LN+2hqdXHlLZhoU9nw6evwHnfg5xCd4MzxoS1YCaCbGB7wPMy/7qOLheRVSIyW0RyO3sjEZklIsUiUlxZWRmMWIOuqKSC+GgPEwv6QvU2+Oe3IXcinHOv26EZY8Kc2yeL3wDyVHUs8C7wXGeNVPVJVS1U1cKMjN43AZuqsqCkgrOHpBMTAbz2DVAfXPYkeOzmbmOMu4KZCMqBwD38HP+6dqpapapN/qdPA6cHMR7XlOyqY0dNozMstPgR2LYYLvgVpOa5HZoxxgQ1ESwDhopIvohEA1cDcwIbiEj/gKeXAOuCGI9rivyXjU7t3+jcMzDiIjj1apejMsYYR9DGJVS1VURuB94GPMAzqrpGRB4EilV1DnCniFwCtAJ7gRuDFY+bFpRUMDq7Dxkf/AgkAmb+wgrNGGN6jKAOUKvqXGBuh3U/DFi+H7g/mDG4bd/+ZlZs28dvTt0FJW/CtB9Dco7bYRljTDu3TxaHvEUbKonSZi4o/19IHw6TbnU7JGOMOYRdshJkRSUV3BM3l5i6bfDlORAZ7XZIxhhziOM6IhCRxO4OJBS1en2Url/FTfwdRl8OBee5HZIxxnzO8Q4NdZwmwnRixdZ9fMf7DOKJhC/+zO1wjDGmU10ODYnIt7vaBNgRwVHYvmQ2l3tW0njug0T26X/kFxhjjAsOd0TwcyAVSOrwSDzC6wxAcwNnlf6abZF5xJ5lJ4iNMT3X4U4WrwD+rqrLO24I9emiu0Ptuw/RTyt4Y8zTDPREuR2OMcZ06XB79uXAVhG5q5NtNl3m4ewpJaH4Mf7mPZtRZ850OxpjjDmswyWCUUA08HURSRWRvm0PoOXkhNcLqcK/vksTUfw56WYK0hPcjsgYYw7rcENDTwDzgQJgOc5J4jbqX286WvsP2FTEb7w3Mm7EsINFaIwxpofq8ohAVX+vqiNx5ggqUNX8gIclgc401cPb/01dykiebZnK1JGZR36NMca47IhX/6jqt05GICHhvV9CbTkvpN9FTHQ0E/L7uh2RMcYckV0G2l0q18OSR9Fx1/Pc9kynCE2kx+2ojDHmiCwRdAdVePNeiE5k49jvsLOm0YaFjDG9hk061x1W/w22vA8XPsy7W70ATB5uicAY0zvYEcGJaqyFt78PA8bD6TdS5C9Ck9kn1u3IjDHmqFgiOFGLfgH1u+HC37DvgJdPtu1jyogst6MyxpijZongROxeA0v/AKd/FbJPZ9GGSnyKU6TeGGN6CUsEx0sV3vwOxCbD1AcAmF9SQXpiNGOzk10Ozhhjjl5QE4GIzBCR9SJSKiL3Habd5SKiItJ75jBa9TJsWwzTfgTxfWn1+li0voLzh2cSEWF3Extjeo+gJQIR8QCPAjNx5i26RkRGddIuCbgL+ChYsXS7A9Xwzg8guxDG3wDAim3V1Da22rCQMabXCeYRwQSgVFU3q2oz8BJwaSftfgL8AmgMYizda8HPoaEKLvwNRDhf4fyS3URGCGcPTXc5OGOMOTbBTATZwPaA52X+de1E5DQgV1XfPNwbicgsESkWkeLKysruj/RY7Pw3LHsKCm+CAePaVy8oqWBCfl/6xFrtAWNM7+LayWIRiQAeBu49UltVfVJVC1W1MCMjI/jBdcXnc04Qx/WFKd9vX719bwMbdtfbsJAxplcKZiIoB3IDnuf417VJAkYDC0VkCzAJmNOjTxivfAHKPoYv/gTiUttXL1hfAcBkSwTGmF4omIlgGTBURPJFJBq4GpjTtlFVa1Q1XVXzVDUPWApcoqrFQYzp+DXshXkPQO4kGHv1IZuKSirIS4u3IjTGmF4paIlAVVuB24G3gXXAK6q6RkQeFJFLgvW5QVP0E+dqoYATxAANza0s3lTF5BGZVoTGGNMrBXXSOVWdC8ztsO6HXbQ9P5ixnJDyFVD8J5j4Teg3+pBNi0uraG712fkBY0yvZXcWH4nP60wxnZgJk+//3Oai9RUkRHusCI0xpteyaaiPZMXzsGMFXPa0M51EAFVlQUkFZw+1IjTGmN7LjggOZ38VzP8x5J0DY6743OZ1O+vYWdNow0LGmF7NEsHhzP8RNNXBBb+CTk4Et182akVojDG9mCWCrmxf5gwLTfoWZI7stElRSQVjspOtCI0xplezRNAZnxfe/DYkDYDzvtdpk737m1mxbZ/dRGaM6fXsZHFnip+BXavgij9BTFKnTRZtqEAVploiMMb0cnZE0FF9Bcz/CRScD6d8uctmRSWVpCfGMMaK0BhjejlLBB29+wC0NMAFv+70BDEQUIQmw4rQGGN6PUsEgbYugX+/CGfeAelDu2y2fOs+ahtbbVjIGBMSLBG08bY6dxAn58K53zls06L1FUR5rAiNMSY02MniNh8/CRVr4Kq/QPThZxEtWlfBGXl9SbIiNMaYEGBHBAB1u5zyk0OmwYiLDtt0+94GNlZYERpjTOiwRABOIXpvE8z8ZZcniNu03U1sicAYEyosEXz2Pnz6Kpx1N6QNPmLz+ev8RWgyEoMfmzHGnAThnQi8LTD3O5AyEM759hGbNzS3smRzFVNGZJ2E4Iwx5uQI75PFS/8AlSVwzUsQFXfE5laExhgTisL3iKCmHBY+BMNmwvCZR/WS+SVWhMYYE3rCNxG8831QL8x86KiaqyoL11dwztAMoiPD92szxoSeoPZoIjJDRNaLSKmI3NfJ9m+KyKcislJEPhCRUcGMp92mBbDmdTjnXkjNO6qXWBEaY0yoCloiEBEP8CgwExgFXNNJR/+iqo5R1XHAL4GHgxVPu9Ym5wRx3wI4886jfllRyW4Azh+REazIjDHGFcE8IpgAlKrqZlVtBl4CLg1soKq1AU8TAA1iPI4l/wdVpTDzVxB19AVlikoqGJuTTGaSFaExxoSWYCaCbGB7wPMy/7pDiMhtIrIJ54jg6HfRj0f1Nlj0Kxh5MQyddtQv27u/mU+2V1tJSmNMSHL9rKeqPqqqg4HvAT/orI2IzBKRYhEprqysPP4Pe+t+587h//ifY3pZWxEaOz9gjAlFwUwE5UBuwPMc/7quvAR8qbMNqvqkqhaqamFGxnGO0W98F0r+Ced+F1Jyj9w+wPx1FVaExhgTsoKZCJYBQ0UkX0SigauBOYENRCRw0v8LgY1Bi6apDnInwhduP6aXtXp9vLehkslWhMYYE6KCdmexqraKyO3A24AHeEZV14jIg0Cxqs4BbheRaUALsA/4arDiYfRlTunJI0wq11FbERobFjLGhKqgTjGhqnOBuR3W/TBg+a5gfv7nHGMSAOdqIStCY4wJZa6fLO7pikoqmJBvRWiMMaHLEsFhHCxCY7ONGmNClyWCwygqsSI0xpjQZ4ngMIpKKshPTyA//fA1jI0xpjezRNCFg0Vo7GjAGBPaLBF04UMrQmOMCROWCLpQVFJBYkwkZ+RZERpjTGizRNAJVWVBSQXnDE23IjTGmJBnvVwn1u6sZVdtI5NtWMgYEwYsEXRigf+y0fOHWxEaY0zos0TQiaKSCk61IjTGmDBhiaCDqvompwiNDQsZY8KEJYIOFm2otCI0xpiwYomgg6KSCjKSYhg9wIrQGGPCgyWCAC1eH4usCI0xJsxYIgiwfOs+6qwIjTEmzFgiCLCgvQiNXTZqjAkflggCzC+pYGJ+GokxQS3cZowxPYolAr/texsorai3y0aNMWHHEoFfWxGaqZYIjDFhJqiJQERmiMh6ESkVkfs62f5tEVkrIqtEZL6IDApmPIczv6SCgvQE8qwIjTEmzAQtEYiIB3gUmAmMAq4RkVEdmn0CFKrqWGA28MtgxXM4Dc2tLN1cZcNCxpiwFMwjgglAqapuVtVm4CXg0sAGqrpAVRv8T5cCOUGMp0ttRWhsWMgYE46CmQiyge0Bz8v867pyE/CvzjaIyCwRKRaR4srKym4M0VFUspvEmEgKrQiNMSYM9YiTxSJyPVAI/Kqz7ar6pKoWqmphRkb3XuPvFKGptCI0xpiwFcwL5suB3IDnOf51hxCRacD3gfNUtSmI8XSqrQiN3U1sjAlXwdwFXgYMFZF8EYkGrgbmBDYQkfHAE8AlqloRxFi6VLSurQiNJQJjTHgKWiJQ1VbgduBtYB3wiqquEZEHReQSf7NfAYnAqyKyUkTmdPF2QVO03ilCk5EUc7I/2hhjeoSgzqWgqnOBuR3W/TBgeVowP/9IquqbWLm9mrunDnMzDGOMcVVYnx1duN6K0BhjTFgngqL1ThGaUwb0cTsUY4xxTdgmghavj/c2VDJleKYVoTHGhLWwTQTFW5wiNDathDEm3IVtIliwvq0ITbrboRhjjKvCNhEUlVQwqcCK0BhjTFgmgm1V/iI0dhOZMcaEZyIoKtkN2GWjxhgD4ZoI1ldSkGFFaIwxBsIwEexvamXppiqm2LCQMcYAYZgIPizdQ7PXZ8NCxhjjF3aJYMH6CpKsCI0xxrQLq0SgqhSVVHDOMCtCY4wxbcKqN1yzo5bdtU122agxxgQIq0SwoKQCEStCY4wxgcIqERStr2BsTooVoTHGmABhkwjaitDYZaPGGHOosEkEbUVopo60RGCMMYHCJhH0iYti+qgsK0JjjDEdBDURiMgMEVkvIqUicl8n288VkRUi0ioiVwQzlumjsnjqPwsRsSI0xhgTKGiJQEQ8wKPATGAUcI2IjOrQbBtwI/BisOIwxhhzeMGcjH8CUKqqmwFE5CXgUmBtWwNV3eLf5gtiHMYYYw4jmEND2cD2gOdl/nXHTERmiUixiBRXVlZ2S3DGGGMcveJksao+qaqFqlqYkZHhdjjGGBNSgpkIyoHcgOc5/nXGGGN6kGAmgmXAUBHJF5Fo4GpgThA/zxhjzHEIWiJQ1VbgduBtYB3wiqquEZEHReQSABE5Q0TKgCuBJ0RkTbDiMcYY07lgXjWEqs4F5nZY98OA5WU4Q0bGGGNcIqrqdgzHREQqga3H+fJ0YE83htPb2fdxKPs+DrLv4lCh8H0MUtVOr7bpdYngRIhIsaoWuh1HT2Hfx6Hs+zjIvotDhfr30SsuHzXGGBM8lgiMMSbMhVsieNLtAHoY+z4OZd/HQfZdHCqkv4+wOkdgjDHm88LtiMAYY0wHlgiMMSbMhU0iOFKRnHAhIrkiskBE1orIGhG5y+2YegIR8YjIJyLyT7djcZuIpIjIbBEpEZF1IvIFt2Nyi4jc4/87WS0ifxWRWLdjCoawSARHWSQnXLQC96rqKGAScFsYfxeB7sKZCsXA74C3VHUEcCph+r2ISDZwJ1CoqqMBD86caSEnLBIBAUVyVLUZaCuSE3ZUdaeqrvAv1+H8kR9XnYhQISI5wIXA027H4jYRSQbOBf4IoKrNqlrtalDuigTiRCQSiAd2uBxPUIRLIui2IjmhRETygPHARy6H4rb/Bf4LsEp5kA9UAn/yD5U9LSIJbgflBlUtB36NU1J3J1Cjqu+4G1VwhEsiMB2ISCLwN+BuVa11Ox63iMhFQIWqLnc7lh4iEjgN+IOqjgf2A2F5Tk1EUnFGDvKBAUCCiFzvblTBES6JwIrkBBCRKJwk8IKqvuZ2PC47C7hERLbgDBlOEZG/uBuSq8qAMlVtO0qcjZMYwtE04DNVrVTVFuA14EyXYwqKcEkEViTHT0QEZ/x3nao+7HY8blPV+1U1R1XzcP5fFKlqSO71HQ1V3QVsF5Hh/lVTgbUuhuSmbcAkEYn3/91MJURPnAe1HkFPoaqtItJWJMcDPKOq4VoE5yzgBuBTEVnpX/ff/toRxgDcAbzg32naDHzN5XhcoaofichsYAXO1XafEKJTTdgUE8YYE+bCZWjIGGNMFywRGGNMmLNEYIwxYc4SgTHGhDlLBMYYE+YsERhzEonI+TbDqelpLBEYY0yYs0RgTCdE5HoR+VhEVorIE/56BfUi8lv//PTzRSTD33aciCwVkVUi8rp/jhpEZIiIzBORf4vIChEZ7H/7xID5/l/w37VqjGssERjTgYiMBK4CzlLVcYAXuA5IAIpV9RRgEfCA/yXPA99T1bHApwHrXwAeVdVTceao2elfPx64G6c2RgHO3d7GuCYsppgw5hhNBU4Hlvl31uOACpxpql/2t/kL8Jp//v4UVV3kX/8c8KqIJAHZqvo6gKo2Avjf72NVLfM/XwnkAR8E/bcypguWCIz5PAGeU9X7D1kp8v86tDve+VmaApa92N+hcZkNDRnzefOBK0QkE0BE+orIIJy/lyv8ba4FPlDVGmCfiJzjX38DsMhf/a1MRL7kf48YEYk/mb+EMUfL9kSM6UBV14rID4B3RCQCaAFuwynSMsG/rQLnPALAV4HH/R194GydNwBPiMiD/ve48iT+GsYcNZt91JijJCL1qprodhzGdDcbGjLGmDBnRwTGGBPm7IjAGGPCnCUCY4wJc5YIjDEmzFkiMMaYMGeJwBhjwtz/B0Ux5h+dWvKaAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(model.history.history.keys())\n",
    "# summarize history for accuracy\n",
    "plt.plot(model.history.history['f1'])\n",
    "plt.plot(model.history.history['val_f1'])\n",
    "plt.title('model f1')\n",
    "plt.ylabel('f1')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'val'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Еще один параметр в сверточном слое - padding. Так как мы проходим каким-то окном по последовательности, на выходе последовательность сокращается. Но мы можем добавить нулей в начало и конец, чтобы на выходе было такое же число (padding='same').\n",
    "\n",
    "Так мы можем наслаивать огромное количество слоев (она не уменьшается на каждом шаге).\n",
    "\n",
    "А еще это позволяет, например, сделать вот такую сеть:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs = tf.keras.layers.Input(shape=(MAX_LEN,))\n",
    "embeddings = tf.keras.layers.Embedding(input_dim=len(word2id), output_dim=100)(inputs, )\n",
    "convs = []\n",
    "\n",
    "# мы делаем несколько параллельных сверточных слоев (даже пар слоев с пулингом) с разным окном\n",
    "# но из-за того, что бы сделали паддинг, длинна получается одинаковая и мы можем сконкатенировать \n",
    "# векторы по второй размерности (так же как вы склеивали эбмединги от фастекста и ворд2века в предыдущей домашке)\n",
    "\n",
    "for ks in [3,4,5,7,10]:\n",
    "    conv1 = tf.keras.layers.Conv1D(kernel_size=ks, filters=32, padding='same',strides=1)(embeddings)\n",
    "    conv2 = tf.keras.layers.Conv1D(kernel_size=ks, filters=32, padding='same',strides=1, \n",
    "                                  kernel_regularizer='l2', activation='relu')(conv1)\n",
    "    convs.append(conv2)\n",
    "\n",
    "\n",
    "concat = tf.keras.layers.concatenate(convs, axis=2)\n",
    "\n",
    "# а тут мы можем поставить еще свертку\n",
    "conv_global = tf.keras.layers.Conv1D(kernel_size=5, filters=32, strides=1)(concat)\n",
    "flatten = tf.keras.layers.Flatten()(conv_global)\n",
    "dense = tf.keras.layers.Dense(50, activation='relu')(flatten)\n",
    "outputs = tf.keras.layers.Dense(len(label2id), activation='softmax')(dense)\n",
    "\n",
    "model = tf.keras.Model(inputs=inputs, outputs=outputs)\n",
    "optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)\n",
    "model.compile(optimizer=optimizer,\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=[f1])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_17\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_22 (InputLayer)           [(None, 201)]        0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding_20 (Embedding)        (None, 201, 100)     5530600     input_22[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_35 (Conv1D)              (None, 201, 32)      9632        embedding_20[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_37 (Conv1D)              (None, 201, 32)      12832       embedding_20[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_39 (Conv1D)              (None, 201, 32)      16032       embedding_20[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_41 (Conv1D)              (None, 201, 32)      22432       embedding_20[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_43 (Conv1D)              (None, 201, 32)      32032       embedding_20[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_36 (Conv1D)              (None, 201, 32)      3104        conv1d_35[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_38 (Conv1D)              (None, 201, 32)      4128        conv1d_37[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_40 (Conv1D)              (None, 201, 32)      5152        conv1d_39[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_42 (Conv1D)              (None, 201, 32)      7200        conv1d_41[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_44 (Conv1D)              (None, 201, 32)      10272       conv1d_43[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_1 (Concatenate)     (None, 201, 160)     0           conv1d_36[0][0]                  \n",
      "                                                                 conv1d_38[0][0]                  \n",
      "                                                                 conv1d_40[0][0]                  \n",
      "                                                                 conv1d_42[0][0]                  \n",
      "                                                                 conv1d_44[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_45 (Conv1D)              (None, 197, 32)      25632       concatenate_1[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "flatten_17 (Flatten)            (None, 6304)         0           conv1d_45[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dense_34 (Dense)                (None, 50)           315250      flatten_17[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "dense_35 (Dense)                (None, 19)           969         dense_34[0][0]                   \n",
      "==================================================================================================\n",
      "Total params: 5,995,267\n",
      "Trainable params: 5,995,267\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "71/71 [==============================] - 25s 321ms/step - loss: 1.4183 - f1: 0.0072 - val_loss: 0.4284 - val_f1: 0.0084\n",
      "Epoch 2/5\n",
      "71/71 [==============================] - 24s 337ms/step - loss: 0.3309 - f1: 0.1004 - val_loss: 0.1513 - val_f1: 0.5782\n",
      "Epoch 3/5\n",
      "71/71 [==============================] - 24s 344ms/step - loss: 0.1236 - f1: 0.6519 - val_loss: 0.0908 - val_f1: 0.7120\n",
      "Epoch 4/5\n",
      "71/71 [==============================] - 24s 342ms/step - loss: 0.0767 - f1: 0.7619 - val_loss: 0.0831 - val_f1: 0.7228\n",
      "Epoch 5/5\n",
      "71/71 [==============================] - 24s 338ms/step - loss: 0.0617 - f1: 0.8082 - val_loss: 0.0819 - val_f1: 0.7297\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f7028b68198>"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train, y_train, \n",
    "          validation_data=(X_valid, y_valid),\n",
    "          batch_size=2000,\n",
    "         epochs=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['loss', 'f1', 'val_loss', 'val_f1'])\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAzAUlEQVR4nO3deXxU5fX48c9JCCQkLFnYTAhJFQUBBQwBRS0VbVEU3MGlSl34tdWqbUXRVhGq1traWiuKqLi0rqB+v+gXi4KAC8imgLKHPawhBAiEkO38/rgTHEICWebOncyc9+uVFzP3PjP3ZMg8597z3PtcUVWMMcZEriivAzDGGOMtSwTGGBPhLBEYY0yEs0RgjDERzhKBMcZEOEsExhgT4SwRGNNAIvKqiDxay7YbReTCGtbFiciHIrJPRCYHNkpjamaJwJjQcTXQDkhW1WtEpIOITBWRbSKiIpLhcXwmTFkiMCZ0dALWqGqZ73kF8F/gKu9CMpHAEoGJCL6SzCgRWSYiB0XkZRFpJyIfi0ihiMwQkUS/9kNEZLmI7BWR2SLS1W9dLxH5xve6d4DYKtu6VESW+F47V0TOqEV8Y4GHgWEickBEblXVnar6HLAwcJ+EMceyRGAiyVXARcCpwGXAx8CDQBuc78JdACJyKvAWcI9v3TTgQxFpKiJNgf8B/g0kAZPx22MXkV7AJOD/AcnAC8BUEWl2vMBUdQzwOPCOqiao6ssB+Y2NqQVLBCaS/Mu3l70V+AKYr6rfqmox8AHQy9duGPB/qvqpqpYCfwPigHOAfkAM8LSqlqrqFI7eYx8JvKCq81W1XFVfAw77XmdMSGridQDGBNFOv8eHqnme4Ht8ErCpcoWqVojIFiAVKAe26tGzNW7ye9wJuFlEfuO3rKnvPY0JSZYIjDnWNqBH5RMREaAjsBVQIFVExC8ZpAPrfI+3AI+p6mNBjNeYBrHSkDHHehcYLCIDRSQG+D1OeWcuMA8oA+4SkRgRuRLI9nvti8AvRaSvOOJFZLCItKhPICISC1SOLzTzPTcmoCwRGFOFqq4GbgT+BezGGVi+TFVLVLUEuBIYAezBGU943++1i4DbgWeBAiDH17a+DgEHfI9X+Z4bE1BiN6YxxpjIZkcExhgT4SwRGGNMhLNEYIwxEc4SgTHGRLhGdx1BSkqKZmRkeB2GMcY0KosXL96tqm2qW9foEkFGRgaLFi3yOgxjjGlURGRTTeusNGSMMRHOEoExxkQ4SwTGGBPhGt0YQXVKS0vJzc2luLjY61BcFRsbS1paGjExMV6HYowJI2GRCHJzc2nRogUZGRk4E0WGH1UlPz+f3NxcMjMzvQ7HGBNGwqI0VFxcTHJyctgmAQARITk5OeyPeowxwRcWiQAI6yRQKRJ+R2NM8LmaCERkkIisFpEcERldzfp0EZklIt/6bip+iZvxGGNMY7KvqJQlW/byv0u28vSMNXyXu8+V7bg2RiAi0cB4nJuF5wILRWSqqq7wa/ZH4F1VfV5ETse5SXiGWzG5Ze/evbz55pv8+te/rtPrLrnkEt58801at27tTmDGmJC3t6iEDbsPsim/yPfvQTbmF7Ex/yB7i0qPtBOBlIRm9EhrFfAY3BwszgZyVHU9gIi8DQwF/BOBAi19j1vh3CKw0dm7dy/PPffcMYmgrKyMJk1q/oinTZvmdmjGmBBQcLCEjfkHnZ/dRb7HRWzcfZB9h47u7E9qFUdGSnMG9+hARnI8GSnxZCQ3p2NSc2Jjol2Jz81EkIpz/9ZKuUDfKm0eAT7x3eg7HriwujcSkZHASID09PSAB9pQo0ePZt26dfTs2ZOYmBhiY2NJTExk1apVrFmzhssvv5wtW7ZQXFzM3XffzciRI4Efpss4cOAAF198Meeeey5z584lNTWV//3f/yUuLs7j38wYUxuqyt6iUjbkO3v0G3YXOXv2u50Ov7rOPjMlnkvP6EBmSjydkuPJTGlOWqJ7nf3xeH366HXAq6r6lIicDfxbRLqraoV/I1WdCEwEyMrKOu4t1cZ+uJwV2/YHNMjTT2rJmMu61bj+iSee4Pvvv2fJkiXMnj2bwYMH8/333x85zXPSpEkkJSVx6NAh+vTpw1VXXUVycvJR77F27VreeustXnzxRa699lree+89brzxxoD+HsaY+lNVCopKfXv1P+zROx3/QfYXlx1pKwKprZ3O/rIzfXv2yfFkpDh79s2aBL+zPx43E8FWoKPf8zTfMn+3AoMAVHWe78bcKcAuF+NyXXZ29lHn+j/zzDN88MEHAGzZsoW1a9cekwgyMzPp2bMnAGeddRYbN24MVrjGGB9VZc/BkqM7+fwf9u79O/sogdTEODKS4xnaM5VOyc2P7N13TIoLuc7+eNxMBAuBziKSiZMAhgPXV2mzGRgIvCoiXYFYIK8hGz3ennuwxMfHH3k8e/ZsZsyYwbx582jevDkDBgyo9lqAZs2aHXkcHR3NoUN2j3Jj3PBDZ39svX5j/kEKj9PZV9brM1LiSUtsXJ398biWCFS1TETuBKYD0cAkVV0uIuOARao6Ffg98KKI/BZn4HiEqh639BOKWrRoQWFhYbXr9u3bR2JiIs2bN2fVqlV8/fXXQY7OmMijquQfLDmqXl95Zk51nX1aYnM6JTfnivTUI/X6TsnxdExsTtMmYXO5VY1cHSNQ1Wk4p4T6L3vY7/EKoL+bMQRDcnIy/fv3p3v37sTFxdGuXbsj6wYNGsSECRPo2rUrp512Gv369fMwUmPCR2Vn71+vrzwzZ9PuIgoPH9vZZ6TE0yu99ZF6fUZyPGkR0tkfjzS2HfCsrCytemOalStX0rVrV48iCq5I+l2NUVV2HyjxG6B1Ov1NvrLOAb/OPjpKSEuMc/bok5v79uzj6ZTc3Dp7QEQWq2pWdeu8PmvIGBPhVJW8A4ePvqDKV7vflH9sZ9/R19lndUqik69e7+zZxxETHdmdfX1ZIjDGBE15hbJy+34WbNjDN5sLWJ/ndPwHS8qPtKns7DNS4umTkURGcnM6pcSTmRxPqnX2rrBEYIxxTXFpOcty97FgQz4LNhbwzaaCI3v4qa3j6NwugezMpCNn4mRYZ+8JSwTGmIDZX1zK4k0FLNywh4Ub97B0yz5Kyp3rQ09tl8DQnieRnZlEn4wkTmptV86HCksExph6yys8zMKNe1jg6/hXbt9PhTrlne6prbj5nE70yXA6/sT4pl6Ha2pgicAYUyuqypY9h1iwcc+RPf71uw8CEBsTRa+Oidx5QWeyM5Lold6a+GbWvTQW9j/lgYSEBA4cOOB1GMYcV0WFsmZXIQs37GG+r+Pfuf8wAC1jm9AnI4lr+3QkOzOJ7ie1ivjTMxszSwTGGABKyir4fts+Fm5wSj2LNhUcmTWzXctmZGcmk52RSJ/MJE5t24KoKLtjXriwRBAAo0ePpmPHjtxxxx0APPLIIzRp0oRZs2ZRUFBAaWkpjz76KEOHDvU4UmN+UFRSxreb9zp7+xv28O2WAopLnYHdH6XEM6hbe/pkJpGdkUTHpDi7VWoYC79E8PFo2PFdYN+zfQ+4+IkaVw8bNox77rnnSCJ49913mT59OnfddRctW7Zk9+7d9OvXjyFDhtiXyXim4GAJCzc6JZ4FGwtYvnUfZRVKlEDXDi0Z3ied7MwksjISadsi1utwQ4cqVJRDeYnzU1H2w+PysjouL/X9+B5XlNZt+dl3QJfA39E3/BKBB3r16sWuXbvYtm0beXl5JCYm0r59e37729/y+eefExUVxdatW9m5cyft27f3OlwTIbbtPXTUGT1rdjrjUk2jozizYytGnv8j+mQmcVanRFrGxgQvMFUoOwxlxT90chVVOrxql9en4w3QcjdFxUB0U4iO8f34Hle3HHemBAq/RHCcPXc3XXPNNUyZMoUdO3YwbNgw3njjDfLy8li8eDExMTFkZGRUO/20MYGgqqzLO+js8W/Yw4KNe8gtcKYyT2jWhN6dEhly5klkZyZzRlqro++CVVEBJQehtBjKDgXo32IoLap5nUsdmtNxNoWoJj88jm5S/fKmzSG69fHb19QhB2p5VBPnLjYeC79E4JFhw4Zx++23s3v3bubMmcO7775L27ZtiYmJYdasWWzatMnrEE1joOrsAZ+goy0vKWLb7r1s3pXPtrwCdu3ZS0XpIWIp4eyYcq5OgPadICW2goToMqLKimHtIVhZDKWHfB217/0asscb3QxiYqFJ3LH/xraChPZVlsdCTNwP/x7pGOvQgde0PCo6JDrVxsgSQYB069aNwsJCUlNT6dChAzfccAOXXXYZPXr0ICsriy5dungdonFT0R5Y+wkU7/+h0y4t8nW4h2r4t4aO/ug7tVYrGuf2f/63ANQmgjaJRWLiEImD0ljAr9NNaHtsR9yQf5vEQpSdMhoOXE0EIjII+CfO3+1LqvpElfX/AH7ie9ocaKuqrd2MyU3ffffDIHVKSgrz5s2rtp1dQxBGdq2E+RNg6TtOJ+4vKsav86yyt9w0Hpqn1Lg3XUwMmwqVdXvKWJ1fypr8cg5UNKFYm9IuqTWnpqXQNb0dZ2S0p11Sa4iJQ6Kb2skIpl5cSwQiEg2MBy4CcoGFIjLVdzMaAFT1t37tfwP0ciseYwKmosLZ+5//PKyf7XT0Pa6BrFugdfoPe8vRtf967SosZuGGgiODu6t2OFM1NPFN1ZB9jjNNQ1anRJuqwQScm0cE2UCOqq4HEJG3gaHAihraXweMcTEeYxqmeD8seQPmvwAFG6DFSXDBQ3DWLyA+udZvUzlVw/wN+b7TOQvY4DdVQ+/0RH5zQWeyM52pGpo3tQqucZebf2GpwBa/57lA3+oaikgnIBP4rIb1I4GRAOnp6dVuTFXD/rC4sd1NLmzkr4MFE+HbN6CkENKyYeBD0HWIM9B5AhUVyuqdhUedylk5VUOruBj6ZCQyvHKqhtRWNgWzCbpQ2dUYDkxR1fLqVqrqRGAiOLeqrLo+NjaW/Px8kpOTwzYZqCr5+fnExtqFPkGh6pR95k+ANdOds1O6XQH9fgmpZx33pSVlFXy3dd+RUzn9p2po3zLWmarBd8Vu57YJNlWD8ZybiWArR5/UkOZbVp3hwB313VBaWhq5ubnk5eXV9y0ahdjYWNLS0rwOI7yVFMGyd5zyT95KZ0D3/FHQ51ZoUfPFgIdKynnpi/XMXZd/9FQNbeK5uHt7+mQkkZ2ZRFqiTdVgQo+biWAh0FlEMnESwHDg+qqNRKQLkAhUf4pNLcTExJCZmVnflxsD+3JhwYvwzWtwqMCZVmToc9D9KudMnhP488creX3eJrqntuS67HSyM5LIykiiTYtmQQjemIZxLRGoapmI3AlMxzl9dJKqLheRccAiVZ3qazoceFutAG6CTRW2zIevn4eVHwIKXQZD319Bp3NqfXHSvHX5vD5vE7/on8GYy7q5G7MxLpDG1v9mZWXpokWLvA7DNGZlh+H79536//YlzhWwvW+CPrdDYqc6vVVRSRk/e/pzokT4+O7z7AwfE7JEZLGqZlW3zv5qTeQo3AmLJjk/B3dBymkw+O9w5nDnAq96+MvHq8gtOMQ7I8+2JGAaLfvLNeFv27fw9QRY/r4zr07nn0LfX8LJFzRobpqv1+fz2rxNjDgng+zMpAAGbExwWSIw4am8DFZ96CSALV9DTDycNQKy/x+knNLgty8qKeO+KcvolNyc+wad1vB4jfGQJQITXor2OGf+LHgJ9udC607ws8eh143OWECAPPnf1WzeU8Q7I/tZScg0evYXbMJD1cnfMs6DS56EUwc50xMH0Nfr83l17kZGnJNB3x/VfmoJY0KVJQLTeFU3+dsZ1zrln/bdXdlkZUkoPclKQiZ8WCIwjU91k78NfBh6j6jT5G/1UVkSettKQiaM2F+yaTwaOPlbQ833lYRuPrsT/awkZMKIJQIT2how+VsgFZWUcd97Tkno/ovtbnMmvFgiMKGpnpO/ueWv01ezKb+It263kpAJP/YXbUJLAyd/c8OCDXt4de5Gbjq7E2efbCUhE34sERjvBWjyNzccKinnvilLSUuM4/5BVhIy4ckSgfFO2WFY/oGTAConfzv71/Wa/M0tf52+mo35Rbx5e1/im9nXxYQn+8s2wVe4Exa/AgtfDtjkb25YuHEPr8zdwM/7deKck1O8DscY11giMMHj0uRvbjhUUs6oyUtJbR3HaDtLyIQ5SwTGXS5P/uaWv33iKwndZiUhE/5c/QsXkUHAP3HuUPaSqj5RTZtrgUcABZaq6jG3szSNUJAmf3PDoo17mPTVBm7sl845p1hJyIQ/1xKBiEQD44GLgFxgoYhMVdUVfm06Aw8A/VW1QETauhWPCZIgTv7mhkMl5YyasozU1nE8cHFXr8MxJijcPCLIBnJUdT2AiLwNDAVW+LW5HRivqgUAqrrLxXiMWzyY/M0tT32ymg27D1pJyEQUN//SU4Etfs9zgb5V2pwKICJf4ZSPHlHV/1Z9IxEZCYwESE9PdyVYUw8eTv7mhsWb9vCylYRMBPJ6l6cJ0BkYAKQBn4tID1Xd699IVScCE8G5eX2QYzRVeTz5mxuKS8sZNXkZJ7WKY7SVhEyEcTMRbAU6+j1P8y3zlwvMV9VSYIOIrMFJDAtdjMvUR4hM/uaWpz5ZzfrdB3njtr4kWEnIRBg3/+IXAp1FJBMnAQwHqp4R9D/AdcArIpKCUypa72JMpq5CbPI3NyzetIeXvtzADX3T6W8lIROBXEsEqlomIncC03Hq/5NUdbmIjAMWqepU37qfisgKoBwYpar5bsVk6qDkIHz+V1j8ashM/uYG/5LQA5dYSchEJlePgVV1GjCtyrKH/R4r8Dvfjwklnz3qzAHU9dKQmPzNLX//dA3rdx/kP7daSchELvvLN8cq2OhMBd3rRhj6rNfRuGbxpgJe+mI91/dN59zOVhIykSvK6wBMCPrsUefirwEPeB2Ja4pLyxk1ZSkdWsXxgM0lZCKcJQJztO1L4bvJ0O9X0CrV62hc849P17A+7yBPXNWDFrGN85RXYwLFEoE52qdjIC4R+t/jdSSu+WZzAS9+sZ7rstM5r3Mbr8MxxnOWCMwP1n0G62fBefdCXGuvo3FFcWk5905eSvuWsTx4iZWEjAEbLDaVKiqco4FW6ZB9u9fRuOYfM5yS0Ou3ZFtJyBgfSwTG8f17sGMZXDERmjTzOhpXfLO5gBc/X8912R05/1QrCRlTyUpDxrl38GfjoF0P6HGN19G4wrlwrLIkZBeOGePPjggMLJoEezfDje9BVHjuGzw9Yy3r8g7ympWEjDlGeH7rTe0V74M5T0Lmj+HkgV5H44pvNxcw8fN1DO/TkR9bSciYY1giiHRfPQOH9sBFY8NyCgnnwrFltGsZy4ODrSRkTHWsNBTJ9m+HeeOdieRO6uV1NK7458y15Ow6wGu3ZNPSSkLGVMuOCCLZ7D9DRRlc8JDXkbhiyZa9vDBnHcOyrCRkzPFYIohUeWvg239D1i2QlOl1NAFXeZZQu5ax/OFSKwkZczxWGopUM8dCTDz8+D6vI3HFMzPXsnbXAV79RR8rCRlzAq4eEYjIIBFZLSI5IjK6mvUjRCRPRJb4fm5zMx7js/lrWPUR9L8b4sNv+uWlW/YyYc46rs1KY8Bpbb0Ox5iQ59oRgYhEA+OBi3DuTbxQRKaq6ooqTd9R1TvdisNUoepMJZHQDs7+tdfRBNzhMmcuobYtYvnD4NO9DseYRsHNI4JsIEdV16tqCfA2MNTF7ZnaWD0NtnwNA0ZD03ivowm4ypLQn6/qQas4KwkZUxtuJoJUYIvf81zfsqquEpFlIjJFRDq6GI8pL4MZYyH5FOh1k9fRBNyy3L1MmLOea85K4ydWEjKm1rw+a+hDIENVzwA+BV6rrpGIjBSRRSKyKC8vL6gBhpUlb8Du1TBwDESH13kClSWhNgnN+OOlVhIypi7cTARbAf89/DTfsiNUNV9VD/uevgScVd0bqepEVc1S1aw2bex88HopKXKuG0jrA10v8zqagPvXzBzW7DzAn6+0kpAxdeVmIlgIdBaRTBFpCgwHpvo3EJEOfk+HACtdjCeyzX8eCrfDRePCbiqJZbl7eX7OOq4+K42fdLGSkDF15Vp9QFXLROROYDoQDUxS1eUiMg5YpKpTgbtEZAhQBuwBRrgVT0Q7mA9fPg2nXgydzvE6moA6XFbOqMnLSEloykNWEjKmXlwtFKvqNGBalWUP+z1+AHjAzRgM8MXfoOQAXDjG60gC7tnPcli9s5BJI7KsJGRMPXk9WGzcVrARFrwIPa+HtuE11cJ3uft4bvY6ruqdxgVd2nkdjjGNliWCcPfZYxAVDQMe9DqSgCopq+DeyUtJSWjKw1YSMqZBLBGEs+1L4bt3od+voFV1l3A0Xs9+tpbVOwuds4SaW0nImIawRBDOZjwCcYnQ/x6vIwmo77fuY/zsdVzZO9VKQsYEgCWCcLVuFqz7DM67F+Jaex1NwFSWhJLjmzLm0m5eh2NMWAivy0uNo6ICPn0YWqVD9u1eRxNQz362llU7CnnppiwrCRkTIJYIwtHy92HHMrjiBWjSzOtoAuZISahXKheebiUhYwLFSkPhpuwwzBwH7XpAj2u9jiZgjioJXWYlIWMCyY4Iws2iV2DvJrjxPYgKnzz/7KwcKwkZ45Lw6SkMFO+Hz5+EzPPh5IFeRxMw32/dx3OzcrjCSkLGuMISQTj56p9QlB9WE8uVlFUwasoyEuObMuYyu3DMGDdYaShcFO6AeeOh+1VwUi+vowmY8bNyWLl9Py/elEXr5k29DseYsGRHBOFi9p+hogwu+KPXkQTM8m37GD8rh8t7nsRFVhIyxjWWCMJB3hr45t+QdQsk/cjraAKitLyCeycvo3XzpjwyxM4SMsZNVhoKBzPHQkxz+PF9XkcSMJUloYk/P8tKQsa4zI4IGrvN82HVR9D/LohP8TqagFixbT/PfpbD0J4n8dNu7b0Ox5iwV69EICIJtWw3SERWi0iOiIw+TrurRERFJKs+8UQsVWcqifi2cPYdXkcTEE5JaKlTErILx4wJivoeEaw4UQMRiQbGAxcDpwPXicgx5/+JSAvgbmB+PWOJXKs/hi1fw4DR0DTe62gC4rlZ61ixfT+PXdGdxHgrCRkTDDWOEYjI72paBdTmiCAbyFHV9b73exsYyrFJ5E/AX4BRtXhPU6m8zJlmOvkU6H2T19EExIpt+/nXZ2sZcuZJ/MxKQsYEzfGOCB4HEoEWVX4STvC6SqnAFr/nub5lR4hIb6Cjqv7f8d5IREaKyCIRWZSXl1eLTUeApW/C7tUwcAxEN/4pF0rLKxg1ZSmtm8cw1s4SMiaojnfW0DfA/6jq4qorROS2hm5YRKKAvwMjTtRWVScCEwGysrK0odtu9EqKYNbjkNYHul7mdTQB8fzsdSzftp8JN55lJSFjgux4e/ZbgU0icnc162ozqLsV6Oj3PM23rFILoDswW0Q2Av2AqTZgXAvzn4fC7WEzlcTK7U5J6LIzT2JQdysJGRNsx0sEpwNNgVtEJFFEkip/gNJavPdCoLOIZIpIU2A4MLVyparuU9UUVc1Q1Qzga2CIqi6q928TCYr2wJdPw6kXQ6dzvI6mwSpLQq3irCRkjFeOVxp6AZgJ/AhYjDNIXEl9y2ukqmUicicwHYgGJqnqchEZByxS1anHe72pwed/g5IDcOEYryMJiAmz1/H91v1MuLE3SVYSMsYTNSYCVX0GeEZEnlfVX9XnzVV1GjCtyrKHa2g7oD7biCgFm2Dhi9Dzemjb1etoGmzVjv0889laLj2jA4O6d/A6HGMi1gnP/qlvEjAumPUYSBQMeNDrSBqs8sKxlrExjBva3etwjIloNsVEY7F9GSx7F/r+Elqlnrh9iHthjlMSevTy7lYSMsZjlggaixljIK41nPtbryNpsNU7CvnnTKckdHEPKwkZ4zVLBI3Bulmw7jM4714nGTRiZX4lITtLyJjQYNNQh7qKCudooFVH6NPg6/g898Ln6/lu6z6eu6E3yQnNvA7HGIMlgtC3/H3YvhSueAFiYr2OpkFW7yjk6RlrGHxGBy6xkpAxIcNKQ6GsrARmjoN2PaDHtV5H0yBlvgvHWsTGMM5KQsaEFDsiCGWLJsHeTXDDexDVuHP2C5+vZ1nuPsZfbyUhY0JN4+5dwlnxfvj8Scg8H04Z6HU0DbJmZyH/nLGWwT06MPgMKwkZE2osEYSquc9AUT5cOLZRTyxXeZZQQmwTxg61kpAxochKQ6GocAfMGw/droTU3l5H0yATv3BKQs9e34sUKwkZE5LsiCAUzX4Cyktg4ENeR9Iga3cW8vSna7mkR3suPeMkr8MxxtTAEkGo2b0Wvnkdsm6BpONO8BrSKktC8c2ibS4hY0KclYZCzYxHICYOzr/P60ga5MUvNrA0dx//us5KQsaEOjsiCCVbFsCqj6D/3ZDQxuto6m3tzkL+8ekaLu7enkvtLCFjQp4lglChCp8+DPFtod+vvY6m3srKK7h3yrIjJSFpxGc8GRMpXE0EIjJIRFaLSI6IjK5m/S9F5DsRWSIiX4rI6W7GE9JWfwyb58GA0dAsweto6u2lLzewdMtexg7tTpsWVhIypjFwLRGISDQwHrgY5/7H11XT0b+pqj1UtSfwJPB3t+IJaeVlMHMsJJ8CvW/yOpp6y9lVyN8/XcOgbu25zEpCxjQabh4RZAM5qrpeVUuAt4Gh/g1Udb/f03iceyFHnqVvQt4qGPgwRMd4HU29lFco905eRnzTaP50uZWEjGlM3DxrKBXY4vc8F+hbtZGI3AH8DmgKXFDdG4nISGAkQHp6esAD9VRJEcx6HNL6QNchXkdTby99sZ4lW/byzHW9rCRkTCPj+WCxqo5X1ZOB+4E/1tBmoqpmqWpWmzaN92yaas2fAIXbG/VUEjm7DvDUp2v4Wbd2VhIyphFyMxFsBTr6PU/zLavJ28DlLsYTeor2wJdPw6mDIKO/19HUS3mFMmrKUppbSciYRsvNRLAQ6CwimSLSFBgOTPVvICKd/Z4OBta6GE/o+eIpKCmECx/xOpJ6e/nL9Xy7eS9jh3SjbYvGfeMcYyKVa2MEqlomIncC04FoYJKqLheRccAiVZ0K3CkiFwKlQAFws1vxhJyCTbBgIvS8Htp29TqaesnZdYC/fbKGn57ejiFn2lxCxjRWrk4xoarTgGlVlj3s9/huN7cf0mY9BhIFAx70OpJ6Ka9Q7vOVhB69wkpCxjRmng8WR6Tty2DZu9D3l9Aq1eto6mXSlxv4xkpCxoQFSwRemPEIxLaCc3/rdST1si7vAH/7ZDUXWUnImLBgiSDY1s+GdTPh/HshrrXX0dRZeYUyavJSYmOieczOEjImLNg01MFUUQGfjoFWHaHP7V5HUy+vfOWUhJ4e1pO2La0kZEw4sEQQTMvfh+1L4IoXIKbxdaLr8w7w1+mrubBrO4b2tJKQMeHCSkPBUlYCn/0J2nWHHtd4HU2dOReOLSM2JprH7SwhY8KKHREEy+JXoGAj3PAeREV7HU2dvfLVBhZvKuAfw860kpAxYcaOCIKheD/M+QtknAenDPQ6mjrbsPugryTUlst7Ns7TXY0xNbNEEAxz/wVF+XDRuEY3sVzlWULNmkTx2BU9rCRkTBiy0pDbCnfAvGeh25WQ2tvraOrs1bkbWbSpgL9feybtrCRkTFiyIwK3zX4Cykvggmpn2A5pTkloFQO7tOWKXlYSMiZcWSJw0+618M3rkHULJJ/sdTR1UlkSahodxeNXWknImHBmicBNM8dCTBycf5/XkdTJvqJSfvPWNyzaVMCYy7pZSciYMGdjBG7ZsgBWfujMLprQeO6qNm9dPr97dwl5hYcZfXEXruxtJSFjwp0lAjeoOlNJxLeFs+/wOppaKSmr4B8z1jBhzjoyk+P54Nf96ZHWyuuwjDFB4GppSEQGichqEckRkdHVrP+diKwQkWUiMlNEOrkZT9Cs+S9sngsDRkOzBK+jOaH1eQe4esJcnp+9jmFZHfnornMtCRgTQVw7IhCRaGA8cBGQCywUkamqusKv2bdAlqoWicivgCeBYW7FFBTlZc4008mnQO+bvI7muFSVdxdt4ZGpK2jaJIrnb+jNxT3s5vPGRBo3S0PZQI6qrgcQkbeBocCRRKCqs/zafw3c6GI8wbH0LchbBde+DtExXkdTo71FJTzw/nd8/P0Ozjk5maeuPZMOreK8DssY4wE3E0EqsMXveS7Q9zjtbwU+djEe95UUwazHITULug7xOpoazV23m9+9s5T8g4d54OIu3H7ej4iKstNDjYlUITFYLCI3AlnAj2tYPxIYCZCenh7EyOpo/gQo3AZXvRSSU0mUlFXw90/X8MLnzoDwizfZgLAxxt1EsBXo6Pc8zbfsKCJyIfAH4Meqeri6N1LVicBEgKysLA18qAFQtAe+fBpOHQQZ/b2O5hjr8g5wz9tL+G7rPq7LTuehS7vSvGlI7AcYYzzmZk+wEOgsIpk4CWA4cL1/AxHpBbwADFLVXS7G4r4vnoKSQhg4xutIjqKqvL1wC+M+XEGzmCgm3HgWg7q39zosY0wIcS0RqGqZiNwJTAeigUmqulxExgGLVHUq8FcgAZjsm8Jgs6qGbnG9Jns3w4KJcOb10O50r6M5ouBgCaPfX8b05Tvpf0oyT13Tk/at7CphY8zRXK0NqOo0YFqVZQ/7Pb7Qze0HzWePgUTBTx70OpIj5ubs5nfvOgPCD17ShdvOtQFhY0z1rEjcUDu+g2XvQP+7oZX30zGUlFXw1CermfjFejJT4nnp5v50T7UBYWNMzSwRNNSMRyC2FZx7j9eRkLPrAPe88y3fb93P9X3TeWjw6cQ1bXy3xTTGBJclgoZYPwdyZsBPH4W4RM/CUFXeWrCFcR8tJy4mmok/P4ufdrMBYWNM7VgiqK+KCvj0YWjVEfrc7lkYBQdLuP+9ZXyyYifnnpLCU3YnMWNMHVkiqK8VH8D2JXD5BIjxpuP9cu1ufj95CXsOlvCHS7py67mZNiBsjKkzSwT1UVYCM8dBu+5wxrVB3/zhsnKe+mQNEz9fz8lt4pk0og/dTrIBYWNM/VgiqI/Fr0DBRrhhCkQFdzA2Z9cB7n77W5Zv28+N/dL5wyU2IGyMaRhLBHVVvB/m/AUyzoNTgncZhKry5oLN/OmjFTRv2oQXb8riotPbBW37xpjwZYmgrub+C4ry4aKxQZtYbo9vQPjTFTs5r3MKT11zJm1tQNgYEyCWCOqicAfMexa6XQGpZwVlk1+szeP37y5lb1EpfxzclVv624CwMSawLBHUxZy/QHkJXPCQ65s6XFbOX/+7mpe+3MApbRN45Rc2IGyMcYclgtravRYWvwZ9boXkk13dVM6uQu56awkrtu/n5/068eAlXW1A2BjjGksEtTVzHMTEwfn3ubYJVeU/8zfz6EcriG/WhJdvzmJgVxsQNsa4yxJBbWxZCCunwoAHIaGNK5vIP3CY+99bxoyVuzj/1Db87ZozaNvCBoSNMe6zRHAiqs5UEvFt4ew7XNnE52vy+P3kpewrKuXhS09nxDkZNiBsjAkaSwQnsmY6bJ4Lg5+CZgkBfevDZeU8+d/VvPzlBjq3TeD1W7Lp2qFlQLdhjDEnEuXmm4vIIBFZLSI5IjK6mvXni8g3IlImIle7GUu9VJQ700wnnwK9bw7oW6/dWcjQZ7/i5S83cPPZnfjwN+daEjDGeMK1IwIRiQbGAxcBucBCEZmqqiv8mm0GRgD3uhVHgyx5E/JWwrWvQ3RMQN5SVfnP15t49P9WktCsCZNGZHFBFxsQNsZ4x83SUDaQo6rrAUTkbWAocCQRqOpG37oKF+Oon9JDMOtxSM2CroG5jfLuA4e5f8oyZq7axYDT2vDXq8+kTYtmAXlvY4ypLzcTQSqwxe95LtC3Pm8kIiOBkQDp6ekNj6w25k+Awm1w1UsBmUpizhrnCuH9xaWMucwZEJYgTVFhjDHH0ygGi1V1IjARICsrS13fYNEe+OIfcOogyOjfoLcqLnUGhCd9tYFT2yXwn9uy6dLexgKMMaHDzUSwFejo9zzNtyz0ffEUlBTCwDENeps1Owu5661vWbWjkBHnZDD64i7ExtgVwsaY0OJmIlgIdBaRTJwEMBy43sXtBcbezbBgIpx5PbQ7vV5voaq8Pm8Tj09bSYvYJrwyog8/6dI2wIEaY0xguJYIVLVMRO4EpgPRwCRVXS4i44BFqjpVRPoAHwCJwGUiMlZVu7kVU63MehwkCn7yQL1evvvAYUZNXsqs1Xn85LQ2PGkDwsaYEOfqGIGqTgOmVVn2sN/jhTglo9Cw4ztY+jb0vwta1T2sWat3MWryUvYXlzF2SDduOruTDQgbY0JeoxgsDpoZj0BsKzj3t3V6WXFpOU98vIpX526kS/sWvHFbP05r38KdGI0xJsAsEVRaPwdyZsBFf4K4xFq/bPUOZ0B49c5CftE/g/sH2YCwMaZxsUQAUFEBM8ZAq46QPbJWL1FVXpu7kcc/XkXL2Bhe+UUffnKaDQgbYxofSwQAKz6Abd/C5RMg5sRTP+cVHmbUlKXMXp3HBV3a8uTVZ5CSYAPCxpjGyRJBWQnM/BO07QZnXHvC5rNW7WLUlKUUFpcxbmg3ft7PBoSNMY2bJYLFr0LBBrhhCkTVXNuvOiD85u39OLWdDQgbYxq/yE4ExfudG9JnnAenXFhjs1U79nP3W0tYvbOQW8/NZNTPTrMBYWNM2IjsRDDvWSjaDReNrXZiOVXlla828sR/nQHh127J5senunOrSmOM8UrkJoLCnTD3Weh2BaSedczqXYXFjJq8jDlr8hjoGxBOtgFhY0wYitxEMOcJKD8MFzx0zKqZK3dy35RlHDhcxp8u786NfdNtQNgYE7YiMxHszoHFr0HWLZB88pHFxaXlPD5tJa/P20TXDi15e3hPOtuAsDEmzEVmIpg5FmLi4Mf3H1m0cvt+7nrrW9buOsBt52YyatBpNGtiA8LGmPAXeYlgy0JYORUGPAgJbaioUF6Zu5G/fLyKVs1jeP2WbM63AWFjTASJrESg6kwlEd8Wzr6DXfuLuXfKMj5fk8eFXdvx5NVnkBTf1OsojTEmqCIrEayZDpu+gsFPMWPdQe57bxlFJWU8enl3brABYWNMhIqcRFBRDjMeoSLpZB7JPYvX5y/i9A4teea6npzS1gaEjTGRK8rNNxeRQSKyWkRyRGR0Neubicg7vvXzRSTDtWCWvgV5K/nToat5ff42bj8vkw/uOMeSgDEm4rl2RCAi0cB44CIgF1goIlNVdYVfs1uBAlU9RUSGA38BhrkRz6xdCeRVXMD/lfbhP7f24tzOKW5sxhhjGh03S0PZQI6qrgcQkbeBoYB/IhgKPOJ7PAV4VkREVTXQwbQ47Xze2dWR/17ZwwaEjTHGj5uJIBXY4vc8F+hbUxvfze73AcnAbv9GIjISGAmQnp5er2CyMpLIykiq12uNMSacuTpGECiqOlFVs1Q1q00bO8ffGGMCyc1EsBXo6Pc8zbes2jYi0gRoBeS7GJMxxpgq3EwEC4HOIpIpIk2B4cDUKm2mAjf7Hl8NfObG+IAxxpiauTZG4Kv53wlMB6KBSaq6XETGAYtUdSrwMvBvEckB9uAkC2OMMUHk6gVlqjoNmFZl2cN+j4uBa9yMwRhjzPE1isFiY4wx7rFEYIwxEc4SgTHGRDhpbCfpiEgesKmeL0+hysVqIcLiqhuLq+5CNTaLq24aElcnVa32QqxGlwgaQkQWqWqW13FUZXHVjcVVd6Eam8VVN27FZaUhY4yJcJYIjDEmwkVaIpjodQA1sLjqxuKqu1CNzeKqG1fiiqgxAmOMMceKtCMCY4wxVVgiMMaYCBeWiSCk7pVct7hGiEieiCzx/dwWpLgmicguEfm+hvUiIs/44l4mIr1DJK4BIrLP7/N6uLp2AY6po4jMEpEVIrJcRO6upk3QP69axuXF5xUrIgtEZKkvrrHVtAn697GWcXnyffRtO1pEvhWRj6pZF/jPS1XD6gdnptN1wI+ApsBS4PQqbX4NTPA9Hg68EyJxjQCe9eAzOx/oDXxfw/pLgI8BAfoB80MkrgHAR0H+rDoAvX2PWwBrqvl/DPrnVcu4vPi8BEjwPY4B5gP9qrTx4vtYm7g8+T76tv074M3q/r/c+LzC8YjgyL2SVbUEqLxXsr+hwGu+x1OAgSIiIRCXJ1T1c5xpwGsyFHhdHV8DrUWkQwjEFXSqul1Vv/E9LgRW4txy1V/QP69axhV0vs/ggO9pjO+n6hkqQf8+1jIuT4hIGjAYeKmGJgH/vMIxEVR3r+SqX4ij7pUMVN4r2eu4AK7ylROmiEjHatZ7obaxe+Fs3+H9xyLSLZgb9h2S98LZm/Tn6ed1nLjAg8/LV+ZYAuwCPlXVGj+vIH4faxMXePN9fBq4D6ioYX3AP69wTASN2YdAhqqeAXzKD1nfVO8bnPlTzgT+BfxPsDYsIgnAe8A9qro/WNs9kRPE5cnnparlqtoT53a12SLSPRjbPZFaxBX076OIXArsUtXFbm/LXzgmglC9V/IJ41LVfFU97Hv6EnCWyzHVVm0+06BT1f2Vh/fq3AQpRkRS3N6uiMTgdLZvqOr71TTx5PM6UVxefV5+298LzAIGVVnl6b3La4rLo+9jf2CIiGzEKR9fICL/qdIm4J9XOCaCUL1X8gnjqlJHHoJT5w0FU4GbfGfD9AP2qep2r4MSkfaVtVERycb5e3a1A/Ft72Vgpar+vYZmQf+8ahOXR59XGxFp7XscB1wErKrSLOjfx9rE5cX3UVUfUNU0Vc3A6SM+U9UbqzQL+Ofl6q0qvaAheq/kWsZ1l4gMAcp8cY1wOy4AEXkL54ySFBHJBcbgDJ6hqhNwbjd6CZADFAG/CJG4rgZ+JSJlwCFgeBASen/g58B3vvoywINAul9cXnxetYnLi8+rA/CaiETjJJ53VfUjr7+PtYzLk+9jddz+vGyKCWOMiXDhWBoyxhhTB5YIjDEmwlkiMMaYCGeJwBhjIpwlAmOMiXCWCIwJInFmAD1mRkljvGSJwBhjIpwlAmOqISI3+uarXyIiL/gmKDsgIv/wzV8/U0Ta+Nr2FJGvfZOTfSAiib7lp4jIDN8kb9+IyMm+t0/wTWK2SkTecHumTWNOxBKBMVWISFdgGNDfNylZOXADEI9zdWc3YA7Olc4ArwP3+yYn+85v+RvAeN8kb+cAldNM9ALuAU7HuT9Ff5d/JWOOK+ymmDAmAAbiTDC20LezHoczVXEF8I6vzX+A90WkFdBaVef4lr8GTBaRFkCqqn4AoKrFAL73W6Cqub7nS4AM4EvXfytjamCJwJhjCfCaqj5w1EKRh6q0q+/8LIf9Hpdj30PjMSsNGXOsmcDVItIWQESSRKQTzvflal+b64EvVXUfUCAi5/mW/xyY47tLWK6IXO57j2Yi0jyYv4QxtWV7IsZUoaorROSPwCciEgWUAncAB3FuYPJHnFLRMN9LbgYm+Dr69fww2+jPgRd8M0eWAtcE8dcwptZs9lFjaklEDqhqgtdxGBNoVhoyxpgIZ0cExhgT4eyIwBhjIpwlAmOMiXCWCIwxJsJZIjDGmAhnicAYYyLc/wfCxfEFZ5R7dwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(model.history.history.keys())\n",
    "# summarize history for accuracy\n",
    "plt.plot(model.history.history['f1'])\n",
    "plt.plot(model.history.history['val_f1'])\n",
    "plt.title('model f1')\n",
    "plt.ylabel('f1')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'val'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "В Conv1D только один вариант паддинга - нулями. Но есть и другие способы (посмотрите лекцию по ссылке в начале). В функции tf.pad дает больше выбора паддинга. Его можно вставить в tf.keras.layers.Lambda, чтобы использовать в сети. \n",
    "\n",
    "Для текстов это наверное не так актуально, но при работе с картинками может пригодиться."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.layers.core.Lambda at 0x7f4545d80940>"
      ]
     },
     "execution_count": 138,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# пример\n",
    "tf.keras.layers.Lambda(lambda x: tf.pad(x, \n",
    "                                        [[0,0], [1,1], [0,0]], \n",
    "                                        mode='CONSTANT'))\n",
    "\n",
    "# x тут это то, что будет на входе, например последовательность эмбедингов\n",
    "#\n",
    "# второй параметр - это значения паддинга для каждой из размерности слева и справа\n",
    "\n",
    "# у нас будут последовательности вида (batch_size, max_len, embedding_size)\n",
    "\n",
    "# с размером батча ничего не делаем, поэтому [0, 0]\n",
    "\n",
    "# длину последовательности меняем, она связана с kernel_size как раз и меняется - [1, 1], \n",
    "# т.е. паддим одну колонку слева и одну справа (была 134, станет 136, например)\n",
    "\n",
    "# размер эмбединга тоже не трогаем, т.к. фильтра применяется на всю глубину\n",
    "#\n",
    "\n",
    "# mode - это тип паддинга, CONSTANT - это паддинг нулями\n",
    "# еще есть SYMMETRIC, REFLECT\n",
    "\n",
    "# вот примеры, как это работает\n",
    "\n",
    "# tf.pad(t, paddings, \"CONSTANT\")  # [[0, 0, 0, 0, 0, 0, 0],\n",
    "#                                  #  [0, 0, 1, 2, 3, 0, 0],\n",
    "#                                  #  [0, 0, 4, 5, 6, 0, 0],\n",
    "#                                  #  [0, 0, 0, 0, 0, 0, 0]]\n",
    "\n",
    "# tf.pad(t, paddings, \"REFLECT\")  # [[6, 5, 4, 5, 6, 5, 4],\n",
    "#                                 #  [3, 2, 1, 2, 3, 2, 1],\n",
    "#                                 #  [6, 5, 4, 5, 6, 5, 4],\n",
    "#                                 #  [3, 2, 1, 2, 3, 2, 1]]\n",
    "\n",
    "# tf.pad(t, paddings, \"SYMMETRIC\")  # [[2, 1, 1, 2, 3, 3, 2],\n",
    "#                                   #  [2, 1, 1, 2, 3, 3, 2],\n",
    "#                                   #  [5, 4, 4, 5, 6, 6, 5],\n",
    "#                                   #  [5, 4, 4, 5, 6, 6, 5]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Попробуйте подбирать паддинги, чтобы понять, как это работает, на игрушечном тензоре"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = [[[1,2,3],[3, 4,5]], [[6,7,8], [9,10, 11]]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[ 1,  2,  3],\n",
       "        [ 3,  4,  5]],\n",
       "\n",
       "       [[ 6,  7,  8],\n",
       "        [ 9, 10, 11]]])"
      ]
     },
     "execution_count": 144,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.array(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2, 2, 3)"
      ]
     },
     "execution_count": 146,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.array(x).shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 4, 3), dtype=int32, numpy=\n",
       "array([[[ 0,  0,  0],\n",
       "        [ 1,  2,  3],\n",
       "        [ 3,  4,  5],\n",
       "        [ 0,  0,  0]],\n",
       "\n",
       "       [[ 0,  0,  0],\n",
       "        [ 6,  7,  8],\n",
       "        [ 9, 10, 11],\n",
       "        [ 0,  0,  0]]], dtype=int32)>"
      ]
     },
     "execution_count": 140,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.pad(x, [[0, 0], [1, 1], [0, 0]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Это еще не все сложности.\n",
    "\n",
    "Понадобится ещё разобраться в том, как расчитывается размер выхода из свертки в зависимости от размера фильтра.\n",
    "\n",
    "В лекции была такая формула -   \n",
    "output = (shape-kernel_size)/stride + 1\n",
    "\n",
    "\n",
    "Из неё нам нужно вывести размер паддинга с каждой стороны нашей матрицы. Для этого нам нужно найти такой shape, при котором выход из свертки будет равняться нашему текущему shape.   \n",
    "shape = (X-kernel_size)/stride + 1\n",
    "\n",
    "Потом его нужно разделить на 2 и получить размер паддинга с каждой стороны. \n",
    "\n",
    "Получается что-то такое -   \n",
    "X = ((((shape - 1) * stride) + kernel_size) - shape) / 2 (может еще упрощается, но я давно не считал руками)\n",
    "\n",
    "Тут еще нужно учесть, что не при любом значении kernel_size будут получаться четные числа, а это значит нужно будет паддить не ровно. Это вы можете доделать самостоятельно. Я просто взял значения kernel_size (3,5), при которых все сходится."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Еще небольшая проблема в том, что мы не можем задать сетку, как это делали выше (через цикл). Я не знаю, почему так, но если развернуть цикл и написать все без переменных, то работает."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 129,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Паддинг для kernel_size=3\n",
    "ks = 3\n",
    "((((MAX_LEN - 1) * 1) + ks) -  MAX_LEN) / 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.0"
      ]
     },
     "execution_count": 130,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Паддинг для kernel_size=5\n",
    "ks = 5\n",
    "((((MAX_LEN - 1) * 1) + ks) -  MAX_LEN) / 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "inputs = tf.keras.layers.Input(shape=(MAX_LEN,))\n",
    "embeddings = tf.keras.layers.Embedding(input_dim=len(word2id), output_dim=100)(inputs, )\n",
    "convs = []\n",
    "\n",
    "\n",
    "# kernel_size = 3\n",
    "pad1 = tf.keras.layers.Lambda(lambda x: tf.pad(x, \n",
    "                                               [[0,0], [1,1], [0, 0]], \n",
    "                                               mode='REFLECT'))(embeddings)\n",
    "\n",
    "conv1 = tf.keras.layers.Conv1D(kernel_size=3, filters=32, strides=1)(pad1)\n",
    "\n",
    "pad2 = tf.keras.layers.Lambda(lambda x: tf.pad(x, \n",
    "                                               [[0,0], [1,1], [0, 0]], \n",
    "                                               mode='REFLECT'))(conv1)\n",
    "\n",
    "conv2 = tf.keras.layers.Conv1D(kernel_size=3, filters=32,strides=1, \n",
    "                              kernel_regularizer='l2', activation='relu')(pad2)\n",
    "\n",
    "\n",
    "#kernel_size = 5\n",
    "pad3 = tf.keras.layers.Lambda(lambda x: tf.pad(x, \n",
    "                                               [[0,0], [2,2], [0, 0]], \n",
    "                                               mode='REFLECT'))(embeddings)\n",
    "\n",
    "conv3 = tf.keras.layers.Conv1D(kernel_size=5, filters=32, strides=1)(pad3)\n",
    "\n",
    "pad4 = tf.keras.layers.Lambda(lambda x: tf.pad(x, \n",
    "                                               [[0,0], [2,2], [0, 0]], \n",
    "                                               mode='REFLECT'))(conv3)\n",
    "\n",
    "conv4 = tf.keras.layers.Conv1D(kernel_size=5, filters=32,strides=1, \n",
    "                              kernel_regularizer='l2', activation='relu')(pad4)\n",
    "\n",
    "\n",
    "\n",
    "concat = tf.keras.layers.concatenate([conv2, conv4])\n",
    "\n",
    "conv_global = tf.keras.layers.Conv1D(kernel_size=5, filters=32, strides=1)(concat)\n",
    "flatten = tf.keras.layers.Flatten()(conv_global)\n",
    "dense = tf.keras.layers.Dense(50, activation='relu')(flatten)\n",
    "outputs = tf.keras.layers.Dense(len(label2id), activation='softmax')(dense)\n",
    "\n",
    "model = tf.keras.Model(inputs=inputs, outputs=outputs)\n",
    "optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)\n",
    "model.compile(optimizer=optimizer,\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=[f1])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_18\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_23 (InputLayer)           [(None, 201)]        0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding_21 (Embedding)        (None, 201, 100)     5530600     input_23[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "lambda (Lambda)                 (None, 203, 100)     0           embedding_21[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "lambda_2 (Lambda)               (None, 205, 100)     0           embedding_21[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_46 (Conv1D)              (None, 201, 32)      9632        lambda[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_48 (Conv1D)              (None, 201, 32)      16032       lambda_2[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "lambda_1 (Lambda)               (None, 203, 32)      0           conv1d_46[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "lambda_3 (Lambda)               (None, 205, 32)      0           conv1d_48[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_47 (Conv1D)              (None, 201, 32)      3104        lambda_1[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_49 (Conv1D)              (None, 201, 32)      5152        lambda_3[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_2 (Concatenate)     (None, 201, 64)      0           conv1d_47[0][0]                  \n",
      "                                                                 conv1d_49[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_50 (Conv1D)              (None, 197, 32)      10272       concatenate_2[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "flatten_18 (Flatten)            (None, 6304)         0           conv1d_50[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "dense_36 (Dense)                (None, 50)           315250      flatten_18[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "dense_37 (Dense)                (None, 19)           969         dense_36[0][0]                   \n",
      "==================================================================================================\n",
      "Total params: 5,891,011\n",
      "Trainable params: 5,891,011\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "71/71 [==============================] - 17s 212ms/step - loss: 0.7792 - f1: 2.6767e-05 - val_loss: 0.3100 - val_f1: 0.0000e+00\n",
      "Epoch 2/5\n",
      "71/71 [==============================] - 15s 207ms/step - loss: 0.2570 - f1: 0.1484 - val_loss: 0.1517 - val_f1: 0.4862\n",
      "Epoch 3/5\n",
      "71/71 [==============================] - 15s 208ms/step - loss: 0.1313 - f1: 0.5529 - val_loss: 0.1043 - val_f1: 0.6510\n",
      "Epoch 4/5\n",
      "71/71 [==============================] - 15s 206ms/step - loss: 0.0909 - f1: 0.7038 - val_loss: 0.0882 - val_f1: 0.7033\n",
      "Epoch 5/5\n",
      "71/71 [==============================] - 15s 208ms/step - loss: 0.0721 - f1: 0.7719 - val_loss: 0.0845 - val_f1: 0.7213\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f7042cc71d0>"
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train, y_train, \n",
    "          validation_data=(X_valid, y_valid),\n",
    "          batch_size=2000,\n",
    "         epochs=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['loss', 'f1', 'val_loss', 'val_f1'])\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA24ElEQVR4nO3dd3gVZfbA8e8hBEIJEEJPCL2DEggBwYJtxQYoFmygqKyrLra1d1dXd93VXfeHqyiI2LCAiopixYqSEHoPCCShhUBCgPSc3x9zwRgChHDnzk3u+TxPHu/MvJk5Gblzpr3nFVXFGGNM6KrldQDGGGO8ZYnAGGNCnCUCY4wJcZYIjDEmxFkiMMaYEGeJwBhjQpwlAmOOkYhMFZHHK9l2g4iccYhl9UTkIxHJEZF3/RulMYdmicCY4HER0BKIVtWLRaS1iMwSkc0ioiLS3uP4TA1licCY4NEOWKOqxb7pUuAzYJR3IZlQYInAhATfLZk7RWSJiOwVkcki0lJEPhWRXBH5UkSiyrQfLiLLRSRbROaKSI8yy+JFJMX3e28DEeW2dZ6ILPL97k8iclwl4nsUeAi4VET2iMi1qrpNVZ8Hkvy3J4w5mCUCE0pGAWcCXYHzgU+B+4DmON+FCQAi0hV4C7jVt2w28JGI1BGROsAHwGtAU+Bdypyxi0g8MAX4IxANvAjMEpG6hwtMVR8G/ga8raoNVXWyX/5iYyrBEoEJJf/1nWVnAN8Dv6jqQlXNB94H4n3tLgU+UdUvVLUI+CdQDxgMDALCgX+rapGqvsfvz9jHAy+q6i+qWqKqrwIFvt8zJijV9joAYwJoW5nPeRVMN/R9bgNs3L9AVUtFJA2IAUqADP19tcaNZT63A8aKyJ/LzKvjW6cxQckSgTEH2wz02T8hIgK0BTIABWJERMokgzhgne9zGvCEqj4RwHiNOSZ2a8iYg70DnCsip4tIOHAHzu2dn4B5QDEwQUTCReRCILHM774E3CAiA8XRQETOFZHIqgQiIhHA/ucLdX3TxviVJQJjylHV1cCVwH+BHTgPls9X1UJVLQQuBK4GduI8T5hZ5neTgeuB/wN2Aam+tlWVB+zxfV7lmzbGr8QGpjHGmNBmVwTGGBPiLBEYY0yIs0RgjDEhzhKBMcaEuGrXj6BZs2bavn17r8MwxphqZcGCBTtUtXlFy1xNBCIyDPgPEAa8rKpPlVseB7wKNPG1uUdVZx9une3btyc5OdmdgI0xpoYSkY2HWubarSERCQMmAmcDPYHLRKRnuWYPAO+oajwwGnjerXiMMcZUzM1nBIlAqqqu93XCmQ6MKNdGgUa+z41xuvYbY4wJIDcTQQxO3ZX90n3zynoEuFJE0nFK/f6ZCojIeBFJFpHkzMxMN2I1xpiQ5fXD4suAqar6LxE5AXhNRHqramnZRqo6CZgEkJCQcFBX6KKiItLT08nPzw9I0F6JiIggNjaW8PBwr0MxxtQgbiaCDJyKjfvF+uaVdS0wDEBV5/kKajUDth/NhtLT04mMjKR9+/Y4hSJrHlUlKyuL9PR0OnTo4HU4xpgaxM1bQ0lAFxHp4BvVaTQwq1ybTcDpAL6hACOAo773k5+fT3R0dI1NAgAiQnR0dI2/6jHGBJ5ricA3APfNwBxgJc7bQctF5DERGe5rdgdwvYgsxhka8GqtYhW8mpwE9guFv9EYE3iuPiPw9QmYXW7eQ2U+rwCGuBmDMcZUZ7vzi1iclk3KxmxO79GC3jGN/b4Nrx8W1wjZ2dm8+eab3HjjjUf1e+eccw5vvvkmTZo0cScwY0y1UlKqrN2ey8JN2SzctIuFm7JJzdyDKohA04Z1LBEEq+zsbJ5//vmDEkFxcTG1ax96F8+efdhO1MaYGi5rTwGL0rKdA3/aLhan5bCnoBiAqPrhxMdFMfz4NsTHRXF828ZERrjzxqAlAj+45557WLduHX379iU8PJyIiAiioqJYtWoVa9asYeTIkaSlpZGfn88tt9zC+PHjgd/KZezZs4ezzz6bE088kZ9++omYmBg+/PBD6tWr5/FfZozxl6KSUlZtyWVhmnOmn7JpFxuz9gEQVkvo0TqSC+JjiI9rQr+4KNpF1w/Yc8Ealwge/Wg5Kzbv9us6e7ZpxMPn9zrk8qeeeoply5axaNEi5s6dy7nnnsuyZcsOvOY5ZcoUmjZtSl5eHgMGDGDUqFFER0f/bh1r167lrbfe4qWXXuKSSy5hxowZXHnllX79O4wxgbNtd/6B2zspm3axJD2HgmKni1TzyLr0i2vCZYlx9IuLok9MY+rVCfMs1hqXCIJBYmLi7971f+6553j//fcBSEtLY+3atQclgg4dOtC3b18A+vfvz4YNGwIVrjHmGOUXlbB88+4DB/6Fm3axOcd51btOWC16xTTiykHtiI9rQnxcFG0aRwTVW4A1LhEc7sw9UBo0aHDg89y5c/nyyy+ZN28e9evXZ+jQoRX2Bahbt+6Bz2FhYeTl2RjlxgQjVSV9Vx4p+w/6adms2JxDUYnz5ntsVD36t2/KdW2bEB/XhJ5tGlG3tndn+5VR4xKBFyIjI8nNza1wWU5ODlFRUdSvX59Vq1bx888/Bzg6Y8yx2FtQzJL0nAP39hduymbHngIA6oWHcVxsY649saNztt+2CS0aRXgc8dGzROAH0dHRDBkyhN69e1OvXj1atmx5YNmwYcN44YUX6NGjB926dWPQoEEeRmqMORxVZf2Ovb97fXPV1t2U+rq5dmzWgJO7NiM+Lop+cU3o1jKS2mHVf6BHqWJHXs8kJCRo+YFpVq5cSY8ePTyKKLBC6W81xm05eU5nrf2vby7clE1OXhEAkXVr09d3lh/fLoq+sU2IalDH44irTkQWqGpCRcvsisAYExLKdtZK2biLhWnZpG7fAzidtbq2iOScPq2IbxtFfFwTOjVvSK1awfNA102WCIwxNdL+zlr7H+ouTstmb2EJ4HTW6hcXxci+Tmet42Ld66xVHVgiMMZUe0Ulpazcsvu3e/tp2b/rrNWzdSNG9Y/1PdANbGet6sASgTGm2tmak3/ggL+wXGetFpF16RcXxeWJccQHQWet6sASgTEmqDmdtXIOvLqZsmkXW8p01urt66zVL865t986yDprVUppKRTugfwcKNjt/Dd/d5nPvp8ew6HtAL9v3hKBMSZoHNRZa9MuVmzZ/bvOWgntm9LP10O3R+vI4OisVZRf7iBe7oB+YHr3oZdxhDc4a0dAdGdLBDVFw4YN2bNnj9dhGOO5sp21UjZmsyhtFzv2FAJOZ63j2zbmupM6Et+2CX3jmtAi0oXOWiXFzoG4wjPxsgfu7EMvKyk8/DakFtRtBBGNIKIx1G0MTdo503V98w4sK9du/7LadQ+/jWPgaiIQkWHAf4Aw4GVVfarc8meBU32T9YEWqtrEzZiMMd7anJ3HB4sy+GTJFlZuKdNZq3kDTunawlePp5KdtVShcO8hDtwVnJlX1K6wEidl4Q1+f6CuHw1NO1ZwEC9/QPdN12novKMapFxLBCISBkwEzgTSgSQRmeUblQwAVb2tTPs/A/FuxeOme+65h7Zt23LTTTcB8Mgjj1C7dm2++eYbdu3aRVFREY8//jgjRozwOFJjvLG3oJjPlm1l5sJ0flqXhSr0bxfFhKHtSWhViz7R0FjyIH8nFGyArbthwxHul++/1aIlh994rfCDz7abdf792XaFZ+W+5XUjIaxmv1rq5hVBIpCqqusBRGQ6MAJYcYj2lwEPH/NWP70Hti495tX8Tqs+cPZTh1x86aWXcuuttx5IBO+88w5z5sxhwoQJNGrUiB07djBo0CCGDx9e/R5iGVNFJaXKvHVZzExJZ/6yVcSVbOSEBlu5N24HXXQTdXelwrxKnI3XLXegbtQGmnc/zEG83AE9vF5Qn40HAzcTQQyQVmY6HRhYUUMRaQd0AL52MR7XxMfHs337djZv3kxmZiZRUVG0atWK2267je+++45atWqRkZHBtm3baNWqldfhGuOegj1sWp3C0oXzyN24mLZFG7g/LI3osN3ODeIiYE9zaNED2l8BDZof5v54I+dsvFYQPAyu4YLlYfFo4D3Viq/xRGQ8MB4gLi7u8Gs6zJm7my6++GLee+89tm7dyqWXXsobb7xBZmYmCxYsIDw8nPbt21dYftqYaqmkGHaug23LYfsKCjYvpSBjGQ3zMohDiQMKJIK8Zl1oGHc+tOoNLXtCi17QsLnX0Zty3EwEGUDbMtOxvnkVGQ3cdKgVqeokYBI4Ref8FaA/XXrppVx//fXs2LGDb7/9lnfeeYcWLVoQHh7ON998w8aNG70O0Zijpwq7N8P2FQcO+mxbATtWH3hTppRapGkrVpW2JbvhUFp37Uf8gCE0bdOVurWqf2XOUOBmIkgCuohIB5wEMBq4vHwjEekORAHzXIzFdb169SI3N5eYmBhat27NFVdcwfnnn0+fPn1ISEige/fuXodozOHlZcP2lbB9uXOw3+77yc850EQj25DTqAuLm47is8xoFhe0YU/DjpzTrwMX9ouha8tI7+I3VeZaIlDVYhG5GZiDc3dwiqouF5HHgGRVneVrOhqYrtWtHnYFli797SF1s2bNmDev4txmfQiMp4oLYMca38F+/0F/JexO/61N3UbQoif0HgUterI1ohMzMxoxfWkum9bto154GGf3bsV9/WI5oVM0YSFSpbOmcvUZgarOBmaXm/dQuelH3IzBmJBVWgrZG3+7nbN9uXPA37H2t1cua4VD827QbrDvHr7vp3EsOXnFfLx0M+8vyCB54y5E8hjSqRm3ntGFs3q1okHdYHnEaI6V/Z80pibYu6PMPXzfAX/7Sija+1ubJu2gZS/ofp7z1k7LXk7JgjLvyBeVlPLt6kxmfpLClyu2U1hSSucWDbl7WHdGxrehdeN6Hvxxxm01JhGoao1/R78G3D0zx6pwH2Su/P09/G0rYO/239rUj3bO6uOv/O1NnRbdnVcxK6CqLMvYzYyUdD5avJmsvYVEN6jD5QPjGNUvlt4xjWr8dyvU1YhEEBERQVZWFtHR0TX2H6yqkpWVRURE9RsY21RBSTHsXP/b7Zz9Z/s7f+VAcbLa9ZwDfJcznQP/gdczW1SqA9WWnDw+WLiZmSnprN2+hzphtTizZ0su7BfDyV2bE14DxuI1lVMjEkFsbCzp6elkZmZ6HYqrIiIiiI2N9ToM40+qkLul3Bn+cshcDSUFThup5dS1adkbjrvUd9DvBVHtj7qzVUWlHhLaRfG3C/pwbp/WNK5fs0spmIrViEQQHh5Ohw4dvA7DmMPLz/Hdu1/x24F/23KnquV+DVs5Z/YdTnYO9i16Og9zw6t+b75sqYdPl20lr6iEuKb1mXBaFy7sF0O76AbH/reZaq1GJAJjgkpxofN6Zvl38nPKVFypE+k8sO010rmds/+NnfpN/RbGmm25zEzJ4IOFGWzdnU9kRG1Gxscwql8M/dtF1djbqOboWSIwxh/SF8AvLzgFD7PWQmmxM79WbWjWFdoOhIRrfjvoN27rSiG0HXsKmLVoM+8vzGBpRg5htYShXZvz4Hk9Ob1HCyLCrW6POZglAmOOxa4N8NVjsGwG1ItyDvjdzvbd1ukB0V2gdh1XQ8gvKuGrlduZmZLO3DWZlJQqvWMa8dB5PRnetw3NGro3oImpGSwRGFMV+3bC9/+C+ZNAwuDkO2HILYd8RdPfVJUFG3cxIyWDj5dsJje/mFaNIrj+pI5W6sEcNUsExhyN4gLn4P/dP52Hv/FXwKn3OzXyA2Bj1l7eX5jBzJQMNu38rdTDhVbqwRwDSwTGVIaqc/vnq8ecsg2dToczH3PKK7ssJ6+IT5ZsYWZKuq/UAwzuFM0tp3dhWG8r9WCOnf0LMuZINvwInz8Am1OgZR+46n3odJqrmywqKeW7NZnMTMngi5XbKCy2Ug/GPZYIjDmUHWvhi4dh9ScQ2QZG/s/p0OXSiFmqyvLNTqmHWYucUg9NG9Th8kQr9WDcZYnAmPL2ZMLcJ2HBVAivD6c9CINuhDr1XdlcRaUezujZggvjYzmlm5V6MO6zRGDMfoX74OeJ8MO/oSgPEsbBKXe7MrTi3oJi5izfysyUDH5ctwNV6N8uiicu6M15fdpYqQcTUJYIjCktgcVvwddPQO5mp0zzGY9Asy5+3UxJqfLz+ixmpKTz2bKt7CssoW3Tekw4rQsXxMfQvpmVejDesERgQlvqV/DFQ7BtGcQkwEWTnUFa/GjttlxmlCv1MKJvGy7sF0uClXowQcDVRCAiw4D/4AxV+bKqPlVBm0uAR3Bq6y5W1YPGNTbG77Yugy8ehHVfO1U8L3oFel3gt7IPO/YU8NHizcxM+X2phwfO68EZPVpaqQcTVFxLBCISBkwEzgTSgSQRmaWqK8q06QLcCwxR1V0i0sKteIwBICcDvnkCFr0JEY3hrCdhwLVQ+9jLMOwv9fD+wnTmrs6k2Eo9mGrCzSuCRCBVVdcDiMh0YASwokyb64GJqroLQFW3H7QWY/whfzf8+B+YN9EZr3fwzXDSHU59ID/4ad0Obnwjhex9RbRsVJdrT+rAhfGxdGtlpR5M8HMzEcQAZerukg4MLNemK4CI/Ihz++gRVf2s/IpEZDwwHiAuLs6VYE0NVVLkvAY69ynYtwP6XOy8DhrVzm+bWJaRw/hpC2jdOIL/XhbP4E7NrNSDqVa8flhcG+gCDAVige9EpI+qZpdtpKqTgEkACQkJNnCvOTJVWD3b6RCWtRbanQh/+CvE9PPrZjZm7eXqV+bTKKI2065NtB6/plpyMxFkAG3LTMf65pWVDvyiqkXAryKyBicxJLkYl6np0hc4JSE2/eSMBXDZdOg6zO/1/7fn5nPV5PmUlCrTxg+0JGCqLTcTQRLQRUQ64CSA0UD5N4I+AC4DXhGRZji3ita7GJOpycqODdCgOZz7DPQbC2H+/2eem1/E1VOSyMwt4M3rB9K5RUO/b8OYQHEtEahqsYjcDMzBuf8/RVWXi8hjQLKqzvIt+4OIrABKgDtVNcutmEwNddDYAHfBkAmujQ2QX1TC+GkLWLMtl5fHJhAf558HzsZ4RVSr1y33hIQETU5O9joMEww8GBugpFT581spzF66lWcvPZ4L4mNd25Yx/iQiC1Q1oaJlXj8sNubolZbC8pnw1aOQvQk6n+GMDdCyl6ubVVUenrWM2Uu38sC5PSwJmBrDEoGpXsqODdCqD1z1AXQ6NSCbfu6rVF7/eRN/PKUj153UMSDbNCYQLBGY6iFzDXz5sPNKaKMYGPmCb2yAwJRofv3njTz75RpG9YvlnmHdA7JNYwLFEoEJbnu2O53B9o8NcPpDztgA4YF7VfPTpVt48MNlnNa9BU+N6mNF4kyNY4nABKfCfU45iB//DcX5Tj2gU+6GBs0CGsa8dVncMn0R8W2bMPHyfjZIjKmRLBGY4HJgbIDHIXeLb2yAR6FZ54CHsnxzDuOnJRMXXZ8pVw+gXh2rGGpqJksEJnikfgmfPwTbl/vGBngF2p3gSSibsvYxdkoSDSNqM21cIk3q1/EkDmMCwRKB8d7WpfD5g7D+G2dsgIunQs+Rfi8JUVmZuQVcNeUXiktLmT7+BNo0sdIRpmazRGC8U3ZsgHpN/Do2QFXl5hdxzdT5bNudzxvXDaJzCysjbWo+SwQm8PJ3Ow+B5z3vGxvgz3DS7X4bG6CqCopLuOH1BazcksvLYxLo385KR5jQYInABE4Axgaocmilyu3vLObH1Cz+dfHxnNrdBsszocMSgXGfKqz6xOkQlpUK7U9ySkL4eWyAqlJVHv1oOZ8s2cJ953RnVH8rHWFCiyUC4670ZOdB8KafoFk3uOxt6HqWZw+CK/J/X6cybd5Grj+pA+NP7uR1OMYEnCUC446dvzpjAyyfCQ1awHnPQvwYV8YGOBZvzd/Ev75Yw4XxMdx7dg+vwzHGE8H1rTTV376dTlno+ZMgLNzpDTz4z66NDXAsPlu2lfvfX8rQbs35+0XHUcvGGTYhyhKB8Y8DYwM8DQW5EH8lDL0PGrX2OrIK/bI+iwnTF3JcbBOev8JKR5jQ5uq/fhEZJiKrRSRVRO6pYPnVIpIpIot8P9e5GY9xQWkpLH0P/i/BKQ8dmwg3/AjD/xu0SWDllt1cNy2ZtlH1eOXqAdSvY+dDJrS59g0QkTBgInAmziD1SSIyS1VXlGv6tqre7FYcxkXlxwYY8yF0HOp1VIeVtnMfY6bMp0Gd2ky7diBRDax0hDFungolAqmquh5ARKYDI4DyicBUNx6PDVBVO/YUMGbKfAqLS3n3hhOIsdIRxgDuJoIYIK3MdDowsIJ2o0TkZGANcJuqplXQxgSDsmMD1GkApz8Mg/4U0LEBqmpPQTHjpiaxJSePN64bSNeWwffw2hiveH1z9CPgLVUtEJE/Aq8Cp5VvJCLjgfEAcXFxgY3QVDA2wHVwyl0BHxugqgqLS7nhtQUs37ybSVf1p3+7pl6HZExQcTMRZABty0zH+uYdoKpZZSZfBv5R0YpUdRIwCSAhIUH9G6Y5pPJjA/Q43xkbILr6dLoqLVXueHcxP6Tu4OmLjuP0Hi29DsmYoONmIkgCuohIB5wEMBq4vGwDEWmtqlt8k8OBlS7GY47GzvXw9lWwbRnEDnBKQ8cN8jqqo6KqPPbxCj5avJm7h3Xn4oS2R/4lY0KQa4lAVYtF5GZgDhAGTFHV5SLyGJCsqrOACSIyHCgGdgJXuxWPOQqqMGsC5KTBxa9CzxFBVRKisp6fu46pP23g2hM7cMMpHb0Ox5ig5eozAlWdDcwuN++hMp/vBe51MwZTBUvehg3fw3n/hl4jvY6mSt5O2sTTc1Yzsm8b7j+nhw04b8xhBPf7fibw8nbBnPud20H9xnodTZV8vnwr985cysldm/OPi4630hHGHIHXbw2ZYPPlo04yOO+DoO8XUJH5v+7kz28tpE9sE/53RT/q1K5+f4MxgWbfEvOb9GSnj8DAG5yewtXMqq27ufbVJGJ8pSMa1LXzHGMqwxKBcZQUw8e3QmRrOLX6PbZJ27mPMZPnU79OGNPGJdLUSkcYU2l2ymQc8yfB1qVwybSgLBl9OFl7Chg7ZT75RSW8e8NgYqPqex2SMdWKJQIDuzfDN09A5zOhx3Cvozkqe32lIzKy83j9uoF0a1W9kpgxwcASgYHP7oXSYjjn6WrVX6CwuJQbXl/Ass27eeHK/gxob6UjjKkKe0YQ6tZ+CSs+gJP/Ak07eB1NpZWWKne+t5jv1+7gyQv6cGZPKx1hTFVZIghlRXkw+w6I7gKDJ3gdTaWpKo9/spIPF23mzrO6cckAKx1hzLGwW0Oh7PtnYNcGGPsR1K7rdTSV9sK365ny469cM6Q9Nw6tPgXwjAlWdkUQqnashR+edQaU6XCy19FU2jvJafz9s1UMP74ND57b00pHGOMHlghCkSp8cjvUqQ9/eNzraCrtyxXbuHfmUk7q0ox/XmylI4zxF7s1FIqWvge/fgfnPgMNW3gdTaUkb9jJTW+m0KtNI/53ZX8rHWGMH9m3KdTkZcOceyGmP/S/xutoKmXNtlzGTU2iTROndERDKx1hjF/ZNyrUfP1X2JcFV86oFkXlMrLzGDN5PhHhTumI6IbV56G2MdVF8B8JjP9kLICkyZA4Hlof73U0R7RzbyFXTf6FvYXFvDoukbZNrXSEMW6wK4JQUVoCH98GDVvCqfd7Hc0R7St0Skek78rjtXGJ9GjdyOuQjKmxXL0iEJFhIrJaRFJF5J7DtBslIioiCW7GE9KSXoYti2HYkxAR3AfVopJS/vR6CkvSs/nvZfEM7BjtdUjG1GiuJQIRCQMmAmcDPYHLRKRnBe0igVuAX9yKJeTlboWvH4dOp0GvC7yO5rBKS5W73lvCt2sy+dsFfTirVyuvQzKmxnPziiARSFXV9apaCEwHRlTQ7q/A34F8F2MJbXPug+ICOOefQV9U7slPV/L+wgz+8oeujE6M8zocY0KCm4kgBkgrM53um3eAiPQD2qrqJ4dbkYiMF5FkEUnOzMz0f6Q12bqvYdkMOOkOiA7ucgwvfruOl77/lbEntOOmUzt7HY4xIaNKiUBEGh7rhkWkFvAMcMeR2qrqJFVNUNWE5s2bH+umQ0dRPnxyBzTtBCfe6nU0h/XegnSe/HQV5x7XmofP72WlI4wJoKpeEayoRJsMoGxZyFjfvP0igd7AXBHZAAwCZtkDYz/68d+wcz2c+6+gLir39apt3D1jCUM6R/PMJVY6wphAO+TroyJy+6EWAZW5IkgCuohIB5wEMBq4fP9CVc0BmpXZ3lzgL6qaXIl1myPJWgff/wt6XwSdTvU6mkNasHEXN76RQs/WjXjxqgTq1g7zOiRjQs7hrgj+BkThnLmX/Wl4hN8DQFWLgZuBOcBK4B1VXS4ij4lI9RoPsbpRdW4J1Y6As/7mdTSHtNZXOqJVowheucZKRxjjlcN981KAD1R1QfkFInJdZVauqrOB2eXmPXSItkMrs05TCctnwvpvnLeEIoNz5K7N2XmMmTKfOrVr8dq1A2lmpSOM8czhzuwzgI0icksFy+w+frDKz3HGIG4TDwnjvI6mQrv2FjJmynz25Bfz6jVWOsIYrx0uEfQE6gDjRCRKRJru/wGKAhOeOWpfPwF7tjslpmsF3/32fYXFjHs1iU079/HS2AR6tgnuXs7GhILD3Rp6EfgK6AgswHlIvJ/65ptgsnkhJL0EA66DmH5eR3OQopJSbnojhcVp2Tx/RT8GWekIY4LCIa8IVPU5Ve0BTFHVjqraocyPJYFgs7+oXIPmcPqDXkdzEFXl7hlL+GZ1Jo+P7MOw3q29DskY43PE1zRU9U+BCMQco+QpzhXBqMkQ0djraA7y1KermJmSwe1nduXygVY6wphgYuMR1AS52+Crv0KHU6D3KK+jOchL363nxe/WM+aEdvz5NCsdYUywsURQE3z+ABTnOQ+Ig6w0w8yUdJ6YvZJz+1jpCGOClSWC6m79XFj6Dpx4GzQLrrPtb1Zv5673ljC4UzTPXHo8YVY6wpigZImgOisucHoQR3WAEw9VEcQbCzft4sbXU+jWKpIXr+pvpSOMCWLWp786+/E5yEp1BqIPj/A6mgNSt+dyzdQkWjSqy9RrEomMCPc6JGPMYdgVQXW1cz1897Qz4ljnM7yO5oAtOXmMmTyf2rVqMW1cIs0jrXSEMcHOEkF1pAqz74SwOnDWk15Hc0D2vkLGTJ7P7vxipl4zgHbRDbwOyRhTCZYIqqMVH0Lql3DaA9AoODpm5RWWcO2ryWzM2sekMf3pHRN8fRmMMRWzZwTVTf5u+OweaHWcU0oiCBSVlHLzmymkbNrFxMv7MbhTsyP/kjEmaFgiqG7mPgm5W+HSNyDM+/99qsq9M5fy1artPD6yN+f0CY4rFGNM5dmtoepky2L45QVIuAZi+3sdDQB//2w17y1I55bTu3DloHZeh2OMqQJXE4GIDBOR1SKSKiL3VLD8BhFZKiKLROQHEenpZjzVWmkpfHw71I+G0ysc2yfgXv5+PS98u44rBsZx6xldvA7HGFNFriUCEQkDJgJn44xtcFkFB/o3VbWPqvYF/gE841Y81V7KVMhIhj88AfWivI6GDxZm8PgnKzm7dyseG9HbSkcYU425eUWQCKSq6npVLQSmAyPKNlDV3WUmG+CMc2DK25MJXz4C7U+C4y7xOhq+XZPJX95dzKCOTXn20r5WOsKYas7Np40xQFqZ6XRgYPlGInITcDvOaGinuRhP9fX5A1C4LyiKyi1Ky+ZPry+gS8tIJo1JICLcSkcYU915/rBYVSeqaifgbuCBitqIyHgRSRaR5MzMzMAG6LVfv4cl02HILdC8q6ehrMvcwzWvzCe6YR1eHTeARlY6wpgawc1EkAG0LTMd65t3KNOBkRUtUNVJqpqgqgnNmzf3X4TBrrjQKSrXpB2c/BdPQ9mak8+YyfMJqyW8Nm4gLSKDp7aRMebYuJkIkoAuItJBROoAo4FZZRuISNlXTc4F1roYT/Uz77+wYzWc808Ir+dZGDn7ihg7ZT7Z+wqZek0i7ZtZ6QhjahLXnhGoarGI3AzMAcJwxj5eLiKPAcmqOgu4WUTOAIqAXcBYt+KpdnZtgG//AT2GQ9c/eBZGflEJ101L4tcde3nlmgFWOsKYGsjVrqmqOhuYXW7eQ2U+3+Lm9qstVZh9F9SqDcOe8iyM4pJSbn5zIckbd/Hfy+IZ0tlKRxhTE3n+sNhUYNXHsHYOnHofNI7xJARV5b73l/Llym08OrwX5x3XxpM4jDHus0QQbAr2wKd3Q8s+kPhHT0IoKVUe+GAZ7ySnM+G0zow5ob0ncRhjAsP7qmXm9+Y+Cbsz4OKpnhSVKygu4ba3FzF76Vb+eEpHbjvT21dWjTHus0QQTLYug5//B/3GQtvEgG8+N7+I8dMWMG99Fg+c24PrTuoY8BiMMYFniSBYlJbCx7dBvSZwxiMB33xmbgFXvzKf1VtzeeaS47mwX2zAYzDGeMMSQbBY+Bqkz4eR/4P6TQO66U1Z+7hqyi9s313AS2MTOLVbi4Bu3xjjLUsEwWDvDvjyYWg3BI6/LKCbXr45h7FTkiguLeWN6wfSL877yqbGmMCyRBAMvngICnIDXlRu3rosxk9LpmFEbaaPP4HOLSIDtm1jTPCwROC1jT/BojfgxNugRfeAbfazZVuYMH0RcU3rM21cIm2aeFfCwhjjLUsEXioudEYdaxwHJ98VsM2+NX8T97+/lOPbNmHK2AFENagTsG0bY4KPJQIv/TwRMlfCZW9Dnfqub05V+b+vU/nXF2sY2q05z1/Rj/p17J+AMaHOjgJeyd7kFJXrfh50G+b65kpLlUc/Ws6r8zZyQXwM/7joOMLDrGO5McYSgXc+vdv5bwCKyhUUl3DHO4v5eMkWrjuxA/ed04NaNrykMcbHEoEXVn0Cq2fDmX+FJm2P3P4Y7Cko5obXFvBD6g7uPbs7fzylk6vbM8ZUP5YIAq1wr3M10KInDPqTq5vK2lPANVOTWL55N09fdBwXJ7ibdIwx1ZMlgkD79u+Qkwbj5kCYe2P+pu3cx5gp89mcnceLV/bnjJ4tXduWMaZ6s0QQSNtWwLyJEH8VxA1ybTOrtu5mzOT55BeV8MZ1A0loH9iSFcaY6sXV10ZEZJiIrBaRVBG5p4Llt4vIChFZIiJfiUg7N+PxVGkpfHI71G0EZz7m2maSNuzkkhfmIQLv3jDYkoAx5ohcSwQiEgZMBM4GegKXiUjPcs0WAgmqehzwHvAPt+Lx3OI3YdM8Jwm4VFTuixXbuPLlX2jWsC4z/jSYbq2sZIQx5sjcvCJIBFJVdb2qFgLTgRFlG6jqN6q6zzf5M1Azax/v2wmfPwhtB0HfK1zZxDtJadzw+gK6t4rk3RtOIDbK/Q5qxpiawc1EEAOklZlO9807lGuBTytaICLjRSRZRJIzMzP9GGKAfPEQFOyG856BWv7d5arK83NTuWvGEgZ3iubN6wcR3bCuX7dhjKnZguJhsYhcCSQAp1S0XFUnAZMAEhISNIChHbtNPztjDQyeAC17+XXVpaXK45+sZMqPvzL8+Db88+LjqVPbegsbY46Om4kgAyj74nqsb97viMgZwP3AKapa4GI8gVdS5BSVaxQLp9zt11UXFpdy13uL+WDRZq4e3J6HzutpvYWNMVXiZiJIArqISAecBDAauLxsAxGJB14Ehqnqdhdj8cbP/4Pty2H0m1C3od9Wu6+wmBteT+G7NZnceVY3bhzaCQngOAbGmJrFtUSgqsUicjMwBwgDpqjqchF5DEhW1VnA00BD4F3fgWyTqg53K6aAykmHuU9B17Oh+7l+W+2uvYVcMzWJJenZPHVhH0Ynxvlt3caY0OTqMwJVnQ3MLjfvoTKfz3Bz+5769G7QUjj7735bZUZ2HmMm/0Larjz+d2V/zurVym/rNsaErqB4WFzjrP4MVn0MZzwCUf7pI7d2Wy5jpsxnT0Exr41LZGDHaL+s1xhjLBH4W+E++PROaN4dBt3kl1Uu2LiLcVOTqFO7Fu/88QR6tG7kl/UaYwxYIvC/7552Bp25ejbUPvYhIL9etY0b30ihVaMIXrt2IG2bWkcxY4x/WSLwp+2r4KfnnN7D7Ycc8+pmLEjnrhlL6NE6kqnXJNLMOooZY1xgicBfVOGTO6BOQ78UlZv03Tr+NnsVgztF8+JV/YmMcK9ktTEmtFki8JfF02HjD3D+f6BBsyqvRlV56tNVvPjdes7t05pnLj2eurXD/BioMcb8niUCf9i3Ez5/AGITIX5MlVdTVFLKPTOWMiMlnasGteOR4b0Is97CxhiXWSLwh68ehbxdx1RULq+whJveTOHrVdu57YyuTDi9s/UWNsYEhCWCY5WWBAumwgk3Q6s+VVpF9r5Crn01mZRNu3h8ZG+uHFRzx+cxxgQfSwTHoqQYPr4NItvA0IMGYKuULTl5jJ0ynw079vH85f04u09rPwdpjDGHZ4ngWMx/EbYthUteg7pHPxpY6vY9jJn8C7vzi5k6bgCDO1X9IbMxxlSVJYKqysmAb/4GXf4APc4/6l9fuMnpLRxWS5g+fhC9Yxq7EKQxxhyZJYKqmnMvlBbD2f+Ao3yo++2aTG54bQHNI+sybVwi7Zs1cClIY4w5MhvOqirWfgErPoST74SmHY7qVz9clMG1U5No36wB7/3pBEsCxhjP2RXB0SrKg9l/gWZdneEnj8KUH37lsY9XMKhjUyaNSaCR9RY2xgQBSwRH6/t/wa4NMPajSheVU1WenrOa5+euY1ivVvx7dF8iwq23sDEmOLh6a0hEhonIahFJFZGD3q8UkZNFJEVEikXkIjdj8YvMNfDDv+G40dDh5Er9SrGvt/Dzc9dxWWIcE6/oZ0nAGBNUXLsiEJEwYCJwJpAOJInILFVdUabZJuBq4C9uxeE3qvDJ7VCnPvzh8Ur9Sn5RCX9+ayFfrNjGhNM6c9uZXa23sDEm6Lh5aygRSFXV9QAiMh0YARxIBKq6wbes1MU4/GPpu7Dhezj3GWjY/IjNc/KKuP7VZJI27uTR4b0YO7i9+zEaY0wVuJkIYoC0MtPpwMCqrEhExgPjAeLiPBisPW8XzLkPYvpD/2uO2Hzb7nzGTpnPusw9PDc6nvOPbxOAII0xpmqqxeujqjpJVRNUNaF58yOfjfvdV3+FfVlw3rNHLCq3PnMPo/73E2k79/HK1YmWBIwxQc/NK4IMoG2Z6VjfvOolfQEkT4GBN0Dr4w/bdGl6Dle/Mh+At8YP4rjYJgEI0Bhjjo2bVwRJQBcR6SAidYDRwCwXt+d/JcXw8a0Q2QpOve+wTX9Yu4PRk+YRER7GuzecYEnAGFNtuJYIVLUYuBmYA6wE3lHV5SLymIgMBxCRASKSDlwMvCgiy92Kp0qSXoatS2DYkxDR6JDNPl6ymWumzqdt0/rMvHEwHZs3DGCQxhhzbFztUKaqs4HZ5eY9VOZzEs4to+Czewt8/Th0Oh16jjxks2nzNvDwrOUktIvi5bEDaFzPegsbY6oX61l8KHPug5JCOOfpCovKqSrPfrGG575O5YweLfm/y+Oto5gxplqyRFCR1K9g+Uw49X6I7nTQ4pJS5YEPlvHW/E1ckhDL3y7oQ+2wavECljHGHMQSQXlF+U5RuejOMOSWgxbnF5Vw6/RFfLZ8KzcO7cSdZ3Wz3sLGmGrNEkF5PzwLO9fDmA+hdt3fLdqdX8T4acn8vH4nD53Xk3EnHl0JamOMCUaWCMrakQo/PAN9LoaOQ3+3aHtuPldPSWLNtlz+M7ovI/rGeBOjMcb4mSWC/VRh9h1Qux784YnfLdqYtZerJs8nM7eAl8cmMLRbC4+CNMYY/7NEsN+yGbB+LpzzT4hs+dvsjByufiWJktJS3rx+IPFxUd7FaIwxLrBEAJCf47wu2iYeEsYdmD1vXRbXT0umUURtpo0fTOcW1lHMGFPzWCIAp+PY3ky4/G2o5fQF+HTpFm6Zvoh20fWZdm0irRvX8zhIY4xxhyWCjBSnlMSA65wrAuCNXzbywAfL6BcXxeSxCTSpX7khKY0xpjoK7URQWgIf3wYNmsNpD6CqPPdVKs9+uYbTurdg4uX9qFfHegsbY2q20E4EyVNgyyIYNZmSOo14dNZyps3byKh+sTw1qg/h1lvYGBMCQjcR5G6Drx6DjkMp6D6S26cv5JMlW/jjyR255+zu1lvYGBMyQjcRfH4/FOez94y/M/7VZH5MzeK+c7oz/uSDawsZY0xNFpqJYN03sPRd9p3wF0bP2MGKLbv518XHM6p/cFbENsYYN4VeIigugNl/oahxB0YuGcim3bm8NKY/p3VveeTfNcaYGij0EsGP/4GsVG4Pe5BtpfDGdQPp366p11EZY4xnXH0tRkSGichqEUkVkXsqWF5XRN72Lf9FRNq7GQ9Z6yj99mk+ZTBJYfG8e8MJlgSMMSHPtUQgImHAROBsoCdwmYj0LNfsWmCXqnYGngX+7lY8qLLjnQnsKwnjlQbXMePGwXRtGena5owxprpw84ogEUhV1fWqWghMB0aUazMCeNX3+T3gdHHpvc2fP55Cs20/8FbDMbxw4/nENLGSEcYYA+4mghggrcx0um9ehW1UtRjIAaLLr0hExotIsogkZ2ZmVimYplFNWVR/MJff/BhNG1jJCGOM2a9aPCxW1UnAJICEhAStyjq6nngBnHiBX+MyxpiawM0rggygbZnpWN+8CtuISG2gMZDlYkzGGGPKcTMRJAFdRKSDiNQBRgOzyrWZBYz1fb4I+FpVq3TGb4wxpmpcuzWkqsUicjMwBwgDpqjqchF5DEhW1VnAZOA1EUkFduIkC2OMMQHk6jMCVZ0NzC4376Eyn/OBi92MwRhjzOFZnWVjjAlxlgiMMSbEWSIwxpgQZ4nAGGNCnFS3tzVFJBPYWMVfbwbs8GM4/mJxHR2L6+gFa2wW19E5lrjaqWrzihZUu0RwLEQkWVUTvI6jPIvr6FhcRy9YY7O4jo5bcdmtIWOMCXGWCIwxJsSFWiKY5HUAh2BxHR2L6+gFa2wW19FxJa6QekZgjDHmYKF2RWCMMaYcSwTGGBPiamQiEJFhIrJaRFJF5J4KltcVkbd9y38RkfZBEtfVIpIpIot8P9cFKK4pIrJdRJYdYrmIyHO+uJeISL8giWuoiOSU2V8PVdTOzzG1FZFvRGSFiCwXkVsqaBPw/VXJuLzYXxEiMl9EFvvierSCNgH/PlYyLk++j75th4nIQhH5uIJl/t9fqlqjfnBKXq8DOgJ1gMVAz3JtbgRe8H0eDbwdJHFdDfyfB/vsZKAfsOwQy88BPgUEGAT8EiRxDQU+DvC+ag30832OBNZU8P8x4PurknF5sb8EaOj7HA78Agwq18aL72Nl4vLk++jb9u3AmxX9/3Jjf9XEK4JEIFVV16tqITAdGFGuzQjgVd/n94DTRUSCIC5PqOp3OONBHMoIYJo6fgaaiEjrIIgr4FR1i6qm+D7nAis5eCzugO+vSsYVcL59sMc3Ge77Kf+GSsC/j5WMyxMiEgucC7x8iCZ+3181MRHEAGllptM5+AtxoI2qFgM5QHQQxAUwync74T0RaVvBci9UNnYvnOC7vP9URHoFcsO+S/J4nLPJsjzdX4eJCzzYX77bHIuA7cAXqnrI/RXA72Nl4gJvvo//Bu4CSg+x3O/7qyYmgursI6C9qh4HfMFvWd9ULAWnfsrxwH+BDwK1YRFpCMwAblXV3YHa7pEcIS5P9peqlqhqX5xxyxNFpHcgtnsklYgr4N9HETkP2K6qC9zeVlk1MRFkAGUzd6xvXoVtRKQ20BjI8jouVc1S1QLf5MtAf5djqqzK7NOAU9Xd+y/v1RkNL1xEmrm9XREJxznYvqGqMyto4sn+OlJcXu2vMtvPBr4BhpVb5MX38YhxefR9HAIMF5ENOLePTxOR18u18fv+qomJIAnoIiIdRKQOzsOUWeXazALG+j5fBHytvicvXsZV7j7ycJz7vMFgFjDG9zbMICBHVbd4HZSItNp/b1REEnH+Pbt6APFtbzKwUlWfOUSzgO+vysTl0f5qLiJNfJ/rAWcCq8o1C/j3sTJxefF9VNV7VTVWVdvjHCO+VtUryzXz+/5ydcxiL6hqsYjcDMzBeVNniqouF5HHgGRVnYXzhXlNRFJxHkaODpK4JojIcKDYF9fVbscFICJv4bxR0kxE0oGHcR6eoaov4Iw7fQ6QCuwDrgmSuC4C/iQixUAeMDoACX0IcBWw1Hd/GeA+IK5MXF7sr8rE5cX+ag28KiJhOInnHVX92OvvYyXj8uT7WBG395eVmDDGmBBXE28NGWOMOQqWCIwxJsRZIjDGmBBnicAYY0KcJQJjjAlxlgiMCSBxKoAeVFHSGC9ZIjDGmBBnicCYCojIlb569YtE5EVfgbI9IvKsr379VyLS3Ne2r4j87CtO9r6IRPnmdxaRL31F3lJEpJNv9Q19RcxWicgbblfaNOZILBEYU46I9AAuBYb4ipKVAFcADXB6d/YCvsXp6QwwDbjbV5xsaZn5bwATfUXeBgP7y0zEA7cCPXHGpxji8p9kzGHVuBITxvjB6TgFxpJ8J+v1cEoVlwJv+9q8DswUkcZAE1X91jf/VeBdEYkEYlT1fQBVzQfwrW++qqb7phcB7YEfXP+rjDkESwTGHEyAV1X13t/NFHmwXLuq1mcpKPO5BPseGo/ZrSFjDvYVcJGItAAQkaYi0g7n+3KRr83lwA+qmgPsEpGTfPOvAr71jRKWLiIjfeuoKyL1A/lHGFNZdiZiTDmqukJEHgA+F5FaQBFwE7AXZwCTB3BuFV3q+5WxwAu+A/16fqs2ehXwoq9yZBFwcQD/DGMqzaqPGlNJIrJHVRt6HYcx/ma3howxJsTZFYExxoQ4uyIwxpgQZ4nAGGNCnCUCY4wJcZYIjDEmxFkiMMaYEPf/hA87NN/xso0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(model.history.history.keys())\n",
    "# summarize history for accuracy\n",
    "plt.plot(model.history.history['f1'])\n",
    "plt.plot(model.history.history['val_f1'])\n",
    "plt.title('model f1')\n",
    "plt.ylabel('f1')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'val'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Сохранение и загрузка модели"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Когда какая-то модель получилась, её может потребоваться сохранить."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Assets written to: model/assets\n"
     ]
    }
   ],
   "source": [
    "model.save('model')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# сохраненную модель можно загрузить вот так, НО\n",
    "# модель с метрикой ф1 не будет загружаться, так как она не стандартная\n",
    "# model = tf.keras.models.load_model('model')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Можно сохранить только веса, но потом придется задать модель и только после этого загрузить веса."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save_weights('model.weights')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs = tf.keras.layers.Input(shape=(MAX_LEN,))\n",
    "embeddings = tf.keras.layers.Embedding(input_dim=len(word2id), output_dim=100)(inputs, )\n",
    "convs = []\n",
    "\n",
    "for ks in [2,3,4,5]:\n",
    "    conv1 = tf.keras.layers.Conv1D(kernel_size=ks, filters=32, padding='same',strides=1)(embeddings)\n",
    "    conv2 = tf.keras.layers.Conv1D(kernel_size=ks, filters=32, padding='same',strides=1, \n",
    "                                  kernel_regularizer='l2', activation='relu')(conv1)\n",
    "    convs.append(conv2)\n",
    "concat = tf.keras.layers.concatenate(convs, axis=2)\n",
    "conv_global = tf.keras.layers.Conv1D(kernel_size=5, filters=32, strides=1)(concat)\n",
    "flatten = tf.keras.layers.Flatten()(conv_global)\n",
    "dense = tf.keras.layers.Dense(50, activation='relu')(flatten)\n",
    "outputs = tf.keras.layers.Dense(len(label2id), activation='softmax')(dense)\n",
    "\n",
    "model = tf.keras.Model(inputs=inputs, outputs=outputs)\n",
    "optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)\n",
    "model.compile(optimizer=optimizer,\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=[f1])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.training.tracking.util.CheckpointLoadStatus at 0x7f491073d5c0>"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.load_weights('model.weights')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Callbacks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Еще есть такая штука как колбеки. С помощью них можно сохранять модель с лучшим результатом, останавливать обучение, если уже не обучается или менять learning_rate в зависимости от текущего состояния. С колбеками можно поставить нейронку обучаться и заняться какими-то другими делами, зная, что лучшие веса сохранятся.\n",
    "\n",
    "Описание стандартных колбеков - https://www.tensorflow.org/api_docs/python/tf/keras/callbacks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ModelCheckpoint сохраняет лучшие версии моделей\n",
    "checkpoint = tf.keras.callbacks.ModelCheckpoint('model.weights', # названия файла \n",
    "                                                monitor='val_f1', # за какой метрикой следить\n",
    "                                                verbose=1, # будет печатать что происходит\n",
    "                                                save_weights_only=True, # если нужно только веса сохранить\n",
    "                                                save_best_only=True, # сохранять только лучшие\n",
    "                                                mode='max', # если метрика должна расти, то тут max и min если наоборот\n",
    "                                                save_freq='epoch' # как часто вызывать\n",
    "                                               )\n",
    "\n",
    "# EarlyStopping позволяет автоматически остановить обучение, если качество не улучшается \n",
    "early_stop = tf.keras.callbacks.EarlyStopping(monitor='val_f1', \n",
    "                                              min_delta=0.01, # какая разница считается как улучшение\n",
    "                                              patience=3, # сколько эпох терпеть отсутствие улучшений\n",
    "                                              verbose=1, \n",
    "                                              mode='max',\n",
    "                                              )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs = tf.keras.layers.Input(shape=(MAX_LEN,))\n",
    "\n",
    "embeddings = tf.keras.layers.Embedding(input_dim=len(word2id), output_dim=100)(inputs)\n",
    "drop1 = tf.keras.layers.Dropout(0.3)(embeddings)\n",
    "conv1 = tf.keras.layers.Conv1D(kernel_size=3, filters=128, strides=1, \n",
    "                                               kernel_regularizer='l2',\n",
    "                                              activation='relu')(embeddings)\n",
    "conv2 = tf.keras.layers.Conv1D(kernel_size=5, filters=128, strides=2, \n",
    "                                               kernel_regularizer='l2',\n",
    "                                              activation='relu')(conv1)\n",
    "\n",
    "drop2 = tf.keras.layers.Dropout(0.5)(conv2) # это значит что случайно заменяется половина всех значений\n",
    "\n",
    "flatten = tf.keras.layers.Flatten()(drop2)\n",
    "dense = tf.keras.layers.Dense(50, activation='relu')(flatten)\n",
    "outputs = tf.keras.layers.Dense(len(label2id), activation='softmax')(dense)\n",
    "\n",
    "model = tf.keras.Model(inputs=inputs, outputs=outputs)\n",
    "optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)\n",
    "model.compile(optimizer=optimizer,\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=[f1])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "621/621 [==============================] - ETA: 0s - loss: 0.2668 - f1: 0.3060\n",
      "Epoch 00001: val_f1 improved from -inf to 0.52367, saving model to model.weights\n",
      "621/621 [==============================] - 65s 105ms/step - loss: 0.2668 - f1: 0.3060 - val_loss: 0.1307 - val_f1: 0.5237\n",
      "Epoch 2/10\n",
      "621/621 [==============================] - ETA: 0s - loss: 0.1262 - f1: 0.5597\n",
      "Epoch 00002: val_f1 did not improve from 0.52367\n",
      "621/621 [==============================] - 65s 105ms/step - loss: 0.1262 - f1: 0.5597 - val_loss: 0.1279 - val_f1: 0.5177\n",
      "Epoch 3/10\n",
      "621/621 [==============================] - ETA: 0s - loss: 0.1187 - f1: 0.5877\n",
      "Epoch 00003: val_f1 improved from 0.52367 to 0.56685, saving model to model.weights\n",
      "621/621 [==============================] - 65s 104ms/step - loss: 0.1187 - f1: 0.5877 - val_loss: 0.1242 - val_f1: 0.5669\n",
      "Epoch 4/10\n",
      "621/621 [==============================] - ETA: 0s - loss: 0.1142 - f1: 0.6055\n",
      "Epoch 00004: val_f1 did not improve from 0.56685\n",
      "621/621 [==============================] - 65s 104ms/step - loss: 0.1142 - f1: 0.6055 - val_loss: 0.1260 - val_f1: 0.5379\n",
      "Epoch 5/10\n",
      "621/621 [==============================] - ETA: 0s - loss: 0.1105 - f1: 0.6206\n",
      "Epoch 00005: val_f1 improved from 0.56685 to 0.58214, saving model to model.weights\n",
      "621/621 [==============================] - 65s 104ms/step - loss: 0.1105 - f1: 0.6206 - val_loss: 0.1261 - val_f1: 0.5821\n",
      "Epoch 6/10\n",
      "621/621 [==============================] - ETA: 0s - loss: 0.1076 - f1: 0.6317\n",
      "Epoch 00006: val_f1 improved from 0.58214 to 0.59003, saving model to model.weights\n",
      "621/621 [==============================] - 64s 104ms/step - loss: 0.1076 - f1: 0.6317 - val_loss: 0.1280 - val_f1: 0.5900\n",
      "Epoch 7/10\n",
      "621/621 [==============================] - ETA: 0s - loss: 0.1048 - f1: 0.6454\n",
      "Epoch 00007: val_f1 did not improve from 0.59003\n",
      "621/621 [==============================] - 64s 104ms/step - loss: 0.1048 - f1: 0.6454 - val_loss: 0.1292 - val_f1: 0.5811\n",
      "Epoch 8/10\n",
      "621/621 [==============================] - ETA: 0s - loss: 0.1022 - f1: 0.6595\n",
      "Epoch 00008: val_f1 did not improve from 0.59003\n",
      "621/621 [==============================] - 68s 109ms/step - loss: 0.1022 - f1: 0.6595 - val_loss: 0.1309 - val_f1: 0.5722\n",
      "Epoch 9/10\n",
      "621/621 [==============================] - ETA: 0s - loss: 0.0994 - f1: 0.6712\n",
      "Epoch 00009: val_f1 improved from 0.59003 to 0.60026, saving model to model.weights\n",
      "621/621 [==============================] - 65s 104ms/step - loss: 0.0994 - f1: 0.6712 - val_loss: 0.1319 - val_f1: 0.6003\n",
      "Epoch 10/10\n",
      "621/621 [==============================] - ETA: 0s - loss: 0.0971 - f1: 0.6834\n",
      "Epoch 00010: val_f1 did not improve from 0.60026\n",
      "621/621 [==============================] - 64s 104ms/step - loss: 0.0971 - f1: 0.6834 - val_loss: 0.1375 - val_f1: 0.5873\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7fe840025ba8>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "model.fit(X_train, y_train, \n",
    "          validation_data=(X_valid, y_valid),\n",
    "          batch_size=2000,\n",
    "         epochs=10,\n",
    "         callbacks=[checkpoint, early_stop])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['loss', 'f1', 'val_loss', 'val_f1'])\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deXzcZb33/9cnW7PvaZNmL6R7oaWhlFUUkCLSoiBUREE9Vo8g6PHnLZzbWxGXH+d4bj0ueAAR5BxZRBAtnmoPqIDQpCSlG92bdEnSLZ3sbbN/7j++36STMG3Tdr6ZZObzfDzm0ZnvMnPNPJp5z3Vd3+u6RFUxxhhjhosKdQGMMcaMTRYQxhhjArKAMMYYE5AFhDHGmIAsIIwxxgRkAWGMMSYgCwhjPCIivxKR747w2N0icvUJ9iWIyMsi0ioivw1uKY05MQsIY8a+m4FJQJaqfkxE8kRkuYjsExEVkZLQFs+EKwsIY8a+YmC7qva6j/uBPwM3ha5IJhJYQJiI5jbtfE1ENojIERH5pYhMEpE/iUi7iLwqIhl+xy8WkU0i0iIir4nIDL9980TkHfe83wDxw17rwyKyzj13lYicN4LyfRv4JnCriHSIyGdV9aCq/hyoCt4nYcx7WUAY4/wSvwaYCtwA/An4ZyAH52/kHgARmQo8C3zZ3bcCeFlE4kQkDvg98F9AJvBb/H7hi8g84Ang80AW8CiwXEQmnKxgqvot4PvAb1Q1WVV/GaT3bMwpWUAYAz91f5U3AH8HVqvqWlXtBF4C5rnH3Qr8t6q+oqo9wL8BCcAlwEIgFvh3Ve1R1RcY+gt/GfCoqq5W1T5VfQrocs8zZkyKCXUBjBkDDvrdPxbgcbJ7fzKwZ2CHqvaLSB2QD/QBDTp09ss9fveLgTtE5Et+2+Lc5zRmTLKAMGbk9gFzBh6IiACFQAOgQL6IiF9IFAE17v064Huq+r1RLK8xZ8WamIwZueeB60XkKhGJBb6K00y0CqgAeoF7RCRWRD4KLPA79xfAF0TkInEkicj1IpJyJgURkXhgoP9igvvYmKCygDBmhFR1G3A78FPgME6H9g2q2q2q3cBHgTuBJpz+it/5nVsNfA74GdAM7HSPPVPHgA73/lb3sTFBJbZgkDHGmECsBmGMMSYgCwhjjDEBeRoQIrJIRLaJyE4RuS/A/h+5I0vXich2EWnx23eHiOxwb3d4WU5jjDHv5VkfhIhEA9txRqjW4wwa+riqbj7B8V8C5qnqZ0QkE6gGynEuH1wDzFfVZk8Ka4wx5j28HAexANipqrUAIvIcsAQIGBDAx4FvufevBV5R1Sb33FeARTjTHASUnZ2tJSUlwSm5McZEiDVr1hxW1ZxA+7wMiHycwUED6oGLAh0oIsVAKfDXk5ybf7IXKykpobq6+owLa4wxkUhE9pxo31jppF4KvKCqfadzkogsE5FqEalubGz0qGjGGBOZvAyIBpxpCAYUuNsCWcrQ5qMRnauqj6lquaqW5+QErCEZY4w5Q14GRBVQJiKl7lTIS4Hlww8SkelABs5UBQNWAh8UkQx3Lv4PutuMMcaMEs/6IFS1V0TuxvlijwaeUNVNIvIgUK2qA2GxFHjOfxZMVW0Ske9wfLrkBwc6rE9HT08P9fX1dHZ2nt2bGQfi4+MpKCggNjY21EUxxoSJsJlqo7y8XId3Uu/atYuUlBSysrJwJt4MT6qKz+ejvb2d0tLSUBfHGDOOiMgaVS0PtG+sdFJ7orOzM+zDAUBEyMrKioiakjFm9IR1QABhHw4DIuV9GmNGjy0YZIwx41B7Zw9bD7SzeV8bsdFR3HZRUdBfwwLCYy0tLTzzzDN88YtfPK3zPvShD/HMM8+Qnp7uUcmMMeOBqnKwrYvN+1vZ1NDG5v3ObY/v6OAx84rSLSDGo5aWFn7+85+/JyB6e3uJiTnxx79ixQqvi2aMGWN6+/rZdfgIm/e3sWlfG5v3OWHQdKR78JjirERm5qVy8wUFzMpPZWZeGpNSJ5zkWc+cBYTH7rvvPmpqapg7dy6xsbHEx8eTkZHB1q1b2b59OzfeeCN1dXV0dnZy7733smzZMuD41CEdHR1cd911XHbZZaxatYr8/Hz+8Ic/kJCQEOJ3Zow5G0e6etl64HgIbN7XxtYD7XT19gMQFx3F1NxkrpkxiZmTU5k5OZXpuSmkxI/epewRExDffnkTm/e1BfU5Z05O5Vs3zDrpMQ899BDvvvsu69at47XXXuP666/n3XffHbwc9YknniAzM5Njx45x4YUXctNNN5GVlTXkOXbs2MGzzz7LL37xC2655RZefPFFbr/99qC+F2OMdw61dbJp//Ew2LKvjV2+IwyMMkhLiGXW5FQ+ubB4MAzOyUkmNjq01xFFTECMFQsWLBgyVuEnP/kJL730EgB1dXXs2LHjPQFRWlrK3LlzAZg/fz67d+8etfIaY0aur18Hm4j8awaHO7oGjynMTGBmXipL5uYPhsHktPgxeSVixATEqX7pj5akpKTB+6+99hqvvvoqFRUVJCYmcuWVVwYcyzBhwvH2xejoaI4ds/XpjQm1o929bDvQPiQMtu5v51iPM+dobLRQNjGFK6flMDPPCYIZeamkJYyf2Q4iJiBCJSUlhfb29oD7WltbycjIIDExka1bt1JZWTnKpTPGnIqqsq+1ky372tiyv40tB5wg8G8iSomPYWZeKksXFDIzL5VZk9M4d2IycTHje6iZBYTHsrKyuPTSS5k9ezYJCQlMmjRpcN+iRYt45JFHmDFjBtOmTWPhwoUhLKkxprOnj20H2tmy3+kwdmoFbbR19g4eU5SZyIy8FG44fzIz8lKZNTmVgoyEMdlEdLbCei6mLVu2MGPGjBCVaPRF2vs15kypKvtbO9l6oI0t+48Hwa7DR+h3vxIT46KZnpvC9DynaWhmXgpTJ43uVUSj4WRzMVkNwhgT1jp7+thxsGOweWigdtBytGfwmMLMBKbnpnL9eZOZmZfC9NxUijITiYoKv1rB6bCAMMaEhYERx8eDwGkq2nX4CH1utSAhNpppuSlcNzvPCYK8VKblppAaZrWCYLGAMMaMO509few85NYK9g/0GbTR7FcryE9PYEZeKh+anTvYTFRstYLTYgFhjBnTVJ2xBevqWli7t4W1dc1s3d9Or1sriI+NYlpuKotm5zI91wmCabkp4+py0rHKAsIYM6a0Hu1hXX0L69wwWFfXMthfkDwhhvML01h2xRRmTU5jRl4KxVlJRFutwBMWEMaYkOnt62fbwfbjtYO9zdQ0HgFABKZOTGHRrFzmFaUzryiDc3KSLQxGkacBISKLgB/jrEn9uKo+FOCYW4AHAAXWq+pt7vY+YKN72F5VXexlWceK5ORkOjo6Ql0MYzxxqK2Td/a2uIHQzIb61sGRx1lJccwrSuejFxQwrzCdOQVpYXdJ6XjjWUCISDTwMHANUA9UichyVd3sd0wZcD9wqao2i8hEv6c4pqpzvSqfMcZbnT19bNrX6vYbOE1GDS3ONDGx0cLMyWncemEh84rSuaAoI2wHm41nXtYgFgA7VbUWQESeA5YAm/2O+RzwsKo2A6jqIQ/LExL33XcfhYWF3HXXXQA88MADxMTE8Le//Y3m5mZ6enr47ne/y5IlS0JcUmPOnKqyt+koa/1qB5v3t9HT53Qk56cnMK8onc9cVsq8onRm5qUSHxsd4lKbU/EyIPKBOr/H9cBFw46ZCiAib+E0Qz2gqn9298WLSDXQCzykqr8f/gIisgxYBlBUdIrVlP50HxzYePJjTlfuHLjuPa1mQ9x66618+ctfHgyI559/npUrV3LPPfeQmprK4cOHWbhwIYsXL7ZfT2bcaOvsYUNdK+vqmgdrCAOL2iTGRXNeQRr/cPkU5hWmM7conYkp8SEusTkToe6kjgHKgCuBAuANEZmjqi1Asao2iMgU4K8islFVa/xPVtXHgMfAmWpjdIs+MvPmzePQoUPs27ePxsZGMjIyyM3N5Stf+QpvvPEGUVFRNDQ0cPDgQXJzc0NdXGPeY6Ajeb0bCOvqWthxqGNworpzJyZz1fSJzCvKYF5ROmUTk4kJ8ToGJji8DIgGoNDvcYG7zV89sFpVe4BdIrIdJzCqVLUBQFVrReQ1YB5Qw5k6xS99L33sYx/jhRde4MCBA9x66608/fTTNDY2smbNGmJjYykpKQk4zbcxo01VqW8+xrq6FtbXtbC+voWNDa109jirnKUnxjKvMJ0PnzeZeUXpnFeQbuMNwpiXAVEFlIlIKU4wLAVuG3bM74GPA0+KSDZOk1OtiGQAR1W1y91+KfCvHpbVU7feeiuf+9znOHz4MK+//jrPP/88EydOJDY2lr/97W/s2bMn1EU0EarlaDfr61tZX9cyGAo+t6loQkwUs/PTuG1BMecXpjG3MJ2izERrCo0gngWEqvaKyN3ASpz+hSdUdZOIPAhUq+pyd98HRWQz0Ad8TVV9InIJ8KiI9ANROH0Qm0/wUmPerFmzaG9vJz8/n7y8PD7xiU9www03MGfOHMrLy5k+fXqoi2giQGdPH5v3tzk1g7oW1te3suvw8TEH5+Yk8/7pE5lbmM7cwnSm5aaEfMlLE1o23XcYibT3a06sv1+pPXzkeM2gvoUtflcV5abGc35hGue7YTAn38YcRCqb7tuYMHeorXMwCNbVtbChrpX2LmeRm+QJMYNXFZ1f4ARCbppdVWROzQLCmHHmSFcvG+pbWV/fMthctK/VucghJkqYnpfC4rmTB5uKptj0FOYMhX1AqGpEdKqFS1OhGar1aA/bD7Wz7UA7G+pbWF/Xyo5D7YOrnhVnJVJekuk2FaUxa3KaDUAzQRPWAREfH4/P5yMrKyusQ0JV8fl8xMdbs8F4dbS7lx0HO9h2sJ0dB9vZdrCD7QfaOdB2/PLnjMRYzi9M57o5uZxfmM75BelkJsWFsNQm3IV1QBQUFFBfX09jY2Ooi+K5+Ph4CgoKQl0McwpdvX3UNh5h+0GnVrD9YAfbD7ZT13x0cODZhJgoyiYlc8m5WUydlMK0SSmUTUomP93mKjKjK6wDIjY2ltLS0lAXw0Sg3r5+9jQdZfuBdrYdbGf7QScM/Je/jIkSpuQkcV5BGjfPL3DCIDeFosxE6zMwY0JYB4QxXuvvVxpajjk1goPtbHdrBTsbO+judUYfi0BxZiJTJzlrG0zNdWoFpdlJxMXYOAMzdllAGDMCqkpjexfbBpuGnH6CnQfbOdLdN3jc5LR4puamcFlZ9mDz0LkTk0mIs45jM/5YQBgTwKG2Tqp2N1O1u4nN+9rYfqh9cNlLgOzkOKZOSuFj5YVu01AyZZNSSLXBZiaMWECYiKeq1DR2DAZC9e5m9jYdBSA+NoqZealcNzuPaZOSmZqbwtRJKWQnTwhxqY3xngWEiTjdvf1sbGilencTVbubWbOniWa3dpCVFEd5SQafuriY8pJMZk1OtfmITMSygDBhr/VYD+/sbR4MhPV1LXS5Hcil2UlcPWMSF5ZkUl6SQWl2kl1KaozLAsKEnX0txwabiqp2N7HtYDuqEB0lzJ6cyu0Li7mwJIP5xZnkpFhTkTEnYgFhxrX+fmX7oXaqdjs1hOrdzTS0HAMgKS6aC4ozuG52HheWZDC3KJ3EOPsvb8xI2V+LGVc6e/rYUN/q1hCaWLOnmbZOZ9bSnJQJLCjJ5B8uL+XCkkym56bY0pfGnAULCDOmtRztZs2e5sEawob6Vrr7nP6Dcycmc/15eZQXZ3JhSSaFmTYVhTHBZAFhxpSOrl4qa3y8saORylof2w92ABAbLczJT+PTl5ZQXpLJ/OIMm6jOGI95GhAisgj4Mc6So4+r6kMBjrkFeABQYL2q3uZuvwP4hnvYd1X1KS/LakKjv1/ZtK+NN3Y08sb2Rt7Z20xPn5IQG82C0kyWzM2nvDiD8wvTbRprY0aZZwEhItHAw8A1QD1QJSLL/deWFpEy4H7gUlVtFpGJ7vZM4FtAOU5wrHHPbfaqvGb0HGrr5O87DvPGjkbe3HEY35FuAGbmpfLZy6ZwRVk280symBBjgWBMKHlZg1gA7FTVWgAReQ5YAmz2O+ZzwMMDX/yqesjdfi3wiqo2uee+AiwCnvWwvMYjnT19VO9u5u87Gnl9eyNbD7QDznQVV0zN4fKybC4ry2Ziiq1nYcaB/n7Y+QqsfgQOvAu5syFvLkyeB5PnQlqhM0NjGPAyIPKBOr/H9cBFw46ZCiAib+E0Qz2gqn8+wbn5w19ARJYBywCKioqCVnBzdgamrnh9+2H+7vYldPb0ExstlBdn8vVF07m8LJuZealE2bTWZrzoaod1z8DqR6GpBlLy4Jz3w6HNsOon0O9cTUdilhMW/qGRmj8uQyPUndQxQBlwJVAAvCEic0Z6sqo+BjwGUF5ebmtuhlDr0R7e3OkEwhvbGwfXSJ6SncTSC4u4vCybhVOySJoQ6v9yxpympl3w9i9g7X9BVxvkl8NNv4SZSyDanZyxpxMOboL9a2HfWti3Dmp+BOrO9JuUMzQwJs9zAmaMh4aXf60NQKHf4wJ3m796YLWq9gC7RGQ7TmA04ISG/7mveVZSc9p6+/pZX98yWEtYX9dCv0JKfAyXnpPN3R9wmo4KMxNDXVRjTp8q7H4TKv8Dtq2AqGiYeSMs/EcoKH/v8bHxUDDfuQ3oOeY0Qe1f5xcafwF1LtMmaeLQwMibC6l5o/P+Rki8WuxeRGKA7cBVOF/4VcBtqrrJ75hFwMdV9Q4RyQbWAnNxO6aBC9xD3wHmD/RJBFJeXq7V1dWevBfjqG8+yhvbD/PG9kbeqjlMe2cvUQLnFaRzxdQc3jc1m/ML0m1wmhm/ejph42+d/oWD70JCJpR/Bi78LKROPvvn7z7qPO9AYOxbC4e3HQ+N5NyhgTF5HqRMOvvXPQkRWaOqAVLPwxqEqvaKyN3ASpz+hSdUdZOIPAhUq+pyd98HRWQz0Ad8TVV9bqG/gxMqAA+eLByMN45291JZ6xsMhdrDRwDIS4vnQ7PzuGJqDpeem0V6oo1HCKr+fjhyCFrqoHWv+289xKdBZilkToGMUkjJHfNNFONG236oehzWPAlHfTBxFiz+Kcz5GMQmBO914hKhcIFzG9B9BA5sHBoa21fi/E7GaYoa3qeRPDF4ZToJz2oQo81qEMHR0HKM5ev28fcdjVTvbqa7r5/42CguKs3iiqk5XFGWzbkTk23E8tno64X2/dBaBy17/YLALwz6uoaeMyENujuOt2kDxCZCRokTFpmlQ8MjrRCirb/nlOrXwOr/gE0vQX8fTPsQLPwClFwe2vDt6oADG44Hxv51cHgHg6GRmj80MPLmQnLOGb3UyWoQFhAGVaV6TzNPvrWLlZsO0tevTM9NcQMhh/KSDBukdjp6u5wv+dY65wu/Ze/Q+20NQ7/owWmPTi90vtjTi5zb4P1CmJACfT3O+c27nI7Tpl3u/Vpo3g29ncefLyrGOTfDDQ3/8MgocdrMI1VfD2z+g9OMVF8FE1Jh3u2w4HPOZzRWdba9NzR8O519k2bDP751Rk8bkiYmM/Z19fbx8vr9/GrVLt5taCM1PoZ/uKyU2xcWW+fyyXQfcX/pD9QA/AKgtQ7aDzD4Sw9AopxmgvQiKFrofOH7B0BawciaMaJjIesc5zZcf79TKxkIjKaB4NjlfAl2tfkdLE57eqCaR2ap05QVjo42OU1Ibz8O7fuc93zdv8Lc25wAHuviU6HkMuc2oLMV9m9wfpR4wGoQEehQeye/rtzLM6v3cLijm3MnJvPpS0v4yLx8mw57uG1/ht1/HxoER31Dj4mKhbR89wu/2K8m4AZAav7xyyFDQdX5cgwUHk27nP4Of4lZgWsemaXO5ZrjrXnx4GanGWnD804ta8qVsPCLcO41EGUXVFgNwgCwob6FJ9/azR837KOnT/nA9Il8+tISLjs32/oUhjvaBCv+P3j3RYiJP/5rf2CkrH8QpOQ6l0GOVSKQlOXcAl2i2dXuNFEND4+9lc4VPf61obhkNyxKhgZHRqlTExorn0N/P+xYCZU/h11vQEwCnL8ULvoCTJwR6tKNGxYQYa6nr5+Vmw7w5Fu7WbOnmaS4aD5xUTF3XFJCaXZSqIs3Nu14Bf5wNxw9DO//33DZP4V3h++EFMid49yG6+1yak7Dw+PQVudKm77u48dGxUJG8dDQGKiFpBePTr9HZxuse9oZ7dy8y6m9Xf0AXHAHJGZ6//phJoz/10e25iPdPPP2Xn5duYf9rZ0UZyXyzQ/P5GPlBaTEh7C5YyzraoeV/xveeQpypsNtv3FqDJEsZgJklzm34fr7nA73wc5yv07zvZXQ3e53cIB+D/8gSUg/u3L6auDtx2Dt087rFl4EV30TZtwQ2ua9cc4CIsxsPdDGr97azUtrG+jq7efSc7P4zpLZvH/6RKJt3qMT27MKXvqC82v5ki/B+78R2Vf6jERU9PErrnjf0H2qTl+Nf3gM1D62r3xvv0dC5ntrHQMBkjwpcL+HKux6HSofge1/dq7cmv1Rpxkp/4L3Hm9OmwVEGOjrV/669RBPvrWLVTU+4mOj+OgFBdx5SQnTcsfB1Rmh1NMJf/surPqZ80X36RVQfEmoSzX+iUBStnMrvPC9+7s6jvd7+Nc+6qtg0++OjywGd7zHQG2jxPlXFaqfcCbKS8yGK77mjHZOyR2tdxgRLCDGsbbOHp6vquM/K/awt+kok9Pi+fqi6Sy9sJAMW23t1Patc2oNjVtg/p3wwe+Oj8sdw8GEZGea7NzZ793X2+1cLTa89uHbCTtfPT7eI3cOLPk5zL7JansesYAYh2obO3hq1W5eWFPPke4+yosz+Pqi6Vw7a5LNgzQSfb3w5g/h9X9xfn1+4gUouybUpTIDYuJOPt6j44DTX5Q9dfxdcjvOWECME6rKGzsO8+Rbu3htWyOx0cIN503m05eWMqcgiAObWuudDsaya8JzwFTjdnjp87DvHZh9M3zoB3Z1y3gSFRWcSfPMiFhAjHFHu3t58Z0GfvXWLmoaj5CdPIEvX13GbRcVBX8Ftu0r4XfLoLMF4lLggk8589Kkh8FiTP39ztQKf/m2M2r55iedDk1jzAlZQIxR9c1H+c+KPTz39l7aOnuZk5/GD285n+vPywv+Ws19vU5H7Zs/gklz4COPOgPE3n7U+VKduRgu/tLQue7Hk+Y98Ie7nBHRZdfC4p9YZ6YxI2ABMYaoKqt3NfGrt3bzP5sPICIsmp3LZy4t4YKiDG9GO7cfgBc+C3vedAYTXfcvzi/saYvg6m85A47WPOXMdll0MVx8lzPj5VgZMXsyqs6gqT/dB6gzffO8T1q7tTEjZHMxjRG+ji7ufLKKjQ2tpCfG8vEFRXxyYTGT04M4F/1wu95wwqG7Az78I2cqgkC62mHtr51pC1r2OtepL/yiM8lZ3Bgdjd1+EF6+F7b/CYovgxsfdi6RNMYMYXMxjQMrNu5nY0Mr3148i1vKC0mI8/AXen8/vPl/4W/fh6xz4Y7lJ5+fZkKKs9TihZ+DrX+Eip858xT99bvOalsLlo2tpRI3/R7++BVn1tVrvw8X/aNNymbMGbCAGCNW1fiYnBbPpy4u9nbivKNNTkf0zlecq3hu+LFzTfpIRMfArBud297VTlC89e+w6qcw52an+SnQfD6j5VgzrPhfsPF5ZwGVjzwKE6eHrjzGjHOeBoS75vSPcZYcfVxVHxq2/07gBzhrVgP8TFUfd/f1ARvd7XtVdbGXZQ2l/n6lstbHB6ZP8jYc6qrgt3c60xxc/3+h/LNn3h5fdJFza6p1pjpY+2tY/yyUvs+ZquLcq0e3rX/nq/CHL0HHQbjyfrj8qzYHjzFnybOAEJFo4GHgGqAeqBKR5aq6edihv1HVuwM8xTFVjYiZ0rYdbKf5aA8Xn5PlzQuoOlcj/c83nGvIP7MyeHPVZE6BD/0rvP9+WPMrp1P76Zudye4WfhHOu9XbUa5dHfDK/3GmXcieBkuftnl4jAkSLxtmFwA7VbVWVbuB54AlHr7euFVR4yxA40lAdLbC85+CP98HZR+Ez7/hzRdoQgZc9hW4dwN85DHn1/vL98CPZsFrD8GRw8F/zb2V8MhlUP0kXHw3fP51CwdjgsjLgMgH6vwe17vbhrtJRDaIyAsiUui3PV5EqkWkUkRuDPQCIrLMPaa6sbExiEUfXRW1PooyE8kP9hVLBzbCY1fC1v+Gax6Epc84X+ReiomD82+Fz/8dPrXc+cJ+7f93guLle52RzGertwte+SY8schZ2/nOP8K13xvZsp3GmBELdSf1y8CzqtolIp8HngI+4O4rVtUGEZkC/FVENqpqjf/JqvoY8Bg4l7mOZsGDpa9fWV3r47rZQbwKSBXW/hes+JoTCHf+cfRnKBWBKe9zbo3boOJhWPes0wxVdi1ccjeUXH76/RT7NzgT7B3a5Iz0vvb7NsGeMR7xsgbRAPjXCAo43hkNgKr6VHVgte3Hgfl++xrcf2uB14B5HpY1ZLbsb6Otszd4zUvdR+H3X4TlX3IWTfn830M/fXXONGf08lc2OR3IDWvgqRvg0Stg/W+c2TtPpa8X3vgB/OIDzkpvtz3vDHyzcDDGM14GRBVQJiKlIhIHLAWW+x8gIv4/mxcDW9ztGSIywb2fDVwKDO/cDgurapy2+aAExOEd8PhVztVE7/s6fPIlSM45++cNluQcuPI++Mq7cMNPnKail5bBj893pvk41hz4vMM74YlrnXEXM26AL1bC1GtHt+zGRCDPmphUtVdE7gZW4lzm+oSqbhKRB4FqVV0O3CMii4FeoAm40z19BvCoiPTjhNhDAa5+CgsVNT6m5CQxKfUsr/R590VYfo+zROTtL8K5VwWngF6ITYD5dzjTXux8FSp+Cq8+AK//AObd7gzKyyx1BvRV/QJe+Zbzvm76pTPewhgzKmyqjRDq7etn7oOvsGTuZL73kTMcYNbbBSv/Gaoed5qUbn4S0gJdCzDG7d/g9FO8+4Kzmtj0DztXYO16Hc69xmlOGkujtY0JEzbVxhi1saGVjq6z6H9o3gO/vQP2rXUu87z6gfE7OCzvPPjoo84EgW8/5oxr6O9zRnpfcIdNsGdMCFhAhFBFrTP+YeGUMwiIbX9yFr5R4P8+gy8AABR3SURBVNanYcaHg1u4UEmd7ATd+77u1CTG6mSAxkQAC4gQqqjxMXVSMtnJE0Z+Ul8v/PVBeOvHkHse3PKUM5o53NiYBmNCzgIiRLp7+6ne3cwt5QUjP6ltP7zwGdi7CuZ/GhY9ZIu1G2M8YwERIuvrWzjW0zfy/ofa1+HFzzpTWH/kMWe0sjHGeMgCIkQqanyIwEWlpwiI/n74+78501VklcEdf7QprI0xo8ICIkQqanzMyE0lIynuxAcd8cHvPgc1f4E5tzirvo107QZjjDlLFhBdHfAfl0DyJEie6Pevez9p4vHHQeo47ezpY83eZj65sPjEB9W97a7d0OgEw/xP26WexphRZQHR1w2FC5yFZnw1sGcVHGsKfOyENGe6CP8wSRp4POn4vqSck45HWLu3he7efi4OdHmrqrP28yvfhNR8+OwrMDkilsUwxowxFhCJmXDT40O39XY7v9yPHIKOQ054dPjdP9LoTKXd8RfoajvB82b51T6G1koatvcwM6qDiybNc/oYBtZL7myFP9wFW152RhIveRgS0r19/8YYcwI21cbZ6jk2LDxOECodh6D32HvPl2hIynbC44jPOfaabzsjo61JyRjjMZtqw0uxCZBR7NxORhW6O+hs3s+dP/sjt0yfwEfLYoeGSmIWfOxJKFo4OmU3xpiTsIAYLSIwIYWq9k4qe6fx+QsvhGkTQ10qY4w5IS/XgzABVNT4iIkSLizJDHVRjDHmpCwgRllFrY/zCtJInmCVN2PM2GYBMYo6unrZUN8avOVFjTHGQxYQo6hqdxN9/crFU7JDXRRjjDklTwNCRBaJyDYR2Ski9wXYf6eINIrIOvf2D3777hCRHe7tDi/LOVoqa3zERgvzizNCXRRjjDklzxrCRSQaeBi4BqgHqkRkeYC1pX+jqncPOzcT+BZQjrMkzhr33BOsaj8+VNT6mFeYQUJcdKiLYowxp+RlDWIBsFNVa1W1G3gOWDLCc68FXlHVJjcUXgEWeVTOUdF6rId3G1pZaP0PxphxwsuAyAfq/B7Xu9uGu0lENojICyJSeDrnisgyEakWkerGxsZgldsTb+9qol/hEgsIY8w4cUYBISLBmnP6ZaBEVc/DqSU8dTonq+pjqlququU5OTlBKpI3Kmp8TIiJYl6Rza1kjBkfzrQGMbwfIZAGoNDvcYG7bZCq+lS1y334ODB/pOeONxW1PuYXZzAhxvofjDHjwwk7qUXkn060CxhJDaIKKBORUpwv96XAbcNeI09V97sPFwNb3Psrge+LyMDlPh8E7h/Ba45JzUe62bK/ja9eMzXURTHGmBE72VVM3wd+APQG2HfKmoeq9orI3Thf9tHAE6q6SUQeBKpVdTlwj4gsdl+jCbjTPbdJRL6DEzIAD6rqCRZpGPtW7/IB2AA5Y8y4crKAeAf4vaquGb7Df7zCyajqCmDFsG3f9Lt/PyeoGajqE8ATI3mdsa6ixkdCbDTnFVj/gzFm/DhZTaAB2CMi9wbYF3DucBPYqhof5SUZxMXYwHVjzPhxsm+smUAc8BkRyRCRzIEb0DM6xRv/Gtu72HGog0vOsek1jDHjy8mamB4F/gJMAdbgdE4PUHe7OYXKWut/MMaMTyesQajqT1R1Bk7n8hRVLfW7WTiMUEWtj+QJMcyenBrqohhjzGkZydVI/zgaBQlXlTU+FpRmEhNt/Q/GmPHFvrU8dLCtk9rDR7h4ijUvGWPGHwsID1XUWP+DMWb8soDwUEWNj9T4GGbkWf+DMWb8sYDw0Kraw1w0JYvoKDn1wcYYM8ZYQHikvvkodU3HbHpvY8y4ZQHhEet/MMaMdxYQHqmo9ZGZFMfUiSmhLooxxpwRCwgPqCqVNT4WTskkyvofjDHjlAWEB/Y2HWVfa6eNfzDGjGsWEB6w/gdjTDiwgPDAqhofOSkTOCcnWEt3G2PM6LOACDJVpaLWx8VTshCx/gdjzPjlaUCIyCIR2SYiO0XkvpMcd5OIqIiUu49LROSYiKxzb494Wc5gqmk8QmN7lzUvGWPGvZOtB3FWRCQaeBi4BqgHqkRkuapuHnZcCnAvsHrYU9So6lyvyueVioH1H6yD2hgzznlZg1gA7FTVWlXtBp4DlgQ47jvAvwCdHpZl1FTW+MhLi6c4KzHURTHGmLPiZUDkA3V+j+vdbYNE5AKgUFX/O8D5pSKyVkReF5HLPSxn0Kgqldb/YIwJE541MZ2KiEQBPwTuDLB7P1Ckqj4RmQ/8XkRmqWrbsOdYBiwDKCoq8rjEp7b9YAe+I90stP4HY0wY8LIG0QAU+j0ucLcNSAFmA6+JyG5gIbBcRMpVtUtVfQCqugaoAaYOfwFVfUxVy1W1PCcnx6O3MXKrag4D1v9gjAkPXgZEFVAmIqUiEgcsBZYP7FTVVlXNVtUSVS0BKoHFqlotIjluJzciMgUoA2o9LGtQVNT4KMxMoDDT+h+MMeOfZwGhqr3A3cBKYAvwvKpuEpEHRWTxKU6/AtggIuuAF4AvqGqTV2UNhv5+ZfWuJqs9GGPChqd9EKq6AlgxbNs3T3DslX73XwRe9LJswbZ5fxutx3ps/IMxJmzYSOogqRwc/5Ad4pIYY0xwWEAESUWNj9LsJHLT4kNdFGOMCQoLiCDo7evn7V1NLLT+B2NMGLGACIJ397XR3tVr/Q/GmLBiAREEg+s/WA3CGBNGLCCCoKLWR9nEZHJSJoS6KMYYEzQWEGepp6+f6t1N1rxkjAk7FhBnaUN9C0e7+6x5yRgTdiwgztJA/8NFFhDGmDBjAXGWKmp9TM9NITMpLtRFMcaYoLKAOAtdvX1U7262/gdjTFiygDgLa/e20NXbzyXn2PQaxpjwYwFxFipqfEQJLCjNDHVRjDEm6CwgzkJFrY9Zk9NIS4gNdVGMMSboLCDOUGdPH+v2tlj/gzEmbFlAnKE1e5rp7uu38Q/GmLBlAXGGKmp8REcJF1r/gzEmTFlAnKGKWh9z8tNInuDponzGGBMyngaEiCwSkW0islNE7jvJcTeJiIpIud+2+93ztonItV6W83Qd6eplfZ31PxhjwptnP39FJBp4GLgGqAeqRGS5qm4edlwKcC+w2m/bTGApMAuYDLwqIlNVtc+r8p6Oqt1N9PYrl1hAGGPCmJc1iAXATlWtVdVu4DlgSYDjvgP8C9Dpt20J8JyqdqnqLmCn+3xjQkWtj9hoobzY+h+MMeHLy4DIB+r8Hte72waJyAVAoar+9+me656/TESqRaS6sbExOKUegcoaH3ML00mIix611zTGmNEWsk5qEYkCfgh89UyfQ1UfU9VyVS3PyckJXuFOoq2zh40NrXZ5qzEm7Hl5CU4DUOj3uMDdNiAFmA28JiIAucByEVk8gnNDpmpXE/0KC63/wRgT5rysQVQBZSJSKiJxOJ3Oywd2qmqrqmaraomqlgCVwGJVrXaPWyoiE0SkFCgD3vawrCNWUeMjLiaKC4oyQl0UY4zxlGc1CFXtFZG7gZVANPCEqm4SkQeBalVdfpJzN4nI88BmoBe4a6xcwbSqxscFRenEx1r/gzEmvHk6yktVVwArhm375gmOvXLY4+8B3/OscGeg5Wg3Ww608ZWrp4a6KMYY4zkbSX0aKmubUMUGyBljIoIFxGmorPWREBvN+QXpoS6KMcZ4zgLiNFTU+CgvySAuxj42Y0z4s2+6EfJ1dLHtYDsLbfyDMSZCWECMUGVtE2D9D8aYyGEBMUKrag6TFBfNnPy0UBfFGGNGhQXECFXU+lhQmklstH1kxpjIYN92I3CwrZPaxiPWvGSMiSgWECNQWesD4OIp2SEuiTHGjB4LiBGoqPGRGh/DzMmpoS6KMcaMGguIEXD6H7KIjpJQF8UYY0aNBcQp7Gs5xh7fUet/MMZEHAuIU6ioGeh/sIAwxkQWC4hTWFXjIyMxlum5KaEuijHGjCoLiJNQVSprfSyckkWU9T8YYyKMBcRJ1DUdo6HlmPU/GGMikgXESVTUHgas/8EYE5k8DQgRWSQi20Rkp4jcF2D/F0Rko4isE5E3RWSmu71ERI6529eJyCNelvNEKmp8ZCdP4NyJyaF4eWOMCSnPlhwVkWjgYeAaoB6oEpHlqrrZ77BnVPUR9/jFwA+BRe6+GlWd61X5TkVVqaj1sXBKJiLW/2CMiTxe1iAWADtVtVZVu4HngCX+B6hqm9/DJEA9LM9p2XX4CAfbuqz/wRgTsbwMiHygzu9xvbttCBG5S0RqgH8F7vHbVSoia0XkdRG5PNALiMgyEakWkerGxsZglp1V7viHS86x+ZeMMZEp5J3Uqvqwqp4DfB34hrt5P1CkqvOAfwKeEZH3TISkqo+parmqlufk5AS1XBW1PnJT4ynJSgzq8xpjzHjhZUA0AIV+jwvcbSfyHHAjgKp2qarPvb8GqAGmelTO91BVVtf6uPicLOt/MMZELC8DogooE5FSEYkDlgLL/Q8QkTK/h9cDO9ztOW4nNyIyBSgDaj0s6xA7DnVwuKPbLm81xkQ0z65iUtVeEbkbWAlEA0+o6iYReRCoVtXlwN0icjXQAzQDd7inXwE8KCI9QD/wBVVt8qqsww3Ov2Qd1MaYCOZZQACo6gpgxbBt3/S7f+8JznsReNHLsp1MRY2P/PQECjOt/8EYE7lC3kk91vT3K5W7fFZ7MMZEPAuIYbYcaKPlaI/1PxhjIp4FxDDW/2CMMQ4LiGEqa32UZCUyOT0h1EUxxpiQsoDw09evrN7VZLUHY4zBAmKITftaae/sZaH1PxhjjAWEP1t/2hhjjrOA8FNR6+OcnCQmpsaHuijGGBNyFhCunr5+qqz/wRhjBllAuDbUt3Kku8+m9zbGGJcFhKuy1ul/sA5qY4xxWEC4Kmp8TM9NITMpLtRFMcaYMcECAujq7aN6T5PVHowxxo8FBLC+rpXOnn7roDbGGD8WEDjNSyKwsNQCwhhjBlhAABW1h5mZl0paYmyoi2KMMWNGxAdEZ08f7+xp4RJrXjLGmCE8DQgRWSQi20Rkp4jcF2D/F0Rko4isE5E3RWSm37773fO2ici1XpWxrbOH6+bk8v5pE716CWOMGZdEVb15YpFoYDtwDVAPVAEfV9XNfsekqmqbe38x8EVVXeQGxbPAAmAy8CowVVX7TvR65eXlWl1d7cl7McaYcCUia1S1PNA+L2sQC4Cdqlqrqt3Ac8AS/wMGwsGVBAyk1RLgOVXtUtVdwE73+YwxxoySGA+fOx+o83tcD1w0/CARuQv4JyAO+IDfuZXDzs0PcO4yYBlAUVFRUAptjDHGEfJOalV9WFXPAb4OfOM0z31MVctVtTwnJ8ebAhpjTITyMiAagEK/xwXuthN5DrjxDM81xhgTZF4GRBVQJiKlIhIHLAWW+x8gImV+D68Hdrj3lwNLRWSCiJQCZcDbHpbVGGPMMJ71Qahqr4jcDawEooEnVHWTiDwIVKvqcuBuEbka6AGagTvcczeJyPPAZqAXuOtkVzAZY4wJPs8ucx1tdpmrMcacvlBd5mqMMWYcC5sahIg0AnvO4imygcNBKs54Z5/FUPZ5DGWfx3Hh8FkUq2rAy0DDJiDOlohUn6iaFWnssxjKPo+h7PM4Ltw/C2tiMsYYE5AFhDHGmIAsII57LNQFGEPssxjKPo+h7PM4Lqw/C+uDMMYYE5DVIIwxxgRkAWGMMSagiA+IU616F0lEpFBE/iYim0Vkk4jcG+oyhZqIRIvIWhH5Y6jLEmoiki4iL4jIVhHZIiIXh7pMoSQiX3H/Tt4VkWdFJD7UZQq2iA4Id9W7h4HrgJnAx/2XPY1AvcBXVXUmsBC4K8I/D4B7gS2hLsQY8WPgz6o6HTifCP5cRCQfuAcoV9XZOPPNLQ1tqYIvogOCEax6F0lUdb+qvuPeb8f5AnjPQk2RQkQKcGYZfjzUZQk1EUkDrgB+CaCq3araEtpShVwMkCAiMUAisC/E5Qm6SA+IQKveRewXoj8RKQHmAatDW5KQ+nfgfwH9oS7IGFAKNAJPuk1uj4tIUqgLFSqq2gD8G7AX2A+0qur/hLZUwRfpAWECEJFk4EXgy8PWDY8YIvJh4JCqrgl1WcaIGOAC4D9UdR5wBIjYPjsRycBpbSgFJgNJInJ7aEsVfJEeELZy3TAiEosTDk+r6u9CXZ4QuhRYLCK7cZoePyAivw5tkUKqHqhX1YEa5Qs4gRGprgZ2qWqjqvYAvwMuCXGZgi7SA+KUq95FEhERnDbmLar6w1CXJ5RU9X5VLVDVEpz/F39V1bD7hThSqnoAqBORae6mq3AW9IpUe4GFIpLo/t1cRRh22nu2otx4cKJV70JcrFC6FPgksFFE1rnb/llVV4SwTGbs+BLwtPtjqhb4dIjLEzKqulpEXgDewbn6by1hOO2GTbVhjDEmoEhvYjLGGHMCFhDGGGMCsoAwxhgTkAWEMcaYgCwgjDHGBGQBYcwYICJX2oyxZqyxgDDGGBOQBYQxp0FEbheRt0VknYg86q4X0SEiP3LXBviLiOS4x84VkUoR2SAiL7nz9yAi54rIqyKyXkTeEZFz3KdP9ltv4Wl3hK4xIWMBYcwIicgM4FbgUlWdC/QBnwCSgGpVnQW8DnzLPeU/ga+r6nnARr/tTwMPq+r5OPP37He3zwO+jLM2yRScke3GhExET7VhzGm6CpgPVLk/7hOAQzjTgf/GPebXwO/c9RPSVfV1d/tTwG9FJAXIV9WXAFS1E8B9vrdVtd59vA4oAd70/m0ZE5gFhDEjJ8BTqnr/kI0i/2fYcWc6f02X3/0+7O/ThJg1MRkzcn8BbhaRiQAikikixTh/Rze7x9wGvKmqrUCziFzubv8k8Lq7Ul+9iNzoPscEEUkc1XdhzAjZLxRjRkhVN4vIN4D/EZEooAe4C2fxnAXuvkM4/RQAdwCPuAHgP/vpJ4FHReRB9zk+Nopvw5gRs9lcjTlLItKhqsmhLocxwWZNTMYYYwKyGoQxxpiArAZhjDEmIAsIY4wxAVlAGGOMCcgCwhhjTEAWEMYYYwL6f2xfEzlU0aSAAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(model.history.history.keys())\n",
    "# summarize history for accuracy\n",
    "plt.plot(model.history.history['f1'])\n",
    "plt.plot(model.history.history['val_f1'])\n",
    "plt.title('model f1')\n",
    "plt.ylabel('f1')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'val'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Домашка\n",
    "\n",
    "\n",
    "Обучите модель с минимум 15 слоями, где у каждого слоя разные параметры (Dropout, Conv1d и Pooling считаются слоями, остальное нет). Как мимнимум 4 слоя должны быть наложены друг на друга и как минимум 2 параллельных слоя (последовательности слоев). Должен быть хотя бы один слой каждого типа.\n",
    "\n",
    "При обучении используйте колбек для отслеживания лучшей модели. Ориентируйтесь на ф1 меру. Должно получиться сопоставимое качество с моделями выше.\n",
    "\n",
    "Советы: Начните с небольших сетей и постепенно добавляйте, не пытайтесь сразу собрать все слои. Иногда кернел может крашиться просто так или из-за слишком больших матриц.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}