notebooks/word_embeddings/nn_intro_torch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Дисклеймер\n",
    "Эту тетрадку лучше запускать в колабе или в vast.ai. Не мучатесь с установкой библиотек и с обучением на cpu."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting pandas\n",
      "  Downloading pandas-1.2.4-cp38-cp38-manylinux1_x86_64.whl (9.7 MB)\n",
      "\u001b[K     |████████████████████████████████| 9.7 MB 14.9 MB/s eta 0:00:01     |███████████████▎                | 4.6 MB 14.9 MB/s eta 0:00:01███████████████████████▉      | 7.8 MB 14.9 MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting scikit-learn\n",
      "  Downloading scikit_learn-0.24.1-cp38-cp38-manylinux2010_x86_64.whl (24.9 MB)\n",
      "\u001b[K     |████████████████████████████████| 24.9 MB 18.1 MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting gensim\n",
      "  Downloading gensim-4.0.1-cp38-cp38-manylinux1_x86_64.whl (23.9 MB)\n",
      "\u001b[K     |████████████████████████████████| 23.9 MB 100.6 MB/s eta 0:00:01                      | 2.0 MB 100.6 MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting matplotlib\n",
      "  Downloading matplotlib-3.4.1-cp38-cp38-manylinux1_x86_64.whl (10.3 MB)\n",
      "\u001b[K     |████████████████████████████████| 10.3 MB 20.1 MB/s eta 0:00:01    |███████▌                        | 2.4 MB 20.1 MB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: numpy>=1.16.5 in /opt/conda/lib/python3.8/site-packages (from pandas) (1.19.2)\n",
      "Requirement already satisfied: pytz>=2017.3 in /opt/conda/lib/python3.8/site-packages (from pandas) (2021.1)\n",
      "Requirement already satisfied: python-dateutil>=2.7.3 in /opt/conda/lib/python3.8/site-packages (from pandas) (2.8.1)\n",
      "Collecting joblib>=0.11\n",
      "  Downloading joblib-1.0.1-py3-none-any.whl (303 kB)\n",
      "\u001b[K     |████████████████████████████████| 303 kB 66.9 MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting scipy>=0.19.1\n",
      "  Downloading scipy-1.6.2-cp38-cp38-manylinux1_x86_64.whl (27.2 MB)\n",
      "\u001b[K     |████████████████████████████████| 27.2 MB 21.3 MB/s eta 0:00:01�██████████████████         | 19.5 MB 21.3 MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting threadpoolctl>=2.0.0\n",
      "  Downloading threadpoolctl-2.1.0-py3-none-any.whl (12 kB)\n",
      "Collecting smart-open>=1.8.1\n",
      "  Downloading smart_open-5.0.0-py3-none-any.whl (56 kB)\n",
      "\u001b[K     |████████████████████████████████| 56 kB 5.5 MB/s  eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: pillow>=6.2.0 in /opt/conda/lib/python3.8/site-packages (from matplotlib) (8.1.2)\n",
      "Collecting kiwisolver>=1.0.1\n",
      "  Downloading kiwisolver-1.3.1-cp38-cp38-manylinux1_x86_64.whl (1.2 MB)\n",
      "\u001b[K     |████████████████████████████████| 1.2 MB 67.0 MB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: pyparsing>=2.2.1 in /opt/conda/lib/python3.8/site-packages (from matplotlib) (2.4.7)\n",
      "Collecting cycler>=0.10\n",
      "  Downloading cycler-0.10.0-py2.py3-none-any.whl (6.5 kB)\n",
      "Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas) (1.15.0)\n",
      "Installing collected packages: pandas, joblib, scipy, threadpoolctl, scikit-learn, smart-open, gensim, kiwisolver, cycler, matplotlib\n",
      "Successfully installed cycler-0.10.0 gensim-4.0.1 joblib-1.0.1 kiwisolver-1.3.1 matplotlib-3.4.1 pandas-1.2.4 scikit-learn-0.24.1 scipy-1.6.2 smart-open-5.0.0 threadpoolctl-2.1.0\n"
     ]
    }
   ],
   "source": [
    "!pip install pandas scikit-learn gensim matplotlib"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Введение в нейронные сети "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Все задачи, которые мы решали до этого, можно решать с помощью нейронных сетей. Часто нейронные сети будут наилучшим выбором, особенно когда у нас текстовые или картиночные данные и их достаточно много (примерно от 50к). Но в любом случае, перед тем как доставать нейронки лучше прогнать несколько стандартных моделей. Часто они решат задачу на приемлимом уровне и время, затраченное на обучение нейронки, не окупит себя. Также с табличными данными лучше просто использовать бустинги."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Для написания нейронок есть два основных фреймворка -  tensorflow и pytorch. Сказать, что лучше, тяжело. В целом, тензорфлоу с встроенным keras (раньше был отдельным фреймворком) сильно проще для новичков, но многое в нём происходит \"под капотом\". Когда все работает это даже хорошо, но когда что-то ломается, очень сложно понять, что именно.  Pytorch сложнее в освоении и многие вещи там приходится делать вручную, но это заставляет разобраться в технических деталях и дает большую свободу. \n",
    "\n",
    "\n",
    "В этой тетрадке код написан на торче"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Для наглядности возьмем две разные задачи - классификации и регресии. В обоих задачах мы будем работать с текстом и по большей части весь пайплайн у нас будет одинаков. Отличаться будут только последние штрихи в самих моделях."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Регрессия на данных Avito"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Данные из соревнования - https://www.kaggle.com/c/avito-demand-prediction/overview  \n",
    "Я взял только поле description, так как наша задача разобраться с нейронками, а не добиться самого большого качества."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from string import punctuation\n",
    "from sklearn.model_selection import train_test_split\n",
    "from collections import Counter\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Во всех предыдущих семинарах мы обрезали данные Авито, так как модели не могли столько переварить. В случае нейронок такой проблемы нет, модель обучается на батчах (кусках данных) и можно использовать неограниченное количество данных. Но такие большие датасеты лучше не грузить в память целиком и генерировать по необходимости.\n",
    "\n",
    "Чтобы разобрать предобработку пока возьмем 10% от данных, а позже напишем логику генерации."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "avito = pd.read_csv('avito_data.csv').dropna().sample(frac=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(138715, 2)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "avito.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Предобработка достаточно простая. Главное - получить из текстов токены."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess(text):\n",
    "    tokens = text.lower().split()\n",
    "    tokens = [token.strip(punctuation) for token in tokens]\n",
    "    return tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Теперь нам нужно собрать все уникальные слова в словарь. Лучше сразу посчитать количество упоминаний, чтобы отсеять самые редкие."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab = Counter()\n",
    "\n",
    "for text in avito.description:\n",
    "    vocab.update(preprocess(text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "почти 300к слов - это много, учитывая, что это только 10% данных"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1429855"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Возьмем только те, что встретились больше 30 раз."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "filtered_vocab = set()\n",
    "\n",
    "for word in vocab:\n",
    "    if vocab[word] > 30:\n",
    "        filtered_vocab.add(word)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "11к - уже нормальная цифра"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "54266"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(filtered_vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Теперь нам нужно заменить в каждом тексте слова на числа (индексы в словаре). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Создадим для этого специальный словарь с индексами. В этот словарь нам нужно положить дополнительный токен - PAD (чуть ниже станет понятнее)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "word2id = {'PAD':0}\n",
    "\n",
    "for word in filtered_vocab:\n",
    "    word2id[word] = len(word2id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "В нашей задаче нам не понадобится раскодировать индексы, но если бы мы делали языковую модель, нам бы понадобился ещё и обратный словарь."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "id2word = {i:word for word, i in word2id.items()}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "А теперь про паддинг. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Все тексты в нашем трейне разной длины и это проблема, так как модель ожидает последовательности одной длины. Для этого есть паддинг - т.е. забивание последовательности нулями до определенной длинны (мы не просто так присвоили PAD нулевой индекс - это важно)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Лучше всего сделать паддиг до самого длинного текста, но это зачастую не оптимально или даже невозможно из-за нехватки ресурсов. Поэтому значение паддинга приходится подбирать на глаз. \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_LEN = 0\n",
    "\n",
    "for text in avito.description:\n",
    "    tokens = preprocess(text)\n",
    "    MAX_LEN = max(len(tokens), MAX_LEN)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "711"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "MAX_LEN # реальная максимальная длина"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Значение в 100 кажется адекватным "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_LEN = 100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Паддинг в торче устроен немного по-другому (по сравнении с tf). Воспроизвести такой-же результат как и в tf можно двумя способами. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Паддинг каждого в отдельности"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "X = torch.LongTensor(size=(avito.shape[0], 100))\n",
    "\n",
    "for i, text in enumerate(avito.description):\n",
    "    tokens = preprocess(text) # токенизируем\n",
    "    \n",
    "    ids = [word2id[token] for token in tokens if token in word2id][:MAX_LEN] # заменяем токены на индексы в словаре\n",
    "                                                                             # и сразу обрезаем если > MAX_LEN\n",
    "    \n",
    "    # паддим до 100 справа\n",
    "    # нужно каждый раз указывать сколько 0 добавлять\n",
    "    # для этого вычитаем из 100 длину текущего текста\n",
    "    # \n",
    "    ids = torch.nn.functional.pad(torch.LongTensor(ids), \n",
    "                            (0, MAX_LEN-len(ids)), # можно задать паттерн паддинга туплом - \n",
    "                                                           #  (паддинг слева, паддинг справа); \n",
    "                                                           # еще можно задать паддинг с 4-х сторон - \n",
    "                                                           # (слева, справа, сверху, снизу)\n",
    "                            mode='constant', # еще можно по-другому паддить (не нулями), об этом на следующем занятии\n",
    "                            value=0)\n",
    "    X[i] = ids"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([138715, 100])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.size() # в торче не shape а size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Паддинг сразу всех и обрезание"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = []\n",
    "\n",
    "for text in avito.description:\n",
    "    tokens = preprocess(text) # токенизируем\n",
    "    ids = torch.LongTensor([word2id[token] for token in tokens if token in word2id]) # заменяем токены на индексы\n",
    "                                                                                     # переводим в тензор\n",
    "    X.append(ids)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = torch.nn.utils.rnn.pad_sequence(X, batch_first=True) # паддим все до максимальной длины"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([138715, 456])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.size() # значение отличается от MAX_LEN выше из-за того, что не все слова есть в словаре"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = X[:, :MAX_LEN] # обрезаем до нужной длины"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([138715, 100])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.size()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Чтобы итерироваться по данным нужно создать вот такой класс, в котором будет функция \\_\\_get_item\\_\\_\n",
    "\n",
    "В нее будет передавать индекс и мы можем задать вручную, что по этому индексу будет возвращаться  \n",
    "\n",
    "В этот класс мы вставим предобработку, которую делали выше. Так мы сможем генерировать данные батчами и не грузить все целиком."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Dataset(torch.utils.data.Dataset):\n",
    "\n",
    "    def __init__(self, dataset, word2id, MAX_LEN, text_field, target_field):\n",
    "        \n",
    "        # храним сырой датасет\n",
    "        # еще лучше можно сделать, если считывать из файла при генерации\n",
    "        self.dataset = dataset[text_field].values\n",
    "        \n",
    "        # мы не считаем словарь прямо тут, т.к. нам нужно шерить словарь между трейном и валидацией\n",
    "        self.word2id = word2id\n",
    "        self.MAX_LEN = MAX_LEN\n",
    "        self.length = dataset.shape[0]\n",
    "        # таргеты можно сразу целиком выделить \n",
    "        self.target = torch.Tensor(dataset[target_field].values)\n",
    "\n",
    "    def __len__(self):\n",
    "        # нужно уметь возвращать длину датасета, в нашем случае это просто сделать\n",
    "        # если не загружать данные целиком, а читать из файла - это может быть сложнее\n",
    "        return self.length\n",
    "\n",
    "    def __getitem__(self, index):\n",
    "\n",
    "        # в index будут индексы от 0 до self.length\n",
    "        # по ним мы достанем тексты, предобработаем,переведем в векторы, западим и вернем\n",
    "\n",
    "\n",
    "        tokens = self.preprocess(self.dataset[index]) # токенизируем\n",
    "        ids = [self.word2id[token] for token in tokens if token in self.word2id][:MAX_LEN]\n",
    "        ids = torch.nn.functional.pad(torch.LongTensor(ids), \n",
    "                                (0, self.MAX_LEN-len(ids)), \n",
    "                                mode='constant',\n",
    "                                value=0)\n",
    "\n",
    "        y = self.target[index]\n",
    "\n",
    "        return ids, y\n",
    "    \n",
    "    def preprocess(self, text):\n",
    "        tokens = text.lower().split()\n",
    "        tokens = [token.strip(punctuation) for token in tokens]\n",
    "        tokens = [token for token in tokens if token]\n",
    "        return tokens\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "avito = pd.read_csv('avito_data.csv').dropna() # берем все данные\n",
    "train_dataset, valid_dataset = train_test_split(avito, test_size=0.05)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_set = Dataset(train_dataset, word2id, MAX_LEN, 'description', 'deal_probability')\n",
    "training_generator = torch.utils.data.DataLoader(training_set, batch_size=256)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "valid_set = Dataset(valid_dataset, word2id, MAX_LEN, 'description', 'deal_probability')\n",
    "valid_generator = torch.utils.data.DataLoader(valid_set, batch_size=256)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Теперь нужно прописать логику обучения. Этот момент в tf и torch отличается наверное заментнее всего."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# функция для трейна\n",
    "# логика такая:  1) достаем батч данных, \n",
    "#              2) прогоняем через модель \n",
    "#              3) считаем функцию потерь\n",
    "#              4) считаем и сохраняем градиенты \n",
    "#              5) обновляем параметры модели \n",
    "\n",
    "def train(model, iterator, optimizer, criterion, print_every=1000):\n",
    "    epoch_loss = 0 # тут будет хранить лосс на каждом батче, чтобы потом посчитать средний лосс по эпохе\n",
    "    model.train()  # в торче нужно указать, что сейчас мы обучаемся и нужно хранить градиенты для апдейта модели\n",
    "\n",
    "    for i, (texts, ys) in enumerate(iterator):\n",
    "        \n",
    "        # в торче нужно явно обнулять градиенты\n",
    "        # если этого не сделать, то градиенты будут накапливаться со всех батчей\n",
    "        # а нам нужно чтобы каждый батч рассматривался в отдельности\n",
    "        # эта функция может пригодиться если данные помещаются очень маленькими батчами или вообще по 1\n",
    "        # так можно накопить градиенты и сделать апдейт после нескольких прогонов через модель и только потом обнулиться \n",
    "        optimizer.zero_grad()   \n",
    "        \n",
    "        # прогоняем тексты через модель\n",
    "        predictions = model(texts.to(device)).squeeze()  \n",
    "        \n",
    "        # считаем лосс\n",
    "        loss = criterion(predictions.float(), ys.to(device).float())        \n",
    "        \n",
    "        # градиенты\n",
    "        loss.backward()     \n",
    "        \n",
    "        # обновляем модель \n",
    "        optimizer.step()      \n",
    "        \n",
    "        # сохраняем значение лосса\n",
    "        epoch_loss += loss.item() \n",
    "        \n",
    "        # можно вставить промежуточные принты, чтобы понимать что происходит\n",
    "        if not (i+1) % print_every:\n",
    "            print(f'Loss: {epoch_loss/i}')\n",
    "        \n",
    "    return epoch_loss / len(iterator) # возвращаем усредненное значение лосса по всем батчам"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# функция для валидации отличается тем, что мы не считаем градиенты и не обновляем модель\n",
    "\n",
    "def evaluate(model, iterator, criterion):\n",
    "    \n",
    "    epoch_loss = 0\n",
    "    epoch_acc = 0\n",
    "    \n",
    "    model.eval()  # режим валидации (нужно если в модели есть Dropout )\n",
    "\n",
    "    with torch.no_grad(): # не считаем градиенты\n",
    "        for texts, ys in iterator:   \n",
    "\n",
    "            predictions = model(texts.to(device)).squeeze()  # прогоняем тексты\n",
    "            loss = criterion(predictions.float(), ys.to(device).float())   # считаем лосс  \n",
    "\n",
    "            # сохраняем лосс\n",
    "            epoch_loss += loss.item()  \n",
    "        \n",
    "    return epoch_loss / len(iterator) # возвращаем средний лосс"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Теперь создаем модель"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Regression(nn.Module):\n",
    "    \n",
    "    def __init__(self, vocab_size, embedding_dim, output_dim):\n",
    "        \n",
    "        super().__init__()          \n",
    "        \n",
    "        # Создаем слои и указываем параметры\n",
    "        self.embedding = nn.Embedding(vocab_size, embedding_dim)\n",
    "        self.fc = nn.Linear(embedding_dim, output_dim) # то же самое что Dense в tf keras\n",
    "        self.act = nn.Sigmoid() # активация отдельно, мы предсказываем число, поэтому можно было и не ставить активацию\n",
    "                                # но наш таргет от 0 до 1, поэтому и предсказания лучше ограничить\n",
    "                                # сигмоида - как раз функция, которая маппит число в интервал от 0 до 1\n",
    "        \n",
    "        \n",
    "    def forward(self, text):\n",
    "        # описываем как данные проходят по слоям\n",
    "        embedded = self.embedding(text)   # эмбедим тексты\n",
    "        hidden = torch.mean(embedded, dim=1) # усредняем вектора последовательности\n",
    "        dense_outputs=self.fc(hidden) # пропускаем через полносвязный слой \n",
    "\n",
    "        # применяем активацию, чтобы получить предикт в нужном интервале\n",
    "        outputs=self.act(dense_outputs)\n",
    "        \n",
    "        return outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Создаем оптимайзер и класс для лосса  \n",
    "Инициализируем модель и отправляем ее на gpu (это нужно делать вручную, обратите внимание, что выше в функциях train  и evaluate данные также отправляются на видеокарту вручную, данные и модель должны быть в одном месте - либо на gpu либо на cpu)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')  \n",
    "model = Regression(len(word2id), 30, 1)\n",
    "\n",
    "\n",
    "optimizer = optim.Adam(model.parameters(), lr=0.01)\n",
    "criterion = nn.MSELoss() # лосс для регрессии - mse (mean squared error)\n",
    "\n",
    "# отправляем модель и класс лосса на гпу\n",
    "model = model.to(device)\n",
    "criterion = criterion.to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Запускаем обучение"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "Loss: 0.06477599534708578\n",
      "Loss: 0.06260758320083017\n",
      "Loss: 0.06176662701453794\n",
      "Loss: 0.061212924311170876\n",
      "Loss: 0.060750814953233555\n",
      "1\n",
      "Loss: 0.057957255166840624\n",
      "Loss: 0.057846035528788274\n",
      "Loss: 0.0579212444557095\n",
      "Loss: 0.057922998638406754\n",
      "Loss: 0.05783559635555799\n",
      "2\n",
      "Loss: 0.056886465423875626\n",
      "Loss: 0.05681015573080448\n",
      "Loss: 0.0569264260979345\n",
      "Loss: 0.05696758405622407\n",
      "Loss: 0.05687612739778574\n"
     ]
    }
   ],
   "source": [
    "losses = []\n",
    "losses_eval = []\n",
    "\n",
    "for i in range(3):\n",
    "    print(i)\n",
    "    losses.append(train(model, training_generator, optimizer, criterion))\n",
    "    losses_eval.append(evaluate(model, valid_generator, criterion))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Хороший график должен выглядеть как-то так (и трейн и валидация снижаются, между ними нет слишком сильной разницы), но разница между начальными и конечными числами могла бы быть побольше."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEWCAYAAABxMXBSAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Z1A+gAAAACXBIWXMAAAsTAAALEwEAmpwYAAA3w0lEQVR4nO3deXxU1fn48c8z2VcCIWwJqwKybwHBBbVqRa24iyiyqdQiWuvX9ktrF2tt69ft54KKaFlSFRQsLVbcRXEBISBbQNlkCSAEEEgC2Z/fH/cmDjE7M5lJ8rxfr3lx59xz7zxzM8wz9557zhFVxRhjjKkpT6ADMMYY07BY4jDGGFMrljiMMcbUiiUOY4wxtWKJwxhjTK1Y4jDGGFMrljiM8RERmS0iD9Ww7g4RucjfMRnjD5Y4jAkybgIqEJEcETksIu+LyBle68eLiIrI/yu33ZVu+WyvsltF5GsRyRaR/SKyWETiKnid0sfaenujpsGyxGFMcHpEVWOBZGAP8I9y67cBN4hIqFfZOGBz6RMROQ/4GzBaVeOAHsBrFb2O16Ofr9+IaXwscZgmxb1E9GsRWSciuSLyDxFpLSJvu7/KPxCR5l71R4pIhogcEZGPRaSH17oBIrLa3e41ILLca/1MRNa4234hIn1rG6+qngBeB/qXW/UdsB64xH2tFsBZwCKvOoOBZar6lbuvw6o6R1WzaxuHMd4scZim6FrgYqAbcAXwNvA7IAnn/8TdACLSDZgL3OOuWwy8KSLhIhIO/Bv4J9ACmO/uF3fbAcBM4OdAIvACsEhEImoTqIjEAKOBrRWsTgPGuss3Av8B8r3WfwlcIiJ/FpGza/vaxlTGEodpip5R1f2qugf4FPhSVb9S1TxgITDArTcKeEtV31fVQuAxIArnl/1QIAx4UlULVXUBsNLrNSYBL6jql6parKpzcL7Uh9YwxvtE5AiQDZwD3FJBnYXA+SLSDCeBpHmvVNVPgWuAgcBbwCEReUJEQsq/jtdjTg3jM02YJQ7TFO33Wj5RwfNYd7kdsLN0haqWALtx2h3aAXv05FFCd3otdwT+x/tLGWjvblcTj6lqAtDJjal7+QruZay3gN8Diar6eQV13lbVK3DOiq4ExgO3lX8dr8e4GsZnmjBLHMZUbi9OAgBARATny38PsA9IdstKdfBa3g38tdyXcrSqzq1NAKq6C/gl8JSIRFVQJQ34H+DlavZToqofAh8BvWsTgzHlWeIwpnKvA5eLyIUiEobzBZ0PfAEsA4qAu0UkTESuAYZ4bfsicIeInCmOGBG5vPRW2NpQ1fdxktikClZ/gtNe80z5Fe7tuTeKSHM3hiHAecDy2sZgjDdLHMZUQlW/AcbgfCkfxGlIv0JVC1S1AKf9YDxwGKc95F9e26YDtwPTgO9xGrfHn0I4jwK/Kd/ArY4PVfVwBdt878awBTiGc1byqKq+4lXnN+X6cRw8hRhNEyE2kZMxxpjasDMOY4wxtWKJwxhjTK1Y4jDGGFMrljiMMcbUSmj1VRq+li1baqdOnQIdhjHGNCirVq06qKpJ5cubROLo1KkT6enpgQ7DGGMaFBHZWVG5XaoyxhhTK5Y4jDHG1IolDmOMMbXSJNo4KlJYWEhmZiZ5eXmBDsWvIiMjSUlJISwsLNChGGMaiSabODIzM4mLi6NTp06cPMBp46GqHDp0iMzMTDp37hzocIwxjUSTvVSVl5dHYmJio00aACJCYmJioz+rMsbUL78mDhEZISLfiMhWEZlawfp7RWSjO//zhyLSsdz6eBHJFJFpXmWjRWS9u807ItLyFOKr66YNRlN4j8aY+uW3xOFOT/kscCnQExgtIj3LVfsKSFXVvsAC4JFy6/8CLPXaZyjwFHCBu806YIp/3gEczs0nO6/QX7s3xpgGyZ9nHEOAraq63Z27YB7O1JVlVHWJqh53ny4HUkrXicggoDXwntcm4j5i3JnX4nEmuPG5ElUO5RSw89BxjhcU+Xz/R44c4bnnnqv1dpdddhlHjhzxeTzGGFNT/kwcyTjTZ5bKdMsqcyvwNoCIeIDHgfu8K6hqIfALYD1OwugJ/KOinYnIJBFJF5H0rKysWgfvEaFTyxhCQ4QdB4+TX1hc631UpbLEUVRUdZJavHgxCQkJPo3FGGNqIygax0VkDJCKM8sZwGRgsapmlqsXhpM4BgDtcC5V/baifarqDFVNVdXUpKQfDbVSI2EhHjonxgDw7cFcCotL6rSfikydOpVt27bRv39/Bg8ezLnnnsvIkSPp2dO5mnfVVVcxaNAgevXqxYwZM8q269SpEwcPHmTHjh306NGD22+/nV69evHTn/6UEydO+Cw+Y4ypjD9vx90DtPd6nuKWnURELgLuB85T1Xy3eBhwrohMBmKBcBHJAd4AUNVt7ravAz9qdK+tP7+Zwca9xypdX6LKicJiPCJEhYXUaJ8928Xzpyt6Vbr+4YcfZsOGDaxZs4aPP/6Yyy+/nA0bNpTdNjtz5kxatGjBiRMnGDx4MNdeey2JiYkn7WPLli3MnTuXF198kRtuuIE33niDMWPG1Cg+Y4ypK38mjpVAVxHpjJMwbgRu8q4gIgOAF4ARqnqgtFxVb/aqMx6nAX2qiLQDeopIkqpmARcDm/z4HgDnslVkaAh5hcXkFRYTWcPkURtDhgw5qa/F008/zcKFCwHYvXs3W7Zs+VHi6Ny5M/379wdg0KBB7Nixw+dxGWNMeX5LHKpaJCJTgHeBEGCmqmaIyINAuqouwrk0FQvMd28b3aWqI6vY514R+TOwVEQKgZ3A+FONtaozA2/fHy9g9+HjNIsKo0OLaJ/e6hoTE1O2/PHHH/PBBx+wbNkyoqOjOf/88yvsixEREVG2HBISYpeqjDH1wq89x1V1MbC4XNkfvZYvqsE+ZgOzvZ5PB6b7LMhaaB4dTlGxsu/oCfYeyaNdQmSdk0dcXBzZ2dkVrjt69CjNmzcnOjqar7/+muXLl59K2MYY41NNdsiRukqKi6CopISs7HxCQ4TW8ZF12k9iYiJnn302vXv3JioqitatW5etGzFiBNOnT6dHjx50796doUOH+ip8Y4w5ZaKqgY7B71JTU7X8RE6bNm2iR48eddqfqpL5/Qm+P15ASvMoWsREVL9RAJ3KezXGNF0iskpVU8uX2xlHHYgIyc2jKCpR9nx/glCPh/goG33WGNM0BEU/jobII0KHFtFEhYew6/BxcvN937vcGGOCkSWOUxDiETolxhAW4mHHoVzyfNy73BhjgpEljlMUGuKhc0vn1txvD+ZSUOS73uXGGBOMLHH4QHhoCJ0TYygpUb49mEuRD4cmMcaYYGOJw0eiwkPomBhDQXEJOw4dp6Sk8d+tZoxpmixx+FBsZCgdmkdxvKCIXYeP48tbnWNjY322L2OMORWWOHysWXQ4yQlRHMsrZM/3J3yaPIwxJhhYPw4/SIyNoLBYOZCdR2iIhzbNfty7fOrUqbRv354777wTgAceeIDQ0FCWLFnC999/T2FhIQ899BBXXnnlj7Y1xphAssQB8PZU+G69T3fZuk1viob+yU0eQsvYk3uXjxo1invuuacscbz++uu8++673H333cTHx3Pw4EGGDh3KyJEjbd5wY0xQscThJ4KQnBBFUbGy98gJQj1CQnR42foBAwZw4MAB9u7dS1ZWFs2bN6dNmzb86le/YunSpXg8Hvbs2cP+/ftp06ZNAN+JMcaczBIHwKUP+2W3AnRoEc23B3PZ/b2TPGIjfxia5Prrr2fBggV89913jBo1ildeeYWsrCxWrVpFWFgYnTp1qnA4dWOMCSRrHPczj0fomBhNRIiHnYeOc6Lgh6FJRo0axbx581iwYAHXX389R48epVWrVoSFhbFkyRJ27twZwMiNMaZiljjqQWiIh04tY/B4hG8PHaegyBmapFevXmRnZ5OcnEzbtm25+eabSU9Pp0+fPqSlpXHGGWcEOHJjjPkxu1RVT8JDPXRuGcO2rBy+PZhLl6RYwkI8rF//Q6N8y5YtWbZsWYXb5+Tk1FeoxhhTJTvjqEeRYSF0SoyhsFjZcSiXYutdboxpgCxx1LOYiFA6tIgmr6CEnYdyKbEOgsaYBqZJJ45A9eqOjwojuXkkOflFZPq5d7n1XDfG+FqTTRyRkZEcOnQoYF+sLWIiaBMfyZHjBXx31D+33Koqhw4dIjKybvOiG2NMRZps43hKSgqZmZlkZWUFNI7c4wXs31XM7qhQ4iJ9P/1sZGQkKSkpPt+vMabparKJIywsjM6dOwc6DIpLlLvnfsVb63fzxA39uGagfckbY4Jbk00cwSLEIzwxqh+Hcwv4zYJ1tIgJ5/zurQIdljHGVKrJtnEEk4jQEF4YO4iureOY/Mpq1u4+EuiQjDGmUpY4gkR8ZBhzJgwmMTacCbNXsj3LOvwZY4KTJY4g0io+krSJZyLA2JkrOHDMBjg0xgQfSxxBpnPLGGZNGMzh3ALGzVrJsbzCQIdkjDEnscQRhPqmJDB9zCC27M9mUlo6eYXFgQ7JGGPKWOIIUsO7JfHY9f1Yvv0w976+xsa1MsYEDbsdN4hdNSCZgzn5PPTWJlrGZvDnkb1sGlljTMBZ4ghyt53bhQPZ+cxYup2k2AjuurBroEMyxjRxfr1UJSIjROQbEdkqIlMrWH+viGwUkXUi8qGIdCy3Pl5EMkVkmvs8TkTWeD0OisiT/nwPwWDqiDO4ekAyj7+/mXkrdgU6HGNME+e3Mw4RCQGeBS4GMoGVIrJIVTd6VfsKSFXV4yLyC+ARYJTX+r8AS0ufqGo20N/rNVYB//LXewgWHo/wyHV9OZRbwO8WricxNoKLe7YOdFjGmCbKn2ccQ4CtqrpdVQuAecCV3hVUdYmqHnefLgfKBmoSkUFAa+C9inYuIt2AVsCnfog96ISFeHj+5oH0SW7GlFdXk77jcKBDMsY0Uf5MHMnAbq/nmW5ZZW4F3gYQEQ/wOHBfFfVvBF7TSsZFF5FJIpIuIumBHgHXV2IiQpk5fjDtEqKYOHslm/dnBzokY0wTFBS344rIGCAVeNQtmgwsVtXMKja7EZhb2UpVnaGqqaqampSU5LtgAywxNoK0iUOICAth3MwV7D1yItAhGWOaGH8mjj1Ae6/nKW7ZSUTkIuB+YKSq5rvFw4ApIrIDeAwYKyIPe23TDwhV1VV+ij2otW8RzZwJQ8jJK2LszBUcOV4Q6JCMMU2IPxPHSqCriHQWkXCcM4RF3hVEZADwAk7SOFBarqo3q2oHVe2Ec7kqTVW978oaTRVnG01Bz3bxzBibyq5Dx7l1TjonCqx3uTGmfvgtcahqETAFeBfYBLyuqhki8qCIjHSrPQrEAvPd22sXVbK78m6giScOgGGnJfLkjf1Zvet77pq7mqLikkCHZIxpAiRQc27Xp9TUVE1PTw90GH7zz2U7+MN/MrghNYX/u7av9S43xviEiKxS1dTy5dZzvBG4ZVgnDmTn88xHW2kVF8l9l3QPdEjGmEbMEkcjce/F3cjKzmfakq0kxUUw7qxOgQ7JGNNIWeJoJESEh67qzcGcAh54M4OWsRFc3rdtoMMyxjRCQdGPw/hGaIiHaTcNYFCH5vzqtTV8se1goEMyxjRCljgamciwEF4al0rHxGgmpa0iY+/RQIdkjGlkLHE0QgnR4cyZOIS4yFDGz1rJ7sPHq9/IGGNqyBJHI9UuIYq0iUMoKCph7MwVHMrJr34jY4ypAUscjVjX1nHMHJ/KvqMnmDB7Jbn5RYEOyRjTCFjiaOQGdWzBtNEDydh7jDteXkVBkfUuN8acGkscTcBFPVvzt6t78+mWg/xmwVpKShr/aAHGGP+xfhxNxKjBHTiYU8Cj735DUlwE91/eM9AhGWMaKEscTcjk80/jwLE8Xvz0W5LiIpg0/LRAh2SMaYAscTQhIsIfr+jFwZwC/rb4a1rGRnDNwJTqNzTGGC+WOJqYEI/wxKh+HM4t4DcL1tEiJpzzu7cKdFjGmAbEGser8uUL8PVb0MiGno8IDeGFsYPo2jqOya+sZu3uI4EOyRjTgFjiqExJMaxOg3k3wYsXwJb3G1UCiY8MY86EwSTGhjNh9kq2Z+UEOiRjTANhiaMynhCY9AmMnAa5h+CV62DmJbD9k0BH5jOt4iNJm3gmAoyduYIDx/ICHZIxpgGwxFGVkFAYeAvctQoufwKO7Ia0kTD7Z7BzWaCj84nOLWOYNWEwh3MLGDdrJcfyCgMdkjEmyFniqInQcBh8K9z9FYz4P8j6BmaNgH9eDZkNf0ravikJTB8ziC37s5mUlk5eYXGgQzLGBDFLHLURFglD74BfroWL/wL71sJLF8IrN8DeNYGO7pQM75bEY9f3Y/n2w9z7+hqKrXe5MaYSljjqIjwazr7bSSA/+QPs/hJmnAfzbob9GYGOrs6uGpDM7y/vweL13/HnNzPQRnQzgDHGdyxxnIqIOBh+H9yzDs6bCt8uhefPhgUT4eCWQEdXJ7ed24VJw7uQtmwn0z7aGuhwjDFByBKHL0Q2gwt+65yBnPMr+OYdeHYILLwDDm8PdHS1NnXEGVw9IJnH39/MvBW7Ah2OMSbIWOLwpegWcNGfnAQydDJkLIRnUmHRXXCk4XwBezzCI9f1ZXi3JH63cD3vb9wf6JCMMUHEEoc/xCbBJX91Esjg22DtPHh6ILz1P3Bsb6Cjq5GwEA/P3zyQPsnNmPLqatJ3HA50SMaYIGGJw5/i2sBljzi38Q64GVbNhqf6wzu/hZwDgY6uWjERocwcP5h2CVHcOiedzfuzAx2SMSYIWOKoD81S4IqnYEo69LkOvpwOT/WD9/8Ix4P7l3xibARpE4cQHuph3MwV7D1yItAhGWMCzBJHfWrRGa56Du5cCWdcDp8/DU/2gY/+CieOBDq6SrVvEc2cCUPIySti7MwVHDleEOiQjDEBZIkjEFqeDte+BJOXwekXwtJH4Km+8MmjkB+cl4N6totnxthUdh06zq1z0jlRYL3LjWmqLHEEUqsecEMa/PxT6Hg2LHkInuwLnz0JBbmBju5Hhp2WyJM39mf1ru+5a+5qiopLAh2SMSYALHEEg7Z9YfRcuP0jSB4IH/zJaQNZ9hwUBteItZf1acuDI3vxwaYD/G7heutdbkwTZIkjmCQPgjFvwMR3nbORd38LT/eHFS9CUX6goytzy7BO3PWT03k9PZPH39sc6HCMMfXMr4lDREaIyDcislVEplaw/l4R2Sgi60TkQxHpWG59vIhkisg0r7JwEZkhIptF5GsRudaf7yEgOgyFcW86j4SOsPg+eGYQrJoDxcEx7Pm9F3fjxsHtmbZkK3O+2BHocIwx9chviUNEQoBngUuBnsBoEelZrtpXQKqq9gUWAI+UW/8XYGm5svuBA6razd1v45lZqbzOw2HiO85ZSEwSvHk3TBsMa+Y6MxQGkIjw0FW9uahHax54M4O31u0LaDzGmPrjzzOOIcBWVd2uqgXAPOBK7wqqukRVj7tPlwMppetEZBDQGniv3H4nAn93ty9R1YN+ij84iMDpFzntH6PnQUQs/PsOePZM2PAGlASugTo0xMO0mwYwqENzfvXaGr7Y1rj/FMYYhz8TRzKw2+t5pltWmVuBtwFExAM8DtznXUFEEtzFv4jIahGZLyKtK9qZiEwSkXQRSc/KyqrjWwgiItD9Upi01LkTyxPqjMI7/RzY9GbA5kOPDAvhpXGpdEyMZlLaKjL2Hg1IHMaY+hMUjeMiMgZIBR51iyYDi1U1s1zVUJyzki9UdSCwDHison2q6gxVTVXV1KSkJD9FHgAeD/S8En7xOVz7DyjOh9fGOPOBbH43IAkkITqcOROHEBcZyvhZK9l9+Hj1GxljGix/Jo49QHuv5ylu2UlE5CKcdouRqlp669AwYIqI7MBJDGNF5GHgEHAc+Jdbbz4w0C/RBztPiDN8yeQv4arnnZ7nr94AL10E2z6q9wTSLiGKtIlDKCgqYezMFRzKCZ67wIwxvuXPxLES6CoinUUkHLgRWORdQUQGAC/gJI2yUf9U9WZV7aCqnXAuV6Wp6lR1Og28CZzvVr0Q2OjH9xD8QkKh/01w1ypnPKzs75y50GddBjs+q9dQuraOY+b4VPYdPcGE2SvJzS+q19c3xtQPvyUOVS0CpgDvApuA11U1Q0QeFJGRbrVHgVhgvoisEZFFlezO2/8CD4jIOuAW4H/8EH7DExIGg8bD3avhssecCaRmXw5zRsLuFfUWxqCOLZg2eiAZe49xx8urKCiy3uXGNDZSVc9fEfmJqn7kLndW1W+91l2jqv+qdOMgkpqaqunp6YEOo34VnoD0mfDZ/4PcLDj9Yrjgd07P9Hrw2spd/O8b67mqfzueuKE/Ho/Uy+saY3xHRFapamr58urOOLwbnt8ot+73pxyV8Z+wKBh2pzOZ1EUPwJ50ePECmHsTfLfB7y8/anAHfn1Jd/69Zi9/f3uT31/PGFN/qkscUslyRc9NMAqPceZB/+U6uOB+p91j+tnw+jg48LVfX3ry+acxblhHXvz0W15c2vDmXjfGVKy6xKGVLFf03ASzyHg47zdwz1o49z7Y+gE8NxTeuB0ObfPLS4oIf7yiF5f3actfF29i4Vfl7642xjREodWs7+I2WIvXMu7zzn6NzPhHVHO48A8wdDJ8/qQzgOKGN6DfaCexNO9Y7S5qI8QjPDGqH4dzC/j1/HU0jw7n/O6tfPoaxpj6VV3j+HlVbayqDWKcqCbZOF5T2fudBvT0maDFMHCsc0bSrKpO/rV3LK+QUS8sZ+ehXObePpR+7RN8un9jjO9V1jheZeKoYCdhQG9gj3e/i2BniaMGju6BTx+H1WkgHkidAOfcC3EVjuhSJweO5XHt9C/IzS9mwR3D6JIU67N9G2N8r053VYnIdBHp5S43A9YCacBXIjLaL5GawGiWDD97wulI2PcG5xLWU/3gvd9Drm8GL2wVH0naxDMRYOzMFRw4FlyTVBljaqa6xvFzVTXDXZ4AbFbVPsAg4Dd+jcwERvOOcOU0mLLSGRNr2bPOdLYfPgjHD5/y7ju3jGHWhMEczi1g3KyVHMsLjvlFjDE1V13iKPBavhj4N4CqfuevgEyQSDwNrnkBJi+Hbpc4l7Ge6gcfPwx5pzYCbt+UBKaPGcSW/dlMSksnrzCwc4sYY2qnusRxRER+5o4pdTbwDoCIhAJR/g7OBIGk7nD9LLjjc2diqY//7pyBfPo45OfUebfDuyXx2PX9WL79MPe+vobiEru725iGorrE8XOc8aZmAfd4nWlcCLzlz8BMkGnTG258BSZ9DO2HOJeunuoHXzzjDG9SB1cNSOb3l/dg8frv+PObGdTmRg1jTODU6q6qhsruqvKD3StgyV9h+8cQ29q5hXfQOAiNqPWu/rZ4EzOWbue+n3Zjyk+6+j5WY0ydVHZXVZUdAEXk6arWq+rdpxqYaaDaD4Gx/4EdnzsJ5O1fw+dPwfD7YMAYZ7TeGpo64gyysvN57L3NtIyN4MYhHfwYuDHmVFV3qeoO4BxgL5AOrCr3ME1dp7Nh/Ftwy78hvi389x54ZhB89QoU12w+Do9HeOS6vgzvlsTvFq7n/Y37/RqyMebUVJc42gIzgEtw5r4IA/6jqnNUdY6/gzMNhAicdgHc+j7cNB+iEuA/k+G5M2HdfCip/q6psBAPz988kD7JzZjy6mrSd5z6rb/GGP+oMnGo6iFVna6qF+D040gANorILfURnGlgRKDbT2HSJzDqFQiJgH/dBs+fBRn/hpKqJ3WKiQhl5vjBtEuI4tY56Wzen10/cRtjaqVGMwCKyEDgl8AY4G3sMpWpigj0+Bnc8RlcNwu0BOaPgxeGw9eLq5wPPTE2grSJQwgP9TBu5gr2HqnbHVvGGP+pbsiRB0VkFXAv8AmQqqq3qmrTnufb1IzHA72vcToRXj0DCnNh3mh48Sew5YNKE0j7FtHMmTCEnLwixs5cwZHjBRXWM8YERnWj45YA3wLH3aLSygKoqvb1b3i+YbfjBoniIlg7Fz55BI7ugvZnOpNLdal4EOZl2w4xbuYK+qQ04+VbzyQqPKSeAzamaavT6LgiUuXkDKq60wex+Z0ljiBTVABf/ROWPgbZe6HTuU4C6TjsR1UXr9/Hna+u5sIzWjF9zCBCQ2p0ddUY4wN1Gh1XVXdW9AB249yma0zthYbD4Fvh7q9gxMOQ9Q3MGgH/vBoyT24+u6xPWx4c2YsPNh3g/oUbrHe5MUGgujaOeBH5rYhME5GfiuMuYDtwQ/2EaBqtsEgY+gv45Vq4+EHYuwZe+gm8Ogr2rS2rdsuwTtz1k9N5LX03j7+3OXDxGmOA6u+q+ifQHVgP3AYsAa4DrlLVK/0cm2kqwqPh7F/CPevgJ7+HXcucO7BeuwUObALg3ou7cePg9kxbspU5X+wIbLzGNHHVzjnuzr+BiLwE7AM6qKrNwGN8LyIOhv8aBt8Oy5+DZc/Bpjeh97XI+VN56KreHMwp4IE3M2gZG8HlfdsGOmJjmqTqzjjKZtlR1WIg05KG8buoBLjgd84ZyDn3wDeL4dkhhC66k2cvbc6gDs351Wtr+GKbb2YmNMbUTnV3VRUDuaVPcebgOM4Pt+PG+z1CH7C7qhq4nCz4/ElY+RKUFJHfezQTtp/Puuw4Xvv5UHq1axboCI1plOp0O25jYYmjkTi2Dz57AlbNRoE39EL+4bmGGZN/RvsW0YGOzphGp0634xoTVOLbwmWPwl2rkX6juZb3WVg0meXP/5zD+zMDHZ0xTYYlDtPwJLSHkU8jU9LJOX0k1xS8SfTzAyl45w9w3EbVNcbfLHGYhqtFZ1qOmcmXl77Ne8WDCF3+DPpkX1jyNzhxJNDRGdNoWeIwDd5ZQ4dxYuQLjMh/mLURA+CT/4On+sLSRyHfhmY3xtcscZhGYdTgDoy8+EKuyrqDl3rOhg5nwUcPwZN9nSltC45Xuw9jTM34NXGIyAgR+UZEtorI1ArW3ysiG0VknYh8WH5QRXfIk0wRmeZV9rG7zzXuo5U/34NpOO684HTGDevIQ6vDeTHlb3DbR9BuALz/R3iqHyx/HgqtG5Ixp8pviUNEQoBngUuBnsBoEelZrtpXOHN89AUWAI+UW/8XYGkFu79ZVfu7jwM+Dt00UCLCH6/oxeV92vLXxZtYmNUabvkXTHgHkrrDO1Ph6QFOf5Aim+PDmLry5xnHEGCrqm5X1QJgHnDS+FaqukRVS68hLAdSSteJyCCgNfCeH2M0jUyIR3hiVD+GdUnk1/PX8fE3B5zh2sf/F8Yucu7Ieut/4JlBsDoNigur36kx5iT+TBzJOMOvl8p0yypzK860tIiIB3gcuK+SurPcy1R/EBGpqIKITBKRdBFJz8rKqn30psGKCA3hhbGD6No6jsmvrGbt7iPOii7nwcR34eY3ICYRFt0F0wbD2nlQUhzQmI1pSIKicVxExgCpwKNu0WRgsapW1KvrZnfgxXPdxy0V7VNVZ6hqqqqmJiUl+SNsE8TiI8OYM2EwibHhTJi9ku1ZOc4KEeh6Edy+BG6cC+GxsPDn8NxQ2PAvKCkJbODGNAD+TBx7gPZez1PcspOIyEXA/cBIVc13i4cBU0RkB/AYMFZEHgZQ1T3uv9nAqziXxIz5kVbxkaRNPBMBxs5cwYFjXg3jInDGZfDzpXD9HBAPLJgA089xRuRtAkPxGFNX/kwcK4GuItJZRMKBG4FF3hVEZADwAk7SKGvkVtWbVbWDqnbCuVyVpqpTRSRURFq624YBPwM2+PE9mAauc8sYZk0YzOHcAsbNWsmxvHJtGh4P9LoKfvEFXPMSFOXBa2Ngxnmw+V1LIMZUwG+JQ1WLgCnAu8Am4HVVzRCRB0VkpFvtUSAWmO+2WSyqZHelIoB3RWQdsAbnDOZFv7wB02j0TUlg+phBbNmfzaS0dPIKK2jP8IRA3+vhzhVw5XNOz/NXb4B/XAzbPrIEYowXGx3XNBn//moP97y2hsv6tOGZ0QMJ8VR4X4WjuBC+ehmWPgbHMqHj2XDB/dDp7PoL2JgAs9FxTZN31YBkfn95Dxav/44/v5lBlT+aQsIgdQLcvRoufRQObYPZl0HalbB7Zf0FbUwQssRhmpTbzu3CpOFdSFu2k2eXbK1+g9AIOHMS/HIN/PSv8N0G+MdF8Mr1sPcrv8drTDCyxGGanKkjzuDqAck89t5m5q3YVbONwqLgrCnwy7Vw4Z9g9wqYcT7Mu9lJJsY0IZY4TJPj8QiPXNeX4d2S+N3C9by/cX/NN46IhXPvhXvWw/m/g2+XwvSzYf54yPrGbzEbE0yscdw0Wbn5Rdz04nK+/i6bV247k9ROLWq/k+OHYdk0WD4dik5An+vhvP+FxNN8H7BpnIqLoDDXGcG50H0UHHfKCk+UW85163gt/2i7csv373Pa7OrA5hy3xGEqcCgnn+umL+NwbgHz7xhGt9ZxddtR7kH4/ElY8RIUF0D/0TD8N9C8Y7WbmiBXUuz1Je1+gVe6XFECKJcMCo6fXLekluOliQfCYiA8GsLcR1XL5//WaaurA0scljhMJXYfPs41z39BqEd44xdn0S4hqu47y94Pnz0B6TOdvh8Dx8Lw+yC+ne8CNicrKa7Zr/RKv9DLJ4Byv+iLazuSskB4TPVf6GExTttZad2Tlr3qh7v1SpdDwp2RD+qBJQ5LHKYKG/ceY9QLy2jdLJIFdwwjITr81HZ4dA98+his/qfzCzF1IpzzK4hr7ZuAG5KSkiq+pOv4K917uTi/+hhOIpV/qZd9SZf+oq9sufRLv4IEEBpRb1/s/maJwxKHqcaybYcYN3MFfVKa8fKtZxIVHnLqO/1+Jyx9BNbMdX4pDrkdzr7HGZ03WJSUOO0zFf1ir/JXerlf5pUtF9Vh8qwa/zKPquTXfblf6d5f+qGRjeaL3d8scVjiMDWweP0+7nx1NRee0YrpYwYRGuKjGw8PbXPmQl/3uvNFduYdzu29Uc2r31a1gksp1TWKVvMr3btu0Ynav59Qr1/bVf4yr+4yTfSP64ZF2Rd7kLDEYYnD1NA/l+3gD//JYFRqex6+tg+VTPlSN1nfwMd/h4yFENEMul3iXEOv6k6ZwjrMlx4aVcNf5lX9Sq/kSz80yhkc0jR6lSWO0EAEY0wwu2VYJw5k5/PMR1tJiovgvku6+27nSd3h+tlw7n3wycOw84uTf6VHt4CwlMp/pdfkF31YtH2xG7+yxGFMBe69uBtZ2flMW+Ikj3FndfLtC7TpDaNe9u0+jaknljiMqYCI8NBVvTmYU8ADb2bQMjaCy/u2DXRYxgQFO581phKhIR6m3TSAQR2a86vX1vDFtoOBDsmYoGCJw5gqRIaF8NK4VDomRvPztFVk7D0a6JCMCThLHMZUIyE6nDkThxAbGcr4WSvZfbgOdzkZ04hY4jCmBtolRJE2cQgFRSWMnbmCQzm17a1sTONhicOYGuraOo6Z41PZd/QEE2evJDe/KNAhGRMQljiMqYVBHVswbfRANuw9xh0vr6KgqCTQIRlT7yxxGFNLF/Vszd+u7s2nWw7ymwVrKSlp/KMvGOPN+nEYUwejBncgKzufx97bTFJcBPdf3jPQIRlTbyxxGFNHd15wOlnZ+bz46be0iovk9uFdAh2SMfXCEocxdSQi/PGKXhzMKeCvizfRMi6cqwekBDosY/zOEocxpyDEIzwxqh+Hcwv49fx1tIiJ4LxuSYEOyxi/ssZxY05RRGgIL4wdRNfWcfzi5VWs3X0k0CEZ41eWOIzxgfjIMOZMGExibDgTZq9ke1ZOoEMyxm8scRjjI63iI0mbeCYCjJ25ggPH6jBlqjENgCUOY3yoc8sYZk0YzOHcAsbNWsmxvMJAh2SMz1niMMbH+qYkMH3MILbsz2ZSWjp5hcWBDskYn7LEYYwfDO+WxGPX92P59sPc+/oaiq13uWlE7HZcY/zkqgHJHMzJ56G3NtEyNoM/j+yFiAQ6LGNOmV/POERkhIh8IyJbRWRqBevvFZGNIrJORD4UkY7l1seLSKaITKtg20UissGf8Rtzqm47twuThnchbdlOnl2yNdDhGOMTfkscIhICPAtcCvQERotI+QF9vgJSVbUvsAB4pNz6vwBLK9j3NYDd72gahKkjzuDqAck89t5m5q3YFehwjDll/jzjGAJsVdXtqloAzAOu9K6gqktUtXQ6teVA2XgNIjIIaA28572NiMQC9wIP+TF2Y3zG4xEeua4vw7sl8buF63l/4/5Ah2TMKfFn4kgGdns9z3TLKnMr8DaAiHiAx4H7Kqj3F3ddlfN3isgkEUkXkfSsrKzaxG2Mz4WFeHj+5oH0SW7GlFdXk77jcKBDMqbOguKuKhEZA6QCj7pFk4HFqppZrl5/4DRVXVjdPlV1hqqmqmpqUpKNHWQCLyYilJnjB9MuIYpb56SzeX92oEMypk78mTj2AO29nqe4ZScRkYuA+4GRqlo6kfMwYIqI7AAeA8aKyMNueapb/hnQTUQ+9tcbMMbXEmMjSJs4hPBQD+NmrmDvkROBDsmYWhNV/9xfLiKhwGbgQpyEsRK4SVUzvOoMwGkUH6GqWyrZz3icBvQp5co7Af9V1d7VxZKamqrp6el1fCfG+N7GvccY9cIyilXp2TaeXu3i6dWuGT3bxdOtdRzhoUFxMcA0cSKySlVTy5f7rR+HqhaJyBTgXSAEmKmqGSLyIJCuqotwLk3FAvPd+9t3qepIf8VkTLDo2S6euZOGMj99Nxl7j7FgVSZzlu0EICxE6Noqzk0m8fRObkaPtvHERFi3KxMc/HbGEUzsjMMEu5ISZcehXDL2HnMfR9m49xiHcgsAEIHOiTH0dM9MSpNKYmxEgCM3jVm9n3EYY2rO4xG6JMXSJSmWK/q1A0BV2X8sn4y9R9mwx0kmX+06wn/X7Svbrk18ZFkS6dmuGb2T40lOiLIe6savLHEYE6REhDbNImnTLJILe7QuKz9yvICNXmcmGXuPseSbA5QOh9UsKqwsmZSenXRJiiXEY8nE+IYlDmMamITocM46vSVnnd6yrOxEQTFff3dyMpmzbCcFRSUARIZ5OKPNycmke5s4IsNCAvU2TANmbRzGNFKFxSVsy8ohY49Xu8m+Y2TnFQHOfOldW8We1G7Ss1088ZFhAY7cBIvK2jgscRjThKgquw+fKDsr2eD+m5WdX1anQ4voH13qahUfGcCoTaBY47gxBhGhQ2I0HRKjubRP27LyA9l5ZOw95radOMnk7Q3fla1Piov4UTLp0CLaGuGbKEscxhhaxUXSqnskF3RvVVZ2LK+QTWWN8E5C+XTLwbJJqeIiQulRLpmc3iqWsBDrvNjYWeIwxlQoPjKMM7skcmaXxLKyvMJiNu/PPqkRfu6KXeQVOo3w4aEezmgTV3Z7cK928fRoE09UuDXCNyaWOIwxNRYZFkLflAT6piSUlRWXKN8ezDnpzGTx+u+Yu8IZHNsj0CUp9keXuhKiwwP0LsypssZxY4zPqSp7jpwoSyYb3bOTfUfzyuokJ0SdlEh6JcfTJj7S2k2CiDWOG2PqjYiQ0jyalObRXNKrTVn5oZx8Nu47ud3k/U37Kf392iImvOy24NKE0jkxBo91XgwqljiMMfUmMTaCc7smcW7XH+bIyc0vYtO+kzsvzvzsWwqLnWwSHR5iIwgHGbtUZYwJOgVFJWw5kH3SLcIb9x4jt6AY+PEIwr3cEYRjbQRhn7JLVcaYBiM81ONeqmpWVlbRCMIffX2A+auciUJtBOH6Y4nDGNMgVDeCcGkyWbO76hGEe7WLJ6W5jSB8KixxGGMaLBtBODAscRhjGp3qRxB2bhG2EYTrxhrHjTFNVlFxCduyctmw54dLXeVHED49KZZeyU1zBGEbHdcShzGmBsqPIFz674EmOIKw3VVljDE1UNkIwlnZ+T9KJt4jCLeMdUYQ7p3c+EcQtsRhjDE1kBQXwfndW3F+NSMIf771IEWNfARhSxzGGFNHlY0gvGV/Dhl7j5ZNlDVvxW5OFO4AnD4q3VvHlXVcbIgjCFviMMYYH4oMC6FPSjP6pPzQebGiEYTfyfiOeSsb5gjC1jhujDEBoKrsPZpHxp6T203KjyDc000mvds1q/cRhK1x3BhjgoiIkJwQRXJCFD/1GkH4cG6BVyO8k1A+CLIRhC1xGGNMEGkRE17hCMJlnRf3HGPD3qM/GkG4R9kIwk5C8ecIwnapyhhjGqCajiA89/ahNIuuW4dFu1RljDGNSGUjCO88fLzsUtf2rBzio3z/NW+JwxhjGgmPR+jcMobOLWP4Wd92/nsdv+3ZGGNMo2SJwxhjTK1Y4jDGGFMrfk0cIjJCRL4Rka0iMrWC9feKyEYRWSciH4pIx3Lr40UkU0SmeZW9IyJrRSRDRKaLSMPpp2+MMY2A3xKH+4X+LHAp0BMYLSI9y1X7CkhV1b7AAuCRcuv/AiwtV3aDqvYDegNJwPW+jt0YY0zl/HnGMQTYqqrbVbUAmAdc6V1BVZeo6nH36XIgpXSdiAwCWgPvldvmmLsYCoQDjb8jijHGBBF/Jo5kYLfX80y3rDK3Am8DiIgHeBy4r6KKIvIucADIxjlTMcYYU0+ConFcRMYAqcCjbtFkYLGqZlZUX1UvAdoCEcBPKtnnJBFJF5H0rKwsP0RtjDFNkz87AO4B2ns9T3HLTiIiFwH3A+epauncjMOAc0VkMhALhItIjqqWNbCrap6I/Afn8tf75ferqjOAGe5rZInIzjq+j5bAwTpu608WV+1YXLVjcdVOY42rY0WFfhurSkRCgc3AhTgJYyVwk6pmeNUZgHOpaYSqbqlkP+NxGtCniEgsEKeq+9z9vwJ8qqrTKtrWR+8jvaKxWgLN4qodi6t2LK7aaWpx+e2MQ1WLRGQK8C4QAsxU1QwReRBIV9VFOJemYoH57vjyu1R1ZBW7jQEWiUgEzmW2JcB0f70HY4wxP+bXsapUdTGwuFzZH72WL6rBPmYDs93l/cBgnwZpjDGmVoKicTzIzQh0AJWwuGrH4qodi6t2mlRcTWI+DmOMMb5jZxzGGGNqxRKHMcaYWmnSiaMGgzBGiMhr7vovRaST17rfuuXfiMgl9RhTpQNDikixiKxxH4t8FVMtYhvv9pkpjeE2r3XjRGSL+xhXz3H9P6+YNovIEa91fjlmIjJTRA6IyIZK1ouIPO3GvE5EBnqt8+exqi6um9141ovIFyLSz2vdDrd8jYj4dC7mGsR1vogc9fpb/dFrXZV/fz/H9WuvmDa4n6cW7jp/Hq/2IrLE/S7IEJFfVlDHf58xVW2SD5xbhLcBXXDGvFoL9CxXZzIw3V2+EXjNXe7p1o8AOrv7CamnmC4Aot3lX5TG5D7PCfDxGg9Mq2DbFsB299/m7nLz+oqrXP27cG4N9+sxA4YDA4ENlay/DGeIHQGGAl/6+1jVMK6zSl8PZ4DSL73W7QBaBuh4nQ/891T//r6Oq1zdK4CP6ul4tQUGustxOH3myv9/9NtnrCmfcVQ7CKP7fI67vAC4UJwOJ1cC81Q1X1W/Bba6+/N7TFrFwJB+VpPjVZlLgPdV9bCqfo/T039EgOIaDcz10WtXSlWXAoerqHIlkKaO5UCCiLTFv8eq2rhU9Qv3daEeP181OF6VOZXPpa/jqpfPFoCq7lPV1e5yNrCJH48F6LfPWFNOHDUZhLGsjqoWAUeBxBpu66+YvJUNDOmKFGd8ruUicpUP4qlLbNe6p8ULRKR0yBl/Ha9a7du9rNcZ+Mir2J/HrCqVxe3PY1Vb5T9fCrwnIqtEZFIA4hkmzlw8b4tIL7csKI6XiETjfPm+4VVcL8dLnEvoA4Avy63y22fMrx0Ajf/IDwNDnudV3FFV94hIF+AjEVmvqtvqMaw3gbmqmi8iP8c5W6twEMoAuRFYoKrFXmWBPmZBSUQuwEkc53gVn+Meq1bA+yLytfuLvD6sxvlb5YjIZcC/ga719No1cQXwuap6n534/XiJMwzTG8A9+sOUE37XlM84ajIIY1kdccbGagYcquG2/orJe2DIkfrDwJCo6h733+3Axzi/Qnyl2thU9ZBXPC8Bg2q6rT/j8nIj5S4l+PmYVaWyuP15rGpERPri/P2uVNVDpeVex+oAsBDfXJ6tEVU9pqo57vJiIExEWhIEx8tV1WfLL8dLRMJwksYrqvqvCqr47zPmj4abhvDAOdvajnPporRRrVe5OndycuP46+5yL05uHN+ObxrHaxLTAJzGwK7lypsDEe5yS2ALvm0krElsbb2WrwaW6w+Ncd+6MTZ3l1vUV1xuvTNwGiulHo9ZJypv7L2ckxsuV/j7WNUwrg44bXZnlSuPwRlgtHT5C5zBSesrrjalfzucL+Bd7rGr0d/fX3G565vhtIPE1Nfxct97GvBkFXX89hnz2cFtiA+cuw4243wR3++WPYjzSx4gEpjv/kdaAXTx2vZ+d7tvgEvrMaYPgP3AGvexyC0/C1jv/sdZD9wagOP1dyDDjWEJcIbXthPd47gVmFCfcbnPHwAeLred344Zzq/PfUAhzjXkW4E7gDvc9YIztfI297VT6+lYVRfXS8D3Xp+vdLe8i3uc1rp/4/vrOa4pXp+t5Xgltor+/vUVl1tnPM7NMt7b+ft4nYPThrLO6291WX19xmzIEWOMMbXSlNs4jDHG1IElDmOMMbViicMYY0ytWOIwxhhTK5Y4jDHG1IolDmOCmDsq7H8DHYcx3ixxGGOMqRVLHMb4gIiMEZEV7twLL4hIiIjkiDMXSIY4c6ckuXX7u4MqrhORhSLS3C0/XUQ+cAfyWy0ip7m7j3UHjfxaRF5xR2g2JmAscRhzikSkBzAKOFtV+wPFwM04Q02kq2ov4BPgT+4macD/qmpfnB69peWvAM+qaj+cXu373PIBwD0488B0Ac7281sypko2Oq4xp+5CnAEdV7onA1HAAaAEeM2t8zLwLxFpBiSo6idu+RxgvojEAcmquhBAVfMA3P2tUNVM9/kanLGTPvP7uzKmEpY4jDl1AsxR1d+eVCjyh3L16jq+T77XcjH2/9YEmF2qMubUfQhc5867gIi0cCeN8gDXuXVuAj5T1aPA9yJyrlt+C/CJOrO4ZZZOJiXOfPfR9fkmjKkp++VizClS1Y0i8nuc2d48OCOp3gnkAkPcdQdw2kEAxgHT3cSwHZjglt8CvCAiD7r7uL4e34YxNWaj4xrjJyKSo6qxgY7DGF+zS1XGGGNqxc44jDHG1IqdcRhjjKkVSxzGGGNqxRKHMcaYWrHEYYwxplYscRhjjKmV/w/XqxY4ERh2uQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Нарисуем график с RMSE (это просто корень из лосса)\n",
    "plt.plot(np.sqrt(losses))\n",
    "plt.plot(np.sqrt(losses_eval))\n",
    "plt.title('model RMSE')\n",
    "plt.ylabel('RMSE')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'val'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Обучать эмбединги под задачу в теории правильнее всего, но на практике бывает лучше использовать предобученные эмбединги (word2vec, fastext, glove). Отдельно эмбединги обучаются быстрее и можно использовать гораздо больше данных, так как нам не нужна разметка."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Обучим фастекст модель."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.8/site-packages/gensim/similarities/__init__.py:15: UserWarning: The gensim.similarities.levenshtein submodule is disabled, because the optional Levenshtein package <https://pypi.org/project/python-Levenshtein/> is unavailable. Install Levenhstein (e.g. `pip install python-Levenshtein`) to suppress this warning.\n",
      "  warnings.warn(msg)\n"
     ]
    }
   ],
   "source": [
    "import gensim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "texts = avito.sample(frac=0.1).description.apply(preprocess).tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 4min 11s, sys: 7.05 s, total: 4min 18s\n",
      "Wall time: 1min 41s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "ft = gensim.models.FastText(texts, vector_size=100, epochs=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('коляска-люлька', 0.9760681390762329),\n",
       " ('кресло-коляска', 0.9455839395523071),\n",
       " ('санки-коляска', 0.9428970813751221),\n",
       " ('колясочка', 0.937743604183197),\n",
       " ('коляску,в', 0.9363677501678467),\n",
       " ('колясках', 0.9245208501815796),\n",
       " ('коляска-трансформер', 0.9208803176879883),\n",
       " ('аляска', 0.9043774604797363),\n",
       " ('кoляска', 0.8985717296600342),\n",
       " ('коляску-трансформер', 0.8910782337188721)]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ft.wv.most_similar('коляска')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Теперь нам нужно сделать матрицу с векторами. Индексы строчек в этой матрице должны совпадать с индексами слов в словаре."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "weights = np.zeros((len(word2id), 100))\n",
    "\n",
    "for word, i in word2id.items():\n",
    "    # вектор паддинга оставим нулевым\n",
    "    if word == 'PAD':\n",
    "        continue\n",
    "    \n",
    "    try:\n",
    "        weights[i] = ft.wv[word]\n",
    "    \n",
    "    \n",
    "    except KeyError:\n",
    "        # для слов, которых нет в модели тоже возьмем какой-то  рандомный вектор\n",
    "        weights[i] = ft.wv['опрагпллирао']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Regression_Emb(nn.Module):\n",
    "    \n",
    "    def __init__(self, vocab_size, embedding_dim, embeddings_weights, output_dim):\n",
    "        \n",
    "        super().__init__()          \n",
    "        \n",
    "        self.embedding = nn.Embedding(vocab_size, embedding_dim)\n",
    "        # вот так можно загрузить готовые вектора и заморозить обучение\n",
    "        self.embedding.from_pretrained(torch.tensor(embeddings_weights), freeze=True)\n",
    "        self.fc = nn.Linear(embedding_dim, output_dim)\n",
    "        self.act = nn.Sigmoid()\n",
    "        \n",
    "    def forward(self, text):\n",
    "        embedded = self.embedding(text)   \n",
    "        hidden = torch.mean(embedded, dim=1)\n",
    "        dense_outputs=self.fc(hidden)\n",
    "        outputs=self.act(dense_outputs)\n",
    "        \n",
    "        return outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# в остальном все также\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')  \n",
    "model = Regression_Emb(len(word2id), 100, weights, 1)\n",
    "\n",
    "optimizer = optim.Adam(model.parameters())\n",
    "criterion = nn.MSELoss()\n",
    "\n",
    "\n",
    "model = model.to(device)\n",
    "criterion = criterion.to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "Loss: 0.07252339075680252\n",
      "Loss: 0.06893570427449719\n",
      "Loss: 0.06691501865603798\n",
      "Loss: 0.0655519172789589\n",
      "Loss: 0.06463247046372395\n",
      "1\n",
      "Loss: 0.06081590337214527\n",
      "Loss: 0.060488868683666035\n",
      "Loss: 0.060373216442622196\n",
      "Loss: 0.060233215822610715\n",
      "Loss: 0.060133176877167634\n",
      "2\n",
      "Loss: 0.05988390226517473\n",
      "Loss: 0.05963499575905528\n",
      "Loss: 0.05958813311741447\n",
      "Loss: 0.05951087957256942\n",
      "Loss: 0.05946526806305995\n"
     ]
    }
   ],
   "source": [
    "losses = []\n",
    "losses_eval = []\n",
    "for i in range(3):\n",
    "    print(i)\n",
    "    losses.append(train(model, training_generator, optimizer, criterion))\n",
    "    losses_eval.append(evaluate(model, valid_generator, criterion))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEWCAYAAABxMXBSAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Z1A+gAAAACXBIWXMAAAsTAAALEwEAmpwYAAA0TElEQVR4nO3deXhV5bn+8e+TmQyMCYOAzFVAGSSAQwdtbdXaorV1xhFqbWsn29PSnx1tT+upHT3aqnVErROd6KnWVovaVgGDIgiiDIKAQMKciYzP74+1QlZiQrIhOzvJvj/XtS/2XtN+shNyZ633Xe9r7o6IiEh7pSS6ABER6V4UHCIiEhMFh4iIxETBISIiMVFwiIhITBQcIiISEwWHSAcxs/vM7Ift3HajmZ0e75pE4kHBIdLFhAFUbWZlZrbbzP5hZsdG1l9pZm5mv2i23znh8vsiy+aY2RozKzWzHWb2hJnltfA+DY9XO+0LlW5LwSHSNf3E3XOBocBW4O5m69cDF5hZWmTZFcCbDS/M7APAj4CL3T0PGA882tL7RB6TO/oLkZ5HwSFJJbxE9F9mtsLMys3sbjMbZGZPhn+VP21m/SLbzzKzVWa218yeNbPxkXVTzezlcL9Hgaxm7/UxM1se7vuCmU2KtV53rwQeA6Y0W7UdWAmcEb5Xf+BkYGFkm+nAi+7+Snis3e5+v7uXxlqHSJSCQ5LRJ4EPA+8BPg48Cfw/oIDg/8QXAczsPcDDwJfDdU8AfzGzDDPLAP4EPAD0Bx4Pj0u471TgHuAzwADgDmChmWXGUqiZ5QAXA+taWD0fuDx8fhHwZ6Aqsn4JcIaZfd/MTon1vUVao+CQZPS/7r7D3bcC/wKWuPsr7n4A+CMwNdzuQuCv7v4Pd68Bfgr0IvjL/kQgHfilu9e4+wLgpch7XAPc4e5L3L3O3e8n+KV+Yjtr/JqZ7QVKgfcCl7WwzR+BU82sD0GAzI+udPd/AecBJwB/BXaZ2c/NLLX5+0Qe97ezPkliCg5JRjsizytbeJ0bPj8K2NSwwt3rgc0E7Q5HAVu96SihmyLPRwBfjf5SBoaH+7XHT929LzAyrOmY5huEl7H+CnwLGODu/2lhmyfd/eMEZ0XnAFcCc5u/T+RxRTvrkySm4BBp3TsEAQCAmRnBL/+twDZgaLiswdGR55uB/272Sznb3R+OpQB3fxv4EvArM+vVwibzga8CD7ZxnHp3fwb4J3BcLDWINKfgEGndY8DZZvYhM0sn+AVdBbwAvAjUAl80s3QzOw+YEdn3t8C1ZjbTAjlmdnZDV9hYuPs/CELsmhZWP0fQXvO/zVeE3XMvMrN+YQ0zgA8Ai2OtQSRKwSHSCnd/A5hN8Et5J0FD+sfdvdrdqwnaD64EdhO0h/whsm8R8GngVmAPQeP2lUdQzs3A15s3cHvgGXff3cI+e8Ia1gL7Cc5Kbnb3hyLbfL3ZfRw7j6BGSRKmiZxERCQWOuMQEZGYKDhERCQmCg4REYmJgkNERGKS1vYm3V9+fr6PHDky0WWIiHQry5Yt2+nuBc2XJ0VwjBw5kqKiokSXISLSrZjZppaW61KViIjERMEhIiIxUXCIiEhMkqKNoyU1NTVs2bKFAwcOJLqUuMrKymLYsGGkp6cnuhQR6SGSNji2bNlCXl4eI0eOpOkApz2Hu7Nr1y62bNnCqFGjEl2OiPQQcb1UZWZnmtkbZrbOzOa1sP56M1sdTuP5jJlFh7CuC6fdXG5mC1vY9xYzKzvc2g4cOMCAAQN6bGgAmBkDBgzo8WdVItK54nbGEc4ydhvBkM9bgJfMbKG7r45s9gpQ6O4VZvZZ4CcEo4wCVLr7lFaOXQj0a2ldjDUe6SG6vGT4GkWkc8XzjGMGsM7dN4RDUD9CMAPZQe6+yN0rwpeLgWFtHTQMpJuBr3dwve+yv7KG3eXV8X4bEZFuJZ7BMZRgFrQGW8JlrZkDPBl5nWVmRWa22MzOjSy/Dljo7tsO9eZmdk24f1FJSUmMpQftA7vLq9m6p4K9FR0fHnv37uXXv/51zPt99KMfZe/evR1ej4hIe3WJ7rhmNhsoJDiTaDDC3QuBS4BfmtkYMzsKOJ8WZjtrzt3vdPdCdy8sKHjXHfPtqYnh/bPJzkhj855K9lfWxHyMQ2ktOGpraw+53xNPPEHfvn07tBYRkVjEMzi2EszP3GBYuKwJMzsduAGY5e5VDcvdfWv47wbgWWBq+BgLrDOzjUC2ma2LU/2kphgj87PJSkvh7d0VlFUd+pd6LObNm8f69euZMmUK06dP533vex+zZs1iwoQJAJx77rlMmzaNiRMncueddx7cb+TIkezcuZONGzcyfvx4Pv3pTzNx4kQ+8pGPUFlZ2WH1iYi0Jp7dcV8CxpnZKILAuIjg7OEgM5sK3AGc6e7FkeX9gAp3rzKzfOAU4Cdhw/rgyHZl7j72SAv9/l9Wsfqd/a2ud+BAdR31OL3SU0lpR4PzhKN6892PT2x1/U033cRrr73G8uXLefbZZzn77LN57bXXDnabveeee+jfvz+VlZVMnz6dT37ykwwYMKDJMdauXcvDDz/Mb3/7Wy644AJ+//vfM3v27PZ90SIihyluweHutWZ2HfAUkArc4+6rzOxGoMjdFxJcmsoFHg97/7zt7rOA8cAdZlZPcFZ0U7PeWJ3KgKz0VCpr6jhQU0dWO8MjFjNmzGhyr8Utt9zCH//4RwA2b97M2rVr3xUco0aNYsqUKQBMmzaNjRs3dmhNIiItiesNgO7+BPBEs2XfiTw/vZX9XgCOb8fxc4+0RuCQZwZRVbV1bCgpxx3GFOSQmZ7aEW8PQE5OzsHnzz77LE8//TQvvvgi2dnZnHrqqS3ei5GZmXnweWpqqi5ViUin6BKN491FZloqo/KDX/Bv7Synurb+sI+Vl5dHaWlpi+v27dtHv379yM7OZs2aNSxevPiw30dEpKMl7ZAjhysrPZVR+dlsKCnnrZ3ljC7IIT019vwdMGAAp5xyCscddxy9evVi0KBBB9edeeaZ3H777YwfP55jjjmGE088sSO/BBGRI2Lunuga4q6wsNCbT+T0+uuvM378+MM+ZnlVLW/tLCcjLYXR+TmkHUZ4dJYj/VpFJDmZ2bLwtogmuu5vuy4uJzONEQOyqaqtZ+OuCurqe34Ai4iAguOI5GWlc3T/bCqr69i0q5x6hYeIJAEFxxHq0yudYf17UVZVy9u7K6hPgkt/IpLcFBwdoF92BkP79mL/gRq27K4kGdqNRCR5qVdVBxmQm0mdO9v3HSBlLwzt20tDmotIj6Tg6EAD87Kor4fi0gOkmDGkT5bCQ0R6HF2q6mCDemeSn5vJzrIqikur2t6hnXJzO+QmeRGRI6Yzjg5m4ZlGXb2zY39w5lGQl9n2jiIi3YSCIw7MjGH9elHvzrZ9laSmQP+cpuExb948hg8fzuc//3kAvve975GWlsaiRYvYs2cPNTU1/PCHP+Scc85p6S1ERBJGwQHw5DzYvrJDD2mDj2f4mT+mflcFW/ZUkmJG3+yMg+svvPBCvvzlLx8Mjscee4ynnnqKL37xi/Tu3ZudO3dy4oknMmvWLLWTiEiXouCIoxQzRvTP5q2d5WzeHYRH717pAEydOpXi4mLeeecdSkpK6NevH4MHD+YrX/kKzz//PCkpKWzdupUdO3YwePDgNt5JRKTzKDgAzropbodOCWcR3LCznE27Kxg1IJvcrCA8zj//fBYsWMD27du58MILeeihhygpKWHZsmWkp6czcuTIFodTFxFJJPWq6gSpKSmMGpBDZloKG3dVUB5OQXvhhRfyyCOPsGDBAs4//3z27dvHwIEDSU9PZ9GiRWzatCnBlYuIvJuCo5OkpaYwKj+HtFRj465yKqvrmDhxIqWlpQwdOpQhQ4Zw6aWXUlRUxPHHH8/8+fM59thjE122iMi76FJVJ0pPDYZgXx/O5TGmIIeVKxsb5fPz83nxxRdb3LesrKyzyhQROSSdcXSyjMgsghuOcBZBEZFEUHAkQMMsgvXuvLWzjJo6hYeIdB9JHRyJHMW2V0YaIwfkUFPnvLWznNo4hYdG6hWRjpa0wZGVlcWuXbsS+os1JzONkXGcRdDd2bVrF1lZWR16XBFJbknbOD5s2DC2bNlCSUlJokuhpqaOHWXVvLMxhfzcjA69UzwrK4thw4Z12PFERJI2ONLT0xk1alSiyzjoz8u38ulHl3PaMQO5ffY0MtKS9mRQRLo4/XbqIs6ZMpQfnnsc/1xTzFceW97hl61ERDpK0p5xdEWXzhxBeVUtP3piDTkZqdx03iRSUjTAoYh0LQqOLuaa94+h7EAtt/xzHTmZaXznYxM0Oq6IdClxvVRlZmea2Rtmts7M5rWw/nozW21mK8zsGTMbEVlXZ2bLw8fCyPKHwmO+Zmb3mFl6PL+GRPjKh9/DVaeM5N7/bOQXT69NdDkiIk3ELTjMLBW4DTgLmABcbGYTmm32ClDo7pOABcBPIusq3X1K+JgVWf4QcCxwPNALmBuvryFRzIxvnz2BCwqHccsza7nz+fWJLklE5KB4XqqaAaxz9w0AZvYIcA6wumEDd18U2X4xMLutg7r7Ew3PzWwp0CP7mqakGD8+bxLlVXX86Ik15Gamc8nMoxNdlohIXC9VDQU2R15vCZe1Zg7wZOR1lpkVmdliMzu3+cbhJarLgL+1dDAzuybcv6gr3KtxOFJTjF9cOIXTjinghj+t5M/Ltya6JBGRrtEd18xmA4XAzZHFI9y9ELgE+KWZjWm226+B5939Xy0d093vdPdCdy8sKCiIS92dISMthd/MnsaMkf25/rFX+fuq7YkuSUSSXDyDYyswPPJ6WLisCTM7HbgBmOXuVQ3L3X1r+O8G4FlgamSf7wIFwPXxKLyryUpP5e4rp3Pc0D5c97tX+M+6nYkuSUSSWDyD4yVgnJmNMrMM4CJgYXQDM5sK3EEQGsWR5f3MLDN8ng+cQtg2YmZzgTOAi909aYaVzc1M4/6rpjMqP4dPzy9i2aY9iS5JRJJU3ILD3WuB64CngNeBx9x9lZndaGYNvaRuBnKBx5t1ux0PFJnZq8Ai4CZ3b2hUvx0YBLwY7vOdeH0NXU3f7AwemDuDgXmZXHXvUla9sy/RJYlIErJkGHa7sLDQi4qKEl1Gh9myp4ILbn+Rqtp6Hrv2JMYU5Ca6JBHpgcxsWdjW3ESXaByX2Azrl82Dc2diBrPvWsLm3RWJLklEkoiCo5saXZDLA3NmUl5Vy+y7l1C8/0CiSxKRJKHg6MbGD+nNfVfPoKS0itl3L2FPeXWiSxKRJKDg6OZOOLofd11eyMZdFVxx71JKD9QkuiQR6eEUHD3AyWPz+fUlJ7D6nf3Mub+Iyuq6RJckIj2YgqOHOH3CIH5+4RRe2ribax9cRnVt0tziIiKdTMHRg8yafBQ//sTxPPdmCV965BVq6xQeItLxFBw9zEUzjuZbZ4/nyde2M+8PK6nXFLQi0sE0A2APNPd9oymrquWXT68lNzON735cswiKSMdRcPRQX/rQOMoO1HLXv98iNzONr51xTKJLEpEeQsHRQ5kZN5w9nrKqWm5dFMxf/tlTm49MLyISOwVHD2Zm/Pcnjqe8uo7/+dsacrPSuOzEEW3vKCJyCAqOHi41xfj5BZOprK7lO39+jdzMVD4xtUfOtisinUS9qpJAemoKt15yAieNHsDXHl/BU5pFUESOgIIjSWSlp/LbywuZNKwPX/jdK/xrbfech11EEk/BkURyMtO478oZjC7I4Zr5yyjauDvRJYlIN6TgSDJ9stN5YM5MhvTJ4qp7X+K1rZpFUERio+BIQgV5mTw4dya9e6Vz+T1LWVdcmuiSRKQbUXAkqaP69uLBuTNJMeNSzSIoIjFQcCSxUfk5PDh3Bgdq6rnkrsVs36dZBEWkbQqOJHfs4N7cf/UMdpdVM/vuJezWLIIi0gYFhzBleF/uumI6m3dXcPk9S9ivWQRF5BAUHALASWMGcPvsaazZVsrV975ERXVtoksSkS5KwSEHnXbsQH510VRefnsPn3lgGVW1moJWRN5NwSFNnD1pCDedN4l/rd3JFx/WLIIi8m4KDnmXC6YP5zsfm8BTq3bw9QUrNIugiDSh0XGlRVe/dxTlVbX87B9vkpOZxo3nTNQsgiICKDjkEK774FjKqmq54/kN5Gal8Y0zj010SSLSBcT1UpWZnWlmb5jZOjOb18L6681stZmtMLNnzGxEZF2dmS0PHwsjy0eZ2ZLwmI+aWUY8v4ZkZmbMO+tYLp15NL95dj23LVqX6JJEpAuIW3CYWSpwG3AWMAG42MwmNNvsFaDQ3ScBC4CfRNZVuvuU8DErsvx/gF+4+1hgDzAnXl+DBOHxg3OO49wpR3HzU28w/8WNiS5JRBIsnmccM4B17r7B3auBR4Bzohu4+yJ3bxgkaTFwyKnpLLjI/kGCkAG4Hzi3I4uWd0tJMW4+fzIfnjCI7/x5FQuWbUl0SSKSQPEMjqHA5sjrLeGy1swBnoy8zjKzIjNbbGbnhssGAHvdveHutFaPaWbXhPsXlZRo0qIjlZ6awv9ePJX3js3n6wte5cmV2xJdkogkSJfojmtms4FC4ObI4hHuXghcAvzSzMbEckx3v9PdC929sKCgoAOrTV5Z6ancefk0pgzvyxcfeYXn3lQgiySjeAbHVmB45PWwcFkTZnY6cAMwy92rGpa7+9bw3w3As8BUYBfQ18waeoO1eEyJn+yMNO69agbjBubxmQeKWPqWZhEUSTbxDI6XgHFhL6gM4CJgYXQDM5sK3EEQGsWR5f3MLDN8ng+cAqx2dwcWAZ8KN70C+HMcvwZpQZ9e6cyfM4Oj+vbi6vteYsWWvYkuSUQ6UdyCI2yHuA54CngdeMzdV5nZjWbW0EvqZiAXeLxZt9vxQJGZvUoQFDe5++pw3TeA681sHUGbx93x+hqkdfm5mTw0dyZ9eqVzxT1LeXOHZhEUSRYW/BHfsxUWFnpRUVGiy+iRNu0q5/zbXwTg8WtPYsSAnARXJCIdxcyWhW3NTXSJxnHpvkYMyOHBuTOprqvn0ruWsG1fZaJLEpE4U3DIEXvPoDzmXz2DvRU1zL5rCTvLqtreSUS6LQWHdIhJw/pyz5XT2bq3ksvvXsq+Ss0iKNJTKTikw8wY1Z/bZ09jbXEpV927lPIqzSIo0hMpOKRDnXrMQG65aCrLN+/lmgeKOFCjWQRFehoFh3S4s44fwk8+NZn/rNvFFx5+hRrNIijSoyg4JC4+NW0Y3581kX+s3sF/Pf6qZhEU6UE0kZPEzRUnj6Ssqpabn3qDnMw0fnjucZpFUKQHUHBIXH3+tGAWwd88u57czDTmnXWswkOkmzvkpSoz+2Dk+ahm686LV1HSs3z9jGO47MQR3PH8Bs0iKNIDtNXG8dPI8983W/etDq5Feigz4/uzJnLe1KH89O9vcu9/3kp0SSJyBNq6VGWtPG/ptUirUlKMn3xqEuXVtXz/L6vJyUzjgsLhbe8oIl1OW2cc3srzll6LHFJaagq3XDyV943LZ97vV/DXFZpFUKQ7auuMY3Q41LlFnhO+HtX6biIty0xL5Y7LpnH53Uv58qOvkJ2RymnHDkx0WSISg0MOq25mHzjUzu7+XIdXFAcaVr3r2X+ghkt+u5i1O8q4/+oZnDh6QKJLEpFmDmtYdXd/LvoAXgD2A693l9CQrql3Vjr3XzWD4f2zmXPfSyzfvDfRJYlIO7XVHfd2M5sYPu8DvArMB14xs4s7oT7pwQbkZvLgnJn0z83ginuWsmb7/kSXJCLt0Fbj+PvcfVX4/CrgTXc/HpgGfD2ulUlSGNwni4fmnEhWegqz71rKWzvLE12SiLShreCojjz/MPAnAHffHq+CJPkcPSCbB+fMpN6d2Xct4Z29mkVQpCtrKzj2mtnHzGwqcArwNwAzSwN6xbs4SR7jwlkE91cGswiWlGoWQZGuqq3g+AxwHXAv8OXImcaHgL/GszBJPscN7cO9V01n274DXHb3EvZVaBZBka6orV5Vb7r7me4+xd3viyx/yt2/GvfqJOkUjuzPnZdPY0NJOVfep1kERbqiQ94AaGa3HGq9u3+xY8sRgfeNK+CWi6fy+d+9zKfnF3HPldPJSk9NdFkiEmrrUtW1wHuBd4AiYFmzh0hcnHncYH56/iReWL+L6373smYRFOlC2hpyZAhwPnAhUAs8Cixw971xrkuET0wdRtmBWr7951V89bFX+cWFU0hN0diaIonWVhvHLne/3d1PI7iPoy+w2swu64ziRC47aSTfOPNYFr76Dt/600oONUSOiHSOds0AaGYnABcT3MvxJLpMJZ3os6eOoayqhtsWrScnI40bzh6vWQRFEqitIUduNLNlwPXAc0Chu89x99XtObiZnWlmb5jZOjOb18L6681stZmtMLNnzGxEs/W9zWyLmd0aWXaxma0M9/mbmeW36yuVbu1rHzmGK04awV3/fotbntEsgiKJ1Fbj+LcILk9NBn4MvBz+wl5pZisOtaOZpQK3AWcBE4CLzWxCs81eIQijScAC4CfN1v8AeD5yzDTgV8Bp4T4rCO4zkR7OzPjuxyfyyROG8Yun3+Suf21IdEkiSautS1VHMufGDGCdu28AMLNHgHOAg2cr7r4osv1iYHbDCzObBgwiuFu9YVhfCx85ZrYL6A3oz88kkZJi/M8nj6eiupYf/vV1cjPTuGjG0YkuSyTptNU4vqmlB7CZoJvuoQwNt2uwJVzWmjkE7SeYWQrwM+BrzeqpAT4LrCToIjwBuLulg5nZNWZWZGZFJSUlbZQq3UVaagq/vGgKH3hPAd/840r+8uo7iS5JJOm01cbR28y+aWa3mtlHLPAFYANwQUcVYWazCc4qbg4XfQ54wt23NNsunSA4pgJHEVyq+mZLx3T3O9290N0LCwoKOqpU6QIy01K5ffY0po/oz1ceXc4zr+9IdEkiSaWtNo4HgGMI/sKfCywCPgWc6+7ntLHvVmB45PWwcFkTZnY6cAMwy90bRrY7CbjOzDYCPwUuN7ObgCkA7r7eg36ZjwEnt1GH9EC9MlK5+8pCJhzVm88+9DIvrN+Z6JJEkkZbwTHa3a909zsIuuNOAM5w9+XtOPZLwDgzG2VmGcBFwMLoBuGou3cQhEZxw3J3v9Tdj3b3kQSXq+a7+zyC4JlgZg2nEB8GXm9HLdID5YWzCI4ckM3c+4t4+e09iS5JJCm0FRwHhyd19zpgi7sfaM+B3b2WoMfTUwS/3B9z91VhF99Z4WY3A7nA42a23MwWtnK4hmO+A3wfeD7s1TUF+FF76pGeqV9OBg/OmUlBXiZX3rOU17dpFkGReLND3YlrZnVAw5RsRjAHR0X43N29d9wr7ACFhYVeVFSU6DIkjjbvruD821+ktr6exz5zEqMLchNdkki3Z2bL3L2w+fK2elWlunvv8JHn7mmR590iNCQ5DO+fzYNzZ+IOs+9awlbNIigSN21dqhLpNsYOzGX+nBmUVtVy6W8XU1zarquqIhIjBYf0KBOP6sN9V82guLSKy+9eyt6K6kSXJNLjKDikx5k2oh93XlbIhpJyrrj3Jco0i6BIh1JwSI/03nH53HrJVF7buo+597/EgZq6RJck0mMoOKTH+sjEwfzs/MkseWs3n3voZaprNYugSEdQcEiPdu7Uofzw3OP455pivvLYcurqNRGUyJFq10ROIt3ZpTNHUF5Vy4+eWENORio3nTeJFE1BK3LYFBySFK55/xjKDtRyyz/XkZOZxnc+NkGzCIocJgWHJI2vfPg9lFbVcu9/NpKXlc71H35PoksS6ZYUHJI0zIxvnz2B8qpabnlmLbmZqVzz/jGJLkuk21FwSFJJSTF+fN4kyqvq+NETa8jNTOeSmZpFUCQWCg5JOqkpxi8unEJFdS03/GklOZmpnDPlUJNTikiUuuNKUspIS+E3s6cxY2R/rn/sVf6+anuiSxLpNhQckrSy0lO5+8rpHDe0D9f97hX+vVazCIq0h4JDklpuZhr3XzWdUfk5fHp+Ecs2aRZBkbYoOCTp9c3O4IG5MxjUO5Or7l3Kqnf2JbokkS5NwSECDMzL4sG5M8nNTOPyu5eyvqQs0SWJdFkKDpHQsH7BLIJmwSyCm3dXJLokkS5JwSESMbogl/lXz6S8qpbZdy+heL9mERRpTsEh0syEo3pz39UzKCmtYvbdS9hTrlkERaIUHCItOOHoftx1eSEbd1Vwxb1LKT1Qk+iSRLoMBYdIK04em8+vLzmB1e/sZ879RVRWaxZBEVBwiBzS6RMG8fMLp/DSxt1c++AyzSIogoJDpE2zJh/Fjz9xPM+9WcKXHnmF2jqFhyQ3BYdIO1w042i+dfZ4nnxtO/P+sJJ6TUErSUyj44q009z3jaasqpZfPr2W3Mw0vvtxzSIoyUnBIRKDL31oHGUHarnr32+Rm5nG1844JtEliXS6uF6qMrMzzewNM1tnZvNaWH+9ma02sxVm9oyZjWi2vreZbTGzWyPLMszsTjN708zWmNkn4/k1iESZGTecPZ6Lpg/n1kXr+M2z6xNdkkini9sZh5mlArcBHwa2AC+Z2UJ3Xx3Z7BWg0N0rzOyzwE+ACyPrfwA83+zQNwDF7v4eM0sB+sfraxBpiZnx3584nvLqOv7nb2vIzUzlspNGJroskU4Tz0tVM4B17r4BwMweAc4BDgaHuy+KbL8YmN3wwsymAYOAvwGFke2uBo4N968HNImCdLrUFOPnF0ymsrqWb/95FblZaXxi6rBElyXSKeJ5qWoosDnyeku4rDVzgCcBwjOJnwFfi25gZn3Dpz8ws5fN7HEzG9TSwczsGjMrMrOikpKSw/wSRFqXnprCrZecwMljBvC1x1fwlGYRlCTRJbrjmtlsgrOKm8NFnwOecPctzTZNA4YBL7j7CcCLwE9bOqa73+nuhe5eWFBQEKfKJdllpafy28sLOX5oH77wu1f411r9kSI9XzyDYyswPPJ6WLisCTM7naDdYpa7V4WLTwKuM7ONBMFwuZndBOwCKoA/hNs9DpwQl+pF2iknM437rprO6IIcrpm/jKKNuxNdkkhcxTM4XgLGmdkoM8sALgIWRjcws6nAHQShUdyw3N0vdfej3X0kweWq+e4+z90d+Atwarjph4i0mYgkSt/sDB6YM5PBfbK46t6XeG2rZhGUnituweHutcB1wFPA68Bj7r7KzG40s1nhZjcDucDjZrbczBa2criobwDfM7MVwGXAV+NQvkjMCvIyeXDuTHr3Sufye5ayrrg00SWJxIUFf8T3bIWFhV5UVJToMiRJvLWznPNvf5HUFFhw7ckM75+d6JJEDouZLXP3wubLu0TjuEhPMio/hwfnzuBATT2X3LWY7fs0i6D0LAoOkTg4dnBv7r96BrvLqpl99xJ2axZB6UEUHCJxMmV4X+66Yjqbd1dw+T1L2K9ZBKWHUHCIxNFJYwZw++xprNlWytX3vkRFdW2iSxI5YgoOkTg77diB/Oqiqbz89h4+88Ayqmo1Ba10bwoOkU5w9qQh3HTeJP61didffFizCEr3puAQ6SQXTB/Odz42gadW7eDrC1ZoFkHptjSR06Hs2wq9+kJGTqIrkR7i6veOoryqlp/9401yMtO48ZyJmkVQuh0Fx6H89avw5t8gfxwMmRw8Bk+CIZOgV79EVyfd1HUfHEtZVS13PL+B2vp6zpg4mLEDczmqTy9SUhQi0vUpOA7lxGvhqCmw7VXY9CKsfLxxXd8RQYAMmQxDpgSBktfiCO8iTZgZ8846lgM1ddz/4iYeXhrMPpCdkcqYglzGDcxl7KBcxhbkMm5QHsP79SItVVeVpevQkCOxKN8ZhMj2FcG/216F3Rsa1+cODoNkUuPZSd+jQZcipBW7yqpYV1zGupIy1u4oY3347/b9jXebZ6SmMLoghzEDw1AZmMu4gXmMzM8mMy01gdVLT9fakCMKjiN1YB9sf61poJSsAQ97zWT1jYTJlOB5/zGQor8gpXX7D9SwvrgsCJXiMtaG/27eU0HDf9nUFGNE/2zGNoTJoFzGFuQxZmAO2Rm6mCBHTsHRmYMc1lTCjtWwbXljoOxYBXXhsBPpOTD4+KZnJwXHQmp659Uo3VJldR0bdkYCZUdwtrJxZzm1kV5aQ/v2CoMkDJSBQaj0ydbPmLSfgiPRo+PW1QRnItsil7m2r4Sa8mB9agYMnNDYCD9kMgyaCOm9Elu3dAvVtfW8vbs8CJLwDGVtcRkbSsqoqm28Z6QgL5NxkUteYwfmMXZgLvm5GerdJe+i4Eh0cLSkvi5oI2kIkobHgb3BekuFgmPCnlwNvbqOh6zeCS1buo+6emfLnooml7vWFpexvriMsqrG4U/6ZqcfPDsZEzbKBz29shQoSUzB0RWDoyXusG9zJEjCM5Sy7Y3b9B8d6RocBkpOfuJqlm7H3dm+/0CTy13rdpSxtriUPRWNgzHmZKQyJtIgPzY8WxneP5tUdR3u8RQc3SU4WlO6I2x8X94YKHs3Na7vPfTdYdL7KPXokpjtKqs6eHbS2Dhfyo79VQe3yUhLYXR+TtNAGZTLyAE5ZKSp40dPoeDo7sHRkso9QYBEuwfvXAuE39PsAU3bTAZPgn6j1KNLDsv+AzVNwqQhULbsqWza02tAdpNuw2MHBpe/emWo63B3o+DoicHRkqqyoAfXtldhexgmxWugPrz8kNm78e73hjDJfw+kqvumHJ7K6jrWlzQNk3XFZWzcVUFd2NPLLOzpFQmUhktgfXqpp1dXpeBIluBoSW0VFL/e9F6T7a9BbWWwPi0LBh0XuRN+ctDDKy0zsXVLt1ZdW8+mXeVNGuXXFQc3OVZHenoN6p3ZJEwawmVAjnp6JZqCI5mDoyV1tbBrbWPje0OgVO0P1qekQcH4pveaDDoOMnMTW7d0e3X1zubdTXt6rQvPUsqrG+cq6Zed3qTLcEOgDFFPr06j4FBwtK2+HvZubHqvybZXoWJnuIHBgLGRdpNJwaWu7P6JrFp6CHdn274D7wqUtcVl7I309MrNTGNMQQ5jB+Y1uclxWD/19OpoCg4Fx+Fxh9JtTbsGb3sV9m9p3Kbv0WG7yZTGQMkbnLCSpWdxd3aVVzcGyo7Sg2N7FZc29vTKTEthdEFuk7OTcQNzGaGeXodNwaHg6Fjluxob3xsCZff6xvW5g97dPVgDPkoH21cZ9PRaH2mUX1tcxpY9lQe3STvY06ux2/CYAvX0ag8Fh4Ij/g7shx2vNb15sWQNeHjdOqtv4+WthrOTAWMgRf95pWNVVNeyoaS8MUzCmxw3NevpNaxfr4OBEn30zlJPL1BwKDgSpWHAx+2RNpMdq6EuvMSQnt044OPgyICPaRmJrVt6pKraOjbtqoiM6RUEy4ad5e/q6RUNlIM9vXKTq6ehgkPB0XXU1UDJG826B6+E6rJgfWoGDBwfuXExHPAxIzuxdUuP1dDTa22ze1HWFZdREenp1T8ng7EF0Ym2gkAZ3Ltn9vRKSHCY2ZnAr4BU4C53v6nZ+uuBuUAtUAJc7e6bIut7A6uBP7n7dc32XQiMdvfj2qpDwdEN1NeHAz4ubxoolXuC9ZYC+cc0vddk8PGQ1SehZUvP5u6809DTa0fpwYm21haXsa+yaU+v5mcn4wbmMbRfr27d06vTg8PMUoE3gQ8DW4CXgIvdfXVkm9OAJe5eYWafBU519wsj638FFAC7o8FhZucBnwImKTh6sIMDPja716R0W+M2/UY1m3VxMuQWJK5mSQruzs6y6iZdhhsa5kua9fQa07yn16Cgp1d6N5gOuLXgiOc4EzOAde6+ISzgEeAcgjMIANx9UWT7xcDshhdmNg0YBPwNKIwszwWuB64BHotj/ZJoZkFPrL5Hw/iPNS4vKw7DZHnYbrIcVv+pcX3eUU3vNRkyORgEsgdeSpDEMDMK8jIpyMvkpDEDmqzbV1HDupKmjfLLNu1h4avvHNwmLcUYmZ8TmRcl9+CYXlnpXb+zSDyDYyiwOfJ6CzDzENvPAZ4EMLMU4GcEQXJ6s+1+EK6r6LBKpXvJHQjjTg8eDSr3BO0k0XtN1j7VOIVvr/5NB3wcMlkDPkpc9MlOZ9qI/kwb0fTG2IrqWtYXl7OupPRg4/wb20v5++odTXp6De+X/a5AGTswl7wu1NOrS4xsZ2azCc4qPhAu+hzwhLtviTY4mdkUYIy7f8XMRrZxzGsIzko4+uij41C1dCm9+sGo9wePBtXljQM+NjxevK1xwMeMvEj34PDsJP8YDfgocZGdkcbxw/pw/LCm7XJVtXVs3FnR5D6U9cVl/GvtTqrrGnt6De6d1TgNcGTk4f45nd8DMZ5tHCcB33P3M8LX3wRw9x832+504H+BD7h7cbjsIeB9QD2QC2QAvwY2Ad8GqglCbyDwgrufeqha1MYhB9VWQ8nrTe812b6y2YCPE5veuDhwAqRnJbZuSTq1dfVs3lPJ2vBO+XUNE2416+k1ICejyeCQDYEyqHfmEff0SkTjeBpB4/iHgK0EjeOXuPuqyDZTgQXAme6+tpXjXAkUttCraiTwf2oclyNWXxfMYxKd12TbCqjaF6xPSQvuLYneazL4OMjMS2zdkpTq65139lU2mxcl6PW1/0DjdMB5mWmMGZjLzy+YzOiCwxuctNMbx9291syuA54i6I57j7uvMrMbgSJ3XwjcTHBG8XiYjG+7+6x41STSopRUGHhs8Jh0QbDMHfZsbNqba+3fYflD4U4W3PUenSRryGQN+Chxl5JiDOuXzbB+2Zx6zMCDy92dkrKqpoGyo4y+2R1/KUs3AIq0lzuUbm88K2kIlH2RPiB9jm52r0k44KN6dEk3lIjuuCI9ixn0HhI8jjmzcXnF7qYN8NtXwJr/a1yfM7Bp1+Ahk6HvCIWJdFsKDpEjld0fxpwWPBocHPAx0m6y/p+RAR/7BGcjBccEbSWZeUEvr8xcyMgN/23htXp8SRegn0KReMjqDSNODh4Nag5A8aqmYbJyQTBGV31t68eKSss6RLDkhgHU7HWTZZF9MnJ01iOHRcEh0lnSs2DotOAR5R7MC19dBlWl4b9l735dVQrVpZF14b/lJbDnrcbXDYNFtsmahcwhwqe119EzJc1RnzQUHCKJZhaESnoW5OQf+fHq66GmvOXwac/rvW83XVZX1fZ7AqSkx3Ym1NZrzdPSZSk4RHqalJTGS1Qdoa4m9vBpeH1gP+x/J1weni15XdvvCZDWK/w6jiB8Gl6nZ+uyXAdScIjIoaWmBx0AOuIeFXeoPRBeetsfYxiVQdl22BV5XVPevve1lLBdJ3qJraWwaaVTQvMwSvKJxhQcItJ5zCC9V/DoiOHv6+uCMckOBkzzNqDSIIDetayhfWhn09d11e1739SMNoKmHT3kDnZSyO12g20qOESk+0pJDXqwZfXumOPVVrdwtnOI4ImeHVXuCW4GjXZS8Pq23xMgPaeNoGl2lnSoMErLivtlOQWHiEiDtAxI68DLcjUVh99JYf87TV/XtHMmCUttGiQX/S4YHqcDKThEROLBLLhXJiOHYE66I1RX23gmE0sYZeQc+Xs3o+AQEekOUtOgV9/gkWDdq0VGREQSTsEhIiIxUXCIiEhMFBwiIhITBYeIiMREwSEiIjFRcIiISEwUHCIiEhNz90TXEHdmVgJsOszd84GdHVhOR1FdsVFdsVFdsempdY1w93eNRpkUwXEkzKzI3QsTXUdzqis2qis2qis2yVaXLlWJiEhMFBwiIhITBUfb7kx0Aa1QXbFRXbFRXbFJqrrUxiEiIjHRGYeIiMREwSEiIjFJ6uAwszPN7A0zW2dm81pYn2lmj4brl5jZyMi6b4bL3zCzMzq5ruvNbLWZrTCzZ8xsRGRdnZktDx8LO7muK82sJPL+cyPrrjCzteHjik6u6xeRmt40s72RdXH5vMzsHjMrNrPXWllvZnZLWPMKMzshsi6en1VbdV0a1rPSzF4ws8mRdRvD5cvNrKiT6zrVzPZFvlffiaw75Pc/znX9V6Sm18Kfp/7hunh+XsPNbFH4e2CVmX2phW3i9zPm7kn5AFKB9cBoIAN4FZjQbJvPAbeHzy8CHg2fTwi3zwRGhcdJ7cS6TgOyw+efbagrfF2WwM/rSuDWFvbtD2wI/+0XPu/XWXU12/4LwD2d8Hm9HzgBeK2V9R8FngQMOBFYEu/Pqp11ndzwfsBZDXWFrzcC+Qn6vE4F/u9Iv/8dXVezbT8O/LOTPq8hwAnh8zzgzRb+P8btZyyZzzhmAOvcfYO7VwOPAOc02+Yc4P7w+QLgQ2Zm4fJH3L3K3d8C1oXH65S63H2RuzfMXL8YGNZB731EdR3CGcA/3H23u+8B/gGcmaC6LgYe7qD3bpW7Pw/sPsQm5wDzPbAY6GtmQ4jvZ9VmXe7+Qvi+0Hk/W+35vFpzJD+XHV1Xp/xsAbj7Nnd/OXxeCrwODG22Wdx+xpI5OIYCmyOvt/DuD/7gNu5eC+wDBrRz33jWFTWH4K+KBllmVmRmi83s3A6qKZa6PhmeFi8ws+Ex7hvPuggv6Y0C/hlZHK/Pqy2t1R3PzypWzX+2HPi7mS0zs2sSUM9JZvaqmT1pZhPDZV3i8zKzbIJfvr+PLO6Uz8uCS+hTgSXNVsXtZywt5iqlyzCz2UAh8IHI4hHuvtXMRgP/NLOV7r6+k0r6C/Cwu1eZ2WcIztY+2Env3R4XAQvcvS6yLJGfV5dlZqcRBMd7I4vfG35WA4F/mNma8C/yzvAywfeqzMw+CvwJGNdJ790eHwf+4+7Rs5O4f15mlksQVl929/0deexDSeYzjq3A8MjrYeGyFrcxszSgD7CrnfvGsy7M7HTgBmCWu1c1LHf3reG/G4BnCf4S6ZS63H1XpJa7gGnt3TeedUVcRLNLCXH8vNrSWt3x/KzaxcwmEXz/znH3XQ3LI59VMfBHOu7ybJvcfb+7l4XPnwDSzSyfLvB5hQ71sxWXz8vM0glC4yF3/0MLm8TvZyweDTfd4UFwtrWB4NJFQ6PaxGbbfJ6mjeOPhc8n0rRxfAMd1zjenrqmEjQIjmu2vB+QGT7PB9bSQQ2F7axrSOT5J4DF3tgY91ZYX7/wef/Oqivc7liCxkrrjM8rPOZIWm/sPZumDZdL4/1ZtbOuowna7E5utjwHyIs8fwE4sxPrGtzwvSP4Bfx2+Nm16/sfr7rC9X0I2kFyOuvzCr/2+cAvD7FN3H7GOuzD7Y4Pgl4HbxL8Er4hXHYjwV/xAFnA4+F/pKXA6Mi+N4T7vQGc1cl1PQ3sAJaHj4Xh8pOBleF/npXAnE6u68fAqvD9FwHHRva9Ovwc1wFXdWZd4evvATc12y9unxfBX5/bgBqCa8hzgGuBa8P1BtwW1rwSKOykz6qtuu4C9kR+torC5aPDz+nV8Ht8QyfXdV3kZ2sxkWBr6fvfWXWF21xJ0Fkmul+8P6/3ErShrIh8rz7aWT9jGnJERERiksxtHCIichgUHCIiEhMFh4iIxETBISIiMVFwiIhITBQcIl1cODLs/yW6DpEGCg4REYmJgkOkg5jZbDNbGs6/cIeZpZpZmQXzgayyYO6UgnDbKeHAiivM7I9m1i9cPtbMng4H83vZzMaEh88NB45cY2YPhaM0iySEgkOkA5jZeOBC4BR3nwLUAZcSDDdR5O4TgeeA74a7zAe+4e6TCO7qbVj+EHCbu08muLN9W7h8KvBlgrlgRgOnxPlLEmmVRscV6RgfIhjU8aXwZKAXUAzUA4+G2zwI/MHM+gB93f25cPn9wONmlgcMdfc/Arj7AYDweEvdfUv4ejnB+En/jvtXJdICBYdIxzDgfnf/ZpOFZt9utt3hjvFTFXleh/7vSgLpUpVIx3gG+FQ49wJm1j+cOCoF+FS4zSXAv919H7DHzN4XLr8MeM6Dmdy2NEwoZcGc99md+UWItIf+ahHpAO6+2sy+RTDjWwrBaKqfB8qBGeG6YoJ2EIArgNvDYNgAXBUuvwy4w8xuDI9xfid+GSLtotFxReLIzMrcPTfRdYh0JF2qEhGRmOiMQ0REYqIzDhERiYmCQ0REYqLgEBGRmCg4REQkJgoOERGJyf8HejpklbeBaFQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(np.sqrt(losses))\n",
    "plt.plot(np.sqrt(losses_eval))\n",
    "plt.title('model RMSE')\n",
    "plt.ylabel('RMSE')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'val'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Классификация"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Теперь проделаем тоже самое с классификацией. \n",
    "\n",
    "Данные возьмем из соревнования - https://www.kaggle.com/c/quora-insincere-questions-classification/overview  \n",
    "В этом соревновании нужно было предсказывать токсичность вопроса."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "quora = pd.read_csv('quora.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>qid</th>\n",
       "      <th>question_text</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>00002165364db923c7e6</td>\n",
       "      <td>How did Quebec nationalists see their province...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>000032939017120e6e44</td>\n",
       "      <td>Do you have an adopted dog, how would you enco...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0000412ca6e4628ce2cf</td>\n",
       "      <td>Why does velocity affect time? Does velocity a...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>000042bf85aa498cd78e</td>\n",
       "      <td>How did Otto von Guericke used the Magdeburg h...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0000455dfa3e01eae3af</td>\n",
       "      <td>Can I convert montra helicon D to a mountain b...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1306117</th>\n",
       "      <td>ffffcc4e2331aaf1e41e</td>\n",
       "      <td>What other technical skills do you need as a c...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1306118</th>\n",
       "      <td>ffffd431801e5a2f4861</td>\n",
       "      <td>Does MS in ECE have good job prospects in USA ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1306119</th>\n",
       "      <td>ffffd48fb36b63db010c</td>\n",
       "      <td>Is foam insulation toxic?</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1306120</th>\n",
       "      <td>ffffec519fa37cf60c78</td>\n",
       "      <td>How can one start a research project based on ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1306121</th>\n",
       "      <td>ffffed09fedb5088744a</td>\n",
       "      <td>Who wins in a battle between a Wolverine and a...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1306122 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                          qid  \\\n",
       "0        00002165364db923c7e6   \n",
       "1        000032939017120e6e44   \n",
       "2        0000412ca6e4628ce2cf   \n",
       "3        000042bf85aa498cd78e   \n",
       "4        0000455dfa3e01eae3af   \n",
       "...                       ...   \n",
       "1306117  ffffcc4e2331aaf1e41e   \n",
       "1306118  ffffd431801e5a2f4861   \n",
       "1306119  ffffd48fb36b63db010c   \n",
       "1306120  ffffec519fa37cf60c78   \n",
       "1306121  ffffed09fedb5088744a   \n",
       "\n",
       "                                             question_text  target  \n",
       "0        How did Quebec nationalists see their province...       0  \n",
       "1        Do you have an adopted dog, how would you enco...       0  \n",
       "2        Why does velocity affect time? Does velocity a...       0  \n",
       "3        How did Otto von Guericke used the Magdeburg h...       0  \n",
       "4        Can I convert montra helicon D to a mountain b...       0  \n",
       "...                                                    ...     ...  \n",
       "1306117  What other technical skills do you need as a c...       0  \n",
       "1306118  Does MS in ECE have good job prospects in USA ...       0  \n",
       "1306119                          Is foam insulation toxic?       0  \n",
       "1306120  How can one start a research project based on ...       0  \n",
       "1306121  Who wins in a battle between a Wolverine and a...       0  \n",
       "\n",
       "[1306122 rows x 3 columns]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "quora"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Предобработка точно такая же"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# словарь\n",
    "vocab = Counter()\n",
    "\n",
    "for text in quora.question_text:\n",
    "    vocab.update(preprocess(text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "273056"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# отфильтрованный словарь\n",
    "filtered_vocab = set()\n",
    "\n",
    "for word in vocab:\n",
    "    if vocab[word] > 50:\n",
    "        filtered_vocab.add(word)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "14083"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(filtered_vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "# индексируем слова\n",
    "word2id = {'PAD':0}\n",
    "\n",
    "for word in filtered_vocab:\n",
    "    word2id[word] = len(word2id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "id2word = {i:word for word, i in word2id.items()}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "# поставит тоже 100\n",
    "MAX_LEN = 100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_dataset, valid_dataset = train_test_split(quora, test_size=0.05)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Разлчия в самой модели тоже минимальные - нужно поменять активацию в последнем слое, лосс и можно добавить метрику."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_set = Dataset(train_dataset, word2id, MAX_LEN, 'question_text', 'target')\n",
    "training_generator = torch.utils.data.DataLoader(training_set, batch_size=256)\n",
    "\n",
    "valid_set = Dataset(valid_dataset, word2id, MAX_LEN, 'question_text', 'target')\n",
    "valid_generator = torch.utils.data.DataLoader(valid_set, batch_size=256)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Как выбирать batch_size? Размер батча может влиять на качество модели и точно влияет на скорость обучения. Обычно, чем больше батч, тем быстрее будет обучаться модель. Размер батча ограничен памятью видеокарты, поэтому чем лучше gpu, тем больше может быть batch_size. \n",
    "\n",
    "Примерное максимальное значение можно найти быстрым перебором. Если появляется ошибка, то слишком большой batch_size, а если обучается, то можно попробовать поставить побольше."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Clf(nn.Module):\n",
    "    \n",
    "    def __init__(self, vocab_size, embedding_dim, output_dim):\n",
    "        \n",
    "        super().__init__()          \n",
    "        self.embedding = nn.Embedding(vocab_size, embedding_dim)\n",
    "        self.flatten = nn.Flatten() # вместо усреднения склеим все в 1 вектор\n",
    "        self.fc = nn.Linear(embedding_dim*MAX_LEN, output_dim) # размер склееного вектора - размер эмбединга на MAX_LEN\n",
    "        self.act = nn.Sigmoid() # активацию менять даже не пришлось, так как у нас бинарная классификация\n",
    "                                # если классов больше 2, то нужно поставить nn.Softmax()\n",
    "        \n",
    "    def forward(self, text):\n",
    "        \n",
    "        embedded = self.embedding(text)   \n",
    "        hidden = self.flatten(embedded)\n",
    "        dense_outputs=self.fc(hidden)\n",
    "        outputs=self.act(dense_outputs)\n",
    "        \n",
    "        return outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Немного перепишем функции тренировки и обучения, чтобы еще и accuracy посчитать"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "def binary_accuracy(preds, y):\n",
    "    rounded_preds = torch.round(preds)\n",
    "    \n",
    "    correct = (rounded_preds == y).float() \n",
    "    acc = correct.sum() / len(correct)\n",
    "    return acc\n",
    "\n",
    "\n",
    "def train(model, iterator, optimizer, criterion, print_every=1000):\n",
    "    \n",
    "    epoch_loss = 0\n",
    "    epoch_acc = 0\n",
    "    \n",
    "    model.train()  \n",
    "\n",
    "    for i, (texts, ys) in enumerate(iterator):\n",
    "        \n",
    "        optimizer.zero_grad()   \n",
    "        predictions = model(texts.to(device)).squeeze()  \n",
    "        loss = criterion(predictions.float(), ys.to(device).float())        \n",
    "        acc = binary_accuracy(predictions.float(), ys.to(device).float())\n",
    "        loss.backward()     \n",
    "        \n",
    "        optimizer.step()      \n",
    "        \n",
    "        #лосс и еще и метрика теперь\n",
    "        epoch_loss += loss.item()  \n",
    "        epoch_acc += acc.item()\n",
    "        \n",
    "        if not (i+1) % print_every:\n",
    "            print(f'Loss: {epoch_loss/i}; Accuracy: {epoch_acc/i}')\n",
    "        \n",
    "    return epoch_acc / len(iterator)\n",
    "\n",
    "\n",
    "def evaluate(model, iterator, criterion):\n",
    "    \n",
    "    epoch_loss = 0\n",
    "    epoch_acc = 0\n",
    "    \n",
    "    model.eval()  \n",
    "    with torch.no_grad():\n",
    "        for texts, ys in iterator:   \n",
    "\n",
    "            predictions = model(texts.to(device)).squeeze()  \n",
    "            loss = criterion(predictions.float(), ys.to(device).float())        \n",
    "            acc = binary_accuracy(predictions.float(), ys.to(device).float())\n",
    "\n",
    "            epoch_loss += loss.item()  \n",
    "            epoch_acc += acc.item()\n",
    "            \n",
    "    return epoch_acc / len(iterator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.optim as optim\n",
    "\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')  \n",
    "model = Clf(len(word2id), 30, 1)\n",
    "\n",
    "optimizer = optim.Adam(model.parameters())\n",
    "criterion = nn.BCELoss() # другой лосс - Binary cross-entropy\n",
    "\n",
    "model = model.to(device)\n",
    "criterion = criterion.to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "Loss: 0.1853803273913142; Accuracy: 0.9401041666666666\n",
      "Loss: 0.1686513783254583; Accuracy: 0.942557216108054\n",
      "Loss: 0.1597724282724653; Accuracy: 0.94420536012004\n",
      "Loss: 0.15332563948906283; Accuracy: 0.9457422949487372\n",
      "1\n",
      "Loss: 0.12715587534152112; Accuracy: 0.9525932182182182\n",
      "Loss: 0.12566945568219073; Accuracy: 0.9526071629564783\n",
      "Loss: 0.12528697234833389; Accuracy: 0.9525597074024675\n",
      "Loss: 0.12430719831442798; Accuracy: 0.9528339116029008\n",
      "2\n",
      "Loss: 0.12027006918887; Accuracy: 0.9544779154154154\n",
      "Loss: 0.11945270453246609; Accuracy: 0.954348267883942\n",
      "Loss: 0.11944647719533931; Accuracy: 0.954176131627209\n",
      "Loss: 0.11887520184182888; Accuracy: 0.9543421011502876\n"
     ]
    }
   ],
   "source": [
    "accs = []\n",
    "accs_eval = []\n",
    "for i in range(3):\n",
    "    print(i)\n",
    "    accs.append(train(model, training_generator, optimizer, criterion))\n",
    "    accs_eval.append(evaluate(model, valid_generator, criterion))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "это тоже нормальный график, но тут acc должна расти"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEWCAYAAABxMXBSAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Z1A+gAAAACXBIWXMAAAsTAAALEwEAmpwYAAAx20lEQVR4nO3deXwV5dn/8c9FCIQlQMIOgQQQ2QSBRMSl7m1R664IrrhAW/Wx2lUfu/rUp/bXfbFPi4piVXBf2mq1KmqtgiQsguwgS1gDJIEAIdv1+2Mm4RADJJCTk+R8369XXsyZuWfOdSZhvue+55wZc3dERERqq0WsCxARkaZFwSEiInWi4BARkTpRcIiISJ0oOEREpE4UHCIiUicKDpHDMLPHzeyntWy71szOi3ZNIrGm4BBpQGb2YzNzMzs51rWIHC0Fh0gDMTMDbgB2hv825HO3bMjnk+ZNwSFNXjhE9B0z+8TM9pjZo2bW3cxeN7PdZvaWmaVEtL/YzD41swIze9fMhkQsG2Vm88L1ngGSqj3XV8xsQbjuh2Y2og6lfgHoCdwJTDCzVhHbbWNmvzKzdWZWaGYfmFmbcNnp4XMVmNkGM5sUzn/XzG6N2MYkM/sg4rGb2e1mthJYGc77XbiNXWaWY2ZfiGifYGb/bWarw9efY2Z9zOwhM/tVtf3wqpndXYfXLs2IgkOaiyuALwLHAxcBrwP/DXQl+Du/E8DMjgdmAHeFy14D/mZmrcID+cvAX4FU4Llwu4TrjgKmAV8FOgN/AV41s9a1rPFG4G/As+HjiyKW/RLIBE4Nn/u7QIWZpYev5Q9hvSOBBbV8PoBLgZOBoeHjueE2UoGngefMrDIcvwlMBC4AOgA3A3uB6cBEM2sBYGZdgPPC9SUOKTikufiDu291943Av4E57j7f3YuBl4BRYburgX+4+7/cvZTggN2G4IA9FkgEfuvupe7+PMGBttIU4C/uPsfdy919OrA/XO+wzKwtcBXwdPi8zxMOV4UH5JuBb7j7xnDbH7r7fuAa4C13nxHWtMPdF9Rhv/zM3Xe6+z4Ad38y3EaZu/8KaA0MCtveCnzf3Zd7YGHY9mOgEDg3bDcBeNfdt9ahDmlGFBzSXEQexPbV8Lh9ON0LWFe5wN0rgA1A73DZRj/4yp/rIqbTgW+FQ0YFZlYA9AnXO5LLgDKCHg7AU8D5ZtYV6EIwJLa6hvX6HGJ+bW2IfGBm3zazpeFwWAHQMXz+Iz3XdOC6cPo6gl6ZxCkFh8SbTQQBAFSdsO4DbAQ2A73DeZX6RkxvAB5w904RP23dfUYtnvdGgvBab2ZbCIbBEgl6FNuBYmBADettOMR8gD1A24jHPWpoUxWC4fmM7wLjgRR370TQk6h8vYd7rieBS8zsRGAIwZCexCkFh8SbZ4ELzexcM0sEvkUw3PQh8BFBr+BOM0s0s8uBMRHrPgx8zcxOtkA7M7vQzJIP94Rm1ptgmOcrBOcXRgInAj8Hbgh7PdOAX5tZr/Ak9SnhuZOngPPMbLyZtTSzzmY2Mtz0AuByM2trZscBtxzhtSeHry8PaGlmPyQ4l1HpEeB/zGxg+PpGmFlnAHfPJRi2+yvwQuXQl8QnBYfEFXdfTjDU8geCd/oXARe5e4m7lwCXA5MIPjJ7NfBixLrZwGTgj0A+sCpseyTXAwvc/U1331L5A/weGGFmJwDfBhYRHJx3EoRKC3dfT3Cy+lvh/AUEoQPwG6CEYFhuOkHIHM4bwD+BFQRDcMUcPJT1a4JgfRPYBTxKcP6n0nRgOBqminumGzmJSG2Y2RkEQ1bprgNHXFOPQ0SOKBzW+wbwiEJDFBwicljhFyQLCL68+NuYFiONgoaqRESkTtTjEBGROomLC5916dLFMzIyYl2GiEiTkpOTs93du1afHxfBkZGRQXZ2dqzLEBFpUsxsXU3zNVQlIiJ1ouAQEZE6UXCIiEidxMU5jpqUlpaSm5tLcXFxrEuJqqSkJNLS0khMTIx1KSLSTMRtcOTm5pKcnExGRgYHXwy1+XB3duzYQW5uLv369Yt1OSLSTMTtUFVxcTGdO3dutqEBYGZ07ty52feqRKRhxW1wAM06NCrFw2sUkYYVt0NVIiKNljuUl0DZ/vDf4mrTJVC+P5hXtr/adLU2J38d2nWu1/IUHDFSUFDA008/zW233Van9S644AKefvppOnXqFJ3CROKVO5SX1v6AXNcDeF3b1AuD4Vc1reAws3HA74AEgssxP1hteTrBnc+6Etyk5jp3zzWzswluUlNpMDDB3V82s38T3MkMoBvwsbtfGs3XEQ0FBQX86U9/+lxwlJWV0bLloX8tr7322iGXiTQ57lBRdowH5Jra1+Zdeg3bqRcGLVtDQuvg35atIaEVtEyClq3C+UmQ1DFifk1tWtduO4dr36IlRGG4OmrBYWYJwEPAF4FcYK6ZveruSyKa/RJ4wt2nm9k5wM+A6919FsHtNTGzVII7rb0J4O5fiHiOF4BXovUaoumee+5h9erVjBw5ksTERJKSkkhJSWHZsmWsWLGCSy+9lA0bNlBcXMw3vvENpkyZAhy4fEpRURHnn38+p59+Oh9++CG9e/fmlVdeoU2bNkd4Zol7VQfrmg7K+4MDaVlxtemSWrSvbFNT+8MEAfV0he6WSeFBMzy4JrSqdjBtBUkdIuYfos0Rt1OtffUDeEJiVA7WjUk0exxjgFXuvgbAzGYClwCRwTEU+GY4PQt4uYbtXAm87u57I2eaWQfgHOCmYy30J3/7lCWbdh3rZg4ytFcHfnTRsEMuf/DBB1m8eDELFizg3Xff5cILL2Tx4sVVH5udNm0aqamp7Nu3j5NOOokrrriCzp0P7m6uXLmSGTNm8PDDDzN+/HheeOEFrrvuunp9HRJF+4tg16aId8ZHeUA+4kG+hjb1dbA+5LveiHfGrZOP4t1zHQ/gcXCwbkyiGRy9Ofh+xrnAydXaLCS4x/PvgMuAZDPr7O47ItpMILgXcnWXAm+7e/0e8WNkzJgxB33X4ve//z0vvfQSABs2bGDlypWfC45+/foxcuRIADIzM1m7dm1DlSt1UbIH8pZD3jLYtjT8dxkUrj+67dX0rrf6gbVV+3oe/qhhSEUH67gV65Pj3wb+aGaTgPeBjUB55UIz6wkMB96oYd2JwCOH2rCZTQGmAPTt2/ewRRyuZ9BQ2rVrVzX97rvv8tZbb/HRRx/Rtm1bzjrrrBq/i9G6deuq6YSEBPbt29cgtcohlO6rISCWQsF6qt7hJ7SCLsdDnzGQeQOk9DtwQD5kEERMJ7TSwVpiLprBsRHoE/E4LZxXxd03EfQ4MLP2wBXuXhDRZDzwkruXRq5nZl0IhsIuO9STu/tUYCpAVlZWo7vNYXJyMrt3765xWWFhISkpKbRt25Zly5Yxe/bsBq5ODqu0GHasDHoNeUsP/LvzM6oCokUidD4Oeo+GkddCt8HQbWgQFAmxfr8mcmyi+Rc8FxhoZv0IAmMCcE1kgzAAdrp7BXAvwSesIk0M51d3JfB3d2+yX4nu3Lkzp512GieccAJt2rShe/fuVcvGjRvHn//8Z4YMGcKgQYMYO3ZsDCuNY2UlYUAsPbgXsXMNeEXQpkVLSB0APYbD8PFBQHQdAp0HBEM5Is1QVO85bmYXENzcPgGY5u4PmNn9QLa7v2pmVxJ8ksoJhqpud/f94boZwH+APmGwRG73XeBBd/9nberIysry6jdyWrp0KUOGDDmGV9d0xNNrPSrlpbBj1ecDYsdq8HDk1BIgtf+BYKgKiOOCoSWRZsjMctw9q/r8qPaZ3f014LVq834YMf088Pwh1l1LcIK9pmVn1VuREj/Ky4LeQuTw0rZlQWhUVI6GGqT2C0JhyEXB8FLXwdBlYHCOQURifnJcpP5VlAfnGz4XECuDj6oCYJCSHgTEoHEHehFdjodEfRdGmofi0nKSEhPqfbsKDmm6Ksohf+3nP+a6fcXBl2zo1DcIhoHnRQTEIGjVNmali0TDtt3F5KzNJ3td8PPpxkL+/b2z6dmxft8MKTik8auoCL7zENl72LYEtq+EsoiPIHdIC0Kh/5nQbUjw02UQtG4fu9pFoqSiwlm5rYjsdTurwmL9zuB70q1btuDEtE5MPqM/LZrSJUdE6swdCjd8/mOueSugdM+Bdsm9goDI+MKBk9RdBwWXkxBppvaWlLFgQwE5a/PJWZ/PvHX57CouA6BL+1Zkpqdw/dh0MjNSOKFXR1q1jN5dMxQc0vDcYdfGGgJiOZQUHWjXvkcQDKNvODgg2nSKWekiDWXrrmKy1+YHPYp1+SzZtIuyiuBTsMd3b8+FI3qRlZ5CZnoK6Z3bNui9dxQcTUT79u0pKio6csPGxB12b/n8Seq8ZbA/4kox7boGn1waeU3wb7chwb9tU2NXu0gDKq9wVmzdTfa6fHLW7iR7XT65+cEwbFJiMOz01TP7k5Weyui+KXRsG9vvCCk45Ni5Q9G2GgJiKRQXHmjXtnPQaxgxPiIghtT7vQJEGrs9+4Nhp8oexYL1BezeHww7dU1uTVZ6CpNOzSArI5VhvTqQmNC4btaq4IiRe+65hz59+nD77bcD8OMf/5iWLVsya9Ys8vPzKS0t5ac//SmXXHJJjCutpiiv5oDYl3+gTZuUIBBOuOLgL8u17xq7ukViaHPhPrLX5pOzLgiKpZt3U17hmMGg7slcPLIXWRkpZKWnkpbSptHf8lnBAfD6PbBlUf1us8dwOP/BQy6++uqrueuuu6qC49lnn+WNN97gzjvvpEOHDmzfvp2xY8dy8cUXx+aPaO/O8COulZ9iCqf3Rly4uHXHIBSGXHxgeKnbEGjfXRfik7hVXuEs3bwrDIngJPbGgmDYqU1iAiP7dOK2swaQmZ7CqL4pdGzT9C5No+CIkVGjRrFt2zY2bdpEXl4eKSkp9OjRg7vvvpv333+fFi1asHHjRrZu3UqPHj2iV8i+/M/3HrYtgz3bDrRplRwExKALDg6I5J4KCIl7RfvLmL8+v6pHMX99PntKgkvV9OiQRGZGCrd+oR9Z6akM7pnc6IadjoaCAw7bM4imq666iueff54tW7Zw9dVX89RTT5GXl0dOTg6JiYlkZGTUeDn1o1JcWHNAFG050KZV++BTSwO/dPA1mTr0VkCIAO7OxoJ95KwLh53W5rNsyy4qHFoYDOrRgctHp5GVEXzaqXenxj/sdDQUHDF09dVXM3nyZLZv3857773Hs88+S7du3UhMTGTWrFmsW7eu7hutKA/u8FZaHN5Zbl9wl7kHI66wm9g2uLTGgLPD3sPQMCDSoEXTfzckUl/KyitYunk32et2hp94ymfLruDNXLtWCYzqm8Id5wwkKz2FUX07kZzU9IadjoaCI4aGDRvG7t276d27Nz179uTaa6/loosuYvjw4WRlZTF48OBDr1wZEAeFRHHEtZgA7MBNgs790YFhpk7pCgiRGuwqLmX++oKqj8Qu2FDA3nDYqVfHJE7ql1r13YnBPZJp2QyGnY6GgiPGFi06cFK+S5cufPTRRwc3qCiHkr0UbVsXfGmu9DAB0aodtOwcTCeG92s2gx1LIfObiMgB7k5u/r6gNxGen1i+dTceDjsN6dmBqzLTyMwIwqJXJ138spKCo7GoqDjQaygrDm5DWmNAtA6GmtqGAVHZo2iG46gi9am0vIIlm3YFQ05hWGzbHVwMs33rlozq24nzT+hJVkYKI/t0ol1rHR4PRXumoXkFlO0/EAyVQ02RV3M9KCBSw3BoE9wwyOKzayxSV4V7S5m3/sAlOxZuKGRfaTDs1LtTG04Z0DkcdkplUI9kElrozVdtxXVwuHv0PvFQGRBV5yD21RAQhAGRFHxpLjGyB1E/ARHNOzyKNBbuzvqde8NvYgc9ihVbg0v0JLQwhvXqwIQxfchMD75k16NjUowrbtriNjiSkpLYsWMHnTt3Prbw8Irg3tRl+6qdqN5PcEfcUEJlQHQKz0G0qdeAqLE0d3bs2EFSkv6TSPNSUlbBp5sKqz4Sm70un+1FwZuy5KSWjO6bwkUjepEZDju1bRW3h7qoiNu9mZaWRm5uLnl5ebVbwR0qyoJbjJaHPxVlwb+RAdEiERJaQkJiOB3+awbsD38Ka36OKEhKSiItLa3Bnk8kGgr2llR9EztnbT4LcwvYX1YBQN/UtnxhYJegN5GRwvHdkmmhYaeoitvgSExMpF+/fodvlDMdPns/uJrr9hUHn6julH7wt6i7hrcd1V3lRI6Ju7N2x16y1+6sCotV24Jhp5YtjGG9O3Ld2PSqj8V266AedUOL2+ColdXvwMacIBQGnH3wbUd1VzmRerG/rJzFGwurhpzmrctnx57gTVqHpJZkpqdw2ajeZKancGJaJ9q0qv97aEvdKDgO58pp0EJ/pCL1aeeekqqrxOaszeeTjYWUhMNOGZ3bctagblWX7Diua3sNOzVCCo7DUWiIHBN3Z3XeHuaFQZG9Lp81ecFtgBMTjBN6d+TGU9LJTE8lMz2FrsmtY1yx1IaCQ0TqTXFpOYvCYaec8PsT+XtLAUhpm0hmegpXZqaRlZ7KiLSOJCXqzVlTpOAQkaO2vWh/xJVid7J44y5KyoNhp/5d2nHekO7hsFMqA7q2a5ZXio1HCg4RqZWKCmd1XhHZ6/KrehRrd+wFoFVCC4andeSm0zLIDD/t1Lm9hp2aKwWHiNSouLSchRsKwm9iBz+F+4Jhp9R2rchMT2HimL5kZaQwrJeGneKJgkNEANi2u5icqvti5/PppkJKy4Mvtw7o2o5xw3qQmZFCVnoK/bpo2CmeKThE4lBFhbNyW1HVR2Kz1+Wzfmc47NSyBSemdeSW0/tXfckupV2rGFcsjYmCQyQO7C0pY+GGwuBy4uGX7HYVlwHQpX0w7HT92HQyM1I4oVdHWrXUVZjl0BQcIs3Q1l3F4Texg4/ELtm0i7KKYNjp+O7tuXBETzLTgxsUpXduq2EnqRMFh0gTV17hrNi6O7wAYNCjyM3fB0BSYgtOTOvEV8/sT1Z6KqP7ptCxbXzcF1uiR8Eh0sTs2V/Ggg0FwUdi1+czf10+u/cHw05dk1uTlZ7CpFMzyMpIZWjPDhp2knqn4BBp5DYX7qu6J3b2up0s3byb8grHDAZ1T+bikb2qblDUJ7WNhp0k6hQcIo1IeYWzdPOu4JanYVhsLAiGndokJjCyTyduO2sAmekpjOqbQsc2GnaShqfgEImhov1lzI8Iifnr89lTEtwXu3uH1mSlp3LL6f3IykhhSM8OJCZo2EliT8EhEgPuzi/eWM6f31tNhQc3iBzcowOXj06ruqR4704adpLGScEh0sDcnQf+sZRHPviMS0f24vLRaYzq24nkJA07SdMQ1X6vmY0zs+VmtsrM7qlhebqZvW1mn5jZu2aWFs4/28wWRPwUm9ml4TIzswfMbIWZLTWzO6P5GkTqk7vzP38PQmPSqRn85uqRnHF8V4WGNClR63GYWQLwEPBFIBeYa2avuvuSiGa/BJ5w9+lmdg7wM+B6d58FjAy3kwqsAt4M15kE9AEGu3uFmXWL1msQqU/uzv1/X8Jj/1nLTadl8MOvDNVQlDRJ0exxjAFWufsady8BZgKXVGszFHgnnJ5Vw3KAK4HX3X1v+PjrwP3uXgHg7tvqvXKReubu/ORvQWjcfFo/hYY0adEMjt7AhojHueG8SAuBy8Ppy4BkM+tcrc0EYEbE4wHA1WaWbWavm9nAmp7czKaEbbLz8vKO+kWIHCt350evfsrjH67l1tP78YOvDFFoSJMW68/2fRs408zmA2cCG4HyyoVm1hMYDrwRsU5roNjds4CHgWk1bdjdp7p7lrtnde3aNVr1ixxWRYXzg1cW88RH65hyRn/uu1ChIU1fND9VtZHgXESltHBeFXffRNjjMLP2wBXuXhDRZDzwkruXRszLBV4Mp18CHqvfskXqR2VoPDVnPV89oz/3nD9YoSHNQjR7HHOBgWbWz8xaEQw5vRrZwMy6mFllDffy+d7DRA4epgJ4GTg7nD4TWFGfRYvUh4oK5/thaHztzAEKDWlWohYc7l4G3EEwzLQUeNbdPzWz+83s4rDZWcByM1sBdAceqFzfzDIIeizvVdv0g8AVZraI4FNYt0brNYgcjYoK576XF/H0nPXcdtYAvjdukEJDmhVz91jXEHVZWVmenZ0d6zIkDlRUOPe+uIhnsjdwx9nH8a0vHa/QkCbLzHLC88kH0TfHRepJRYXzvRc+4bmcXO485zju/qJCQ5onBYdIPSgPQ+P5nFzuPHcgd583UKEhzZaCQ+QYlVc4333+E16Yl8td5w3krvOOj3VJIlGl4BA5BuUVzneeW8iL8zdy93nH843zavw+qkizouAQOUrlFc63nl3Ayws28a0vHs9/navQkPig4BA5CmXlFXzruYW8smAT3/nyIG4/+7hYlyTSYBQcInVUVl7B3c8u5G8LFRoSnxQcInVQVl7BXc8s4O+fbOZ74wbz9bMGxLokkQan4BCppdLyCu6auYB/LNrMvecP5qtnKjQkPik4RGqhtLyCb8ycz2uLtnDfBUOYfEb/WJckEjMKDpEjKC2v4M4Z83l98Ra+f+EQbv2CQkPim4JD5DBKyir4rxnzeOPTrfzgK0O55fR+sS5JJOYUHCKHUFJWwR1Pz+PNJVv50UVDuek0hYYIKDhEalRSVsFtT83jraVb+cnFw7jx1IxYlyTSaCg4RKrZX1bO7U/N462l27j/kmHccEpGrEsSaVQUHCIR9peV8/Un5/HOsm38z6UncP3Y9FiXJNLoKDhEQsWl5Xz9yRxmLc/jp5eewHUKDZEaKThECELja0/m8O7yPP73suFcc3LfWJck0mgpOCTuFZeWM+WvOby/Io8HLx/OhDEKDZHDUXBIXCsuLWfyE9l8sGo7/++KEYw/qU+sSxJp9BQcErciQ+PnV4xgfJZCQ6Q2FBwSl/aVBKHxn9VBT+MqhYZIrSk4JO7sKynnlulz+WjNDn555YlckZkW65JEmhQFh8SVvSVl3PJ4NnM+28Gvx5/IZaMUGiJ1peCQuLG3pIybHpvL3LU7+fX4kVw6qnesSxJpkhQcEhf27C/jpsfnkr12J7+5eiSXjFRoiBwtBYc0e3v2Bz2N7HU7+e2EUVx8Yq9YlyTSpCk4pFkr2l/GTY99zLz1BfxuwiguUmiIHDMFhzRbu4tLmfTYXBZsKOD3E0Zx4YiesS5JpFlQcEiztLu4lBunfcwnuYX8ceIozh+u0BCpLwoOaXZ2haGxKLeQP14zinEnKDRE6pOCQ5qVXcWl3PDoxyzeWMgfrxnNuBN6xLokkWZHwSHNRuG+Um6Y9jFLNhXyp2tH86VhCg2RaFBwSLNQuLeU66fNYenmXfzp2ky+OLR7rEsSabYUHNLkFe4t5bpH57B8y27+fF0m5w5RaIhEk4JDmrSCvSVc9+gcVmwp4s/Xj+acwQoNkWhrcaQGZtbOzFpEPG5hZm2jW5bIkRXsLeHaR4LQ+Mv1mQoNkQZyxOAA3gYig6It8FZtNm5m48xsuZmtMrN7aliebmZvm9knZvaumaWF8882swURP8Vmdmm47HEz+yxi2cja1CLNS/6eEq55eA4rtxUx9YZMzh7cLdYlicSN2gxVJbl7UeUDdy+qTY/DzBKAh4AvArnAXDN71d2XRDT7JfCEu083s3OAnwHXu/ssYGS4nVRgFfBmxHrfcffna1G7NEM79wQ9jdV5RTx8QxZnHt811iWJxJXa9Dj2mNnoygdmlgnsq8V6Y4BV7r7G3UuAmcAl1doMBd4Jp2fVsBzgSuB1d99bi+eUZm7nnhKueXg2a/KKeEShIRITtQmOu4DnzOzfZvYB8AxwRy3W6w1siHicG86LtBC4PJy+DEg2s87V2kwAZlSb90A4vPUbM2tdi1qkGdhRtJ9rHp7NZ9v38OiNJ3GGQkMkJo4YHO4+FxgMfB34GjDE3XPq6fm/DZxpZvOBM4GNQHnlQjPrCQwH3ohY596wnpOAVOB7NW3YzKaYWbaZZefl5dVTuRIr24v2c83Dc1i7Yw/TJp3E6QO7xLokkbhVm09V3Q60c/fF7r4YaG9mt9Vi2xuBPhGP08J5Vdx9k7tf7u6jgPvCeQURTcYDL7l7acQ6mz2wH3iMYEjsc9x9qrtnuXtW1656Z9qU5e3ez8Sps1m3cw/TbjyJ045TaIjEUm2GqiZHHszdPR+YXIv15gIDzayfmbUiGHJ6NbKBmXWJ+KjvvcC0atuYSLVhqrAXgpkZcCmwuBa1SBO1bXcxEx+eTW7+Ph6bNIZTFRoiMVeb4EgID9JA1aelWh1pJXcvIzgX8gawFHjW3T81s/vN7OKw2VnAcjNbAXQHHoh4ngyCHst71Tb9lJktAhYBXYCf1uI1SBO0bXcxE6fOZmP+Ph676SROGVD99JeIxIK5++EbmP0CSAf+Es76KrDe3b8d5drqTVZWlmdnZ8e6DKmDbbuCnsbmwmIem3QSJ/dXaIg0NDPLcfes6vNr8z2O7wFTCE6MA3wC6LKjEjVbdwU9jS27inn8pjGM6Zca65JEJEJtPlVVAcwB1hKciD6HYOhJpN5tKSxmwtTZbN1VzPSbFRoijdEhexxmdjzByemJwHaC72/g7mc3TGkSbzYX7mPi1NlsLyrhiVvGkJmu0BBpjA43VLUM+DfwFXdfBWBmdzdIVRJ3NhfuY8LU2ewoKmH6zWPITE+JdUkicgiHG6q6HNgMzDKzh83sXMAO017kqGwqCEJjZ1VPQ6Eh0pgdMjjc/WV3n0DwLe1ZBJce6WZm/2dmX2qg+qSZ21gtNEb3VWiINHa1OTm+x92fdveLCL79PZ9DXOZDpC5y8/cyYepH5O8t4a+3nswohYZIk1CbLwBWcff88FIe50arIIkPG3buZcLU2RTuLeWpW09mZJ9OsS5JRGpJt46VBlcZGkX7y3jq1rEMT+sY65JEpA4UHNKgDg6Nkzmht0JDpKlRcEiDWb8jOKext7RcoSHShCk4pEGs27GHCVNnsy8MjWG9FBoiTZWCQ6Ju7fYgNPaXlfP0rWMZ2qtDrEsSkWOg4JCo+mz7HiZM/YjScufpyWMZ0lOhIdLUKTgkatbkFTHx4dmUlTszJo9lUI/kWJckIvVAwSFRsTqviIlTZ1NeEfQ0FBoizYeCQ+rdqm1BT8PdmTFlLMd3V2iINCcKDqlXq7btZsLUOQDMmDyWgQoNkWZHwSH1ZuXW3Ux8eDZmxozJYzmuW/tYlyQiUVCna1WJHMqKMDRamDFzikJDpDlTcMgxW75lNxOnBqExY8pYBnRVaIg0ZwoOOSbLtuxi4sOzaZkQ9DQUGiLNn4JDjtqSTbuYOHU2rRJaMHPKKfRXaIjEBZ0cl6OyZNMurn1kNkmJCcyYPJaMLu1iXZKINBD1OKTOPt1UyDWPzKZNYgIzpyg0ROKNehxSJ4s3FnLtI3No37olMyaPpW/ntrEuSUQamIJDam1RbiHXPRqExswpY+mTqtAQiUcaqpJa+SS3gGsfma3QEBH1OOTIFm4o4PpH59ChTSIzp4wlLUWhIRLPFBxyWAvC0OjUNpGZU06hd6c2sS5JRGJMwSGHNH99Pjc8+jEp7Voxc8pYeik0RASd45BDyFmXz/WPfkxqe4WGiBxMPQ75nJx1O7lx2ly6tG/FjClj6dlRoSEiB6jHIQfJXruTGx79mK7JrZk55RSFhoh8joJDqsxdu5Mbp31M9w5JzJwylh4dk2Jdkog0QhqqEgA+/mwnkx77mB4dk5g5eSzdOig0RKRm6nEIs9fsYNJjH9NToSEitaDgiHMfrd7BTY/NpVenNsyYotAQkSOLanCY2TgzW25mq8zsnhqWp5vZ22b2iZm9a2Zp4fyzzWxBxE+xmV1abd3fm1lRNOtv7j5cvZ2bH59LWkobZkweS7dkhYaIHFnUgsPMEoCHgPOBocBEMxtardkvgSfcfQRwP/AzAHef5e4j3X0kcA6wF3gzYttZQEq0ao8HH64KQqNPatDT6JrcOtYliUgTEc0exxhglbuvcfcSYCZwSbU2Q4F3wulZNSwHuBJ43d33QlUg/QL4blSqjgMfrNzOTY/PJT21HTMmj6VLe4WGiNReNIOjN7Ah4nFuOC/SQuDycPoyINnMOldrMwGYEfH4DuBVd998uCc3sylmlm1m2Xl5eXUuvrn698o8bpk+l35d2vH05JPprNAQkTqK9cnxbwNnmtl84ExgI1BeudDMegLDgTfCx72Aq4A/HGnD7j7V3bPcPatr167RqL3JeX9FHrdMzw5DY6xCQ0SOSjS/x7ER6BPxOC2cV8XdNxH2OMysPXCFuxdENBkPvOTupeHjUcBxwCozA2hrZqvc/biovIJm5L0VeUx+IpvjurbnqVtPJqVdq1iXJCJNVDR7HHOBgWbWz8xaEQw5vRrZwMy6mFllDfcC06ptYyIRw1Tu/g937+HuGe6eAexVaBzZrOXbmPxENgO7KTRE5NhFLTjcvYzgfMQbwFLgWXf/1MzuN7OLw2ZnAcvNbAXQHXigcn0zyyDosbwXrRrjwaxl2/jqEzkc312hISL1w9w91jVEXVZWlmdnZ8e6jAb39tKtfP3JeQzqkcxfbxlDp7YKDRGpPTPLcfes6vNjfXJcouStJVv52pM5DO6ZzJO3nKzQEJF6o+Bohv61ZCtffyqHoT078NdbTqZj28RYlyQizYiujtvMvPHpFu54eh7DenXkiVvG0CFJoSEi9Us9jmbkn4u3cPtT8ziht0JDRKJHPY5m4vVFm/mvGfMZkdaR6TePIVmhISJRoh5HM/Daos3cMWM+J/bppNAQkahTj6OJ+8cnm7lz5nxG9enE4zePoX1r/UpFJLrU42jC/rZwE3fOnE9m3xSFhog0GAVHE/XKgo18Y+Z8MtNTeOymkxQaItJgFBxN0CsLNnL3Mws4KSOVxyadRDuFhog0IB1xmpiX52/km88uYEy/VKZNOom2rfQrFJGGpaNOE/LivFy+/dxCxvbvzKM3nkSbVgmxLklE4pCGqpqI53Ny+dZzCzllgEJDRGJLPY4m4LnsDXz3hU84bUAXHrkxi6REhYaIxI56HI3cs3OD0Dj9OIWGiDQO6nE0Ys/MXc89Ly7iCwO7MvX6TIWGiDQK6nE0UjM+Xs/3XljEGQoNEWlkFByN0NNz1nPvi4s4e1BX/qLQEJFGRkNVjcyTs9fx/ZcXc87gbvzfdaNp3VKhISKNi3ocjchfP1rL919ezLkKDRFpxBQcjcQTH63lB698ynlDuvMnhYaINGIaqmoEHv/PZ/z4b0v44tDuPHTNaFq1VJ6LSOOl4IixaR98xv1/X8KXh3XnDxMVGiLS+Ck4YuiRf6/hp/9YyrhhPfjDNaNITFBoiEjjpyNVjFSGxvknKDREpGlRjyMGHn5/DQ+8tpQLh/fktxNGKjREpElRcDSwv7y3mp+9voyvjOjJb68eSUuFhog0MQqOBvR/767m5/9cxkUn9uI3409UaIhIk6TgaCAPzVrFL95YziUje/GrqxQaItJ0KTgawB/fWckv31zBpSN78avxI0loYbEuSUTkqCk4ouwPb6/kV/9aweWjevOLq05UaIhIk6fgiKLfvbWS37y1gstH9+YXVyo0RKR5UHBEyW/+tYLfvb2SKzPT+PkVIxQaItJsKDjqmbvzm7dW8vu3VzI+K40HLx9BC4WGiDQjCo565O78+l8r+MM7q7g6qw8/u3y4QkNEmh0FRz1xd3715gr+OGsVE07qw/9eptAQkeZJwVEP3J1fvLGcP727molj+vLApScoNESk2Yrqt9DMbJyZLTezVWZ2Tw3L083sbTP7xMzeNbO0cP7ZZrYg4qfYzC4Nlz1qZgvDdZ43s/bRfA1H4u78/J9BaFx7skJDRJq/qAWHmSUADwHnA0OBiWY2tFqzXwJPuPsI4H7gZwDuPsvdR7r7SOAcYC/wZrjO3e5+YrjOeuCOaL2GI3F3Hnx9GX9+bzXXj03npwoNEYkD0exxjAFWufsady8BZgKXVGszFHgnnJ5Vw3KAK4HX3X0vgLvvAjAzA9oAHoXaj8jd+d/XlvKX99dwwynp3H/JMIKSRESat2gGR29gQ8Tj3HBepIXA5eH0ZUCymXWu1mYCMCNyhpk9BmwBBgN/qK+Ca8vdeeAfS3n4358x6dQMfnKxQkNE4kesr7T3beBMM5sPnAlsBMorF5pZT2A48EbkSu5+E9ALWApcXdOGzWyKmWWbWXZeXl69Fezu/M/fl/LIB59x02kZ/OiioQoNEYkr0QyOjUCfiMdp4bwq7r7J3S9391HAfeG8gogm44GX3L20+sbdvZxg+OuKmp7c3ae6e5a7Z3Xt2vWYXkjENvnJ35Yw7T+fcfNp/fjhVxQaIhJ/ohkcc4GBZtbPzFoRDDm9GtnAzLqYWWUN9wLTqm1jIhHDVBY4rnIauBhYFqX6D1IZGo9/uJZbT+/HD74yRKEhInEpasHh7mUEn3h6g2BI6Vl3/9TM7jezi8NmZwHLzWwF0B14oHJ9M8sg6LG8F7FZA6ab2SJgEdCT4NNYUeXu/OjVT3n8w7VMOaM/912o0BCR+GXuMflQUoPKysry7Ozso1q3osL54auLeXL2er56Rn/uOX+wQkNE4oKZ5bh7VvX5+ub4YVRUOD94ZTFPzVnP184cwPfGDVJoiEjcU3Acgrvz/VcW8/Sc9dx21gC+82WFhogIKDgOycwY0LU9d5x9HN/60vEKDRGRkILjMG45vV+sSxARaXRi/QVAERFpYhQcIiJSJwoOERGpEwWHiIjUiYJDRETqRMEhIiJ1ouAQEZE6UXCIiEidxMVFDs0sD1h3lKt3AbbXYzn1RXXVjeqqG9VVN821rnR3/9wNjeIiOI6FmWXXdHXIWFNddaO66kZ11U281aWhKhERqRMFh4iI1ImC48imxrqAQ1BddaO66kZ11U1c1aVzHCIiUifqcYiISJ0oOEREpE7iOjjMbJyZLTezVWZ2Tw3LW5vZM+HyOWaWEbHs3nD+cjP7cgPX9U0zW2Jmn5jZ22aWHrGs3MwWhD+vNnBdk8wsL+L5b41YdqOZrQx/bmzgun4TUdMKMyuIWBaV/WVm08xsm5ktPsRyM7PfhzV/YmajI5ZFc18dqa5rw3oWmdmHZnZixLK14fwFZpbdwHWdZWaFEb+rH0YsO+zvP8p1fSeipsXh31NquCya+6uPmc0KjwOfmtk3amgTvb8xd4/LHyABWA30B1oBC4Gh1drcBvw5nJ4APBNODw3btwb6hdtJaMC6zgbahtNfr6wrfFwUw/01CfhjDeumAmvCf1PC6ZSGqqta+/8CpjXA/joDGA0sPsTyC4DXAQPGAnOiva9qWdeplc8HnF9ZV/h4LdAlRvvrLODvx/r7r++6qrW9CHingfZXT2B0OJ0MrKjh/2PU/sbiuccxBljl7mvcvQSYCVxSrc0lwPRw+nngXDOzcP5Md9/v7p8Bq8LtNUhd7j7L3feGD2cDafX03MdU12F8GfiXu+9093zgX8C4GNU1EZhRT899SO7+PrDzME0uAZ7wwGygk5n1JLr76oh1ufuH4fNCw/1t1WZ/Hcqx/F3Wd10N8rcF4O6b3X1eOL0bWAr0rtYsan9j8RwcvYENEY9z+fyOr2rj7mVAIdC5lutGs65ItxC8q6iUZGbZZjbbzC6tp5rqUtcVYbf4eTPrU8d1o1kX4ZBeP+CdiNnR2l9Hcqi6o7mv6qr635YDb5pZjplNiUE9p5jZQjN73cyGhfMaxf4ys7YEB98XImY3yP6yYAh9FDCn2qKo/Y21rHOV0miY2XVAFnBmxOx0d99oZv2Bd8xskbuvbqCS/gbMcPf9ZvZVgt7aOQ303LUxAXje3csj5sVyfzVaZnY2QXCcHjH79HBfdQP+ZWbLwnfkDWEewe+qyMwuAF4GBjbQc9fGRcB/3D2ydxL1/WVm7QnC6i5331Wf2z6ceO5xbAT6RDxOC+fV2MbMWgIdgR21XDeadWFm5wH3ARe7+/7K+e6+Mfx3DfAuwTuRBqnL3XdE1PIIkFnbdaNZV4QJVBtKiOL+OpJD1R3NfVUrZjaC4Pd3ibvvqJwfsa+2AS9Rf8OzR+Tuu9y9KJx+DUg0sy40gv0VOtzfVlT2l5klEoTGU+7+Yg1Novc3Fo0TN03hh6C3tYZg6KLypNqwam1u5+CT48+G08M4+OT4Gurv5Hht6hpFcEJwYLX5KUDrcLoLsJJ6OlFYy7p6RkxfBsz2AyfjPgvrSwmnUxuqrrDdYIKTldYQ+yvcZgaHPtl7IQefuPw42vuqlnX1JThnd2q1+e2A5IjpD4FxDVhXj8rfHcEBeH2472r1+49WXeHyjgTnQdo11P4KX/sTwG8P0yZqf2P1tnOb4g/Bpw5WEByE7wvn3U/wLh4gCXgu/I/0MdA/Yt37wvWWA+c3cF1vAVuBBeHPq+H8U4FF4X+eRcAtDVzXz4BPw+efBQyOWPfmcD+uAm5qyLrCxz8GHqy2XtT2F8G7z81AKcEY8i3A14CvhcsNeCiseRGQ1UD76kh1PQLkR/xtZYfz+4f7aWH4O76vgeu6I+JvazYRwVbT77+h6grbTCL4sEzketHeX6cTnEP5JOJ3dUFD/Y3pkiMiIlIn8XyOQ0REjoKCQ0RE6kTBISIidaLgEBGROlFwiIhInSg4RBq58Mqwf491HSKVFBwiIlInCg6RemJm15nZx+H9F/5iZglmVmTB/UA+teDeKV3DtiPDCyt+YmYvmVlKOP84M3srvJjfPDMbEG6+fXjhyGVm9lR4lWaRmFBwiNQDMxsCXA2c5u4jgXLgWoLLTWS7+zDgPeBH4SpPAN9z9xEE3+qtnP8U8JC7n0jwzfbN4fxRwF0E94LpD5wW5Zckcki6Oq5I/TiX4KKOc8POQBtgG1ABPBO2eRJ40cw6Ap3c/b1w/nTgOTNLBnq7+0sA7l4MEG7vY3fPDR8vILh+0gdRf1UiNVBwiNQPA6a7+70HzTT7QbV2R3uNn/0R0+Xo/67EkIaqROrH28CV4b0XMLPU8MZRLYArwzbXAB+4eyGQb2ZfCOdfD7znwZ3ccitvKGXBPe/bNuSLEKkNvWsRqQfuvsTMvk9wx7cWBFdTvR3YA4wJl20jOA8CcCPw5zAY1gA3hfOvB/5iZveH27iqAV+GSK3o6rgiUWRmRe7ePtZ1iNQnDVWJiEidqMchIiJ1oh6HiIjUiYJDRETqRMEhIiJ1ouAQEZE6UXCIiEid/H+1EtT8ZSfnRQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(np.sqrt(accs))\n",
    "plt.plot(np.sqrt(accs_eval))\n",
    "plt.title('model Accuracy')\n",
    "plt.ylabel('Acc')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'val'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [],
   "source": [
    "preds = []\n",
    "targets = []\n",
    "for texts, ys in valid_generator:\n",
    "    preds += model(texts.to(device)).squeeze().detach().to('cpu').numpy().tolist()\n",
    "    targets += ys.numpy().tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Можно вытащить предикты и посчитать все через sklearn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import classification_report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       0.97      0.99      0.98     61330\n",
      "         1.0       0.67      0.47      0.55      3977\n",
      "\n",
      "    accuracy                           0.95     65307\n",
      "   macro avg       0.82      0.73      0.76     65307\n",
      "weighted avg       0.95      0.95      0.95     65307\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# можно подбирать порог для отнесения к токсичному классу\n",
    "print(classification_report(targets, (np.array(preds) > 0.5).astype(int)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Домашка\n",
    "\n",
    "1) На одной из задач сравните, что лучше: а) конкатенировать эмбединги в один большой эмбединг, б) усреднять эмбединги, в) складывать эмбединги. Зафиксируйте разбиение на train_test  через random_seed, чтобы обучаться и оцениваться на одних и тех же данных. Параметры нейронок не нужно делать одинаковыми. Просто попробуйте добиться максимума от каждого из подходов. \n",
    "\n",
    "2) В одной из задач используйте сразу две предобученные эмбединг модели (можно фастекст и ворд2век, а можно просто 2 фастекста с разными параметрами) в одной нейронке (имеется в виду предобученные до обучения нейронки, т.е вы можете сами обучить фастекст/ворд2век и не искать готовые модели в интернете). Попробуйте усреднять эмбединги и конкатенировать (общие эмбединги потом в один вектор собирайте любым способом - например, тем который оказался лучше в первой задаче).\n",
    "Подсказка: сделайте два Embedding слоя и к их выходам примените torch.cat(dim=2) или torch.mean(dim=2)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Бонусный балл можно получить за добавление к пайплайну subword токенизации (через библиотеку tokenizers например)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Лучше делать задание на второй задаче, так как в Авито в любом случае разница будет в несколько десятых процентов."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}