notebooks/word_embeddings/Word_embeddings.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4d4afc86",
   "metadata": {},
   "source": [
    "## Векторные представления слов"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9aecffe8",
   "metadata": {},
   "source": [
    "Векторные представления слов (или embeddings) - это наборы чисел, которые должны как-то отражать смысл/значение слов. Формально значение определить очень сложно, поэтому в NLP используются очень упрощенные и утилитарные подходы к пониманию значения. Например, можно считать, что векторные представления хорошо передают смысл слов, если близость (по какой-то выбранной метрике) между похожими словами большая, а между противоположными - большая. Другой пример - с помощью векторных представлений получается хорошо решить задачу, которая требует понимания смысла (т.е. почти любую NLP задачу), значит эти векторные представления хорошо кодируют смысл. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "113ce347",
   "metadata": {},
   "source": [
    "Есть еще такие тонкости как многозначность и зависимость смысла от контекста, но это уже сильно сложнее и мы оставим их на следующие семинары, а пока просто полностью проигнорируем. К тому же, как и с мешком слов, часто упрощенного подхода будет достаточно."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad148a50",
   "metadata": {},
   "source": [
    "Единственная окололингвистическая теоретическая часть, о которой стоит сказать это дистрибутивная гипотеза. Она состоит в том, что лингвистические единицы, встречающиеся в схожих контекстах, имеют близкие значения. Обычно в научных статьях цитируют John Firth (Ферс) - '*You shall know a word by the company it keeps*' (Firth, J. R. 1957:11) \n",
    "Также в \"Филосовских исследованиях\" Витгенштейна есть такое -  *“For a large class of cases—though not for all—in which we employ the word ‘meaning’ it can be defined thus: **the meaning of a word is its use in the language.**”* (1951)\n",
    "\n",
    "Саму область иногда называют дистрибутивная семантика."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41f39810",
   "metadata": {},
   "source": [
    "Давайте посмотрим, как можно строить векторные представления слов."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "22ec086b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install matplotlib pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "952c8c5a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from string import punctuation\n",
    "from sklearn.model_selection import train_test_split\n",
    "from collections import Counter\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.decomposition import TruncatedSVD\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.metrics.pairwise import cosine_distances\n",
    "\n",
    "from IPython.display import Image\n",
    "from IPython.core.display import HTML \n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee85cb52",
   "metadata": {},
   "source": [
    "### Матричные разложения "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "069eaa65",
   "metadata": {},
   "source": [
    "Для начала вспомним матричные разложения. В двух методах, которые мы использовали для тематического моделирования одна из матриц была размерности (количество слов в словаре, количество тем), т.е. каждую строчку в этой матрице можно рассматривать как векторное представление слова! Давайте разложим матрицу документы-слова, собранную по википедии и посмотрим на близость векторов слов."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "f682013c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"https://miro.medium.com/max/1362/1*AQ3ohYYQtJLIGWOc5UBrDQ.webp\" width=\"1000\" height=\"900\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# на картинке TruncatedSVD\n",
    "Image(url=\"https://miro.medium.com/max/1362/1*AQ3ohYYQtJLIGWOc5UBrDQ.webp\",\n",
    "     width=1000, height=900)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b036f980",
   "metadata": {},
   "outputs": [],
   "source": [
    "# в нашем корпусе 20к текстов\n",
    "wiki = open('wiki_data.txt').read().split('\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "65bb2fd9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20003"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(wiki)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "4b97780b",
   "metadata": {},
   "outputs": [],
   "source": [
    "cv = CountVectorizer(max_features=10000)\n",
    "svd = TruncatedSVD(200)\n",
    "\n",
    "X = cv.fit_transform(wiki)\n",
    "X_svd = svd.fit_transform(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8779634",
   "metadata": {},
   "source": [
    "Получившаяся матрица X нас не интересует. Нам нужно вытащить матрицу U, она лежит в svd.components_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "d86a188d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# изначально U размерности (темы, слова) и для удобства ее нужно перевернуть - транспонировать\n",
    "embeddings = svd.components_.T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "59ecdf45",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10000, 200)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embeddings.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bda1c46b",
   "metadata": {},
   "source": [
    "Теперь вытаскиваем соответствия слов индексам и наоборот"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "2f90e0e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "id2word = cv.get_feature_names_out()\n",
    "word2id = {word:i for i,word in enumerate(id2word)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "f4714f62",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7070"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word2id['птица']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4801087",
   "metadata": {},
   "source": [
    "Напишем простую функцию, которая для заданного слова найдет ближайшие ему слова в матрице"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "3529ebfa",
   "metadata": {},
   "outputs": [],
   "source": [
    "def most_similar(word, embeddings):\n",
    "    similar = [id2word[i] for i in \n",
    "               cosine_distances(embeddings[word2id[word]].reshape(1, -1), embeddings).argsort()[0][:10]]\n",
    "    return similar"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ebab8e9c",
   "metadata": {},
   "source": [
    "Испытаем ее на нескольких рандомных словах"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "2a1adc3c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['птица',\n",
       " 'самцы',\n",
       " 'самцов',\n",
       " 'самки',\n",
       " 'птицы',\n",
       " 'самок',\n",
       " 'самка',\n",
       " 'обитает',\n",
       " 'пятнами',\n",
       " 'хвост']"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "most_similar('птица', embeddings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "8b6cf967",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['церковь',\n",
       " 'чудотворца',\n",
       " 'каменная',\n",
       " 'храм',\n",
       " 'прихожан',\n",
       " 'орла',\n",
       " 'храма',\n",
       " 'деревянная',\n",
       " 'богослужения',\n",
       " 'троицы']"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "most_similar('церковь', embeddings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "3c42970e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['москва',\n",
       " 'выставка',\n",
       " 'арт',\n",
       " 'галерея',\n",
       " 'центральный',\n",
       " 'выставке',\n",
       " 'выставках',\n",
       " 'галерее',\n",
       " 'художника',\n",
       " 'постоянная']"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "most_similar('москва', embeddings)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "346ce174",
   "metadata": {},
   "source": [
    "Кажется, что получается неплохо. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08395b4f",
   "metadata": {},
   "source": [
    "Но матричные разложения для этой задачи обычно не применяют. Основной метод для создания векторых представлений слов - это word2vec и его улучшение fastext. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c9f4420",
   "metadata": {},
   "source": [
    "## Word2Vec"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37de9786",
   "metadata": {},
   "source": [
    "Word2Vec был создан в 2013 году в Google командой под руководством Tomas Mikolov. \n",
    "\n",
    "Он был описан в двух статьях: \n",
    "1) https://arxiv.org/abs/1301.3781 (https://papers.nips.cc/paper_files/paper/2013/hash/9aa42b31882ec039965f3c4923ce901b-Abstract.html)   \n",
    "2) https://aclanthology.org/N13-1090/\n",
    "\n",
    "С word2vec началась transfer learning и self supervised революция в NLP. Вместо обучения моделей с нуля под каждую конкретную задачу люди стали предобучать общие модели на сыром тексте и затем дообучать их под нужные задачи, используя размеченные данные.  У такого подхода сразу несколько преимуществ: 1) он просто лучше работает, так как помимо самих размеченных данных, модель косвенно использует большое количество знаний из предобучающего корпуса; 2) размеченных данных нужно сильно меньше (для достижения качества сопостовимого с моделью обученной с нуля), так как предобученной модели не нужно выучивать базовое понимание языка, она сразу может фокусироваться на специфичных для задачи паттернах; 3) одну предобученную модель можно переиспользовать в разных задачах (часто такие модели создаются крупными компаниями/лабами с огромными вычислительными ресурсами, а затем выкладываются в открытый доступ) . \n",
    "\n",
    "Также важным достоинством самого word2vec была его эффективность (и по скорости и по потреблению ресурсов), что с одной строны позволило большим компаниям обучить модели на огромных корпусах, а с другой дало возможность отдельным исследователям строить качественные специфичные модели (например, на корпусе научных статей по математике), используя небольшие сервера или вообще личные компьютеры. \n",
    "\n",
    "Сейчас word2vec уже конечно устарел (но он может быть хорошим бейзлайном), транформерные модели лучше. Но через word2vec удобно начать переходить к нейронным сетям (так как это на самом деле однойслойная нейронка). Поэтому мы попробуем воспроизвести базовые составляющие алгоритма на tensorflow, параллельно разобрав и основы deep learning. В Word2Vec очень много деталей и оптимизаций, которые мы опустим, при решении конкретной задачи лучше использовать готовую реализацию (ниже мы это сделаем через gensim)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0133ad1",
   "metadata": {},
   "source": [
    "### word2vec в tensorflow"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae6d6286",
   "metadata": {},
   "source": [
    "Для начала нам нужно построить словарь."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1248f8ae",
   "metadata": {},
   "source": [
    "Предобработка нам не так важна. Главное - получить из текстов токены."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "17f8648d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "def preprocess(text):\n",
    "    tokens = re.sub('#+', ' ', text.lower()).split()\n",
    "    tokens = [token.strip(punctuation) for token in tokens]\n",
    "    tokens = [token for token in tokens if token]\n",
    "    return tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9964e652",
   "metadata": {},
   "source": [
    "Лучше сразу посчитать количество упоминаний, чтобы отсеять самые редкие."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "6a7314a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab = Counter()\n",
    "\n",
    "for text in wiki:\n",
    "    vocab.update(preprocess(text))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2289def",
   "metadata": {},
   "source": [
    "443к слов - это многовато"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "f175e7af",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "443922"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ba6655d",
   "metadata": {},
   "source": [
    "Возьмем только те, что встретились больше 30 раз."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "334cdb3a",
   "metadata": {},
   "outputs": [],
   "source": [
    "filtered_vocab = set()\n",
    "\n",
    "for word in vocab:\n",
    "    if vocab[word] > 30:\n",
    "        filtered_vocab.add(word)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "78bbe728",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "17016"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(filtered_vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc18a6ee",
   "metadata": {},
   "source": [
    "~17к - уже нормальная цифра"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec88f467",
   "metadata": {},
   "source": [
    "Теперь нам нужно заменить в каждом тексте слова на числа (индексы в словаре). "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d9ae638",
   "metadata": {},
   "source": [
    "Создадим для этого специальный словарь с индексами. В этот словарь нам нужно положить дополнительный токен - PAD (чуть ниже станет понятнее)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "1ad31ef0",
   "metadata": {},
   "outputs": [],
   "source": [
    "word2id = {'PAD':0}\n",
    "\n",
    "for word in filtered_vocab:\n",
    "    word2id[word] = len(word2id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "3b797373",
   "metadata": {},
   "outputs": [],
   "source": [
    "id2word = {i:word for word, i in word2id.items()}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84373999",
   "metadata": {},
   "source": [
    "Заменяем слова на индексы."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "2f115901",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "sentences = []\n",
    "\n",
    "for text in wiki:\n",
    "    tokens = preprocess(text)\n",
    "    if not tokens:\n",
    "        continue\n",
    "    ids = [word2id[token] for token in tokens if token in word2id]\n",
    "    sentences.append(ids)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "id": "9096af09",
   "metadata": {},
   "outputs": [],
   "source": [
    "# sentences[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c68ae26a",
   "metadata": {},
   "source": [
    "Далее нужно преобразовать токенизированные тексты в формат для обучения."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "710994eb",
   "metadata": {},
   "source": [
    "Word2vec обучается решать синтетическую задачу, а точнее две зеркальные задачи - предсказывать слово по его окружению и предсказывать окружение по слову. \n",
    "\n",
    "У такой задачи есть какое-то теоретическое обоснование (см. выше про дистрибутивную гипотезу), но оно скорее для красоты - просто это работает. В более поздних моделях используются немного другие синтетические задачи - заполнение пропусков или просто генерация следующего слова.\n",
    "\n",
    "Обучающие данные для таких задач можно просто гененировать из любого текста практически бесконечно, поэтому такой подход называется self-supervised."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19ee0baf",
   "metadata": {},
   "source": [
    "Давайте разберем 2 алгоритма word2vec: skip gram и CBOW (continuos bag of words). В обоих алгоритмах происходит итерирование по кусочкам текстов фиксированной длинны (окнам). В каждом окне выбирается центральное целевое слово (target), а слова слева и справа от него считаются контекстом (context). \n",
    "\n",
    "**В Skip-gram по слову предсказывается окружение, а в CBOW по окружению предсказывается целевое слово.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f254a38",
   "metadata": {},
   "source": [
    "### Skip Gram"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "db2a735c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"https://leimao.github.io/images/article/2019-08-23-Word2Vec-Classic/word2vec.png\" width=\"600\" height=\"600\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url=\"https://leimao.github.io/images/article/2019-08-23-Word2Vec-Classic/word2vec.png\",\n",
    "     width=600, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85d45db6",
   "metadata": {},
   "source": [
    "  Давайте соберем датасет для skip gram. Датасетом будут просто пары - (целевое слово, контекстное слово). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "56e6e7f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# skip gram\n",
    "X = []\n",
    "y = []\n",
    "\n",
    "window = 5\n",
    "for sent in sentences[:1000]:\n",
    "    for i in range(len(sent)-1):\n",
    "        word = sent[i] # target\n",
    "        context = sent[max(0, i-window):i] + sent[i+1:i+window]  # context (слова до и после целевого)\n",
    "        # в начале и в конце окно будет неполным, но это не страшно\n",
    "\n",
    "        for context_word in context:\n",
    "            X.append(word)\n",
    "            y.append(context_word)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "ae28a5a3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(7069, 8518)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X[1], y[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "8030c45a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1765036, 1765036)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(X), len(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "840ec4b0",
   "metadata": {},
   "source": [
    "Как и в любой задаче машинного обучения нужно тестироваться на отложенной выборке"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "d71be564",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_valid, y_train, y_valid = train_test_split(X, y, test_size=0.05)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c201693",
   "metadata": {},
   "source": [
    "### Создаем нейронную сеть"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7c0b1ef",
   "metadata": {},
   "source": [
    "Для введения в нейронные сети посмотрите:\n",
    "1) серию видео на канале 3blue1brown - https://www.youtube.com/watch?v=aircAruvnKk  \n",
    "2) лекцию курса dlcoure.ai - https://www.youtube.com/watch?v=kWTC1NvL894"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62b91edc",
   "metadata": {},
   "source": [
    "Word2Vec можно рассматривать как очень маленькую нейронную сеть - тут всего 1 слой. Сама задача сводится к классификации. На вход мы получаем индекс слова, преобразуем его в векторное представление, \n",
    "\n",
    "В tf.keras два способа построения моделей: Sequential и Functional. В первом мы как будто создаем список и добавляем в него слои, а во втором вручную применяем каждый слой к результату предыдущего слоя (как функцию в питоне).\n",
    "\n",
    "Соберем нейронную сеть первым способом."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "4921a30b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# инициализируем модель\n",
    "# это что-то типа model = [] \n",
    "model = tf.keras.Sequential() \n",
    "\n",
    "# первым добавляем эмбединг слой\n",
    "# он каждому слову (его индексу) сопоставит вектор заданой размерности\n",
    "# также нам нужно передать параметры нашей выборки - размер словаря (input_dim) и длину текста (input_length)\n",
    "# в нашем случае мы подаем только 1 слово поэтому длина - 1\n",
    "model.add(tf.keras.layers.Embedding(input_dim=len(word2id), \n",
    "                                    input_length=1, \n",
    "                                    output_dim=100)) # также указываем желаемый размер эмбединга\n",
    "\n",
    "# на выходе из Embedding мы получим массив размерности (1, 100)\n",
    "# единичная размерность нам не нужна и ее можно схлопнуть\n",
    "# это можно сделать специальным слоем которые называется Flatten\n",
    "# чтобы было понятнее, вот пример на списках\n",
    "# флаттен делает что-то такое\n",
    "# [1, ] - 1\n",
    "# [[1], [2], [3]] -> [1, 2, 3]\n",
    "model.add(tf.keras.layers.Flatten()) \n",
    "\n",
    "# И теперь 1 реальный слой, который называется полносвязным\n",
    "# Полносвязный слой это матрица, применение полносвязного слоя - это умножение на эту матрицу\n",
    "# На выходе получится вектор или матрица (по правилу умножения матриц, размерность в итоге будет - (n, m) * (m, k) = n*k)\n",
    "# В нашем случае n - это 1, так как у нас 1 вектор 1 целевого слова\n",
    "# m - это размерность нашего вектора, мы указали 100\n",
    "# Полносвязный слоя сам подстроится под m, но вот k - нужно указать вручную\n",
    "# в нашем случае мы хотим получить по 1 числу на 1 слово в словаре, т.е. k = размер словаря\n",
    "# это число будет соответствовать вероятности предсказания этого слова\n",
    "\n",
    "# умножение вектора на матрицу можно интепретировать как нахождение близости\n",
    "# с каждым элементом матрицы; т.е. мы берем эмбединг и находим близость с эмбедингами всех других слов в словаре\n",
    "\n",
    "# По умолчанию полносвязный слой не выдает вероятности, чтобы их получить нужно использовать softmax\n",
    "# Софтмакс - это одна из функций активации\n",
    "# Функции аквтивации преобразуют числа каким-то нелинейным способом, которого не добьешься обычным умножением на матрицу\n",
    "# сочетание линейных операций с матрицами и векторами и нелинейных функций - это то что делает нейронные сети такими мощными\n",
    "model.add(tf.keras.layers.Dense(len(word2id), \n",
    "                                activation='softmax'))\n",
    "\n",
    "# компилируем модель\n",
    "# выбираем лосс - функция, по которой модель будет оценивать качество и обновлять веса\n",
    "# для классификации стандартно используется categorical_crossentropy\n",
    "# У нас стоит sparse_categorical_crossentropy потому что мы подаем индексы, а по умолчанию нужно подавать \n",
    "# one-hot векторы или вероятности\n",
    "\n",
    "# выбираем оптимизатор - он влияет на то, как будет обновляться модель \n",
    "# (подробнее вот тут - \n",
    "# https://medium.com/datadriveninvestor/overview-of-different-optimizers-for-neural-networks-e0ed119440c3 ).\n",
    "# По умолчанию можно ставить adam и менять только learning_rate\n",
    "\n",
    "model.compile(, optimizer=tf.keras.optimizers.Adam()\n",
    "              loss='sparse_categorical_crossentropy',\n",
    "              \n",
    "              )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a41e6bf7",
   "metadata": {},
   "source": [
    "Можно посмотреть на размерности, которые будут получаться после каждого слоя"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "6b57cd8c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " embedding (Embedding)       (None, 1, 100)            1701700   \n",
      "                                                                 \n",
      " flatten (Flatten)           (None, 100)               0         \n",
      "                                                                 \n",
      " dense (Dense)               (None, 17017)             1718717   \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 3,420,417\n",
      "Trainable params: 3,420,417\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "print(model.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "541ea7f9",
   "metadata": {},
   "source": [
    "Осталось только обучить"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "6410a849",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/3\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-01-11 14:10:10.262168: W tensorflow/core/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1677/1677 [==============================] - 84s 50ms/step - loss: 7.9027 - val_loss: 7.5715\n",
      "Epoch 2/3\n",
      "1677/1677 [==============================] - 86s 51ms/step - loss: 7.4750 - val_loss: 7.4310\n",
      "Epoch 3/3\n",
      "1677/1677 [==============================] - 85s 51ms/step - loss: 7.3195 - val_loss: 7.3206\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x2f770e970>"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# нейронные сети обучаются не на всех данных сразу\n",
    "# а по кусочкам (батчам)\n",
    "# размер батча зависит от размера нейронной сети и ресурсов\n",
    "# обычно это значение от 4 до нескольких тысяч\n",
    "model.fit(X_train, y_train, \n",
    "          validation_data=(X_valid, y_valid),\n",
    "          batch_size=1000, \n",
    "         epochs=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "730a8313",
   "metadata": {},
   "source": [
    "Посмотрим на графики функции потерь. Она должна уменьшаться и на трейне и на тесте"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "27b3bb36",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['loss', 'val_loss'])\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAHHCAYAAABDUnkqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy89olMNAAAACXBIWXMAAA9hAAAPYQGoP6dpAABkrElEQVR4nO3deVhTZ9oG8DsJhH0R2TcRUNwAN7AurbZqcanV1rpgW0W0Tlv7TW3Hbk47Vp2pzkz3qdpOR7Fq3feqtRWrta7gviMoEkAUBdkhgeR8f0QTIxABgZOE+3dduWrOeZM8h6Pk7nuWRyIIggAiIiIiCyEVuwAiIiKixsRwQ0RERBaF4YaIiIgsCsMNERERWRSGGyIiIrIoDDdERERkURhuiIiIyKIw3BAREZFFYbghIiIii8JwQ0Qm79q1a5BIJFi2bFm9X7tv3z5IJBLs27fP6Lhly5ZBIpHg2rVrDaqRiEwHww0RERFZFIYbIiIisigMN0RERGRRGG6I6KE+/vhjSCQSXL58GS+99BJcXFzg4eGBjz76CIIgIDMzEyNHjoSzszO8vb3x2WefVXuP3NxcTJkyBV5eXrC1tUVkZCR++OGHauMKCgoQFxcHFxcXuLq6YtKkSSgoKKixrkuXLuGFF16Am5sbbG1t0bNnT2zbtq1Rt33RokXo3LkzbGxs4Ovri+nTp1erJzU1FaNHj4a3tzdsbW3h7++P8ePHo7CwUDdm9+7d6NevH1xdXeHo6IiwsDDMmjWrUWslIi0rsQsgIvMxbtw4dOzYEQsWLMCOHTvw97//HW5ubvjuu+/w1FNP4Z///Cd+/PFHzJw5E1FRUXjiiScAAOXl5RgwYADS0tLwxhtvoG3btli/fj3i4uJQUFCAN998EwAgCAJGjhyJAwcO4NVXX0XHjh2xefNmTJo0qVot58+fR9++feHn54f3338fDg4OWLduHUaNGoWNGzfiueeee+Tt/fjjjzFnzhwMGjQIr732GlJSUrB48WIkJyfj4MGDsLa2hkqlQkxMDJRKJf7v//4P3t7eyM7Oxvbt21FQUAAXFxecP38ezzzzDCIiIjB37lzY2NggLS0NBw8efOQaiagGAhHRQ8yePVsAIEybNk23rKqqSvD39xckEomwYMEC3fI7d+4IdnZ2wqRJk3TLvvzySwGAsHLlSt0ylUol9O7dW3B0dBSKiooEQRCELVu2CACEf/3rXwaf8/jjjwsAhISEBN3ygQMHCuHh4UJFRYVumUajEfr06SO0a9dOt2zv3r0CAGHv3r1GtzEhIUEAIKSnpwuCIAi5ubmCXC4Xnn76aUGtVuvGffPNNwIAYenSpYIgCMLJkycFAML69etrfe8vvvhCACDcunXLaA1E1Dh4WIqI6mzq1Km6P8tkMvTs2ROCIGDKlCm65a6urggLC8PVq1d1y3bu3Alvb2/ExsbqlllbW+PPf/4zSkpK8Pvvv+vGWVlZ4bXXXjP4nP/7v/8zqCM/Px+//fYbxo4di+LiYty+fRu3b99GXl4eYmJikJqaiuzs7Efa1sTERKhUKsyYMQNSqf5X5SuvvAJnZ2fs2LEDAODi4gIA+OWXX1BWVlbje7m6ugIAtm7dCo1G80h1EdHDMdwQUZ0FBgYaPHdxcYGtrS3c3d2rLb9z547ueUZGBtq1a2cQEgCgY8eOuvX3/uvj4wNHR0eDcWFhYQbP09LSIAgCPvroI3h4eBg8Zs+eDUB7js+juFfTg58tl8sRHBysW9+2bVu8/fbb+N///gd3d3fExMRg4cKFBufbjBs3Dn379sXUqVPh5eWF8ePHY926dQw6RE2E59wQUZ3JZLI6LQO05880lXuhYObMmYiJialxTGhoaJN9/oM+++wzxMXFYevWrfj111/x5z//GfPnz8eRI0fg7+8POzs77N+/H3v37sWOHTuwa9curF27Fk899RR+/fXXWn+GRNQwnLkhoibXpk0bpKamVpupuHTpkm79vf/m5OSgpKTEYFxKSorB8+DgYADaQ1uDBg2q8eHk5PTINdf02SqVCunp6br194SHh+PDDz/E/v378ccffyA7Oxvffvutbr1UKsXAgQPx+eef48KFC/jHP/6B3377DXv37n2kOomoOoYbImpyw4YNw40bN7B27VrdsqqqKvznP/+Bo6Mj+vfvrxtXVVWFxYsX68ap1Wr85z//MXg/T09PDBgwAN999x1ycnKqfd6tW7ceueZBgwZBLpfj66+/NpiFWrJkCQoLCzF8+HAAQFFREaqqqgxeGx4eDqlUCqVSCUB7jtCDunbtCgC6MUTUeHhYioia3LRp0/Ddd98hLi4Ox48fR1BQEDZs2ICDBw/iyy+/1M2yjBgxAn379sX777+Pa9euoVOnTti0aZPB+Sv3LFy4EP369UN4eDheeeUVBAcH4+bNmzh8+DCysrJw+vTpR6rZw8MDH3zwAebMmYMhQ4bg2WefRUpKChYtWoSoqCi89NJLAIDffvsNb7zxBsaMGYP27dujqqoKK1asgEwmw+jRowEAc+fOxf79+zF8+HC0adMGubm5WLRoEfz9/dGvX79HqpOIqmO4IaImZ2dnh3379uH999/HDz/8gKKiIoSFhSEhIQFxcXG6cVKpFNu2bcOMGTOwcuVKSCQSPPvss/jss8/QrVs3g/fs1KkTjh07hjlz5mDZsmXIy8uDp6cnunXrhr/97W+NUvfHH38MDw8PfPPNN3jrrbfg5uaGadOm4ZNPPoG1tTUAIDIyEjExMfjpp5+QnZ0Ne3t7REZG4ueff8Zjjz0GAHj22Wdx7do1LF26FLdv34a7uzv69++POXPm6K62IqLGIxGa8qw/IiIiombGc26IiIjIojDcEBERkUVhuCEiIiKLwnBDREREFoXhhoiIiCwKww0RERFZlBZ3nxuNRoPr16/DyckJEolE7HKIiIioDgRBQHFxMXx9fas14X1Qiws3169fR0BAgNhlEBERUQNkZmbC39/f6JgWF27u3eY9MzMTzs7OIldDREREdVFUVISAgIA6NcVtceHm3qEoZ2dnhhsiIiIzU5dTSnhCMREREVkUhhsiIiKyKAw3REREZFFa3Dk3daVWq1FZWSl2GWbJ2toaMplM7DKIiKiFYrh5gCAIuHHjBgoKCsQuxay5urrC29ub9xIiIqJmx3DzgHvBxtPTE/b29vxyridBEFBWVobc3FwAgI+Pj8gVERFRS8Nwcx+1Wq0LNq1btxa7HLNlZ2cHAMjNzYWnpycPURERUbPiCcX3uXeOjb29vciVmL97P0Oet0RERM2N4aYGPBT16PgzJCIisTDcEBERkUVhuKFqgoKC8OWXX4pdBhERUYOIGm6CgoIgkUiqPaZPn17j+MrKSsydOxchISGwtbVFZGQkdu3a1cxVm6YBAwZgxowZjfJeycnJmDZtWqO8FxERUXMT9Wqp5ORkqNVq3fNz585h8ODBGDNmTI3jP/zwQ6xcuRLff/89OnTogF9++QXPPfccDh06hG7dujVX2bUqVVZBbiWFtcz0JsQEQYBarYaV1cN3uYeHRzNURERE1DRE/Rb28PCAt7e37rF9+3aEhISgf//+NY5fsWIFZs2ahWHDhiE4OBivvfYahg0bhs8++6yZK6+uuKIS6bdLce12KdQaTbN+dlxcHH7//Xd89dVXutmvZcuWQSKR4Oeff0aPHj1gY2ODAwcO4MqVKxg5ciS8vLzg6OiIqKgoJCYmGrzfg4elJBIJ/ve//+G5556Dvb092rVrh23btjXrNhIREdWVyUwxqFQqrFy5EvHx8bVeaaNUKmFra2uwzM7ODgcOHKj1fZVKJYqKigwe9SEIAspUVQ99VKk1UFVpcKdMhUs3ilGirKzT64w9BEGoU41fffUVevfujVdeeQU5OTnIyclBQEAAAOD999/HggULcPHiRURERKCkpATDhg3Dnj17cPLkSQwZMgQjRoyAQqEw+hlz5szB2LFjcebMGQwbNgwvvvgi8vPz6/WzJCIiag4mcxO/LVu2oKCgAHFxcbWOiYmJweeff44nnngCISEh2LNnDzZt2mRwaOtB8+fPx5w5cxpcV3mlGp3+9kuDX/8oLsyNgb384bvIxcUFcrkc9vb28Pb2BgBcunQJADB37lwMHjxYN9bNzQ2RkZG65/PmzcPmzZuxbds2vPHGG7V+RlxcHGJjYwEAn3zyCb7++mskJSVhyJAhDdo2IiKipmIyMzdLlizB0KFD4evrW+uYr776Cu3atUOHDh0gl8vxxhtvYPLkyZBKa9+MDz74AIWFhbpHZmZmU5Rvsnr27GnwvKSkBDNnzkTHjh3h6uoKR0dHXLx48aEzNxEREbo/Ozg4wNnZWddigYiIyJSYxMxNRkYGEhMTsWnTJqPjPDw8sGXLFlRUVCAvLw++vr54//33ERwcXOtrbGxsYGNj0+Da7KxluDA3pl6vKSyrROadMm3NjrbwcmnY59tZP3rbAgcHB4PnM2fOxO7du/Hpp58iNDQUdnZ2eOGFF6BSqYy+j7W1tcFziUQCTTOfW0RERFQXJhFuEhIS4OnpieHDh9dpvK2tLfz8/FBZWYmNGzdi7NixTVabRCKp06Gh+9nLrSC3kiK7oBzFyko4qazg7tjwgFUXcrnc6OG5ew4ePIi4uDg899xzALQzOdeuXWvS2oiIiJqT6IelNBoNEhISMGnSpGqXKU+cOBEffPCB7vnRo0exadMmXL16FX/88QeGDBkCjUaDd999t7nLfqjWjjbwctae/Hy9oBwFZcZnRh5VUFAQjh49imvXruH27du1zqq0a9cOmzZtwqlTp3D69GlMmDCBMzBERGRRRA83iYmJUCgUiI+Pr7ZOoVAgJydH97yiogIffvghOnXqhOeeew5+fn44cOAAXF1dm7HiuvN0stHN2GTml6O4oumaSM6cORMymQydOnWCh4dHrefQfP7552jVqhX69OmDESNGICYmBt27d2+yuoiIiJqbRKjr9cYWoqioCC4uLigsLISzs7PBuoqKCqSnp6Nt27bVLjlvKEEQkJlfjoJyFaQSCYLdHWBvYxJHA5tUU/wsiYio5TL2/f0g0WduLJ1EIoG/mx2cbK2hEQSk55WiovLh58YQERFRwzDcNAOpRIJAN3vYy62g1ghIv10KVRXPcyEiImoKDDfNRCaVIKi1PWysZKhUa5B+uxRVagYcIiKixsZw04ysZFK0dXeAtUwKZZUa1/LKoNa0qFOeiIiImhzDTTOTW2kDjpVUgjJVFTLySqFpWed0ExERNSmGGxHYWssQ5O4AqUSCEmUVsvLL69wkk4iIiIxjuBGJvdwKbVrbQyKRoKBcheuFFQw4REREjYDhRkROttYIaGUHAMgrUSK3WClyRUREROaP4UZkrvZy+LpqA87NogrklTDgEBERPQqGGxPg7mgDz7t9qLKboQ9VTYKCgvDll182++cSERE1NoYbE+HlZIPWDnIAQOadpu1DRUREZMkYbkyERCKBr6sdXOysIQgCMvLKUKaqErssIiIis8NwY0IkEgkC3OzhaGMFjSDg2u0yKOvQh+q///0vfH19odEY3vF45MiRiI+Px5UrVzBy5Eh4eXnB0dERUVFRSExMbKrNICIiEhXDzcMIAqAqbbaHtLIMbZwAeyhRpVYj/XYpKh/Sh2rMmDHIy8vD3r17dcvy8/Oxa9cuvPjiiygpKcGwYcOwZ88enDx5EkOGDMGIESOgUCia+qdHRETU7KzELsDkVZYBn/g260fKAIQCuDzlMirUNkjPK0WwuwOsZDVn0VatWmHo0KFYtWoVBg4cCADYsGED3N3d8eSTT0IqlSIyMlI3ft68edi8eTO2bduGN954oxm2iIiIqPlw5saEBbW2g7VMiopKbR8qjZE+VC+++CI2btwIpVJ7KfmPP/6I8ePHQyqVoqSkBDNnzkTHjh3h6uoKR0dHXLx4kTM3RERkkThz8zDW9sCs66J8tNzaHm2tNbhyq0Tbhyq/DG1a20MqkVQbO2LECAiCgB07diAqKgp//PEHvvjiCwDAzJkzsXv3bnz66acIDQ2FnZ0dXnjhBahUzX/JORERUVNjuHkYiQSQO4j28bbWMgS1dkD67VIUV1Qi+045/FvZQfJAwLG1tcXzzz+PH3/8EWlpaQgLC0P37t0BAAcPHkRcXByee+45AEBJSQmuXbvW3JtCRETULBhuzICDjRUC3eyRkVeGO2UqyKQS+LjYVgs4L774Ip555hmcP38eL730km55u3btsGnTJowYMQISiQQfffRRtSuriIiILAXPuTETznbW8HfTtmm4XaLErRraNDz11FNwc3NDSkoKJkyYoFv++eefo1WrVujTpw9GjBiBmJgY3awOERGRpeHMjRlpZS9HlVpATmE5bhRWwEoqgZuDjW69VCrF9evVzw8KCgrCb7/9ZrBs+vTpBs95mIqIiCwFZ27MjIeTDTydtIEm+045CsvZpoGIiOh+DDdmyMvZFm4OcggAFPllKKlgmwYiIqJ7GG7MkEQigZ9BH6pSlLMPFREREQCGG7MlkUgQ0MoeDjZWUAsC0uvYh4qIiMjSMdzUQBBqvxOwKZFKJQhqbQ87axmqNBqk55WiUm0al3iby8+QiIgsD8PNfaytrQEAZWVlIldSdzKpFEHuDpBbSaGq0iD9dimqTOAeNvd+hvd+pkRERM2Fl4LfRyaTwdXVFbm5uQAAe3v7ajfKM1W+jjIo8lUoL1fhao4K/q3sIZU2f+2CIKCsrAy5ublwdXWFTCZr9hqIiKhlY7h5gLe3NwDoAo450ag1uF2shEYAcq9L4eYgFy2cubq66n6WREREzYnh5gESiQQ+Pj7w9PREZaX53UOmIqsA7204A5Vag6c7e+PdmLBmDzjW1tacsSEiItEw3NRCJpOZ5Rd0dKg3Zj0rwasrjyPhSDbkNjb4YGhHscsiIiJqNjyh2AIN7uSFBc+HAwC++/0q/rv/isgVERERNR+GGws1pmcAPhjaAQDwyc5LWH8sU+SKiIiImgfDjQX7U/8QTHsiGADw/qazSLxwU+SKiIiImh7DjYX7YGgHjO7uD7VGwPRVJ5B8LV/skoiIiJoUw42Fk0gk+OfocAzs4AlllQbxy5JxMadI7LKIiIiaDMNNC2Alk+KbCd0RFdQKxRVVmLg0CZn55nMXZiIiovpguGkh7OQy/G9iFDp4O+FWsRIvLzmKW8VKscsiIiJqdAw3LYiLvTV+iI+Gfys7XMsrQ1xCEoorzO9GhURERMYw3LQwXs62WDGlF9wd5Th/vQivLD+Gikq12GURERE1GoabFqituwOWTY6Go40VjlzNx5trTkKtEcQui4iIqFEw3LRQXfxc8N+JPSCXSfHL+Zv4cMtZCAIDDhERmT+GmxasT4g7vo7tCqkEWJ2Uic9+vSx2SURERI+M4aaFG9LFB/94TtuH6pu9aVh6IF3kioiIiB4Nww0hNjoQ78SEAQDmbr+ALSezRa6IiIio4RhuCADw+oAQTO4bBACYuf409qbkilsQERFRAzHcEABtm4aPhnfCqK6+qNIIeG3lcRzPuCN2WURERPUmargJCgqCRCKp9pg+fXqtr/nyyy8RFhYGOzs7BAQE4K233kJFRUUzVm25pFIJ/j0mEgPCPFBRqe1DdflmsdhlERER1Yuo4SY5ORk5OTm6x+7duwEAY8aMqXH8qlWr8P7772P27Nm4ePEilixZgrVr12LWrFnNWbZFs5ZJsejF7ugW6IrC8kpMXJKErDvsQ0VEROZD1HDj4eEBb29v3WP79u0ICQlB//79axx/6NAh9O3bFxMmTEBQUBCefvppxMbGIikpqZkrt2z2ciskxEWhnacjbhRVYOKSJOSVsA8VERGZB5M550alUmHlypWIj4+HRCKpcUyfPn1w/PhxXZi5evUqdu7ciWHDhtX6vkqlEkVFRQYPejhXezmWT4mGn6sdrt4uxeRlyShRVoldFhER0UOZTLjZsmULCgoKEBcXV+uYCRMmYO7cuejXrx+sra0REhKCAQMGGD0sNX/+fLi4uOgeAQEBTVC9ZfJxscPyKdFwc5DjTFYh/rTiGJRV7ENFRESmzWTCzZIlSzB06FD4+vrWOmbfvn345JNPsGjRIpw4cQKbNm3Cjh07MG/evFpf88EHH6CwsFD3yMzMbIryLVaIhyMS4qJgL5fhYFoe3l57mn2oiIjIpEkEE2golJGRgeDgYGzatAkjR46sddzjjz+Oxx57DP/+9791y1auXIlp06ahpKQEUunDs1pRURFcXFxQWFgIZ2fnRqm/JTiQehuTlyWhUi3g5cfaYO7IzrUePiQiImps9fn+NomZm4SEBHh6emL48OFGx5WVlVULMDKZDADY9LGJ9Wvnji/GdYVEAqw4koEvE1PFLomIiKhGoocbjUaDhIQETJo0CVZWVgbrJk6ciA8++ED3fMSIEVi8eDHWrFmD9PR07N69Gx999BFGjBihCznUdJ6J8MXckV0AAF/tScXyw9fELYiIiKgGVg8f0rQSExOhUCgQHx9fbZ1CoTCYqfnwww8hkUjw4YcfIjs7Gx4eHhgxYgT+8Y9/NGfJLdrLj7VBXokSXyamYva282hlL8eIyNrPkyIiImpuJnHOTXPiOTePThAEzN52HssPZ8BaJsHSuCg83s5D7LKIiMiCmd05N2ReJBIJPh7RGc9E+KBSLeBPK47jVGaB2GUREREBYLihBpJKJfh8bFc83s4dZSo1JickIS23ROyyiIiIGG6o4eRWUnz7Ug9E+rvgTlklJi45ipzCcrHLIiKiFo7hhh6Jg40VEiZHI9jDAdcLK/DykiTcKVWJXRYREbVgDDf0yNwc5FgxpRe8nW2RlluCycuSUaZiHyoiIhIHww01Cj9XO6yYEg0XO2ucyizAqytPQFWlEbssIiJqgRhuqNG083JCwuQo2FnLsP/yLbyz4TQ07ENFRETNjOGGGlX3wFZY/FJ3WEkl2HrqOuZuv8DWGERE1KwYbqjRDQjzxGdjIwEAyw5dw8K9aSJXRERELQnDDTWJkV39MHtEJwDAp79exqqjCpErIiKiloLhhprM5L5t8caToQCAD7ecxc9nc0SuiIiIWgKGG2pSf3m6PWKjA6ERgDfXnMKhtNtil0RERBaO4YaalEQiwd9HdcGQzt5QqTV4ZfkxnM0qFLssIiKyYAw31ORkUgm+HN8VvYNbo1SlRlxCEtJvl4pdFhERWSiGG2oWttYy/HdiD3Txc0ZeqQovLzmKm0UVYpdFREQWiOGGmo2TrTWWTY5GUGt7ZN0px8QlSSgsqxS7LCIisjAMN9Ss3B1tsGJKL3g62SDlZjGm/JCMcpVa7LKIiMiCMNxQswtws8fyKdFwtrXCsYw7mL7qBCrV7ENFRESNg+GGRNHB2xlL4qJgYyXFb5dy8d7GM+xDRUREjYLhhkQTFeSGRS92h0wqwaYT2fhk50X2oSIiokfGcEOiGtjRC/8aHQEA+N+BdHy3/6rIFRERkbljuCHRje7hj78O6wgAWPDzJaxLzhS5IiIiMmcMN2QSXnkiGK/2DwEAvL/pDH49f0PkioiIyFwx3JDJeG9IGMb29IdGAN5YfRJHr+aJXRIREZkhhhsyGRKJBJ88F45BHb2gqtJg6g/HcOF6kdhlERGRmWG4IZNiJZPimwndEN3WDcXKKkxcmgRFXpnYZRERkRlhuCGTY2stw/8m9URHH2fcLlHipSVHkVvMPlRERFQ3DDdkkpxtrfFDfBQC3eyhyC/DpKXJKKpgHyoiIno4hhsyWZ5OtlgxJRrujja4mFOEqT8cQ0Ul+1AREZFxDDdk0tq0dsAP8VFwsrFCUno+/m/1SVSxDxURERnBcEMmr7OvC76f1BNyKyl2X7iJWZvPsk0DERHViuGGzMJjwa3xn9hukEqAdcey8K9fUsQuiYiITBTDDZmNmM7emP98OABg8b4r+N8f7ENFRETVMdyQWRkXFYh3h4QBAP6+4yI2Hs8SuSIiIjI1DDdkdl7rH4Kp/doCAN7deAa/XbopckVERGRKGG7I7EgkEswa1hHPd/ODWiPg9R9P4Ni1fLHLIiIiE8FwQ2ZJKpXgny9E4MkwD1RUahC/LBkpN4rFLouIiEwAww2ZLWuZFIte7IEebVqhqKIKE5ceRWY++1AREbV0DDdk1uzkMiydFIUwLyfcLFJi4tIk3C5Ril0WERGJiOGGzJ6LvTV+iI+Gn6sd0m+XIi4hCcXsQ0VE1GIx3JBF8HbR9qFq7SDHuewi/GnFcSir2IeKiKglYrghixHs4Yhlk6PhIJfh0JU8zFhzCmoN2zQQEbU0DDdkUcL9XfD9xJ6Qy6T4+dwNfLT1HPtQERG1MAw3ZHH6hLrjy/FdIZEAq44q8MXuy2KXREREzYjhhizSsHAf/H1UFwDA17+lIeFgusgVERFRc2G4IYv1Yq82+Mvg9gCAOT9dwNZT2SJXREREzUHUcBMUFASJRFLtMX369BrHDxgwoMbxw4cPb+bKyVy88VQo4voEAQD+su409qXkilsQERE1OVHDTXJyMnJycnSP3bt3AwDGjBlT4/hNmzYZjD937hxkMlmt44kkEgn+9kwnPBvpiyqNgNdWnsBJxR2xyyIioiYkarjx8PCAt7e37rF9+3aEhISgf//+NY53c3MzGL97927Y29sz3JBRUqkEn46JxBPtPVBeqcbkZclIvck+VERElspkzrlRqVRYuXIl4uPjIZFI6vSaJUuWYPz48XBwcGji6sjcya2k+Pal7uga4IqCskpMXJqE7IJyscsiIqImYDLhZsuWLSgoKEBcXFydxiclJeHcuXOYOnWq0XFKpRJFRUUGD2qZ7OVWSIiLQqinI3IKKzBxyVHkl6rELouIiBqZyYSbJUuWYOjQofD19a3z+PDwcERHRxsdN3/+fLi4uOgeAQEBjVEumalWDnIsj4+Gj4strtwqxeRlyShVVoldFhERNSKTCDcZGRlITEx86CzMPaWlpVizZg2mTJny0LEffPABCgsLdY/MzMxHLZfMnK+rHVZMiUYre2uczizAqyuPQ1WlEbssIiJqJCYRbhISEuDp6VnnS7rXr18PpVKJl1566aFjbWxs4OzsbPAgCvV0QsLkaNjLZfgj9TbeXncKGvahIiKyCKKHG41Gg4SEBEyaNAlWVlYG6yZOnIgPPvig2muWLFmCUaNGoXXr1s1VJlmgrgGu+PalHrCWSbD9TA7m/HSefaiIiCyA6OEmMTERCoUC8fHx1dYpFArk5OQYLEtJScGBAwfqdEiK6GGeaO+BT8dEQiIBfjicga/3pIldEhERPSKJ0ML+V7WoqAguLi4oLCzkISrS+eHQNczedh4AMG9UF7z8WBuRKyIiovvV5/tb9JkbIlMwqU8Q/jywHQDgb1vPYceZnIe8goiITBXDDdFdbw1qhxd7BUIQgBlrT+JA6m2xSyIiogZguCG6SyKRYO7ILhgW7o1KtYA/rTiGM1kFYpdFRET1xHBDdB+ZVIIvxnVF39DWKFWpEZeQjCu3SsQui4iI6oHhhugBNlYyfPdyT4T7uSC/VIWJS5Jwo7BC7LKIiKiOGG6IauBoY4Vlk6MQ7O6A7IJyTFx6FAVl7ENFRGQOGG6IatHa0QbLp0TDy9kGl2+WIH5ZMspU7ENFRGTqGG6IjPBvZY8VU3rBxc4aJxQFeP3HE6hUsw8VEZEpY7gheoj2Xk5YGtcTttZS7Eu5hXc3nGEfKiIiE8ZwQ1QHPdq4YfGLPSCTSrD5ZDb+vuMi+1AREZkohhuiOnqygyc+HRMBAFh6MB2L9l0RuSIiIqoJww1RPTzXzR8fPdMJAPDvX1KwJkkhckVERPQghhuieprSry1eHxACAJi1+Sx2nbshckVERHQ/hhuiBngnJgzjegZAIwB/XnMSh6/kiV0SERHdxXBD1AASiQT/eK4Lnu7kBVWVBq8sP4Zz2YVil0VERGC4IWowK5kUX8d2Q6+2bihRViEuIQnXbpeKXRYRUYvHcEP0CGytZfh+Uk908nHG7RIVXl56FLlF7ENFRCQmhhuiR+Rsa40f4qPRprU9MvPLMXFpEgrLK8Uui4ioxWK4IWoEHk42WBHfCx5ONrh0oxhTf0hGRaVa7LKIiFokhhuiRhLY2h7L46PhZGuF5Gt38MaqE6hiHyoiombHcEPUiDr6OGPJpCjYWEmReDEX7286yzYNRETNjOGGqJFFt3XDNxO6QyaVYMPxLCz4+ZLYJRERtSgMN0RNYHAnLyx4PhwA8N3+q/jud/ahIiJqLgw3RE1kTM8AfDC0AwBg/s+XsP5YpsgVERG1DAw3RE3oT/1DMO2JYADA+5vOIvHCTZErIiKyfAw3RE3sg6EdMLq7P9QaAdNXnUBSer7YJRERWTSGG6ImJpFI8M/R4RjYwRPKKg2m/JCMizlFYpdFRGSxGG6ImoGVTIpvJnRHVFArFFdUYeLSJGTml4ldFhGRRWK4IWomdnIZ/jcxCh28nXCrWImXlhzFrWKl2GUREVkchhuiZuRir+1D5d/KDhl5ZZi0NAlFFexDRUTUmBhuiJqZl7MtVkzpBXdHOS7kFGHa8mPsQ0VE1IgYbohE0NbdAcsmR8PRxgpHrubjzTUnodawTQMRUWNguCESSRc/F/x3Yg/IZVL8cv4m/rqZfaiIiBoDww2RiPqEuOPr2K6QSoA1yZn49NcUsUsiIjJ7DDdEIhvSxQf/eE7bh2rh3itYciBd5IqIiMxbg8LNDz/8gB07duiev/vuu3B1dUWfPn2QkZHRaMURtRSx0YF4JyYMADBv+wVsOZktckVEROarQeHmk08+gZ2dHQDg8OHDWLhwIf71r3/B3d0db731VqMWSNRSvD4gBJP7BgEAZq4/jb0pueIWRERkphoUbjIzMxEaGgoA2LJlC0aPHo1p06Zh/vz5+OOPPxq1QKKWQiKR4KPhnTCqqy+qNAJeW3kcxzPuiF0WEZHZaVC4cXR0RF5eHgDg119/xeDBgwEAtra2KC8vb7zqiFoYqVSCf4+JxIAwD1RUahC/LBmXbxaLXRYRkVlpULgZPHgwpk6diqlTp+Ly5csYNmwYAOD8+fMICgpqzPqIWhxrmRSLXuyOboGuKCyvxMQlSci6wz5URER11aBws3DhQvTu3Ru3bt3Cxo0b0bp1awDA8ePHERsb26gFErVE9nIrJMRFoZ2nI24UVWDikiTklbAPFRFRXUiEFnbXsKKiIri4uKCwsBDOzs5il0NkVE5hOV5YfBjZBeWI8HfBqlceg6ONldhlERE1u/p8fzdo5mbXrl04cOCA7vnChQvRtWtXTJgwAXfu8ARIosbi42KH5VOi4eYgx5msQvxpxTEoq9iHiojImAaFm3feeQdFRUUAgLNnz+Ivf/kLhg0bhvT0dLz99tuNWiBRSxfi4YiEuCjYy2U4mJaHt9eeZh8qIiIjGhRu0tPT0alTJwDAxo0b8cwzz+CTTz7BwoUL8fPPPzdqgUQERAa44r8v94S1TIIdZ3Pwt63n2IeKiKgWDQo3crkcZWXaqzcSExPx9NNPAwDc3Nx0MzpE1Lj6tXPHF+O6QiIBfjyqwJeJqWKXRERkkhp0ZmK/fv3w9ttvo2/fvkhKSsLatWsBAJcvX4a/v3+jFkhEes9E+OJOWSU+2nIOX+1JRWtHOSb2DhK7LCIik9KgmZtvvvkGVlZW2LBhAxYvXgw/Pz8AwM8//4whQ4bU+X2CgoIgkUiqPaZPn17rawoKCjB9+nT4+PjAxsYG7du3x86dOxuyGURm6eXH2mDGoHYAgNnbzmPb6esiV0REZFpEvRT81q1bUKv1V36cO3cOgwcPxt69ezFgwIBq41UqFfr27QtPT0/MmjULfn5+yMjIgKurKyIjI+v0mbwUnCyBIAiYve08lh/OgLVMgiWTovBEew+xyyIiajL1+f5u8A0z1Go1tmzZgosXLwIAOnfujGeffRYymazO7+HhYfjLeMGCBQgJCUH//v1rHL906VLk5+fj0KFDsLa2BgDeEZlaJIlEgo9HdEZ+qQrbz+Tg1ZXHseqVx9A1wFXs0oiIRNegmZu0tDQMGzYM2dnZCAsLAwCkpKQgICAAO3bsQEhISL0LUalU8PX1xdtvv41Zs2bVOGbYsGFwc3ODvb09tm7dCg8PD0yYMAHvvfderaFKqVRCqdTf2bWoqAgBAQGcuSGLoKrSYMoPyfgj9TZa2Vtj/at9EOrpKHZZRESNrslv4vfnP/8ZISEhyMzMxIkTJ3DixAkoFAq0bdsWf/7znxtU9JYtW1BQUIC4uLhax1y9ehUbNmyAWq3Gzp078dFHH+Gzzz7D3//+91pfM3/+fLi4uOgeAQEBDaqPyBTJraRY/FIPRPq74E5ZJSYuOYrrBWxeS0QtW4NmbhwcHHDkyBGEh4cbLD99+jT69u2LkpKSehcSExMDuVyOn376qdYx7du3R0VFBdLT03UzNZ9//jn+/e9/Iycnp8bXcOaGWoL8UhVe+PYQrt4qRainI9b/qTdaOcjFLouIqNE0+cyNjY0NiouLqy0vKSmBXF7/X6gZGRlITEzE1KlTjY7z8fFB+/btDQ5BdezYETdu3IBKpaq1VmdnZ4MHkaVxc5BjxZRe8Ha2RVpuCSYvS0aZqkrssoiIRNGgcPPMM89g2rRpOHr0KARBgCAIOHLkCF599VU8++yz9X6/hIQEeHp6Yvjw4UbH9e3bF2lpadBoNLplly9fho+PT4NCFZEl8XO1w4op0XCxs8apzAK8uvIEVFWah7+QiMjCNCjcfP311wgJCUHv3r1ha2sLW1tb9OnTB6Ghofjyyy/r9V4ajQYJCQmYNGkSrKwML96aOHEiPvjgA93z1157Dfn5+XjzzTdx+fJl7NixA5988onR++IQtSTtvJyQMDkKdtYy7L98CzPXn4aGfaiIqIVp0KXgrq6u2Lp1K9LS0nSXgnfs2BGhoaH1fq/ExEQoFArEx8dXW6dQKCCV6vNXQEAAfvnlF7z11luIiIiAn58f3nzzTbz33nsN2Qwii9Q9sBUWv9QdU384hm2nr8PNQY7ZIzpBIpGIXRoRUbOo8wnF9en2/fnnnze4oKbGm/hRS7H1VDbeXHMKADDz6fZ446l24hZERPQImuQmfidPnqzTOP7fIZFpGNnVD/mlKsz56QI+/fUyWjnI8WKvNmKXRUTU5Oocbvbu3duUdRBRE5jcty3ySlT4Zm8aPtxyDq3s5RgW7iN2WURETapBJxQTkfn4y9PtERsdCEEAZqw5hUNpt8UuiYioSTHcEFk4iUSCv4/qgiGdvaFSa/DK8mM4m1UodllERE2G4YaoBZBJJfhyfFf0Dm6NUpUacQlJuHqr/ncSJyIyBww3RC2ErbUM/53YA138nJFXqsLLS5Jws6hC7LKIiBodww1RC+Jka41lk6MR1Noe2QXlmLgkCYVllWKXRUTUqBhuiFoYd0cbrJjSC55ONki5WYwpPySjXKUWuywiokbDcNNYBAFI+h64kyF2JUQPFeBmj+VTouFsa4VjGXcwfdUJVKrZh4qILAPDTWPJOQ3snAl8FQEsHQocWwqU5YtdFVGtOng7Y0lcFGyspPjtUi7e23CGfaiIyCIw3DQWTRXQtj8ACaA4BGx/C/i0PbDmReDCVqCSJ26S6YkKcsOiF7tDJpVg08lsfLLzIurYkYWIyGTVubeUpWjy3lJF14GzG4Az64CbZ/XLbVyAziOBiHFAYB9AylxJpmPj8Sz8Zf1pAMB7QzrgtQEhIldERGSoPt/fDDdN6eZ5bcg5ux4oytYvd/YHIsZog45nx6atgaiOvt9/Ff/YeREA8M/R4RgXFShyRUREegw3RojSFVyjATIOAmfWag9RKYv067zDtSGnywuAM3v+kLjm/3wR3/1+FVIJsPilHojp7C12SUREABhujBIl3NyvsgK4vEs7m3P5F0Bz7x4jEqDtE9qg03EEYCtCbdTiCYKA9zaewbpjWZBbSbEiPhq9gluLXRYREcONMaKHm/uV5QMXtmgPXSkO65db2QJhw7RBJ3QgILMWrURqearUGry68gQSL96Ek40V1vzpMXT2dRG7LCJq4RhujDCpcHO/O9e0szmn1wJ5qfrl9q2Bzs9rg45/T0AiEa1EajkqKtWYuDQJSen5cHe0wcbXeqNNawexyyKiFozhxgiTDTf3CAKQc+ruicgbgNJc/bpWbbUhJ2Is0JpXs1DTKqqoxLjvjuBiThEC3eyx4bXe8HSyFbssImqhGG6MMPlwcz91FZC+Txt0Lm4HKkv16/x6aINO5+cBRw/RSiTLlltcgRcWH4YivwwdfZyx9k+PwdmWh0mJqPkx3BhhVuHmfqpS4NJO7RVXV34DhLu9gCQy7Xk5EeO05+nI7cWtkyxORl4pRi8+jNslSkS3dcPy+GjYWsvELouIWhiGGyPMNtzcryQXOLdJG3Sun9Avlztqr7SKGKu9W7KUX0DUOM5fL8T4746gWFmFwZ28sPjF7rCS8UaURNR8GG6MsIhwc7/bqdrDVmfWAgX3Ne109NLeOydiLOATyROR6ZEduZqHiUuToKrSYGxPf/xzdAQk/HtFRM2E4cYIiws39wgCkJmkDTnnNwHld/Tr3MO0ISd8DNCqjXg1ktn75fwNvLbyODQC8Gr/ELw/tIPYJRFRC8FwY4TFhpv7VamAK3u0QefSTkCt1K8L7KMNOp1HAXatRCuRzNfaZAXe26jtm/bXYR3xyhPBIldERC0Bw40RLSLc3K+iELj4kzbopP8B4O7ulsmBdk9rT0Ru9zRgzUt8qe4W7UvDv3alAAA+GxOJ0T38Ra6IiCwdw40RLS7c3K8wGzh3r2P5Of1yWxegEzuWU90JgoB/7LiI/x1Ih0wqwfcTe+CpDl5il0VEFozhxogWHW7ud+MccHYdcGY9UHxdv5wdy6mONBoBM9efxqaT2bC1lmLllF7oGeQmdllEZKEYboxguHmARgNkHNDO5rBjOdVTpVqDacuPYW/KLTjbWmHdq73RwZv/roio8THcGMFwY0RlubZT+Zl1QOqvhh3Lg/trg06HZ9ixnAyUq9R4aclRHM+4A08nG2x8rQ8C3HgzSSJqXAw3RjDc1FFZPnB+szboZB7RL7eyAzoMA8LHsmM56RSWVWLsd4eRcrMYbd0dsP7V3nB3tBG7LCKyIAw3RjDcNEB+uraJ5xl2LKfa3SiswOjFh5BdUI4ufs5Y/cpjcGIfKiJqJAw3RjDcPAJBAK6f1M7mnNvIjuVUzdVbJXjh28PIL1Whd3BrJEyOYh8qImoUDDdGMNw0EoOO5T8BlWX6dX49756I/Dzg4C5aiSSOs1mFGP/fwyhVqTG0ize+mdAdMiln9Yjo0TDcGMFw0wSUJUDK/R3LNdrlEhkQOkg7m8OO5S3KobTbiEtIhkqtwYRegfjHqC7sQ0VEj4ThxgiGmyZWfFPb2+rMWu0hrHvYsbzF2Xk2B9NXnYAgAP/3VCj+8nSY2CURkRljuDGC4aYZ3bp890aBa4EChX65ozcQfrdjuXcET0S2YD8ezcBfN2vvhj17RCdM7ttW5IqIyFwx3BjBcCMCYx3LPTroO5a7BopXIzWZ/+xJxWe7LwMAvhrfFSO7+olcERGZI4YbIxhuRFalAtIStUEn5WfDjuVt+mqDTqeR7FhuQQRBwJyfLmDZoWuwkkrwv0k9MSDMU+yyiMjMMNwYwXBjQioKgQvbtEHn2gHU2LG8fQxgxZvBmTuNRsCMtaew7fR12FnL8OMrvdA9kAGWiOqO4cYIhhsTVZh190aB64Dc8/rlti5Ap1F3O5b3ZsdyM6aq0mDq8mPYf/kWXO2tsf5PvdHOy0nssojITDDcGMFwYwZq61juEqA9NydiHODZQbz6qMHKVFWY8P1RnMosgI+LLTa81gd+rnZil0VEZoDhxgiGGzOiUQMZB7WHrS5se6BjecTdGwWOZsdyM3OnVIUx3x1GWm4Jgj0csOHVPnBzkItdFhGZOIYbIxhuzFRlOXB5130dy6u0yyVSoO0T2qDTcQRgw8Mc5uB6QTlGLz6EnMIKRPq7YNUrj8HBxkrssojIhDHcGMFwYwFK84AL9zqWH9Uvv9exPGIcEPIUO5abuLTcYoz59jDulFXi8XbuWDIpCnIrnlNFRDVjuDGC4cbC6DqWrwHy0vTL7VtrD1lFjAP8evBGgSbqVGYBJnx/BGUqNZ6J8MHX47tByj5URFQDhhsjGG4slEHH8g1A6S39OrdgbcgJH8OO5SZo/+VbmPJDMirVAib2boM5z3ZmHyoiqobhxgiGmxZAXQVc3ac9EfnSdsOO5f5RQPhYdiw3MVtPZWPG2lMQBOCtQe3x5qB2YpdERCamPt/foh7gDgoKgkQiqfaYPn16jeOXLVtWbaytrW0zV00mT2YFtBsEjP4emJkKPPdfIGSg9uTjrGTg53eAz8KAH8dqD2mpyh7+ntSkRnb1w8cjOgMAvki8jBVHMkSuiIjMmaiXJyQnJ0OtVuuenzt3DoMHD8aYMWNqfY2zszNSUlJ0zzl9TUbZOAKR47SP4pvAuY3ae+hcPwmk/qJ9yB2Bjs/e7Vj+BDuWi2RSnyDklarw9Z5U/G3rObjZyzE8gpf5E1H9iRpuPDw8DJ4vWLAAISEh6N+/f62vkUgk8Pb2burSyBI5eQG9X9c+HuxYfnqV9qHrWD4O8A7nicjN7K1B7ZBXosSPRxWYsfYkXOys0a8dDx8SUf2YzHWXKpUKK1euRHx8vNHZmJKSErRp0wYBAQEYOXIkzp8/X+tYAFAqlSgqKjJ4EMGjPfDUh8CbZ4D4X4Ce8YCtK1ByAzj8DfDd48Ci3sAfnwMFmWJX22JIJBLMHdkFw8K9UakWMG3FMZzOLBC7LCIyMyZzQvG6deswYcIEKBQK+Pr61jjm8OHDSE1NRUREBAoLC/Hpp59i//79OH/+PPz9/Wt8zccff4w5c+ZUW84TiqmaKhWQtvtux/Jd7FguImWVGvHLknEwLQ9uDnKsf7U3QjwcxS6LiERklldLxcTEQC6X46effqrzayorK9GxY0fExsZi3rx5NY5RKpVQKvVfUkVFRQgICGC4IePKC4CL27SXlj/Ysbx9jPawVbun2bG8CZUoqxD73yM4m10IP1c7bHitN3xc2IeKqKUyu3CTkZGB4OBgbNq0CSNHjqzXa8eMGQMrKyusXr26TuN5KTjVm65j+Vog94J+ua0L0Pk5bdAJeIwdy5tAXokSY749jKu3S9HO0xHrX+0NV3v2oSJqiczmUvB7EhIS4OnpieHDh9frdWq1GmfPnoWPD6+ooCbk4g/0mwG8fhh49SDQ58+Akw9QUQgcXwYkDAW+igT2zAVyL4ldrUVp7WiD5VOi4eVsg9TcEsQvS0aZqkrssojIxIk+c6PRaNC2bVvExsZiwYIFBusmTpwIPz8/zJ8/HwAwd+5cPPbYYwgNDUVBQQH+/e9/Y8uWLTh+/Dg6depUp8/jzA01Co1ae7jqzDrgwlZAVaxfd69jefgLgBOv7GsMl29q+1AVlldiQJgHvp/YE9Yyk/h/MyJqJmY1c5OYmAiFQoH4+Phq6xQKBXJycnTP79y5g1deeQUdO3bEsGHDUFRUhEOHDtU52BA1GqkMCO4PjFoIvJMKvJAAtB8KSK2AG2eAX/8KfN4RWD4KOLUaUBY/9C2pdu29nLA0ridsraXYl3IL76w/DY1G9CPqRGSiRJ+5aW6cuaEmZbRj+fC7HcufZMfyBtp7KRdTlx+DWiNgct8g/O2ZTryRJ1ELYXYnFDcnhhtqNvlX9SciG3Qsd7+vY3l33iiwnjafzMJba08DAN6JCcP0J0NFroiImgPDjREMN9TsBAG4fkI7m3N2A1B2W7/OLUR7/xx2LK+XJQfSMW+79sq1Bc+HY3x0oMgVEVFTY7gxguGGRKWu1Hcsv7gdqCrXr/OP0s7mdH6OHcvr4F+7LmHRviuQSoBFL3bHkC68apLIkjHcGMFwQyZDWQxc2qGd0bm6FxA02uVSKyB0kHZGp/1QQG4vbp0mShAEvL/xLNYey4RcJsWy+Cj0CWEoJLJUDDdGMNyQSbrXsfzMWiDnlH653AnodLdjedDj7Fj+gCq1Bq//eAK/XrgJRxsrrJn2GLr4uYhdFhE1AYYbIxhuyOTdStHO5pxZBxQq9MudfLT3zgkfy47l96moVGPS0iQcTc+Hu6McG17tgyB3B7HLIqJGxnBjBMMNmQ2NRns5+Zm1wPnNQEWBfp1HR/2JyK4BopVoKooqKjH+uyO4kFMEP1c7vDe0A2I6e8HGijNdRJaC4cYIhhsyS1VKIPVux/LLvzzQsbzffR3LXUUrUWy3ipV44dtDyMgrAwC0srfG6O7+GB8diFBPdhQnMncMN0Yw3JDZM+hY/od+uUwOtB9yt2P54BbZsTyvRInlhzOwNjkTN4oqdMuj27phQnQghnTxhq01Z3OIzBHDjREMN2RRCjKBcxuA02uBWxf1y21dgc6jWmzH8iq1Br9fvoXVSQr8dikX9zo1uNhZ4/nufoiNDkR7LydxiySiemG4MYLhhiySIAA3z2kPW53dABTre7LBJRCIGKMNOh5h4tUokpzCcqw/loW1yZnILtDfV6hnm1aIjQ7EsHAf2Mk5m0Nk6hhujGC4IYunUWsPV51ZB1zYZtix3CdSG3K6jG5xHcvVGgH7U29h9VEF9lzKhfrudI6TrRWe7+aH2F6B6ODN3wlEporhxgiGG2pRKsuBlJ+1QSdtN6Cp0i6XSIHgAdrLyjs+A9i0rEM0uUUVWH88C6uTFMi6o5/N6RboitjoQDwT4QN7uZWIFRLRgxhujGC4oRarNA84v0kbdLKS9MtbcMdyjUbAgbTbWJOswK/nb6Lq3myOjRVGdvNFbHQgOvvypoBEpoDhxgiGGyJoO5afWa89Ryf/in55C+5YfqtYiQ3Hs7AmWaG7nBwAIv1dEBsdiBGRvnCw4WwOkVgYboxguCG6jyAA2Se0Iefcxho6lo/TnozsFixejc1MoxFw5GoeViUp8Mv5G6hUa39FOshleLarHyZEByLcn7M5RM2N4cYIhhuiWqgrgSt7gbPrauhYHq29UWDn5wGH1uLV2MzySpTYeCILa5IycfV2qW55Fz9njI8KxMiuvnCybTmH8YjExHBjBMMNUR3oOpavBa7uq7ljedgwwNpO1DKbiyAIOJqej9VJCvx89gZUau3Pw85ahmcjfRHbKxCR/i6QtKDDeETNjeHGCIYbonoqvnFfx/LT+uUttGP5nVKVdjYnORNpuSW65R28nTChVyBGdvWDix1nc4gaG8ONEQw3RI8g95L2sNWZ9TV3LI8YB3h1aREnIguCgGMZd7D6qALbz+ZAVaWdzbG1luKZCO2VVt0DXTmbQ9RIGG6MYLghagQaDZB5RHtZ+YMdyz076TuWu/iLVmJzKihTYfPJbKxOUuDyTf1sTnsvR8RGB+L5bv5wsedsDtGjYLgxguGGqJEZdCzfBahV+nUtrGO5IAg4oSjA6iQFtp+5jopK7WyOjZUUw8N9MD46EFFBrTibQ9QADDdGMNwQNaHyO9qWD2fWARkH9MtlNkD7mBbVsbywvBJbT2Vj1VEFLt3Qt8AI8XBAbHQgRnf3RysHuYgVEpkXhhsjGG6ImklBJnD27o0Cb13SL7d1BTo/d7djeS+L71guCAJOZxVi9VEFtp2+jvJKNQBALpNiSBdvxEYH4rFgN87mED0Ew40RDDdEzUwQgBtn9TcKvL9juWugtr9VxNgW0bG8uKIS205fx6qjCpy/XqRbHuzugPHRARjd3R+tHS1/VouoIRhujGC4IRKRQcfyrYBKf/ItfLpqQ04L6Vh+NqsQq5IU2HYqG6Uq7WyOtUyCpzt7Y0J0IHoHt4ZUytkconsYboxguCEyEaoy4PK9juWJ1TuWR4wDOjwD2DiKWmZTK1VW4afT17E6SYHTWYW65W1a22N8VCBe6OEPDyfO5hAx3BjBcENkgkpvay8pP7MWyErWL7e213csD34SkFl248rz1wuxJikTW05mo1ipDXtWUgkGd/JCbHQg+oW6czaHWiyGGyMYbohMXN4V/YnI+Vf1yx087nYsHwv4WnbH8jJVFbafycHqJAVOKgp0y/1b2WF8VADG9AyAl7OteAUSiYDhxgiGGyIzYaxjeetQ7WxO+BjAra14NTaDSzeKsCYpExtPZKG4QjubI5NKMLCDJ2J7BeKJdh6QcTaHWgCGGyMYbojM0L2O5WfWaht6tsCO5eUqNXae1c7mHMu4o1vu52qHcVEBGNszAN4unM0hy8VwYwTDDZGZUxYDF7drg0767w90LB+s7XEVOsii74icerMYq+/O5hSWVwIApBLgqQ6eiI0ORP/2HrCSWfb9g6jlYbgxguGGyIIU5eg7lt84o18ukQH+UdqQ024Q4B1pkTcLrKhUY9e5G1idpMDR9Hzdch8XW4zpGYBxUQHwc7UTsUKixsNwYwTDDZGFyr1097DVduD2ZcN19u5A6EBt2Al5CnBwF6fGJpSWW4K1yQpsOJ6FO2Xa2RyJBBjQ3gOx0YF4qoMnZ3PIrDHcGMFwQ9QC3MkAruwB0vYAV/cZ3iwQEsC3mzbohA4C/HpY1CXmyio1fj1/E6uTFDh0JU+33NPJBmPvzuYEuNmLWCFRwzDcGMFwQ9TCVKmAzKPaGwWm7QFunjVcb+sKhDx5d1ZnIODsI0qZTSH9dinWJCuw4VgW8kq13dolEuDxdh6IjQrAoE5esOZsDpkJhhsjGG6IWriinLuzOonAld+AikLD9V5d9LM6Ab0AK/Pv3K2q0iDxonY2549U/SX17o42GNPTH+OjAtCmtYOIFRI9HMONEQw3RKSjrgKunwBSd2vDzvWTAO77lSh3BNr2156UHDpI2+jTzCnyyrAmWYH1x7Nwq1ipW94v1B3jowPwdCdvyK04m0Omh+HGCIYbIqpV6W3t/XTSdmsPYd1/40AAcG9/d1ZnINCmH2BtvveVqVRrsOdiLlYnKbA/9RbufRO0dpDjhR7+GBcVgGAPy+7rReaF4cYIhhsiqhONBrhxWjujk5oIZCXp76kDAFZ2QFA//SGs1iFm2xIiM78M649lYu2xTNws0s/mPBbshtjoQAzp4g0bK5mIFRIx3BjFcENEDVJ+B7j6u/7E5OLrhutbBemDTtDjZtnNvEqtwd6UW1idpMC+lFxo7n47tLK3xuju/hgfHYhQT/PbLrIMDDdGMNwQ0SMTBCD3wt2gkwhkHAY0lfr1MjkQ2Fsfdjw7mt2szvWCcqw7lom1yZnIKazQLY8OckNsrwAM7eIDW2vO5lDzYbgxguGGiBqdsgRI33837OwGChSG65399DcRbNvfrFpDqDUCfr+ci1VHM7E3JRfqu9M5LnbWeK6bHyb0CkR7LyeRq6SWgOHGCIYbImpSggDkXdEHnWsHgCr9zAckMiAg+m7YGQx4R5hNa4gbhRVYfywTa5IzkV2gb17ao00rxEYHYni4D+zknM2hpsFwYwTDDRE1q8pyIOOg9jyd1N1AXqrhegcP7c0Dda0hTL+zuVoj4I/UW1iTlIndF2/qZnOcbK3wXDc/xEYHoqMPf79S42K4MYLhhohEdeeaNuik7dF2NX+wNYRfd8PWEFLTngnJLarA+uNZWJOsQGa+fjana4ArJkQH4plIH9jLLae9BYnHbMJNUFAQMjIyqi1//fXXsXDhQqOvXbNmDWJjYzFy5Ehs2bKlzp/JcENEJqNKBWQeua81xDnD9bau2tmce/fWcfIWpcy60GgEHLxyG6uTFPj1/E1U3Z3NcbSxwsiuvoiNDkQXPxeRqyRzZjbh5tatW1Cr1brn586dw+DBg7F3714MGDCg1tddu3YN/fr1Q3BwMNzc3BhuiMgyFF2/O6uTCFzdW0NriHD93ZIDegEya3HqfIhbxUpsPJGFNUkKXMsr0y2P8HdBbHQgRkT6wtGGszlUP2YTbh40Y8YMbN++HampqZDUctmkWq3GE088gfj4ePzxxx8oKChguCEiy6OuArKP371b8r3WEPeROwHB/fWzOibYGkKjEXDkah5WJ2di17kcVKq1XzcOchmevTubE+7nUuvve6L7mWW4UalU8PX1xdtvv41Zs2bVOm727Nk4c+YMNm/ejLi4OIYbImoZSm5pZ3PuHcKq1hoiTBt02g0CAvuYXGuIvBIlNp3IxuokBa7eLtUt7+zrjNjoQIzs6gsnW9OciSLTYJbhZt26dZgwYQIUCgV8fX1rHHPgwAGMHz8ep06dgru7e53CjVKphFKpv514UVERAgICGG6IyHxpNEDOKf0hrJpaQ7R93LA1hIkQBAFJ6flYnaTAznM3oKrS1m1nLcOISB/ERgeia4ArZ3OoGrMMNzExMZDL5fjpp59qXF9cXIyIiAgsWrQIQ4cOBYA6hZuPP/4Yc+bMqbac4YaILEb5HeDqvvtaQ+QYrm/VVh902j4OyB1EKfNBd0pV2HRSO5uTlqu/aqyDtxNiowMxqpsfXOw4m0NaZhduMjIyEBwcjE2bNmHkyJE1jjl16hS6desGmUx/WaRGo038UqkUKSkpCAmp/n8nnLkhohZFEICb5/WtIRRHqreGaNNHH3Y8OojeGkIQBBzPuINVSQrsOJMD5d3ZHFtrKYaH+2JCrwB0D2zF2ZwWzuzCzccff4zvvvsOmZmZsLKq+Qz6iooKpKWlGSz78MMPUVxcjK+++grt27eHXC5/6GfxnBsialGUxUD6H9oTk1MTgcIHW0P461tDBPcHbMW9XLuwrBKbT2ZhdVImUm4W65a393LE+KhAPN/dD672D/9dT5bHrMKNRqNB27ZtERsbiwULFhismzhxIvz8/DB//vwaX8sTiomI6kEQgLw07YxO6t3WEGr9zLa2NUQvbdhpN1h76blIrSEEQcDJzAKsPqrAT2euo6JSO5sjt5JieLj23JyoIM7mtCT1+f4W/UYDiYmJUCgUiI+Pr7ZOoVBAaiY9V4iITJ5EAri30z4eew1QlQEZh/SHsPJSAcUh7eO3eYCDp35WJ+QpwN6tGUuVoHtgK3QPbIWPRnTC1lPXseqoAhdzirD5ZDY2n8xGiIcDYqMD8Xx3f7g5cDaH9ESfuWlunLkhIqpFfjpw5W5riKu/A5Wl962UaNtB6FpDdG/21hCCIOBMViFWJymw7fR1lKm0N4GVy6QY0sUb46MD0Du4NWdzLJRZHZZqbgw3RER1UKXUnox87wqs3POG6+1a6VtDhAwEnLyatbwSZRW2nbqO1UkKnM3W38m5rbsDxkcFYHQPf7g72jRrTdS0GG6MYLghImoAXWuI3cCVfYDygdYQ3hH6WZ2A6GZtDXEuWzubs/XUdZQoqwAA1jIJnu7kjdjoQPQJaQ2plLM55o7hxgiGGyKiR6SuArKP6U9MzjlluP5ea4h2g7WzOq4BzVJWqbIK289cx6qkTJzOLNAtD3Szx/joALzQwx+eTqZ152aqO4YbIxhuiIgaWckt4Mpv2rBzZQ9Qlme43qODflanTR/AqukPF124XoQ1yQpsPpGN4ruzOVZSCQZ19EJsr0A8HurO2Rwzw3BjBMMNEVET0miAnJP3tYZINmwNYW0PBD2ub/jZxK0hylRV2HEmB6uTFDihKNAt929lh/FRARjTMwBezpzNMQcMN0Yw3BARNaPyO8CVvfqwU3LDcL1bsH5WJ6hfk7aGSLlRjNVJCmw6kYWiCu1sjkwqwVMdPDEhOhBPtPeAjLM5JovhxgiGGyIikehaQ+zWhh3FYUBTpV8vs3mgNURYk7SGqKhUY+fZHKxJykTStXzdcl8XW4yNCsDYngHwdbVr9M+lR8NwYwTDDRGRiVAWA+n7756YXENrCJcA/U0E2/YHbBv/d3ZabjFWJ2Vi44ksFJRpe3BJJcCTYZ6IjQ7EgDAPWMl4M1lTwHBjBMMNEZEJEgTgdurd++rsBq4dNGwNIbXSt4YIHQx4hzfqrE5FpRq/nL+B1UkKHLmqn83xdtbO5oyLCoAfZ3NExXBjBMMNEZEZUJUBGQfvaw1h2DgZjl7ay8xDBzZ6a4grt0qwNjkTG45nIb9UBUCbo/q390BsdCCe6uAJa87mNDuGGyMYboiIzFB+uv5uyen7DVtDSKSGrSF8uzVKawhllRq7L9zE6iQFDqbpL2/3dLLBmJ7+GB8ViAA3+0f+HKobhhsjGG6IiMxclVJ7MrKuNcQFw/V2bvrWEKEDAUfPR/7Ia7dLsSY5ExuOZ+J2iX42p1+oOyZEB2JQJy/O5jQxhhsjGG6IiCxMYbb25oGpu4Gr+wBlkeF67wjt3ZJDBwH+UY/UGkJVpcGeizexKkmBP1Jv65a7O8rxQo8AjI8KQJB7013O3pIx3BjBcENEZMHUlUDWMf2JyTmnDdfbOGtbQ4QO1s7quPg3+KMy88uwJlmBdceycKtYf/Jz39DWiI0OxOBOXrCxat7O6ZaM4cYIhhsiohakJFffGiJtD1Ceb7jeo6M25LQbDAT2blBriEq1Br9dysXqJAV+v3wL975V3RzkeKGHP8ZHBSDYw7ERNqZlY7gxguGGiKiF0qiB66f0V2BlH6veGqLtE/oTk93a1vsjsu6UYV1yJtYdy8KNogrd8seC3RAbHYiYzt6wteZsTkMw3BjBcENERACAsnzg6v2tIW4arncLeaA1RN2vjKpSa7Av5RZWJymwNyUXmrvftK721hjd3R+x0QEI9XRqxI2xfAw3RjDcEBFRNYIA3Dynv1ty5pHqrSGC+urDjnv7Ot9EMKewHOuSs7A2WYHrhfrZnKigVoiNDsSwcB/O5tQBw40RDDdERPRQFUX61hBpiUBhpuF6l8D7WkM8UafWEGqNgP2Xb2FVkgK/XcqF+u50jrOtFZ7v7o/Y6ECEeXM2pzYMN0Yw3BARUb0IAnD7sj7o1Nga4jGg3d1ZHa8uD53VuVlUgfXHMrE6KRPZBeW65d0DXREbHYhnInxhJ+dszv0YboxguCEiokeiKtUGnHthJ/+K4XpH77uzOgOB4CeNtobQaAT8kXYbq48qkHjxJqruzuY42VrhuW5+GB8ViE6+/K4CGG6MYrghIqJGlX9Vf1Jy+n6gsky/TiIF/Ho+0Bqi5jsZ5xZXYMPxLKxJyoQiX/8ekQGumBAdgGcifOFgY9XUW2OyGG6MYLghIqImc681ROpubeC5ddFwvX1rfWuIkKdqbA2h0Qg4dCUPq5MV+PX8DVSqtV/TjjZWeLarLyZEB6KLn0tzbI1JYbgxguGGiIiaTWGWflanptYQPl31szr+UYDMcGbmdokSG49nYXWSAtfy9LM54X4uiI0OxLNdfeHYQmZzGG6MYLghIiJRqCuBrGT9uTrVWkO4aFtDtBsMhAwEXPx0qwRBwJGr+VidpMCuczegUmtvPmgvl+HZSF/ERgciwt8Fkjpenm6OGG6MYLghIiKTUHxT3xriyh6g/I7hes9Od09MHgwEPqZrDZFfqsKmE9rZnCu3SnXDO/k4I7ZXIEZ29YWzbcObg5oqhhsjGG6IiMjkaNTA9ZP6WZ2sYwDu+3q2drjbGuJuH6xWQRAEAcnX7mB1kgI7zuZAVaWdzbGzluGZCB/E9gpEtwBXi5nNYbgxguGGiIhM3r3WEKl3w05pruH61qH6c3Xa9EVBlRU2ncjG6iQFUnNLdMM6eDshNjoQo7r5wcXOvGdzGG6MYLghIiKzotHoW0OkJQKZRw1bQ1jZAm20rSGE0IE4UeqOVUlZ2H7mOpR3Z3NsrKQYHuGDCdGB6NGmlVnO5jDcGMFwQ0REZq2iUN8aIjURKMoyXO8aCIQOQmngAGwtDMXy43m4dKNYt7qdpyPGRwdidHc/uNrLm7n4hmO4MYLhhoiILIYgALdS9LM6GQcBtUq/XmoNIbAXrrv3w9qCMHyfYofySu1sjtxKimFdvBEbHYjotm4mP5vDcGMEww0REVksVSlw7cB9rSGuGqzWOHrhqktvrC8Iw+q8EBTBEQAQ7OGA2KhAjO7hDzcH05zNYbgxguGGiIhajLwr+psIXvvDoDWEIJEiw64zfirpiN2V4TgrtIW1zAoxXbwRGx2A3sGtTWo2h+HGCIYbIiJqkSortK0h7s3q3LpksLpQ4ozfqsLxuzoSf2jC4dTaB+OjA/FCD3+4O9qIVLQew40RDDdEREQACjK1Nw9MSwSu/l6tNcQZTVvs00TiILrCs0M/jO0VhL4h7pBKxZnNYbgxguGGiIjoAepKIDNJP6tz44zB6iLBHn9ouuCMXRR8uj+DYX27w9PJtllLZLgxguGGiIjoIYpv6mZ1qlL3wEpZYLD6kiYAGW694dV9BMJ7x0Bm3fSHrRhujGC4ISIiqoe7rSEqL/2ConO70KrgLKT3tYYogy1uuEWjddfhcAkfArQKapIyGG6MYLghIiJ6BGX5yDmxEzdPbId//iG4o9BgdalTW9h1Hg5pzN+BRrzaiuHGCIYbIiKixlGhqsShg/tw4/h2hBQdQQ/JZVhJNDgj64TOfz0EWSOefFyf72+rRvtUIiIialFs5dZ46snBwJODkZZbjC8OXcSN07+go58HIkS6qgrgzI3Y5RAREVkUZZUaJRVVaN3I98bhzA0RERGJwsZKBhtHmag1SEX9dCIiIqJGxnBDREREFoXhhoiIiCwKww0RERFZFFHDTVBQECQSSbXH9OnTaxy/adMm9OzZE66urnBwcEDXrl2xYsWKZq6aiIiITJmoV0slJydDrVbrnp87dw6DBw/GmDFjahzv5uaGv/71r+jQoQPkcjm2b9+OyZMnw9PTEzExMc1VNhEREZkwk7rPzYwZM7B9+3akpqZCUsdbNnfv3h3Dhw/HvHnz6jSe97khIiIyP/X5/jaZc25UKhVWrlyJ+Pj4OgUbQRCwZ88epKSk4Iknnqh1nFKpRFFRkcGDiIiILJfJ3MRvy5YtKCgoQFxcnNFxhYWF8PPzg1KphEwmw6JFizB48OBax8+fPx9z5sxp5GqJiIjIVJnMYamYmBjI5XL89NNPRsdpNBpcvXoVJSUl2LNnD+bNm4ctW7ZgwIABNY5XKpVQKpW650VFRQgICOBhKSIiIjNidu0XMjIykJiYiE2bNj10rFQqRWhoKACga9euuHjxIubPn19ruLGxsYGNTeP2tyAiIiLTZRLn3CQkJMDT0xPDhw+v92s1Go3BzAwRERG1bKLP3Gg0GiQkJGDSpEmwsjIsZ+LEifDz88P8+fMBaM+f6dmzJ0JCQqBUKrFz506sWLECixcvFqN0IiIiMkGih5vExEQoFArEx8dXW6dQKCCV6ieXSktL8frrryMrKwt2dnbo0KEDVq5ciXHjxtX58+6dYsSrpoiIiMzHve/tupwqbDInFDeXrKwsBAQEiF0GERERNUBmZib8/f2Njmlx4Uaj0eD69etwcnKq840C6+relViZmZkWeSWWpW8fYPnbyO0zf5a+jdw+89dU2ygIAoqLi+Hr62twVKcmoh+Wam5SqfShie9ROTs7W+xfWsDytw+w/G3k9pk/S99Gbp/5a4ptdHFxqdM4k7haioiIiKixMNwQERGRRWG4aUQ2NjaYPXu2xd400NK3D7D8beT2mT9L30Zun/kzhW1scScUExERkWXjzA0RERFZFIYbIiIisigMN0RERGRRGG6IiIjIojDcGLFw4UIEBQXB1tYWvXr1QlJSktHx69evR4cOHWBra4vw8HDs3LnTYL0gCPjb3/4GHx8f2NnZYdCgQUhNTW3KTXio+mzj999/j8cffxytWrVCq1atMGjQoGrj4+LiIJFIDB5Dhgxp6s2oVX22b9myZdVqt7W1NRhjavuwPts3YMCAatsnkUgwfPhw3RhT2n/79+/HiBEj4OvrC4lEgi1btjz0Nfv27UP37t1hY2OD0NBQLFu2rNqY+v67bkr13cZNmzZh8ODB8PDwgLOzM3r37o1ffvnFYMzHH39cbR926NChCbeidvXdvn379tX4d/TGjRsG48x5H9b0b0wikaBz5866MaayD+fPn4+oqCg4OTnB09MTo0aNQkpKykNfZwrfhQw3tVi7di3efvttzJ49GydOnEBkZCRiYmKQm5tb4/hDhw4hNjYWU6ZMwcmTJzFq1CiMGjUK586d043517/+ha+//hrffvstjh49CgcHB8TExKCioqK5NstAfbdx3759iI2Nxd69e3H48GEEBATg6aefRnZ2tsG4IUOGICcnR/dYvXp1c2xONfXdPkB7R837a8/IyDBYb0r7sL7bt2nTJoNtO3fuHGQyGcaMGWMwzlT2X2lpKSIjI7Fw4cI6jU9PT8fw4cPx5JNP4tSpU5gxYwamTp1q8OXfkL8TTam+27h//34MHjwYO3fuxPHjx/Hkk09ixIgROHnypMG4zp07G+zDAwcONEX5D1Xf7bsnJSXFoH5PT0/dOnPfh1999ZXBtmVmZsLNza3av0NT2Ie///47pk+fjiNHjmD37t2orKzE008/jdLS0lpfYzLfhQLVKDo6Wpg+fbruuVqtFnx9fYX58+fXOH7s2LHC8OHDDZb16tVL+NOf/iQIgiBoNBrB29tb+Pe//61bX1BQINjY2AirV69ugi14uPpu44OqqqoEJycn4YcfftAtmzRpkjBy5MjGLrVB6rt9CQkJgouLS63vZ2r78FH33xdffCE4OTkJJSUlumWmtP/uB0DYvHmz0THvvvuu0LlzZ4Nl48aNE2JiYnTPH/Vn1pTqso016dSpkzBnzhzd89mzZwuRkZGNV1gjqcv27d27VwAg3Llzp9YxlrYPN2/eLEgkEuHatWu6Zaa6D3NzcwUAwu+//17rGFP5LuTMTQ1UKhWOHz+OQYMG6ZZJpVIMGjQIhw8frvE1hw8fNhgPADExMbrx6enpuHHjhsEYFxcX9OrVq9b3bEoN2cYHlZWVobKyEm5ubgbL9+3bB09PT4SFheG1115DXl5eo9ZeFw3dvpKSErRp0wYBAQEYOXIkzp8/r1tnSvuwMfbfkiVLMH78eDg4OBgsN4X91xAP+zfYGD8zU6PRaFBcXFzt32Bqaip8fX0RHByMF198EQqFQqQKG6Zr167w8fHB4MGDcfDgQd1yS9yHS5YswaBBg9CmTRuD5aa4DwsLCwGg2t+3+5nKdyHDTQ1u374NtVoNLy8vg+VeXl7Vjv3ec+PGDaPj7/23Pu/ZlBqyjQ9677334Ovra/CXdMiQIVi+fDn27NmDf/7zn/j9998xdOhQqNXqRq3/YRqyfWFhYVi6dCm2bt2KlStXQqPRoE+fPsjKygJgWvvwUfdfUlISzp07h6lTpxosN5X91xC1/RssKipCeXl5o/ydNzWffvopSkpKMHbsWN2yXr16YdmyZdi1axcWL16M9PR0PP744yguLhax0rrx8fHBt99+i40bN2Ljxo0ICAjAgAEDcOLECQCN83vLlFy/fh0///xztX+HprgPNRoNZsyYgb59+6JLly61jjOV78IW1xWcGseCBQuwZs0a7Nu3z+Ck2/Hjx+v+HB4ejoiICISEhGDfvn0YOHCgGKXWWe/evdG7d2/d8z59+qBjx4747rvvMG/ePBEra3xLlixBeHg4oqOjDZab8/5raVatWoU5c+Zg69atBuekDB06VPfniIgI9OrVC23atMG6deswZcoUMUqts7CwMISFheme9+nTB1euXMEXX3yBFStWiFhZ0/jhhx/g6uqKUaNGGSw3xX04ffp0nDt3TrTzt+qLMzc1cHd3h0wmw82bNw2W37x5E97e3jW+xtvb2+j4e/+tz3s2pYZs4z2ffvopFixYgF9//RURERFGxwYHB8Pd3R1paWmPXHN9PMr23WNtbY1u3brpajelffgo21daWoo1a9bU6ZekWPuvIWr7N+js7Aw7O7tG+TthKtasWYOpU6di3bp11Q4BPMjV1RXt27c3i31Yk+joaF3tlrQPBUHA0qVL8fLLL0MulxsdK/Y+fOONN7B9+3bs3bsX/v7+Rseaynchw00N5HI5evTogT179uiWaTQa7Nmzx+D/7O/Xu3dvg/EAsHv3bt34tm3bwtvb22BMUVERjh49Wut7NqWGbCOgPct93rx52LVrF3r27PnQz8nKykJeXh58fHwape66auj23U+tVuPs2bO62k1pHz7K9q1fvx5KpRIvvfTSQz9HrP3XEA/7N9gYfydMwerVqzF58mSsXr3a4DL+2pSUlODKlStmsQ9rcurUKV3tlrIPAe2VSGlpaXX6nwyx9qEgCHjjjTewefNm/Pbbb2jbtu1DX2My34WNdmqyhVmzZo1gY2MjLFu2TLhw4YIwbdo0wdXVVbhx44YgCILw8ssvC++//75u/MGDBwUrKyvh008/FS5evCjMnj1bsLa2Fs6ePasbs2DBAsHV1VXYunWrcObMGWHkyJFC27ZthfLy8mbfPkGo/zYuWLBAkMvlwoYNG4ScnBzdo7i4WBAEQSguLhZmzpwpHD58WEhPTxcSExOF7t27C+3atRMqKipMfvvmzJkj/PLLL8KVK1eE48ePC+PHjxdsbW2F8+fP68aY0j6s7/bd069fP2HcuHHVlpva/isuLhZOnjwpnDx5UgAgfP7558LJkyeFjIwMQRAE4f333xdefvll3firV68K9vb2wjvvvCNcvHhRWLhwoSCTyYRdu3bpxjzsZ9bc6ruNP/74o2BlZSUsXLjQ4N9gQUGBbsxf/vIXYd++fUJ6erpw8OBBYdCgQYK7u7uQm5tr8tv3xRdfCFu2bBFSU1OFs2fPCm+++aYglUqFxMRE3Rhz34f3vPTSS0KvXr1qfE9T2Yevvfaa4OLiIuzbt8/g71tZWZlujKl+FzLcGPGf//xHCAwMFORyuRAdHS0cOXJEt65///7CpEmTDMavW7dOaN++vSCXy4XOnTsLO3bsMFiv0WiEjz76SPDy8hJsbGyEgQMHCikpKc2xKbWqzza2adNGAFDtMXv2bEEQBKGsrEx4+umnBQ8PD8Ha2lpo06aN8Morr4j2S0cQ6rd9M2bM0I318vIShg0bJpw4ccLg/UxtH9b37+ilS5cEAMKvv/5a7b1Mbf/duyz4wce9bZo0aZLQv3//aq/p2rWrIJfLheDgYCEhIaHa+xr7mTW3+m5j//79jY4XBO3l7z4+PoJcLhf8/PyEcePGCWlpac27YXfVd/v++c9/CiEhIYKtra3g5uYmDBgwQPjtt9+qva8570NB0F76bGdnJ/z3v/+t8T1NZR/WtF0ADP5dmep3oeTuBhARERFZBJ5zQ0RERBaF4YaIiIgsCsMNERERWRSGGyIiIrIoDDdERERkURhuiIiIyKIw3BAREZFFYbghohZv3759kEgkKCgoELsUImoEDDdERERkURhuiIiIyKIw3BCR6DQaDebPn4+2bdvCzs4OkZGR2LBhAwD9IaMdO3YgIiICtra2eOyxx3Du3DmD99i4cSM6d+4MGxsbBAUF4bPPPjNYr1Qq8d577yEgIAA2NjYIDQ3FkiVLDMYcP34cPXv2hL29Pfr06YOUlJSm3XAiahIMN0Qkuvnz52P58uX49ttvcf78ebz11lt46aWX8Pvvv+vGvPPOO/jss8+QnJwMDw8PjBgxApWVlQC0oWTs2LEYP348zp49i48//hgfffQRli1bpnv9xIkTsXr1anz99de4ePEivvvuOzg6OhrU8de//hWfffYZjh07BisrK8THxzfL9hNR42LjTCISlVKphJubGxITE9G7d2/d8qlTp6KsrAzTpk3Dk08+iTVr1mDcuHEAgPz8fPj7+2PZsmUYO3YsXnzxRdy6dQu//vqr7vXvvvsuduzYgfPnz+Py5csICwvD7t27MWjQoGo17Nu3D08++SQSExMxcOBAAMDOnTsxfPhwlJeXw9bWtol/CkTUmDhzQ0SiSktLQ1lZGQYPHgxHR0fdY/ny5bhy5Ypu3P3Bx83NDWFhYbh48SIA4OLFi+jbt6/B+/bt2xepqalQq9U4deoUZDIZ+vfvb7SWiIgI3Z99fHwAALm5uY+8jUTUvKzELoCIWraSkhIAwI4dO+Dn52ewzsbGxiDgNJSdnV2dxllbW+v+LJFIAGjPByIi88KZGyISVadOnWBjYwOFQoHQ0FCDR0BAgG7ckSNHdH++c+cOLl++jI4dOwIAOnbsiIMHDxq878GDB9G+fXvIZDKEh4dDo9EYnMNDRJaLMzdEJConJyfMnDkTb731FjQaDfr164fCwkIcPHgQzs7OaNOmDQBg7ty5aN26Nby8vPDXv/4V7u7uGDVqFADgL3/5C6KiojBv3jyMGzcOhw8fxjfffINFixYBAIKCgjBp0iTEx8fj66+/RmRkJDIyMpCbm4uxY8eKtelE1EQYbohIdPPmzYOHhwfmz5+Pq1evwtXVFd27d8esWbN0h4UWLFiAN998E6mpqejatSt++uknyOVyAED37t2xbt06/O1vf8O8efPg4+ODuXPnIi4uTvcZixcvxqxZs/D6668jLy8PgYGBmDVrlhibS0RNjFdLEZFJu3cl0507d+Dq6ip2OURkBnjODREREVkUhhsiIiKyKDwsRURERBaFMzdERERkURhuiIiIyKIw3BAREZFFYbghIiIii8JwQ0RERBaF4YaIiIgsCsMNERERWRSGGyIiIrIoDDdERERkUf4fW8zIZBUyxMIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(model.history.history.keys())\n",
    "# summarize history for accuracy\n",
    "plt.plot(model.history.history['loss'])\n",
    "plt.plot(model.history.history['val_loss'])\n",
    "plt.title('model loss')\n",
    "plt.ylabel('loss')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'val'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22d65d78",
   "metadata": {},
   "source": [
    "Но нас интересует не сама задача классификации, а получившиеся векторные представления."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1793a448",
   "metadata": {},
   "source": [
    "У нас есть два слоя с матрицами размера (длина словаря, 100). Можно взять векторные представления из любого из этих слоев или взять усредненный вектор."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "5b903e82",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<keras.layers.core.embedding.Embedding at 0x30c86e520>,\n",
       " <keras.layers.reshaping.flatten.Flatten at 0x30c86e5b0>,\n",
       " <keras.layers.core.dense.Dense at 0x30c849940>]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "879f2852",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(17017, 100)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# веса embedding слоя\n",
    "model.layers[0].get_weights()[0].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "6c789439",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(100, 17017)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# веса полносвязного слоя\n",
    "model.layers[2].get_weights()[0].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "b10b6910",
   "metadata": {},
   "outputs": [],
   "source": [
    "embeddings_1 = model.layers[0].get_weights()[0]\n",
    "embeddings_2 = model.layers[2].get_weights()[0].T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "bb1a3248",
   "metadata": {},
   "outputs": [],
   "source": [
    "embeddings = np.mean([embeddings_1, embeddings_2], axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "7b6d5cdc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(17017, 100)"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embeddings.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "f19673bd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['птица',\n",
       " 'небольшая',\n",
       " 'ареала',\n",
       " 'верхнем',\n",
       " '•',\n",
       " 'шанхай',\n",
       " 'магистрали',\n",
       " 'крупная',\n",
       " 'некотором',\n",
       " 'узкой']"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "most_similar('птица', embeddings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "fe05034b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['церковь',\n",
       " 'которой',\n",
       " 'святого',\n",
       " 'книга',\n",
       " 'здание',\n",
       " 'которая',\n",
       " 'при',\n",
       " 'монастырь',\n",
       " 'её',\n",
       " 'здания']"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "most_similar('церковь', embeddings)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c84ed4f",
   "metadata": {},
   "source": [
    "### CBOW\n",
    "\n",
    "\n",
    "Теперь попробуем CBOW. В нем нужно по контексту предсказать целевое слово. В отличие от skip-gram тут сразу используется весь контекст, а векторы слов в контексте усредняются в 1 вектор контекста."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "449ee211",
   "metadata": {},
   "source": [
    "Собрать датасет очень просто."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "b79059e0",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "X = []\n",
    "y = []\n",
    "\n",
    "window = 5\n",
    "for sent in sentences[:10000]:\n",
    "    for i in range(len(sent)-1):\n",
    "        word = sent[i]\n",
    "        context = sent[max(0, i-window):i] + sent[i+1:i+window]\n",
    "        \n",
    "        X.append(context)\n",
    "        y.append(word)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2358400",
   "metadata": {},
   "source": [
    "Но нужно его немного поправить. В начале и конце текстов окна будут не полные и соответственно длина контекста будет не одинаковая. Но для преобразования в массив, длины должны быть одинаковыми. Решение этой проблемы - паддинг, добавление недостающего количества нулей  (может быть и другой индекс, но чаще всего именно 0). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "8bf10994",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[14675, 8518, 9192, 8800],\n",
       " [7069, 8518, 9192, 8800, 2861],\n",
       " [7069, 14675, 9192, 8800, 2861, 7897],\n",
       " [7069, 14675, 8518, 8800, 2861, 7897, 2759],\n",
       " [7069, 14675, 8518, 9192, 2861, 7897, 2759, 1863],\n",
       " [7069, 14675, 8518, 9192, 8800, 7897, 2759, 1863, 8800],\n",
       " [14675, 8518, 9192, 8800, 2861, 2759, 1863, 8800, 16161],\n",
       " [8518, 9192, 8800, 2861, 7897, 1863, 8800, 16161, 6972],\n",
       " [9192, 8800, 2861, 7897, 2759, 8800, 16161, 6972, 9192],\n",
       " [8800, 2861, 7897, 2759, 1863, 16161, 6972, 9192, 14573]]"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# оригинальные контексты\n",
    "X[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "8091932e",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/28/11vpccl53qqbbg2772qsny6m0000gn/T/ipykernel_99487/1088767851.py:1: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.\n",
      "  np.array(X[:10])\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([list([14675, 8518, 9192, 8800]),\n",
       "       list([7069, 8518, 9192, 8800, 2861]),\n",
       "       list([7069, 14675, 9192, 8800, 2861, 7897]),\n",
       "       list([7069, 14675, 8518, 8800, 2861, 7897, 2759]),\n",
       "       list([7069, 14675, 8518, 9192, 2861, 7897, 2759, 1863]),\n",
       "       list([7069, 14675, 8518, 9192, 8800, 7897, 2759, 1863, 8800]),\n",
       "       list([14675, 8518, 9192, 8800, 2861, 2759, 1863, 8800, 16161]),\n",
       "       list([8518, 9192, 8800, 2861, 7897, 1863, 8800, 16161, 6972]),\n",
       "       list([9192, 8800, 2861, 7897, 2759, 8800, 16161, 6972, 9192]),\n",
       "       list([8800, 2861, 7897, 2759, 1863, 16161, 6972, 9192, 14573])],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.array(X[:10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a2b44ae6",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "a423d7cd",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = tf.keras.preprocessing.sequence.pad_sequences(X, maxlen=10, padding='post')\n",
    "y = np.array(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "993b116b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[14675,  8518,  9192, ...,     0,     0,     0],\n",
       "       [ 7069,  8518,  9192, ...,     0,     0,     0],\n",
       "       [ 7069, 14675,  9192, ...,     0,     0,     0],\n",
       "       ...,\n",
       "       [ 1626, 16630,  8800, ...,  5966,     0,     0],\n",
       "       [16630,  8800,  8800, ...,     0,     0,     0],\n",
       "       [ 8800,  8800, 13277, ...,     0,     0,     0]], dtype=int32)"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# контексты после паддинга\n",
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "41d5dc4d",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_valid, y_train, y_valid = train_test_split(X, y, test_size=0.05)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08a4639c",
   "metadata": {},
   "source": [
    "Сама модель почти полностью совпадает"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "21f963c5",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/mnefedov/miniforge3/lib/python3.9/site-packages/keras/optimizers/optimizer_v2/adam.py:110: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.\n",
      "  super(Adam, self).__init__(name, **kwargs)\n"
     ]
    }
   ],
   "source": [
    "model = tf.keras.Sequential()\n",
    "\n",
    "# 1 отличие - то что на вход подается не 1 слово а целое окно контекста\n",
    "# но на уровне embedding слоя нужно изменить только input_length\n",
    "# она будет равна числу до которого мы делали паддинг\n",
    "model.add(tf.keras.layers.Embedding(input_dim=len(word2id), \n",
    "                                    input_length=10, \n",
    "                                    output_dim=100))\n",
    "\n",
    "# 2 отличие - нужно усреднить контекст в 1 вектор\n",
    "model.add(tf.keras.layers.Lambda(lambda x: tf.keras.backend.sum(x, axis=1)))\n",
    "\n",
    "model.add(tf.keras.layers.Dense(len(word2id), activation='softmax'))\n",
    "model.compile(optimizer=tf.keras.optimizers.Adam(lr=0.001), loss='sparse_categorical_crossentropy',\n",
    "              \n",
    "              )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "864e36f3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " embedding_1 (Embedding)     (None, 10, 100)           1701700   \n",
      "                                                                 \n",
      " lambda (Lambda)             (None, 100)               0         \n",
      "                                                                 \n",
      " dense_1 (Dense)             (None, 17017)             1718717   \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 3,420,417\n",
      "Trainable params: 3,420,417\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "print(model.summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "34d67f13",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "1729/1729 [==============================] - 95s 55ms/step - loss: 7.1615 - val_loss: 6.5477\n",
      "Epoch 2/5\n",
      "1729/1729 [==============================] - 98s 57ms/step - loss: 6.1930 - val_loss: 6.0831\n",
      "Epoch 3/5\n",
      "1729/1729 [==============================] - 100s 58ms/step - loss: 5.7482 - val_loss: 5.8858\n",
      "Epoch 4/5\n",
      "1729/1729 [==============================] - 98s 56ms/step - loss: 5.4635 - val_loss: 5.7834\n",
      "Epoch 5/5\n",
      "1729/1729 [==============================] - 101s 59ms/step - loss: 5.2501 - val_loss: 5.7327\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x30c849d90>"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "model.fit(X_train, y_train, \n",
    "          validation_data=(X_valid, y_valid),\n",
    "          batch_size=1000,\n",
    "         epochs=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "9873dcf4",
   "metadata": {},
   "outputs": [],
   "source": [
    "embeddings_1 = model.layers[0].get_weights()[0]\n",
    "embeddings_2 = model.layers[2].get_weights()[0].T\n",
    "embeddings = np.mean([embeddings_1, embeddings_2], axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "5a2896c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "embeddings = np.mean([embeddings_1, embeddings_2], axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "7b8f0da5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(17017, 100)"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embeddings.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "b7f2dbfe",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['птица',\n",
       " 'кустарников',\n",
       " 'змей',\n",
       " 'блестящий',\n",
       " 'рыба',\n",
       " 'змея',\n",
       " 'птиц',\n",
       " 'встречается',\n",
       " 'семейства',\n",
       " 'подсемейства']"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "most_similar('птица', embeddings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f96b7ef8",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "40dac186",
   "metadata": {},
   "source": [
    "### Negative sampling"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "637385f3",
   "metadata": {},
   "source": [
    "Одна из основных оптимизаций в word2vec - negative sampling. Задача из многоклассовой классификации преобразуется в бинарную классификацию. На вход подается сразу и целевое слово и контекст, а на выходе ожидается вероятность того, что они употребляются вместе. Положительные примеры просто берутся из корпуса, а для того, чтобы собрать негативные примеры, из словаря просто случайно выбираются слова. Возможна ситуация, когда для контекста случайно выбранное слово на самом деле может встретиться в этом контексте, но за счет большого количества примеров, такие неточности не повлияют на итоговый результат."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec8312ee",
   "metadata": {},
   "source": [
    "В этот раз не будем сразу генерировать весь датасет. Он получается очень большой и эффективнее генерировать примеры на ходу (т.е. использовать генератор)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "1928254d",
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab_size = len(id2word)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0663f708",
   "metadata": {},
   "source": [
    "Эта функция просто генерирует обучающее примеры батчами. В tf генераторы можно напрямую модавать в fit модели"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "e850e1d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# skip gram\n",
    "def gen_batches_sg(sentences, window = 5, batch_size=1000):\n",
    "    \n",
    "    # параметр window задает его целиком\n",
    "    # нам нужно поделить его пополам на левую и правую часть\n",
    "    # когда делится неровно, то левая часть больше на 1\n",
    "    left_context_length = (window/2).__ceil__() # округлить в большую сторону\n",
    "    right_context_length = window // 2 # округлить в меньшую сторону\n",
    "    \n",
    "    while True:\n",
    "        X_target = []\n",
    "        X_context = []\n",
    "        y = []\n",
    "\n",
    "        for sent in sentences:\n",
    "            for i in range(len(sent)-1):\n",
    "                word = sent[i]\n",
    "                context = sent[max(0, i-left_context_length):i] + sent[i+1:i+right_context_length]\n",
    "                for context_word in context:\n",
    "                    X_target.append(word)\n",
    "                    X_context.append(context_word)\n",
    "                    y.append(1)\n",
    "                    \n",
    "                    X_target.append(word)\n",
    "                    X_context.append(np.random.randint(vocab_size))\n",
    "                    y.append(0)\n",
    "                    \n",
    "                    if len(X_target) >= batch_size:\n",
    "                        X_target = np.array(X_target)\n",
    "                        X_context = np.array(X_context)\n",
    "                        y = np.array(y)\n",
    "                        yield ((X_target, X_context), y)\n",
    "                        X_target = []\n",
    "                        X_context = []\n",
    "                        y = []\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "33541b1f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # cbow \n",
    "def gen_batches_cbow(sentences, window = 5, batch_size=1000):\n",
    "    \n",
    "    # параметр window задает его целиком\n",
    "    # нам нужно поделить его пополам на левую и правую часть\n",
    "    # когда делится неровно, то левая часть больше на 1\n",
    "    left_context_length = (window/2).__ceil__() # округлить в большую сторону\n",
    "    right_context_length = window // 2 # округлить в меньшую сторону\n",
    "    \n",
    "    while True:\n",
    "        X_target = []\n",
    "        X_context = []\n",
    "        y = []\n",
    "\n",
    "        for sent in sentences:\n",
    "            for i in range(len(sent)-1):\n",
    "                word = sent[i]\n",
    "                context = sent[max(0, i-left_context_length):i] + sent[i+1:i+right_context_length]\n",
    "\n",
    "                X_target.append(word)\n",
    "                X_context.append(context)\n",
    "                y.append(1)\n",
    "                \n",
    "                X_target.append(np.random.randint(vocab_size))\n",
    "                X_context.append(context)\n",
    "                y.append(0)\n",
    "\n",
    "                if len(X_target) == batch_size:\n",
    "                    X_target = np.array(X_target)\n",
    "                    X_context = tf.keras.preprocessing.sequence.pad_sequences(X_context, maxlen=window)\n",
    "                    y = np.array(y)\n",
    "                    yield ((X_target, X_context), y)\n",
    "                    X_target = []\n",
    "                    X_context = []\n",
    "                    y = []"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74b09b8a",
   "metadata": {},
   "source": [
    "Саму модель нужно уже задать через Functional API, т.к. у нас будет два входа"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "6e086b72",
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs_target = tf.keras.layers.Input(shape=(1,))\n",
    "inputs_context = tf.keras.layers.Input(shape=(1,))\n",
    "\n",
    "\n",
    "embeddings_target = tf.keras.layers.Embedding(input_dim=len(word2id), output_dim=300)(inputs_target, )\n",
    "embeddings_context = tf.keras.layers.Embedding(input_dim=len(word2id), output_dim=300)(inputs_context, )\n",
    "\n",
    "target = tf.keras.layers.Flatten()(embeddings_target)\n",
    "context = tf.keras.layers.Flatten()(embeddings_context)\n",
    "\n",
    "dot = tf.keras.layers.Dot(1)([target, context])\n",
    "outputs = tf.keras.layers.Activation(activation='sigmoid')(dot)\n",
    "\n",
    "model = tf.keras.Model(inputs=[inputs_target, inputs_context], \n",
    "                       outputs=outputs)\n",
    "optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)\n",
    "model.compile(optimizer=optimizer,\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "952c5ab4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "10000/10000 [==============================] - 156s 16ms/step - loss: 0.4743 - accuracy: 0.7791 - val_loss: 0.5320 - val_accuracy: 0.7565\n",
      "Epoch 2/2\n",
      "10000/10000 [==============================] - 156s 16ms/step - loss: 0.4505 - accuracy: 0.7969 - val_loss: 0.4750 - val_accuracy: 0.7860\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x30e8e9ee0>"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(gen_batches_sg(sentences[:19000], window=5),\n",
    "          validation_data=gen_batches_sg(sentences[19000:],  window=5),\n",
    "          batch_size=1000,\n",
    "          steps_per_epoch=10000,\n",
    "          validation_steps=30,\n",
    "         epochs=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "1a56360e",
   "metadata": {},
   "outputs": [],
   "source": [
    "embeddings = model.layers[2].get_weights()[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "f06af355",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['церковь',\n",
       " 'построена',\n",
       " 'храм',\n",
       " 'чудотворца',\n",
       " 'часовня',\n",
       " 'селе',\n",
       " 'деревни',\n",
       " 'святителя',\n",
       " 'николая',\n",
       " 'села']"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "most_similar('церковь', embeddings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5c539e13",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "7255ae35",
   "metadata": {},
   "outputs": [],
   "source": [
    "#cbow negative sampling\n",
    "inputs_target = tf.keras.layers.Input(shape=(1,))\n",
    "inputs_context = tf.keras.layers.Input(shape=(10,))\n",
    "\n",
    "\n",
    "embeddings_target = tf.keras.layers.Embedding(input_dim=len(word2id), output_dim=300)(inputs_target, )\n",
    "embeddings_context = tf.keras.layers.Embedding(input_dim=len(word2id), output_dim=300)(inputs_context, )\n",
    "\n",
    "target = tf.keras.layers.Flatten()(embeddings_target)\n",
    "context = tf.keras.layers.Lambda(lambda x: tf.keras.backend.sum(x, axis=1))(embeddings_context)\n",
    "dot = tf.keras.layers.Dot(1)([target, context])\n",
    "\n",
    "# полученную близость нужно преобразовать в вероятность\n",
    "# когда она одна используется не софтмакс и сигмоида\n",
    "outputs = tf.keras.layers.Activation(activation='sigmoid')(dot)\n",
    "\n",
    "model = tf.keras.Model(inputs=[inputs_target, inputs_context], \n",
    "                       outputs=outputs)\n",
    "\n",
    "\n",
    "optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)\n",
    "model.compile(optimizer=optimizer,\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=['accuracy'])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "36a09425",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "5000/5000 [==============================] - 81s 16ms/step - loss: 0.3749 - accuracy: 0.8336 - val_loss: 0.3023 - val_accuracy: 0.8737\n",
      "Epoch 2/2\n",
      "5000/5000 [==============================] - 79s 16ms/step - loss: 0.2654 - accuracy: 0.8895 - val_loss: 0.2520 - val_accuracy: 0.8952\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x30eec8910>"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(gen_batches_cbow(sentences[:19000], window=5),\n",
    "          validation_data=gen_batches_cbow(sentences[19000:],  window=5),\n",
    "          batch_size=1000,\n",
    "          steps_per_epoch=5000,\n",
    "          validation_steps=30,\n",
    "         epochs=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "778836d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "embeddings = model.layers[2].get_weights()[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "953fe0eb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['семья',\n",
       " 'уехала',\n",
       " 'переехала',\n",
       " 'сестру',\n",
       " 'проживала',\n",
       " 'жила',\n",
       " 'уезжает',\n",
       " 'родители',\n",
       " 'уехал',\n",
       " 'жену']"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "most_similar('семья', embeddings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "07e6b428",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['церковь',\n",
       " 'часовня',\n",
       " 'церкви',\n",
       " 'храма',\n",
       " 'монастыря',\n",
       " 'здание',\n",
       " 'святого',\n",
       " 'монастырь',\n",
       " 'храм',\n",
       " 'часовни']"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "most_similar('церковь', embeddings)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8dc9cac",
   "metadata": {},
   "source": [
    "Другая значимая оптимизация в word2vec - это иерархический софтмакс. Когда мы считали модель без негативного семплирования, то у нас был большой полносвязный слой с софтмаксом. Иерархический софтмакс упрощает расчеты в этом слое, но его уже так просто не воспроизведешь.\n",
    "\n",
    "На этом месте уже лучше перейти к готовой реализации."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7af3bf3",
   "metadata": {},
   "source": [
    "## Word2Vec в gensim"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10b28433",
   "metadata": {},
   "source": [
    "В gensim есть питоновская обертка к оригинальному ворд2веку, которой очень удобно пользоваться."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "90fa2ab4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install gensim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "d916424d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import gensim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "0b744492",
   "metadata": {},
   "outputs": [],
   "source": [
    "?gensim.models.Word2Vec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "a942a031",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'4.3.0'"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gensim.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5414972b",
   "metadata": {},
   "source": [
    "На вход нужно подавать список списков (токенизированный текст)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "28b573f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "wiki = open('wiki_data.txt').read().split('\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "912f3adf",
   "metadata": {},
   "outputs": [],
   "source": [
    "texts = [preprocess(text) for text in wiki]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "ce4b4186",
   "metadata": {},
   "outputs": [],
   "source": [
    "# f = open('corpus.txt', 'w')\n",
    "# for text in texts:\n",
    "#     f.write(' '.join(text) + '\\n')\n",
    "# f.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "118e3ee0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# corpus = gensim.models.word2vec.LineSentence('corpus.txt', )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91051ba2",
   "metadata": {},
   "source": [
    "В саму модель нужно передать корпус для обучения, для остальных параметров есть значения по умолчанию"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "71504b2b",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 37.8 s, sys: 144 ms, total: 37.9 s\n",
      "Wall time: 14 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "w2v = gensim.models.Word2Vec(texts, )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "524499f8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('деревянная', 0.8377265334129333),\n",
       " ('часовня', 0.8358676433563232),\n",
       " ('богородицы', 0.8274974822998047),\n",
       " ('построена', 0.821127712726593),\n",
       " ('православная', 0.8120527267456055),\n",
       " ('храм', 0.8012527227401733),\n",
       " ('икона', 0.8005196452140808),\n",
       " ('сгорела', 0.777483344078064),\n",
       " ('построенная', 0.7769498229026794),\n",
       " ('храма', 0.7756760716438293)]"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w2v.wv.most_similar('церковь')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3363b11e",
   "metadata": {},
   "source": [
    "Давайте теперь разберемся с параметрами:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6128bdd",
   "metadata": {},
   "source": [
    "**vector_size** - размер вектора, обычно используют значение 100-1000  \n",
    "\n",
    "**min_count** - минимальная частота слова в корпусе (выше мы тоже фильтровали по частототе 30)  \n",
    "\n",
    "**max_vocab_size** - максимальный размер словаря\n",
    "\n",
    "**window** - размер окна (выше мы использовали значение 10)\n",
    "\n",
    "**epochs** - количество эпох (полных итераций по корпусу)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "6ea31ac0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 26.1 s, sys: 1.86 s, total: 28 s\n",
      "Wall time: 9.59 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "w2v = gensim.models.Word2Vec(texts, \n",
    "                             vector_size=300, \n",
    "                             min_count=30, \n",
    "                             max_vocab_size=10000,\n",
    "                             window=5,\n",
    "                             epochs=7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "304b59ab",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('храм', 0.7495061755180359),\n",
       " ('церкви', 0.6722915768623352),\n",
       " ('монастыря', 0.6711115837097168),\n",
       " ('монастырь', 0.6551513075828552),\n",
       " ('храма', 0.6509529948234558),\n",
       " ('собор', 0.633480429649353),\n",
       " ('святого', 0.6160135865211487),\n",
       " ('собора', 0.6044516563415527),\n",
       " ('епархии', 0.574800968170166),\n",
       " ('николая', 0.5695961713790894)]"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w2v.wv.most_similar('церковь')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "004aa64a",
   "metadata": {},
   "source": [
    "CBOW или Skip-gram выбираются через параметр **sg** - 1 это skip-gram, а 0 - cbow (по умолчанию всегда cbow)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "6982b4b4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1min 30s, sys: 452 ms, total: 1min 30s\n",
      "Wall time: 30.9 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "w2v = gensim.models.Word2Vec(texts,\n",
    "                             vector_size=300, \n",
    "                             min_count=30, \n",
    "                             max_vocab_size=10000,\n",
    "                             window=5,\n",
    "                             epochs=7,\n",
    "                             sg=1\n",
    "                            )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a516ad01",
   "metadata": {},
   "source": [
    "Скип-грам обучается значительно дольше "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "240d2fbe",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('храм', 0.6280046105384827),\n",
       " ('церкви', 0.5492430329322815),\n",
       " ('собор', 0.5290341973304749),\n",
       " ('храма', 0.5172240734100342),\n",
       " ('монастырь', 0.5040321350097656),\n",
       " ('монастыря', 0.4657866954803467),\n",
       " ('епархии', 0.45769721269607544),\n",
       " ('здание', 0.44704288244247437),\n",
       " ('собора', 0.44320300221443176),\n",
       " ('школа', 0.426459938287735)]"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w2v.wv.most_similar('церковь')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5180b31b",
   "metadata": {},
   "source": [
    "Увеличения window на cbow практически не влияет"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "a6b81c04",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 31 s, sys: 1.67 s, total: 32.7 s\n",
      "Wall time: 11.2 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "w2v = gensim.models.Word2Vec(texts,\n",
    "                             vector_size=300, \n",
    "                             min_count=30, \n",
    "                             max_vocab_size=10000,\n",
    "                             window=10,\n",
    "                             epochs=7,\n",
    "                             \n",
    "                            )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d8a6dc5",
   "metadata": {},
   "source": [
    "А вот skip-gram сильно замедляется с увеличением окна"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "6cef32e5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2min 34s, sys: 238 ms, total: 2min 34s\n",
      "Wall time: 52.4 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "w2v = gensim.models.Word2Vec(texts,\n",
    "                             vector_size=300, \n",
    "                             min_count=30, \n",
    "                             max_vocab_size=10000,\n",
    "                             window=10,\n",
    "                             epochs=7,\n",
    "                             sg=1\n",
    "                            )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b79d046d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "6a9ec418",
   "metadata": {},
   "source": [
    "Выбор между negative_sampling и hierarchical softmax осуществляется через 2 параметра **hs** и **negative**:   \n",
    "1) **hs=0, negative=0** - ни то ни другое не используется  \n",
    "2) **hs=1** - используется hierarchical softmax  \n",
    "3) **hs=0, negative > 0** - используется negative_sampling и число в negative указывает сколько негативных примеров семплировать (в документации советуют значения от 5 до 20)  (по умолчанию)   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "6fa3ed28",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 7.88 s, sys: 125 ms, total: 8.01 s\n",
      "Wall time: 8 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "w2v = gensim.models.Word2Vec(texts,\n",
    "                             hs=0,\n",
    "                             negative=0\n",
    "                            )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ab7c517",
   "metadata": {},
   "source": [
    "Работает быстрее, но качество сильно страдает"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "6868493d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('отрицательных', 0.39327651262283325),\n",
       " ('инвестиционный', 0.37610307335853577),\n",
       " ('даль', 0.3691421449184418),\n",
       " ('приходили', 0.36865225434303284),\n",
       " ('восточно-казахстанской', 0.3680453598499298),\n",
       " ('пояснил', 0.36741116642951965),\n",
       " ('заклинания', 0.36411765217781067),\n",
       " ('постепенным', 0.35541704297065735),\n",
       " ('проходка', 0.3530977964401245),\n",
       " ('brm', 0.35309237241744995)]"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w2v.wv.most_similar('церковь')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4008e899",
   "metadata": {},
   "source": [
    "Иерархический софтмакс работает сильно дольше"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "b9c0725e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1min 22s, sys: 1.57 s, total: 1min 24s\n",
      "Wall time: 30.5 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "w2v = gensim.models.Word2Vec(texts, \n",
    "                             hs=1\n",
    "                            )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "ea742df9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('часовня', 0.8145325183868408),\n",
       " ('храм', 0.7590469717979431),\n",
       " ('церкви', 0.7268047332763672),\n",
       " ('деревянная', 0.6939830183982849),\n",
       " ('икона', 0.6819214224815369),\n",
       " ('колокольня', 0.679404616355896),\n",
       " ('монастырь', 0.6711443066596985),\n",
       " ('храма', 0.6586366891860962),\n",
       " ('каменная', 0.642469584941864),\n",
       " ('святого', 0.6422722935676575)]"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w2v.wv.most_similar('церковь')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96c3a389",
   "metadata": {},
   "source": [
    "Дефолтная настройка работает достаточно быстро и хорошо"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "bbc7b815",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 39.5 s, sys: 1.48 s, total: 41 s\n",
      "Wall time: 14.5 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "w2v = gensim.models.Word2Vec(texts, \n",
    "                             hs=0,\n",
    "                             negative=5\n",
    "                            )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "id": "266440a9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('часовня', 0.8631377816200256),\n",
       " ('деревянная', 0.8212272524833679),\n",
       " ('богородицы', 0.8176008462905884),\n",
       " ('храм', 0.8152698278427124),\n",
       " ('православная', 0.809910774230957),\n",
       " ('построена', 0.8091415762901306),\n",
       " ('икона', 0.7990718483924866),\n",
       " ('монастыря', 0.7944340109825134),\n",
       " ('построенная', 0.7885655760765076),\n",
       " ('пресвятой', 0.7850500345230103)]"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w2v.wv.most_similar('церковь')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1da5a8c",
   "metadata": {},
   "source": [
    "Повышение negative влияет на время обучения"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "id": "ee61c82b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1min 35s, sys: 565 ms, total: 1min 36s\n",
      "Wall time: 33.2 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "w2v = gensim.models.Word2Vec(texts, \n",
    "                             hs=0,\n",
    "                             negative=20\n",
    "                            )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "id": "c297b275",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('часовня', 0.831301212310791),\n",
       " ('деревянная', 0.8208282589912415),\n",
       " ('храм', 0.7935220003128052),\n",
       " ('богородицы', 0.7933349609375),\n",
       " ('икона', 0.784454882144928),\n",
       " ('построена', 0.7777988910675049),\n",
       " ('колокольня', 0.7736459970474243),\n",
       " ('монастырь', 0.7726446390151978),\n",
       " ('каменная', 0.7674807906150818),\n",
       " ('пресвятой', 0.7669936418533325)]"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w2v.wv.most_similar('церковь')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f40a7e24",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "b6b5357b",
   "metadata": {},
   "source": [
    "Есть еще несколько параметров, но уже не таких влиятельных:\n",
    "\n",
    "1) **sample** - отвечает за downsampling частотных слов; чем меньше, тем сильнее штрафуются частотные слова (по умолчанию стоит 1e-3, но можно поставить 1e-4, 1e-5). Для sg это будет значительно влиять на скорость (1e-5 будет быстрее)  \n",
    "2) **ns_exponent** -  регулирет downsampling частотных слов в negative sampling; 0 - никакого штрафа за частотность (как у нас выше с полным рандомом), 1 - штрафует пропорционально частотности, значения 0-1 что-то посередине (по умолчанию стоит 0.75)  \n",
    "3) **cbow_mean** - отвечает за функцию агрегации в cbow; 1 - это среднее, 0 - сумма (у нас выше среднее, оно же стоит по умолчанию)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "id": "0d1a5345",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1min 24s, sys: 739 ms, total: 1min 25s\n",
      "Wall time: 29.6 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "w2v = gensim.models.Word2Vec(texts, \n",
    "                             sample=1e-5,\n",
    "                             sg=1\n",
    "                            )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "id": "1c3928c9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('храм', 0.9655008316040039),\n",
       " ('часовня', 0.9532762765884399),\n",
       " ('храма', 0.9530141353607178),\n",
       " ('храме', 0.9416210055351257),\n",
       " ('рождества', 0.9381084442138672),\n",
       " ('деревянная', 0.9327593445777893),\n",
       " ('собор', 0.9324644804000854),\n",
       " ('освящена', 0.9293673038482666),\n",
       " ('чудотворца', 0.9289676547050476),\n",
       " ('построена', 0.9267347455024719)]"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w2v.wv.most_similar('церковь')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "id": "a6e9b5e7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 37.5 s, sys: 1.51 s, total: 39 s\n",
      "Wall time: 13.7 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "w2v = gensim.models.Word2Vec(texts, \n",
    "                             ns_exponent=0.84\n",
    "                            )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "id": "29ad129d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('деревянная', 0.8359008431434631),\n",
       " ('часовня', 0.8308195471763611),\n",
       " ('построена', 0.8174147605895996),\n",
       " ('храм', 0.7996065020561218),\n",
       " ('православная', 0.797082245349884),\n",
       " ('богородицы', 0.782943069934845),\n",
       " ('построенная', 0.7801513075828552),\n",
       " ('монастырь', 0.7759658098220825),\n",
       " ('икона', 0.7669859528541565),\n",
       " ('церкви', 0.7601329684257507)]"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w2v.wv.most_similar('церковь')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0cb2fda1",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "id": "fd8c2eba",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 37.9 s, sys: 1.57 s, total: 39.4 s\n",
      "Wall time: 14 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "w2v = gensim.models.Word2Vec(texts, \n",
    "                             cbow_mean=0\n",
    "                            )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "id": "c08ade15",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('деревянная', 0.939599335193634),\n",
       " ('пресвятой', 0.9233182668685913),\n",
       " ('богородицы', 0.9191804528236389),\n",
       " ('часовня', 0.9166568517684937),\n",
       " ('икона', 0.9163088798522949),\n",
       " ('построена', 0.9091044068336487),\n",
       " ('храм', 0.908804714679718),\n",
       " ('каменная', 0.9076648354530334),\n",
       " ('рождества', 0.9051446318626404),\n",
       " ('община', 0.9050907492637634)]"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w2v.wv.most_similar('церковь')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31490cc2",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "7f383ad3",
   "metadata": {},
   "source": [
    "## FastText"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77a79c3b",
   "metadata": {},
   "source": [
    "В 2015 почти та же команда выпустила улучшение - FastText (статья - https://arxiv.org/abs/1607.04606). Главная доработка в нем - использование символьных нграммов. В оригинальном word2vec словарь обучается вначале и фиксируется, добавлять новые слова к обученной модели нельзя. Соответственно, если слова нет в словаре, то и вектора для него не будет.\n",
    "\n",
    "В FastText каждое слово дополнительно разбивается на символьные нграммы и для каждого символьного нграмма тоже есть свой отдельный эмбеддинг. При предсказании векторы символьных нграммов и целого слова усредняются в 1 вектор. В дальнейшем это позволяет генерировать векторы для слов, которых нет в изначальном словаре. Нужно только чтобы новое слово состояло уже из известных символьных нграмм."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17faa699",
   "metadata": {},
   "source": [
    "Интерфейс обучения и большинство параметров - точно такие же"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "id": "ea91a3c8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2min 47s, sys: 1.69 s, total: 2min 48s\n",
      "Wall time: 1min\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "ft = gensim.models.FastText(texts)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "102ab1e1",
   "metadata": {},
   "source": [
    "Использование символьных нграммов сразу заметно - близким считаются слова похожие по написанию, а не только по смыслу"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "id": "b43dc9e5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('«церковь', 0.9903956055641174),\n",
       " ('церковь»', 0.9901298880577087),\n",
       " ('церковью', 0.9594478011131287),\n",
       " ('це́рковь', 0.9570942521095276),\n",
       " ('церкви»', 0.8876131772994995),\n",
       " ('церкви', 0.8861965537071228),\n",
       " ('церквям', 0.8725695610046387),\n",
       " ('храма', 0.8415743708610535),\n",
       " ('монастыря»', 0.8240010142326355),\n",
       " ('храму', 0.817355751991272)]"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ft.wv.most_similar('церковь')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b6b70a2",
   "metadata": {},
   "source": [
    "Дополнительные параметры:\n",
    "\n",
    "1) **min_n** - минимальная длина для символьных нграмм (по умолчанию 3)  \n",
    "2) **max_n** - максимальная длина для символьных нграмм, если поставить меньше min_n, то символьные нграммы не будут использоваться (по умолчанию 6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "id": "454cde27",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 4min 25s, sys: 2.11 s, total: 4min 27s\n",
      "Wall time: 1min 35s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "ft = gensim.models.FastText(texts, min_n=2, max_n=8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "id": "f0df4504",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('це́рковь', 0.9439380764961243),\n",
       " ('«церковь', 0.92202228307724),\n",
       " ('церковь»', 0.9158356785774231),\n",
       " ('церковью', 0.9080968499183655),\n",
       " ('церкви', 0.8009109497070312),\n",
       " ('монастырь', 0.7924925684928894),\n",
       " ('церкви»', 0.78168785572052),\n",
       " ('монастырская', 0.7621025443077087),\n",
       " ('церковная', 0.7567659616470337),\n",
       " ('святынь', 0.755405604839325)]"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ft.wv.most_similar('церковь')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5badb355",
   "metadata": {},
   "source": [
    "### Использование предобученных моделей"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc374a8f",
   "metadata": {},
   "source": [
    "В открытом доступе есть много предобученных моделей. Они как правило обучены на неспецифических, но очень больших корпусах (новостных, википедии). Если в решаемой задаче тексты похожи, то лучше использовать предобученные модели. Свои модели имеет смысл обучать, когда данные очень специфичные.\n",
    "\n",
    "Для русского готовые модели можно взять на сайте https://rusvectores.org/ru/models/ \n",
    "\n",
    "Нужно скачать какую-то модель, распаковать и открыть в генсиме.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "id": "5e2684a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# я скачал вот эту araneum_upos_skipgram_300_2_2018 \n",
    "# и распаковал архив в папку с ноутбуком\n",
    "import gensim\n",
    "model = gensim.models.KeyedVectors.load_word2vec_format('araneum_upos_skipgram_300_2_2018.vec', binary=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "id": "be30d5b5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.077049,  0.047464,  0.06009 , -0.074989,  0.044796, -0.001967,\n",
       "       -0.029693,  0.043408,  0.064917, -0.040833, -0.061425, -0.029607,\n",
       "       -0.029905,  0.064318, -0.165209, -0.048543,  0.094033,  0.098251,\n",
       "        0.120046,  0.023853,  0.016249, -0.015089, -0.05077 , -0.01809 ,\n",
       "       -0.064637, -0.039266, -0.022106, -0.019366,  0.016014, -0.051354,\n",
       "        0.046586, -0.024155,  0.062748,  0.000681,  0.05134 ,  0.049167,\n",
       "        0.005723, -0.072041, -0.049427, -0.094474, -0.075772,  0.087238,\n",
       "       -0.023207, -0.043947, -0.005014,  0.048574, -0.008169, -0.016334,\n",
       "       -0.006644,  0.026402,  0.046584, -0.040629,  0.008602,  0.047264,\n",
       "        0.00828 ,  0.089396, -0.030152, -0.011632,  0.016811,  0.081198,\n",
       "       -0.042167, -0.004215,  0.079515,  0.061092,  0.065421, -0.090715,\n",
       "       -0.07259 ,  0.061831,  0.036428,  0.024443,  0.040917,  0.043281,\n",
       "        0.050277,  0.02531 ,  0.040681, -0.073628,  0.032021, -0.020285,\n",
       "       -0.001634, -0.073642, -0.092016, -0.010889,  0.014986,  0.083961,\n",
       "       -0.037115,  0.073271,  0.003354, -0.020166, -0.011193, -0.153913,\n",
       "        0.077515,  0.048687, -0.10863 ,  0.080093,  0.04904 , -0.015638,\n",
       "       -0.011593,  0.023486, -0.054422, -0.019362,  0.060243,  0.004193,\n",
       "        0.055972,  0.02419 , -0.035099, -0.037713, -0.05796 ,  0.044854,\n",
       "       -0.009581,  0.006213,  0.091667,  0.035143, -0.058419, -0.077955,\n",
       "        0.010467, -0.045589,  0.067338,  0.049547,  0.028813, -0.053835,\n",
       "       -0.040958, -0.059349, -0.051708, -0.01979 ,  0.060717, -0.064267,\n",
       "        0.000969,  0.010098,  0.007866,  0.037205, -0.035548,  0.031603,\n",
       "       -0.101909,  0.016248,  0.004228, -0.029127, -0.102244,  0.03937 ,\n",
       "       -0.011886, -0.048208, -0.040977,  0.017401,  0.041162, -0.030116,\n",
       "       -0.074647, -0.031459, -0.001456,  0.049032,  0.099571, -0.049697,\n",
       "        0.000542, -0.01695 ,  0.043604, -0.017133, -0.003281, -0.002051,\n",
       "       -0.068998,  0.010618, -0.051367, -0.080354,  0.016654,  0.029355,\n",
       "       -0.044999, -0.08382 , -0.067588,  0.007303,  0.018462,  0.047438,\n",
       "       -0.132176,  0.01172 ,  0.030804, -0.001147, -0.097051, -0.013091,\n",
       "       -0.051683,  0.107505,  0.001299,  0.081465, -0.106922, -0.02635 ,\n",
       "       -0.060651, -0.015804, -0.090327,  0.01674 , -0.019938,  0.053004,\n",
       "        0.0802  ,  0.075273, -0.011325,  0.104902,  0.118813,  0.014863,\n",
       "       -0.031739, -0.036848,  0.007057, -0.017402,  0.023001,  0.044251,\n",
       "        0.004787,  0.08071 , -0.158897, -0.081702, -0.040502, -0.022916,\n",
       "        0.022795, -0.027717,  0.046067,  0.029809, -0.050003, -0.033388,\n",
       "       -0.007351, -0.045664, -0.089034, -0.038753, -0.00454 , -0.072393,\n",
       "        0.023547,  0.028502, -0.019941,  0.036289, -0.054831,  0.03609 ,\n",
       "        0.117796, -0.036494,  0.054533,  0.010218, -0.05965 , -0.041402,\n",
       "        0.061536,  0.061666, -0.086688,  0.004461, -0.065211, -0.196897,\n",
       "       -0.064875, -0.000384,  0.00934 , -0.018347,  0.016305,  0.00207 ,\n",
       "        0.025788,  0.025661, -0.024847,  0.001329,  0.017222,  0.023123,\n",
       "        0.020163,  0.037764,  0.097089, -0.076315,  0.062176,  0.074422,\n",
       "       -0.104194, -0.054039,  0.004654,  0.034065,  0.054541,  0.022959,\n",
       "        0.040029,  0.037736, -0.041228, -0.077804,  0.103979, -0.055198,\n",
       "        0.032196,  0.020032,  0.055866, -0.03335 ,  0.064627, -0.117019,\n",
       "       -0.019819, -0.091563,  0.016277, -0.158784,  0.090109,  0.052189,\n",
       "        0.026157,  0.043388,  0.09961 ,  0.040325,  0.01243 ,  0.062557,\n",
       "       -0.112546, -0.045751,  0.071704, -0.02192 , -0.028775, -0.010755,\n",
       "       -0.102415,  0.036397, -0.141866, -0.00391 ,  0.008874, -0.055684,\n",
       "       -0.062073, -0.0665  , -0.090125,  0.026231,  0.029068, -0.121298],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model['март_NOUN']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "id": "bb122eda",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('март_NOUN', 0.9506457448005676),\n",
       " ('ноябрь_NOUN', 0.9416142702102661),\n",
       " ('декабрь_NOUN', 0.9391435980796814),\n",
       " ('январь_NOUN', 0.9299512505531311),\n",
       " ('апрель_NOUN', 0.9271649718284607),\n",
       " ('октябрь_NOUN', 0.920407772064209),\n",
       " ('сентябрь_NOUN', 0.9028568863868713),\n",
       " ('май_NOUN', 0.8944932818412781),\n",
       " ('июнь_NOUN', 0.8860151767730713),\n",
       " ('марта_NOUN', 0.8700184226036072)]"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.most_similar('февраль_NOUN')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd8f36dd",
   "metadata": {},
   "source": [
    "Слова в модели хранятся в нормализованном виде с прибавленной частью речи. Для пребобработки там использовался udpipe, вызывать его в тетрадке неудобно, поэтому можно заменить его на майстем (по большей части все будет совпадать). На сайте есть ссылка на файл с соответствием тэгов майстема тэгам Udpipe (https://github.com/akutuzov/universal-pos-tags/blob/4653e8a9154e93fe2f417c7fdb7a357b7d6ce333/ru-rnc.map)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "id": "2d5dcca5",
   "metadata": {},
   "outputs": [],
   "source": [
    "mapping = {}\n",
    "\n",
    "for line in open('./ru-rnc.map'):\n",
    "    ms, ud = line.strip('\\n').split()\n",
    "    mapping[ms] = ud"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "id": "a590f675",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'A': 'ADJ',\n",
       " 'ADV': 'ADV',\n",
       " 'ADVPRO': 'ADV',\n",
       " 'ANUM': 'ADJ',\n",
       " 'APRO': 'DET',\n",
       " 'COM': 'ADJ',\n",
       " 'CONJ': 'SCONJ',\n",
       " 'INTJ': 'INTJ',\n",
       " 'NONLEX': 'X',\n",
       " 'NUM': 'NUM',\n",
       " 'PART': 'PART',\n",
       " 'PR': 'ADP',\n",
       " 'S': 'NOUN',\n",
       " 'SPRO': 'PRON',\n",
       " 'UNKN': 'X',\n",
       " 'V': 'VERB'}"
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mapping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "id": "67203ef9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install pymystem3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "id": "d09b0994",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pymystem3 import Mystem\n",
    "m = Mystem()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "id": "a189810b",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "def normalize_mystem(text):\n",
    "    tokens = []\n",
    "    norm_words = m.analyze(text)\n",
    "    for norm_word in norm_words:\n",
    "        if 'analysis' not in norm_word:\n",
    "            continue\n",
    "            \n",
    "        if not len(norm_word['analysis']):\n",
    "            lemma = norm_word['text']\n",
    "            pos = 'UNKN'\n",
    "        else:\n",
    "            lemma = norm_word[\"analysis\"][0][\"lex\"].lower().strip()\n",
    "            pos = norm_word[\"analysis\"][0][\"gr\"].split(',')[0]\n",
    "            pos = pos.split('=')[0].strip()\n",
    "        pos = mapping[pos]\n",
    "        tokens.append(lemma+'_'+pos)\n",
    "\n",
    "    return tokens\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c5490ed",
   "metadata": {},
   "source": [
    "Эту функцию можно применять к любому тексты и получать на выходе токены, по которым можно обращаться к модели напрямую."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "id": "fd4cbd0e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['обучать_VERB',\n",
       " 'классификатор_NOUN',\n",
       " 'парафраз_NOUN',\n",
       " 'на_ADP',\n",
       " 'предобученный_ADJ',\n",
       " 'модель_NOUN',\n",
       " 'вы_PRON',\n",
       " 'нужно_ADV',\n",
       " 'быть_VERB',\n",
       " 'дома_ADV']"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "normalize_mystem('Обучить классификатор парафразов на предобученной модели вам нужно будет дома')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "id": "2e4f1b39",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(300,)"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model['обучать_VERB'].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c94d61b3",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9295afc6",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7eb1ff00",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "38dee586",
   "metadata": {},
   "source": [
    "## Использование для классификации"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65ecfe1f",
   "metadata": {},
   "source": [
    "До этого мы почти никак не оценивали качество эмбедингов, только смотрели на ближайшие по нескольким словам. На практике нужно всегда ориентироваться на задачу, которую нужно решить. Давайте возьмем датасет для классификации и обучим несколько моделей с использованием векторных представлений."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "id": "df109f99",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.read_csv('labeled.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "id": "f85b7d97",
   "metadata": {},
   "outputs": [],
   "source": [
    "data['norm_text'] = data.comment.apply(preprocess)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "id": "a9ae9e6f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>comment</th>\n",
       "      <th>toxic</th>\n",
       "      <th>norm_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Верблюдов-то за что? Дебилы, бл...\\n</td>\n",
       "      <td>1.0</td>\n",
       "      <td>[верблюдов-то, за, что, дебилы, бл]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Хохлы, это отдушина затюканого россиянина, мол...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>[хохлы, это, отдушина, затюканого, россиянина,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Собаке - собачья смерть\\n</td>\n",
       "      <td>1.0</td>\n",
       "      <td>[собаке, собачья, смерть]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Страницу обнови, дебил. Это тоже не оскорблени...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>[страницу, обнови, дебил, это, тоже, не, оскор...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>тебя не убедил 6-страничный пдф в том, что Скр...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>[тебя, не, убедил, 6-страничный, пдф, в, том, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14407</th>\n",
       "      <td>Вонючий совковый скот прибежал и ноет. А вот и...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>[вонючий, совковый, скот, прибежал, и, ноет, а...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14408</th>\n",
       "      <td>А кого любить? Гоблина тупорылого что-ли? Или ...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>[а, кого, любить, гоблина, тупорылого, что-ли,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14409</th>\n",
       "      <td>Посмотрел Утомленных солнцем 2. И оказалось, ч...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>[посмотрел, утомленных, солнцем, 2, и, оказало...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14410</th>\n",
       "      <td>КРЫМОТРЕД НАРУШАЕТ ПРАВИЛА РАЗДЕЛА Т.К В НЕМ Н...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>[крымотред, нарушает, правила, раздела, т.к, в...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14411</th>\n",
       "      <td>До сих пор пересматриваю его видео. Орамбо кст...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>[до, сих, пор, пересматриваю, его, видео, орам...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>14412 rows × 3 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 comment  toxic  \\\n",
       "0                   Верблюдов-то за что? Дебилы, бл...\\n    1.0   \n",
       "1      Хохлы, это отдушина затюканого россиянина, мол...    1.0   \n",
       "2                              Собаке - собачья смерть\\n    1.0   \n",
       "3      Страницу обнови, дебил. Это тоже не оскорблени...    1.0   \n",
       "4      тебя не убедил 6-страничный пдф в том, что Скр...    1.0   \n",
       "...                                                  ...    ...   \n",
       "14407  Вонючий совковый скот прибежал и ноет. А вот и...    1.0   \n",
       "14408  А кого любить? Гоблина тупорылого что-ли? Или ...    1.0   \n",
       "14409  Посмотрел Утомленных солнцем 2. И оказалось, ч...    0.0   \n",
       "14410  КРЫМОТРЕД НАРУШАЕТ ПРАВИЛА РАЗДЕЛА Т.К В НЕМ Н...    1.0   \n",
       "14411  До сих пор пересматриваю его видео. Орамбо кст...    0.0   \n",
       "\n",
       "                                               norm_text  \n",
       "0                    [верблюдов-то, за, что, дебилы, бл]  \n",
       "1      [хохлы, это, отдушина, затюканого, россиянина,...  \n",
       "2                              [собаке, собачья, смерть]  \n",
       "3      [страницу, обнови, дебил, это, тоже, не, оскор...  \n",
       "4      [тебя, не, убедил, 6-страничный, пдф, в, том, ...  \n",
       "...                                                  ...  \n",
       "14407  [вонючий, совковый, скот, прибежал, и, ноет, а...  \n",
       "14408  [а, кого, любить, гоблина, тупорылого, что-ли,...  \n",
       "14409  [посмотрел, утомленных, солнцем, 2, и, оказало...  \n",
       "14410  [крымотред, нарушает, правила, раздела, т.к, в...  \n",
       "14411  [до, сих, пор, пересматриваю, его, видео, орам...  \n",
       "\n",
       "[14412 rows x 3 columns]"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "801380ad",
   "metadata": {},
   "source": [
    "Также как и в начале собираем словарь"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "id": "5e6f3dac",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7231"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab = Counter()\n",
    "\n",
    "for text in data['norm_text']:\n",
    "    vocab.update(text)\n",
    "    \n",
    "filtered_vocab = set()\n",
    "\n",
    "for word in vocab:\n",
    "    if vocab[word] > 5:\n",
    "        filtered_vocab.add(word)\n",
    "\n",
    "len(filtered_vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "id": "5600572a",
   "metadata": {},
   "outputs": [],
   "source": [
    "word2id = { 'PAD':0}\n",
    "\n",
    "for word in filtered_vocab:\n",
    "    word2id[word] = len(word2id)\n",
    "id2word = {i:word for word, i in word2id.items()}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce34a3b8",
   "metadata": {},
   "source": [
    "Переводим слова в индексы"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "id": "de33c88c",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = []\n",
    "\n",
    "for tokens in data['norm_text']:\n",
    "    ids = [word2id[token] for token in tokens if token in word2id]\n",
    "    X.append(ids)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "id": "ecbca663",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = tf.keras.preprocessing.sequence.pad_sequences(X, maxlen=100)\n",
    "y = data.toxic.values"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5703e724",
   "metadata": {},
   "source": [
    "Разбиваем на трейн и тест"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "id": "0997f420",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_valid, y_train, y_valid = train_test_split(X, y, test_size=0.05)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c169d9b6",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "435e8ed5",
   "metadata": {},
   "source": [
    "Теперь мы можем уже обучать модель"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "id": "90385246",
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs = tf.keras.layers.Input(shape=(100,))\n",
    "\n",
    "embeddings = tf.keras.layers.Embedding(input_dim=len(word2id), output_dim=100)(inputs, )\n",
    "# embedding слой возвращает последовательность векторов\n",
    "# а нам нужно классифицировать сразу весь текст\n",
    "# стандартный подход в этом случае - усреднить единичные вектора в 1 вектор текста\n",
    "mean = tf.keras.layers.Lambda(lambda x: tf.keras.backend.mean(x,  axis=1))(embeddings)\n",
    "# к усредненному вектору мы уже применяем полносвязный слой, который вернет вероятность токсичности\n",
    "outputs = tf.keras.layers.Dense(1, activation='sigmoid')(mean)\n",
    "\n",
    "model = tf.keras.Model(inputs=inputs, outputs=outputs)\n",
    "optimizer = tf.keras.optimizers.Adam()\n",
    "model.compile(optimizer=optimizer,\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=['accuracy'])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "id": "84de337e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "428/428 [==============================] - 2s 3ms/step - loss: 0.6200 - accuracy: 0.6657 - val_loss: 0.5884 - val_accuracy: 0.6560\n",
      "Epoch 2/10\n",
      "428/428 [==============================] - 1s 3ms/step - loss: 0.5472 - accuracy: 0.6962 - val_loss: 0.5134 - val_accuracy: 0.7184\n",
      "Epoch 3/10\n",
      "428/428 [==============================] - 1s 3ms/step - loss: 0.4632 - accuracy: 0.7868 - val_loss: 0.4434 - val_accuracy: 0.7864\n",
      "Epoch 4/10\n",
      "428/428 [==============================] - 1s 3ms/step - loss: 0.3938 - accuracy: 0.8428 - val_loss: 0.3975 - val_accuracy: 0.8405\n",
      "Epoch 5/10\n",
      "428/428 [==============================] - 1s 3ms/step - loss: 0.3465 - accuracy: 0.8704 - val_loss: 0.3717 - val_accuracy: 0.8433\n",
      "Epoch 6/10\n",
      "428/428 [==============================] - 1s 3ms/step - loss: 0.3116 - accuracy: 0.8868 - val_loss: 0.3561 - val_accuracy: 0.8627\n",
      "Epoch 7/10\n",
      "428/428 [==============================] - 2s 4ms/step - loss: 0.2854 - accuracy: 0.8940 - val_loss: 0.3472 - val_accuracy: 0.8613\n",
      "Epoch 8/10\n",
      "428/428 [==============================] - 1s 3ms/step - loss: 0.2621 - accuracy: 0.9046 - val_loss: 0.3421 - val_accuracy: 0.8627\n",
      "Epoch 9/10\n",
      "428/428 [==============================] - 1s 3ms/step - loss: 0.2436 - accuracy: 0.9097 - val_loss: 0.3403 - val_accuracy: 0.8669\n",
      "Epoch 10/10\n",
      "428/428 [==============================] - 1s 3ms/step - loss: 0.2269 - accuracy: 0.9149 - val_loss: 0.3401 - val_accuracy: 0.8599\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x3095f2730>"
      ]
     },
     "execution_count": 119,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train, y_train, \n",
    "          validation_data=(X_valid, y_valid),\n",
    "          batch_size=32,\n",
    "         epochs=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2d4fc1c",
   "metadata": {},
   "source": [
    "В этой модели векторные представления обучаются с нуля. Можно подставить в Embedding слой уже готовые вектора из word2vec/fastext"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c346a1a4",
   "metadata": {},
   "source": [
    "Обучаем fastext. Можно обучать на размеченных данных, можно использовать другой более объемный корпус (например, вики, который мы использовали выше), а можно и на том и на другом"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "id": "b664e3f4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 9min 30s, sys: 819 ms, total: 9min 31s\n",
      "Wall time: 3min 17s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "ft = gensim.models.FastText(texts + data['norm_text'].values.tolist(), window=10, epochs=10)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86e45c8d",
   "metadata": {},
   "source": [
    "Теперь нам нужно сделать матрицу с векторами. Индексы строчек в этой матрице должны совпадать с индексами слов в словаре."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "id": "098485ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "weights = np.zeros((len(word2id), 100))\n",
    "\n",
    "for word, i in word2id.items():\n",
    "    # вектор паддинга оставим нулевым\n",
    "    if word == 'PAD':\n",
    "        continue\n",
    "\n",
    "    try:\n",
    "        weights[i] = ft.wv[word]\n",
    "    \n",
    "    \n",
    "    except KeyError:\n",
    "        # для слов, которых нет в модели тоже возьмем какой-то  рандомный вектор\n",
    "        continue\n",
    "        weights[i] = ft.wv['опрагпллирао']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "id": "e703d96d",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "inputs = tf.keras.layers.Input(shape=(100,))\n",
    "\n",
    "# передаем матрицу в эмбединг слой\n",
    "# указываем параметр trainable=False, чтобы вектора не обучались\n",
    "# Пояснение: кажется, что обучение векторов отключать не стоит, так как модель должна взять готовые \n",
    "# эмбединги и подогнать их к нашей задаче. На практитке это однако не работает - эмбединги просто \n",
    "# # обучатся заново\n",
    "embeddings = tf.keras.layers.Embedding(input_dim=len(word2id), output_dim=100, \n",
    "                                       trainable=False,\n",
    "                                       weights=[weights])(inputs, )\n",
    "mean = tf.keras.layers.Lambda(lambda x: tf.keras.backend.mean(x,  axis=1))(embeddings)\n",
    "\n",
    "outputs = tf.keras.layers.Dense(1, activation='sigmoid')(mean)\n",
    "\n",
    "model = tf.keras.Model(inputs=inputs, outputs=outputs)\n",
    "optimizer = tf.keras.optimizers.Adam()\n",
    "model.compile(optimizer=optimizer,\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=['accuracy'])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa8459b5",
   "metadata": {},
   "source": [
    "Так как слой с эмбедингами не обучается, вся модель обучается сильно быстрее"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "id": "0e7f4630",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/30\n",
      "428/428 [==============================] - 2s 3ms/step - loss: 0.6112 - accuracy: 0.6591 - val_loss: 0.5827 - val_accuracy: 0.6630\n",
      "Epoch 2/30\n",
      "428/428 [==============================] - 1s 3ms/step - loss: 0.5607 - accuracy: 0.6928 - val_loss: 0.5400 - val_accuracy: 0.6879\n",
      "Epoch 3/30\n",
      "428/428 [==============================] - 1s 3ms/step - loss: 0.5248 - accuracy: 0.7297 - val_loss: 0.5043 - val_accuracy: 0.7448\n",
      "Epoch 4/30\n",
      "428/428 [==============================] - 1s 3ms/step - loss: 0.4926 - accuracy: 0.7639 - val_loss: 0.4734 - val_accuracy: 0.7836\n",
      "Epoch 5/30\n",
      "428/428 [==============================] - 1s 3ms/step - loss: 0.4638 - accuracy: 0.7879 - val_loss: 0.4479 - val_accuracy: 0.7933\n",
      "Epoch 6/30\n",
      "428/428 [==============================] - 1s 3ms/step - loss: 0.4372 - accuracy: 0.8075 - val_loss: 0.4268 - val_accuracy: 0.8280\n",
      "Epoch 7/30\n",
      "428/428 [==============================] - 1s 3ms/step - loss: 0.4142 - accuracy: 0.8265 - val_loss: 0.4083 - val_accuracy: 0.8322\n",
      "Epoch 8/30\n",
      "428/428 [==============================] - 1s 3ms/step - loss: 0.3934 - accuracy: 0.8381 - val_loss: 0.3962 - val_accuracy: 0.8100\n",
      "Epoch 9/30\n",
      "428/428 [==============================] - 1s 3ms/step - loss: 0.3744 - accuracy: 0.8507 - val_loss: 0.3807 - val_accuracy: 0.8474\n",
      "Epoch 10/30\n",
      "428/428 [==============================] - 1s 3ms/step - loss: 0.3572 - accuracy: 0.8603 - val_loss: 0.3693 - val_accuracy: 0.8585\n",
      "Epoch 11/30\n",
      "428/428 [==============================] - 1s 3ms/step - loss: 0.3419 - accuracy: 0.8690 - val_loss: 0.3633 - val_accuracy: 0.8391\n",
      "Epoch 12/30\n",
      "428/428 [==============================] - 1s 3ms/step - loss: 0.3271 - accuracy: 0.8763 - val_loss: 0.3534 - val_accuracy: 0.8447\n",
      "Epoch 13/30\n",
      "428/428 [==============================] - 1s 3ms/step - loss: 0.3143 - accuracy: 0.8804 - val_loss: 0.3475 - val_accuracy: 0.8682\n",
      "Epoch 14/30\n",
      "428/428 [==============================] - 1s 3ms/step - loss: 0.3017 - accuracy: 0.8864 - val_loss: 0.3418 - val_accuracy: 0.8724\n",
      "Epoch 15/30\n",
      "428/428 [==============================] - 1s 3ms/step - loss: 0.2904 - accuracy: 0.8898 - val_loss: 0.3381 - val_accuracy: 0.8710\n",
      "Epoch 16/30\n",
      "428/428 [==============================] - 1s 3ms/step - loss: 0.2795 - accuracy: 0.8923 - val_loss: 0.3354 - val_accuracy: 0.8779\n",
      "Epoch 17/30\n",
      "428/428 [==============================] - 1s 3ms/step - loss: 0.2698 - accuracy: 0.8977 - val_loss: 0.3360 - val_accuracy: 0.8682\n",
      "Epoch 18/30\n",
      "428/428 [==============================] - 1s 3ms/step - loss: 0.2602 - accuracy: 0.9040 - val_loss: 0.3336 - val_accuracy: 0.8710\n",
      "Epoch 19/30\n",
      "428/428 [==============================] - 1s 3ms/step - loss: 0.2513 - accuracy: 0.9060 - val_loss: 0.3395 - val_accuracy: 0.8433\n",
      "Epoch 20/30\n",
      "428/428 [==============================] - 1s 3ms/step - loss: 0.2430 - accuracy: 0.9086 - val_loss: 0.3340 - val_accuracy: 0.8696\n",
      "Epoch 21/30\n",
      "428/428 [==============================] - 1s 3ms/step - loss: 0.2358 - accuracy: 0.9121 - val_loss: 0.3315 - val_accuracy: 0.8696\n",
      "Epoch 22/30\n",
      "428/428 [==============================] - 1s 3ms/step - loss: 0.2281 - accuracy: 0.9148 - val_loss: 0.3331 - val_accuracy: 0.8752\n",
      "Epoch 23/30\n",
      "428/428 [==============================] - 1s 3ms/step - loss: 0.2209 - accuracy: 0.9178 - val_loss: 0.3442 - val_accuracy: 0.8571\n",
      "Epoch 24/30\n",
      "428/428 [==============================] - 1s 3ms/step - loss: 0.2146 - accuracy: 0.9176 - val_loss: 0.3362 - val_accuracy: 0.8669\n",
      "Epoch 25/30\n",
      "428/428 [==============================] - 1s 3ms/step - loss: 0.2084 - accuracy: 0.9229 - val_loss: 0.3388 - val_accuracy: 0.8682\n",
      "Epoch 26/30\n",
      "428/428 [==============================] - 1s 3ms/step - loss: 0.2022 - accuracy: 0.9246 - val_loss: 0.3387 - val_accuracy: 0.8710\n",
      "Epoch 27/30\n",
      "428/428 [==============================] - 1s 3ms/step - loss: 0.1964 - accuracy: 0.9251 - val_loss: 0.3408 - val_accuracy: 0.8669\n",
      "Epoch 28/30\n",
      "428/428 [==============================] - 1s 3ms/step - loss: 0.1913 - accuracy: 0.9295 - val_loss: 0.3436 - val_accuracy: 0.8627\n",
      "Epoch 29/30\n",
      "428/428 [==============================] - 1s 3ms/step - loss: 0.1860 - accuracy: 0.9304 - val_loss: 0.3464 - val_accuracy: 0.8655\n",
      "Epoch 30/30\n",
      "428/428 [==============================] - 1s 3ms/step - loss: 0.1808 - accuracy: 0.9318 - val_loss: 0.3531 - val_accuracy: 0.8669\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x2eee6bd60>"
      ]
     },
     "execution_count": 123,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train, y_train, \n",
    "          validation_data=(X_valid, y_valid),\n",
    "          batch_size=32,\n",
    "         epochs=30)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba5d9d51",
   "metadata": {},
   "source": [
    "Качество получается хуже, но скорее всего это из-за того, что обучающих данных слишком мало для fastext'а"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "id": "afaaac4c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['loss', 'accuracy', 'val_loss', 'val_accuracy'])\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkEAAAHHCAYAAAC4BYz1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy89olMNAAAACXBIWXMAAA9hAAAPYQGoP6dpAAByFElEQVR4nO3deVxU9f7H8dewgwiIIAii4IY75pprlpRmec3KtM2lspvteeuWldrNbla3+tlut6ttt1uWaVmWZW7lnvu+4YIbmwvIDjPn98eBUQQVEBhg3s/HYx4czpw585lxdN5+v9/z/VoMwzAQERERcTIuji5ARERExBEUgkRERMQpKQSJiIiIU1IIEhEREaekECQiIiJOSSFIREREnJJCkIiIiDglhSARERFxSgpBIiIi4pQUgkSkSh08eBCLxcInn3xS5scuXboUi8XC0qVLK7wuEXE+CkEiIiLilBSCRERExCkpBImIOFhGRoajSxBxSgpBIk7mhRdewGKxsGfPHu666y78/f0JDg5m4sSJGIbB4cOHGTJkCH5+foSGhvLGG28UO0dSUhL33nsvISEheHl5ERMTw6efflrsuNOnTzN69Gj8/f0JCAhg1KhRnD59usS6du3axa233kpgYCBeXl506dKFefPmles1Hjp0iAcffJDo6Gi8vb2pX78+w4YN4+DBgyXW+MQTTxAZGYmnpyeNGjVi5MiRpKSk2I/Jzs7mhRdeoGXLlnh5edGwYUNuvvlm4uLigAuPVSpp/NPo0aPx9fUlLi6OQYMGUbduXe68804A/vjjD4YNG0bjxo3x9PQkIiKCJ554gqysrBLfr9tuu43g4GC8vb2Jjo7mueeeA2DJkiVYLBbmzp1b7HH/+9//sFgsrFq1qqxvq0it4+boAkTEMYYPH07r1q155ZVXmD9/Pi+99BKBgYF8+OGHXHPNNbz66qt88cUXPPnkk3Tt2pW+ffsCkJWVRb9+/di3bx8PP/wwUVFRfPPNN4wePZrTp0/z2GOPAWAYBkOGDGH58uU88MADtG7dmrlz5zJq1KhitWzfvp1evXoRHh7OM888Q506dfj666+56aab+Pbbbxk6dGiZXtuff/7JypUrGTFiBI0aNeLgwYN88MEH9OvXjx07duDj4wNAeno6ffr0YefOndxzzz106tSJlJQU5s2bx5EjRwgKCsJqtXLjjTeyaNEiRowYwWOPPcaZM2dYuHAh27Zto1mzZmV+7/Pz8xkwYAC9e/fm9ddft9fzzTffkJmZybhx46hfvz5r167lnXfe4ciRI3zzzTf2x2/ZsoU+ffrg7u7O/fffT2RkJHFxcfzwww/885//pF+/fkRERPDFF18Ue++++OILmjVrRo8ePcpct0itY4iIU5k8ebIBGPfff799X35+vtGoUSPDYrEYr7zyin3/qVOnDG9vb2PUqFH2fdOmTTMA47///a99X25urtGjRw/D19fXSEtLMwzDML777jsDMF577bUiz9OnTx8DMD7++GP7/v79+xvt27c3srOz7ftsNpvRs2dPo0WLFvZ9S5YsMQBjyZIlF32NmZmZxfatWrXKAIzPPvvMvm/SpEkGYMyZM6fY8TabzTAMw5g5c6YBGG+++eYFj7lQXQcOHCj2WkeNGmUAxjPPPFOquqdOnWpYLBbj0KFD9n19+/Y16tatW2TfufUYhmFMmDDB8PT0NE6fPm3fl5SUZLi5uRmTJ08u9jwizkjdYSJO6r777rNvu7q60qVLFwzD4N5777XvDwgIIDo6mv3799v3/fTTT4SGhnL77bfb97m7u/Poo4+Snp7OsmXL7Me5ubkxbty4Is/zyCOPFKnj5MmTLF68mNtuu40zZ86QkpJCSkoKJ06cYMCAAezdu5ejR4+W6bV5e3vbt/Py8jhx4gTNmzcnICCADRs22O/79ttviYmJKbGlyWKx2I8JCgoqVve5x5THue9LSXVnZGSQkpJCz549MQyDjRs3ApCcnMzvv//OPffcQ+PGjS9Yz8iRI8nJyWH27Nn2fbNmzSI/P5+77rqr3HWL1CYKQSJO6vwvUH9/f7y8vAgKCiq2/9SpU/bfDx06RIsWLXBxKfrPR+vWre33F/5s2LAhvr6+RY6Ljo4u8vu+ffswDIOJEycSHBxc5DZ58mTAHINUFllZWUyaNImIiAg8PT0JCgoiODiY06dPk5qaaj8uLi6Odu3aXfRccXFxREdH4+ZWcaMH3NzcaNSoUbH98fHxjB49msDAQHx9fQkODuaqq64CsNddGEgvVXerVq3o2rUrX3zxhX3fF198wZVXXknz5s0r6qWI1GgaEyTipFxdXUu1D8zxPZXFZrMB8OSTTzJgwIASjynrl/YjjzzCxx9/zOOPP06PHj3w9/fHYrEwYsQI+/NVpAu1CFmt1hL3e3p6FguRVquVa6+9lpMnT/L000/TqlUr6tSpw9GjRxk9enS56h45ciSPPfYYR44cIScnh9WrV/Puu++W+TwitZVCkIiUSZMmTdiyZQs2m63IF/muXbvs9xf+XLRoEenp6UVag3bv3l3kfE2bNgXMLrXY2NgKqXH27NmMGjWqyJVt2dnZxa5Ma9asGdu2bbvouZo1a8aaNWvIy8vD3d29xGPq1asHUOz8ha1ipbF161b27NnDp59+ysiRI+37Fy5cWOS4wvfrUnUDjBgxgvHjx/Pll1+SlZWFu7s7w4cPL3VNIrWdusNEpEwGDRpEQkICs2bNsu/Lz8/nnXfewdfX1959M2jQIPLz8/nggw/sx1mtVt55550i52vQoAH9+vXjww8/5Pjx48WeLzk5ucw1urq6Fmu9euedd4q1zNxyyy1s3ry5xEvJCx9/yy23kJKSUmILSuExTZo0wdXVld9//73I/e+//36Zaj73nIXbb731VpHjgoOD6du3LzNnziQ+Pr7EegoFBQVx/fXX89///pcvvviCgQMHFuvuFHFmagkSkTK5//77+fDDDxk9ejTr168nMjKS2bNns2LFCqZNm0bdunUBGDx4ML169eKZZ57h4MGDtGnThjlz5hQZk1Povffeo3fv3rRv356xY8fStGlTEhMTWbVqFUeOHGHz5s1lqvHGG2/k888/x9/fnzZt2rBq1Sp+++036tevX+S4p556itmzZzNs2DDuueceOnfuzMmTJ5k3bx7Tp08nJiaGkSNH8tlnnzF+/HjWrl1Lnz59yMjI4LfffuPBBx9kyJAh+Pv7M2zYMN555x0sFgvNmjXjxx9/LNNYplatWtGsWTOefPJJjh49ip+fH99++22R8ViF3n77bXr37k2nTp24//77iYqK4uDBg8yfP59NmzYVOXbkyJHceuutAEyZMqVM76NIreeoy9JExDEKL5FPTk4usn/UqFFGnTp1ih1/1VVXGW3bti2yLzEx0RgzZowRFBRkeHh4GO3bty9yGXihEydOGHfffbfh5+dn+Pv7G3fffbexcePGYpeNG4ZhxMXFGSNHjjRCQ0MNd3d3Izw83LjxxhuN2bNn248p7SXyp06dstfn6+trDBgwwNi1a5fRpEmTIpf7F9b48MMPG+Hh4YaHh4fRqFEjY9SoUUZKSor9mMzMTOO5554zoqKiDHd3dyM0NNS49dZbjbi4OPsxycnJxi233GL4+PgY9erVM/76178a27ZtK/ES+ZLeZ8MwjB07dhixsbGGr6+vERQUZIwdO9bYvHlzie/Xtm3bjKFDhxoBAQGGl5eXER0dbUycOLHYOXNycox69eoZ/v7+RlZW1kXfNxFnYzGMShzxKCIiDpWfn09YWBiDBw9mxowZji5HpFrRmCARkVrsu+++Izk5uchgaxExqSVIRKQWWrNmDVu2bGHKlCkEBQUVmSRSRExqCRIRqYU++OADxo0bR4MGDfjss88cXY5ItaSWIBEREXFKagkSERERp6QQJCIiIk5JkyWWwGazcezYMerWrXtZq0SLiIhI1TEMgzNnzhAWFlZsfb6SKASV4NixY0RERDi6DBERESmHw4cP06hRo0sepxBUgsJp/w8fPoyfn5+DqxEREZHSSEtLIyIiwv49fikKQSUo7ALz8/NTCBIREalhSjuURQOjRURExCkpBImIiIhTUggSERERp6QxQZfBarWSl5fn6DJqJA8Pj1JdvigiIlJZFILKwTAMEhISOH36tKNLqbFcXFyIiorCw8PD0aWIiIiTUggqh8IA1KBBA3x8fDShYhkVTkZ5/PhxGjdurPdPREQcQiGojKxWqz0A1a9f39Hl1FjBwcEcO3aM/Px83N3dHV2OiIg4IQ3KKKPCMUA+Pj4OrqRmK+wGs1qtDq5ERESclUJQOakL5/Lo/RMREUdTCBIRERGnpBAk5RIZGcm0adMcXYaIiEi5aWC0E+nXrx8dO3askPDy559/UqdOncsvSkRExEEUgsTOMAysVitubpf+WAQHB1dBRSIiUtvk5FtJSsvB28OVIF9Ph9ai7jAnMXr0aJYtW8Zbb72FxWLBYrHwySefYLFY+Pnnn+ncuTOenp4sX76cuLg4hgwZQkhICL6+vnTt2pXffvutyPnO7w6zWCz85z//YejQofj4+NCiRQvmzZtXxa9SREQcKTvPysGUDFbFnWDuxiO8v3QfE7/bxn2fruPGd/6gy0sLiX5+AX1eW8JXa+MdXa5agiqCYRhk5VX9pd7e7q6lvsrqrbfeYs+ePbRr144XX3wRgO3btwPwzDPP8Prrr9O0aVPq1avH4cOHGTRoEP/85z/x9PTks88+Y/DgwezevZvGjRtf8Dn+8Y9/8Nprr/Gvf/2Ld955hzvvvJNDhw4RGBh4+S9WRESqnGEYZOfZSM3K43RWLqcz8zidmUdqVi7JZ3I4nppNQmo2x1KzSUjN4lRm6ZaS8nRzITvPVsnVX5pCUAXIyrPSZtIvVf68O14cgI9H6f4I/f398fDwwMfHh9DQUAB27doFwIsvvsi1115rPzYwMJCYmBj771OmTGHu3LnMmzePhx9++ILPMXr0aG6//XYAXn75Zd5++23Wrl3LwIEDy/zaRESk8iSlZbP24ElOZeSaASczj9NZZwPOufty88sWVrzcXQjz9ybU34tQfy/7dsOC3xv6e1PPx71aTJWiECR06dKlyO/p6em88MILzJ8/n+PHj5Ofn09WVhbx8RdvuuzQoYN9u06dOvj5+ZGUlFQpNYuISNmczszl520JzNt0jNUHTmAYpX+sq4uFAG93/L3d8fdxJ8Dbnfq+noT5exHq720POGH+3vh5u1WLgFMaCkEVwNvdlR0vDnDI81aE86/yevLJJ1m4cCGvv/46zZs3x9vbm1tvvZXc3NyLnuf85S8sFgs2m+ObO0VEnFVGTj6/7Uxk3qZj/L43mTzr2eTTPtyfsAAvArw9CPAxw42/t/vZ3wtCT4CPO76eNSfYlIVCUAWwWCyl7pZyJA8Pj1ItU7FixQpGjx7N0KFDAbNl6ODBg5VcnYiIVIScfCvLdiczb/MxftuZWGTsTavQuvylYxiDO4QREajln6r/N7dUmMjISNasWcPBgwfx9fW9YCtNixYtmDNnDoMHD8ZisTBx4kS16IiIVGNWm8GquBPM23yUBdsSSMvOt9/XpL4Pf4kJ4y8xYbQIqevAKqsfhSAn8uSTTzJq1CjatGlDVlYWH3/8cYnHvfnmm9xzzz307NmToKAgnn76adLS0qq4WhERuRjDMNgQf4p5m44xf2sCKek59vtC/Dy5sYMZfDo08q+VXVkVwWIYZRka5RzS0tLw9/cnNTUVPz+/IvdlZ2dz4MABoqKi8PLyclCFNZ/eRxFxVoZhkJqVR/zJTBJSs8nKs5KVayUz10pWnpXsvHO2z9mfVfizYDstO48z57T4BPi4M6h9QwZ3CKNbVCCuLs4XfC72/V0StQSJiIhUsNx8G8dOZxF/MpP4k5kcLvhZeDs3vFyOOh6uXNc2lL/EhNG7RRDurpoDuSwUgkRERMrBMAx2J55hb2J6kaBz6EQmx1OzsF2inyXEz5NQf298PV3xdnfF28MNb3eXc7Zd8fFwxcvD1b5t3nf2Z+NAH7wq6EphZ6QQJCIiUgbZeVZ+2HyMz1YdYuvR1Ase5+XuQuNAHxoH+hBR8LPw1qieD94eCi+OphAkIiJSCkdOZfLFmni+WhtvXx7Cw82FDuH+NK5fNOQ0ru9DsK+nBiRXcwpBIiIiF2AYBivjTvDpyoP8tjPR3sUVHuDNXVc2YXjXCALreDi2SCk3hSAREZHzpOfkM3fDET5ddYh9Sen2/b2a12dkj0j6t2qAmwYh13gKQSIiIgXiktP5fNUhZq8/QnqOeQVXHQ9XbunciLuvbKLJBmsZhSAREXFqVpvBkl1JfLrqIH/sTbHvbxpUh5E9mnBL50bU9XK/yBmkplIIEhGRGsVqM9h5PI3V+0+w9sBJNh85TZ7VwM3FYt5cXQp+WnB1ccHd1YKriwV3FxdcC/afe9zWo6kcOZUFgMUC/VuFMKpnE3o1C8LFCSccdCYKQVJqkZGRPP744zz++OOOLkVEnEhuvo2tR0+z5sBJ1h44yfqDpziTUzGTDRby93ZnRNcI7rqyiRYWdSIKQSIiUq1k51nZGH+aNQfMlp4N8aeKrIQOUNfLja6RgXSLCqRrZD3qermTZ7VhtRnk2wzyrQb5Nhv5VgOrzbDfl2czsNps5BXsz7fa8Pfx4NrWIZq3xwkpBImIiEOdyc5jQ/xp1pzXvXWuwDoedCsIPd2iAmnd0M8p18aSiqUQ5CT+/e9/88ILL3DkyBFcXM5e1jlkyBDq16/Pc889x/jx41m9ejUZGRm0bt2aqVOnEhsb68CqRaS2yc6zsvN4GpsPn2bLkVQ2HzlNXHJGseNC/DzpHlWfblGBdI8KpHkDX008KBVOIagiGAbkZVb987r7mKP4SmHYsGE88sgjLFmyhP79+wNw8uRJFixYwE8//UR6ejqDBg3in//8J56ennz22WcMHjyY3bt307hx48p8FSJSS1ltBnuTzrDlsBl2Nh85ze6EM8VaeQAiAr2LhJ7GgT4KPVLpFIIqQl4mvBxW9c/77DHwqFOqQ+vVq8f111/P//73P3sImj17NkFBQVx99dW4uLgQExNjP37KlCnMnTuXefPm8fDDD1dK+SJSexiGweGTWWbYKWjl2XYslcxca7Fj69fxoEMjf2IiAohpFECHRv7U9/V0QNXi7BSCnMidd97J2LFjef/99/H09OSLL75gxIgRuLi4kJ6ezgsvvMD8+fM5fvw4+fn5ZGVlER8f7+iyRaSaSkjN5vc9ySzbm8yquBOczMgtdkwdD1fahfvTMSKADgWBp1E9b7XySLWgEFQR3H3MVhlHPG8ZDB48GMMwmD9/Pl27duWPP/7g//7v/wB48sknWbhwIa+//jrNmzfH29ubW2+9ldzc4v+oiYhzys6zsvbASX7fk8zve5PZk5he5H53VwttGvrZw07HiACaBvtqALNUWwpBFcFiKXW3lCN5eXlx880388UXX7Bv3z6io6Pp1KkTACtWrGD06NEMHToUgPT0dA4ePOjAakXE0QzDIC45nWV7Uvh9TzJrDpwocqm6xQIdGgVwVYsg+rYMpn0jfzzddJm51BwKQU7mzjvv5MYbb2T79u3cdddd9v0tWrRgzpw5DB48GIvFwsSJE7HZbBc5k4jURqlZeazYZ4ae3/ckcyw1u8j9IX6e9G0RTN+WwfRuHkQ9raAuNZhCkJO55pprCAwMZPfu3dxxxx32/W+++Sb33HMPPXv2JCgoiKeffpq0tDQHVioiVSHpTLb96q2VcSfYGH8K2zkXb3m4udAtMpC+LYO4qmUDWoboUnWpPRSCnIyLiwvHjhUfvxQZGcnixYuL7HvooYeK/K7uMZGaLS07j21HUtl05DRbDqey5cjpYi09AM2C69C3pdnac2VUfc2kLLWWQpCISC1U2kkJLRZoHuxLTEQAnZvUo0+LIBrV09pZ4hwUgkREari07Dz2J2ewJ+EMm4+YoWdXQlqJkxI2qudNTKMAYiL86dAogHbh/vh66qtAnJM++SIiNYDNZnA8LZu4pHTikgtuSRnEJaeTdCanxMfUr+NBTIR5ubomJRQpTiFIRKQayc6zciAlo0jIiUtOZ39yBll5xWdfLhTi50mzYF/ah/vbg094gCYlFLkYhaByMozizcxSenr/RM6y2gx+3HKMD5ftZ2dCGhf66+HuaiGyfh2aBfvSrEHBz2BfmgbXoa6Xe9UWLVILKASVkbu7+Q9NZmYm3t7eDq6m5iqcidrVVVediPMqDD9vL9pbZNCyv7c7zRv40iz4bNBp1sCXiHreuLm6OLBikdrF4SHovffe41//+hcJCQnExMTwzjvv0K1btxKPzcvLY+rUqXz66accPXqU6OhoXn31VQYOHFjuc5aVq6srAQEBJCUlAeDjo5WOy8pms5GcnIyPjw9ubg7/CIpUOavN4IfNx3h78V72F4Qff2937usdxfBuEQT7eurfFZEq4NBvoFmzZjF+/HimT59O9+7dmTZtGgMGDGD37t00aNCg2PHPP/88//3vf/noo49o1aoVv/zyC0OHDmXlypVcccUV5TpneYSGhgLYg5CUnYuLC40bN9Y/9OJU8q02fthyjHcW7ysSfsb2iWJUz0h1aYlUMYvhwMEZ3bt3p2vXrrz77ruA2UIQERHBI488wjPPPFPs+LCwMJ577rkik/jdcssteHt789///rdc5yxJWloa/v7+pKam4ufnd8HjrFYreXl5pX69cpaHhwcuLmrWF+eQb7Uxb/Mx3l28j/0pZvgJ8HFnbJ+mjOzRROFHpIKU9vu7kMNagnJzc1m/fj0TJkyw73NxcSE2NpZVq1aV+JicnBy8vLyK7PP29mb58uXlPmfheXNyzl5iWtrlIlxdXTWmRUQuKN9q4/tNx3h3yT4OFISfej7u3NenKaN6Rmp+HhEHc9jfwJSUFKxWKyEhIUX2h4SEsGvXrhIfM2DAAN5880369u1Ls2bNWLRoEXPmzMFqtZb7nABTp07lH//4x2W+IhERU77VxnebjvHu4r0cPJEJmOFnbN+mjOyh8CNSXdSov4lvvfUWY8eOpVWrVlgsFpo1a8aYMWOYOXPmZZ13woQJjB8/3v57WloaERERl1uuiDiZPKuN7zYe5d0l+zhUEH4C63jYu73qKPyIVCsO+xsZFBSEq6sriYmJRfYnJibaBx6fLzg4mO+++47s7GxOnDhBWFgYzzzzDE2bNi33OQE8PT3x9NQsqiJSPqcycvnyz3g+X3WI4wULkgbW8eD+vk25+0qFH5HqymEjUz08POjcuTOLFi2y77PZbCxatIgePXpc9LFeXl6Eh4eTn5/Pt99+y5AhQy77nCIiZbU74QwT5myhxyuLeG3Bbo6nZhPk68GE61vxx9+v5oGrmikAiVRjDv3bOX78eEaNGkWXLl3o1q0b06ZNIyMjgzFjxgAwcuRIwsPDmTp1KgBr1qzh6NGjdOzYkaNHj/LCCy9gs9n4+9//XupziohcDpvNYMnuJGauOMCKfSfs+9uG+TGmVxSDYxri6aYLJkRqAoeGoOHDh5OcnMykSZNISEigY8eOLFiwwD6wOT4+vshl1NnZ2Tz//PPs378fX19fBg0axOeff05AQECpzykiUh5nsvOYvf4In648aB/s7GKBAW1DGdMriq6R9TTvlUgN49B5gqqrss4zICK116ETGXyy8iDfrDtCek4+AH5ebtzerTF392hCo3o+Dq5QRArVmHmCRESqK8MwWBV3gpkrDrBoV5J9QdNmwXUY3SuKWzqF4+Ohfz5Fajr9LRYRAXLzbexJPMOG+FN8sTqe3Yln7Pf1iw5mTK8o+jQPwsVFXV4itYVCkIg4new8K7sTzrDtWCrbjqay7WgauxPOkGu12Y/x8XDl1s6NGNUzkmbBvg6sVkQqi0KQiNRqWblWdhxPKwg7qWw7lsbexDPk24oPh/T3dqdduB9XRzdgWJcI/L21ppdIbaYQJCK1htVmsP1YKn8ePMX2o6lsPZpKXHI6JeQdAut40C7cn3ZhfrQP96dduD+N6nnrCi8RJ6IQJCI1lmEYxJ/M5I+9KazYl8LKuBOkZuUVOy64rqcZdML8zOAT7k9Dfy8FHhEnpxAkIjXKyYxcVuwzQ8/yfSkcOZVV5P66nm50iwqkQ6MA2jfyo12YPw38vBxUrYhUZwpBIlKtZeVa+fPgSXvo2X4srcj97q4WOjWuR+/mQfRqEUSHcH/cXB22IpCI1CAKQSJS7Ww7msqyPcms2JfCuoOnily1BdAqtC69mwfRu0UQ3aICNWePiJSL/uUQkWrBMAyW7knm/SX7+PPgqSL3hfl70btFEL2aB9GzWRDBdT0dVKWI1CYKQSLiUDabwa87Enh3yT62HTW7ujxcXegXHUyfguATFVRHg5hFpMIpBImIQ+Rbbfyw5RjvL4ljb1I6AN7urtx1ZWPG9mmqwcwiUukUgkSkSuXkW/l2/VGmL4sj/qS5GntdLzdG94xkTK8oAut4OLhCEXEWCkEiUiUyc/P5cu1hPvp9Pwlp2QDUr+PBPb2juLtHE/y8NDuziFQthSARqVRp2Xl8vuoQM5Yf4GRGLgChfl7c37cpt3drjLeHq4MrFBFnpRAkIpXiZEYuM5cf4NNVBzmTnQ9A40AfxvVrxs2dwvF0U/gREcdSCBKRCpNntbEq7gQ/bzvOdxuPkZVnBaBFA18euro5N3ZoqIkMRaTaUAgSkcuSnWdl+d4Uft6WwG87E4us3dU+3J+Hrm7OdW1CcHHRJe4iUr0oBIlImWXm5rN0dzI/b0tg8c5EMnKt9vuCfD24rm0oN3ZoSI+m9TW/j4hUWwpBIlIqadl5LNqZyIJtCSzbk0x23tmlLBr6ezGgbSjXtwulS2QgrjW11efwn7BiGjTsCH3Gg4vGLYnUZgpBInJBJzNyWbgjgZ+3JbBiXwp5VsN+X+NAH65vF8rAdqHENAqo2d1dJ+Jg0T9gx/fm77t+hIO/wy0zwLeBY2sTkUqjECQixazZf4J3Fu9j1f4TWG1ng0/zBr724NOmoV/N7+rKOAG/vwZ/zgBbHmCB1oNh3yI48Dt82BeGfQKNr3R0pSJSCRSCRMTuyKlMpv68i/lbjtv3tWnox/XtQrm+fSjNG9R1YHUVKC8LVn8Ay/8Pcsz1ymh+LVz7DwhpC8m7YdbdkLIbPrkBrn0RrnwQanroq64StoFhhYYxjq5EnIzFMAzj0oc5l7S0NPz9/UlNTcXPz8/R5YhUuqxcKx8si+PDZXHk5NtwscCIbo35a9+mNKlfx9HlVRybFbbMgsUvQdpRc19oB7huCjTtV/TYnHT44VHY9q35e+u/wJD3wEv/JlQImxV2zYfV70P8KsACo3+EyN6OrkxqsLJ+fysElUAhSJyFYRj8sOU4U3/ayfFUcymL7lGBTBrchrZh/g6uroLFLYZfJ0HiVvN3/wi4ZiK0HwYuF5i7yDBg7Ufwy7Nmd1lgMxj+udlaJOWTnQob/wtrpsPp+KL3+UfAuBXgVcs+e1JlFIIqgEKQOIOtR1L5xw/bWXfoFADhAd48d0Nrrm8XWvPH+pwrYSssnGSGIABPf/PKr+4PgHspV6o/sg6+HgVpR8DNGwZPg5gRlVZyrXRyP6z5txmAcs+Y+7wDocs9EHM7fHErnDoAHYbDzf92bK1SYykEVQCFIKnNks/k8Povu/l6/WEMA7zdXXmwXzPG9m2Kl7sDLwm35pmDkXd8Zw5MdnUHv3DwCyu4nbddp8GFW3AAUo+a3V6bvwQMcHGHbmOh71PgE1j2+jJOwJz7zoapzmNg4CulD1LOyDDg0Apz/NWu+UDB101wK7hyHLS/DTx8zH2H18LMAWDY4NaZ0O4Wh5UtNZdCUAVQCJLaKDffxqcrD/L2or2cyTHX8hrSMYxnrm9FQ39vxxSVnwsHlsH278zL0rNPl/6xLm5QN+ycYHROUDq+2Rxrkm928dH2Zug/EQKbXl69Nissew2WvQoY5nxCt30K9SIv77yF5z4RZ16S7x1w+edzpPwc2DbH/DNI2HJ2f/NYc4B5s2tKHmS++J/m1Xpe/jBuFfiHV13NjpCeBBkpZii8WKCXUlMIqgAKQVLbLN6VyJQfd3IgJQMwl7OYPLgNXSLL0SJyufJzIG6JOSfP7vnmGJFCdYLNS9RbDwZ3H3PwctqxglvBdupRSE8wWwwupXFPc9Bzoy4V+xr2/Qbf3gdZp8ArwOy+aTmgbOfISDG72Y78ad6ObjC7iQKbwUNrzJawmiYjBdbNhD//A+mJ5j43b7Pr8MpxEBx98cdb82DGdXBsA0RdBXd/V3vCQX6u2TVb+Od9ZO3ZMVF+4ebA+zZDIKJ77XnNDqAQVAEUgqS22JeUzpQfd7BsTzIAQb6e/H1gNLd2alS1kxvmZUPcooLg8/PZy9IBfEPML4C2N0HjHqWbpdmab37JnhuOzv1pcYWeD0P0oMq7rP30YfhmFBxdb/7e50m4+tmS67fmQeK2oqHn5P4Ln3vovyFmeOXUfSE7f4Sj68r/+LTjsH0uWHPM3+s2NLsfO48pW/djyl6Y3gfys2DAVOjxYPlrcqTUIwV/1gV/5sc2nX1v7Czg5mW+1kK+odDmL9DmJnN+Ks1aXiYKQRVAIUhquuw8K28u3MPM5QfItxm4u1q4p3cUD1/dnLpeVdTCkJcFexeawWfPAshNP3tf3Ybm/3rt//Otof/Q5+fAL8/Bnx+Zv0f1hVtmmnPeHPnTHOdyZB0c21j0i65QUEto1M1sqWrUFXb/BEv+CSHt4YE/qm5eouTd8F537GN2LkfYFXDlQ+afrZtH+c7x5wyYPx5cPeH+pRDS5vLrqkx5WWYX7OG1Z4PPmWPFj/MONP+cG3WFiK4Q1glcPWB/Qcvorp8g59yW0QZmq2jbm8xWTVdN7XcpCkEVQCFIarLNh08z/utNxCWbXV+xrRvw3A1tiAqqwvl+lk8zx87kZZzd5xdeEHxuMr8EalOT/5ZvzDmF8jLNLzVrbvFjvPwhvAtEFISe8M7gXa/oMZkn4f/ame/b3XPNsTNVYc5fYctX5mSFTXqV7xwubmbLW+MrLz+8GQb8bzjs/cUMhGMXgZvn5Z3zcuRlndMte6xoN21qPCTtBFt+0cdYXM2pFCK6nQ0+gU0v/t7k58D+ZQWB6Lwxcj5BZiBqMwQi+ygQXYBCUAVQCJKaKDffxruL9/Le0jisNoPgup68cnN7+rcOqdpCEraa3RkY4N/YbNpvO9T8X29tCj7nS9oFX98NKXvA4gIN2pxt4WnUDeo3L93r//lpcw6dZteYQaiynToIb3cyW6/GLoHwTpX/nKVxJhE+6AGZJ6DXY+as3ZXBZjO7JtOOFA84hduZJy59njoNigaesI7gcRn/8cjPNdevK7xoIOvU2fu8A6H1jWYgirqq5o0fs+bBmQRw94Y6QRV6aoWgCqAQJDXN7oQzjP96E9uPmWNtBseE8eJf2lKvTjm7Iy7Hf2+FfQvNFp9hnzjXUhP5OWbXUmAUeJZziZFTB+HtK8yB3w8sh9D2FVpiMT8+YQ5mrqrQVRa75sNXd1Bps0mfiDPHdSVsvfSx7j7nTNNw3nQNDVpDQOPK+6xb8+DgH2YL0c4fioYyrwBodaPZZRZ1Vfm7ICtKfg6cOX72IoZiY/aOFQyaN+Ca580pKyqQQlAFUAiSmsJqM/jPH/t549c95FptBPi489JN7bixQ5hjCjq43Fxry8UNHloL9Zs5po6a7psxsH0OdBgBN39Yec9zJgGmtTe770b/BJHl7AqrTN8/DBs/r/jZpHfMg+8fMgfpu3lBQJOiAcc/vGjY8QqoHoHemm/OvbTje9g5DzKSz97n6Q+tBpn/AWl2deV1IaYeMcc9nYwr3mp2bj0X4+IOPR+B2MkVWppCUAVQCJKa4NCJDJ78ZjN/HjSbya9p1YBXbm5PAz8HTd5nGPCfWPMKoy73wo1vOqaO2uDoBvjoajNMPral8ubL+eU5WPUuRFwJ9yyoHl/y58s5A9N7my1kFTGbtDUPfnvBfN1gXpF468fg1/ByK616Nqu57tr278xAVDgtAYCnH0Rfb3aZNetf/kk987LMK9vsl/ZfYND3udy8Sm4xO3fbJ6hSuscVgiqAQpBUZ4Zh8N818bw8fydZeVbqeLgyeXBbhnVp5NjlLnb+ALPuMrsNHt0Edat4LFJt88mNZhdIz0fgupcq/vznDsK+cza0uLbin6OiFJlN+mNod3P5zpN2HGbfA/Erzd97PAyxL9S8MTUlsVnh8BqzhWjH92aXVCEPX2g50AxEzWPPztJ9PsMww6Y98PxpdhWWNOg7tJ057q1YF2G4OSWCg/4tUgiqAApBUl0dT83i77O38MfeFACubBrIv26NISLwAv+oVRVrvjmINWWPOV9O/4mOrac22PML/O828KgL47dX/KKiS142Z74O7QB//b16tgKdyz6bdACMW1n21rEDf5gBKCPJfE9vet8ctF8b2WxmgNnxnRmI0o6evc+9DrS8zgxETXpB8q6C6RwKQk9mSvHz+YacHfBdEYO+K5FCUAVQCJLqxjAMvtt0lEnfb+dMdj6ebi48PbAVo3tGVu2khxey4TOY94h51cpjm7QKeEWw2cxgmbwLrp0CvR6tuHNnp8G0duZs3cM+NQfVVnflnU3aZoMV02DxFLMlqUFbGP6584xXs9nM92z7XHMcVGr8xY93cTenSiicy6hRV3M8VnUPyQXK+v2tiQZEqrkT6Tk8N3cbC7YnABATEcAbw2Jo3sDXwZUVyMuCJVPN7T5/UwCqKC4uZnfNvIfNBUi7P1BxV/6sm2kGoKCW5mzdNYGruzkeaHofc725NdMvPZt01imYOw72/Gz+HnMH3PDGhbuDaiMXl4KpGrqY3arHNpitQ9u/g9OHzIBjn8qhq9ky6ESLAisEiVRjv25P4Nm5W0lJz8XNxcLjsS144KpmuLlWo/l21v7bHCjp1wi63ufoamqXDreZLRhnjplXi8WMuPxz5mXBqvfM7d5P1Ky5m4JawICXYP7fzMHNTftdeDbp45th1t3mF72rJwx6DTqNqjEtGpXCYjEn6QzvDLH/MAedezl3b0cN+vSLOI/sPCvPzt3K/Z+vJyU9l+iQunz3UC8evqZF9QpAWafhj4KrwK5+1qn+B1kl3Dyh+1/N7ZXvmANXL9fG/5rjYgIaQ/thl3++qtblXmhxnbkO15z7zXlpzmUYsP5T+M+1ZgAKaAz3/gqdRzt3ADqfxeL0AQgUgkSqnfgTmdw6fSX/WxOPxQJ/vaop8x7pRbvwatjNtGKaObV/cOuKaaWQ4rrcYw5mTdwGcYsv71zWPFjxlrnd67GaeVWUxQJ/eRd86kPiVnOttUK5mebcPz88aoaklgPNQd9hHR1WrlRvCkEi1cjCHYnc8M4fbDuaxlDvjWxs/DYTWp/E060aLjCadhxWTze3+0+quYugVnfe9aDTSHN75TuXd64tsyD1sHm1T8e7Lr82R6kbAoPfNrdXvG1O0nkiDmZcC5u+MJct6T8JRnxZfH02kXMoBIlUA/lWG1N/3snYz9ZxJjufiUFLedN4nYDE1fDtvWa3U3Wz7BVzZfSIK81J2aTyXDnOnJtl/5LSLfFQEpsVlv+fud3j4Zrfddn6RrjiLsCA2ffCv/uZrWV1gmHk9+Yg/Zo03kkcQp8QEQdLTMvmjo/W8OGy/Viw8VXj77k3/d9YMMxukDPHYeEkR5dZVMpe2PC5uR37gsZaVLZ6Tc5exl7e1qAd38OJfeY8O13GVFRljjXwFagXCekJ5vIXEVea3V9RfR1dmdQQCkEiDrRyXwo3vP0Haw+eJNDTxspmn3Nl0izzzth/wJ3fmNsbPoX9yxxX6PkWTzFXHW85EJr0cHQ1zqHHw+bPbd+aazeVhWGcHcB+5bjyL+5a3XjWhVtnmnP/9HrcXGTVz0Hr5kmNpBAk4gA2m8G7i/dy14w1pKTn0rWBwcqwt2h49Bdw9YBbZkDvx80FLQsvO//hUcjNcGjdABxdb7YqYDHHXUjVCO8EkX3MJQxWf1C2x+791RxE7OEL3e6vnPocJbwzPLgSrv1HzRzoLQ6lECRSxU5l5HLPp3/y+q97sBnw1w4uzHJ/Aa/jf5oTDd49F9rfevYB/Sebc/CcOmgudeBIhmHOzwLm1WAhbR1ajtPpWTBr9PpPzckOS8Mw4PfXze0u95jrOokIoBAkUqU2xp/ixneWs3R3Mp5uLnwU68KEo4/gcmKvGXTu+QUiexd9kJcfDJ5mbq9+31zF2VHiFsOB383WqqufdVwdzqp5LAS3gtwzsP6T0j3m4HI4stacMLCwS01EgGoQgt577z0iIyPx8vKie/furF279qLHT5s2jejoaLy9vYmIiOCJJ54gOzvbfv8LL7yAxWIpcmvVqlVlvwyRizIMg09XHuS2D1dx9HQWkfV9WHhDFteuuQcykiG0Pdz3GzRoXfIJWlwLHUaYax99/1DxCeKqgs12thWo633mJHRStVxczFXlwZyeID/30o/5o6AVqNPd5qXlImLn0BA0a9Ysxo8fz+TJk9mwYQMxMTEMGDCApKSkEo//3//+xzPPPMPkyZPZuXMnM2bMYNasWTz7bNH/kbZt25bjx4/bb8uXL6+KlyNSovScfB7+ciOT520nz2pwfbtQfu69j8a/3gt5mdDsGhjzM/g1vPiJBk41L/9N3nV2kGtV2j4HEraYK3D3ebLqn19M7YeBb6i5lMa2by9+7JH1sH8puLiZkyOKSBEODUFvvvkmY8eOZcyYMbRp04bp06fj4+PDzJkzSzx+5cqV9OrVizvuuIPIyEiuu+46br/99mKtR25uboSGhtpvQUFBVfFyRIrZnXCGv7y7nPlbjuPmYmHSDa15P/RHvH950mzV6XgX3PF16a7W8QmEQf8yt/94HRK3V27x58rPhcUvmdu9HoU69avuuaWosiyl8ccb5s/2t6nlTqQEDgtBubm5rF+/ntjY2LPFuLgQGxvLqlWrSnxMz549Wb9+vT307N+/n59++olBgwYVOW7v3r2EhYXRtGlT7rzzTuLj4yvvhYhcwJ8HT3Lz+yvYn5xBQ38vvr6vM/ckTcWyvKAVp98EGPJu2a5oaXMTtLrRvELo+4fAml8ptRez4VM4dQDqNIArL7Fyt1S+LmPMOaSStl94KY3EHbB7PmCBPuOrtDyRmsJhISglJQWr1UpISNE+6pCQEBISEkp8zB133MGLL75I7969cXd3p1mzZvTr169Id1j37t355JNPWLBgAR988AEHDhygT58+nDlz5oK15OTkkJaWVuQmcjn+PHiS0TPXkpFr5cqmgfz01xg6/XEfbP3anPn3L+9Cv2fKPsmgxQI3vAGe/nBsI6wp46XS5ZGTDsteM7ev+jt4+lb+c8rFFVlK4+2SjykM222GmKuvi0gxDh8YXRZLly7l5Zdf5v3332fDhg3MmTOH+fPnM2XKFPsx119/PcOGDaNDhw4MGDCAn376idOnT/P1119f8LxTp07F39/ffouIiKiKlyO11LkBqHfzID65OYx6Xw02r6ry8IU7vzYHqZZX3VAYULBo5OKXzDWTKtPqD8xVx+tFQqdRlftcUnr2pTSWwvEtRe87EXd2vJBagUQuyGEhKCgoCFdXVxITE4vsT0xMJDQ0tMTHTJw4kbvvvpv77ruP9u3bM3ToUF5++WWmTp2KzWYr8TEBAQG0bNmSffv2XbCWCRMmkJqaar8dPny4/C9MnNq68wLQfwZ44fXpAEjaYQ5mHfOTeZnz5briLmjaD/KzYd6j5pVblSHjxNlVx6+ZCG4elfM8UnYXW0pjxVvmmLMW10HDmCovTaSmcFgI8vDwoHPnzixatMi+z2azsWjRInr0KHka/szMTFzOWxDP1dVcudq4wODA9PR04uLiaNjwwlfeeHp64ufnV+QmUlbrDp5kVEEA6tW8Ph8Nj8brfzeZa38FRcN9CyvuC8ligcFvgbsPHFpujtmpDH+8Yc5JE9oB2t5cOc8h5Vd4ufy5S2mkHoVN/zO3+/zNMXWJ1BAO7Q4bP348H330EZ9++ik7d+5k3LhxZGRkMGaMubjfyJEjmTBhgv34wYMH88EHH/DVV19x4MABFi5cyMSJExk8eLA9DD355JMsW7aMgwcPsnLlSoYOHYqrqyu33367Q16jOIfzA9B/RnbF+8CvkHUSAprAvb9U/NU59SLPLlvx60Tzy68inY6HPz8yt2Mna0Xu6ijsCnMpDcN6dimNVe+CLQ+a9IbGVzq2PpFqzs2RTz58+HCSk5OZNGkSCQkJdOzYkQULFtgHS8fHxxdp+Xn++eexWCw8//zzHD16lODgYAYPHsw///lP+zFHjhzh9ttv58SJEwQHB9O7d29Wr15NcHBwlb8+cQ7nBqCezQoCkIcrbJ9rHtBhuDmQtTJ0ux+2zTFnBP7xCbhjVsWt6L5kKlhzzS/ZZv0r5pxS8Xo9Bgf/MGeQ7nIPrPvY3N9XrUAil2IxLtSP5MTS0tLw9/cnNTVVXWNyUesPnWTkjLMBaMaoggCUnQr/am6GiHGrIKRN5RWRtAs+7GM+1y0ziq47Vh75ObBlljnWCAPuWwyNOldIqVIJDAPe7wHJO83WxtPxZgvR2CUVF4hFaoiyfn+rfVuknC4YgAB2/2yGkqDoCy+FUVEatIK+fze3f/47ZKSU7zzpybD0Ffi/tjDvEcCAtkMVgKo7i+Xs2KDTBXOi9XlSAUikFBSCRMrhogEIzC4qgHY3V82XUe/HIaQdZJ6An58u22MTt5sTL/5fW1g61VzLrG4YxL4AQ96vjGqlorW/1bz6ECC4NUQPuvjxIgI4eEyQSE20/tBJRs38k4xcKz2alhCAsk6dncW3zU1VU5Sruzn79EfXwLbZ5pdi9PUXPt5mg72/mqvSH1h2dn94Z3NG6DZDyjaTtTiWm6c5SP7np+G6KRrELlJKCkEiZVAYgNJz8unRtD4zR58XgAB2/WRendOgjdlVVVXCrjC7RVa8BT+OhyY9wcu/6DG5Gebl02umw4mCubMsLtD6L9DjIYjoVnX1SsW64k7zJiKlphAkUkrrD50qEoBmjO5SPADB2avC2g6t2gLBXI9s549wMg4WTjLnEgJzDpm1/zavIMpONfd5+kPnkeYVZlpcU0SckEKQSCmYAWhtkQDk41HCX5/Mk7B/ibntiBDk7g1/eQc+GWQGnpB2cGgl7PjenEsGILApdB8HHe/QOmAi4tQUgkQu4dwAdGXTwAsHIIBdP5orvIe0d9yilZG9oMu9sG4G/PTk2f1Rfc3xPi0GaMyIiAgKQSIXde4YoCubBjJzdNcLByA4pyvspiqp74JiXzAHPJ+Oh/a3wZUPQGh7x9YkIlLNKASJXMCS3UmM++96svNspQtAGSdgf8GVVo7oCjuXlx88sBxsVnV5iYhcgEKQSAnmbjzCU99sId9mcFXLYD64q9PFAxDAznnmuJuGMVC/WdUUejHu3o6uQESkWlMIEjnPjOUHmPLjDgBu6hjGv4bF4O5aijE0jrwqTEREykwhSKSAYRi8/utu3lsSB8CYXpFMvKENLi6lmPE5PdlcxBKqboJEERG5LApBIkC+1cbz323jqz8PA/DUgGge7NcMS2mXvNj5PRg2c8LCwKhKrFRERCqKQpA4vew8K499tZFftifiYoGXh7ZnRLcyTh64/TvzZ9ubK7w+ERGpHApBUrvlZcHBFeZA5RJaaNKy8xj76TrWHDiJh5sLb4+4goHtQsv2HGcS4eByc9vRl8aLiEipKQRJ7ZObAXsXwo7vYM+vkJcBdRrAI+uKrKWVfCaHUTPXsuN4Gr6ebnw0sgs9mtUv+/PtnAcYEN5Fy0+IiNQgCkFSO+Skw95fzOUh9i6EvMyz91lcICMJ/ngDrn0RgPgTmdw9cw2HTmQS5OvBJ2O60S7c/wInv4Rtc8yfuipMRKRGUQiSmis7Dfb8Yrb47PsN8rPP3hfQBNoMMbunMlLgf7fB6g+g82h2ZAcx6uO1JJ/JISLQm8/v6U5kUJ3y1ZB2DOJXmdvqChMRqVEUgqRmyU6F3QsKgs8isOacva9elBlE2txkTlhYeGWXYUCz/hC3iJPfTWB4/D2cyc6nVWhdPrunGw38vMpfz46CrrCI7uDfqPznERGRKqcQJNVffi5sm21egbV/CVhzz95Xv7kZetoMMdfGKumSdosFBvwT2wdLCYxfQJucrhiRvfhoVBf8vd0vrzZNkCgiUmMpBEn198OjsPnLs78HRRe0+AyBBm1KDj7n+Trel5z8a7jbdSGv1f2SkDGP4uV5mQEo9QgcXg1YzFpERKRGUQiS6i3jBGz9xtzu8yS0HwYNWpXpFB8ui2Pqz7uoxy3c6r6KJrn7YMfXcMVdl1fbju/Nn417gF/Y5Z1LRESqXCkWRBJxoO1zwJZvjvHpP7HMAejX7QlM/XkXALdd1RGv2GfMOxa9CDlnLrM2dYWJiNRkCkFSvRV2g3UYUeaHns7M5dm52wC4r3cUE65vjaXbXyGwKaQnwvJp5a/rdDwc+ROzK+wv5T+PiIg4jEKQVF8pe+HoerC4Qvtby/zwf/ywg5T0HJo38OXJAdHmTjcPuO4lc3vlO2aYKY/CZTKa9IK6ZZxhWkREqgWFIKm+tswyfza7BnwblOmhC3ckMnfjUVws8K9bO+Dl7nr2zuhBENnHvLz+txfKV1thV1g7dYWJiNRUCkFSPdlsZ0NQTNm6wsxusK0AjO3blCsa1yt6gMUCA6cCFtj2LcSvKVttpw7CsQ3mTNSt1RUmIlJTKQRJ9XR4tdlV5VEXWt1Qpoe++OMOks/k0Cy4Dk/Etiz5oND20Oluc3vBM2boKq3CVqDI3mVuoRIRkepDIUiqp81fmT/bDAF371I/bNHOROZsKOgGGxZTtBvsfNdMNEPWsQ1nL8MvDV0VJiJSKygESfWTl3124HHM8FI/LDUzz94Ndl+fpnQ6vxvsfL4NoM94c/u3F8zV5y/lRBwc32wO1lZXmIhIjaYQJNXPnp8hJxX8GkGT3qV+2JT5O0hMy6FpUB3GX3uBbrDzXfkgBDSGM8fMq8UuZcd35s+ovlAnqNS1iYhI9aMQJNXP5oIB0R2GgUvpPqJLdiUxe/0RLBb417AOF+8GO5e7F1z7orm9fBqkHr348dvUFSYiUluUKwQtWbKkousQMWWkwL6F5nYpJ0hMzcpjwhyzG+zeXlF0bhJYtudsc5O59EV+ljmT9IWk7IXEreDiBq0Hl+05RESk2ilXCBo4cCDNmjXjpZde4vDhwxVdkzizbYXLZHQs9RIZ/5y/g4S0bKKC6vC366LL/pznXjK/5Ss4sr7k4wrHKTXtBz5lDFoiIlLtlCsEHT16lIcffpjZs2fTtGlTBgwYwNdff01ubm5F1yfOZkvBVWGlnBto6e4kvl5X0A12awe8PUrZDXa+sCsg5nZz+5cJYBjFj9FVYSIitUq5QlBQUBBPPPEEmzZtYs2aNbRs2ZIHH3yQsLAwHn30UTZv3lzRdYozOHeZjHaXXiYjLftsN9g9vaLoEnmZrTP9J4G7DxxeYy7ceq7k3ZC0HVzcyzxvkYiIVE+XPTC6U6dOTJgwgYcffpj09HRmzpxJ586d6dOnD9u3b6+IGsVZFM4N1Lw/+AZf8vCX5+/keGo2kfV9eLI83WDn82sIvZ8wtxe+AHlZZ+8rbAVqdjV4X+LSexERqRHKHYLy8vKYPXs2gwYNokmTJvzyyy+8++67JCYmsm/fPpo0acKwYcMqslapzWw22PK1ud3h0nMD/b4nma/+PIzFAq/dGlP+brDz9XjYvDQ/NR5WvXd2v70r7OaKeR4REXG4coWgRx55hIYNG/LXv/6Vli1bsnHjRlatWsV9991HnTp1iIyM5PXXX2fXrl0VXa/UVvGrzOBRimUyzmTn8cy3WwAY3TOSblEVOEjZwwdiXzC3/3gTziRA0k5I3gWuHhB9fcU9l4iIOJRbeR60Y8cO3nnnHW6++WY8PT1LPCYoKEiX0kvpFQ6IbnvpZTJe/mknx1KzaVLfh6cGVEA32Pna3wprP4Qjf8LiKVA3zNzfrD94B1T884mIiEOUKwQtWrTo0id2c+Oqq64qz+nF2eRlw/bvze1LzA30+55kvlxrTsvw2i0d8PEo10f44iwWGDAVZsTCxi/Ozgytq8JERGqVcnWHTZ06lZkzZxbbP3PmTF599dXLLkqcTOEyGf4R0KTXBQ87c87VYKN7RtK9af3KqymiK7QfBhiQkQyunuoKExGpZcoVgj788ENatSo+kV3btm2ZPn36ZRclTqbwqrD2F18mY+rPuzh6OovGgT78fWAldIOdr/9kcPMyt1tcC15+lf+cIiJSZcoVghISEmjYsGGx/cHBwRw/fvyyixInkpEC+34zty8yQeLyvSn8b008AK9WVjfY+QIizEHSbt7Q/a+V/3wiIlKlyhWCIiIiWLFiRbH9K1asICws7LKLEiey7duzy2QEl9y6k56Tz9MFV4ON7NGEHs0qsRvsfFeOg+eOm6vGi4hIrVKu/06PHTuWxx9/nLy8PK655hrAHCz997//nb/97W8VWqDUcpsvvUzGqwXdYBGB3jw9sHTriVUoi6Xqn1NERCpduULQU089xYkTJ3jwwQft64V5eXnx9NNPM2HChAotUGqxlL1wbMNFl8n48+BJPl99CIBXb+5AHc8q6AYTERGnUK5vFIvFwquvvsrEiRPZuXMn3t7etGjR4oJzBomUyL5MRmyJy2Rk51nt3WAjukbQs3lQVVYnIiK13GX9t9rX15euXbtWVC3iTM5dJiOm5GUy3luyj/3JGQTX9WTCoNZVWJyIiDiDcoegdevW8fXXXxMfH2/vEis0Z86cCzxKpED8SnOZDE8/iB5U7O5dCWl8sDQOgClD2uLv7V7VFYqISC1XrqvDvvrqK3r27MnOnTuZO3cueXl5bN++ncWLF+Pv71/RNUptVNgV1uYvxZbJsNoMnv52K/k2gwFtQxjYrvh0DCIiIperXCHo5Zdf5v/+7//44Ycf8PDw4K233mLXrl3cdtttNG7cuEzneu+994iMjMTLy4vu3buzdu3aix4/bdo0oqOj8fb2JiIigieeeILs7OzLOqdUsbws2HHhZTI+WXmQzYdPU9fLjReHtKvi4kRExFmUKwTFxcVxww3mSt8eHh5kZGRgsVh44okn+Pe//13q88yaNYvx48czefJkNmzYQExMDAMGDCApKanE4//3v//xzDPPMHnyZHbu3MmMGTOYNWsWzz77bLnPKQ6w+2fISStxmYzDJzN5/ZfdADw7qDUhfl6OqFBERJxAuUJQvXr1OHPmDADh4eFs27YNgNOnT5OZmVnq87z55puMHTuWMWPG0KZNG6ZPn46Pj0+J65IBrFy5kl69enHHHXcQGRnJddddx+23316kpaes5xQH2DLL/NnhtiLLZBiGwbNzt5KVZ+XKpoGM6BrhoAJFRMQZlCsE9e3bl4ULFwIwbNgwHnvsMcaOHcvtt99O//79S3WO3Nxc1q9fT2xs7NliXFyIjY1l1apVJT6mZ8+erF+/3h569u/fz08//cSgQYPKfU6AnJwc0tLSitykkqQnn10m47yusLkbj/LH3hQ83FyYenMHLJqkUEREKlG5rg5799137eNwnnvuOdzd3Vm5ciW33HILzz//fKnOkZKSgtVqJSQkpMj+kJAQdu3aVeJj7rjjDlJSUujduzeGYZCfn88DDzxg7w4rzzkBpk6dyj/+8Y9S1S2XqXCZjLArILilfXdKeg4v/rgDgMdjWxAVVMdRFYqIiJMoc0tQfn4+P/74I66uruYJXFx45plnmDdvHm+88Qb16tWr8CILLV26lJdffpn333+fDRs2MGfOHObPn8+UKVMu67wTJkwgNTXVfjt8+HAFVSzFbCm4Kuy8VqAXf9jB6cw82jT0Y2yfpg4oTEREnE2ZW4Lc3Nx44IEH2Llz52U9cVBQEK6uriQmJhbZn5iYSGhoaImPmThxInfffTf33XcfAO3btycjI4P777+f5557rlznBPD09NRs11UheQ8c21iwTMYt9t2LdyUyb/MxXCzmCvHuruXqpRURESmTcn3bdOvWjU2bNl3WE3t4eNC5c2cWLVpk32ez2Vi0aBE9evQo8TGZmZm4uBQtubBFyjCMcp1TqtCW4stkpOfk89xcc2D92D5Nad9I80yJiEjVKNeYoAcffJDx48dz+PBhOnfuTJ06RcdvdOjQoVTnGT9+PKNGjaJLly5069aNadOmkZGRwZgxYwAYOXIk4eHhTJ06FYDBgwfz5ptvcsUVV9C9e3f27dvHxIkTGTx4sD0MXeqc4iBFlsk42xX2rwW7OJ6aTeNAHx6PbXmBB4uIiFS8coWgESPML7FHH33Uvs9isWAYBhaLBavVWqrzDB8+nOTkZCZNmkRCQgIdO3ZkwYIF9oHN8fHxRVp+nn/+eSwWC88//zxHjx4lODiYwYMH889//rPU5xQHiV8JqYcLlsm4HoD1h07yWcEK8VNvbo+3h6sjKxQRESdjMQzDKOuDDh06dNH7mzRpUu6CqoO0tDT8/f1JTU3Fz8/P0eXUDt8/DBs/hyvuhiHvkpNv5Ya3l7MvKZ3bujTitVtjHF2hiIjUcGX9/i5XS1BNDzlSxXIzzi6TUdAV9t6SOPYlpRPk68lzg9o4sDgREXFW5QpBn3322UXvHzlyZLmKkVro1EH4elTBMhmNoXFPdiec4YOl+wB4cUhb/H20QryIiFS9coWgxx57rMjveXl5ZGZm4uHhgY+Pj0KQmPb8AnPuh+zT4B0IQz/AioWnv91CntXg2jYhXN/uwlMXiIiIVKZyhaBTp04V27d3717GjRvHU089ddlFSQ1ns8KSl+GP183fwzvDsE8hIILPVhxg0+HT1PV0Y8qQdloaQ0REHKZcIagkLVq04JVXXuGuu+666BIVUsulJ8O398CB383fu46FAf8EN0+OnMrkXwUrxD8zqBWh/lohXkREHKfCQhCYs0kfO3asIk8pNUn8avhmNJw5Du514C9vQ/tbAXMyy+fmbiMz10q3yEBu79rYsbWKiIjTK1cImjdvXpHfDcPg+PHjvPvuu/Tq1atCCpMaxDBg9fuwcJK5OGpQS7jtc2jQyn7I95uOsWxPsrlC/C3tcXFRN5iIiDhWuULQTTfdVOR3i8VCcHAw11xzDW+88UZF1CU1RXYazHv47CXwbW82W4A869oPOZWRyz9+2A7AY/1b0CzY1xGVioiIFFGuEGSz2Sq6DqmJErfD1yPhxD5wcYcBL0O3sXDeYOf3l+7jVGYerULrcn9frRAvIiLVQ4WOCRInsvkr+OFxyM8Cv3Dz6q+IrsUOO56axaerzBnGJwxqrRXiRUSk2ijXN9Itt9zCq6++Wmz/a6+9xrBhwy67KKnG8rLN8DP3r2YAano1/PX3EgMQwNuL9pGbb6NbVCB9WwRVba0iIiIXUa4Q9PvvvzNo0KBi+6+//np+//33yy5KqqlTh2DmAFj/MWCBq56Bu76FOiWHmwMpGXy97jAAfx8QrTmBRESkWilXd1h6ejoeHh7F9ru7u5OWlnbZRUk1dOB3mHV3wezP9eDm/0CL2Is+5P8W7sFqM7imVQO6RAZWTZ0iIiKlVK6WoPbt2zNr1qxi+7/66ivatNFimLWOYcB3D5oBKKyT2f11iQC041ga8zabc0b97bqWVVCkiIhI2ZSrJWjixIncfPPNxMXFcc011wCwaNEivvzyS7755psKLVCqgaMbIPWwOQHi6Png4XPJh7zxqzkz9OCYMNqG+Vd2hSIiImVWrhA0ePBgvvvuO15++WVmz56Nt7c3HTp04LfffuOqq66q6BrF0XZ8Z/5seV2pAtD6QydZtCsJVxcLT8S2qNzaREREyqncl8jfcMMN3HDDDRVZi1RHhgE7C2YIbzOkFIcbvLbAbAUa1rkRTTUxooiIVFPlGhP0559/smbNmmL716xZw7p16y67KKlGErbCqYPg5gXNr73k4X/sTWHNgZN4uLnwaH+1AomISPVVrhD00EMPcfjw4WL7jx49ykMPPXTZRUk1UtgK1DwWPC/eqmMYhn2V+LuvbEJYgHdlVyciIlJu5QpBO3bsoFOnTsX2X3HFFezYseOyi5JqZEdBCGr9l0seumBbAluPplLHw5UH+zWr5MJEREQuT7lCkKenJ4mJicX2Hz9+HDc3rcRRayTtgpTd5rpgLQdc9FCrzeD1givC7u3TlPq+nlVRoYiISLmVKwRdd911TJgwgdTUVPu+06dP8+yzz3LttZceNyI1RGFXWLOrwTvgoofO2XCEuOQMAnzcGdsnqvJrExERuUzlarZ5/fXX6du3L02aNOGKK64AYNOmTYSEhPD5559XaIHiQKXsCsvJtzLtt70APNivGXW93Cu7MhERkctWrhAUHh7Oli1b+OKLL9i8eTPe3t6MGTOG22+/HXd3fQHWCif3Q+JWsLhCq4tPhfDlmniOns4ixM+TkT0iq6Y+ERGRy1TuATx16tShd+/eNG7cmNzcXAB+/vlnAP7yl0sPopVqrrAVKLI3+Fx43a/M3HzeXbIPgEf7t8DL3bUqqhMREbls5QpB+/fvZ+jQoWzduhWLxYJhGEVWCLdarRVWoDiIfYLEiwfaj1ccJCU9lyb1fbitS0QVFCYiIlIxyjUw+rHHHiMqKoqkpCR8fHzYtm0by5Yto0uXLixdurSCS5Qqd/owHF0PWKDV4AselpqZx4fL4gAYf21L3F3L9XESERFxiHK1BK1atYrFixcTFBSEi4sLrq6u9O7dm6lTp/Loo4+ycePGiq5TqtLOH8yfjXtA3ZALHvbh73GkZefTKrQugzuEVVFxIiIiFaNc/3W3Wq3UrVsXgKCgII4dOwZAkyZN2L17d8VVJ45Riq6wpDPZfLziIABPXheNi4vlgseKiIhUR+VqCWrXrh2bN28mKiqK7t2789prr+Hh4cG///1vmjZtWtE1SlU6kwjxq83t1hfuCntv8T6y8qxc0TiA/q0bVFFxIiIiFadcIej5558nIyMDgBdffJEbb7yRPn36UL9+fWbNmlWhBUoV2/UDYEB4Z/BvVOIhh09m8r+18QA8NSC6yKB4ERGRmqJcIWjAgLNLKDRv3pxdu3Zx8uRJ6tWrpy/Emq4UEyRO+20veVaDPi2C6NksqIoKExERqVgVttBXYOCF55KRGiLjBBxcbm5fYDzQ3sQzzN14BDDHAomIiNRUuqZZzto9HwwrhLaHwJLHdr3x6x5sBgxsG0pMREDV1iciIlKBFILkLHtX2JAS7958+DQLtidgscDfrmtZhYWJiIhUPIUgMWWdhv1Lze0LdIW9/qs5/cHQK8JpEVK3auoSERGpJApBYtrzC9jyICgagouP9VkZl8Ife1Nwd7XwRKxagUREpOZTCBLTJSZIfOu3vQDc3q0xEYE+VVWViIhIpVEIEshJh32/mdttio8HOpiSwZoDJ3GxwLh+zaq4OBERkcqhECSw91fIz4Z6URDSrtjdczYeBaB3i2Aa+ntXdXUiIiKVQiFIinaFnTfZpc1mMGeDOS/QLZ3Cq7oyERGRSqMQ5OzysmDPr+Z2CZfGrz14kiOnsqjr6caAtqFVXJyIiEjlUQhydnGLIS8D/BpBeKdidxe2Ag1q3xAvd9eqrk5ERKTSKAQ5O/sEiYOLdYVl5Vr5aWsCALd0LnkxVRERkZpKIciZ5efC7p/N7RKuCvtlewLpOflEBHrTNbJeFRcnIiJSuRSCnNmBZZCTCr4hENG92N3fFnSF3XxFIyzntRKJiIjUdApBzmzH9+bPVjeCS9GPQkJqNsv3pQBwSyd1hYmISO2jEOSsrPmwa765XcIs0XM3HsUwoFtkII3ra4ZoERGpfRSCnNWhFZB1ErwDoUnvIncZhnG2K0xzA4mISC2lEOSsCidIbHUDuLoVuWvr0VT2JaXj6ebCoA4NHVCciIhI5VMIckY2G+z8wdwu4aqwb9ebrUAD2obi5+VelZWJiIhUmWoRgt577z0iIyPx8vKie/furF279oLH9uvXD4vFUux2ww032I8ZPXp0sfsHDhxYFS+lZji8BtITwdMfoq4qclduvo15m48B6goTEZHaze3Sh1SuWbNmMX78eKZPn0737t2ZNm0aAwYMYPfu3TRo0KDY8XPmzCE3N9f++4kTJ4iJiWHYsGFFjhs4cCAff/yx/XdPT8/KexE1TWFXWPRAcPMoctfiXUmcysyjQV1P+rQIdkBxIiIiVcPhLUFvvvkmY8eOZcyYMbRp04bp06fj4+PDzJkzSzw+MDCQ0NBQ+23hwoX4+PgUC0Genp5FjqtXT5P9AWAYZ7vCWhe/KqxwmYyhV4Tj6qK5gUREpPZyaAjKzc1l/fr1xMbG2ve5uLgQGxvLqlWrSnWOGTNmMGLECOrUqVNk/9KlS2nQoAHR0dGMGzeOEydOXPAcOTk5pKWlFbnVWsc2QOphcK8DzfsXuetkRi5LdicBcLPmBhIRkVrOoSEoJSUFq9VKSEhIkf0hISEkJCRc8vFr165l27Zt3HfffUX2Dxw4kM8++4xFixbx6quvsmzZMq6//nqsVmuJ55k6dSr+/v72W0RERPlfVHVXuFZYy+vA3bvIXfM2HSXPatAu3I/o0LoOKE5ERKTqOHxM0OWYMWMG7du3p1u3bkX2jxgxwr7dvn17OnToQLNmzVi6dCn9+/c//zRMmDCB8ePH239PS0urnUHIMM6OByqpK2zjUUAzRIuIiHNwaEtQUFAQrq6uJCYmFtmfmJhIaGjoRR+bkZHBV199xb333nvJ52natClBQUHs27evxPs9PT3x8/MrcquVErfByf3g5gUtrity197EM2w5koqbi4W/xIQ5qEAREZGq49AQ5OHhQefOnVm0aJF9n81mY9GiRfTo0eOij/3mm2/IycnhrrvuuuTzHDlyhBMnTtCwoZNP/FfYFdasP3j6FrlrdsGA6H7RDajvqyvpRESk9nP41WHjx4/no48+4tNPP2Xnzp2MGzeOjIwMxowZA8DIkSOZMGFCscfNmDGDm266ifr16xfZn56ezlNPPcXq1as5ePAgixYtYsiQITRv3pwBAwZUyWuqtgq7ws5bK8xqM/iuoCvs1s6aG0hERJyDw8cEDR8+nOTkZCZNmkRCQgIdO3ZkwYIF9sHS8fHxuJy3wvnu3btZvnw5v/76a7Hzubq6smXLFj799FNOnz5NWFgY1113HVOmTHHuuYKS90DyLnBxh5ZFJ45csS+FxLQcAnzcubpV8bmZREREaiOHhyCAhx9+mIcffrjE+5YuXVpsX3R0NIZhlHi8t7c3v/zyS0WWVzvs+838GdUHvAOK3FW4WOrgDmF4urlWcWEiIiKO4fDuMKkiRwqWImnSs8juM9l5/LLdnI7gls66KkxERJyHQpCzOLLO/Nmo6HQCP29NIDvPRrPgOsQ08ndAYSIiIo6hEOQM0o6bs0RbXCC8c5G7Cq8Ku7lTIywWLZMhIiLOQyHIGRR2hTVoW+TS+MMnM1l74CQWi1aMFxER56MQ5AwOF4SgiK5Fds/ZYF4W36tZEA39vc9/lIiISK2mEOQMjvxp/mx0NgQZhsGcjYVdYWoFEhER56MQVNvl58KxTeb2OYOi1x06xaETmdTxcGVgu4svUSIiIlIbKQTVdglbwZoD3oFQv5l995yCAdHXt2+Ij0e1mC5KRESkSikE1XaFg6IbdYWCq7+y86z8uPk4oK4wERFxXgpBtV0Jg6J/3ZHImZx8wgO8uTKq/gUeKCIiUrspBNV2JQyK/nb92QHRLi6aG0hERJyTQlBtVsIkiUlp2fyxNxkwJ0gUERFxVgpBtVlhK1CDNuBZF4DvNh3FZkCnxgFEBdVxYHEiIiKOpRBUm507KBpzbqBv15sTJGqxVBERcXYKQbXZ4YKWoAhzfqDtx9LYnXgGDzcXbmwf5sDCREREHE8hqLbKz4VjG83tgpagbwvmBrq2dQj+Pu6OqkxERKRaUAiqrRILJ0msB/Wbk2e1MW/TMQBu6ay5gURERBSCaqvD51wab7GwbHcyJzJyCfL1oG+LYMfWJiIiUg0oBNVW9kHR5nigH7aYrUCDY8Jwc9Ufu4iIiL4Nayv7oOiuZOVaWbgjETBDkIiIiCgE1U5nEiA1HrBAWCeW7E4iM9dKeIA3V0QEOLo6ERGRakEhqDY6d5JELz9+LOgKuzGmIRaLlskQEREBhaDa6ZxFU9Nz8lm0MwmAwR3UFSYiIlJIIag2si+a2o1FOxPJybcRWd+HtmF+jq1LRESkGlEIqm3OmyTxh83HAXNAtLrCREREzlIIqm0St0J+NngFkFqnCcv2mF1hN6orTEREpAiFoNrmyDrzZ6Ou/LojiTyrQcsQX6JD6zq2LhERkWpGIai2sQ+K7sYPW8yuMLUCiYiIFKcQVNsUzBSdFnQFK/alAHBjh4aOrEhERKRaUgiqTc4kwmlzksQFp8Kx2gzahvnRNNjX0ZWJiIhUOwpBtUnhemENWjN3RxqgrjAREZELUQiqTQrmB8oK6cTqAycAdYWJiIhciEJQbVKwaOp6a3MMAzpGBBAR6OPgokRERKonhaDawppnnyTx60SzC0ytQCIiIhemEFRbJGyF/Cxsnv78cNQHi0XjgURERC5GIai2KBgPdLROWwxc6NokkFB/LwcXJSIiUn0pBNUWBSHoj+woAAbHqCtMRETkYhSCaouCmaJ/Ot0YFwsMbKcQJCIicjEKQbVBehKcPoSBhc22ZvRsFkRwXU9HVyUiIlKtKQTVBgWtQIdcIjiDj64KExERKQWFoNqgYKboVbnNcHOxMLBdqIMLEhERqf4UgmqDI+sA2GC0oE+LIAJ8PBxckIiISPWnEFTTWfMwjm4AYIOtheYGEhERKSWFoJoucRuW/CxSDR+OuIZzbdsQR1ckIiJSIygE1XQF64VtsjXnqpYh+Hm5O7ggERGRmkEhqIYzCgZFb7C14MYYdYWJiIiUlkJQDZd7cA0A21yiiW3dwMHViIiI1BwKQTVZejKeZ+KxGRYCWvbAx8PN0RWJiIjUGApBNZjtsNkKtM8I49orWjq4GhERkZpFIagGS9rxBwBbLdH0iw52cDUiIiI1S7UIQe+99x6RkZF4eXnRvXt31q5de8Fj+/Xrh8ViKXa74YYb7McYhsGkSZNo2LAh3t7exMbGsnfv3qp4KVUq+8BqAPLCuuDl7urgakRERGoWh4egWbNmMX78eCZPnsyGDRuIiYlhwIABJCUllXj8nDlzOH78uP22bds2XF1dGTZsmP2Y1157jbfffpvp06ezZs0a6tSpw4ABA8jOzq6ql1XprPl5hKbvACCyYz/HFiMiIlIDOTwEvfnmm4wdO5YxY8bQpk0bpk+fjo+PDzNnzizx+MDAQEJDQ+23hQsX4uPjYw9BhmEwbdo0nn/+eYYMGUKHDh347LPPOHbsGN99910VvrLKtX3DSrzIJQ0fOnXq7uhyREREahyHhqDc3FzWr19PbGysfZ+LiwuxsbGsWrWqVOeYMWMGI0aMoE6dOgAcOHCAhISEIuf09/ene/fupT5nTXBw8xIAEnzb4uGuq8JERETKyqHfnikpKVitVkJCii71EBISwq5duy75+LVr17Jt2zZmzJhh35eQkGA/x/nnLLzvfDk5OeTk5Nh/T0tLK/VrcIQ8qw23Y+aiqZ5RVzq4GhERkZrJ4d1hl2PGjBm0b9+ebt26XdZ5pk6dir+/v/0WERFRQRVWjpVxJ2hr3Q1Ao/ZXObgaERGRmsmhISgoKAhXV1cSExOL7E9MTCQ0NPSij83IyOCrr77i3nvvLbK/8HFlOeeECRNITU213w4fPlzWl1KllqzfThMXc+C4a0RXB1cjIiJSMzk0BHl4eNC5c2cWLVpk32ez2Vi0aBE9evS46GO/+eYbcnJyuOuuu4rsj4qKIjQ0tMg509LSWLNmzQXP6enpiZ+fX5FbdZWTb+Xk7hUAZPq3AO8AxxYkIiJSQzl8RO348eMZNWoUXbp0oVu3bkybNo2MjAzGjBkDwMiRIwkPD2fq1KlFHjdjxgxuuukm6tevX2S/xWLh8ccf56WXXqJFixZERUUxceJEwsLCuOmmm6rqZVWaP/ak0Cp/F7iBV5SuChMRESkvh4eg4cOHk5yczKRJk0hISKBjx44sWLDAPrA5Pj4eF5eiDVa7d+9m+fLl/PrrryWe8+9//zsZGRncf//9nD59mt69e7NgwQK8vLwq/fVUth+3HGOEiznxo0vE5Y2FEhERcWYWwzAMRxdR3aSlpeHv709qamq16hrLzrPSbcoCVlvG4GPJgQdXQ4PWji5LRESkWijr93eNvjrM2SzZlURE3kF8LDkYnn4QFO3okkRERGoshaAa5Mctx7mioCvM0qgLuOiPT0REpLz0LVpDpGXn8dvORDoVhCAa6dJ4ERGRy6EQVEPM33KcnHwb3d32mTsaaVC0iIjI5VAIqiFmrz9CfVIJNwqW/mjU2bEFiYiI1HAKQTXA/uR01h86xRWuBa1AQdHgXc+xRYmIiNRwCkE1wLcbjgBwU/2j5g4tlSEiInLZFIKqOavNYM4GM/z08Nhv7tSgaBERkcumEFTNrYxL4XhqNtFepwk8udHcGXGlY4sSERGpBRSCqrnZ682usJfq/4LFlgdRfaFBKwdXJSIiUvMpBFVjadl5LNiWQCNLMl1Ozjd39nvWsUWJiIjUEgpB1dhPBXMDPef7IxYjH5peDU16OLosERGRWkEhqBqbvf4IEZZEBuQvNndcrVYgERGRiqIQVE0dSMlg3aFTPOr2HS6GFZrHQoRmiRYREakoCkHV1Lfrj9DEksDNrn+YOzQWSEREpEIpBFVDVpvBtxuO8KjbXFyxQYvrtEyGiIhIBVMIqoZWxZ3AO20/N7kuN3f0m+DYgkRERGohhaBqaPb6wzziNhdXDGh5PYR3cnRJIiIitY5CUDWTlp3Hnu3r+YvLSnPH1WoFEhERqQwKQdXMT1uO8wCzcbUYGK1ugIYxji5JRESkVlIIqmZWr1nBjS6rAbBoLJCIiEilUQiqRg6kZBCb9DEuFoPsFjdCaHtHlyQiIlJrKQRVI8t+X8qNrmsA8IrVvEAiIiKVSSGomrDaDBpvexuAY+EDIaStgysSERGp3RSCqoktf/7BNbbV2LBQ/4aJji5HRESk1lMIqiZc/3gVgG0B1+AZ1s7B1YiIiNR+CkHVQPrBdXRIX47VsODeX2OBREREqoJCUDWQ+tMUAJZ49KVVO60RJiIiUhUUghzt6AbCk5ZiNSwkd3oci8Xi6IpEREScgkKQg2X++hIA39l6c02vng6uRkRExHkoBDnSkXX4HFpEvuHC6kb3EuLn5eiKREREnIZCkAMZS6YCMMfah75XdndwNSIiIs5FIchRDq/FEvcbeYYrH7vdyrVtQhxdkYiIiFNRCHKUJS8D8K21D51irsDL3dXBBYmIiDgXhSBHOLQK9i8hz3DlXetQbu3cyNEViYiIOB2FIEdYarYCfWO9Cs+gSDpGBDi2HhERESekEFTVDi6HA7+Tjxvv5t/ErZ0jNDeQiIiIAygEVbWCK8K+zO9HgiWIoVeEO7ggERER56QQVJUO/A6HlpNvcee9/CH0aRFMqL/mBhIREXEEhaCqtHU2AHMtsSRQXwOiRUREHMjN0QU4lcFvsbNuD/71i5W6Xm6aG0hERMSB1BJUlSwW/p3UmiTq8ZeYMM0NJCIi4kAKQVXoTHYeP287DqCuMBEREQdTCKpCP209TnaejabBdTQ3kIiIiIMpBFWhExm5eLu7cmvnRpobSERExME0MLoKPdivOSN7RGIYhqNLERERcXoKQVXM11NvuYiISHWg7jARERFxSgpBIiIi4pQUgkRERMQpKQSJiIiIU3J4CHrvvfeIjIzEy8uL7t27s3bt2osef/r0aR566CEaNmyIp6cnLVu25KeffrLf/8ILL2CxWIrcWrVqVdkvQ0RERGoYh16qNGvWLMaPH8/06dPp3r0706ZNY8CAAezevZsGDRoUOz43N5drr72WBg0aMHv2bMLDwzl06BABAQFFjmvbti2//fab/Xc3N12RJSIiIkU5NB28+eabjB07ljFjxgAwffp05s+fz8yZM3nmmWeKHT9z5kxOnjzJypUrcXd3ByAyMrLYcW5uboSGhlZq7SIiIlKzOaw7LDc3l/Xr1xMbG3u2GBcXYmNjWbVqVYmPmTdvHj169OChhx4iJCSEdu3a8fLLL2O1Wosct3fvXsLCwmjatCl33nkn8fHxF60lJyeHtLS0IjcRERGp3RwWglJSUrBarYSEhBTZHxISQkJCQomP2b9/P7Nnz8ZqtfLTTz8xceJE3njjDV566SX7Md27d+eTTz5hwYIFfPDBBxw4cIA+ffpw5syZC9YydepU/P397beIiIiKeZEiIiJSbdWowTI2m40GDRrw73//G1dXVzp37szRo0f517/+xeTJkwG4/vrr7cd36NCB7t2706RJE77++mvuvffeEs87YcIExo8fb/89LS1NQUhERKSWc1gICgoKwtXVlcTExCL7ExMTLziep2HDhri7u+Pq6mrf17p1axISEsjNzcXDw6PYYwICAmjZsiX79u27YC2enp54enqW85WIiIhITeSw7jAPDw86d+7MokWL7PtsNhuLFi2iR48eJT6mV69e7Nu3D5vNZt+3Z88eGjZsWGIAAkhPTycuLo6GDRtW7AsQERGRGs2h8wSNHz+ejz76iE8//ZSdO3cybtw4MjIy7FeLjRw5kgkTJtiPHzduHCdPnuSxxx5jz549zJ8/n5dffpmHHnrIfsyTTz7JsmXLOHjwICtXrmTo0KG4urpy++23V/nrExERkerLoWOChg8fTnJyMpMmTSIhIYGOHTuyYMEC+2Dp+Ph4XFzO5rSIiAh++eUXnnjiCTp06EB4eDiPPfYYTz/9tP2YI0eOcPvtt3PixAmCg4Pp3bs3q1evJjg4uNR1GYYBoKvEREREapDC7+3C7/FLsRilPdKJHDlyRAOjRUREaqjDhw/TqFGjSx6nEFQCm83GsWPHqFu3LhaLpULPXXjl2eHDh/Hz86vQc9dWes/KR+9b+eh9Kx+9b2Wn96x8Lva+GYbBmTNnCAsLK9KTdCE16hL5quLi4lKqBHk5/Pz89KEvI71n5aP3rXz0vpWP3rey03tWPhd63/z9/Ut9DocvoCoiIiLiCApBIiIi4pQUgqqYp6cnkydP1uSMZaD3rHz0vpWP3rfy0ftWdnrPyqci3zcNjBYRERGnpJYgERERcUoKQSIiIuKUFIJERETEKSkEiYiIiFNSCKpC7733HpGRkXh5edG9e3fWrl3r6JKqtRdeeAGLxVLk1qpVK0eXVe38/vvvDB48mLCwMCwWC999912R+w3DYNKkSTRs2BBvb29iY2PZu3evY4qtRi71vo0ePbrY52/gwIGOKbaamDp1Kl27dqVu3bo0aNCAm266id27dxc5Jjs7m4ceeoj69evj6+vLLbfcQmJiooMqrh5K877169ev2OftgQcecFDFjvfBBx/QoUMH+4SIPXr04Oeff7bfX1GfM4WgKjJr1izGjx/P5MmT2bBhAzExMQwYMICkpCRHl1attW3bluPHj9tvy5cvd3RJ1U5GRgYxMTG89957Jd7/2muv8fbbbzN9+nTWrFlDnTp1GDBgANnZ2VVcafVyqfcNYODAgUU+f19++WUVVlj9LFu2jIceeojVq1ezcOFC8vLyuO6668jIyLAf88QTT/DDDz/wzTffsGzZMo4dO8bNN9/swKodrzTvG8DYsWOLfN5ee+01B1XseI0aNeKVV15h/fr1rFu3jmuuuYYhQ4awfft2oAI/Z4ZUiW7duhkPPfSQ/Xer1WqEhYUZU6dOdWBV1dvkyZONmJgYR5dRowDG3Llz7b/bbDYjNDTU+Ne//mXfd/r0acPT09P48ssvHVBh9XT++2YYhjFq1ChjyJAhDqmnpkhKSjIAY9myZYZhmJ8td3d345tvvrEfs3PnTgMwVq1a5agyq53z3zfDMIyrrrrKeOyxxxxXVA1Qr1494z//+U+Ffs7UElQFcnNzWb9+PbGxsfZ9Li4uxMbGsmrVKgdWVv3t3buXsLAwmjZtyp133kl8fLyjS6pRDhw4QEJCQpHPnr+/P927d9dnrxSWLl1KgwYNiI6OZty4cZw4ccLRJVUrqampAAQGBgKwfv168vLyinzeWrVqRePGjfV5O8f571uhL774gqCgINq1a8eECRPIzMx0RHnVjtVq5auvviIjI4MePXpU6OdMC6hWgZSUFKxWKyEhIUX2h4SEsGvXLgdVVf11796dTz75hOjoaI4fP84//vEP+vTpw7Zt26hbt66jy6sREhISAEr87BXeJyUbOHAgN998M1FRUcTFxfHss89y/fXXs2rVKlxdXR1dnsPZbDYef/xxevXqRbt27QDz8+bh4UFAQECRY/V5O6uk9w3gjjvuoEmTJoSFhbFlyxaefvppdu/ezZw5cxxYrWNt3bqVHj16kJ2dja+vL3PnzqVNmzZs2rSpwj5nCkFSbV1//fX27Q4dOtC9e3eaNGnC119/zb333uvAysQZjBgxwr7dvn17OnToQLNmzVi6dCn9+/d3YGXVw0MPPcS2bds0Tq+MLvS+3X///fbt9u3b07BhQ/r3709cXBzNmjWr6jKrhejoaDZt2kRqaiqzZ89m1KhRLFu2rEKfQ91hVSAoKAhXV9diI9cTExMJDQ11UFU1T0BAAC1btmTfvn2OLqXGKPx86bN3+Zo2bUpQUJA+f8DDDz/Mjz/+yJIlS2jUqJF9f2hoKLm5uZw+fbrI8fq8mS70vpWke/fuAE79efPw8KB58+Z07tyZqVOnEhMTw1tvvVWhnzOFoCrg4eFB586dWbRokX2fzWZj0aJF9OjRw4GV1Szp6enExcXRsGFDR5dSY0RFRREaGlrks5eWlsaaNWv02SujI0eOcOLECaf+/BmGwcMPP8zcuXNZvHgxUVFRRe7v3Lkz7u7uRT5vu3fvJj4+3qk/b5d630qyadMmAKf+vJ3PZrORk5NTsZ+zih27LRfy1VdfGZ6ensYnn3xi7Nixw7j//vuNgIAAIyEhwdGlVVt/+9vfjKVLlxoHDhwwVqxYYcTGxhpBQUFGUlKSo0urVs6cOWNs3LjR2LhxowEYb775prFx40bj0KFDhmEYxiuvvGIEBAQY33//vbFlyxZjyJAhRlRUlJGVleXgyh3rYu/bmTNnjCeffNJYtWqVceDAAeO3334zOnXqZLRo0cLIzs52dOkOM27cOMPf399YunSpcfz4cfstMzPTfswDDzxgNG7c2Fi8eLGxbt06o0ePHkaPHj0cWLXjXep927dvn/Hiiy8a69atMw4cOGB8//33RtOmTY2+ffs6uHLHeeaZZ4xly5YZBw4cMLZs2WI888wzhsViMX799VfDMCruc6YQVIXeeecdo3HjxoaHh4fRrVs3Y/Xq1Y4uqVobPny40bBhQ8PDw8MIDw83hg8fbuzbt8/RZVU7S5YsMYBit1GjRhmGYV4mP3HiRCMkJMTw9PQ0+vfvb+zevduxRVcDF3vfMjMzjeuuu84IDg423N3djSZNmhhjx451+v+0lPR+AcbHH39sPyYrK8t48MEHjXr16hk+Pj7G0KFDjePHjzuu6GrgUu9bfHy80bdvXyMwMNDw9PQ0mjdvbjz11FNGamqqYwt3oHvuucdo0qSJ4eHhYQQHBxv9+/e3ByDDqLjPmcUwDKOcLVMiIiIiNZbGBImIiIhTUggSERERp6QQJCIiIk5JIUhERESckkKQiIiIOCWFIBEREXFKCkEiIiLilBSCRERKYenSpVgslmLrFYlIzaUQJCIiIk5JIUhERESckkKQiNQINpuNqVOnEhUVhbe3NzExMcyePRs421U1f/58OnTogJeXF1deeSXbtm0rco5vv/2Wtm3b4unpSWRkJG+88UaR+3Nycnj66aeJiIjA09OT5s2bM2PGjCLHrF+/ni5duuDj40PPnj3ZvXt35b5wEak0CkEiUiNMnTqVzz77jOnTp7N9+3aeeOIJ7rrrLpYtW2Y/5qmnnuKNN97gzz//JDg4mMGDB5OXlweY4eW2225jxIgRbN26lRdeeIGJEyfyySef2B8/cuRIvvzyS95++2127tzJhx9+iK+vb5E6nnvuOd544w3WrVuHm5sb99xzT5W8fhGpeFpAVUSqvZycHAIDA/ntt9/o0aOHff99991HZmYm999/P1dffTVfffUVw4cPB+DkyZM0atSITz75hNtuu40777yT5ORkfv31V/vj//73vzN//ny2b9/Onj17iI6OZuHChcTGxharYenSpVx99dX89ttv9O/fH4CffvqJG264gaysLLy8vCr5XRCRiqaWIBGp9vbt20dmZibXXnstvr6+9ttnn31GXFyc/bhzA1JgYCDR0dHs3LkTgJ07d9KrV68i5+3Vqxd79+7FarWyadMmXF1dueqqqy5aS4cOHezbDRs2BCApKemyX6OIVD03RxcgInIp6enpAMyfP5/w8PAi93l6ehYJQuXl7e1dquPc3d3t2xaLBTDHK4lIzaOWIBGp9tq0aYOnpyfx8fE0b968yC0iIsJ+3OrVq+3bp06dYs+ePbRu3RqA1q1bs2LFiiLnXbFiBS1btsTV1ZX27dtjs9mKjDESkdpNLUEiUu3VrVuXJ598kieeeAKbzUbv3r1JTU1lxYoV+Pn50aRJEwBefPFF6tevT0hICM899xxBQUHcdNNNAPztb3+ja9euTJkyheHDh7Nq1Sreffdd3n//fQAiIyMZNWoU99xzD2+//TYxMTEcOnSIpKQkbrvtNke9dBGpRApBIlIjTJkyheDgYKZOncr+/fsJCAigU6dOPPvss/buqFdeeYXHHnuMvXv30rFjR3744Qc8PDwA6NSpE19//TWTJk1iypQpNGzYkBdffJHRo0fbn+ODDz7g2Wef5cEHH+TEiRM0btyYZ5991hEvV0SqgK4OE5Ear/DKrVOnThEQEODockSkhtCYIBEREXFKCkEiIiLilNQdJiIiIk5JLUEiIiLilBSCRERExCkpBImIiIhTUggSERERp6QQJCIiIk5JIUhERESckkKQiIiIOCWFIBEREXFKCkEiIiLilP4f+Y9eloW6L3gAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(model.history.history.keys())\n",
    "# summarize history for accuracy\n",
    "plt.plot(model.history.history['accuracy'])\n",
    "plt.plot(model.history.history['val_accuracy'])\n",
    "plt.title('model accuracy')\n",
    "plt.ylabel('accuracy')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'val'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5d437814",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a58b5876",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "274fbc9d",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}