7. Developing Word Embedding.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to Develop Word Embeddings with Gensim."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Modern Embedding algorithms like Word2Vec and GloVe can be used on Natural Language Processing problems like Machine Translation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Word Embeddings."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In word embeddings, we are trying to provide a dense vector representation of words that capture something about their meaning.\n",
    "\n",
    "They are an improvement over simpler bag of words model encoding schemses like word counts and frequencies that result in large and sparce vectors (mostly 0 values) that describe documents but not the meaning of the words.\n",
    "\n",
    "Word embeddings work by using an algorithm to train a set of fixed-length dense and continuous-valued vectors based on a large corpus of text. \n",
    "Each word is repesented by a point in the embedding space and these points are learned and moved around based on the words that surround the target word. This process of defining a word by the company it keeps that allows the word embeddings to learn something about the meaning of the words.\n",
    "\n",
    "This means words with similar meanings are locally clustered within the space."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Gensim Library."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is an open-source library for NLP, with a focus on topic modeling. It is not like the NLTK library, instead it is a focused an efficient suite of NLP tools for topic modeling.\n",
    "\n",
    "It also provides tools for loading pre-trained word embeddings in a few formats and for making use and querying a loaded embedding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Defaulting to user installation because normal site-packages is not writeable\n",
      "Collecting gensim\n",
      "  Downloading gensim-3.8.1-cp37-cp37m-manylinux1_x86_64.whl (24.2 MB)\n",
      "\u001b[K     |████████████████████████████████| 24.2 MB 1.3 MB/s eta 0:00:01     |████████████████▋               | 12.5 MB 1.5 MB/s eta 0:00:08\n",
      "\u001b[?25hRequirement already satisfied, skipping upgrade: numpy>=1.11.3 in /usr/lib/python3/dist-packages (from gensim) (1.16.2)\n",
      "Collecting smart-open>=1.8.1\n",
      "  Downloading smart_open-1.10.0.tar.gz (99 kB)\n",
      "\u001b[K     |████████████████████████████████| 99 kB 3.8 MB/s eta 0:00:011\n",
      "\u001b[?25hRequirement already satisfied, skipping upgrade: scipy>=0.18.1 in /home/michael/.local/lib/python3.7/site-packages (from gensim) (1.4.1)\n",
      "Requirement already satisfied, skipping upgrade: six>=1.5.0 in /usr/lib/python3/dist-packages (from gensim) (1.12.0)\n",
      "Requirement already satisfied, skipping upgrade: requests in /usr/lib/python3/dist-packages (from smart-open>=1.8.1->gensim) (2.21.0)\n",
      "Collecting boto3\n",
      "  Downloading boto3-1.12.31-py2.py3-none-any.whl (128 kB)\n",
      "\u001b[K     |████████████████████████████████| 128 kB 1.2 MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting google-cloud-storage\n",
      "  Downloading google_cloud_storage-1.26.0-py2.py3-none-any.whl (75 kB)\n",
      "\u001b[K     |████████████████████████████████| 75 kB 3.5 MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting jmespath<1.0.0,>=0.7.1\n",
      "  Downloading jmespath-0.9.5-py2.py3-none-any.whl (24 kB)\n",
      "Collecting s3transfer<0.4.0,>=0.3.0\n",
      "  Downloading s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)\n",
      "\u001b[K     |████████████████████████████████| 69 kB 3.9 MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting botocore<1.16.0,>=1.15.31\n",
      "  Downloading botocore-1.15.31-py2.py3-none-any.whl (6.0 MB)\n",
      "\u001b[K     |████████████████████████████████| 6.0 MB 1.0 MB/s eta 0:00:01\n",
      "\u001b[?25hCollecting google-resumable-media<0.6dev,>=0.5.0\n",
      "  Downloading google_resumable_media-0.5.0-py2.py3-none-any.whl (38 kB)\n",
      "Collecting google-cloud-core<2.0dev,>=1.2.0\n",
      "  Downloading google_cloud_core-1.3.0-py2.py3-none-any.whl (26 kB)\n",
      "Collecting google-auth<2.0dev,>=1.11.0\n",
      "  Downloading google_auth-1.12.0-py2.py3-none-any.whl (83 kB)\n",
      "\u001b[K     |████████████████████████████████| 83 kB 1.7 MB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied, skipping upgrade: python-dateutil<3.0.0,>=2.1 in /usr/lib/python3/dist-packages (from botocore<1.16.0,>=1.15.31->boto3->smart-open>=1.8.1->gensim) (2.7.3)\n",
      "Requirement already satisfied, skipping upgrade: docutils<0.16,>=0.10 in /usr/lib/python3/dist-packages (from botocore<1.16.0,>=1.15.31->boto3->smart-open>=1.8.1->gensim) (0.14)\n",
      "Requirement already satisfied, skipping upgrade: urllib3<1.26,>=1.20; python_version != \"3.4\" in /usr/lib/python3/dist-packages (from botocore<1.16.0,>=1.15.31->boto3->smart-open>=1.8.1->gensim) (1.24.1)\n",
      "Collecting google-api-core<2.0.0dev,>=1.16.0\n",
      "  Downloading google_api_core-1.16.0-py2.py3-none-any.whl (70 kB)\n",
      "\u001b[K     |████████████████████████████████| 70 kB 2.6 MB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied, skipping upgrade: rsa<4.1,>=3.1.4 in /home/michael/.local/lib/python3.7/site-packages (from google-auth<2.0dev,>=1.11.0->google-cloud-storage->smart-open>=1.8.1->gensim) (4.0)\n",
      "Requirement already satisfied, skipping upgrade: setuptools>=40.3.0 in /home/michael/.local/lib/python3.7/site-packages (from google-auth<2.0dev,>=1.11.0->google-cloud-storage->smart-open>=1.8.1->gensim) (45.1.0)\n",
      "Requirement already satisfied, skipping upgrade: pyasn1-modules>=0.2.1 in /home/michael/.local/lib/python3.7/site-packages (from google-auth<2.0dev,>=1.11.0->google-cloud-storage->smart-open>=1.8.1->gensim) (0.2.8)\n",
      "Requirement already satisfied, skipping upgrade: cachetools<5.0,>=2.0.0 in /home/michael/.local/lib/python3.7/site-packages (from google-auth<2.0dev,>=1.11.0->google-cloud-storage->smart-open>=1.8.1->gensim) (4.0.0)\n",
      "Collecting googleapis-common-protos<2.0dev,>=1.6.0\n",
      "  Downloading googleapis-common-protos-1.51.0.tar.gz (35 kB)\n",
      "Requirement already satisfied, skipping upgrade: protobuf>=3.4.0 in /home/michael/.local/lib/python3.7/site-packages (from google-api-core<2.0.0dev,>=1.16.0->google-cloud-core<2.0dev,>=1.2.0->google-cloud-storage->smart-open>=1.8.1->gensim) (3.10.0)\n",
      "Requirement already satisfied, skipping upgrade: pytz in /usr/lib/python3/dist-packages (from google-api-core<2.0.0dev,>=1.16.0->google-cloud-core<2.0dev,>=1.2.0->google-cloud-storage->smart-open>=1.8.1->gensim) (2019.1)\n",
      "Requirement already satisfied, skipping upgrade: pyasn1>=0.1.3 in /home/michael/.local/lib/python3.7/site-packages (from rsa<4.1,>=3.1.4->google-auth<2.0dev,>=1.11.0->google-cloud-storage->smart-open>=1.8.1->gensim) (0.4.8)\n",
      "Building wheels for collected packages: smart-open, googleapis-common-protos\n",
      "  Building wheel for smart-open (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for smart-open: filename=smart_open-1.10.0-py3-none-any.whl size=90632 sha256=99ebf569e94197d9763921bf41d2ed7a8d89191458ec46d8836db0c5c124f6aa\n",
      "  Stored in directory: /home/michael/.cache/pip/wheels/1f/e5/fc/7412935a7184efc8ad377e948c81b1cc99b6a02eb8dc7c918c\n",
      "  Building wheel for googleapis-common-protos (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for googleapis-common-protos: filename=googleapis_common_protos-1.51.0-py3-none-any.whl size=77593 sha256=d09d37afc6a3299f660d69c15a05245bbbc42ed5a8e2f29004633c458147aeae\n",
      "  Stored in directory: /home/michael/.cache/pip/wheels/4c/a1/71/5e427276ceeff277fd76878d1b19fbf4587a2845015d86864b\n",
      "Successfully built smart-open googleapis-common-protos\n",
      "Installing collected packages: jmespath, botocore, s3transfer, boto3, google-resumable-media, googleapis-common-protos, google-auth, google-api-core, google-cloud-core, google-cloud-storage, smart-open, gensim\n",
      "  Attempting uninstall: google-auth\n",
      "    Found existing installation: google-auth 1.10.1\n",
      "    Uninstalling google-auth-1.10.1:\n",
      "      Successfully uninstalled google-auth-1.10.1\n",
      "Successfully installed boto3-1.12.31 botocore-1.15.31 gensim-3.8.1 google-api-core-1.16.0 google-auth-1.12.0 google-cloud-core-1.3.0 google-cloud-storage-1.26.0 google-resumable-media-0.5.0 googleapis-common-protos-1.51.0 jmespath-0.9.5 s3transfer-0.3.3 smart-open-1.10.0\n"
     ]
    }
   ],
   "source": [
    "# install gensim\n",
    "!pip install -U gensim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Develop Word2Vec Embeddings."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are 2 Word2Vec algorithms namely the Continuous Bag-of-Words and the Skip-Gram, both of which require a lot of text. \n",
    "\n",
    "Learning a word embedding using the Word2Vec class in Gensim involves loading and organizing the text into sentences and providing them to the constructor of a new Word2Vec() instance.\n",
    "\n",
    "Specifically, each sentence must be tokenized, meaning divided into words and prepared (e.g. perhaps pre-filtered and perhaps converted to a preferred case). \n",
    "\n",
    "The sentences could be text loaded into memory, or an iterator that progressively loads text, required for very large text corpora. There are quite a number of parameters in this constructor, some we may wish to configure are:\n",
    "    \n",
    "    1. size: (default 100) The number of dimensions in the embedding, eg. the length of the dense vector to represent each token.\n",
    "        \n",
    "    2. window: (default 5) The maximum distance between a target word and words around the target word.\n",
    "        \n",
    "    3. min_count: (default 5) The minimum count of words to consider when training the model: words with an occurence less than this count will be ignored.\n",
    "            \n",
    "    4. workers: (default 3) The number of threads to use while training.\n",
    "        \n",
    "    5. sg: (default 0 or CBOW) The training algorithm, either CBOW (0) or skip gram (1).\n",
    "    \n",
    "Let's take a small worked example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load modules\n",
    "from gensim.models import Word2Vec\n",
    "from gensim.test.utils import get_tmpfile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Word2Vec(vocab=14, size=100, alpha=0.025)\n"
     ]
    }
   ],
   "source": [
    "# define training data\n",
    "sentences = [['this', 'is', 'the', 'first', 'sentence', 'for', 'word2vec'], \n",
    "             ['this', 'is', 'the', 'second', 'sentence'],\n",
    "             ['yet', 'another', 'sentence'], \n",
    "             ['one' ,'more', 'sentence'], \n",
    "             ['and', 'the', 'final', 'sentence']]\n",
    "\n",
    "# train a model\n",
    "path = get_tmpfile(\"Models/word2vec.model\")\n",
    "model = Word2Vec(sentences, min_count=1)\n",
    "# summarize the loaded model\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['this', 'is', 'the', 'first', 'sentence', 'for', 'word2vec', 'second', 'yet', 'another', 'one', 'more', 'and', 'final']\n"
     ]
    }
   ],
   "source": [
    "# Summarize the vocabulary\n",
    "words = list(model.wv.vocab)\n",
    "print(words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 2.4445585e-03 -4.0116808e-03  9.8356616e-04 -1.1523644e-03\n",
      "  4.5054639e-03  2.4449022e-03 -2.3396038e-03 -4.1663307e-03\n",
      "  2.9676219e-03  2.4469018e-03 -2.0692858e-03  4.3152613e-03\n",
      "  3.7573215e-03  3.9155057e-05  6.9033972e-04 -1.5007168e-03\n",
      "  3.2499444e-03 -1.2384851e-03 -4.0207501e-03  2.8744114e-03\n",
      "  9.7144727e-04  3.7244020e-03  8.9405745e-04 -1.7689941e-03\n",
      " -8.7716896e-04  3.7607455e-03 -1.6332819e-03 -1.4383604e-03\n",
      " -4.4988231e-03  8.0528739e-04 -3.9712316e-03  4.6293773e-03\n",
      " -1.3304424e-03  1.4792872e-03  1.2532071e-03  2.0937026e-03\n",
      "  5.6169560e-04  3.1592953e-03  3.0715705e-03  3.7911534e-03\n",
      "  2.7528792e-03 -4.7004339e-03 -1.5779417e-03 -2.8022309e-03\n",
      "  4.0371749e-03 -1.0465648e-03 -1.0306865e-03 -4.4579511e-03\n",
      " -6.7581254e-04 -3.4397810e-03  2.3266107e-03 -1.9121204e-03\n",
      " -3.1571425e-04 -6.9215597e-04 -6.5398926e-04  4.0930277e-03\n",
      "  3.6619427e-03 -1.4072107e-03 -2.0808559e-03 -1.4185984e-04\n",
      "  2.2093847e-03 -1.9525071e-03 -2.7522151e-03  2.6156625e-03\n",
      "  3.3582798e-03 -1.2245205e-03 -3.7520952e-03  7.6585612e-04\n",
      "  8.5232122e-04 -4.6780221e-03  1.8195087e-03 -2.2692482e-03\n",
      " -6.2724436e-04 -4.5126774e-03  2.8510175e-03 -4.0786713e-03\n",
      " -3.0988685e-03  4.7600344e-03 -4.0241363e-03  4.1752504e-03\n",
      "  1.3118284e-03  4.8092483e-03  4.0938007e-03 -4.7797542e-03\n",
      "  4.4039357e-03  2.8796685e-03 -7.9250499e-04 -1.0354887e-03\n",
      "  2.0449399e-03 -4.0243412e-03 -2.6729738e-03  4.9696597e-03\n",
      "  9.8226022e-04  4.4495054e-03 -8.5421896e-04  1.4549462e-03\n",
      "  4.0244483e-03 -2.8541901e-03 -1.5987548e-03 -9.2375471e-04]\n"
     ]
    }
   ],
   "source": [
    "# Access the vector for one word\n",
    "print(model.wv['sentence'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the model\n",
    "model.save(\"Models/word2vec.model\") # or we could save to 'model_wv.bin' then remove binary=False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Word2Vec(vocab=14, size=100, alpha=0.025)\n"
     ]
    }
   ],
   "source": [
    "# Load model\n",
    "new_model = Word2Vec.load(\"Models/word2vec.model\")\n",
    "print(new_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train a previous word embedding model on a new sentence.\n",
    "new_model.train([[\"hello\", \"world\"]], total_examples=1, epochs=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4. Visualize Word Embeddings."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After learning word embeddings for your data, it can be nice to explore it with visualizations. You can use classical projection methods to reduce the high-dimensional vectors to two dimensional plots and plot them on a graph. The visualizations can provide a qualitative diagnostic for your learned model.\n",
    "\n",
    "Let's retrieve all the vectors from a trained model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load modules\n",
    "from sklearn.decomposition import PCA\n",
    "from matplotlib import pyplot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fit a 2d PCA model to the vectors\n",
    "X = new_model.wv[new_model.wv.vocab]\n",
    "pca = PCA(n_components=2)\n",
    "result = pca.fit_transform(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(14, 2)"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAAD8CAYAAACCRVh7AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xd8VVW6//HPQ0KJMjRBBAKXeEVKQggmdEQUITpwaYP3oqhYENSxzfxkBPmp2BnxTsGOiqKDgqACPwtFioIikkgRRqQrTYm0kU6S5/dHDpkQcyDhnPTv+/U6r3P22mvv/ayg5zl77bXXNndHREQkLxWKOwARESm5lCRERCQoJQkREQlKSUJERIJSkhARkaCUJEREJCglCRERCUpJQkREglKSEBGRoCKLO4AzUbt2bW/cuHFxhyEiUqqkpqb+7O51CrJNqUwSjRs3JiUlpbjDEBEpVczs+4Juo+4mEREJSklCRESCUpIQEZGglCRERCQoJQkREQlKSUJERIJSkhARkaCUJEREJKiwJAkzu8LMvjOzDWY2Io/1lc1sSmD9UjNrHChva2YrAq+VZtYvHPGIiEh4hJwkzCwCeA64EmgBXG1mLXJVuxnY6+4XAH8F/hwoXw0kuXsCcAXwkpmVyrvARUTKonCcSbQFNrj7Jnc/BkwG+uSq0weYGPg8DehmZubuh9w9PVBeBfAwxCMiImESjiTRANiaY3lboCzPOoGksB84B8DM2pnZGuAb4NYcSeMkZjbUzFLMLCUtLS0MYYuIyOmEI0lYHmW5zwiC1nH3pe4eC7QBRppZlbwO4u7j3T3J3ZPq1CnQJIYiInKGwpEktgENcyxHAzuC1Qlcc6gO7MlZwd2/BQ4CcWGISUREwiAcSWIZ0MTMYsysEjAQmJmrzkxgcODzAGC+u3tgm0gAM/sPoCmwJQwxiYhIGIQ8ksjd083sDmA2EAFMcPc1ZvYIkOLuM4FXgTfNbANZZxADA5t3BkaY2XEgE7jd3X8ONSYREQkPcy99A4qSkpJcDx0SESkYM0t196SCbKM7rkVEJCglCRERCUpJQkREglKSEBGRoJQkREQkKCUJEREJSklCRESCUpIQEZGglCRERCQoJQkREQlKSUJERIJSkhARkaCUJEREJCglCRERCUpJQkREglKSEBGRoJQkREQkKCUJEREJSklCRESCUpIQEZGglCRERCQoJQkREQlKSUJERIIKS5IwsyvM7Dsz22BmI/JYX9nMpgTWLzWzxoHy7maWambfBN4vC0c8IiISHiEnCTOLAJ4DrgRaAFebWYtc1W4G9rr7BcBfgT8Hyn8G/svdWwKDgTdDjUdERMInHGcSbYEN7r7J3Y8Bk4E+uer0ASYGPk8DupmZuftyd98RKF8DVDGzymGISUREwiAcSaIBsDXH8rZAWZ513D0d2A+ck6vO74Dl7n40DDGJiEgYRIZhH5ZHmRekjpnFktUF1SPoQcyGAkMBGjVqVPAoRUSkwMJxJrENaJhjORrYEayOmUUC1YE9geVo4H3genffGOwg7j7e3ZPcPalOnTphCFukbFi4cCG9evUq7jCkjApHklgGNDGzGDOrBAwEZuaqM5OsC9MAA4D57u5mVgP4EBjp7p+HIRYREQmjkJNE4BrDHcBs4FvgHXdfY2aPmFnvQLVXgXPMbAPwR+DEMNk7gAuAB8xsReB1bqgxiRSlgwcP0rNnT1q1akVcXBxTpkwhNTWVSy65hMTERJKTk9m5cycAGzZs4PLLL6dVq1ZcdNFFbNy4EXdn+PDhxMXF0bJlS6ZMmQJknSF07dqVAQMG0KxZMwYNGoR7Vi/trFmzaNasGZ07d+a9994rtrZLOeDupe6VmJjoIiXFtGnTfMiQIdnL+/bt8w4dOviuXbvc3X3y5Ml+4403urt727Zt/b333nN398OHD/vBgwd92rRpfvnll3t6err/+OOP3rBhQ9+xY4cvWLDAq1Wr5lu3bvWMjAxv3769L1q0yA8fPuzR0dG+bt06z8zM9Kuuusp79uxZ9A2XUgdI8QJ+34bjwrVIudayZUvuvfde7rvvPnr16kXNmjVZvXo13bt3ByAjI4N69erxyy+/sH37dvr16wdAlSpVAFi8eDFXX301ERER1K1bl0suuYRly5ZRrVo12rZtS3R0NAAJCQls2bKFqlWrEhMTQ5MmTQC49tprGT9+fDG0XMoDJQmRMzB9+XbGzv6OHfsOU79GFI+89v+wbSsYOXIk3bt3JzY2liVLlpy0zb/+9a889+WeezDgv1Wu/O/bhiIiIkhPTwfALK8BgyLhp7mbRApo+vLtjHzvG7bvO4wD32/dxmOzN1E19lLuvfdeli5dSlpaWnaSOH78OGvWrKFatWpER0czffp0AI4ePcqhQ4fo0qULU6ZMISMjg7S0ND777DPatm0b9PjNmjVj8+bNbNyYNRjw7bffLvQ2S/mlMwmRAho7+zsOH8/IXj6etoXNU19j0MQIWjSoyQsvvEBkZCR33XUX+/fvJz09nXvuuYfY2FjefPNNhg0bxoMPPkjFihWZOnUq/fr1Y8mSJbRq1Qoz46mnnuK8885j7dq1eR6/SpUqjB8/np49e1K7dm06d+7M6tWri6r5Us7YqU51S6qkpCRPSUkp7jCknIoZ8eGv7haFrDtGN4/pWdThiOSbmaW6e1JBtlF3k0gB1a8RVaBykdJMSULKhBMXdIvC8OSmRFWMOKksqmIEw5ObFlkMIkVFSUKK1ZYtW2jWrBlDhgwhLi6OQYMG8cknn9CpUyeaNGnCV199xZ49e+jbty/x8fG0b9+eVatWATB69GiGDh1Kjx49uP7668nIyGD48OG0adOG+Ph4XnrppUKJuW/rBjzZvyUNakRhQIMaUTzZvyV9W+ee11Kk9NOFayl2GzZsYOrUqYwfP542bdrw1ltvsXjxYmbOnMkTTzxBw4YNad26NdOnT2f+/Plcf/31rFixAoDU1FQWL15MVFQU48ePp3r16ixbtoyjR4/SqVMnevToQUxMTNhj7tu6gZKClAs6k5AzNm7cOJo3b07NmjUZM2bMGe8nMzOTli1bUqFCBWJjY+nWrRtmRsuWLdmyZQuLFy/muuuuA+Cyyy5j9+7d7N+/H4DevXsTFZV1LWDOnDm88cYbJCQk0K5dO3bv3s369etDb6hIOaYzCTljzz//PB9//HGBf6nnvBGtlu8/aV2FChWybyCrUKEC6enpREb++j/TEzeTnX322dll7s4zzzxDcnJyQZsiIkHoTELOyK233sqmTZvo3bs3f/3rX7njjjsAuOGGG7jrrrvo2LEj559/PtOmTQPgwIEDdOvWjf9s3pKByZ1Zv2wBDvz0ryO4ZyWOYLp06cKkSZOArEnvateuTbVq1X5VLzk5mRdeeIHjx48DsG7dOg4ePBjmlouUL0oSckZefPFF6tevz4IFC6hZs+ZJ63bu3MnixYv54IMPGDEia8LfKlWq8P7773Pe4L9TZ+Dj7J3/6knTUYyd/V3QY40ePZqUlBTi4+MZMWIEEydOzLPekCFDaNGiBRdddBFxcXEMGzasSEc9iZRF6m6SsOvbty8VKlSgRYsW/PTTT0BWV9D999/Psnc+BDMyDuwm8+A+IqvXxSpWZse+wwC8/vrr2ftp3Lhx9p3EM2bM+NVxRo8efdJyhQoVeOKJJ3jiiScKp2Ei5ZCShORb7kntDh3LyLNezknpTpwtTJo0ibS0NBLvfomdvxxn2ws34RnHsuvpRjSRkkndTZIvuSe1277vMHsPHeOjVTvztf3+/fs599xzue+3sWRuX03Gv3adtF43oomUTEoSki+5J7UDcIdnF2zI1/aDBg0iJSWFx27pQ9MDK4iq0wjIuhGtcmQF3XMgUkJpgj/JF01qJ1L6aYI/KTSa1E6kfFKSkHzRpHYi5ZNGN0m+nLhmkHN00/DkprqWIFLGKUlIvmlSO5HyR91NIiISVFiShJldYWbfmdkGMxuRx/rKZjYlsH6pmTUOlJ9jZgvM7ICZPRuOWEREJHxCThJmFgE8B1wJtACuNrMWuardDOx19wuAvwJ/DpQfAR4A7g01DhERCb9wnEm0BTa4+yZ3PwZMBvrkqtMHODEr2zSgm5mZux9098VkJQsRESlhwpEkGgBbcyxvC5TlWcfd04H9wDlhOLaIiBSicCQJy6Ms9825+alz6oOYDTWzFDNLSUtLK8imIiJyhsKRJLYBDXMsRwM7gtUxs0igOrCnIAdx9/HunuTuSXXq1AkhXBERya9wJIllQBMzizGzSsBAYGauOjOBwYHPA4D5XhonjRIRKWdCvpnO3dPN7A5gNhABTHD3NWb2CJDi7jOBV4E3zWwDWWcQA09sb2ZbgGpAJTPrC/Rw93+GGpeIiIQuLHdcu/tHwEe5yh7M8fkIcFWQbRuHIwYREQk/3XEtIiJBKUmIiEhQShIiIhKUkoSIiASlJCEiIkEpSYiISFBKEiIiEpSShIiIBKUkISIiQSlJiIhIUEoSIlIuTZ8+nX/+89/TxHXt2pWUlJRijKhkUpIQKQXGjRtH8+bNGTRoUHGHUmbkThKhyMjICMt+SiIlCZFS4Pnnn+ejjz5i0qRJp62bnp5eBBEVr759+5KYmEhsbCzjx48HoGrVqowaNYpWrVrRvn17fvrpJwC+//57unXrRnx8PN26deOHH37giy++YObMmQwfPpyEhAQ2btwIwNSpU2nbti0XXnghixYtArISwPDhw2nTpg3x8fG89NJLACxcuJBLL72Ua665hpYtWxbDX6GIuHupeyUmJrpIeTFs2DCvWLGix8XF+dNPP+19+vTxli1bert27XzlypXu7v7QQw/5Lbfc4t27d/err766mCMufLt373Z390OHDnlsbKz//PPPDvjMmTPd3X348OH+6KOPurt7r169/PXXX3d391dffdX79Onj7u6DBw/2qVOnZu/zkksu8T/+8Y/u7v7hhx96t27d3N39pZdeyt7XkSNHPDEx0Tdt2uQLFizws846yzdt2lQELQ4Psh7fUKDv27BMFS4ihefFF19k1qxZLFiwgIcffpjWrVszffp05s+fz/XXX8+KFSsASE1NZfHixURFRRVzxOE3ffl2xs7+jh37DlO/RhQNN3/At1/OA2Dr1q2sX7+eSpUq0atXLwASExOZO3cuAEuWLOG9994D4LrrruNPf/pT0OP0798/e/stW7YAMGfOHFatWsW0adMA2L9/f/bx2rZtS0xMTKG0uaRQkhApRRYvXsy7774LwGWXXcbu3bvZv38/AL179y6zCWLke99w+HhWv//GVUtZvmg2r02Zwf90vICuXbty5MgRKlasiJkBEBEREbTb7USdvFSuXPlX27s7zzzzDMnJySfVXbhwIWeffXbI7SvpdE1CpISavnw7ncbMJ2bEh/y4/wgfrdqJ5/HU3xNfemX1C2vs7O+yEwRA5tFDUPlsxn32A2vXruXLL7885fYdO3Zk8uTJAEyaNInOnTsD8Jvf/IZffvnltMdv0aIFo0eP5vjx4wCsW7eOgwcPnmlzSh0lCZES6MSv5+37DuNAeqbz6If/JLr5RdkXrxcuXEjt2rWpVq1a8QZbyHbsO3zSclRMIp6ZybK/3MwDDzxA+/btT7n9uHHjeO2114iPj+fNN9/k73//OwADBw5k7NixtG7dOvvCdV5iYmI4duwYF110EXFxcQwbNqxcDA44Qd1NIiVQ7l/PAEeOZ7CrSW9SUrK+8M466ywmTpxYTBEWnfo1otieI1FYZEXq/vfDNKgRxdQRl2WXHzhwIPvzgAEDGDBgAAcPHuT3v/89u3fvJjMzk6FDh5KWlsZ1113HgQMHaNiwIa+//jr16tUDskY33X777ezbt48333yTY8eOMXr0aA4fPkyDBg0YOXIkvXr14s477+Sbb74hPT2dGTNm0KdPH15//XVmzpzJoUOH2LhxI/369eOpp54CYNasWdx///1kZGRQu3Zt5s2bx8GDB0/az+jRo+nTp08R/VXzT0lCpATK/es5+rYJAKQdg69mzPhV/dGjRxdFWMVieHLTk65JAERVjGB4ctPTbjtr1izq16/Phx9+CGRddL7yyiuZMWMGderUYcqUKYwaNYoJE7L+vunp6Xz11Vd89NFHPPzww3zyySc88sgjpKSk8OyzzwJw//33c9lllzFhwgT27dtH27ZtufzyywFYsWIFy5cvp3LlyjRt2pQ777yTKlWqcMstt/DZZ58RExPDnj17AHj88cfz3E9J6zZUkhApgXL/es5ZXt70bd0A4KTRTcOTm2aX55ZzJFTN4wfY/tFsat13H7169aJmzZqsXr2a7t27A1n3QJw4i4C8RzflNmfOHGbOnMnTTz8NwJEjR/jhhx8A6NatG9WrVweyrmV8//337N27ly5dumSPgqpVq9Yp99O8efMz/lsVBiUJKfP27dvHW2+9xe23317coeRbKL+ey6K+rRsETQo55R4Jtadibapf/b8c/c1ORo4cSffu3YmNjWXJkiV5bp/X6Kbc3J13332Xpk1P/rdYunRp9vY59+HueY6oCrafkkYXrqXM27dvH88//3xxh1EgfVs34Mn+LWlQIwoDGtSI4sn+LfP1RVme5b6Wk/7Lbo4SybLIOO69916WLl1KWlpadpI4fvw4a9asOeU+c4+CSk5O5plnnskeabZ8+fJTbt+hQwc+/fRTNm/eDJDd3VTQ/RSXsJxJmNkVwN+BCOAVdx+Ta31l4A0gEdgN/I+7bwmsGwncDGQAd7n77HDEJGXbAw88QO3atbn77rsBGDVqFHXr1uXo0aO88847HD16lH79+vHwww8zYsQINm7cSEJCAt27d2fs2LHFHH3+5PfXs/xb7ms5x9O2sGvha+w04/FG5/DCCy8QGRnJXXfdxf79+0lPT+eee+4hNjY26D4vvfRSxowZQ0JCAiNHjuSBBx7gnnvuIT4+HnencePGfPDBB0G3r1OnDuPHj6d///5kZmZy7rnnMnfu3ALvp7hYXuOuC7QDswhgHdAd2AYsA65293/mqHM7EO/ut5rZQKCfu/+PmbUA3gbaAvWBT4AL3f2Us2UlJSW5Zmss37Zs2UL//v35+uuvyczMpEmTJjzxxBPMmzePl156CXend+/e/OlPf6JRo0b06tWL1atX57mvnN1RCxcu5Omnny6R/7PK6XUaMz/PazkNakTxeY6RUOWVmaW6e1JBtglHd1NbYIO7b3L3Y8BkIPc4rj7AibF604BultVJ1weY7O5H3X0zsCGwP5FTaty4Meeccw7Lly9nzpw5tG7dmmXLlmV/vuiii1i7di3r168/7b5KY3eU5G14clOiKkacVFaer+WEQzi6mxoAW3MsbwPaBavj7ulmth84J1D+Za5tdX4tQeUcuVKpTnseGPssZ2cc4KabbmLevHmMHDmSYcOGnbRNsFEqJ+TsjqpYsSJnn302AwYMYPXq1SQmJvKPf/wDMyM1NZU//vGPHDhwgNq1a580vl5KhoKOhJLTC0eSyGsilNx9WMHq5GfbrB2YDQWGAjRq1Kgg8UkZkXvkypEGicyd8Bo1oyJ46623iIyM5IEHHmDQoEFUrVqV7du3U7FixdNOvzBmzBhWr17NihUrWLhwIX369GHNmjXUr1+fTp068fnnn9OuXTvuvPPOoOPrpeTQtZzwCkd30zagYY7laGBHsDpmFglUB/bkc1sA3H28uye5e1KdOnXCEHb5lLNrZeHChdmzZuY2ZMiQsD2QJVxyj1yxiIpUatSSyAs6EhERQY8ePbjmmmvo0KED9evX54ILLuCCCy7g5ZdfplOnTsTFxTF8+PBTHuPHH3+kUaNGREdHU6FCBRISEtiyZQvfffdd9vj6hIQEHnvsMbZt21bYTRYpduE4k1gGNDGzGGA7MBC4JledmcBgYAkwAJjv7m5mM4G3zOwvZF24bgJ8FYaYJIgTSeJ09wy88sorRRRR/uUeueKeydEd30GbEdlld999N3fffTfNmjXj888/DzqNc3p6Oh988xNjZ3/H999vYc/PB5m+fDs//vgju3btyq6Xc6z7qcbXi5RVISeJwDWGO4DZZA2BneDua8zsEbIecDETeBV408w2kHUGMTCw7Rozewf4J5AO/P50I5skNPntf+/atStPP/00rVu35uabbyYlJQUz46abbuIPf/hDscSe8y7kYz//QNq0h4m6sAP/cf4FJ9W79dZb2bRpE7179+amm25i48aNPPvss9xwww3UqlWL5cuXUy36QlbYf7Jz9ouQmcnxPdv509tLOTztefbs2UNCQgKDBw/O3mfTpk2zx9d36NCB48ePs27dulMOnRQpC8Jyn4S7fwR8lKvswRyfjwBXBdn2ceDxcMQhp5ef/vcTUylD1lw027dvzx4+um/fvuIK/aS7kCvVbkSDW1/Nc+RKzof05B7Kum7dOj755BO6jP2UtFdGUKv7bVSJbsGu6WPYPOEeIi1r2oQTD/K54447AKhUqRLTpk0r0Ph6kbJA03KUc23btiU6Ohogu/89Z5I4//zz2bRpE3feeSc9e/akR48exRXqaUeu5Bz5dOL5C7ldddVVREREsGPfYSpHt2Dv/Fc4u0VXal02hMhqtTnywyqS/vV5dv0Tk7pB1t/ns88+K+RWipQsShLlxIkv0Jz97zUgz7lmcqpZsyYrV65k9uzZPPfcc7zzzjvFOqIn2MiV3COfTjx/4cpqe0+qd2KGzfo1ovD2VxH1n204vDGFH9/8P9Qd+Bi1q1aGfxV+O0RKC83dVA7kfICNVYri2OGDjHzvGxavTzvttj///DOZmZn87ne/49FHH+Xrr78ugogLLtjzFz5e/euzCcjquor45Scq1WlM9fYDqFSvCbZ/Bzde0jxfTysTKS90JlEO5PwCjYiqRuUGLdj44jDGVI6ia8IFp9x2+/bt3HjjjWRmZgLw5JNPFnq8ZyL3yKcT9h46nmd539YNeHn3Iha8u5D0TKheL4a//p8b6HtRQz6a8DStWrXihhtuKLaL9CIlRchzNxUHzd1UMDEjPszzDkUDNo/pWdThFArN2SNyesU1d5OUcMEeVFOWHmCjOXtECoeSRDlQHr5A9fwFkcKhaxLlQHmZ9Exz9oiEn5JEOaEvUBE5E+puEhGRoJQkREQkKCUJoWrVqsUdgoTJX/7yF+Li4oiLi+Nvf/sbW7ZsoXnz5txyyy3ExsbSo0cPDh/OGiq8ceNGrrjiChITE7n44otZu3ZtMUcvJZGShEgZkZqaymuvvcbSpUv58ssvefnll9m7dy/r16/n97//PWvWrKFGjRq8++67AAwdOpRnnnmG1NRUnn766dNOHy/lky5clxF9+/Zl69atHDlyhLvvvpuhQ4dStWpV7r77bj744AOioqKYMWMGdevWZfPmzVxzzTWkp6dzxRVXFHfoEiaLFy+mX79+2fNT9e/fn0WLFhETE0NCQgIAiYmJbNmyhQMHDvDFF19w1VX/npz56NGjxRK3lGw6kygjJkyYQGpqKikpKYwbN47du3dz8OBB2rdvz8qVK+nSpQsvv/wykPVgnttuu41ly5Zx3nnnFXPkEorpy7fTacx8YkZ8yN/mrmPtj7+enTCvSRwzMzOpUaMGK1asyH59++23RRm6lBJKEmXEuHHjaNWqFe3bt2fr1q2sX7+eSpUqZT+e9MQvSIDPP/+cq6++GoDrrruuuEKWEOWcuNGBI7UvZOaMGUz5YgMHDx7k/fff5+KLL85z22rVqhETE8PUqVMBcHdWrlxZhNFLaaHuplLsxPTfG1ct5dAX7/LyW+/zPx0voGvXrhw5coSKFStiZsCvpwE/US6lV+6ZbyufdwFnxXbjxv7dOb/22QwZMoSaNWsG3X7SpEncdtttPPbYYxw/fpyBAwfSqlWroghdShEliVIq5/MTMo8eIj0yitEfbyDtx+18+eWXp9y2U6dOTJ48mWuvvZZJkyYVUcQSbnnNfFutbT+qt+3H6hwTN554qiDAvffem/05JiaGWbNmFW6QUuqpu6mUyvkrMiomEc/MZONLtzH6oQdp3779Kbf9+9//znPPPUebNm3Yv39/UYQrhaA8TNwoxU9ThZdS5WH6bzm13E/jg6yJGzWxoQSjqcLLEf2KFM18K0VB1yRKqeHJTfP8FVmWpv+W09PEjVLYlCRKqfIy/beIFC8liVJMvyJFpLCFdE3CzGqZ2VwzWx94z3NQtpkNDtRZb2aDc5Q/bmZbzexAKHGIiEjhCPXC9Qhgnrs3AeYFlk9iZrWAh4B2QFvgoRzJ5P8FykREpAQKNUn0ASYGPk8E+uZRJxmY6+573H0vMBe4AsDdv3T3nSHGICIihSTUJFH3xJd84P3cPOo0ALbmWN4WKCsQMxtqZilmlpKWlnZGwYqISMGc9sK1mX0C5DVV6Kh8HiOvSYIKfAefu48HxkPWzXQF3V5ERArutGcS7n65u8fl8ZoB/GRm9QAC77vy2MU2oGGO5WhgRziCFyluCxcuzJ5pd9KkScTHxxMfH0/Hjh01q6qUCaF2N80EToxWGgzMyKPObKCHmdUMXLDuESgTKXUyMjKCrouJieHTTz9l1apVPPDAAwwdOrQIIxMpHKEmiTFAdzNbD3QPLGNmSWb2CoC77wEeBZYFXo8EyjCzp8xsG3CWmW0zs9EhxiMS1FNPPcW4ceMA+MMf/sBll10GwLx587j22mt5++23admyJXFxcdx3333Z21WtWpUHH3yQdu3asWTJEmbNmkWzZs3o3Lkz7733Xna9jh07Zk/N3b59e7Zt2wbAfffdx/PPP59db/To0fzv//4vAGPHjqVNmzbEx8fz0EMPZdd54403iI+Pp1WrVnrmhxQvdy91r8TERBcpqCVLlviAAQPc3b1z587epk0bP3bsmI8ePdpHjx7tDRs29F27dvnx48f90ksv9ffff9/d3QGfMmWKu7sfPnzYo6Ojfd26dZ6ZmelXXXWV9+zZ81fHGjt2rN98883u7v711197ly5dstc1b97cv//+e589e7bfcsstnpmZ6RkZGd6zZ0//9NNPffXq1X7hhRd6Wlqau7vv3r27UP8uUn4AKV7A71tN8CflRmJiIqmpqfzyyy9UrlyZDh06kJKcty5kAAALkElEQVSSwqJFi6hRowZdu3alTp06REZGMmjQID777DMg64FNv/vd7wBYu3YtMTExNGnSBDPj2muv/dVxFixYwKuvvsqf//xnAFq3bs2uXbvYsWMHK1eupGbNmjRq1Ig5c+YwZ84cWrduzUUXXcTatWtZv3498+fPZ8CAAdSuXRuAWrVqFdFfSOTXNC2HlHknnuC3Y99h9lh1/vDoX+nYsSPx8fEsWLCAjRs30qhRI1JTU/PcvkqVKkRERGQvn+qpfqtWrWLIkCF8/PHHnHPOOdnlAwYMYNq0afz4448MHDgQyDqLHzlyJMOGDTtpH+PGjdOTA6XE0JmElGm5nwNt9Zoz8aVniajfgosvvpgXX3yRhIQE2rdvz6effsrPP/9MRkYGb7/9Npdccsmv9tesWTM2b97Mxo0bAXj77bez1/3www/079+fN998kwsvvPCk7QYOHMjkyZOZNm0aAwYMACA5OZkJEyZw4EDWrDTbt29n165ddOvWjXfeeYfdu3cDsGfPnsL404jki84kpEz71XOgo2PZv+QdPt71Gx6qW5cqVapw8cUXU69ePZ588kkuvfRS3J3f/va39OnT51f7q1KlCuPHj6dnz57Url2bzp07Zz8e9JFHHmH37t3cfvvtAERGRnLi4VixsbH88ssvNGjQgHr16gHQo0cPvv32Wzp06ABkXSD/xz/+QWxsLKNGjeKSSy4hIiKC1q1b8/rrrxfmn0kkKD2ZTso0PcFP5N/0ZDqRXPQEP5HQKElImTY8uSlRFSNOKtMT/ETyT9ckpEzTE/xEQqMkIWVeYT7Br2PHjnzxxReFsm+RkkDdTSIhUIKQsk5JQiQEVatWBWDnzp106dKFhIQE4uLiWLRoUTFHJhIe6m4SCYO33nqL5ORkRo0aRUZGBocOHSrukETCQklCpIByTvNx+HgG05dvp02bNtx0000cP36cvn37kpCQUNxhioSFuptECiD3NB/uMPK9b9jzm//ks88+o0GDBlx33XW88cYbxR2qSFgoSYgUQO5pPgAOH8/g0cmfce6553LLLbdw88038/XXXxdThCLhpe4mkQLYse9wnuU/rF5GQsLjVKxYkapVq+pMQsoMJQmRAqhfI4rtORJFoz9OA+DCi3vx+Yd/Ka6wRAqNuptECkDTfEh5ozMJkQLQNB9S3ihJiBRQYU7zIVLSqLtJRESCUpIQEZGglCRERCSokJKEmdUys7lmtj7wXjNIvcGBOuvNbHCg7Cwz+9DM1prZGjMbE0osIiISfqGeSYwA5rl7E2BeYPkkZlYLeAhoB7QFHsqRTJ5292ZAa6CTmV0ZYjwiIhJGoSaJPsDEwOeJQN886iQDc919j7vvBeYCV7j7IXdfAODux4CvgegQ4xERkTAKNUnUdfedAIH3c/Oo0wDYmmN5W6Asm5nVAP6LrLORPJnZUDNLMbOUtLS0EMMWEZH8OO19Emb2CXBeHqtG5fMYlkeZ59h/JPA2MM7dNwXbibuPB8YDJCUlebB6IiISPqdNEu5+ebB1ZvaTmdVz951mVg/YlUe1bUDXHMvRwMIcy+OB9e7+t3xFLCIiRSbU7qaZwODA58HAjDzqzAZ6mFnNwAXrHoEyzOwxoDpwT4hxiIhIIQg1SYwBupvZeqB7YBkzSzKzVwDcfQ/wKLAs8HrE3feYWTRZXVYtgK/NbIWZDQkxHhERCSNzL33d+0lJSZ6SklLcYYiIlCpmluruSQXZRndci4hIUEoSIiISlJKEiIgEpSQhIiJBKUmIiEhQShIiIhKUkoSIiASlJCEiIkEpSYiISFBKEiIiEpSShIiIBKUkISIiQSlJiIhIUEoSIiISlJKEiIgEpSQhIiJBKUmIiEhQShIiIhKUkoSIiASlJCEiIkEpSYiISFBKEiIiEpSShIiIBBVSkjCzWmY218zWB95rBqk3OFBnvZkNzlE+y8xWmtkaM3vRzCJCiUdERMIr1DOJEcA8d28CzAssn8TMagEPAe2AtsBDOZLJf7t7KyAOqANcFWI8IiISRqEmiT7AxMDniUDfPOokA3PdfY+77wXmAlcAuPu/AnUigUqAhxiPiIiEUahJoq677wQIvJ+bR50GwNYcy9sCZQCY2WxgF/ALMC3EeEREJIwiT1fBzD4Bzstj1ah8HsPyKMs+Y3D3ZDOrAkwCLiPrTCOvOIYCQwEaNWqUz0OLiEgoTpsk3P3yYOvM7Cczq+fuO82sHllnBLltA7rmWI4GFuY6xhEzm0lW91WeScLdxwPjAZKSktQtJSJSBELtbpoJnBitNBiYkUed2UAPM6sZuGDdA5htZlUDiQUziwR+C6wNMR4REQmjUJPEGKC7ma0HugeWMbMkM3sFwN33AI8CywKvRwJlZwMzzWwVsJKss5AXQ4xHRETCyNxLX89NUlKSp6SkFHcYIiKlipmluntSQbbRHdciIhKUkoSIiASlJCEiIkEpSYiISFBKEiIiEpSShIiIBFUqh8CaWRrwfXHHEVAb+Lm4gygiamvZpLaWTXm19T/cvU5BdlIqk0RJYmYpBR13XFqprWWT2lo2haut6m4SEZGglCRERCQoJYnQjS/uAIqQ2lo2qa1lU1jaqmsSIiISlM4kREQkKCWJfDCzWmY218zWB95rBqk3OFBnvZkNDpSdZWYfmtlaM1tjZmOKNvqCCaWtgfLHzWyrmR0ouqgLxsyuMLPvzGyDmY3IY31lM5sSWL/UzBrnWDcyUP6dmSUXZdxn4kzbambnmNkCMztgZs8WddxnIoS2djezVDP7JvB+WVHHXlAhtLWtma0IvFaaWb/THszd9TrNC3gKGBH4PAL4cx51agGbAu81A59rAmcBlwbqVAIWAVcWd5sKo62Bde2BesCB4m5LkPZFABuB8wP/HiuBFrnq3A68GPg8EJgS+NwiUL8yEBPYT0Rxt6mQ2no20Bm4FXi2uNtSyG1tDdQPfI4Dthd3ewqxrWcBkYHPJ54mGnmq4+lMIn/6ABMDnycCffOokwzMdfc97r6XrMewXuHuh9x9AYC7HwO+JusRriXVGbcVwN2/dPedRRLpmWkLbHD3TYF/j8lktTmnnH+DaUA3M7NA+WR3P+rum4ENgf2VVGfcVnc/6O6LgSNFF25IQmnrcnffEShfA1Qxs8pFEvWZCaWth9w9PVBeBTjtRWklifype+KLL/B+bh51GgBbcyxvC5RlM7MawH8B8wopznAIS1tLsPzEnl0n8D/UfuCcfG5bkoTS1tImXG39HbDc3Y8WUpzhEFJbzaydma0BvgFuzZE08hQZpqBLPTP7BDgvj1Wj8ruLPMqys3TgOd5vA+PcfVPBIwyfwm5rCZef2IPVKW3tDqWtpU3IbTWzWODPQI8wxlUYQmqruy8FYs2sOTDRzD5296BnjEoSAe5+ebB1ZvaTmdVz951mdqIfL7dtQNccy9HAwhzL44H17v63MIQbkiJoa0m2DWiYYzka2BGkzrZAcq8O7MnntiVJKG0tbUJqq5lFA+8D17v7xsIPNyRh+Xd192/N7CBZ12GCPg9a3U35MxM4MYJnMDAjjzqzgR5mVjMwIqhHoAwze4ysf6R7iiDWUIXU1lJgGdDEzGLMrBJZF/Vm5qqT828wAJjvWVf6ZgIDAyNHYoAmwFdFFPeZCKWtpc0ZtzXQDfwhMNLdPy+yiM9cKG2NCSQNzOw/gKbAllMerbiv1JeGF1l9efOA9YH3WoHyJOCVHPVuIuti5gbgxkBZNFmned8CKwKvIcXdpsJoa6D8KbJ+xWQG3kcXd5vyaONvgXVkjRAZFSh7BOgd+FwFmBpo21fA+Tm2HRXY7jtK8Ci1MLV1C1m/Pg8E/i1bFHX8RdFW4P8CB3P8/7kCOLe421NIbb2OrIvzK8gaRNP3dMfSHdciIhKUuptERCQoJQkREQlKSUJERIJSkhARkaCUJEREJCglCRERCUpJQkREglKSEBGRoP4//fG5Q2q7v8QAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Create a scatter plot of the pojection \n",
    "pyplot.scatter(result[:, 0], result[:, 1])\n",
    "words = list(model.wv.vocab)\n",
    "for i, word in enumerate(words):\n",
    "    pyplot.annotate(word, xy=(result[i, 0], result[i, 1]))\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TO DO:\n",
    "    \n",
    "    Try on Truncated SVD and T-SNE instead of PCA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.manifold import TSNE\n",
    "X_embedded = TSNE(n_components=2).fit_transform(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(14, 2)"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_embedded.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 the\n"
     ]
    }
   ],
   "source": [
    "for i, word in enumerate(words):\n",
    "    print(i, word)\n",
    "    break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "ename": "IndexError",
     "evalue": "index 14 is out of bounds for axis 0 with size 14",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mIndexError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-75-e8281c63804d>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0;31m#words = list(model.wv.vocab)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mword\u001b[0m \u001b[0;32min\u001b[0m \u001b[0menumerate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mwords\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m     \u001b[0mpyplot\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mannotate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mword\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mxy\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX_embedded\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX_embedded\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      6\u001b[0m \u001b[0mpyplot\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshow\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mIndexError\u001b[0m: index 14 is out of bounds for axis 0 with size 14"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAD8CAYAAACVZ8iyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAHFxJREFUeJzt3X10VPW97/H31zFkDSInUlAhwE1qKWACBhMwyhFZ2hofuIgUz5Wbih5buLT2wfuQ1XBZXcfTrlVTWdci50i9uIqVUz1oESPngKUqcEEKlMQgCDQCIQoJBURDeQg2ib/7x+zJGcIkmSST2ZPk81prVmZ++zfZHzY78539+DPnHCIi0rdd5ncAERHxn4qBiIioGIiIiIqBiIigYiAiIqgYiIgIKgYiIoKKgYiIoGIgIiLA5X4HiMXgwYNdRkaG3zFERHqU8vLyT5xzQ2Lp2yOKQUZGBmVlZX7HEBHpUczso1j7ajeRiIioGIgk2pIlSxg7diyFhYV+RxFp1iN2E4n0JkuXLuXNN98kMzPT7ygizbRlINKNnn76abKzs8nOzmbx4sXMnz+fqqoqpk+fzi9+8Qu/44k06/KWgZmNAFYA1wJfAMucc8+Y2SDgFSADqAb+zjn3mZkZ8AxwD3AeeMQ5915Xc4gkm/Lycl544QV27NiBc46bbrqJ3/zmN/zud79j48aNDB482O+IIs3isWXQCPxP59xYIB94zMyuB4qBd5xzo4B3vNcAdwOjvMc84JdxyCCSNEoraphcsoE7f/R/qbs6h7c+rGPAgAHMnDmTLVu2+B1PJKoubxk4544Bx7znZ8xsP5AO3AdM9bq9CGwCfuS1r3ChIda2m1mamQ31fo9Ij1ZaUcOC1Xuob2jCOThzoZEFq/f4HUukXXE9ZmBmGcAEYAdwTfgD3vt5tdctHTgS8bajXptIj7dofSX1DU0ApI7I4vyB7Zw7f46Sf9vF66+/zq233upzQpHo4nY2kZkNAF4DHnfO/SV0aCB61yhtlwzEbGbzCO1GYuTIkfGKKdKtauvqm5+nXvsVBmTfwZ9X/A/+DDy18L8zYcIE/8KJtCEuxcDMUggVgpecc6u95uPh3T9mNhQ44bUfBUZEvH04UNvydzrnlgHLAPLy8i4pFiLJaFhakJqIgjBw0v0MnHQ/6WlBHn/8dgCqq6t9SifSui7vJvLODvoVsN8593TEpDXAw97zh4E3ItrnWEg+cFrHC6S3KCoYTTAlcFFbMCVAUcFonxKJxCYeWwaTgYeAPWa2y2v730AJ8KqZfQv4GHjAm7aO0GmlBwmdWvr3ccggkhRmTAgd/lq0vpLaunqGpQUpKhjd3C6SrCx0Uk9yy8vLc7pRnYhIx5hZuXMuL5a+ugJZRERUDERERMVARERQMRAREVQMREQEFQMREUHFQEREUDEQERFUDEREBBUDERFBxUBERFAxSIi6ujqWLl3qdwwRkVapGCSAioGIJDsVgwQoLi7m0KFD5OTkUFRURFFREdnZ2YwbN45XXnnF73gifVZGRgbV1dVMnTrV7yi+UzFIgJKSEq677jp27dpFfn4+u3bt4v333+ftt9+mqKiIY8c0to+I+EvFIMHeffddZs+eTSAQ4JprruG2225j586dfscS6ZOGDBlCIBBg0KBBfkfxXVzGQJZLlVbUNI92Ncid5i8XGgHoCYMJifQV4S9iq1evbqdn76ctg25QWlHDgtV7qKmrxwEnLhjHTn5GaUUNU6ZM4ZVXXqGpqYmTJ0+yefNmJk2a5HdkkT6jtKKGySUbyCxey+SSDZRW1PgdKSloy6AbLFpfSX1DU/PrQHAg/dLH8l/v/lsee2gW48eP54YbbsDMeOqpp7j22mt9TCvSd4S/qIX/Pmvq6lmweg9Anx+nWmMgd4PM4rVEW6oGHC65N9FxRMQzuWQDNXX1l7SnpwXZWny7D4m6l8ZA9tmwtGCH2kUkMWqjFIK22vsSFYNuUFQwmmBK4KK2YEqAooLRPiUSEdAXtbaoGHSDGRPSeXLmONLTghihTdAnZ47r8/skRfymL2qt0wHkbjJjQro+/EWSTPhvMnza97C0IEUFo/W3ioqBiPQx+qIWnXYTiYiIioGIiKgYiIgIKgYiIoKKgYiIoGIgIiKoGIiICCoGIiKCioGIiKBiICIixKkYmNlyMzthZh9EtA0ys7fM7ID38yqv3cxsiZkdNLPdZnZjPDKIiEjnxWvL4NfAXS3aioF3nHOjgHe81wB3A6O8xzzgl3HKICIinRSXYuCc2wx82qL5PuBF7/mLwIyI9hUuZDuQZmZD45FDREQ6pzuPGVzjnDsG4P282mtPB45E9DvqtYmIiE/8OIBsUdouGTLYzOaZWZmZlZ08eTIBsURE+q7uLAbHw7t/vJ8nvPajwIiIfsOB2pZvds4tc87lOefyhgwZ0o0xRUSkO4vBGuBh7/nDwBsR7XO8s4rygdPh3UkiIuKPuIx0Zmb/CkwFBpvZUeAfgBLgVTP7FvAx8IDXfR1wD3AQOA/8fTwyiIhI58WlGDjnZrcy6Y4ofR3wWDzmK71faUVN83i1tn897k/v8DfBFNatW8ewYcP8jifSa2gMZElapRU1LFi9h/qGJgDc2AKC4+/hiZnjVAhE4ky3o5CktWh9ZXMhCKtvaGLR+kqfEon0XioGkrRq6+o71C4inadiIElrWFqwQ+0i0nkqBpK0igpGE0wJXNQWTAlQVDDap0QivZcOIEvSmjEhdJeS8NlEw9KCFBWMbm4XkfhRMZCkNmNCuj78RRJAu4lERETFQEREVAxERAQVAxERQcVARERQMRAREVQMREQEFQMREUHFQEREUDEQERF0OwqRPidy9Djd70nCtGUg0oeER4+rqavHATV19RTOvJdfrS/zO5r4TMVApA9pOXqcc1/w+ae1PP/Hkz6mkmSgYiDSh7QcJa7hk4/p/9VbOH7uC58SSbJQMRDpQ1qOEtdvSAaD7pir0eNExUCkL9HocdIanU0k0odo9DhpjYqBSB+j0eMkGu0mEhERFQMREVExEBER+lAxuOWWW/yOICLiCzM7216fPlMM/vCHP/gdQUQkafWZYjBgwAAANm3axNSpU5k1axZjxoyhsLAQ55zP6URE2jZjxgxyc3PJyspi2bJlQOhzbeHChdxwww3k5+dz/PhxAA4fPszNN98MMNbMfhrL7+8zxSBSRUUFixcvZt++fVRVVbF161a/I4mItGn58uWUl5dTVlbGkiVLOHXqFOfOnSM/P5/333+fKVOm8PzzzwPwwx/+kO985zsA+4E/x/L7e3UxKK2oYXLJBjKL11Lf0ERpRQ0AkyZNYvjw4Vx22WXk5ORQXV3tb1ARkXYsWbKkeQvgyJEjHDhwgH79+jFt2jQAcnNzmz/Ltm7dyuzZs8Nv/ZdYfn+vvegsfKve8B0anYMFq/dQOPIMqampzf0CgQCNjY1+xRQRiSpy3IkrPq2kaec6yrdto3///kydOpULFy6QkpKCmQGXfpaF22PVa4tBy1v1AtQ3NLFy5xEy/IkkIhKTll9mT5z6jPPnjN9XfsaY4Mds3769zfdPnjyZlStXhl8WxjJP33YTmdldZlZpZgfNrDjev7/lrXrDPjn7ebxnJSISVy2/zAYzc2lqbKLwnlv58Y9/TH5+fpvvf+aZZ3j22WcBxgJ/E8s8zY8zacwsAHwIfB04CuwEZjvn9kXrn5eX58rKOjYS0+SSDdREKQjpaUG2Ft/e4cwiIomSWbyWaJ/MBhwuuTfm32Nm5c65vFj6+rVlMAk46Jyrcs79FVgJ3BfPGehWvSLSU7U2vkR3jjvhVzFIB45EvD7qtcXNjAnpPDlzHOlpQYzQFsGTM8fpbo0ikvT8+DLr1wHkaIe5L9oqMrN5wDyAkSNHdmomulWviPREfow74VcxOAqMiHg9HKiN7OCcWwYsg9Axg8RFExHxX6K/zPq1m2gnMMrMMs2sH/AgsManLCIifZ4vWwbOuUYz+x6wHggAy51ze/3IIiIiPl505pxbB6zza/4iIvIfevW9iUREJDYqBiIiomIgIiIqBuKjaIN1iIg/eu1dSyX5LV++nEGDBlFfX8/EiRP5xje+wZe+9CW/Y4n0SSoGkjCR92cflhZkxOF/Z//2dwCaB+tQMRDxh4qBJETL+7Mf2r2Dii3reeGVN/gvt3ylebAOEfGHioEkRMv7s3/x+XlIvYIlmz/mhkGN7Q7WISLdS8VAEqLlYEPBzFzOVLzJzqe/xY/L89odrENEupeKgSTEsLTgRYMN2eUpXPN3/0h6WpDfarAhEd/p1FJJCA02JJLctGUgCeHH/dlFJHYqBpIwGmxIJHlpN5GIiKgYiIiIioGIiKBiICIiqBiIiAgqBiIigoqBiHRAXV0dS5cuBWDTpk1MmzatQ+//9a9/TW1tbXdEky5SMRCRmEUWg85QMUheKgYiErPi4mIOHTpETk4ORUVFnD17llmzZjFmzBgKCwtxzgHwk5/8hIkTJ5Kdnc28efNwzrFq1SrKysooLCwkJyeH+vr6duYmCeWcS/pHbm6uExH/HT582GVlZTnnnNu4caMbOHCgO3LkiGtqanL5+fluy5YtzjnnTp061fyeb37zm27NmjXOOeduu+02t3PnzsQH76OAMhfj56y2DESk0yZNmsTw4cO57LLLyMnJobq6GoCNGzdy0003MW7cODZs2MDevXv9DSrt0r2JRKRd4SFLP/qomk8/OUdpRQ1pQGpqanOfQCBAY2MjFy5c4Lvf/S5lZWWMGDGCJ554QqPY9QDaMhCRNoWHLK2pq8f6Bflr/TkWrN7DuwdORu0f/uAfPHgwZ8+eZdWqVc3TrrzySs6cOZOQ3NIx2jIQkTZFDlkaCA4kNf16Dj333yhJDTI15yuX9E9LS2Pu3LmMGzeOjIwMJk6c2DztkUceYf78+QSDQbZt20YwGEzYv0PaZs47+p/M8vLyXFlZmd8xRPqkzOK1RPuUMOBwyb2JjiMdYGblzrm8WPpqN5GItGlYWvRv7621S8+kYiAibdKQpX2DjhmISJs0ZGnfoGIgIu3SkKW9n3YTiYiIioGIiKgYiIgIXSwGZvaAme01sy/MLK/FtAVmdtDMKs2sIKL9Lq/toJkVd2X+IiISH13dMvgAmAlsjmw0s+uBB4Es4C5gqZkFzCwAPAvcDVwPzPb6ioiIj7p0NpFzbj+AmbWcdB+w0jn3OXDYzA4Ck7xpB51zVd77Vnp993Ulh4iIdE13HTNIB45EvD7qtbXWLiIiPmp3y8DM3gaujTJpoXPujdbeFqXNEb34RL05kpnNA+YBjBw5sr2YIiLSBe0WA+fc1zrxe48CIyJeDwfCA5+21t5yvsuAZRC6UV0nMoiISIy6azfRGuBBM0s1s0xgFPBHYCcwyswyzawfoYPMa7opg4iIxKhLB5DN7H7gn4AhwFoz2+WcK3DO7TWzVwkdGG4EHnPONXnv+R6wHggAy51zGg9PRMRnGs9ARKSX0ngGIiLSISoGIiKiYiAiIioGIiKCikGHlVbUMLlkA5nFa5lcsoHSihq/I0kn1NXVsXTpUgA2bdrEtGnTfE4k4i8Vgw4orahhweo91NTV44CaunoWrN6jgtADRRYDkWSQkZHh6/xVDDpg0fpK6huaLmqrb2hi0fpKnxJJZxUXF3Po0CFycnIoKiri7NmzzJo1izFjxlBYWEj4lOvy8nJuu+02cnNzKSgo4NixYz4nF+keKgYdUFtXf9Hr47/9BxrPnLqkXZJfSUkJ1113Hbt27WLRokVUVFSwePFi9u3bR1VVFVu3bqWhoYHvf//7rFq1ivLych599FEWLlzod3TppYYMGQLAsWPHmDJlCjk5OWRnZ7Nly5aEzL9LVyD3NcPSgtREfPBf88A/NrdLz1BaUcOi9ZV89FE1n35yjtKKGtKASZMmMXz4cABycnKorq4mLS2NDz74gK9//esANDU1MXToUB/TS2+2c+dOAF5++WUKCgpYuHAhTU1NnD9/PiHzVzHogKKC0SxYveeiXUXBlABFBaN9TCWxCh/zCf//NTZ9wYLVeygceYbU1NTmfoFAgMbGRpxzZGVlsW3bNr8iSx80ceJEHn30URoaGpgxYwY5OTkJma92E3XAjAnpPDlzHOlpQQxITwvy5MxxzJigIRl6gshjPtYvyBd/rae+oYmVO49E7T969GhOnjzZXAwaGhrYu1e30pL4iXZ24pQpU9i8eTPp6ek89NBDrFixIiFZtGXQQTMmpOvDv4eKPLYTCA4kNf16an/1XezyVDJyv3pJ/379+rFq1Sp+8IMfcPr0aRobG3n88cfJyspKZGzppVpuqYbPTjxRe5RHC3KZO3cu586d47333mPOnDndnkc3qpM+Y3LJhouO+YSlpwXZWny7D4mkL2ttfUw5tJnU/WtJSUlhwIABrFixgszMzE7NoyM3qtOWgfQZOuYjyaS1sxAbr5vCgd/+PMFpdMxA+hAd85Fk0tpZiH6dnagtA+lTdMxHkkWybamqGIiI+CD8pWTR+kpq6+oZlhakqGC0b19WVAxERHySTFuqOmYgIiIqBiIiomIgIiKoGIiICCoGIiKCioGIiKBiICIiqBiIiAgqBiIigoqBiIigYiAiIqgYiIgIKgYiIoKKgYiIoGIgIiKoGIiICCoGIiJCF4uBmS0ysz+Z2W4ze93M0iKmLTCzg2ZWaWYFEe13eW0Hzay4K/MXEZH46OqWwVtAtnNuPPAhsADAzK4HHgSygLuApWYWMLMA8CxwN3A9MNvrKyIiPupSMXDO/d451+i93A4M957fB6x0zn3unDsMHAQmeY+Dzrkq59xfgZVeXxER8VE8jxk8CrzpPU8HjkRMO+q1tdZ+CTObZ2ZlZlZ28uTJOMYUEZGWLm+vg5m9DVwbZdJC59wbXp+FQCPwUvhtUfo7ohcfF22+zrllwDKAvLy8qH1ERCQ+2i0GzrmvtTXdzB4GpgF3OOfCH9pHgRER3YYDtd7z1tpFRMQnXT2b6C7gR8B059z5iElrgAfNLNXMMoFRwB+BncAoM8s0s36EDjKv6UoGERHpuna3DNrxz0Aq8JaZAWx3zs13zu01s1eBfYR2Hz3mnGsCMLPvAeuBALDcObe3ixlERKSL7D/27CSvvLw8V1ZW5ncMEZEexczKnXN5sfTVFcgiIqJiICIiKgYiEqOnn36a7OxssrOzWbx4MdXV1YwdO5a5c+eSlZXFnXfeSX19vd8xpZNUDESkXeXl5bzwwgvs2LGD7du38/zzz/PZZ59x4MABHnvsMfbu3UtaWhqvvfaa31Glk7p6NpGI9FKlFTUsWl9JbV097F3HxJvv4IorrgBg5syZbNmyhczMTHJycgDIzc2lurrax8TSFdoyEJFLlFbUsGD1Hmrq6nHA6fMNbNh/gtKKmov6paamNj8PBAI0NjYiPZOKgYhcYtH6Suobmppfp47I4i+V2yj5t/c5d+4cr7/+OrfeequPCSXetJtIRC5RW3fxgeDUa7/CgOw7eO+fvsNNr17Jt7/9ba666iqf0kl30EVnInKJySUbqKm79Myg9LQgW4tv9yGRdIYuOpOEWrJkCWPHjqWwsNDvKBInRQWjCaYELmoLpgQoKhjtUyLpbtpNJF22dOlS3nzzTTIzM9vt29jYyOWXa7VLdjMmhIYZCZ9NNCwtSFHB6OZ26X30VyldMn/+fKqqqpg+fTqPPPIIW7Zsoaqqiv79+7Ns2TLGjx/PE088QW1tLdXV1QwePJiXX37Z79gSgxkT0vXh34doN5F0yXPPPcewYcPYuHEj1dXVTJgwgd27d/Ozn/2MOXPmNPcrLy/njTfeUCEQSVLaMpC4effdd5uvQL399ts5deoUp0+fBmD69OkEg0E/44lIG1QMpFMir0798+kLrNt9jGhnpnnjXDRfuSoiyUm7iaTDWl6d2viF46dr9zF87I289FJoGOxNmzYxePBgBg4c6G9YEYmJtgykw1penQpwoaGJE6OmU1b2AuPHj6d///68+OKLPiUUkY7SRWfSYZnFa4m21hhwuOTeRMcRkVboojPpVsPSoh8Ibq1dRJKfioF0mK5OFel9dMxAOkxXp4r0PioG0im6OlWkd9FuIhERUTEQEREVAxERQcVARERQMRAREXrIFchmdhL4qEXzYOATH+K0JxlzJWMmSM5cyZgJkjNXMmaC5MzlV6b/5JwbEkvHHlEMojGzslgvs06kZMyVjJkgOXMlYyZIzlzJmAmSM1cyZmpJu4lERETFQEREenYxWOZ3gFYkY65kzATJmSsZM0Fy5krGTJCcuZIx00V67DEDERGJn568ZSAiInHSI4qBmS0ysz+Z2W4ze93M0iKmLTCzg2ZWaWYFEe13eW0Hzay4GzI9YGZ7zewLM8uLaM8ws3oz2+U9nouYlmtme7xMSyw8QHACcnnTfFlWUTI+YWY1EcvonvYyJkKil0MbOaq99WSXmZV5bYPM7C0zO+D9vCoBOZab2Qkz+yCiLWoOC1niLbvdZnZjgnP5uk6Z2Qgz22hm+72/vx967b4vr5g555L+AdwJXO49/znwc+/59cD7QCqQCRwCAt7jEPBloJ/X5/o4ZxoLjAY2AXkR7RnAB62854/AzYQGBXsTuLsbllVruXxbVlEyPgH8ryjtUTMmaB1L+HJoI0s1MLhF21NAsfe8OPw30M05pgA3Rq7PreUA7vHWaQPygR0JzuXrOgUMBW70nl8JfOjN2/flFeujR2wZOOd+75xr9F5uB4Z7z+8DVjrnPnfOHQYOApO8x0HnXJVz7q/ASq9vPDPtd85VxtrfzIYCA51z21xobVgBzIhnpnZy+basOqC1jImQTMshmvuA8KDSL9IN605LzrnNwKcx5rgPWOFCtgNp3jqfqFytScg65Zw75px7z3t+BtgPpJMEyytWPaIYtPAooYoKoYV9JGLaUa+ttfZEyTSzCjP7f2Z2q9eW7uXwK1OyLavveZvHyyN2efj5/+b3OhPJAb83s3Izm+e1XeOcOwahDx7gap+ytZYjGZZfUqxTZpYBTAB2kNzL6yJJM7iNmb0NXBtl0kLn3Bten4VAI/BS+G1R+juiF7kOnzYVS6YojgEjnXOnzCwXKDWzrDaydlgnc3XrsrpkZm1kBH4J/NSbz0+B/0OoyMdtGXWCn/NuabJzrtbMrgbeMrM/+ZSjI/xefkmxTpnZAOA14HHn3F/aOCzo9/K6RNIUA+fc19qabmYPA9OAO7zdLBCqpiMiug0Har3nrbXHLVMr7/kc+Nx7Xm5mh4CvelmHR3TtVKbO5qKbl1VLsWY0s+eBf48hY3fzc94Xcc7Vej9PmNnrhHZrHDezoc65Y97uhBN+ZGsjh6/Lzzl3PPzcr3XKzFIIFYKXnHOrveakXF7R9IjdRGZ2F/AjYLpz7nzEpDXAg2aWamaZwChCB2l3AqPMLNPM+gEPen0TkXWImQW851/2MlV5m4hnzCzfQl8X5gCtfYvvDkmzrFrsG70fCJ8V0lrGRPBtnYlkZleY2ZXh54ROnvjAy/Kw1+1hErvuRGotxxpgjneWTD5wOrx7JBH8Xqe8v+lfAfudc09HTErK5RWV30ewY3kQOuhzBNjlPZ6LmLaQ0BkClUScnUPoaP2H3rSF3ZDpfkLV/XPgOLDea/8GsJfQGQzvAf854j15hFbSQ8A/4130l4hcfi6rKBn/BdgD7Cb0RzG0vYwJWs8SuhxayfBlb91531uPFnrtXwLeAQ54PwclIMu/Etrt2eCtU99qLQeh3R7PestuDxFnsiUol6/rFPC3hHbz7I74nLonGZZXrA9dgSwiIj1jN5GIiHQvFQMREVExEBERFQMREUHFQEREUDEQERFUDEREBBUDEREB/j/aUFAmStfjawAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Create a scatter plot of the pojection \n",
    "pyplot.scatter(X_embedded[:, 0], X_embedded[:, 1])\n",
    "#words = list(model.wv.vocab)\n",
    "for i, word in enumerate(words):\n",
    "    pyplot.annotate(word, xy=(X_embedded[i, 0], X_embedded[i, 1]))\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.decomposition import TruncatedSVD\n",
    "svd = TruncatedSVD(n_components=2, n_iter=7, random_state=42)\n",
    "res = svd.fit_transform(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "ename": "IndexError",
     "evalue": "index 14 is out of bounds for axis 0 with size 14",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mIndexError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-77-089ac40ef759>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0;31m#words = list(model.wv.vocab)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mword\u001b[0m \u001b[0;32min\u001b[0m \u001b[0menumerate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mwords\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m     \u001b[0mpyplot\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mannotate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mword\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mxy\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mres\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mres\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      6\u001b[0m \u001b[0mpyplot\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshow\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mIndexError\u001b[0m: index 14 is out of bounds for axis 0 with size 14"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD8CAYAAABzTgP2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAHdRJREFUeJzt3X10VOXZ7/HvZYAYQIy8VCFgkyqlkKhQBqS1tiwpYGsF5MFHLEvxaOV4PHa1p6fUcNBHSvsoSleLLLCWpRRqtVBZiqkvh+MLVEGrmRiUlxp5SwsJRTSgIkFJvM4f2ckzO0xIYIbMJPl91sqa2ffck33dBPhl733fe8zdERERqXdaqgsQEZH0omAQEZEQBYOIiIQoGEREJETBICIiIQoGEREJUTCIiEiIgkFEREIUDCIiEtIp1QWcjN69e3tubm6qyxARaVNKSkred/c+zfVrk8GQm5tLNBpNdRkiIm2Kmf2jJf10KklEREIUDCIiEqJgEBGREAWDiIiEKBhERCREwSAiIiEKBhERCVEwiIhIiIJBRERCFAwiIhKiYBARkRAFg4iIhCgYREQkJCnBYGaXm1mZmW03s8I4r2ea2crg9dfNLDdoH2lmG4Ovt8zsqmTUIyIiJy/hYDCzDGAx8B1gCHCtmQ1p1O0m4IC7nw/8Brg3aN8MRNx9KHA58Dsza5O3AhcRaS+SccQwEtju7jvd/TNgBTCxUZ+JwPLg+SpgjJmZux9295qg/XTAk1CPiIgkIBnBkAPsjtneE7TF7RMEwYdALwAzu9jMtgCbgFtigkJERFIgGcFgcdoa/+bfZB93f93d84ERwCwzOz3uTsxmmFnUzKL79+9PqGAREWlaMoJhDzAgZrs/UNlUn+AawplAVWwHd/878AlQEG8n7r7E3SPuHunTp9mPLBURkZOUjGAoBgaaWZ6ZdQGmAkWN+hQB04PnU4CX3N2D93QCMLMvAoOA8iTUJCIiJynhGUDuXmNmtwFrgAxgqbtvMbO5QNTdi4CHgUfMbDt1RwpTg7d/Ayg0s6PA58Ct7v5+ojWJiMjJM/e2NxEoEol4NBpNdRkiIm2KmZW4e6S5flr5LCIiIQoGEREJUTCIiEiIgkFEREIUDCIiEqJgEBGREAWDiIiEKBhERCREwSAiIiEKBhERCVEwiIhIiIJBRERCFAwiIhKiYBARkRAFg4iIhCgYREQkRMEgIiIhCgYREQlRMIiISIiCQUREQhQMIiISomAQEZEQBYOIiIQoGEREJETBIK3u4MGDPPDAA6kuQ0SakJRgMLPLzazMzLabWWGc1zPNbGXw+utmlhu0jzWzEjPbFDxelox6JL0pGETSW6dEv4GZZQCLgbHAHqDYzIrcfWtMt5uAA+5+vplNBe4FrgHeB65090ozKwDWADmJ1iTprbCwkB07djB06FDGjh0LwHPPPYeZcccdd3DNNdekuEKRji0ZRwwjge3uvtPdPwNWABMb9ZkILA+erwLGmJm5e6m7VwbtW4DTzSwzCTVJGps3bx7nnXceGzduZNSoUWzcuJG33nqLF154gZkzZ7J3795UlyjSoSUjGHKA3THbezj2t/6GPu5eA3wI9GrU59+AUnf/NN5OzGyGmUXNLLp///4klC3pYP369Vx77bVkZGRw9tln861vfYvi4uJUlyXSoSV8KgmwOG1+In3MLJ+600vjmtqJuy8BlgBEIpHG31/agNWlFcxfU8Y//lFO1fufsLq0Anf9KEXSTTKOGPYAA2K2+wOVTfUxs07AmUBVsN0feBK43t13JKEeSUOrSyuY9cQmKg5WY12y+Kz6E2Y9sYnM/vmsXLmS2tpa9u/fz8svv8zIkSOTss/c3NykfB+RjiYZRwzFwEAzywMqgKnA9xv1KQKmA68BU4CX3N3NLBt4Bpjl7huSUIukqflryqg+WgtARlYPMnOGsOPB/84fB4/i+xdfyEUXXYSZcd9993HOOeekuFqRji3hYHD3GjO7jboZRRnAUnffYmZzgai7FwEPA4+Y2XbqjhSmBm+/DTgfuNPM7gzaxrn7e4nWJeml8mB1aLvPhJlA3TnG+fOuYP78+UnfZ58+fQDYu3cv11xzDR999BE1NTX89re/5dJLL036/kTai2QcMeDuzwLPNmr7j5jnR4Cr47zvl8Avk1GDpLd+2VlUNAqH+vZTpf4i9mOPPcb48eOZPXs2tbW1HD58+JTtU6Q90MpnaRUzxw8iq3NGqC2rcwYzxw865fseMWIEv//975kzZw6bNm3ijDPOOOX7FGnLFAzSKiYNy+GeyReQk52FATnZWdwz+QImDUvuesbVpRVcMu8l8gqf4ZJ5L7G6tIJvfvObvPzyy+Tk5HDdddfxhz/8Ian7FGlvrC1OF4xEIh6NRlNdhqSZ+plP9Re5oe6o5H+NOosbxw+nU6dOLFiwgPLychYsWJDCSkVSw8xK3D3SXL+kXGMQSQexM5/qVR+tZf6yJ1h4+0107tyZ7t2764hBpBkKBmk3Gs98qldz3jfZ9vi9rVyNSNulawzSbjQ1w+lUznwSaY8UDNJupHLmk0h7olNJ0m7Uz3Cav6aMyoPV9MvOYub4QUmf+STS3ikYpF2ZNCxHQSCSIJ1KEhGREAWDiIiEKBhERCREwSAiIiEKBhERCVEwiIhIiIJBRERCFAwiIhKiYBARkRAFg4iIhCgYREQkRMEgIiIhCgYREQlRMIiISIiCQUREQpISDGZ2uZmVmdl2MyuM83qmma0MXn/dzHKD9l5mttbMDpnZomTUIiLHWrhwIYMHD2batGmpLkXagIQ/qMfMMoDFwFhgD1BsZkXuvjWm203AAXc/38ymAvcC1wBHgDuBguBLRE6BBx54gOeee468vLxm+9bU1NCpkz7DqyNLxk9/JLDd3XcCmNkKYCIQGwwTgTnB81XAIjMzd/8EWG9m5yehDhGJ45ZbbmHnzp1MmDCBG264gVdeeYWdO3fStWtXlixZwoUXXsicOXOorKykvLyc3r1789hjj6W6bEmhZJxKygF2x2zvCdri9nH3GuBDoNeJ7MTMZphZ1Myi+/fvT6BckY7lwQcfpF+/fqxdu5by8nKGDRvG22+/zd13383111/f0K+kpISnnnpKoSBJCQaL0+Yn0ee43H2Ju0fcPdKnT58TeauIBNavX891110HwGWXXcYHH3zAhx9+CMCECRPIyspKZXmSJpJxKmkPMCBmuz9Q2USfPWbWCTgTqErCvkWkCatLK5i/pozKg9X868MjPPv2XtyP/X3MrO73tm7durV2iZKmknHEUAwMNLM8M+sCTAWKGvUpAqYHz6cAL3m8v6EikhSrSyuY9cQmKg5W40DN584vntlK/8Ff5dFHHwVg3bp19O7dmx49eqS2WEk7CR8xuHuNmd0GrAEygKXuvsXM5gJRdy8CHgYeMbPt1B0pTK1/v5mVAz2ALmY2CRjXaEaTiJyg+WvKqD5aG2o7crSW9wZOIBr9PRdeeCFdu3Zl+fLlKapQ0pm1xV/cI5GIR6PRVJchkrbyCp+JexHPgF3zrmjtciRNmFmJu0ea66eVzyKtqHv37q2yn37Z8S8iN9UuEkvBINIOzRw/iKzOGaG2rM4ZzBw/KEUVSVuiYBA5QZMmTWL48OHk5+ezZMkSoO5IYPbs2Vx00UWMGjWKffv2AbBr1y6+9rWvMWLECO68887Wq3FYDvdMvoCc7CwMyMnO4p7JFzBpWOMlRiLH0jUGkRNUVVVFz549qa6uZsSIEfz1r3+ld+/eFBUVceWVV/Kzn/2MHj16cMcddzBhwgSmTJnC9ddfz+LFi7n99ts5dOhQqocgHZSuMYicIgsXLmw4Mti9ezfbtm2jS5cufO973wNg+PDhlJeXA7BhwwauvfZagIaFZSLpTnfKEmlG7EKxblVl1BY/S8lrr9G1a1dGjx7NkSNH6Ny5c8NCsYyMDGpqahreX98u0lboiEHkOBovFHvvgwPs/sT4f2UHeOedd/jb3/523PdfcsklrFixAqBhYZlIulMwiBxH44ViWXnDqa2pZdp3L+XOO+9k1KhRx33//fffz+LFixkxYkTDPYlE0p0uPoschxaKSXuii88iSaCFYtIRKRhEjkMLxaQj0qwkkeOoXxBWPyupX3YWM8cP0kKxNBM7c0w/o8QpGESaMWlYjv6TSWP1M8fqJwlUHKxm1hObAPRzO0k6lSQibVrjmWP7VvwfPq7ax/w1ZSmsqm1TMIhIm1Z5sLrhufvnHD2wl9NOPyPULidGwSAibVrsDLGj7/+Trl/+Oqd1ztTMsQQoGESkTYudOdalTy49x9ysmWMJ0sVnEWnTNHMs+RQMItLmaeZYculUkoiIhCgYRKTB17/+9VSXIGlAwSAiDV599dVUlyBpQMEgIg26d+8OwLp16xg9ejRTpkzhK1/5CtOmTaMt3olZTo6CoZ2K94H1IieitLSUBQsWsHXrVnbu3MmGDRtSXZK0kqQEg5ldbmZlZrbdzArjvJ5pZiuD1183s9yY12YF7WVmNj4Z9QgsXbqUkpISotEoCxcu5IMPPkh1SdLGjBw5kv79+3PaaacxdOjQhs+xlvYv4emqZpYBLAbGAnuAYjMrcvetMd1uAg64+/lmNhW4F7jGzIYAU4F8oB/wgpl92d1rkRMWe4fJmuif6fTPYnpkdW74wPpevXqlukRJQ7F/b6qP1rK6tIJsIDMzs6FP48+xlvYtGUcMI4Ht7r7T3T8DVgATG/WZCCwPnq8CxljdJ6RPBFa4+6fuvgvYHnw/OUGxn01c/c+32f9OlC6T7+bny55l2LBhHDlyJNUldjirSyu4ZN5L5BU+wyXzXmJ1aUWqSzpG48+0dodZT2xi/bb9qS5NUigZwZAD7I7Z3hO0xe3j7jXAh0CvFr4XADObYWZRM4vu36+/tI3F3mHy808Pc9rp3fiUzvz8keeb/cB6Sb7G/+HW3wo63cKh8Z1JAaqP1rKieHcT75COIBnBYHHaGk9faKpPS95b1+i+xN0j7h7p06fPCZbY/sXeSTIrbzj++edULr2Nd599qNkPrJfkO+ZW0I/fxcdV7/GT/7iHwYMHM23atBRW918a34H03J+sAuCTnoN4+umnG9oXLVrEDTfc0JqlSQol45YYe4ABMdv9gcom+uwxs07AmUBVC98rLdAvO4uK4B+5derM2f/+cwBysrNYV3hZKkvrkBr/h3v21XU/j92vPsW70ZfJy8tLRVnHiP1707hdOq5kHDEUAwPNLM/MulB3MbmoUZ8iYHrwfArwktdNii4CpgazlvKAgcAbSaipw9FnE6eXftlZfPTGk1Q+fCuVD9/KR8VP8cGaRdR++C8mTJjAb37zm1SXCOjvjcSX8BGDu9eY2W3AGiADWOruW8xsLhB19yLgYeARM9tO3ZHC1OC9W8zsz8BWoAb4n5qRdHJ0h8n0MnnAp8ze/CLnXPdrwPnXH/43OVfNJGvfZtauXUvv3r1TXSKgvzcSn7XF1YyRSMSj0WiqyxBp0v3338+GLbuo+NKEuunDb6xgfGQgL/x5KdFoNG2CQToWMytx90hz/bTyWeQk5ebmUl5ezujRoxva6qeo/rxoC6/t+ICZ4wexa94V3HBJLhf2z05dsSInQMEgkiSxU1QzB+Szb9N6bl9ZzIpXt/Hkk09y6aWXprpEkRbRB/WInKQ+ffqQkZFBz549gfAU1cxzzqd7wRh2Pfwjblx2GncX/ohhw4alslyRFtM1BpEkySt8Ju4iHAN2zbuitcsROUZLrzHoiEHkBMTeV6jxDB6tCZD2QtcYRFqoudtcaE2AtBcKBpEWauq+QvPXlAF1awLumXwBOdlZGHWrzu+ZfIHWBEibo1NJIi3U+DYX8donDctREEibpyMGkRZq6lqBriFIe6NgEGkhXUOQjkKnkkRaSPcVko5CwSByAnQNQToCnUoSEZEQBYOIiIQoGEREJETBICIiIQoGEREJUTCIiEiIgkFEREIUDO1I/cdK9hr3P+jW73zyBuVTWVmZ6rJEpI3RArd2ov6W0NVHaznjq9/jjK9+j9M7Z/DGPmdSv1RXJyJtiY4Y2onmbgktItJSCoZ2oiW3hBYRaQkFQzuhW0KLSLIkFAxm1tPMnjezbcHjWU30mx702WZm02Pa/9PMdpvZoUTqEN0SWkSSJ9EjhkLgRXcfCLwYbIeYWU/gLuBiYCRwV0yA/CVokwSlw8dK/vrXv6agoICCggIWLFhAeXk5gwcP5uabbyY/P59x48ZRXa1TWyJpz91P+gsoA/oGz/sCZXH6XAv8Lmb7d8C1jfocOpH9Dh8+3CW9RKNRLygo8EOHDvnHH3/sQ4YM8TfffNMzMjK8tLTU3d2vvvpqf+SRR1Jcaftx4MABX7x4sbu7r1271q+44ooUVyTpDoh6C/6PTfSI4Wx33xsEzF7gC3H65AC7Y7b3BG3Sjqxfv56rrrqKbt260b17dyZPnswrr7xCXl4eQ4cOBWD48OGUl5enttB25ODBgzzwwAOpLkPaoWbXMZjZC8A5cV6a3cJ9WJw2b+F7Y+uYAcwAOPfcc0/07XKKrC6tYP6aMv7+/Ba6Uc1XSytCp68yMzMbnmdkZOhUUhIVFhayY8cOhg4dSufOnenWrRtTpkxh8+bNDB8+nD/+8Y+YGSUlJfzkJz/h0KFD9O7dm2XLltG3b99Uly9prNkjBnf/trsXxPl6CthnZn0Bgsf34nyLPcCAmO3+wAkvx3X3Je4ecfdInz59TvTtcgrUL6qrOFhN5oB89m1az+0ri1nx6jaefPJJLr300lSX2K7NmzeP8847j40bNzJ//nxKS0tZsGABW7duZefOnWzYsIGjR4/ywx/+kFWrVlFSUsKNN97I7Nkt/Z1OOqpEVz4XAdOBecHjU3H6rAHujrngPA6YleB+JQ3ELqrLPOd8uheMYdfDP+LGZadxd+GPOOusuJPUJEH1R2n/+Ec5Ve9/wurSCrKBkSNH0r9/fwCGDh1KeXk52dnZbN68mbFjxwJQW1urowVpVqLBMA/4s5ndBPwTuBrAzCLALe7+A3evMrNfAMXBe+a6e1XQ7z7g+0BXM9sDPOTucxKsSVpJ48VzPUZeRY+RV2HAj398BQCbN29ueP2nP/1pa5bXLsXe+gSgpvZzZj2xiWnnfnzMabuPPvqIP/3pT+Tn53PPPffwq1/9iqeffrrF+1q2bBnjxo2jXz/dU6WjSSgY3P0DYEyc9ijwg5jtpcDSOP1+BvwskRokdfplZ1ERZ2W1FtWdOrFHadYli88/q6b6aC0rineT26jv4cOHWb16NZ9++ilbtmwB4OjRo7z77rvk5+c3u69ly5ZRUFCgYOiAtPJZTpoW1bW+2KO0jKweZOYMofLhW9n2lweP6fv444+zc+dOMjIyKCwsZN26dfTq1YsxY8Ywbdq0+qnizJ07lxEjRlBQUMCMGTNwd1atWkU0GmXatGkMHTpUkwY6GAWDnLR0WFTX0TQ+GuszYSb9bnqAyI8eDJ0mWrRoEStXruS8886jrKyMv/zlL2RkZLB161YqKysbLk4D3HbbbRQXF7N582aqq6t5+umnmTJlCpFIhEcffZSNGzeSlaWjwI5Et92WhEwalqMgaEUzxw8KXWOAY4/STuTi9De+8Q3Wrl3Lfffdx+HDh6mqqiI/P58rr7yytYcmaUTBINKG1Ifw/DVlVB6spl92FjPHD2poP5GL0zU1NRw5coRbb72VaDTKgAEDmDNnDkeOHGn9gUlaUTCItDHHO0o7kYvTQEMI9O7dm0OHDrFq1SqmTJkCwBlnnMHHH398KoYgaU7BINKONHVx2jplkjv8y8f0z87O5uabb+aCCy4gNzeXESNGNLx2ww03cMstt5CVlcVrr72m6wwdiNXPTGhLIpGIR6PRVJchknYumfdS3CnEOdlZbCi8LAUVSToxsxJ3jzTXT7OSRNoRTSGWZNCpJJF2pLmL0yItoWAQaWc0hVgSpVNJIiISomAQEZEQBYOIiIQoGEREJETBICIiIQoGEREJUTCIiEiIgkFEREIUDCIiEqJgEBGREAWDiIiEKBhERCREwSAiIiEKBhERCUkoGMysp5k9b2bbgsezmug3PeizzcymB21dzewZM3vHzLaY2bxEahERkeRI9IihEHjR3QcCLwbbIWbWE7gLuBgYCdwVEyC/cvevAMOAS8zsOwnWIyIiCUo0GCYCy4Pny4FJcfqMB5539yp3PwA8D1zu7ofdfS2Au38GvAn0T7AeERFJUKLBcLa77wUIHr8Qp08OsDtme0/Q1sDMsoErqTvqEBGRFGr2oz3N7AXgnDgvzW7hPixOm8d8/07An4CF7r7zOHXMAGYAnHvuuS3ctYiInKhmg8Hdv93Ua2a2z8z6uvteM+sLvBen2x5gdMx2f2BdzPYSYJu7L2imjiVBXyKRiB+vr4iInLxETyUVAdOD59OBp+L0WQOMM7OzgovO44I2zOyXwJnAjxOsQ0REkiTRYJgHjDWzbcDYYBszi5jZQwDuXgX8AigOvua6e5WZ9afudNQQ4E0z22hmP0iwHhERSZC5t72zMpFIxKPRaKrLEBFpU8ysxN0jzfXTymcREQlRMIiISIiCQUREQhQMIiISomAQEZEQBYOIiIQoGEREJETBICIiIQoGEREJUTCIiEiIgkFEREIUDCIiEqJgEBGREAWDiIiEKBhERCREwSAiIiEKBhERCVEwiIhIiIJBRERCFAwiIhKiYBARkRAFg4iIhCgYREQkRMEgIiIhCQWDmfU0s+fNbFvweFYT/aYHfbaZ2fSY9v9rZm+Z2RYze9DMMhKpR0REEpfoEUMh8KK7DwReDLZDzKwncBdwMTASuCsmQP7d3S8CCoA+wNUJ1iMiIglKNBgmAsuD58uBSXH6jAeed/cqdz8APA9cDuDuHwV9OgFdAE+wHhERSVCiwXC2u+8FCB6/EKdPDrA7ZntP0AaAma0B3gM+BlYlWI+IiCSoU3MdzOwF4Jw4L81u4T4sTlvDkYG7jzez04FHgcuoO6KIV8cMYEawecjMylq4//akN/B+qotIkY48dujY49fYk+eLLenUbDC4+7ebes3M9plZX3ffa2Z9qfvNv7E9wOiY7f7Aukb7OGJmRdSdmoobDO6+BFjSXL3tmZlF3T2S6jpSoSOPHTr2+DX21h97oqeSioD6WUbTgafi9FkDjDOzs4KLzuOANWbWPQgTzKwT8F3gnQTrERGRBCUaDPOAsWa2DRgbbGNmETN7CMDdq4BfAMXB19ygrRtQZGZvA29Rd7TxYIL1iIhIgpo9lXQ87v4BMCZOexT4Qcz2UmBpoz77gBGJ7L8D6sin0jry2KFjj19jb2XmrhmiIiLyX3RLDBERCVEwpJkk3GbkP81st5kdar2qE2Nml5tZmZltN7N4q+czzWxl8PrrZpYb89qsoL3MzMa3Zt3JcLJjN7NeZrbWzA6Z2aLWrjsZEhj7WDMrMbNNweNlrV17MiQw/pFmtjH4esvMrkp6ce6urzT6Au4DCoPnhcC9cfr0BHYGj2cFz88KXhsF9AUOpXosLRxvBrAD+BJ1q9/fAoY06nMr8GDwfCqwMng+JOifCeQF3ycj1WNqpbF3A74B3AIsSvVYWnnsw4B+wfMCoCLV42nl8XcFOgXP65cJdEpmfTpiSD+J3mbkbx6sRm8jRgLb3X2nu38GrKDuzyBW7J/JKmCMmVnQvsLdP3X3XcD24Pu1FSc9dnf/xN3XA0dar9ykSmTspe5eGbRvAU43s8xWqTp5Ehn/YXevCdpP5xTcSkjBkH4Svs1IG9OSsTT0Cf5BfAj0auF701kiY2/rkjX2fwNK3f3TU1TnqZLQ+M3sYjPbAmwCbokJiqRIaLqqnJxTfZuRNqYlY2mqT1v/c0hk7G1dwmM3s3zgXuoWzbY1CY3f3V8H8s1sMLDczJ5z96QdPSoYUsBb4TYjbcgeYEDMdn+gsok+e4JV8mcCVS18bzpLZOxtXUJjN7P+wJPA9e6+49SXm3RJ+dm7+9/N7BPqrrVEk1WcTiWln5O+zUgr1ZdsxcBAM8szsy7UXWQratQn9s9kCvCS1115KwKmBrM38oCBwButVHcyJDL2tu6kx25m2cAzwCx339BqFSdXIuPPC4ICM/siMAgoT2p1qb46r69jZiv0ou5Dj7YFjz2D9gjwUEy/G6m72Lod+G8x7fdR95vG58HjnFSPqQVj/i7wLnWzNGYHbXOBCcHz04HHg7G+AXwp5r2zg/eVAd9J9Vhaeezl1P0GeSj4WQ9p7fpTMXbgDuATYGPM1xdSPZ5WHP911F103wi8CUxKdm1a+SwiIiE6lSQiIiEKBhERCVEwiIhIiIJBRERCFAwiIhKiYBARkRAFg4iIhCgYREQk5P8DV7WY0EscxgQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Create a scatter plot of the pojection \n",
    "pyplot.scatter(res[:, 0], res[:, 1])\n",
    "#words = list(model.wv.vocab)\n",
    "for i, word in enumerate(words):\n",
    "    pyplot.annotate(word, xy=(res[i, 0], res[i, 1]))\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Turns out TSNE and Truncated SVD dont work as well as PCA for data like this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5. Load Google's Word2Vec Embedding."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training your own word vectors may be the best approach for a given NLP problem. But it can take a long time, a fast computer with a lot of RAM and disk space, and perhaps some expertise in finessing the input data and training algorithm. An alternative is to simply use an existing pre-trained word embedding.\n",
    "\n",
    "A pre-trained model is nothing more tha a file containing tokens and their associated word vectors.\n",
    "The pre-trained Google Word2Vec model was trained on Google news data (about 100 billion words); it contains 3 million words and phrases and was fit using 300-dimensional word vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2020-03-29 16:08:27--  https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit?usp=sharing\n",
      "Resolving drive.google.com (drive.google.com)... 216.58.223.78, 2a00:1450:401a:804::200e\n",
      "Connecting to drive.google.com (drive.google.com)|216.58.223.78|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: unspecified [text/html]\n",
      "Saving to: ‘Models/GoogleNews_vectors_negative300.bin.gz’\n",
      "\n",
      "Models/GoogleNews_v     [ <=>                ]  67.26K  --.-KB/s    in 0.1s    \n",
      "\n",
      "2020-03-29 16:08:29 (474 KB/s) - ‘Models/GoogleNews_vectors_negative300.bin.gz’ saved [68879]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Download the embeddings \n",
    "!wget -O Models/GoogleNews_vectors_negative300.bin.gz https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit?usp=sharing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GoogleNews-vectors-negative300.bin  model_wv.bin  model_wv.txt\tword2vec.model\r\n"
     ]
    }
   ],
   "source": [
    "!ls Models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "# unzip the file downloaded. Alternatively use the gensim loader to download the file\n",
    "!gunzip Models/GoogleNews-vectors-negative300.bin.gz\n",
    "\n",
    "\n",
    "# def load_word2vec():\n",
    "#     \"\"\" Load Word2Vec Vectors\n",
    "#         Return:\n",
    "#             wv_from_bin: All 3 million embeddings, each lengh 300\n",
    "#     \"\"\"\n",
    "#     import gensim.downloader as api\n",
    "#     wv_from_bin = api.load(\"word2vec-google-news-300\")\n",
    "#     vocab = list(wv_from_bin.vocab.keys())\n",
    "#     print(\"Loaded vocab size %i\" % len(vocab))\n",
    "#     return wv_from_bin"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load gensim keyed vectors to take the binary vectors file.\n",
    "from gensim.models import KeyedVectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the google model\n",
    "filename = 'Models/GoogleNews-vectors-negative300.bin'\n",
    "model = KeyedVectors.load_word2vec_format(filename, binary=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can try some tests to see our how our model works with predicting most similar words given a word."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('queen', 0.711819589138031)]\n"
     ]
    }
   ],
   "source": [
    "# Calculate: (king - man) + woman = ?\n",
    "result = model.most_similar(positive=['woman', 'king'], negative=['man'], topn=1)\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is how we'd load and use the google word embedding. for our various problems. Now let's extract the loaded embedding from memory since we'll have to load on the next embedding. If we were working from the cloud vm instance, we may not need to pull it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop the model from ram.\n",
    "del model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6. Load Stanford's GloVe Embedding."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NLP Practitioners prefer GloVe at the moment based on results. It is also a pre-trained word vector. \n",
    "\n",
    "We can download the GloVe pretrained word vectors and load them easily with Gensim. The first step convert the GloVe file format to the Word2Vec file format.\n",
    "\n",
    "The only difference is the addition of a small header line. This can be done by calling the glove2word2vec() function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The smallest GloVe pre-trained model is an 822 Mb zip file with 4 different models (50, 100, 200 and 300-dimensional vectors) trained on data with 6 billion tokens and a 400,000 word vocabulary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import module\n",
    "from gensim.scripts.glove2word2vec import glove2word2vec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2020-03-30 04:03:41--  http://nlp.stanford.edu/data/glove.6B.zip\n",
      "Resolving nlp.stanford.edu (nlp.stanford.edu)... 171.64.67.140\n",
      "Connecting to nlp.stanford.edu (nlp.stanford.edu)|171.64.67.140|:80... connected.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: https://nlp.stanford.edu/data/glove.6B.zip [following]\n",
      "--2020-03-30 04:03:43--  https://nlp.stanford.edu/data/glove.6B.zip\n",
      "Connecting to nlp.stanford.edu (nlp.stanford.edu)|171.64.67.140|:443... connected.\n",
      "HTTP request sent, awaiting response... 301 Moved Permanently\n",
      "Location: http://downloads.cs.stanford.edu/nlp/data/glove.6B.zip [following]\n",
      "--2020-03-30 04:03:44--  http://downloads.cs.stanford.edu/nlp/data/glove.6B.zip\n",
      "Resolving downloads.cs.stanford.edu (downloads.cs.stanford.edu)... 171.64.64.22\n",
      "Connecting to downloads.cs.stanford.edu (downloads.cs.stanford.edu)|171.64.64.22|:80... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 862182613 (822M) [application/zip]\n",
      "Saving to: ‘Models/glove.6B.zip’\n",
      "\n",
      "Models/glove.6B.zip 100%[===================>] 822.24M  1.22MB/s    in 53m 39s \n",
      "\n",
      "2020-03-30 04:57:25 (262 KB/s) - ‘Models/glove.6B.zip’ saved [862182613/862182613]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Download the embedding\n",
    "!wget -O Models/glove.6B.zip http://downloads.cs.stanford.edu/nlp/data/glove.6B.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Archive:  Models/glove.6B.zip\n",
      "  inflating: glove.6B.50d.txt        \n",
      "  inflating: glove.6B.100d.txt       \n",
      "  inflating: glove.6B.200d.txt       \n",
      "  inflating: glove.6B.300d.txt       \n"
     ]
    }
   ],
   "source": [
    "# Unzip the donwloaded files.\n",
    "!unzip Models/glove.6B.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'1. Deep Learning NLP.ipynb'\t\t\t  main.py\r\n",
      "'2. NLP Data Preparation.ipynb'\t\t\t  Model_Imgaes\r\n",
      "'3. Bag-of-Words Model Keras.ipynb'\t\t  model.png\r\n",
      "'4. Sentiment Analysis Data Preparation..ipynb'   Models\r\n",
      "'5. Neural BOW Model for Sentiments..ipynb'\t 'NLP Quiz'\r\n",
      "'6. Word Embeddings.ipynb'\t\t\t  poldata.README.2.0\r\n",
      "'7. Developing Word Embedding.ipynb'\t\t  __pycache__\r\n",
      " Data\t\t\t\t\t\t  requirements.txt\r\n",
      " glove.6B.100d.txt\t\t\t\t  save_processed.py\r\n",
      " glove.6B.200d.txt\t\t\t\t  sent_model1.png\r\n",
      " glove.6B.300d.txt\t\t\t\t  text_prep.py\r\n",
      " glove.6B.50d.txt\t\t\t\t  Weights\r\n"
     ]
    }
   ],
   "source": [
    "!ls "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(400000, 100)"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's use the 100d model\n",
    "glove_input_file = 'Models/glove.6B.100d.txt'\n",
    "word2vec_output_file = 'Models/glove.6B.100d.txt.word2vec'\n",
    "glove2word2vec(glove_input_file, word2vec_output_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have a copy of the GloVe model in Word2Vec format with the filename glove.6B.100d.txt.word2vec. now we can load it and perform the same test we performed on the word2vec model above.\n",
    "\n",
    "**Note** The converted file is ASCII format, not binary, so we set binary=False when loading."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('queen', 0.7698541283607483)]\n"
     ]
    }
   ],
   "source": [
    "# Load modules\n",
    "from gensim.models import KeyedVectors\n",
    "# load the stanford GloVe model\n",
    "filename = 'Models/glove.6B.100d.txt.word2vec'\n",
    "model = KeyedVectors.load_word2vec_format(filename, binary = False)\n",
    "# Calculate: (king - man) + woman = ?\n",
    "result = model.most_similar(positive=['woman', 'king'], negative=['man'], topn=1)\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to Learn and Load Word Embeddings in Keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Word embeddings provide a dense representation of words and their relative meanings. They are an improvement over sparse representations used in simpler BOW model representations.\n",
    "\n",
    "They can be learned from text data and reused among other projects. They can also be learned as part of fitting a neural network on text data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Word Embedding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a class of approaches for representing words and documents using a dense vector representation. It is an improvement over more the traditional BOW model encoding schemes where large sparse vectors were used to represent each word or to score eac word within a vector to represent an entire vocabulary.\n",
    "\n",
    "We will get sparse representations because the vocabularies were vast and a given word or document would be represented by a large vector comprised mostly of zero values.\n",
    "\n",
    "In an embedding, words are represented by dense vectors where a vector represents the projection of the word into a continuous vector space. The position of a word within the vector space is learned from text and is based on the words that surround the word when it is used. The position of a word in a learned vector space is known as it's embedding. Two popular examples of methods of learning word embeddings from text we have seen above include:\n",
    "\n",
    "    1. Word2Vec\n",
    "    2. GloVe.\n",
    "    \n",
    "In addition to these methods, a word embedding can be learned as part of a deep learning model. This can be a slower approach, but tailors the model to a specific training dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Keras Embedding Layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The keras embedding layer requires that the input data be integer encoded, so that each word is represented by a unique integer. This data preparation step can be performed using the Tokenizer API.\n",
    "\n",
    "The embedding layer is initialized with random weights and will learn an embedding for all of the words in the training set.\n",
    "\n",
    "It's also quite flexible and can be used in a variety of ways, such as:\n",
    "    \n",
    "    1. It can be used alone to learn a word embedding that can be saved and used in another model later.\n",
    "    2. It can be used as part of a deep learning model where the embedding is learned along with the model itself.\n",
    "    3. It can be used to load a pre-trained word embedding model, a type of transfer learning.\n",
    "    \n",
    "    The embedding layer is defined as the first hidden layer of a network, with the following arguments as compulsory:\n",
    "        \n",
    "    1. input_dim. This is the size of the vocabulary in the text data. For example, if your data is integer encoded to values between 0-10, then the size of the vocab would be 11 words.\n",
    "    2. output_dim. This is the size of the vector space in which words will be embedded. it defines the size of the output vectors from this layer for each word. For example, it could be 32, 100, or even larger.\n",
    "    3. Input length. This is the length of input sequences, as you would define for any input layer of a keras model.\n",
    "    \n",
    "**Note:** The Embedding layer has weights that are learned if you save your model to file, this will include weights for the Embedding layer. Saving your model to file means this will include weights for the embedding layer. \n",
    "    \n",
    "    \n",
    "    The output of the embedding layer is a 2D vector with one embedding for each word in the input sequence of words (input document). If you wish to connect a dense layer directly to the embedding layer, you must first flatten the 2D output matrix to a 1D vector using the Flatten layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Example of Learning an Embedding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Lets define a simple sentiment analysis problem and see how we can scale up to much larger problems.\n",
    "#Define documents\n",
    "docs = ['Well done!', \n",
    "        'Good work', \n",
    "        'Great effort', \n",
    "        'Nice work', \n",
    "        'Excellent!', \n",
    "        'Weak', \n",
    "        'Poor effort!',\n",
    "        'not good', \n",
    "        'poor work',\n",
    "        'could have done better.']\n",
    "# Define class labels\n",
    "labels = [1,1,1,1,1,0,0,0,0,0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we integer encode each document (one_hot). This means that as input, the embedding layer will have some sequences of integers. Note, we can experiment with other more sophisticated BOW model encoding like counts or TF-IDF.\n",
    "\n",
    "We will estimate the vocabulary of 50 , which is much larger than needed to reduce the probability of collisions from the hash function.\n",
    "\n",
    "Finally, the sequences have different lengths and keras preferes inputs to be vectorized and all inputs to have the same length. We will pad all input sequences to have the length  of 4.\n",
    "\n",
    "Then we define, compile and fit our model keeping the embedding layer at the formost after the input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import modules\n",
    "from keras.preprocessing.text import one_hot\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "from keras.models import Model\n",
    "from keras.layers import Input, Dense, Flatten\n",
    "from keras.layers.embeddings import Embedding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "18"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "h = [i.split() for i in docs]\n",
    "k = [j for i in h for j in i]\n",
    "len(set(k)) # Vocab size should be this length."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[8, 19], [39, 9], [33, 1], [33, 9], [17], [29], [26, 1], [36, 39], [26, 9], [11, 20, 19, 18]]\n"
     ]
    }
   ],
   "source": [
    "# Integer encode the documents\n",
    "vocab_size = 50 # We are choosing 50 because it is quite large and will help us avoid any hashing collisions.\n",
    "encoded_docs = [one_hot(d, vocab_size) for d in docs]\n",
    "print(encoded_docs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the max length of a sentences in the vocab.\n",
    "max_len =0\n",
    "for i in encoded_docs:\n",
    "    if len(i) > max_len:\n",
    "        max_len = len(i)\n",
    "max_len"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 8 19  0  0]\n",
      " [39  9  0  0]\n",
      " [33  1  0  0]\n",
      " [33  9  0  0]\n",
      " [17  0  0  0]\n",
      " [29  0  0  0]\n",
      " [26  1  0  0]\n",
      " [36 39  0  0]\n",
      " [26  9  0  0]\n",
      " [11 20 19 18]]\n"
     ]
    }
   ],
   "source": [
    "# Pad documents.\n",
    "padded_docs = pad_sequences(encoded_docs, maxlen=max_len, padding='post')\n",
    "print(padded_docs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_2\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_3 (InputLayer)         (None, 4)                 0         \n",
      "_________________________________________________________________\n",
      "embedding_2 (Embedding)      (None, 4, 8)              400       \n",
      "_________________________________________________________________\n",
      "flatten_2 (Flatten)          (None, 32)                0         \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 1)                 33        \n",
      "=================================================================\n",
      "Total params: 433\n",
      "Trainable params: 433\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# Define a model\n",
    "def simp_model(vocab_size, max_len, padded_docs, labels):\n",
    "    visible = Input(shape=(max_len, ))\n",
    "    embed1 = Embedding(vocab_size, 8)(visible)\n",
    "    flat1 = Flatten()(embed1)\n",
    "    output = Dense(1, activation='sigmoid')(flat1)\n",
    "    model = Model(inputs = visible, outputs = output)\n",
    "    model.summary()\n",
    "    return model\n",
    "\n",
    "# Load and compile the model\n",
    "model = simp_model(vocab_size, max_len, padded_docs, labels)  \n",
    "model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 100.000000\n"
     ]
    }
   ],
   "source": [
    "# fit the model\n",
    "model.fit(padded_docs, labels, epochs=50, verbose=0)\n",
    "# evaluate the model\n",
    "loss, accuracy = model.evaluate(padded_docs, labels, verbose=0)\n",
    "print('Accuracy: %f'%(accuracy*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can save the learned weights from the embedding layer to a file for later use in other models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4. Example of Using Pre-trained GloVe Embedding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have built an embedding for the task and seen it work for what we wanted it for. Let us attempt using a pre-trained embedding for the same problem.\n",
    "\n",
    "As seen in the previous example on loading a pre-trained embedding, we will do same here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(['Well done!',\n",
       "  'Good work',\n",
       "  'Great effort',\n",
       "  'Nice work',\n",
       "  'Excellent!',\n",
       "  'Weak',\n",
       "  'Poor effort!',\n",
       "  'not good',\n",
       "  'poor work',\n",
       "  'could have done better.'],\n",
       " [1, 1, 1, 1, 1, 0, 0, 0, 0, 0])"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Verify our docs and labels lists havent been altered.\n",
    "docs, labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[6, 2], [3, 1], [7, 4], [8, 1], [9], [10], [5, 4], [11, 3], [5, 1], [12, 13, 2, 14]]\n",
      "[[ 6  2  0  0]\n",
      " [ 3  1  0  0]\n",
      " [ 7  4  0  0]\n",
      " [ 8  1  0  0]\n",
      " [ 9  0  0  0]\n",
      " [10  0  0  0]\n",
      " [ 5  4  0  0]\n",
      " [11  3  0  0]\n",
      " [ 5  1  0  0]\n",
      " [12 13  2 14]]\n"
     ]
    }
   ],
   "source": [
    "# Let's run it all in one script\n",
    "from numpy import asarray, zeros\n",
    "from keras.preprocessing.text import Tokenizer\n",
    "from keras_preprocessing.sequence import pad_sequences\n",
    "from keras.models import Model\n",
    "from keras.layers import Input, Dense, Flatten, Embedding\n",
    "\n",
    "# Define documents: Already done, variable docs and labels\n",
    "\n",
    "# prepare tokenizer\n",
    "t = Tokenizer()\n",
    "t.fit_on_texts(docs)\n",
    "vocab_size = len(t.word_index) + 1 # word_index = Dictionary mapping of words\n",
    "\n",
    "# integer encode the documents\n",
    "encoded_docs = t.texts_to_sequences(docs)\n",
    "print(encoded_docs)\n",
    "\n",
    "# pad documents to a max length of 4 words\n",
    "max_length = 4\n",
    "padded_docs = pad_sequences(encoded_docs, maxlen=max_length, padding='post')\n",
    "print(padded_docs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "glove.6B.100d.txt\t    glove.6B.50d.txt\t\t\tmodel_wv.txt\r\n",
      "glove.6B.100d.txt.word2vec  glove.6B.zip\t\t\tword2vec.model\r\n",
      "glove.6B.200d.txt\t    GoogleNews-vectors-negative300.bin\r\n",
      "glove.6B.300d.txt\t    model_wv.bin\r\n"
     ]
    }
   ],
   "source": [
    "!ls Models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the whole 100d embedding to memory. We are collecting this into a dictionary. It might be faster to filter out only the words in the training set from the word embedding.\n",
    "embeddings_index = dict()\n",
    "f = open('Models/glove.6B.100d.txt', mode='rt', encoding='utf-8')\n",
    "for line in f:\n",
    "    values = line.split()\n",
    "    word = values[0]\n",
    "    coefs = asarray(values[1:], dtype='float32')\n",
    "    embeddings_index[word] = coefs\n",
    "f.close()\n",
    "print('Loaded %s word vectors.'% len(embeddings_index))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "400000"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#\n",
    "len(embeddings_index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a weight matrix for words in training docs\n",
    "embedding_matrix = zeros((vocab_size, 100))\n",
    "for word, i in t.word_index.items():\n",
    "    embedding_vector = embeddings_index.get(word)\n",
    "    if embedding_vector is not None:\n",
    "        embedding_matrix[i] = embedding_vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_3\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_4 (InputLayer)         (None, 4)                 0         \n",
      "_________________________________________________________________\n",
      "embedding_3 (Embedding)      (None, 4, 100)            1500      \n",
      "_________________________________________________________________\n",
      "flatten_3 (Flatten)          (None, 400)               0         \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 1)                 401       \n",
      "=================================================================\n",
      "Total params: 1,901\n",
      "Trainable params: 401\n",
      "Non-trainable params: 1,500\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# Define a model\n",
    "visible = Input(shape=(max_len,))\n",
    "embed1 = Embedding(vocab_size, 100, weights=[embedding_matrix], trainable=False)(visible) # Notice trainable=Flase means we dont want to update the weights of our pretrained embedding. We just want to use as is.\n",
    "flat1 = Flatten()(embed1)\n",
    "output = Dense(1, activation='sigmoid')(flat1)\n",
    "model = Model(inputs = visible, outputs = output)\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 100.000000\n"
     ]
    }
   ],
   "source": [
    "# Compile and fit the model\n",
    "model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])\n",
    "model.fit(padded_docs, labels, epochs=50, verbose=0)\n",
    "# Evaluate the model\n",
    "loss, accuracy = model.evaluate(padded_docs, labels, verbose=0)\n",
    "print('Accuracy: %f' %(accuracy*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**In practice, experiment with learning a word embedding using a pretrained embedding that is fixed and then try to perform learning on top of a pretrained embedding. See what works best for your specific problem.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5. Tips for Cleaning text for Word Embedding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The field of NLP has been moving away from BOW models and word encoding toward word embeddings. \n",
    "\n",
    "The benefit of word embeddings is that they encode each word into a dense vector that captures something about its re;ative meaning within the training text. This means that variations of words like case, spelling, punctuation, and so on will automatically be learned to be similar in the embedding space. \n",
    "\n",
    "In turn, this can mean that the amount of cleaning required from your text may be less and perhaps quite different to classical text cleaning. For example, it might no longer make sense to stem words or remove punctuation for contractions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}