Text_Classification_with_Keras.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Learning for Text Sequences\n",
    "\n",
    "We will explore deep-learning models that can process text (understood as sequences of word or sequences of characters), timeseries, and sequence data in general. The two fundamental deep-learning algorithms for sequence processing are **recurrent neural networks** and **1D convnets**, the one-dimensional version of the 2D convnets.\n",
    "\n",
    "Applications of these algorithms include the following:\n",
    " - Document classification and timeseries classification, such as identifying the topic of an article or the author of a book\n",
    " - Timeseries comparisons, such as estimating how closely related two documents or two stock tickers are\n",
    " - Sequence-to-sequence learning, such as decoding an English sentence into French\n",
    " - Sentiment analysis, such as classifying the sentiment of tweets or movie reviews as positive or negative\n",
    " - Timeseries forecasting, such as predicting the future weather at a certain location, given recent weather data\n",
    " \n",
    "### Working with Text Data\n",
    "\n",
    "Text is one of the most widespread forms of sequence data. It can be understood as either a sequence of characters or a sequence of words, but it’s most common to work at the level of words. The deep-learning sequence-processing models can use text to produce a basic form of natural-language understanding, sufficient for applications including document classification, sentiment analysis, author identification, and even question-answering (QA) (in a constrained context). \n",
    "\n",
    "None of the deeplearning models truly understand text in a human sense; rather, these models can map the statistical structure of written language, which is sufficient to solve many simple textual tasks. Deep learning for natural-language processing is pattern recognition applied to words, sentences, and paragraphs, in much the same way that computer vision is pattern recognition applied to pixels.\n",
    "\n",
    "Like all other neural networks, deep-learning models don’t take as input raw text: they only work with numeric tensors. **Vectorizing** text is the process of transforming text into numeric tensors. This can be done in multiple ways:\n",
    " - Segment text into words, and transform each word into a vector.\n",
    " - Segment text into characters, and transform each character into a vector.\n",
    " - Extract n-grams of words or characters, and transform each n-gram into a vector. **N-grams** are overlapping groups of multiple consecutive words or characters.\n",
    " \n",
    "Collectively, the different units into which we can break down text (words, characters, or n-grams) are called **tokens**, and breaking text into such tokens is called **tokenization**. All text-vectorization processes consist of applying some tokenization scheme and then associating numeric vectors with the generated tokens. These vectors, packed into sequence tensors, are fed into deep neural networks. There are multiple ways to associate a vector with a token. Two major ones: ***one-hot\n",
    "encoding of tokens*** , and ***token embedding (typically used exclusively for words, and called\n",
    "word embedding)***.\n",
    "\n",
    "![capture](https://user-images.githubusercontent.com/13174586/50069599-cf1bdf80-01f0-11e9-9d69-a8065ea71006.JPG)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Understanding n-grams and bag-of-words\n",
    "Word n-grams are groups of N (or fewer) consecutive words that we can extract from a sentence. The same concept may also be applied to characters instead of words. Here’s a simple example. Consider the sentence “The cat sat on the mat.” It may be\n",
    "decomposed into the following set of 2-grams:\n",
    "\n",
    "```{\"The\", \"The cat\", \"cat\", \"cat sat\", \"sat\",```\n",
    "\n",
    "```\"sat on\", \"on\", \"on the\", \"the\", \"the mat\", \"mat\"}```\n",
    "\n",
    "It may also be decomposed into the following set of 3-grams:\n",
    "\n",
    "```{\"The\", \"The cat\", \"cat\", \"cat sat\", \"The cat sat\",```\n",
    "\n",
    "```\"sat\", \"sat on\", \"on\", \"cat sat on\", \"on the\", \"the\",```\n",
    "\n",
    "```\"sat on the\", \"the mat\", \"mat\", \"on the mat\"}```\n",
    "\n",
    "Such a set is called a ***bag-of-2-grams*** or ***bag-of-3-grams***, respectively. The term bag here refers to the fact that you’re dealing with a set of tokens rather than a list or sequence: the tokens have no specific order. This family of tokenization methods is called bag-of-words.\n",
    "\n",
    "Because bag-of-words isn’t an order-preserving tokenization method (the tokens generated are understood as a set, not a sequence, and the general structure of the sentences is lost), it tends to be used in shallow language-processing models rather than in deep-learning models. Extracting n-grams is a form of feature engineering, and deep learning does away with this kind of rigid, brittle approach, replacing it with hierarchical feature learning. One-dimensional convnets and recurrent neural networks,are capable of learning representations for groups of words and characters without being explicitly told about the existence of such groups, by looking at continuous word or character sequences. \n",
    "\n",
    "### One-hot Encoding of Words and Characters\n",
    "One-hot encoding is the most common, most basic way to turn a token into a vector. It consists of associating a unique integer index with every word and then turning this integer index ***i*** into a binary vector of size ***N (the size of the\n",
    "vocabulary)***; the vector is all zeros except for the ***i*** th entry, which is ***1***.\n",
    "\n",
    "Of course, one-hot encoding can be done at the character level, as well. To unambiguously drive home what one-hot encoding is and how to implement it, below are two toy examples: one for words, the other for characters\n",
    "\n",
    "#### Word-level One-hot Encoding (Toy Example)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "[[[0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]\n",
      "\n",
      " [[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]]\n",
      "2\n",
      "[[[0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]\n",
      "\n",
      " [[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]]\n",
      "3\n",
      "[[[0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]\n",
      "\n",
      " [[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]]\n",
      "4\n",
      "[[[0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]\n",
      "\n",
      " [[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]]\n",
      "5\n",
      "[[[0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]\n",
      "\n",
      " [[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]]\n",
      "6\n",
      "[[[0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]\n",
      "\n",
      " [[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]]\n",
      "1\n",
      "[[[0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]\n",
      "\n",
      " [[0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]]\n",
      "7\n",
      "[[[0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]\n",
      "\n",
      " [[0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]]\n",
      "8\n",
      "[[[0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]\n",
      "\n",
      " [[0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]]\n",
      "9\n",
      "[[[0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]\n",
      "\n",
      " [[0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]]\n",
      "10\n",
      "[[[0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]\n",
      "\n",
      " [[0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]]\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "samples=('The cat sat on the mat.', 'The dog ate my homework.') #Initial data: one entry per sample \n",
    "                                                                #(in this example, a sample is a sentence,\n",
    "                                                                #but it could be an entire document)\n",
    "        \n",
    "token_index={} #Build an index of all token in the data\n",
    "for sample in samples:\n",
    "    for word in sample.split(): #Tokenizes the samples via the split method. In real life, we’d also \n",
    "                                #strip punctuation and special characters from the samples.\n",
    "        if word not in token_index:\n",
    "            token_index[word]= len(token_index)+1 #Assigns a unique index to each unique word. Note that we don’t\n",
    "                                                  #attribute index 0 to anything.\n",
    "max_length= 10 #Vectorizes the samples. We’ll only consider the first max_length words in each sample.\n",
    "\n",
    "results= np.zeros(shape=(len(samples), #This is where we store the results\n",
    "                 max_length,\n",
    "                 max(token_index.values()) +1))\n",
    "\n",
    "for i, sample in enumerate(samples):\n",
    "    for j, word in list(enumerate(sample.split()))[:max_length]:\n",
    "        index=token_index.get(word)\n",
    "        print(index)\n",
    "        results[i,j, index]= 1.\n",
    "        print(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Character-level One-hot Encoding (Toy Example)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 The cat sat on the mat.\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "1 The dog ate my homework.\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[1. 1. 1. ... 1. 1. 1.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n",
      "None\n",
      "[[[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  [1. 1. 1. ... 1. 1. 1.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n"
     ]
    }
   ],
   "source": [
    "import string\n",
    "\n",
    "samples=('The cat sat on the mat.', 'The dog ate my homework.')\n",
    "\n",
    "characters= string.printable #All printable ASCII characters\n",
    "\n",
    "token_index= dict(zip(range(1, len(characters)+1), characters))\n",
    "\n",
    "max_length=50\n",
    "results= np.zeros((len(samples), max_length, max(token_index.keys())+1 ))\n",
    "results.shape\n",
    "\n",
    "for i, sample in enumerate(samples):\n",
    "    print(i, sample)\n",
    "    for j, character in enumerate(sample):\n",
    "        index= token_index.get(character)\n",
    "        print(index)\n",
    "        results[i,j, index]= 1\n",
    "        print(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Keras has built-in utilities for doing one-hot encoding of text at the word level or character level, starting from raw text data. You should use these utilities, because they take care of a number of important features such as stripping special characters from strings and only taking into account the N most common words in our dataset (a common restriction, to avoid dealing with very large input vector spaces).\n",
    "\n",
    "#### Using Keras for Word-level One-hot Encoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<keras_preprocessing.text.Tokenizer object at 0x00000206E43B9EF0>\n",
      "[[1, 2, 3, 4, 1, 5], [1, 6, 7, 8, 9]]\n",
      "[[0. 1. 1. ... 0. 0. 0.]\n",
      " [0. 1. 0. ... 0. 0. 0.]]\n",
      "{'the': 1, 'cat': 2, 'sat': 3, 'on': 4, 'mat': 5, 'dog': 6, 'ate': 7, 'my': 8, 'homework': 9}\n",
      "Found 9 unique tokens\n"
     ]
    }
   ],
   "source": [
    "from keras.preprocessing.text import Tokenizer\n",
    "\n",
    "samples=('The cat sat on the mat.', 'The dog ate my homework.')\n",
    "\n",
    "tokenizer= Tokenizer(num_words=1000) #Creates a tokenizer, configured to only take \n",
    "                                    #into account the 1,000 most common words\n",
    "\n",
    "tokenizer.fit_on_texts(samples)  #Builds the word index\n",
    "print(tokenizer)\n",
    "\n",
    "sequences= tokenizer.texts_to_sequences(samples) #Turns strings into lists of integer indices\n",
    "print(sequences)\n",
    "\n",
    "one_hot_results= tokenizer.texts_to_matrix(samples, mode='binary') #We could also directly get the one-hot binary \n",
    "                                                                    #representations. Vectorization modes other than \n",
    "                                                                    #one-hot encoding are supported by this tokenizer.\n",
    "print(one_hot_results)\n",
    "\n",
    "word_index= tokenizer.word_index #How we can recover the word index that was computed\n",
    "print(word_index)\n",
    "\n",
    "print(\"Found %s unique tokens\"%len(word_index))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A variant of one-hot encoding is the so-called **one-hot hashing trick**, which we can use when the number of unique tokens in our vocabulary is too large to handle explicitly. Instead of explicitly assigning an index to each word and keeping a reference of these indices in a dictionary, we can hash words into vectors of fixed size. This is typically done with a very lightweight hashing function. The main advantage of this method is that it does away with maintaining an explicit word index, which saves memory and allows online encoding of the data (we can generate token vectors right away, before we’ve seen all of the available data). The one drawback of this approach is that it’s susceptible to **hash collisions**: two different words may end up with the same hash, and subsequently any machine-learning model looking at these hashes won’t be able to tell the difference between these words. The likelihood of hash collisions decreases when the dimensionality of the hashing space is much larger than the total number of unique tokens being hashed.\n",
    "\n",
    "#### Word-level One-hot Encoding With Hashing Trick (Toy Example)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "378\n",
      "[[[0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]\n",
      "\n",
      " [[0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  ...\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]\n",
      "  [0. 0. 0. ... 0. 0. 0.]]]\n"
     ]
    }
   ],
   "source": [
    "samples=('The cat sat on the mat.', 'The dog ate my homework.')\n",
    "\n",
    "dimensionality= 1000 #Stores the words as vectors of size 1,000. If you have close\n",
    "                    #to 1,000 words (or more), you’ll see many hash collisions,\n",
    "                    #which will decrease the accuracy of this encoding method\n",
    "max_length=10\n",
    "\n",
    "results= np.zeros((len(samples), max_length, dimensionality))\n",
    "\n",
    "for i, sample in enumerate(sample):\n",
    "    for j, word in list(enumerate(sample.split()))[:max_length]:\n",
    "        index= abs(hash(word)%dimensionality) #Hashes the word into a random integer index between 0 and 1,000\n",
    "        print(index)\n",
    "        results[i,j, index]=1\n",
    "        print(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using word embeddings\n",
    "\n",
    "Another popular and powerful way to associate a vector with a word is the use of **dense word vectors**, also called **word embeddings**. Whereas the vectors obtained through one-hot encoding are *binary, sparse (mostly made of zeros), and very high-dimensional* (same dimensionality as the number of words in the vocabulary), word embeddings are low dimensional floating-point vectors (that is, dense vectors, as opposed to sparse vectors). Unlike the word vectors obtained via one-hot encoding, word\n",
    "embeddings are learned from data. It’s common to see word embeddings that are 256-dimensional, 512-dimensional, or 1,024-dimensional when dealing with very large vocabularies. On the other hand, one-hot encoding words generally leads to vectors\n",
    "that are 20,000-dimensional or greater (capturing a vocabulary of 20,000 tokens, in this case). So, word embeddings pack more information into far fewer dimensions.\n",
    "\n",
    "![capture](https://user-images.githubusercontent.com/13174586/50081612-f2588600-0214-11e9-9d6f-ae5450f13b4d.JPG)\n",
    "\n",
    "There are two ways to obtain word embeddings:\n",
    " - Learn word embeddings jointly with the main task we care about (such as document classification or sentiment prediction). In this setup, we start with random word vectors and then learn word vectors in the same way we learn the weights of a neural network.\n",
    " - Load into our model word embeddings that were precomputed using a different machine-learning task than the one we’re trying to solve. These are called **pretrained word embeddings**.\n",
    " \n",
    "Let’s look at both.\n",
    "\n",
    "#### LEARNING WORD EMBEDDINGS WITH THE EMBEDDING LAYER\n",
    "The simplest way to associate a dense vector with a word is to choose the vector at random. The problem with this approach is that the resulting embedding space has no structure: for instance, the words *accurate* and *exact* may end up with completely\n",
    "different embeddings, even though they’re interchangeable in most sentences. It’s difficult for a deep neural network to make sense of such a noisy, unstructured embedding space.\n",
    "\n",
    "To get a bit more abstract, the geometric relationships between word vectors should reflect the semantic relationships between these words. Word embeddings are meant to map human language into a geometric space. For instance, in a reasonable\n",
    "embedding space, we would expect synonyms to be embedded into similar word vectors; and in general, we would expect the geometric distance (such as L2 distance) between any two word vectors to relate to the semantic distance between the associated\n",
    "words (words meaning different things are embedded at points far away from each other, whereas related words are closer). In addition to distance, we may want specific *directions* in the embedding space to be meaningful. To make this clearer, let’s\n",
    "look at a concrete example.\n",
    "\n",
    "Four words are embedded on a 2D plane: cat, dog, wolf, and tiger. With the vector representations we chose here, some semantic relationships between these words can be encoded as geometric transformations. For instance, the same vector allows us to go from cat to tiger and from dog to wolf : this vector could be interpreted as the “from pet to wild animal” vector. Similarly, another vector lets us go from dog to cat and from wolf to tiger, which could be interpreted as a “from canine to feline” vector.\n",
    "![capture](https://user-images.githubusercontent.com/13174586/50082748-b83cb380-0217-11e9-82fe-2ca0f128e7e3.JPG)\n",
    "\n",
    "In real-world word-embedding spaces, common examples of meaningful geometric transformations are “gender” vectors and “plural” vectors. For instance, by adding a “female” vector to the vector “king,” we obtain the vector “queen.” By adding a “plural” vector, we obtain “kings.” Word-embedding spaces typically feature thousands of such interpretable and potentially useful vectors.\n",
    "\n",
    "Is there some ideal word-embedding space that would perfectly map human language and could be used for any natural-language-processing task? Possibly, but we have yet to compute anything of the sort. Also, there is no such a thing as *human language*—\n",
    "there are many different languages, and they aren’t isomorphic, because a language is the reflection of a specific culture and a specific context. But more pragmatically, what makes a good word-embedding space depends heavily on our task: the perfect word-embedding space for an English-language movie-review sentiment analysis model may look different from the perfect embedding space for an English language legal-document-classification model, because the importance of certain semantic relationships varies from task to task.\n",
    "\n",
    "It’s thus reasonable to **learn** a new embedding space with every new task. Fortunately, backpropagation makes this easy, and Keras makes it even easier. It’s about learning the weights of a layer: the ```Embedding``` layer.\n",
    "\n",
    "### Instantiating an Embedding Layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.layers import Embedding\n",
    "\n",
    "embedding= Embedding(1000, 64) #The Embedding layer takes at least two arguments: the number of possible tokens\n",
    "                                #(here, 1,000: 1 + maximum word index) and the dimensionality of the embeddings (here, 64)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ```Embedding``` layer is best understood as a dictionary that maps integer indices (which stand for specific words) to dense vectors. It takes integers as input, it looks up these integers in an internal dictionary, and it returns the associated vectors. It’s effectively a dictionary lookup\n",
    "\n",
    "<span style=\"color:purple\">WORD INDEX -> EMBEDDING LAYER -> CORRESPONDING WORD VECTOR </span>\n",
    "\n",
    "The ```Embedding``` layer takes as input a 2D tensor of integers, of shape ```(samples, sequence_length)```, where each entry is a sequence of integers. It can embed sequences of variable lengths: for instance, we could feed into the Embedding layer in\n",
    "the previous example batches with shapes (32, 10) (batch of 32 sequences of length 10) or (64, 15) (batch of 64 sequences of length 15). All sequences in a batch must have the same length, though (because we need to pack them into a single tensor),\n",
    "so sequences that are shorter than others should be padded with zeros, and sequences that are longer should be truncated.\n",
    "\n",
    "This layer returns a 3D floating-point tensor of shape ```(samples, sequence_length, embedding_dimensionality)```. Such a 3D tensor can then be processed by an **RNN** layer or a **1D convolution** layer (both will be introduced in the following\n",
    "sections).\n",
    "\n",
    "When we instantiate an ```Embedding``` layer, its weights (its internal dictionary of token vectors) are initially random, just as with any other layer. During training, these word vectors are gradually adjusted via backpropagation, structuring the space into something the downstream model can exploit. Once fully trained, the embedding space will show a lot of structure—a kind of structure specialized for the specific problem for which we’re training our model.\n",
    "\n",
    "Let’s apply this idea to the IMDB movie-review sentiment-prediction task that we’re already familiar with. First, we’ll quickly prepare the data. We’ll restrict the movie reviews to the top 10,000 most common words and cut off the reviews after only 20 words. The network will learn 8-dimensional embeddings for each of the 10,000 words, turn the input integer sequences (2D integer tensor) into embedded sequences (3D float tensor), flatten the tensor to 2D, and train a single Dense layer on top for classification.\n",
    "\n",
    "### Load The IMDB Data for Use With an Embedding Layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from keras.datasets import imdb\n",
    "from keras import preprocessing\n",
    "\n",
    "max_features= 10000 #Number of words to consider as features\n",
    "maxlen=20 #Cuts off the text after this number of words (among the max_features most common words)\n",
    "\n",
    "(x_train, y_train), (x_test, y_test)= imdb.load_data(num_words= max_features) #Loads data as list of integers\n",
    "\n",
    "x_train= preprocessing.sequence.pad_sequences(x_train, maxlen=maxlen)    #Turns the lists of integers into a 2D  \n",
    "x_test= preprocessing.sequence.pad_sequences(x_test, maxlen=maxlen)    #integer tensor of shape (samples, maxlen)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Use an Embedding Layer and Classifier on The IMDB Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Flatten, Embedding\n",
    "\n",
    "model= Sequential()\n",
    "model.add(Embedding(10000, 8, input_length=maxlen)) \n",
    "\n",
    "model.add(Flatten())#Flattens the 3D tensor of embeddings into a 2D tensor \n",
    "                    #of shape (samples, maxlen * 8)\n",
    "model.add(Dense(1, activation='sigmoid'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_1 (Embedding)      (None, 20, 8)             80000     \n",
      "_________________________________________________________________\n",
      "flatten_1 (Flatten)          (None, 160)               0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 1)                 161       \n",
      "=================================================================\n",
      "Total params: 80,161\n",
      "Trainable params: 80,161\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['acc'])\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 20000 samples, validate on 5000 samples\n",
      "Epoch 1/10\n",
      "20000/20000 [==============================] - 8s 390us/step - loss: 0.6759 - acc: 0.6050 - val_loss: 0.6398 - val_acc: 0.6814\n",
      "Epoch 2/10\n",
      "20000/20000 [==============================] - 2s 80us/step - loss: 0.5657 - acc: 0.7427 - val_loss: 0.5467 - val_acc: 0.7206\n",
      "Epoch 3/10\n",
      "20000/20000 [==============================] - 2s 79us/step - loss: 0.4752 - acc: 0.7808 - val_loss: 0.5113 - val_acc: 0.7384\n",
      "Epoch 4/10\n",
      "20000/20000 [==============================] - 2s 80us/step - loss: 0.4263 - acc: 0.8077 - val_loss: 0.5008 - val_acc: 0.7452\n",
      "Epoch 5/10\n",
      "20000/20000 [==============================] - 2s 80us/step - loss: 0.3930 - acc: 0.8258 - val_loss: 0.4981 - val_acc: 0.7538\n",
      "Epoch 6/10\n",
      "20000/20000 [==============================] - 2s 78us/step - loss: 0.3668 - acc: 0.8395 - val_loss: 0.5014 - val_acc: 0.7530\n",
      "Epoch 7/10\n",
      "20000/20000 [==============================] - 2s 79us/step - loss: 0.3435 - acc: 0.8533 - val_loss: 0.5052 - val_acc: 0.7520\n",
      "Epoch 8/10\n",
      "20000/20000 [==============================] - 2s 82us/step - loss: 0.3223 - acc: 0.8657 - val_loss: 0.5132 - val_acc: 0.7486\n",
      "Epoch 9/10\n",
      "20000/20000 [==============================] - 2s 79us/step - loss: 0.3022 - acc: 0.8766 - val_loss: 0.5213 - val_acc: 0.7490\n",
      "Epoch 10/10\n",
      "20000/20000 [==============================] - 2s 79us/step - loss: 0.2839 - acc: 0.8860 - val_loss: 0.5303 - val_acc: 0.7466\n"
     ]
    }
   ],
   "source": [
    "history= model.fit(x_train, y_train, epochs=10, batch_size=32, validation_split=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We get to a validation accuracy of ~76%, which is pretty good considering that we’re only looking at the first 20 words in every review. But note that merely flattening the embedded sequences and training a single ```Dense``` layer on top leads to a model that treats each word in the input sequence separately, without considering inter-word relationships and sentence structure (for example, this model would likely treat both “this movie is a bomb” and “this movie is the bomb” as being negative reviews). It’s\n",
    "much better to add ```recurrent``` layers or ```1D convolutional``` layers on top of the embedded sequences to learn features that take into account each sequence as a whole.\n",
    "\n",
    "#### USING PRETRAINED WORD EMBEDDINGS\n",
    "Sometimes, we have so little training data available that we can’t use our data alone to learn an appropriate task-specific embedding of our vocabulary. What do we do then?\n",
    "\n",
    "Instead of learning word embeddings jointly with the problem we want to solve, we can load embedding vectors from a precomputed embedding space that we know is highly structured and exhibits useful properties—that captures generic aspects of language structure. The rationale behind using pretrained word embeddings in natural-language processing is much the same as for using pretrained convnets in image classification: we don’t have enough data available to learn truly powerful features on your own, but we expect the features that we need to be fairly generic—that is, common visual features or semantic features. In this case, it makes sense to reuse features learned on a different problem.\n",
    "\n",
    "Such word embeddings are generally computed using word-occurrence statistics (observations about what words co-occur in sentences or documents), using a variety of techniques, some involving neural networks, others not. The idea of a dense, lowdimensional embedding space for words, computed in an unsupervised way, was initially explored by Bengio et al. in the early 2000s, but it only started to take off in research and industry applications after the release of one of the most famous and successful word-embedding schemes: the **Word2vec** algorithm (https://code.google.com/ archive/p/word2vec), developed by Tomas Mikolov at Google in 2013. Word2vec dimensions capture specific semantic properties, such as gender.\n",
    "\n",
    "There are various precomputed databases of word embeddings that we can download and use in a Keras Embedding layer. Word2vec is one of them. Another popular one is called **Global Vectors for Word Representation** (GloVe, https://nlp.stanford\n",
    ".edu/projects/glove), which was developed by Stanford researchers in 2014. This embedding technique is based on factorizing a matrix of word co-occurrence statistics. Its developers have made available precomputed embeddings for millions of\n",
    "English tokens, obtained from Wikipedia data and Common Crawl data.\n",
    "\n",
    "Let’s look at how we can get started using GloVe embeddings in a Keras model. The same method is valid for Word2vec embeddings or any other word-embedding database.\n",
    "\n",
    "#### DOWNLOADING THE IMDB DATA AS RAW TEXT\n",
    "First, head to http://mng.bz/0tIo and download the raw IMDB dataset. Uncompress it.\n",
    "\n",
    "Now, let’s collect the individual training reviews into a list of strings, one string per review. We’ll also collect the review labels (positive/negative) into a labels list.\n",
    "\n",
    "### Process The Labels of The Raw IMDB Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "imdb_dir= 'IMDB_Data/aclImdb'\n",
    "train_dir= os.path.join(imdb_dir, 'train')\n",
    "\n",
    "\n",
    "labels=[]\n",
    "texts=[]\n",
    "\n",
    "for label_type in ['pos', 'neg']:\n",
    "    dir_name= os.path.join(train_dir, label_type)\n",
    "    for fname in os.listdir(dir_name):\n",
    "        if fname[-4:]=='.txt':\n",
    "            f= open(os.path.join(dir_name,fname), encoding=\"utf8\")\n",
    "            texts.append(f.read())\n",
    "            f.close()\n",
    "            if label_type=='neg':\n",
    "                labels.append(0)\n",
    "            else:\n",
    "                labels.append(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### TOKENIZING THE DATA\n",
    "Let’s vectorize the text and prepare a training and validation split. Because pretrained word embeddings are meant to be particularly useful on problems where little training data is available (otherwise, task-specific embeddings are likely to outperform them), we’ll add the following twist: restricting the training data to the first 200 samples. So we’ll learn to classify movie reviews after looking at just 200 examples.\n",
    "\n",
    "### Tokenize The Text of The Raw IMDB Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<keras_preprocessing.text.Tokenizer object at 0x00000244A2A3EDA0>\n"
     ]
    }
   ],
   "source": [
    "from keras.preprocessing.text import Tokenizer\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "import numpy as np\n",
    "\n",
    "maxlen=100 #Cuts off reviews after 100 words\n",
    "training_samples=20000 #Trains on 200 samples\n",
    "validation_samples=10000 #Validates on 10,000 samples\n",
    "max_words=10000 #Considers only the top 10,000 words in the dataset\n",
    "\n",
    "tokenizer= Tokenizer(num_words=max_words)\n",
    "tokenizer.fit_on_texts(texts)\n",
    "print(tokenizer)\n",
    "sequences= tokenizer.texts_to_sequences(texts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'the': 1, 'and': 2, 'a': 3, 'of': 4, 'to': 5, 'is': 6, 'br': 7, 'in': 8, 'it': 9, 'i': 10}\n",
      "Found 88582 unique tokens\n"
     ]
    }
   ],
   "source": [
    "word_index=tokenizer.word_index\n",
    "print({key:value for key, value in list(word_index.items())[:10]})\n",
    "\n",
    "print(\"Found %s unique tokens\"%len(word_index))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[5924,  482,   69, ...,   12,    9,  215],\n",
       "       [ 111,   10,  255, ...,    5,  335,  405],\n",
       "       [   8, 7576,    1, ...,    6,  176,  396],\n",
       "       ...,\n",
       "       [   1, 2817,   30, ...,   17,   96,   75],\n",
       "       [ 107,    9,   29, ...,  260, 1195,  794],\n",
       "       [ 344,   39,  106, ...,   11,    6, 1350]])"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data= pad_sequences(sequences, maxlen=maxlen)\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of data densor (25000, 100)\n",
      "Shape of labels tensor (25000,)\n"
     ]
    }
   ],
   "source": [
    "labels= np.array(labels)\n",
    "print(\"Shape of data densor\", data.shape)\n",
    "print(\"Shape of labels tensor\", labels.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "indices= np.arange(data.shape[0]) #Splits the data into a training set and a validation set, but first shuffles the data,\n",
    "                                  #because you’re starting with data in which samples are ordered (all negative first, then\n",
    "                                  #all positive)\n",
    "np.random.shuffle(indices)\n",
    "\n",
    "data= data[indices]\n",
    "data\n",
    "labels= labels[indices]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train=data[:training_samples]\n",
    "y_train=labels[:training_samples]\n",
    "x_val= data[training_samples:training_samples+validation_samples]\n",
    "y_val= labels[training_samples:validation_samples+training_samples]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### DOWNLOADING THE GLOVE WORD EMBEDDINGS\n",
    "Go to https://nlp.stanford.edu/projects/glove, and download the precomputed embeddings from 2014 English Wikipedia. It’s an 822 MB zip file called glove.6B.zip, containing 100-dimensional embedding vectors for 400,000 words (or nonword tokens). Unzip it.\n",
    "\n",
    "#### PREPROCESSING THE EMBEDDINGS\n",
    "Let’s parse the unzipped file (a .txt file) to build an index that maps words (as strings) to their vector representation (as number vectors).\n",
    "\n",
    "### Parse The GloVe Word-embeddings File"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 400000 word vectors in GLOVE.6B.100D\n"
     ]
    }
   ],
   "source": [
    "glove_dir= 'GLOVE'\n",
    "embeddings_index= {}\n",
    "f= open(os.path.join(glove_dir, 'glove.6B.100d.txt'), encoding=\"utf8\")\n",
    "for line in f:\n",
    "    values=line.split()\n",
    "    word=values[0]\n",
    "    coefs= np.asarray(values[1:], dtype='float32')\n",
    "    embeddings_index[word]=coefs\n",
    "f.close()\n",
    "\n",
    "print('Found %s word vectors in GLOVE.6B.100D'%len(embeddings_index))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we’ll build an embedding matrix that we can load into an ```Embedding``` layer. It must be a matrix of shape ```(max_words, embedding_dim)```, where each entry i contains the ```embedding_dim``` -dimensional vector for the word of index i in the reference word index (built during tokenization). Note that index 0 isn’t supposed to stand for any word or token—it’s a placeholder.\n",
    "\n",
    "### Prepare The GloVe Word-embeddings Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_dim=100\n",
    "\n",
    "embedding_matrix= np.zeros((max_words, embedding_dim))\n",
    "for word, i in word_index.items():\n",
    "    if i< max_words:\n",
    "        embedding_vector= embeddings_index.get(word)\n",
    "        if embedding_vector is not None:\n",
    "            embedding_matrix[i]= embedding_vector #Words not found in the embedding index will be all zeros.\n",
    "            "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model definition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_8 (Embedding)      (None, 100, 100)          1000000   \n",
      "_________________________________________________________________\n",
      "flatten_8 (Flatten)          (None, 10000)             0         \n",
      "_________________________________________________________________\n",
      "dense_14 (Dense)             (None, 32)                320032    \n",
      "_________________________________________________________________\n",
      "dense_15 (Dense)             (None, 1)                 33        \n",
      "=================================================================\n",
      "Total params: 1,320,065\n",
      "Trainable params: 1,320,065\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Embedding, Dense, Flatten\n",
    "\n",
    "model= Sequential()\n",
    "model.add(Embedding(max_words, embedding_dim, input_length=maxlen))\n",
    "model.add(Flatten())\n",
    "model.add(Dense(32, activation='relu'))\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### LOADING THE GLOVE EMBEDDINGS IN THE MODEL\n",
    "\n",
    "The Embedding layer has a single weight matrix: a 2D float matrix where each entry i is the word vector meant to be associated with index i. Simple enough. Load the GloVe matrix we prepared into the Embedding layer, the first layer in the model.\n",
    "\n",
    "### Load Pretrained Word Embeddings Into The Embedding Layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.layers[0].set_weights([embedding_matrix]) \n",
    "model.layers[0].trainable=False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Additionally, we’ll freeze the ```Embedding``` layer (set its trainable attribute to False), following the same rationale we’re already familiar with in the context of pretrained convnet features: when parts of a model are pretrained (like our Embedding layer) and parts are randomly initialized (like our classifier), the pretrained parts shouldn’t be updated during training, to avoid forgetting what they already know. The large gradient updates triggered by the randomly initialized layers would be disruptive to the already-learned features\n",
    "\n",
    "### Model Training nad Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 20000 samples, validate on 5000 samples\n",
      "Epoch 1/30\n",
      "20000/20000 [==============================] - 2s 106us/step - loss: 0.7086 - acc: 0.4976 - val_loss: 0.6932 - val_acc: 0.4960\n",
      "Epoch 2/30\n",
      "20000/20000 [==============================] - 2s 93us/step - loss: 0.6945 - acc: 0.5057 - val_loss: 0.6930 - val_acc: 0.5052\n",
      "Epoch 3/30\n",
      "20000/20000 [==============================] - 2s 92us/step - loss: 0.6951 - acc: 0.5095 - val_loss: 0.6932 - val_acc: 0.5062\n",
      "Epoch 4/30\n",
      "20000/20000 [==============================] - 2s 105us/step - loss: 0.6938 - acc: 0.5227 - val_loss: 0.6945 - val_acc: 0.5100\n",
      "Epoch 5/30\n",
      "20000/20000 [==============================] - 2s 91us/step - loss: 0.6823 - acc: 0.5533 - val_loss: 0.7096 - val_acc: 0.5024\n",
      "Epoch 6/30\n",
      "20000/20000 [==============================] - 2s 91us/step - loss: 0.6502 - acc: 0.6159 - val_loss: 0.7283 - val_acc: 0.5002\n",
      "Epoch 7/30\n",
      "20000/20000 [==============================] - 2s 91us/step - loss: 0.6020 - acc: 0.6753 - val_loss: 0.8026 - val_acc: 0.5006\n",
      "Epoch 8/30\n",
      "20000/20000 [==============================] - 2s 91us/step - loss: 0.5457 - acc: 0.7241 - val_loss: 0.8457 - val_acc: 0.4912\n",
      "Epoch 9/30\n",
      "20000/20000 [==============================] - 2s 91us/step - loss: 0.4834 - acc: 0.7733 - val_loss: 0.8653 - val_acc: 0.5010\n",
      "Epoch 10/30\n",
      "20000/20000 [==============================] - 2s 91us/step - loss: 0.4221 - acc: 0.8161 - val_loss: 1.0733 - val_acc: 0.4920\n",
      "Epoch 11/30\n",
      "20000/20000 [==============================] - 2s 91us/step - loss: 0.3617 - acc: 0.8559 - val_loss: 1.0522 - val_acc: 0.4942\n",
      "Epoch 12/30\n",
      "20000/20000 [==============================] - 2s 92us/step - loss: 0.3111 - acc: 0.8832 - val_loss: 1.1244 - val_acc: 0.4904\n",
      "Epoch 13/30\n",
      "20000/20000 [==============================] - 2s 92us/step - loss: 0.2684 - acc: 0.9068 - val_loss: 1.3334 - val_acc: 0.4918\n",
      "Epoch 14/30\n",
      "20000/20000 [==============================] - 2s 92us/step - loss: 0.2353 - acc: 0.9217 - val_loss: 1.4397 - val_acc: 0.4982\n",
      "Epoch 15/30\n",
      "20000/20000 [==============================] - 2s 92us/step - loss: 0.2073 - acc: 0.9332 - val_loss: 1.7325 - val_acc: 0.5008\n",
      "Epoch 16/30\n",
      "20000/20000 [==============================] - 2s 91us/step - loss: 0.1826 - acc: 0.9422 - val_loss: 1.9707 - val_acc: 0.4936\n",
      "Epoch 17/30\n",
      "20000/20000 [==============================] - 2s 91us/step - loss: 0.1675 - acc: 0.9482 - val_loss: 1.6196 - val_acc: 0.4932\n",
      "Epoch 18/30\n",
      "20000/20000 [==============================] - 2s 92us/step - loss: 0.1535 - acc: 0.9549 - val_loss: 1.9450 - val_acc: 0.4898\n",
      "Epoch 19/30\n",
      "20000/20000 [==============================] - 2s 92us/step - loss: 0.1410 - acc: 0.9605 - val_loss: 1.8306 - val_acc: 0.4898\n",
      "Epoch 20/30\n",
      "20000/20000 [==============================] - 2s 92us/step - loss: 0.1360 - acc: 0.9611 - val_loss: 1.9079 - val_acc: 0.4876\n",
      "Epoch 21/30\n",
      "20000/20000 [==============================] - 2s 92us/step - loss: 0.1251 - acc: 0.9665 - val_loss: 2.6067 - val_acc: 0.4938\n",
      "Epoch 22/30\n",
      "20000/20000 [==============================] - 2s 92us/step - loss: 0.1238 - acc: 0.9665 - val_loss: 1.9853 - val_acc: 0.4960\n",
      "Epoch 23/30\n",
      "20000/20000 [==============================] - 2s 92us/step - loss: 0.1233 - acc: 0.9665 - val_loss: 2.0304 - val_acc: 0.4948\n",
      "Epoch 24/30\n",
      "20000/20000 [==============================] - 2s 92us/step - loss: 0.1176 - acc: 0.9683 - val_loss: 2.6023 - val_acc: 0.5018\n",
      "Epoch 25/30\n",
      "20000/20000 [==============================] - 2s 92us/step - loss: 0.1135 - acc: 0.9707 - val_loss: 2.2203 - val_acc: 0.4966\n",
      "Epoch 26/30\n",
      "20000/20000 [==============================] - 2s 94us/step - loss: 0.1139 - acc: 0.9707 - val_loss: 2.4453 - val_acc: 0.4936\n",
      "Epoch 27/30\n",
      "20000/20000 [==============================] - 2s 105us/step - loss: 0.1088 - acc: 0.9710 - val_loss: 2.5882 - val_acc: 0.5054\n",
      "Epoch 28/30\n",
      "20000/20000 [==============================] - 2s 109us/step - loss: 0.1098 - acc: 0.9712 - val_loss: 2.3797 - val_acc: 0.4982\n",
      "Epoch 29/30\n",
      "20000/20000 [==============================] - 2s 91us/step - loss: 0.1075 - acc: 0.9722 - val_loss: 2.2779 - val_acc: 0.4928\n",
      "Epoch 30/30\n",
      "20000/20000 [==============================] - 2s 92us/step - loss: 0.1054 - acc: 0.9722 - val_loss: 2.3877 - val_acc: 0.4966\n"
     ]
    }
   ],
   "source": [
    "model.compile(optimizer='rmsprop', loss= 'binary_crossentropy', metrics=['acc'])\n",
    "\n",
    "history= model.fit(x_train, y_train, epochs=30, batch_size=32, validation_data=(x_val, y_val))\n",
    "\n",
    "model.save_weights('pretrained_glove_model.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot the Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xl4FFXWwOHfISAIQRYXQECCDC4QWQKiiCPiwqDjyiIgouAgiqKO24jLOHyMjqi4L4wbrpCIqMgowowsCq6AggooIATZl7CHRULO98etNJ3QSTpJd6q7c97nqSdd1ZXb53Z11+m6t+qWqCrGGGMMQCW/AzDGGBM7LCkYY4wJsKRgjDEmwJKCMcaYAEsKxhhjAiwpGGOMCbCkYIokIkkisktEjovkuvFIRAaJyEzvcZF1DV63lK/1XxHpV9r/N6a0LCkkGG9HlTflisieoPkS72RU9YCqJqvqb5Fct7yJSHUR2SEiZ4V47lkRyShJeZGsq4g8KCKvFyi/q6qOLWvZxbymikhatF7DxCdLCgnG21Elq2oy8BtwcdCyQ3YyIlK5/KMsf6q6G3gXuDp4uYhUAfoAb/gRlx9ERID+wBbgmnJ+7UoiYvudGGYbp4LxfiG+IyLpIrITuEpEOorI1yKyTUTWicgz3s4SEans/aJM8ebf9p7/RER2ishXItK0pOt6z18gIktEZLv3a/0LERkQIubGIrJbRGoFLTtVRDZ6r3mCiHzulbNZRMYVUv03gF4icnjQsguAHOC/Xrn3i8hyL96FInJJIe9jwboeLSIfeUcjXwNNC6z/nIis9p6fIyJneMsvAv4G9POO5uZ5y2fnvRfejvQBEVnp1fl1ETnCe+4PXhxXe+VvEpFhhdQ/TxfgKOCvwJV52zoo1utF5GfvPfhJRFp7y5uIyETvNTaLyNPe8nxHOnkxBc3PFpF/ishXQDZwnNe8tth7jV9FZFCBGLqLyHzv/VomIl1FpK+IfFNgvbtFZEIx9TUloao2JegEZALnFVj2IPA7cDHuR8HhwKnAaUBl4HhgCTDUW78yoECKN/82sBloD1QB3gHeLsW6xwA7gUu9524H9gMDCqnL58DAoPkngee8x+8Cd3v1qQZ0KqQMAZYDfYKWvQuMCpq/AmjglXUlsAuo5z03CJhZSF0nAOlAdaAVsC5vXe/5/kBd7//uBtYAVYO2yesFYp2d914Ag71t0hSoCXwIvOY99wcvjn97dU8D9gHNi/hcvAGMA6oCW4FLgp7rC6wC2nnv1wlAYy/un4BRQA3vc9MpVPx5MRWoSyZwsretK+M+f8d7r3EOsAdo5a1/BrANONfbDo2BE73X3BZcN+BH4FK/v2uJNNmRQsU0W1X/o6q5qrpHVeeo6jeqmqOqy4GXgM5F/P8EVZ2rqvuBsUCbUqx7ETBfVT/0nnsSl0AKMw63w8JrfujtLQOXTFKABqq6V1W/CFWAur3Im3hNSCJSG7dzeiNonfGqus57b8bhdmbti4grrwnqMuDvqrpbVX8A3irw2m+p6hZVzQEeBY7A7TzD0Q+XuFao6k7gXtwv/ODv73Cv7t8BC4HWhcRaA+gBjFPVfcD75G9CGgSMVNV56ixR1VVAR9zRxd2qmu19bkK+z4UYo6qLVXW/9zn7j6ou915jOjAN+KO37l+Al1V1mrcdVqnqL6q6B5fEr/Lq0gaXwCeXIA5TDEsKFdOq4BkROUlEPhaR9SKyAxiB2wEUZn3Q491AcinWPTY4Dm+HvbqIct4F/igi9XDNH3tV9UvvuTtwv0DnisiPIlJUO/mbwPkiUh93VLBIVX/Me1JEBojIAq8pbRtwEkW/FwD1gCTyv68rg1cQkb95TTLbcb/Oa4RRbp5jC5S3EjgMODpvgaqGu016AnuBqd78WOAiEanrzTcGfg3xf42BTFU9EGbMBRX8zF0kIt+IyBbvfe7KwfejsBjAJfC8EyauAt7xflSYCLGkUDEVHBr3RVzTwB9U9QjgAdxhfTStAxrlzYiIAA0LW1lVs4DpQC9cs0560HPrVHWQqjYAbgJeCu67KFDOcuArr4z+uCSRF8PxwGhgCHCkqtYGfqb492IDkIvbmeUJnKoqIl1wzWM9gNpAHVyzVF65xQ1VvBZoUqDs34FNxfxfKNfgjlJWich63Pt4GK6zHdzOu1mI/1sFNBGRpBDPZeOazfLUD7FOcB/D4bjmtodxTXO1cX06ee9HYTGgqrO9MjrhjhzfCrWeKT1LCgZcO/V2IFtETgauL4fX/AhIE5GLxZ0BdStBv3wLMQ63U+vOwaYjROQKEclLKNtwO6CiftG+4b3eacHl4H5dK25nK17n50nFVcT7pToR+D8ROVxEUnEJJ09NXGf2ZtwRzXDckUKeDUCKlxhDSQduF5EUEakJPASkq2pucbEFE3dNxdm4zvU23tQaeJyDTUivAH8TkbbiNBeRxrhEmgX8S9zpvYd7O2aA+UBncScE1AaK6+iuiktEm4ADXmf7uUHPvwoMEpEuXid7IxE5Mej5t3DJO1tVvy7Je2CKZ0nBgGt+uQbX8fsirkM4qlR1A65f4AnczqYZ8D2uk7QwE4EWwG+qujBo+WnAHBHJxrWR36RFXz/wLq6pYqqqbgyK6QfgGeBb3JHMScA3IUs41BDcEcAG3E7ttaDnJgOfAktxfRQ7vPLzvIPbSW4RkW9DlP2yt84sXEf5TlxSK6mrgTleW/36vAl4GmgnIiepajrwiPd6O3DvZx2vL+QiXGfxKtzpzj29cqcAH+A6fb8FJhUVhKpuA27z/meLV85HQc9/CVyH2xbbgRnkPwp7E0jFjhKiQlxTrjH+8pol1gI9VXWW3/GY2OV1lm8EUlV1hd/xJBo7UjC+EZFuIlJLRKoCf8c1sYT6pWxMsJuALywhREeFuJrVxKwzcWe/HIY7jfIy7zRJY0ISkdW4U5Av9TuWRGXNR8YYYwKs+cgYY0xA3DUfHXXUUZqSkpJvWXZ2NjVq1Aj9D3Eo0eoDiVenRKsPJF6dEq0+ULY6zZs3b7OqFnfad/wlhZSUFObOnZtv2cyZMzn77LP9CSgKEq0+kHh1SrT6QOLVKdHqA2Wrk4isLH4taz4yxhgTxJKCMcaYAEsKxhhjAiwpGGOMCbCkYIwxJsCSgjHGmABLCsYYYwLi7joFY4wpV6pw4ADk5Li/ubluWd7f4MfBf/fuhd27i59+//3ga4m4Ke9xgb/JxxwDUb72wpKCMSa+7N8PGzfChg0Hp4LzGzZw+saNUK1a8eUF7/RDTQdKewfSEhBxcRTjiNtui3oolhSMMeHbtQtWrSp+PTi4kyvsb/DjfftgyxbIyir+79atoV+venWoV89Nxx/PtgYNqN+gQXixVq4c3lSp0sEp71d93uOCfw8/3MVU1HT44VClStjv3drPPuOE8GpUapYUjDGh7dsHCxbAnDlumjsXFi92TSPloVYtqFvXTUceCccf7x4fc8zBnX/wlJyc799/njmT+vE6zEXBpqM8laLfDWxJwRgDOTnUWLYMfv31YBL48UfXVANuR3zqqdCzJ5x4Yvg7p8Laxwsuq1LF7fjzEkDt2qF/QZuos6RgTCJRhaVL4Ztv4OuvYdky1+G5b9+h0++/55s/Na+MWrWgfXu44w6XCNq3h8aND/3VahKSJQVj4llWlksAedO33x5sc09OhpNPdu3WRxwBVasWPlWrxqIDB2hxzTXQrFm5NFOY2GRJwZh4oQo//wzTp8NXX7kksGyZe65SJWjZEnr0gNNPh9NOcwkhKSns4jfOnEmL5s2jFLyJF5YUjIllq1bBtGkHp3Xr3PIGDdyOf9Ag97d9+0M6Wo0pDUsKxsSSLVtgxgyXAD791PUPABx9NJxzDpx7rpuaNrU2fhMVlhSM8dP69fDFF2767DP4/nvXTJScDJ07w5AhLgmkplo7vykXlhSMKS+5ue48/7wk8MUX7hRQcFfedugAw4e7JNChg52SaXxhScGYaMnNdZ3BM2e6BPDllwfPDDr6aOjUyR0JdOoEaWlw2GG+hmsMWFIwJrJUYf58SE+HjIyDQ0KcdBJ07w5nnumSwB/+YH0CJiZZUjAmEpYudYkgPd2dNlq5MvzpT/Dww9Ctm7tK15g4YEnBmNJauxbeeQfGjXPjAoHrHP7rX91wEJYITByypGBMSezfDxMm0Pqxx1wzkarrDxg1Cnr3hkaN/I7QmDKxpGBMOLZuhZdfhmefhdWrqXbssfDAA9C3rxsgzpgEYUnBmKL8+is8/TSMGQPZ2dClC4wezTfVq3P2Oef4HZ0xEWdXwxhTkCrMnu3OFmreHP79bzem0Pffu3GHLrrILiQzCcuOFIzJs38/vPcePPGEu59AnTpwzz1w001w7LF+R2dMubCkYIwqvPsu/O1vsHKlOzp44QW4+mqoUcPv6IwpV3YMbCq27793p5H27u3u9vXhh+46gyFDLCGYCsmSgqmYNm2CwYOhXTtYtMj1G8ybB5dcYv0FpkKz5iNTsfz+Ozz/PPzf/7mziW691Z1aWqeO35EZExMsKZiKY8oUd7XxL7+4ISiefNLdncwYE2DHySbxLVniTiO94AI4cAD+8x/45BNLCMaEYEnBJLannnI3qPn8c3jsMVi40CUIG6HUmJCimhREpJuI/CIiy0RkWIjnm4jINBH5QURmiogNHGMiQxWGDYPbboM//9mNYnrnnXbPAmOKEbWkICJJwPPABUALoK+ItCiw2ijgTVVtBYwAHo5WPKYCycmB666DRx6BG26ACROgXj2/ozImLkTzSKEDsExVl6vq70AGcGmBdVoA07zHM0I8b0zJ7N0LvXrBq6/C3//uLkJLSvI7KmPihqhqdAoW6Ql0U9VB3nx/4DRVHRq0zjjgG1V9WkS6A+8BR6lqVoGyBgODAerVq9cuIyMj32vt2rWL5OTkqNTDD4lWHyifOiVlZ5N6//3UmT+fpUOHsqZHj6i9lm2j2Jdo9YGy1alLly7zVLV9sSuqalQmoBfwStB8f+DZAuscC7wPfA88DawGahVVbrt27bSgGTNmHLIsniVafVTLoU7r16u2bataubLq2LHRfS21bRQPEq0+qmWrEzBXw9h3R/M6hdVA46D5RsDaAglpLdAdQESSgR6quj2KMZlEtGIFdO0Ka9bApEnu1FNjTKlEs09hDtBcRJqKyGFAH2BS8AoicpSI5MVwDzAmivGYRPTjj9CpE2RlwbRplhCMKaOoJQVVzQGGAlOBxcB4VV0oIiNE5BJvtbOBX0RkCVAPeCha8ZgE9MUXcNZZ7pqDWbOgY0e/IzIm7kV1mAtVnQxMLrDsgaDHE4AJ0YzBJKiPP3ZnGTVqBP/9L6Sk+B2RMQnBrmg28WfyZLjsMjdMxezZlhCMiSAbEM/El88+c7fGbNXK3RqzVi2/IzImodiRgokfc+a4cYuaNoWpUy0hGBMFlhRMfPjpJ+jWDY4+Gv73PzjqKL8jMiYhWVIwsW/ZMjj/fKhaFT79FBo29DsiYxKW9SmY2LZ6NZx3Huzf74a/Pv54vyMyJqFZUjCxa9Mmd4SwZQvMmAEtCg6ya4yJNEsKJjZt2+ZumZmZ6TqV27XzOyJjKgRLCib2ZGe7G+P89JMby+iss/yOyJgKw5KCiS379sHll8PXX8M777gzjowx5caSgokdOTnQt6875XTMGOjZ0++IjKlw7JRUEzvuuw8++ACeegoGDvQ7GmMqJEsKJjbMnQujRrl7K996q9/RGFNhWVIw/tu/H/7yF6hfHx57zO9ojKnQrE/B+O/RR+GHH2DiRBvPyBif2ZGC8dfPP8OIEe7eCJde6nc0xlR4lhSMf3JzYdAgqFEDnn3W72iMMVjzkfHT6NHulpqvvw716vkdjTEGO1IwfvntNxg2DLp2hauv9jsaY4zHkoIpf6pwww3u74svgojfERljPNZ8ZMrfuHHwySfuIjW7v7IxMcWOFEz52rTJXZx22mkwdKjf0RhjCrCkYMrXrbfCjh3w6quQlOR3NMaYAiwpmPLz0UeQnu7GOGrZ0u9ojDEhWFIw5SIpOxuGDHHJ4J57/A7HGFMI62g25eL4l16CNWtgwgQ47DC/wzHGFMKOFEz0zZpFw0mTDnYwG2NiliUFE1379sGgQeypXx8efNDvaIwxxbCkYKJr1ChYsoSlt93mxjgyxsQ0SwomejIz4aGHoEcPtnTo4Hc0xpgwWFIw0XPbbW4Iiyee8DsSY0yYopoURKSbiPwiIstEZFiI548TkRki8r2I/CAiF0YzHlOOJk92N835+9/huOP8jsYYE6aoJQURSQKeBy4AWgB9RaRFgdXuB8aralugD/BCtOIx5WjvXrjlFjjxRLj9dr+jMcaUQDSvU+gALFPV5QAikgFcCiwKWkeBI7zHtYC1UYzHlJfHHoNff4X//teuSTAmzoiqRqdgkZ5AN1Ud5M33B05T1aFB6zQA/gvUAWoA56nqvBBlDQYGA9SrV69dRkZGvud37dpFcnJyVOrhh3iuT7V16zh1wACyOnZk0fDhgeXxXKdQEq0+kHh1SrT6QNnq1KVLl3mq2r7YFVU1KhPQC3glaL4/8GyBdW4H7vAed8QdRVQqqtx27dppQTNmzDhkWTyL6/pccolqjRqqq1blWxzXdQoh0eqjmnh1SrT6qJatTsBcDWPfHc3mo9VA46D5RhzaPPQXoBuAqn4lItWAo4CNUYzLRMtHH8GkSfDII9Cokd/RGGNKIZpnH80BmotIUxE5DNeRPKnAOr8B5wKIyMlANWBTFGMy0bJnj+tcPvlk+Otf/Y7GGFNKUTtSUNUcERkKTAWSgDGqulBERuAOYyYBdwAvi8htuE7nAd5hjok3jz4KK1bAtGnWuWxMHIvqKKmqOhmYXGDZA0GPFwGdohmDKQfLl8PDD0OfPnDOOX5HY4wpA7ui2ZTdrbdClSpunCNjTFyz+ymYsvnPf1wH86hR0LCh39EYY8rIjhRM6eV1Lrdo4f4aY+JesUlBRIaKSJ3yCMbEmZEj3Uiozz/vmo+MMXEvnCOF+sAcERnvDXAn0Q7KxIFly9z1CFdeCWef7Xc0xpgIKTYpqOr9QHPgVWAAsFRE/iUizaIcm4ll99zjjg4ee8zvSIwxERRWn4J37cB6b8rBjVU0QUQejWJsJlbNmwcTJrgRUI891u9ojDERVOzZRyJyC3ANsBl4BbhLVfeLSCVgKfC36IZoYs7990PdujYsts/279/P6tWr2bt3b0TKq1WrFosXL45IWbEg0eoD4dWpWrVqNGrUiCql7OcL55TUo4DuqroyeKGq5orIRaV6VRO/Zs2CKVPcFcy1avkdTYW2evVqatasSUpKCpHo6tu5cyc1a9aMQGSxIdHqA8XXSVXJyspi9erVNG3atFSvEU7z0WRgS96MiNQUkdO8ABIrDZuiqcK990KDBnDTTX5HU+Ht3buXI488MiIJwSQGEeHII48s09FjOElhNLAraD7bW2YqmqlTYfZsd4vN6tX9jsaAJQRziLJ+JsJJChI8SJ2q5mJXQlc8ubnuKKFpU/jLX/yOxvgsKyuLNm3a0KZNG+rXr0/Dhg0D87///ntYZQwcOJBffvmlyHWef/55xo4dG4mQAdiwYQOVK1fm1VdfjViZiSacnftyr7M57+jgRmB59EIyMen99+H77+HNN20UVMORRx7J/PnzARg+fDjJycnceeed+dbJu2lLpUqhf3u+9tprxb7OTRFupnznnXfo2LEj6enp/CWKP25ycnKoXDk+fzuHc6RwA3AGsAZ345zT8G6NaSqInBzXZNSihbtYzZhCLFu2jNTUVG644QbS0tJYt24dgwcPpn379rRs2ZIRI0YE1j3zzDOZP38+OTk51K5dm2HDhtG6dWs6duzIxo3uPlv3338/Tz31VGD9YcOG0aFDB0488US+/PJLALKzs+nRowetW7emb9++tG/fPpCwCkpPT+epp55i+fLlrF+/PrD8448/Ji0tjdatW9O1a1fAdepec801nHLKKbRq1YqJEycGYs2TkZHBoEGDALjqqqu444476NKlC/feey9ff/01HTt2pG3btnTq1ImlS5cCLmHcdtttpKam0qpVK1544QWmTp1Kr169AuV+8sknXHHFFWXeHqVRbCpT1Y24G+SYiurtt+Hnn+G99yApye9oTCh//SsUsiMM1+EHDuTfvm3agLdDLolFixbx2muv8e9//xuAkSNHUrduXXJycujSpQs9e/akRYsW+f5n+/btdO7cmZEjR3L77bczZswYhg0bdkjZqsq3337LpEmTGDFiBFOmTOHZZ5+lfv36vPfeeyxYsIC0tLSQcWVmZrJ161batWtHz549GT9+PLfccgvr169nyJAhzJo1iyZNmrBlizuvZvjw4Rx99NH8+OOPqCrbtm0rtu6//vor06ZNo1KlSmzfvp3Zs2eTlJTElClTuP/++3nnnXcYPXo0a9euZcGCBSQlJbFlyxZq167NLbfcQlZWFkceeSSvvfYaAwcOLOlbHxHhjH1UTURuEpEXRGRM3lQewZkYsG8fDB8O7drB5Zf7HY2JA82aNePUU08NzKenp5OWlkZaWhqLFy9m0aJFh/zP4YcfzgUXXABAu3btyMzMDFl29+7dD1ln9uzZ9Onjfre2bt2ali1bhvzf9PR0evfuDUCfPn1IT08H4KuvvqJLly40adIEgLp16wLw6aefBpqvRIQ6dYofAq5Xr16B5rJt27bRvXt3UlNTufPOO1m4cGGg3BtuuIEkLwHXrVuXSpUqceWVVzJu3Di2bNnCvHnzAkcs5S2cRq+3gJ+BPwEjgH6AnYpaUbzyCqxcCS+9BHamS+wqxS/6gvZE6Lz+GjVqBB4vXbqUp59+mm+//ZbatWtz1VVXhTxd8rCgfqqkpCRycnJCll21atVD1gn3Zo3p6elkZWXxxhtvALB27VpWrFiBqoY8YyfU8kqVKuV7vYJ1Ca77fffdx5/+9CduvPFGli1bRrdu3QotF+Daa6+lR48eAPTu3TuQNMpbOH0Kf1DVvwPZqvoG8GfglOiGZWJCdjb885/QuTOcf77f0Zg4tGPHDmrWrMkRRxzBunXrmDp1asRf48wzz2T8+PEA/PjjjyGPRBYtWsSBAwdYs2YNmZmZZGZmctddd5GRkUGnTp2YPn06K1e663Pzmo+6du3Kc889B7gd+datW6lUqRJ16tRh6dKl5Obm8sEHHxQa1/bt22no3WPk9ddfDyzv2rUro0eP5sCBA/ler3Hjxhx11FGMHDmSAQMGlO1NKYNwksJ+7+82EUkFagEpUYvIxI7nnoMNG+Chh+wowZRKWloaLVq0IDU1leuuu45OnSJ/992bb76ZNWvW0KpVKx5//HFSU1OpVeBq+3HjxnF5gebPHj16MG7cOOrVq8fo0aO59NJLad26Nf369QPgH//4Bxs2bCA1NZU2bdowa9YsAB555BG6devGueeeS6NGjQqN6+677+auu+46pM7XX3899evXp1WrVrRu3TqQ0ACuvPJKmjZtygknnFCm96RM8k4bK2wCBuEGwDsLdyrqRuD64v4vWlO7du20oBkzZhyyLJ7FRH22blWtU0f1z3+OSHExUacIioX6LFq0KKLl7dixI6LllZf9+/frnj17VFV1yZIlmpKSovv374/L+lx//fX6+uuvF/p8uHUK9dkA5moY+9gi+xS8Qe92qOpW4HPg+KhmKBM7Hn8ctm6FBx/0OxJjirRr1y7OPfdccnJyUFVefPHFuLxGoE2bNtSpU4dnnnnG1ziKfOfUDXo3FBhf1HomwWzcCE8+CVdc4U5LNCaG1a5dm3nz5vkdRpkVdm1FeQunT+F/InKniDQWkbp5U9QjM/4ZOdLdfznoQiNjTMUQzjHWtd7f4OvNFWtKSkyrVsELL8CAAXDiiX5HY4wpZ+Fc0Vy6QblNfPrnP90Q2Q884HckxhgfhHPntatDLVfVNyMfjvHV0qUwZgzceCN4V3caYyqWcPoUTg2a/ggMBy6JYkzGLw88ANWqwX33+R2JiXFnn332IReiPfXUU9x4441F/l9ycjLgribu2bNnoWXPnTu3yHKeeuopdu/eHZi/8MILwxqbKFx5g+tVRMUmBVW9OWi6DmgL2NjJiWb+fMjIcAOr1avndzQmCsaOhZQUqFTJ/S3LbQr69u1LRkZGvmUZGRlh70iPPfZYJkyYUOrXL5gUJk+enG/00rJYvHgxubm5fP7552RnZ0ekzFAKG8rDb+EcKRS0G2ge6UCMz+67D+rUgQJj4pvEMHYsDB7shrFSdX8HDy59YujZsycfffQR+/btA9wIpGvXruXMM88MXDeQlpbGKaecwocffnjI/2dmZpKamgrAnj176NOnD61ataJ3797s2bMnsN6QIUMCw27/4x//AOCZZ55h7dq1dOnShS5dugCQkpLC5s2bAXjiiSdITU3ltNNOCwy7nZmZycknn8x1111Hy5Yt6dq1a77XCTZu3Dj69+9P165dmTRpUmD5smXLOO+882jdujVpaWn8+uuvADz66KOccsoptG7dOjCya/DRzubNm0lJSQHccBe9evXi4osvpmvXrkW+V2+++Wbgquf+/fuzc+dOTjnlFPbvd4NM7Nixg5SUlMB8xBR3dRvwH2CSN32Eu6p5ZDhXxkVjsiuao2DWLFVQHTkyai9h2yjySnJFc5MmbhMXnJo0ObhOSa8AvvDCC3XixImqqvrwww/rnXfeqaruCuPt27erquqmTZu0WbNmmpubq6qqNWrUUFXVFStWaMuWLVVV9fHHH9eBAweqquqCBQs0KSlJ58yZo6qqWVlZqqqak5OjnTt31gULFnj1aaKbNm0Kqp+bnzt3rqampuquXbt07dq12qJFC/3uu+90xYoVmpSUpN9//72qqvbq1UvfeuutkPVq3ry5ZmZm6tSpU/Xiiy8OLO/QoYO+//77qqq6Z88ezc7O1smTJ2vHjh01Ozs7X7ydO3cO1GHTpk3axHujX3vtNW3YsGFgvcLeq59++klPOOGEQB3z1u/Xr59+8MEHqqr64osv6u233x6yDmW5ojmcI4VRwOPe9DBwlqoeOtC5iU+qcM890KAB3Hyz39GYKPntt5ItD0dwE1Jw05Gqcu+999KqVSvOO+881qzbs8s6AAAXaUlEQVRZw4YNGwot5/PPP+eqq64CoFWrVrRq1Srw3Pjx40lLS6Nt27YsXLgw5GB3wWbPns3ll19OjRo1SE5Opnv37oExi5o2bUob72LMwobnnjNnDkcffTRNmjTh3HPP5bvvvmPr1q3s3LmTNWvWBMZPqlatGtWrV+fTTz9l4MCBVPfuWZ437HZRzj///MB6hb1X06dPp2fPnhx11FH5yr3mmmsCd6yL1j0XwkkKvwHfqOpnqvoFkCUiKRGPxPhjyhSYPdvdWc37YJvEc9xxJVsejssuu4xp06bx3XffsWfPnsDNbcaOHcumTZuYN28e8+fPp169eiGHyw4WaijpFStWMGrUKKZNm8YPP/zAn//852LL0SKG0c4bdhsKH547PT2dn3/+mZSUFJo1a8aOHTt47733Ci1XCxkGu3LlyuTm5gJFD69d2HtVWLmnn346mZmZfPbZZxw4cCDQBBdJ4SSFd4HcoPkD3rJiiUg3EflFRJaJyCFHFyLypIjM96YlIhK50wdM8XJz4d57oWlTiOL9ao3/Hnro0JxfvbpbXlrJycmcffbZXHvttfk6mLdv384xxxxDlSpVmDFjRmBI6sKcddZZjPU6N3766Sd++OEHwLWZ16hRg1q1arFhwwY++eSTwP/UrFmTnTt3hixr4sSJ7N69m+zsbD744AP++Mc/hlWf3Nxc3n33XX744YfA8Noffvgh6enpHHHEETRq1IiJEycCsG/fPnbv3k3Xrl0ZM2ZMoNM7bxjslJSUwNAbRXWoF/ZenXvuuYwfP56srKx85QJcffXV9O3bN2p3ZgsnKVRW1d/zZrzHxZ59JCJJwPPABUALoK+I5LsHn6repqptVLUN8CzwfkmCN2U0YYI762jECDjMTihLZP36ufskNWniRkFv0sTNe6NEl1rfvn1ZsGBB4M5n7rX6MXfuXNq3b8/YsWM56aSTiixjyJAh7Nq1i1atWvHoo4/SoUMHwJ0W2rZtW1q2bMm1116bbwjqwYMHc8EFFwQ6mvOkpaUxYMAAOnTowDnnnMOgQYNo27ZtWHX5/PPPadiwYeAeCOCSzKJFi1i3bh1vvfUWzzzzDK1ateKMM85g/fr1dOvWjUsuuYT27dvTpk0bRo0aBcCdd97J6NGjOeOMMwId4KEU9l61bNmS++67j86dO9O6dWtuv/32fP+zdevW6J0yW1ynA/A/4JKg+UuBaWH8X0dgatD8PcA9Raz/JXB+ceVaR3OE7N+vesIJqqmpqjk5UX8520aRZ0NnFy3R6qPq6vTuu+/qVVddVeR6URs623MDMFZEnvPmVwMhr3IuoCGwKmh+NXBaqBVFpAnQFJheyPODgcEA9erVY+bMmfme37Vr1yHL4ll51KfBxx9z4pIl/Pjgg2R5HXHRZNso8mrVqhWyCaW0Dhw4ENHy/JZo9QG44447+PTTT5kwYUKRddu7d2/pP5/hZA6XZEgGapZg/V7AK0Hz/YFnC1n37sKeKzjZkUIE7Nmj2qiR6umnq3qnCkabbaPIsyOFoiVafVTL5yY7xfYpiMi/RKS2qu5S1Z0iUkdEwrnzymqgcdB8I2BtIev2AdLDKNNEwujRsHo1/OtfdptNY0w+4XQ0X6CqgbOC1N2F7cIw/m8O0FxEmorIYbgd/6SCK4nIibjbfX4VXsimTHbudMng/POhQCediT9axCmYpmIq62cinKSQJCKBE3xF5HCgahHrA6CqOcBQYCqwGBivqgtFZISIBA+o1xfIUPt0l48nnoDNm8t2LqKJCdWqVSMrK8sSgwlQVbKysqhWrVqpywino/ltYJqIvObNDwTeCKdwVZ0MTC6w7IEC88PDKctEwObN7t7L3bvDqaf6HY0po0aNGrF69Wo2bdoUkfL27t1bpp1JrEm0+kB4dapWrRqNGjUq9WuEc5OdR0XkB+A8QIApgA22H49GjoTsbHgwnC4hE+uqVKlC06aRuwfWzJkzwz6nPx4kWn2gfOoU7iip63FXNfcAzsU1B5l4sno1PPccXH01nHyy39EYY2JUoUcKInICrnO4L5AFvAOIqlrvZDz65z/dsBbe8MPGGBNKUc1HPwOzgItVdRmAiNxWLlGZyFq6FF591d1m0xvX3RhjQimq+agHrtlohoi8LCLn4voUTLx55BE3tpHdZtMYU4xCk4KqfqCqvYGTgJnAbUA9ERktIl3LKT5TVps3u9trXX213WbTGFOscO7RnK2qY1X1ItxVyfMBu8lOvHjpJdi7F265xe9IjDFxoET3aFbVLar6oqqeE62ATATt3w8vvOCuXm7Rovj1jTEVXjgXr5l49d57sGYNvPii35EYY+JEiY4UTJx5+mlo3hwuuMDvSIwxccKSQqL69lv4+mu4+WaoZJvZGBMe21skqqefhpo1YcAAvyMxxsQRSwqJaO1aGD8err3WJQZjjAmTJYVE9O9/w4EDrunIGGNKwJJCotm71yWFiy6CZs38jsYYE2csKSSajAzYtAluvdXvSIwxcciSQiJRdR3Mqalwjl1faIwpObt4LZHMmgXz57uhLcTGLjTGlJwdKSSSp5+GunWhXz+/IzHGxClLCokiMxMmToTrroPq1f2OxhgTpywpJIrnn3dNRjfd5Hckxpg4ZkkhEWRnwyuvQPfu0Lix39EYY+KYJYVE8OabsG2bnYZqjCkzSwrxLjcXnnkG2rWDM87wOxpjTJyzU1Lj3f/+Bz//7I4W7DRUY0wZ2ZFCvHvmGXfv5Suu8DsSY0wCsKQQz5YsgcmTYcgQqFrV72iMMQnAkkI8e/ZZqFIFrr/e70iMMQnCkkK82rEDXn8d+vSB+vX9jsYYkyAsKcSrt9+GXbtg6FC/IzHGJBBLCvFIFV54AdLS4NRT/Y7GGJNA7JTUeDR7Nixc6K5ittNQjTERFNUjBRHpJiK/iMgyERlWyDpXiMgiEVkoIuOiGU/CGD0aatVy/QnGGBNBUTtSEJEk4HngfGA1MEdEJqnqoqB1mgP3AJ1UdauIHBOteBLGxo0wYYI7DbVGDb+jMcYkmGgeKXQAlqnqclX9HcgALi2wznXA86q6FUBVN0YxnsTw6quwfz/ccIPfkRhjEpCoanQKFukJdFPVQd58f+A0VR0atM5EYAnQCUgChqvqlBBlDQYGA9SrV69dRkZGvud37dpFcnJyVOrhh0Lrc+AAp/frx55jj2XBE0+Uf2BlUGG2URxLtDolWn2gbHXq0qXLPFVtX+yKqhqVCegFvBI03x94tsA6HwEfAFWAprhmptpFlduuXTstaMaMGYcsi2eF1uejj1RBdfz4co0nEirMNopjiVanRKuPatnqBMzVMPbd0Ww+Wg0ED+7fCFgbYp0PVXW/qq4AfgGaRzGm+PbCC+5Ctcsu8zsSY0yCimZSmAM0F5GmInIY0AeYVGCdiUAXABE5CjgBWB7FmOLXihXwySfudptVqvgdjTEmQUUtKahqDjAUmAosBsar6kIRGSEil3irTQWyRGQRMAO4S1WzohVTXHvpJXdNwnXX+R2JMSaBRfXiNVWdDEwusOyBoMcK3O5NpjD79rkL1S6+2G63aYyJKhvmIh689x5s3gw33uh3JMaYBGdJIR6MHg3NmsF55/kdiTEmwVlSiHU//ujGOrrhBqhkm8sYE122l4l1o0e7u6oNHOh3JMaYCsCSQizbuRPeegt694Yjj/Q7GmNMBWBJIZaNHetupDNkiN+RGGMqCEsKsSrvRjpt2sBpp/kdjTGmgrCb7MSqL790ncx5F60ZY0w5sCOFWDV6NBxxBFx5pd+RGGMqEEsKMajKtm3w7rtw9dV2Ix1jTLmypBCD6n/yCfz+u3UwG2PKnSWFWJOby7H/+Q907gwtWvgdjTGmgrGkEGs+/pjD162zowRjjC8sKcSSlSth0CB2N2oEl1/udzTGmArIkkKs2LEDLroI9u3jp4cegsMO8zsiY0wFZNcpxIIDB9ypp4sXw5Qp7K5sm8UY4w87UogFd90FH38Mzz1nw2MbY3xlScFvL70ETz4Jt9zihsc2xhgfWVLw0/TpcNNN0K0bPP6439EYY4wlBd8sWQI9esAJJ0BGBlg/gjEmBlhS8MOWLe5Mo8qV4aOPoFYtvyMyxhjAzj4qf/v3Q8+e7pqE6dOhaVO/IzLGmABLCuVJ1fUhzJgBb74JnTr5HZExxuRjzUfl6amn4OWX4Z57oH9/v6MxxphDWFIoLx9/DHfcAd27w4MP+h2NMcaEZEkh2latctcfXHYZtG3rmo0q2dtujIlNtneKlnXr4Oab4Q9/gDFjYPBgmDLFbppjjIlp1tEcaRs3wiOPwAsvuDONBg6E+++HJk38jswYY4plRwqRkpXlOpCPP951KPfuDb/8Ai+/zNjZTUhJca1GKSkwdqzfwRpjTGiWFEpp7Fi8Hb2SUnsbYxve5Y4QLrkEFi2C11+HZs0YO9a1HK1c6c5IXbnSzVtiMMbEIksKBRzc2Rf4Va8K69fDrFmMvW4Ggwf87u3ohZXbazM45wXGPrwSxo2DE08MlHfffbB7d/7X2L3bLTfGxIdC9wsJyJJCEPerXvP/qr9mH2Ob3ueGomjQAM46i/teacrunPw3wdl9oBr3jW58SJm//Rb6tUItz/vgnXNO55j+4FWkL4iJHX597irc0b6qRm0CugG/AMuAYSGeHwBsAuZ706DiymzXrp0WNGPGjEOWlUhOjurnn2uTmlnqNnv+qUm19apDh6o+84zqlCkqkhtyPZFDi27S5ND1wC0P9vbbqtWr51+nenW3PJS333ZliLi/ZV0vXCWNM0+Zt1GMOPh+5kbk/YwlsbyNSvO5i1R9wv0Ol4ey1AmYq+Hst8NZqTQTkAT8ChwPHAYsAFoUWGcA8FxJyi1NUgi5Y9y7V3XyZNVBg1SPOUYVVDgQ1s6+JB+ScD/M0SiztDvwopT2CxLLOxzV8JJnNN7PWBLJbRTpHyMl/X5EMnGLhH7tUD8C879+ZOqev8zS1ykWkkJHYGrQ/D3APQXWiXpSCPlFTtqrb1cd6GZq1lTt3Vs1I0ObNA6dFMrjV31JPnjhfkGi8QunpF+QPKVK3OUkGok7r1y/6lQakdpGJfl+hFtmuJ+7aHw3o/GDrSQiVWa4SUHcupEnIj2Bbqo6yJvvD5ymqkOD1hkAPOw1IS0BblPVVSHKGgwMBqhXr167jIyMfM/v2rWL5OTkkHH06XM6GzZUO2R5w8M3MvmBMWxNS0MPc/0Dn356DKNGnci+fUmB9apWPcCdd/7CeedtzPf/n356DK+8cjwbN1blmGP2MWjQ8kPWKYnC4qxXby8ZGV/nW3bOOZ1RlUPWFVGmT/+sxOvlCadOJYkz3DJL8r6XRLjbKNw6leT99POzVFpFfY9KUp9w389olFmSz2e4rx+NOPPKjeTnszhdunSZp6rti10xnMxRmgnoBbwSNN8feLbAOkcCVb3HNwDTiyu3pEcKsXDoF46S/BqIxpFCNJqk/PwFXpI4w/2MlCTOeOxLKup7VJK6R+P9DPd9isYRd97r+3VEU9qj84KIh+ajAusnAduLK7ekSSGWOomKE267YTR24NH4goRbZkk+9NFINNHYgfu5Yyx9oin8cxeNnW00frBFI3mVRDR+sEVqHxYLSaEysBxoysGO5pYF1mkQ9Phy4Oviyo1In0KMdw6G0+EX6V+M0fiC+PkLPBqJJm/dcBJ3NOL08wjR7zLDFa0fQpF+/Wh9Povie1JwMXAhrq/gV+A+b9kI4BLv8cPAQi9hzABOKq7MiJ19FMP8OFMnGl+QePkFnhdDST4j4XzmIr1jDLfu0Ug00WjmitYPtkgfcZf+9SNzRFOSOhUlJpJCNKaoXKcQY/yoj99nTUS6SSraR4iROprzuy+ppL9YI/3jKpo/2CJ5xB1ppf18xvV1CtGaLClET3S/9JE5ZzwaiaY0/Din3+++pHgT6/uFSJ8MUBxLCnEs0eqjGtsXRpVGrCfuSCeaeGTfo/zCTQp2PwUTd/r1c1NFFG7dS7IeuAEaf/tNOe444aGHKu77a2xAPGMqvH79IDMTpk//jMxMSwgVnSUFY4wxAZYUjDHGBFhSMMYYE2BJwRhjTIAlBWOMMQFRGzo7WkRkE7CywOKjgM0+hBMtiVYfSLw6JVp9IPHqlGj1gbLVqYmqHl3cSnGXFEIRkbkazjjhcSLR6gOJV6dEqw8kXp0SrT5QPnWy5iNjjDEBlhSMMcYEJEpSeMnvACIs0eoDiVenRKsPJF6dEq0+UA51Sog+BWOMMZGRKEcKxhhjIsCSgjHGmIC4Tgoi0k1EfhGRZSIyzO94IkFEMkXkRxGZLyJz/Y6nNERkjIhsFJGfgpbVFZH/ichS728dP2MsiULqM1xE1njbab6IXOhnjCUhIo1FZIaILBaRhSJyq7c8nrdRYXWKy+0kItVE5FsRWeDV5/+85U1F5BtvG70jIodF/LXjtU9BRJJw938+H1gNzAH6quoiXwMrIxHJBNqratxedCMiZwG7gDdVNdVb9iiwRVVHegm8jqre7Wec4SqkPsOBXao6ys/YSkNEGgANVPU7EakJzAMuAwYQv9uosDpdQRxuJxERoIaq7hKRKsBs4FbgduB9Vc0QkX8DC1R1dCRfO56PFDoAy1R1uar+DmQAl/ockwFU9XNgS4HFlwJveI/fwH1h40Ih9YlbqrpOVb/zHu8EFgMNie9tVFid4pJ3s7Rd3mwVb1LgHGCCtzwq2yiek0JDYFXQ/Gri+EMQRIH/isg8ERnsdzARVE9V14H7AgPH+BxPJAwVkR+85qW4aWoJJiIpQFvgGxJkGxWoE8TpdhKRJBGZD2wE/gf8CmxT1Rxvlajs8+I5KUiIZfHZFpZfJ1VNAy4AbvKaLkzsGQ00A9oA64DH/Q2n5EQkGXgP+Kuq7vA7nkgIUae43U6qekBV2wCNcC0jJ4daLdKvG89JYTXQOGi+EbDWp1giRlXXen83Ah/gPgyJYIPX7pvX/rvR53jKRFU3eF/aXOBl4mw7ee3U7wFjVfV9b3Fcb6NQdYr37QSgqtuAmcDpQG0Rqew9FZV9XjwnhTlAc683/jCgDzDJ55jKRERqeJ1kiEgNoCvwU9H/FTcmAdd4j68BPvQxljLL23l6LieOtpPXifkqsFhVnwh6Km63UWF1itftJCJHi0ht7/HhwHm4fpIZQE9vtahso7g9+wjAO73sKSAJGKOqD/kcUpmIyPG4owOAysC4eKyTiKQDZ+OG+d0A/AOYCIwHjgN+A3qpalx03hZSn7NxTRIKZALX57XHxzoROROYBfwI5HqL78W1wcfrNiqsTn2Jw+0kIq1wHclJuB/v41V1hLePyADqAt8DV6nqvoi+djwnBWOMMZEVz81HxhhjIsySgjHGmABLCsYYYwIsKRhjjAmwpGCMMSbAkoIxHhE5EDSa5vxIjrwrIinBo6waE6sqF7+KMRXGHm9YAWMqLDtSMKYY3j0uHvHGt/9WRP7gLW8iItO8wdamichx3vJ6IvKBNxb+AhE5wysqSURe9sbH/693pSoicouILPLKyfCpmsYAlhSMCXZ4geaj3kHP7VDVDsBzuKvo8R6/qaqtgLHAM97yZ4DPVLU1kAYs9JY3B55X1ZbANqCHt3wY0NYr54ZoVc6YcNgVzcZ4RGSXqiaHWJ4JnKOqy71B19ar6pEishl3Y5f93vJ1qnqUiGwCGgUPP+AN5/w/VW3uzd8NVFHVB0VkCu4mPhOBiUHj6BtT7uxIwZjwaCGPC1snlOAxag5wsE/vz8DzQDtgXtAomMaUO0sKxoSnd9Dfr7zHX+JG5wXoh7tlIsA0YAgEbpRyRGGFikgloLGqzgD+BtQGDjlaMaa82C8SYw463LvTVZ4pqpp3WmpVEfkG90Oqr7fsFmCMiNwFbAIGestvBV4Skb/gjgiG4G7wEkoS8LaI1MLdOOpJb/x8Y3xhfQrGFMPrU2ivqpv9jsWYaLPmI2OMMQF2pGCMMSbAjhSMMcYEWFIwxhgTYEnBGGNMgCUFY4wxAZYUjDHGBPw/1/SI89GiWokAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3XmYVNWd//H3F2hAaBZFdoXGJQs7DZK4IRjNCG4JYpQBjDGEaEw042hCNBMdf/rEMW7RGI1JNDEgaDRRhxBNVIgaowhEUGCiyKIIsqnsoA3f3x/ndlHdVDfVdN2uqtuf1/Pcp6vuvXXqnCq43zrLPcfcHREREYAm+c6AiIgUDgUFERFJUVAQEZEUBQUREUlRUBARkRQFBRERSVFQkNiYWVMz22pmPXJ5bjEys4lmNjt6XGtZ0889wPf6i5mNO9DXS+OmoCAp0YWqcttjZjvSntf5IuPuu9291N3fyeW5Dc3MWpnZZjMbluHYXWY2vS7p5bKsZnaDmf2mWvpfdPep9U07w3tNMbPrcp2uFBYFBUmJLlSl7l4KvAOcmbZvn4uMmTVr+Fw2PHffDvweuCB9v5mVAOcDv81HvkTioKAgWYt+lT5sZtPMbAsw3syONbOXzewjM1tjZndGF0vMrJmZuZmVRc+nRMf/bGZbzOwfZtarrudGx0ea2Ztmtin6tf53M7swQ54PN7PtZtYubd8xZrYues9PmdnzUTobzOyhGor/W+BcMzsobd9IoAL4S5TuD81sWZTfRWZ2Vg2fY/WydjSzGVFt5GWgV7Xzf2Zmq6Ljr5rZcdH+M4DvAeOi2ty8aP+LlZ+FmTUxsx+Z2cqozL8xs7bRsaOifFwQpb/ezCbXUP5amdkJZjY3+hznmNnn0o593cxWRJ/LMjM7P9qf7WcvDUhBQerqy8BDQDvgYcJF8XLgUOB44DTgm7W8/t+B/wIOIdRG/l9dzzWzTsAjwFXR+y4HhmZKwN3fBeYCo6ul+4i7VwA3An8CDgYOA+6uIS8vABuBs9P2TQCmuvvu6PmbhM+gXZTuQ2bWuZbyVboH2AJ0ASYBF1U7/grQn/A5PAr83sxauPsM4OYoD6XuPjhD2hOB8cBw4MionD+tds5xwFHAvwH/bWZHZ5HnFDM7lPAZ3gp0AO4EZprZwVEAug041d3bED6fhdFLs/3spQEpKEhdveju/+vue9x9h7u/6u6vuHuFuy8D7gNOquX1j7r7XHf/BJgKDDyAc88AXnP3J6JjtwMbaknnIWAshF/OwHnRPoBPgDKgq7vvdPe/Z0rAwyRhDxI1IZlZe+BM0pqO3P0Rd18TfTYPASuAIbXkq7IJ6kvAf7n7dndfCPyu2nv/zt0/iILYzUBbwkU8G+OAW9x9ubtvAa4G/j36HCpdF5V9PrAIGJBl2pXOBBa5+7To38EUYBlwemURgL5m1jL6fBZH+7P67KVhKShIXb2b/sTMPmNmfzKz981sM3A94dd7Td5Pe7wdKD2Ac7ul5yO6YK+qJZ3fAydGv9pHADvd/aXo2H8CJcBcM3vdzL5aSzoPAqeaWRfgK8Bid3+98qCZXWhmC6KmtI+Az1D7ZwHQGWhK1c91ZfoJZvY9M/s/M9sEfAi0ziLdSt2qpbcSaA50rNzh7nX5TrJ5j8r36e7umwkB+VLg/aiZ7FPROXX57KWBKChIXVWfVvcXwBvAUe7eFvgRYDHnYQ2huQEAMzOge00nu/tG4DngXELT0bS0Y2vcfaK7dyVcuO5L77uols4y4B9RGhMIQaIyD0cQmoEuATq4e3vg/9j/Z7EW2AMcnrYvNVTVzEYAVwDnAO0JTS1b09Ld3zTHq4Ge1dL+GFi/n9fVRfX3qHyf9wDc/c/ufgrQFVhK+DdTp89eGo6CgtRXG2ATsM3MPkvt/Qm5MgMoN7MzLYyAupy0X741eAj4KqFvIdWhaWZfMbPKgPIR4SK7e9+Xp/w2er/PpadD+HXthIutmdlEQk2hVlHz1+OEtvyDzKwvIeBUakPot9lA+FV9HaGmUGktUBYFxkymAVeYWZmZtSG0409z9z37y1sNmplZy7StOeH76GNm50Wd6P9OaN6aaWZdo++pFSEYbSP6fA/gs5cGoKAg9fWfhIvtFsIvwIfjfkN3X0voF7iN0Pl7JPBPYFctL3sc6A284+6L0vZ/DnjVzLYBfwAu3c/9A78nNN087e7r0vK0kNDBOodQk/kMoYM4G5cQagBrgV8DD6Qdmwk8A7xF6KPYHKVf6WFCc9AHZjYnQ9q/jM55gdDOv4UQ1A7UNcCOtO0v7r4eOAv4PuH7+A/gDHf/gNA0dlWU542ETu1vR2nV9bOXBmBaZEeKnZk1JTRhjHH3F/KdH5FippqCFCUzO83M2plZC8Kw1QrCr3QRqQcFBSlWJxCaQzYQ7o34krvX1nwkIllQ85GIiKSopiAiIilFN6HZoYce6mVlZVX2bdu2jdatW2d+QRFKWnkgeWVKWnkgeWVKWnmgfmWaN2/eBnff39Dt4gsKZWVlzJ07t8q+2bNnM3z48PxkKAZJKw8kr0xJKw8kr0xJKw/Ur0xmVv2u84zUfCQiIikKCiIikqKgICIiKUXXp5CJmbF8+XJ27tyZ76zkRLt27ViyZEm+s5FTNZWpZcuWHHbYYZSUlOQhVyJSXSKCQuvWrWnTpg1lZWXUPC9Y8diyZQtt2rTJdzZyKlOZ3J2NGzeyatUqevXS5JgihSARzUdNmzalQ4cOiQgIjYmZ0aFDh8TU8JJu6lQoK4MmTcLfqfus2i1JkIigACggFCl9b8Vh6lSYNAlWrgT38HfSpIYNDApKDSMxQUFE4nPNNbB9e9V927eH/Q0h30GpMQUkBYUc2LhxIwMHDmTgwIF06dKF7t27p55//PHHWaXxta99jX/961+1nnP33XczNUf/Gk844QRee+21nKQlyfdODasc1LQ/1/IZlPIdkBpaIjqa861Dhw6pC+x1111HaWkpV155ZZVz3B13p0mTzHH4gQceyLg/3aWXXlr/zIocgB49wsUw0/6GkM+gVFtAGjcu/vdvaKopxGjp0qX07duXiy++mPLyctasWcOkSZMYMmQIffr04frrr0+dW/nLvaKigsMPP5zJkyczYMAAjj32WNatCwt8/fCHP+SOO+5InT958mSGDh3Kpz/9aV56KaxDv23bNs455xwGDBjA2LFjGTJkSNY1gh07dvDVr36Vfv36UV5ezvPPPw/A66+/zjHHHMPAgQPp378/y5YtY8uWLYwcOZIBAwbQt29fHn300Vx+dFJgbrwRWrWquq9Vq7C/IdQUfBoiKOW7ltTQkldT+O53IdfNIgMHQnQxrqvFixfzwAMPcO+99wJw0003ccghh1BRUcGIESMYM2YMvXv3rvKaTZs2cdJJJ3HTTTdxxRVXcP/99zN58uR90nZ35syZw5NPPsn111/PU089xV133UWXLl147LHHWLBgAeXl5Vnn9c4776R58+a8/vrrLFq0iFGjRvHWW2/x85//nCuvvJLzzjuPXbt24e488cQTlJWV8ec//zmVZ0muyl/E11wTLoY9eoSA0FC/lG+8MTTZpP9ib6iglO9aUkNTTSFmRx55JMccc0zq+bRp0ygvL6e8vJwlS5awePHifV5z0EEHMXLkSAAGDx7MihUrMqY9evTofc558cUXOf/88wEYMGAAffr0yTqvL774IhMmhDXj+/TpQ7du3Vi6dCnHHXccN9xwAzfffDPvvvsuLVu2pH///jz11FNMnjyZv//977Rr1y7r95HiNG4crFgBe/aEvw3ZdDJuHNx3H/TsCWbh7333NUwe8l1LamjJqykc4C/6uKRPc/vWW2/x05/+lDlz5tC+fXvGjx+fcYx+8+bNU4+bNm1KRUVFxrRbtGixzzn1WTSpptdOmDCBY489lj/96U+ceuqp/Pa3v2XYsGHMnTuXmTNnctVVV3HGGWdw9dVXH/B7i+zPuHH5acPPdy2pocVWUzCzw81slpktMbNFZnZ5hnOGm9kmM3st2n4UV34KwebNm2nTpg1t27ZlzZo1PP300zl/jxNOOIFHHnkECH0BmWoiNRk2bFhqdNOSJUtYs2YNRx11FMuWLeOoo47i8ssv5/TTT2fhwoW89957lJaWMmHCBK644grmz5+f87KIFIp81pIaWpw1hQrgP919vpm1AeaZ2V/dvfpV6gV3PyPGfBSM8vJyevfuTd++fTniiCM4/vjjc/4e3/nOd7jgggvo378/5eXl9O3bt8amnX/7t39LzTl04okncv/99/PNb36Tfv36UVJSwoMPPkjz5s156KGHmDZtGiUlJXTr1o0bbriBl156icmTJ9OkSROaN2+e6jMRkSJXOVQy7g14Aji12r7hwIy6pDN48GCvbv78+fvsK2abN28+4Nd+8sknvmPHDnd3f/PNN72srMw/+eSTXGXtgNVWpsWLFzdgTnJj1qxZ+c5CziWtTEkrj3v9ygTM9SyusQ3Sp2BmZcAg4JUMh481swXAauBKd1+U4fWTgEkAnTt3Zvbs2VWOt23bli1btuQ203m0e/fuAy7PRx99xFlnnUVFRQXuzu23386OHTtynMO6q61MO3fu3Oc7LXRbt24tujzvT9LKlLTyQAOVKZvIUZ8NKAXmAaMzHGsLlEaPRwFv7S891RSKk2oKhS9pZcqmPFOmuPfs6W4W/k6ZEneu6qchagqxDkk1sxLgMWCqu/8hQ0Da7O5bo8czgRIzOzTOPImIQOObviJbcY4+MuDXwBJ3v62Gc7pE52FmQ6P8bIwrTyIilfI9yV+hirNP4XhgAvC6mVXeYnw10APA3e8FxgCXmFkFsAM4P6rmiIjEqrFNX5Gt2IKCu78I1DpZvrv/DPhZXHkQEalJY5u+Ilua5iIHhg8fvs+NaHfccQff+ta3an1daWkpAKtXr2bMmDE1pj137txa07njjjvYnlYPHjVqFB999FE2Wa/Vddddxy233FLvdEQKUVzTV8Sx9kJlmieffFLs6zk0yqCQ6y9t7NixTJ8+vcq+6dOnM3bs2Kxe361bt3rNMlo9KMycOZP27dsfcHoijUEc8ynF0XldNU2LvUO80QWFOL60MWPGMGPGDHbt2gXAihUrWL16NSeccAJbt27lC1/4AuXl5fTr148nnnhin9evWLGCvn37AmH66gsvvJD+/ftz3nnnVbnH4JJLLklNu33ttdcCYWbT1atXM2LECEaMGAFAWVkZGzZsAOC2226jb9++9O3bNzXt9ooVK/jsZz/LN77xDfr06cMXv/jFOt3LkCnNbdu2cfrpp6em0n744YcBmDx5Mr179+bYY4/dZ40JkXzL9fQVcXReN3iHeDbjVgtpq+99Cj17uodwUHXr2TPrJDIaNWqUP/744+7u/uMf/9ivvPJKdw93GG/atMnd3devX+9HHnmk79mzx93dW7du7e7uy5cv9z59+ri7+6233urjx493d/cFCxZ406ZN/dVXX3V3940bN7q7e0VFhZ900km+YMGCqEw9ff369WllDM/nzp3rffv29a1bt/qWLVu8d+/ePn/+fF++fLk3bdrU//nPf7q7+7nnnuu/+93v9inTtdde6z/5yU+q7KspzUcffdQnTpyYOu+jjz7yjRs3+qc+9Snfs2ePb9682T/88MOMn53uUygMSStTPspjlvn6Ypb/NCmE+xQKUVwjDtKbkNKbjtydq6++mv79+3PKKafw3nvvsXbt2hrTef755znvvPMA6N+/P/37908de+SRRygvL2fQoEEsWrRov5Pdvfjii3z5y1+mdevWlJaWMnr0aF544QUAevXqxcCBA4Hap+fONs1+/frxzDPP8P3vf58XXniBdu3a0bZtW1q2bMnEiRN58sknaVW9AVckYeJYDKihFxhqdEEhrg/4S1/6Es8++yzz589nx44dqcVtpk6dyvr165k3bx6vvfYanTt3zjhddrro1o0qli9fzi233MKzzz7LwoULOf300/ebjtcyurdy2m2ofXrubNP81Kc+xbx58+jXrx8/+MEPuP7662nWrBlz5szhnHPOYcaMGZx22mlZvYdIsYqj87qh13NodEEhrg+4tLSU4cOHc9FFF1XpYN60aROdOnWipKSEWbNmsTLTGLg0w4YNS019/cYbb7Bw4UIgTLvdunVr2rVrx9q1a1MrngG0adMm47xCw4YN4/HHH2f79u1s27aNP/7xj5x44on1KmdNaa5evZpWrVoxfvx4rrzySubPn8/WrVvZtGkTo0aN4qabbsp6WVCRYhVH53XVND32BYaSt8jOfsS5YMbYsWMZPXp0lZFI48aN48wzz2TIkCEMHDiQz3zmM7WmcckllzB+/Hj69+/PwIEDGTp0KBBWURs0aBB9+vTZZ9rtSZMmMXLkSLp27cqsWbNS+8vLy7nwwgtTaUycOJFBgwZl3VQEcMMNN6Q6kwFWrVqVMc2nn36aq666iiZNmlBSUsI999zDli1bOPvss9m5cye7d+/m9ttvz/p9RYpVHIsBVaY5e/bfGD58eG4Try6bjodC2jQhXnHShHiFL2llKvTyHMhkfImZOltERPaqHBpfOdS0cmg85H9Vt0bXpyAikm+FPBlfYoKC1zLSRgqXvjdpjAp5Mr5EBIXdu3ezceNGXWCKjLuzceNGWrZsme+siDSohr73oC4S0aewbds2tmzZwvr16/OdlZzYuXNn4i6UNZWpZcuWHHbYYXnIkSTV1KmVowtPyunowly68caqfQoQ770HdZGIoODu9OrVK9/ZyJnZs2czaNCgfGcjp5JYJik8VTtwraA6cNPFOTS+vhLRfCQiAoXdgVtdrifjyxUFBRFJjELuwC0WCgoikhiF3IFbLBQURCQxGnryuCRSUBCRxGjoyeOSSEFBRBKlsgP3uef+VlAduMVCQUFERFIUFEREJEVBQUREUhQUREQkRUFBRERSFBRERCRFQUFERFIUFEQkb6ZOhbIyaNIk/J06Nd85kkRMnS0ixaeQ1yluzGKrKZjZ4WY2y8yWmNkiM7s8wzlmZnea2VIzW2hm5XHlR0QKSzFNc92YxNl8VAH8p7t/Fvg8cKmZ9a52zkjg6GibBNwTY35EpIFk0yykaa4LU2xBwd3XuPv86PEWYAnQvdppZwMPevAy0N7MusaVJxGJX2Wz0MqV4L63Wah6YNA014XJGmKxezMrA54H+rr75rT9M4Cb3P3F6PmzwPfdfW61108i1CTo3Lnz4OnTp1dJf+vWrZSWlsZZhAaVtPJA8sqUTXmeeaYTv/rVEaxb14JOnXYxceIyTjllXQPlsO5y9R2df/7nWbt23/W4O3feyfTpL6eeP/NMJ2655dPs2tU0ta9Fi91ceeW/cvI5Je3fHNSvTCNGjJjn7kP2e6K7x7oBpcA8YHSGY38CTkh7/iwwuLb0Bg8e7NXNmjVrn33FLGnlcU9emfZXnilT3Fu1cg+/lcPWqlXYX6hy9R2ZVS135Wa277lTprj37BmO9eyZ288naf/m3OtXJmCuZ3HNjnVIqpmVAI8BU939DxlOWQUcnvb8MGB1nHkSaQhxdaIWwxDOujQLFeo6xY1ZnKOPDPg1sMTdb6vhtCeBC6JRSJ8HNrn7mrjyJNJQ4uhEzbatPt+0+llxi7OmcDwwATjZzF6LtlFmdrGZXRydMxNYBiwFfgl8K8b8iDSYODpR4659nHzySTmpfVRd/QytflZkYrt5zUPnse3nHAcujSsPIvly441Vb8yC+v9ajrP2EfJpObuBbNw4BYFipWkuRGIQx6/lYqp9SPFSUBCJSa47UeNoq9cNZFKdgoJIkSiW2ocUNwUFkSJSDLUPKW4KCiKNWNXah2ukkCgoiCRRXW5yq6x9PPfc33QDmWg9BZGk0ToFUh+qKYgkjIaZSn0oKIgkjIaZSn0oKIgkjIaZSn0oKIgkjIaZSn0oKIgkjCakk/pQUBCpg1zPKBoXrVMgB0pDUkWyFNeMoiKFRDUFkSxpqKc0BgoKIlnSUE9pDBQURLKkoZ7SGCgoiGRJQz2lMVBQEMmSZhSVxkBBQaQONKOoJJ2CgoiIpCgoiORZXdY+EImbbl4TySOtfSCFRjUFkTzSDXFSaBQURPJIN8RJoVFQEMkj3RAnhUZBQSSPdEOcFBoFBZE80toHUmg0+kgkz8aNUxCQwqGagoiIpCgoiIhISmxBwczuN7N1ZvZGDceHm9kmM3st2n4UV15ERCQ7cfYp/Ab4GfBgLee84O5nxJgHERGpg9hqCu7+PPBBXOmLiEjumbvHl7hZGTDD3ftmODYceAxYBawGrnT3RTWkMwmYBNC5c+fB06dPr3J869atlJaW5jLreZW08kDyypS08kDyypS08kD9yjRixIh57j5kvye6e2wbUAa8UcOxtkBp9HgU8FY2aQ4ePNirmzVr1j77ilnSyuNe+GWaMsW9Z093s/B3ypTazy/08hyIpJUpaeVxr1+ZgLmexTU2b6OP3H2zu2+NHs8ESszs0HzlRxqvyplKV64E970zlWoKa2mMsgoKZnakmbWIHg83s8vMrH193tjMupiZRY+HRnnZWJ80RQ6EZioV2Svb0UePAUPM7Cjg18CTwEOEZp+MzGwaMBw41MxWAdcCJQDufi8wBrjEzCqAHcD5URVHpEFpplKRvbINCnvcvcLMvgzc4e53mdk/a3uBu4/dz/GfEYasiuRVjx6hySjTfpHGJts+hU/MbCzwVWBGtK8kniyJNCzNVCqyV7ZB4WvAscCN7r7czHoBU+LLlkjD0UylIntl1Xzk7ouBywDM7GCgjbvfFGfGRBqSZioVCbIdfTTbzNqa2SHAAuABM7st3qyJiEhDy7b5qJ27bwZGAw+4+2DglPiyJVJ/U6dCWRk0aRL+6r4Dkf3LNig0M7OuwFfY29EsUrB0Q5rIgck2KFwPPA287e6vmtkRwFvxZUukfnRDmsiBybaj+ffA79OeLwPOiStTIvWlG9JEDky2Hc2Hmdkfo0Vz1prZY2Z2WNyZEzlQNd14phvSRGqXbfPRA4SpLboB3YH/jfaJFCTdkCZyYLINCh3d/QF3r4i23wAdY8yXSL3ohjSRA5Pt3EcbzGw8MC16PhbNaCoFTjekidRdtjWFiwjDUd8H1hBmOP1aXJkSEZH8yCoouPs77n6Wu3d0907u/iXCjWwiIpIg9Vl57Yqc5UJERApCfYKC5SwXIiJSEOoTFLRKmohIwtQ6+sjMtpD54m/AQbHkSERE8qbWoODubRoqIyIikn/1aT4SEZGEUVCQoqN1EkTik+0dzSIFoXKdhMppsSvXSQDdvSySC6opSFHROgki8VJQkKKidRJE4qWgILGqbP8/+eSTctL+r3USROKloCCxqbpOsuVknWStkyASLwUFiU1d2v+zHVGkdRJE4qXRRxKbbNv/6zqiSOskiMRHNQWJTbbt/xpRJFI4FBQkNtm2/2tEkUjhiC0omNn9ZrbOzN6o4biZ2Z1mttTMFppZeVx5kfyo2v7vNbb/a0SRSOGIs6bwG+C0Wo6PBI6OtknAPTHmRfJk3DhYsQKee+5vrFiRuS9AI4pECkdsQcHdnwc+qOWUs4EHPXgZaG9mXePKjxQujSgSKRzmHt9aOWZWBsxw974Zjs0AbnL3F6PnzwLfd/e5Gc6dRKhN0Llz58HTp0+vcnzr1q2UlpbmPP/5krTyQPLKlLTyQPLKlLTyQP3KNGLEiHnuPmR/5+VzSGqm5TwzRih3vw+4D2DIkCE+fPjwKsdnz55N9X3FLGnlgeSVKWnlgeSVKWnlgYYpUz5HH60CDk97fhiwOk95ERER8hsUngQuiEYhfR7Y5O5r8pgfEZFGL7bmIzObBgwHDjWzVcC1QAmAu98LzARGAUuB7cDX4sqLiIhkJ7ag4O5j93PcgUvjen8REak73dEsIiIpCgoiIpKioCB1lu001yJSfBQUJCWbi33VhXPIycI5IlI4FBQEyP5ir2muRZJNQUGA7C/2muZaJNkUFATI/mKvaa5Fkk1BIeGy7RTO9mKvaa5Fkk1BIcHq0imc7cVe01yLJJuCQoLVpVO4Lhf7yoVz9uyhxoVzRKQ45XPqbIlZXTuFx43TBV6ksVNNIcHUKSwidaWgkGDqFBaRulJQKFLZjCpSp7CI1JWCQgGpvNCffPJJtQ4frcuoInUKi0hdNIqgUJcJ3LI9N9dpVr3QW60Xek01ISJxSXxQyPir+usVTP3xStiwIeys7dwMF+a6/FKPY04hTTUhInFJ/JDUjBfbXc245mpn3NUdoUUL6N4dunfnmnmPsX17x6rnbodrLtvCuE2/C1d1d6750QVs394283nbpkHTpqFa0LQp11wxmu3bS/c992pn3DhL7avLhb5HjxBcMu0XEamPxAeFGi+21hNuux3eey+1vbO9Q+ZzP2gNl+5dOfQdvlXzed/8ZtV9jK8hXw7tD4bOnaFTJ3q0fIyVOzrtc16mC/2NN4baRnqw06giEcmFxAeFmn9VG3z3u1X3ldVw7mF7YO77YQiPGT3KnZWrajjv5VWhV3f3bti9mx7DdrNy9b6tdD3abYYLLoB162DdOm485BYmvXcd29k7hrQV27nxkJ/DLcCwYTBoEJSUpDqLr7kmBL0ePUJAUCeyiNRX4oNCXX5V13juTc3CL/rK826q5bzu3aumeXMN597dHsbdmdo3DmBqaFZ6513occg2buw3jXGrfwlXvbn3hccdB8OGMW7YMMYtGQoHHXQAn4qISGaJ72iu65w+2ZwbR5qV565YaTz33N9YsaGUcbO+Af/6F6xZA488AhddFGoW114Lw4dD+/YwcmTotd62LSefl4g0bomvKUDd5vTJ9tw40qxRly5w7rlhA/jgA/j732H2bHj0URg/HkpLYfRomDABRowInd0iInWU+JpCIh1yCJx5Jtx6KyxfHoLDeefB44/DqaeGTobvfQ8WLsx3TkWkyCgoFLsmTeCkk+BXv4L33w/NTIMHw+23w4ABYbvlFli9Ot85FZEioKCQJAcdFJqYnnwyBIG77oKWLeGqq+Dww+Gcc0KtIu2GPRGRdAoKSdWxI3z72/DKK/B//wdXXhkCwogR0L8//OIX6pwWkX0oKDQGn/40/M//wKpV8OtfQ7NmcPHFYfin5fIeAAAO40lEQVTsFVfA22/nO4ciUiAUFBqTgw4Kw1rnz4cXXwzDWe+6C44+Gk4/HZ56Ktx4JyKNloJCY2QGxx8P06aFW7h/9KMQKEaOhM98Bu6+e98Jo0SkUVBQaOy6dYPrrgvB4aGHoEOH0BdRVhZu8f7ww3znUEQaUKxBwcxOM7N/mdlSM5uc4fiFZrbezF6Ltolx5kdq0bw5jB0L//gHvPACDB0KP/xhuOfhqqs0pFWkkYgtKJhZU+BuYCTQGxhrZr0znPqwuw+Mtl/FlR+pgxNOgBkzYMECOOssuO026NULvvENePPNfOdORGIUZ01hKLDU3Ze5+8fAdODsGN9Pcq1//zCv0ltvwcSJMGVK6HM491yYNy/fuRORGJjHdCOTmY0BTnP3idHzCcDn3P3baedcCPwYWA+8CfyHu7+bIa1JwCSAzp07D54+fXqV41u3bqW0tLT6y4pWoZan5IMPOOwPf6D744/TbNs2Phg8mHfHjuXD8vLQeV2LQi3TgUpaeSB5ZUpaeaB+ZRoxYsQ8dx+y3xPdPZYNOBf4VdrzCcBd1c7pALSIHl8MPLe/dAcPHuzVzZo1a599xazgy7Npk/vNN7t36RLWohs0yH3aNPdPPqnxJQVfpjpKWnnck1empJXHvX5lAuZ6FtfuOJuPVgGHpz0/DKjSW+nuG919V/T0l8DgGPMjudK2beh8XrEizLm0fXvopD76aPjZzzScVaSIxRkUXgWONrNeZtYcOB94Mv0EM+ua9vQsYEmM+ZFca9ECvv51WLw4zNDatSt85zthxNJ//zds2JDvHIpIHcUWFNy9Avg28DThYv+Iuy8ys+vN7KzotMvMbJGZLQAuAy6MKz8SoyZN4Oyz4aWXwnDW444L9z706BGCxPLl+c6hiGQp1kV23H0mMLPavh+lPf4B8IM48yAN7IQTwrZ4cZiy+xe/gJ//nM8OHw5t2oRpvUWkYOmOZolH795w//2hlnDFFXR4+WUYMgROPhlmztT03SIFSkFB4tW9O/zkJ/zj4YfhJz8JN7+dfjr06wcPPAC7du0/DRFpMAoK0iB2l5aGNR2WLYMHHwxrSF90UbhT+qab4KOP8p1FEUFBQRpa8+YwYQK89ho8/TT06QM/+EFYGe4//iNMzCcieaOgIPlhBl/8Ivz1r/DPf4bRS3fdBUccAaNGwaOPqmlJJA8UFCT/Bg4M8yotWwaTJ8PChWF+pW7d4PLLQ61CRBqEgoIUjh49whoOK1fCn/8Mp5wC994LgwZBeXmoSWzcmO9ciiSagoIUnqZN4bTT4OGHwzoOd90V9l92Wag9fOUrIWjs3p3ffIokkIKCFLbKleDmzw99DxdfDM89F/odunaFSZPC2tIff5zvnIokgoKCFI+BA+GnP4X33gsd0V/4QlhneuRI6NQpjGr64x81IZ9IPSgoSPFp0QLOOScEhPXr4X//F7785XCn9OjR0LEjjBkT1pzevDnfuRUpKrHOfSQSu5Yt4YwzwvbJJ/D88/DYY6HG8Nhj4b6Ik08OtYrhw0Nto5n+2YvURP87JDlKSsLF/wtfCOs6vPxyCAwzZoR+BwhrQZx4YggQChIi+9D/BkmmJk3CFN7HHQe33gpr1sDf/gazZ4ftT38K57VpUzVIDBqkICGNmv71S+PQtSucf37YYG+QqAwUM6MZ3g86KNwTMXTo3q1Xr/2uQS2SFAoK0jhVDxLvvx8CxCuvwJw5cM89cPvt4ViHDnDMMXuDxDHH5C/fIjFTUBAB6NIFzjsvbBA6rRctCgHi1VfD3xtugD17APh8587w+c+HPomBA2HAACgrU41Cip6CgkgmJSV7L/iTJoV927aFm+jmzGHTzJm0fPPNMBw2ChS0axeCw4ABe1/bp08YQitSJBQURLLVunXolD7xRJYMHkzn4cPDjXJvvBEm7VuwIPy9//4QQCB0WvfoEaYGT9/S97VvrxqGFAwFBZH6aNVqb19DpT174O239waJZcvg3XfhhRfC3dgVFVXTaN16b4Do3j3z1qlTGFElEjMFBZFca9IEjj46bGPGVD22ezesXQvvvBMCRfq2ahUsWRJGRlWf7K9Zs9A5XhkkunUL/SDVt06dNKRW6kX/ekQaUtOm4YLerVvoqM6kMnC8917mbdEiePbZzEuYmsGhh+4NEh07hppIq1ZhuG2rVlUfp+0rXbo0LHLUsWPYL42SgoJIoUkPHLUNf925MwSP99/fu61ZU/X522/Djh2h72P79jCqqgZD0p+0bh2CQ8eOIchUPu7YEQ4+ONRGmjYNW5MmNT9u0iQEqsoNqj6v3Jo1C8GpdeuqW/PmOflIJXsKCiLFqmVL6NkzbNmqqAhBIj1Q7NgB27bx+vPP069LlzDJYPq2dm3oTF+/PgSihtSs2d6aTmWgaNUqlL1lyzCyq/Jxta3HqlXw0kvgHtJy37ulP4cQfKrXpjI9b948vGbPnr1b9eeVW5MmYRRbSUkoR6a/JSUF11ekoCDSmDRrFqb2aNNmn0Mb9+wJU33UxD0EkQ8/DE1cu3eHi19tjysvmpkuzOlbRUVIe9u2/W87d4Ztw4a9jyu3XbvC348/5oh4PsHcq6wp1bY1bQrNmnHYiBG1f0c5oKAgItkx2/trvdDt2cPzzzzDsJNOqrnpqvI5hEWaqteeKh+nP9+1a2+zWGXTWPrz9P179oTmuoqKqn8z7du9OzxO3zLs+/jgg2P/6BQURCR5mjRhT/Pm2d842KJF2Nq3jzdf9bRu9mx6x/wehdWYJSIieaWgICIiKQoKIiKSEmtQMLPTzOxfZrbUzCZnON7CzB6Ojr9iZmVx5kdERGoXW1Aws6bA3cBIoDcw1syq95F8HfjQ3Y8Cbgf+J678iIjI/sVZUxgKLHX3Ze7+MTAdOLvaOWcDv40ePwp8wUzTRYqI5It55U0luU7YbAxwmrtPjJ5PAD7n7t9OO+eN6JxV0fO3o3M2VEtrEjAJoHPnzoOnT59e5b22bt1KaWlpLOXIh6SVB5JXpqSVB5JXpqSVB+pXphEjRsxz9yH7Oy/O+xQy/eKvHoGyOQd3vw+4D2DIkCE+vNodfbNnz6b6vmKWtPJA8sqUtPJA8sqUtPJAw5QpzqCwCjg87flhwOoazlllZs2AdsAHtSU6b968DWa2struQ4ENmc4vUkkrDySvTEkrDySvTEkrD9SvTFlNkhVnUHgVONrMegHvAecD/17tnCeBrwL/AMYAz/l+2rPcvWP1fWY2N5tqUbFIWnkgeWVKWnkgeWVKWnmgYcoUW1Bw9woz+zbwNNAUuN/dF5nZ9cBcd38S+DXwOzNbSqghnB9XfkREZP9infvI3WcCM6vt+1Ha453AuXHmQUREspeUO5rvy3cGcixp5YHklSlp5YHklSlp5YEGKFNsQ1JFRKT4JKWmICIiOaCgICIiKUUdFPY34V4xMrMVZva6mb1mZnPznZ8DYWb3m9m66I71yn2HmNlfzeyt6G/8S0jlSA3luc7M3ou+p9fMbFQ+81gXZna4mc0ysyVmtsjMLo/2F/N3VFOZivJ7MrOWZjbHzBZE5fnvaH+vaPLQt6LJRJvn/L2LtU8hmnDvTeBUwk1wrwJj3X1xXjNWT2a2AhhSfaqPYmJmw4CtwIPu3jfadzPwgbvfFAXwg939+/nMZ7ZqKM91wFZ3vyWfeTsQZtYV6Oru882sDTAP+BJwIcX7HdVUpq9QhN9TNAdca3ffamYlwIvA5cAVwB/cfbqZ3QsscPd7cvnexVxTyGbCPckDd3+efe9MT5/88LeE/7BFoYbyFC13X+Pu86PHW4AlQHeK+zuqqUxFyYOt0dOSaHPgZMLkoRDTd1TMQaE78G7a81UU8T+CNA78xczmRRMBJkVnd18D4T8w0CnP+cmFb5vZwqh5qWiaWtJFa5gMAl4hId9RtTJBkX5PZtbUzF4D1gF/Bd4GPnL3iuiUWK55xRwUsppMrwgd7+7lhHUoLo2aLqTw3AMcCQwE1gC35jc7dWdmpcBjwHfdfXO+85MLGcpUtN+Tu+9294GEeeOGAp/NdFqu37eYg0I2E+4VHXdfHf1dB/yR8I8hCdZG7b6V7b/r8pyfenH3tdF/2j3ALymy7ylqp34MmOruf4h2F/V3lKlMxf49Abj7R8Bs4PNA+2jyUIjpmlfMQSE14V7UA38+YYK9omVmraNOMsysNfBF4I3aX1U0Kic/JPr7RB7zUm+VF8/Ilymi7ynqxPw1sMTdb0s7VLTfUU1lKtbvycw6mln76PFBwCmEfpJZhMlDIabvqGhHHwFEw8vuYO+EezfmOUv1YmZHEGoHEOaleqgYy2Rm04DhhGl+1wLXAo8DjwA9gHeAc929KDpvayjPcEKThAMrgG9WtscXOjM7AXgBeB3YE+2+mtAGX6zfUU1lGksRfk9m1p/QkdyU8OP9EXe/PrpGTAcOAf4JjHf3XTl972IOCiIiklvF3HwkIiI5pqAgIiIpCgoiIpKioCAiIikKCiIikqKgIBIxs91ps2m+lsuZd82sLH2WVZFCFesazSJFZkc0rYBIo6Wagsh+RGtc/E80v/0cMzsq2t/TzJ6NJlt71sx6RPs7m9kfo7nwF5jZcVFSTc3sl9H8+H+J7lTFzC4zs8VROtPzVEwRQEFBJN1B1ZqPzks7ttndhwI/I9xFT/T4QXfvD0wF7oz23wn8zd0HAOXAomj/0cDd7t4H+Ag4J9o/GRgUpXNxXIUTyYbuaBaJmNlWdy/NsH8FcLK7L4smXXvf3TuY2QbCwi6fRPvXuPuhZrYeOCx9+oFoOue/uvvR0fPvAyXufoOZPUVYxOdx4PG0efRFGpxqCiLZ8Roe13ROJulz1Oxmb5/e6cDdwGBgXtosmCINTkFBJDvnpf39R/T4JcLsvADjCEsmAjwLXAKphVLa1pSomTUBDnf3WcD3gPbAPrUVkYaiXyQiex0UrXRV6Sl3rxyW2sLMXiH8kBob7bsMuN/MrgLWA1+L9l8O3GdmXyfUCC4hLPCSSVNgipm1IywcdXs0f75IXqhPQWQ/oj6FIe6+Id95EYmbmo9ERCRFNQUREUlRTUFERFIUFEREJEVBQUREUhQUREQkRUFBRERS/j+HOtDOZ7AQPQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "acc= history.history['acc']\n",
    "val_acc= history.history['val_acc']\n",
    "loss= history.history['loss']\n",
    "val_loss= history.history['val_loss']\n",
    "\n",
    "epochs= range(1,len(acc)+1)\n",
    "\n",
    "plt.plot(epochs, acc, color='red', label='Training Accuracy')\n",
    "plt.plot(epochs, val_acc, 'bo', color='blue', label='Validation Accuracy')\n",
    "plt.grid()\n",
    "plt.title('Training vs Validation Accuracy')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend()\n",
    "plt.show()\n",
    "\n",
    "\n",
    "plt.plot(epochs, loss, color='red', label='Training Loss')\n",
    "plt.plot(epochs, val_loss, 'bo', color='blue', label='Validation Loss')\n",
    "plt.grid()\n",
    "plt.title('Training vs Validation Loss')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Loss')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model quickly starts overfitting, which is unsurprising given the small number of training samples. Validation accuracy has high variance for the same reason, but it seems to reach the high 50s.\n",
    "\n",
    "Note that our mileage may vary: because wou have so few training samples, performance is heavily dependent on exactly which 200 samples you choose—and we’re choosing them at random. If this works poorly, will try choosing a different random set of 200 samples.\n",
    "\n",
    "We can also train the same model without loading the pretrained word embeddings and without freezing the embedding layer. In that case, we’ll learn a task specific embedding of the input tokens, which is generally more powerful than pretrained word embeddings when lots of data is available. But in this case, we have only 200 training samples.\n",
    "\n",
    "### Training The Same Model Without Pretrained Word Embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_7 (Embedding)      (None, 100, 100)          1000000   \n",
      "_________________________________________________________________\n",
      "flatten_7 (Flatten)          (None, 10000)             0         \n",
      "_________________________________________________________________\n",
      "dense_12 (Dense)             (None, 32)                320032    \n",
      "_________________________________________________________________\n",
      "dense_13 (Dense)             (None, 1)                 33        \n",
      "=================================================================\n",
      "Total params: 1,320,065\n",
      "Trainable params: 1,320,065\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Flatten, Embedding\n",
    "\n",
    "model=Sequential()\n",
    "model.add(Embedding(max_words, embedding_dim, input_length=maxlen))\n",
    "model.add(Flatten())\n",
    "model.add(Dense(32, activation='relu'))\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 20000 samples, validate on 5000 samples\n",
      "Epoch 1/30\n",
      "20000/20000 [==============================] - 3s 136us/step - loss: 0.6939 - acc: 0.5050 - val_loss: 0.6941 - val_acc: 0.5006\n",
      "Epoch 2/30\n",
      "20000/20000 [==============================] - 2s 118us/step - loss: 0.4667 - acc: 0.7875 - val_loss: 0.9066 - val_acc: 0.5080\n",
      "Epoch 3/30\n",
      "20000/20000 [==============================] - 2s 118us/step - loss: 0.0808 - acc: 0.9763 - val_loss: 1.5239 - val_acc: 0.5172\n",
      "Epoch 4/30\n",
      "20000/20000 [==============================] - 2s 118us/step - loss: 0.0159 - acc: 0.9978 - val_loss: 1.9964 - val_acc: 0.5122\n",
      "Epoch 5/30\n",
      "20000/20000 [==============================] - 2s 118us/step - loss: 0.0088 - acc: 0.9982 - val_loss: 2.2511 - val_acc: 0.5074\n",
      "Epoch 6/30\n",
      "20000/20000 [==============================] - 2s 118us/step - loss: 0.0055 - acc: 0.9986 - val_loss: 2.9330 - val_acc: 0.5074\n",
      "Epoch 7/30\n",
      "20000/20000 [==============================] - 2s 119us/step - loss: 0.0043 - acc: 0.9987 - val_loss: 3.2700 - val_acc: 0.5098\n",
      "Epoch 8/30\n",
      "20000/20000 [==============================] - 2s 118us/step - loss: 0.0037 - acc: 0.9989 - val_loss: 3.3118 - val_acc: 0.5080\n",
      "Epoch 9/30\n",
      "20000/20000 [==============================] - 2s 118us/step - loss: 0.0042 - acc: 0.9990 - val_loss: 3.4650 - val_acc: 0.5088\n",
      "Epoch 10/30\n",
      "20000/20000 [==============================] - 2s 118us/step - loss: 0.0034 - acc: 0.9988 - val_loss: 3.4691 - val_acc: 0.5054\n",
      "Epoch 11/30\n",
      "20000/20000 [==============================] - 2s 118us/step - loss: 0.0032 - acc: 0.9990 - val_loss: 3.5429 - val_acc: 0.5082\n",
      "Epoch 12/30\n",
      "20000/20000 [==============================] - 2s 118us/step - loss: 0.0032 - acc: 0.9989 - val_loss: 3.5626 - val_acc: 0.5062\n",
      "Epoch 13/30\n",
      "20000/20000 [==============================] - 2s 119us/step - loss: 0.0030 - acc: 0.9990 - val_loss: 3.5942 - val_acc: 0.5062\n",
      "Epoch 14/30\n",
      "20000/20000 [==============================] - 2s 119us/step - loss: 0.0029 - acc: 0.9990 - val_loss: 3.6311 - val_acc: 0.5078\n",
      "Epoch 15/30\n",
      "20000/20000 [==============================] - 2s 119us/step - loss: 0.0029 - acc: 0.9990 - val_loss: 3.6367 - val_acc: 0.5078\n",
      "Epoch 16/30\n",
      "20000/20000 [==============================] - 2s 120us/step - loss: 0.0029 - acc: 0.9990 - val_loss: 3.6390 - val_acc: 0.5078\n",
      "Epoch 17/30\n",
      "20000/20000 [==============================] - 2s 120us/step - loss: 0.0029 - acc: 0.9990 - val_loss: 3.6407 - val_acc: 0.5076\n",
      "Epoch 18/30\n",
      "20000/20000 [==============================] - 2s 121us/step - loss: 0.0029 - acc: 0.9990 - val_loss: 3.6416 - val_acc: 0.5078\n",
      "Epoch 19/30\n",
      "20000/20000 [==============================] - 2s 123us/step - loss: 0.0029 - acc: 0.9990 - val_loss: 3.6421 - val_acc: 0.5080\n",
      "Epoch 20/30\n",
      "20000/20000 [==============================] - 2s 123us/step - loss: 0.0029 - acc: 0.9990 - val_loss: 3.6423 - val_acc: 0.5080\n",
      "Epoch 21/30\n",
      "20000/20000 [==============================] - 2s 123us/step - loss: 0.0029 - acc: 0.9990 - val_loss: 3.6423 - val_acc: 0.5080\n",
      "Epoch 22/30\n",
      "20000/20000 [==============================] - 3s 128us/step - loss: 0.0029 - acc: 0.9990 - val_loss: 3.6425 - val_acc: 0.5076\n",
      "Epoch 23/30\n",
      "20000/20000 [==============================] - 2s 123us/step - loss: 0.0029 - acc: 0.9990 - val_loss: 3.6427 - val_acc: 0.5076\n",
      "Epoch 24/30\n",
      "20000/20000 [==============================] - 2s 124us/step - loss: 0.0029 - acc: 0.9990 - val_loss: 3.6427 - val_acc: 0.5076\n",
      "Epoch 25/30\n",
      "20000/20000 [==============================] - 2s 124us/step - loss: 0.0029 - acc: 0.9990 - val_loss: 3.6427 - val_acc: 0.5078\n",
      "Epoch 26/30\n",
      "20000/20000 [==============================] - 2s 123us/step - loss: 0.0029 - acc: 0.9990 - val_loss: 3.6427 - val_acc: 0.5076\n",
      "Epoch 27/30\n",
      "20000/20000 [==============================] - 2s 123us/step - loss: 0.0029 - acc: 0.9990 - val_loss: 3.6427 - val_acc: 0.5078\n",
      "Epoch 28/30\n",
      "20000/20000 [==============================] - 2s 123us/step - loss: 0.0029 - acc: 0.9990 - val_loss: 3.6425 - val_acc: 0.5076\n",
      "Epoch 29/30\n",
      "20000/20000 [==============================] - 2s 124us/step - loss: 0.0029 - acc: 0.9990 - val_loss: 3.6425 - val_acc: 0.5076\n",
      "Epoch 30/30\n",
      "20000/20000 [==============================] - 2s 123us/step - loss: 0.0029 - acc: 0.9990 - val_loss: 3.6427 - val_acc: 0.5076\n"
     ]
    }
   ],
   "source": [
    "model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['acc'])\n",
    "\n",
    "history= model.fit(x_train, y_train, epochs=30, batch_size=32, validation_data=(x_val, y_val))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xt8VNW5//HPQ7jJRUHQqIAJWqwCBgwU66UKYjloq7YKKgXrpYjaWk9rsVprW4+/emp9aWutlkpb8QZBtBWph0orgkprFbCgArUGjMolgOEaQCTk+f2x94yTMJNMkplMZvJ9v177NbP3XrNmPTPJfmZf1trm7oiIiAC0yXQDRESk5VBSEBGRKCUFERGJUlIQEZEoJQUREYlSUhARkSglBamTmeWZWaWZHZ3KstnIzCaa2cLweZ2xxpZt5Hv91czGN/b1Io2lpJBjwg1VZKo2sz0x8w3eyLj7fnfv4u4fpLJsczOzTma2w8zOiLPu12Y2syH1pTJWM/upmT1Sq/5R7j69qXXX855uZsXpeg/JTkoKOSbcUHVx9y7AB8B5McsO2MiYWdvmb2Xzc/fdwFPA12OXm1k74FLg0Uy0KxPMzIDLgC3A5c383m3MTNudFkxfTisT/kJ80sxKzGwnMMHMTjGzf5rZNjPbYGb3hxtLzKxt+IuyMJx/Ilz/FzPbaWavmlnfhpYN159jZv8xs+3hr/W/m9kVcdrcx8x2m9khMcs+Z2abwvc8zsxeDuv5yMxmJAj/UWCsmR0Us+wcoAr4a1jvbWa2JmzvCjM7P8HnWDvWw8zsuXBv5J9A31rlHzCzteH6xWZ2arj8y8D3gfHh3tzScPmiyGcRbkh/bGbvhzE/YmYHh+s+E7bj62H9m83slgTxR4wAegLfAb4W+a5j2nqNmf07/AzeNrNB4fICM5sdvsdHZvarcHmNPZ1Im2LmF5nZ/zOzV4FdwNHh4bVV4XusNrOJtdpwoZktCz+vUjMbZWbjzOy1WuVuNrOn64lXGsLdNeXoBJQBZ9da9lPgE+A8gh8FBwGfA04G2gLHAP8Brg/LtwUcKAznnwA+AoYC7YAngScaUfZwYCdwQbjuRmAfcEWCWF4GroyZ/yXwQPj8KeDmMJ6OwGkJ6jBgDXBpzLKngHti5i8Gjgzr+hpQCeSH6yYCCxPE+jRQAnQCioANkbLh+suAQ8PX3QysAzrEfCeP1GrroshnAUwKv5O+QFfgWWBauO4zYTt+G8ZeDOwF+tXxd/EoMAPoAGwFzo9ZNw74EBgSfl7HAX3Cdr8N3AN0Dv9uTovX/kibasVSBpwQftdtCf7+jgnf4yxgD1AUlj8V2AaMDL+HPsBnw/fcFhsb8BZwQab/13Jp0p5C67TI3f/s7tXuvsfdF7v7a+5e5e5rgKnAmXW8/ml3X+Lu+4DpwOBGlP0ysMzdnw3X/ZIggSQyg2CDRXj44ZJwGQTJpBA40t0/dve/x6vAg63IY4SHkMysG8HG6dGYMrPcfUP42cwg2JgNraNdkUNQXwF+5O673f1N4PFa7/24u29x9yrgbuBggo1nMsYTJK733H0ncCvBL/zY/9/bw9jfAFYAgxK0tTNwETDD3fcCf6LmIaSJwF3uvtQD/3H3D4FTCPYubnb3XeHfTdzPOYGH3X2Vu+8L/87+7O5rwvd4EZgPfCEs+w3gd+4+P/wePnT3d9x9D0ESnxDGMpgggc9tQDukHkoKrdOHsTNmdryZ/Z+ZlZvZDuAOgg1AIuUxz3cDXRpR9qjYdoQb7LV11PMU8AUzyyc4/PGxu/8jXPc9gl+gS8zsLTOr6zj5Y8AXzewIgr2Cle7+VmSlmV1hZsvDQ2nbgOOp+7MAyAfyqPm5vh9bwMy+Hx6S2U7w67xzEvVGHFWrvveB9sBhkQXunux3Mgb4GJgXzk8Hvmxmh4bzfYDVcV7XByhz9/1Jtrm22n9zXzaz18xsS/g5j+LTzyNRGyBI4JELJiYAT4Y/KiRFlBRap9pD4z5EcGjgM+5+MPBjgt36dNoA9I7MmJkBvRIVdvcK4EVgLMFhnZKYdRvcfaK7Hwl8C5gae+6iVj1rgFfDOi4jSBKRNhwDTAGuA3q4ezfg39T/WWwEqgk2ZhHRS1XNbATB4bGLgG5Ad4LDUpF66xuqeD1QUKvuT4DN9bwunssJ9lI+NLNygs+xPcHJdgg23sfGed2HQIGZ5cVZt4vgsFnEEXHKxJ5jOIjgcNvPCA7NdSM4pxP5PBK1AXdfFNZxGsGe4+PxyknjKSkIBMeptwO7zOwE4JpmeM/ngGIzO8+CK6D+m5hfvgnMINioXcinh44ws4vNLJJQthFsgOr6Rfto+H4nx9ZD8OvaCTa2Fp78PL6+QMJfqrOB/zGzg8xsIEHCiehKcDL7I4I9mtsJ9hQiNgKFYWKMpwS40cwKzawrcCdQ4u7V9bUtlgV9KoYTnFwfHE6DgHv59BDS74Hvm9lJFuhnZn0IEmkF8L8WXN57ULhhBlgGnGnBBQHdgPpOdHcgSESbgf3hyfaRMev/AEw0sxHhSfbeZvbZmPWPEyTvXe7+z4Z8BlI/JQWB4PDL5QQnfh8iOCGcVu6+keC8wC8INjbHAv8iOEmayGygP/CBu6+IWX4ysNjMdhEcI/+W191/4CmCQxXz3H1TTJveBO4HXifYkzkeeC1uDQe6jmAPYCPBRm1azLq5wAvAuwTnKHaE9Uc8SbCR3GJmr8ep+3dhmVcITpTvJEhqDfV1YHF4rL48MgG/AoaY2fHuXgL8PHy/HQSfZ/fwXMiXCU4Wf0hwufOYsN7ngWcITvq+DsypqxHuvg34bviaLWE9z8Ws/wdwNcF3sR1YQM29sMeAgWgvIS0sOJQrklnhYYn1wBh3fyXT7ZGWKzxZvgkY6O7vZbo9uUZ7CpIxZjbazA4xsw7AjwgOscT7pSwS61vA35UQ0qNV9GaVFut0gqtf2hNcRvmV8DJJkbjMbC3BJcgXZLotuUqHj0REJEqHj0REJCrrDh/17NnTCwsLayzbtWsXnTt3jv+CLJRr8UDuxZRr8UDuxZRr8UDTYlq6dOlH7l7fZd/ZlxQKCwtZsmRJjWULFy5k+PDhmWlQGuRaPJB7MeVaPJB7MeVaPNC0mMzs/fpL6fCRiIjEUFIQEZEoJQUREYlSUhARkSglBRERiUpbUjCzh8NbB76dYL1ZcKvGUjN703QDcRGRjEvnnsIjwOg61p8D9AunSQRD4YqISAalrZ+Cu79s4U3NE7gAeCy849Y/zaybmR3p7hvqeE322r8f9uyBqqpg2rcv/vOqKrquWgXt2wevqaqK/xh57h5M8Olj7PPYx/37obo6mCLP4y2rrv603romADNo06bexz6lpbB4cXJ1RupN9Bh724HY+OLF3JDHBigsK4MXX2zw61qyXIsp1+IB6HrUUZDmvheZ7LzWi5q36FsbLjsgKZjZJIK9CfLz81m4cGGN9ZWVlQcsa0kOeest+v/P/9ChoiKp8kPS3J5MiHsbrRbEE97fJr4C6r9dWrbJtZhyLR6Adtddl/ZtXSaTQrz/wrjfobtPJbiZPEOHDvXaPfpadM/FqVPhe9+DwkK4+WZo1w7atg2myPNay95csYKi4uJgPi8vmBI9bxMeAaz9izrestjXtGnz6fPYZZEp8os80RTL/dO9i3iP1dW8smgRXzjjjPrrNav7F33s87r2IpJ9jHxUyX2bUS36b66Rci2mXIsHYEszxJTJpLCWmndT6k1wk5XcsG8ffOc78JvfwOjRUFIC3bol9dItXbumfRcxpcyCxFKH/Z06QZdE95IXkZYik5ekzgG+Hl6F9Hlge86cT9i8Gc4+O0gIN90Ezz2XdEIQEcmktO0pmFkJwU3Ce4Y3xvgJwU3LcfffEty39lygFNgNXJmutjSr5cvhggugvBwefxwmTMh0i0REkpbOq4/G1bPeCW6rlzueegquuAK6d4dFi2Do0Ey3SESkQdSjORWqq+FHP4KLL4ZBg4JLL5UQRCQLZd39FFqcHTvgsstgzhy46qrgPEKHDplulYhIoygpNMW6dTBqFLzzDtx/P1x//YGXa4qIZBElhaaYOhX+/W/4619h5MhMt0ZEpMl0TqEp3n0XCgqUEEQkZygpNMXq1fCZz2S6FSIiKaOk0BSlpUoKIpJTlBQaa8uWYFJSEJEcoqTQWKtXB49KCiKSQ5QUGqu0NHhUUhCRHKKk0FilpUGfhGOOyXRLRERSRkmhsUpLoXdv6Ngx0y0REUkZJYXGKi2FY1v6/cRERBpGSaGxdDmqiOQgJYXG2LkTNm1SUhCRnKOk0Bi6HFVEcpSSQmPoclQRyVFKCo0RSQo60SwiOUZJoTFKS+GII6BLl0y3REQkpZQUGkNXHolIjlJSaAwlBRHJUUoKDbV7d3AbTp1PEJEcpKTQUGvWBI/aUxCRHKSk0FC6HFVEcpiSQkPpclQRyWFKCg21ejX06AHdu2e6JSIiKaek0FC68khEcpiSQkMpKYhIDlNSaIi9e+GDD5QURCRnKSk0RFkZVFfrJLOI5Ky0JgUzG21m75hZqZndEmd9gZnNN7M3zWyhmfVOZ3uaTJejikiOS1tSMLM84EHgHKA/MM7M+tcqdg/wmLsXAXcAP0tXe1JCSUFEclw69xSGAaXuvsbdPwFmAhfUKtMfmB8+XxBnfctSWgoHHww9e2a6JSIiaZHOpNAL+DBmfm24LNZy4KLw+VeBrmbWI41taprIlUdmmW6JiEhatE1j3fG2nF5rfjLwgJldAbwMrAOqDqjIbBIwCSA/P5+FCxfWWF9ZWXnAsnQY9tZbVPbrx8o0v1dzxdOcci2mXIsHci+mXIsHmikmd0/LBJwCzIuZ/wHwgzrKdwHW1lfvkCFDvLYFCxYcsCzl9u1zb9vW/dZb0/5WzRJPM8u1mHItHvfciynX4nFvWkzAEk9i253Ow0eLgX5m1tfM2gOXAnNiC5hZTzOLtOEHwMNpbE/TfPABVFXpJLOI5LS0JQV3rwKuB+YBq4BZ7r7CzO4ws/PDYsOBd8zsP0A+cGe62tNkuvJIRFqBdJ5TwN3nAnNrLftxzPOngafT2YaU0eioItIKqEdzskpL4aCD4MgjM90SEZG0UVJIli5HFZFWQEkhWRodVURaASWFZOzfH9xcR0lBRHKckkIy1q2DTz5RUhCRnKekkAxdjioirYSSQjJWrw4elRREJMcpKSSjtBTat4detcfzExHJLUoKySgthWOOgby8TLdERCStlBSSoctRRaSVUFKoj7uSgoi0GkoK9Skvh927lRREpFVQUqiPLkcVkVZESaE+Sgoi0oooKdSntBTatoWCgky3REQk7ZQU6lNaCoWFQWIQEclxSgr1Wb1aN9YRkVZDSaEuuhxVRFoZJYW6VFTA9u1KCiLSaigp1EVXHolIK6OkUBclBRFpZZQU6lJaGtyTuW/fTLdERKRZKCnUpbQUjj4aOnTIdEtERJqFkkJddOWRiLQySgp1KS1VHwURaVWUFBLZti24JFV7CiLSiigpJKL7MotIK6SkkIguRxWRVkhJIZFIUjjmmMy2Q0SkGSkpJFJaCkcdBZ07Z7olIiLNRkkhEV2OKiKtUFqTgpmNNrN3zKzUzG6Js/5oM1tgZv8yszfN7Nx0tqdBlBREpBVKW1IwszzgQeAcoD8wzsz61yp2GzDL3U8CLgV+k672NEhlJZSXKymISKuTzj2FYUCpu69x90+AmcAFtco4cHD4/BBgfRrbk7zI5ajquCYirYy5e90FzK4Hprv71gZVbDYGGO3uE8P5y4CT3f36mDJHAn8FugOdgbPdfWmcuiYBkwDy8/OHzJw5s8b6yspKunTp0pDm1annSy8x8PbbWfLQQ1Qed1zK6k1WquNpCXItplyLB3IvplyLB5oW04gRI5a6+9B6C7p7nRPwU6AUmAWMJkwkSbxuLPD7mPnLgF/XKnMj8L3w+SnASqBNXfUOGTLEa1uwYMEBy5rk5z93B/dt21Jbb5JSHk8LkGsx5Vo87rkXU67F4960mIAlnsS2u97DR+5+G9AP+ANwBfCumf2vmdV3bGUt0CdmvjcHHh76RphscPdXgY5Az/ralHalpXDYYXDIIZluiYhIs0rqnEKYZcrDqYrgcM/TZnZ3HS9bDPQzs75m1p7gRPKcWmU+AEYCmNkJBElhc4MiSAddeSQirVTb+gqY2Q3A5cBHwO+Bm9x9n5m1Ad4Fvh/vde5eFZ6PmAfkAQ+7+wozu4NgN2YO8D3gd2b2XYKTzleECSizSkth+PBMt0KkTvv27WPt2rV8/PHHKanvkEMOYdWqVSmpqyXItXgguZg6duxI7969adeuXaPeo96kQHA450J3fz92obtXm9mX63qhu88F5tZa9uOY5yuB05JvbjOoqoK1a6GgINMtEanT2rVr6dq1K4WFhZhZk+vbuXMnXbt2TUHLWoZciwfqj8ndqaioYO3atfRt5B0jkzl8NBfYEpkxs65mdnLYgNxKwwCbN4M7HHlkplsiUqePP/6YHj16pCQhSG4wM3r06NGkvcdkksIUoDJmfle4LDeVlwePRxyR2XaIJEEJQWpr6t9EMknBYo/zu3s1yR12yk4bNwaPSgoiCVVUVDB48GAGDx7MEUccQa9evaLzn3zySVJ1XHnllbzzzjt1lnnwwQeZPn16KpoMwMaNG2nbti1/+MMfUlZnrklm474mPNkc2Tv4JrAmfU3KMO0piNSrR48eLFu2DIDbb7+dLl26MHny5Bplote9t4n/23PatGn1vs+3vvWtpjc2xpNPPskpp5xCSUkJ3/jGN1Jad6yqqirats3O387J7ClcC5wKrCPoe3AyYe/inBRJCvn5mW2HSBYqLS1l4MCBXHvttRQXF7NhwwYmTZrE0KFDGTBgAHfccUe07Omnn86yZcuoqqqiW7du3HLLLQwaNIhTTjmFTZs2AXDbbbdx3333RcvfcsstDBs2jM9+9rP84x//AGDXrl1cdNFFDBo0iHHjxjF06NBowqqtpKSE++67jzVr1lAe+V8H/u///o/i4mIGDRrEqFGjgOCk7uWXX86JJ55IUVERs2fPjrY1YubMmUycOBGACRMm8L3vfY8RI0Zw66238s9//pNTTjmFk046idNOO413330XCBLGd7/7XQYOHEhRURG/+c1vmDdvHmPHjo3W+5e//IWLL764yd9HY9Sbytx9E0Efg9ahvBy6dtV9FCS7fOc7kGBDmKyD9u+HvLxPFwweDOEGuSFWrlzJtGnT+O1vfwvAXXfdxaGHHkpVVRUjRoxgzJgx9O9fc2zM7du3c+aZZ3LXXXdx44038vDDD3PLLQcMrIy78/rrrzNnzhzuuOMOnn/+eX79619zxBFH8Mc//pHly5dTXFwct11lZWVs3bqVIUOGMGbMGGbNmsUNN9xAeXk51113Ha+88goFBQVs2RJcV3P77bdz2GGH8dZbb+HubNu2rd7YV69ezfz582nTpg3bt29n0aJF5OXl8fzzz3Pbbbfx5JNPMmXKFNavX8/y5cvJy8tjy5YtdOvWjRtuuIGKigp69OjBtGnTuPLKKxv60adEvXsKZtbRzL5lZr8xs4cjU3M0LiPKy3XoSKQJjj32WD73uc9F50tKSiguLqa4uJhVq1axcuXKA15z0EEHcc455wAwZMgQysrK4tZ94YUXHlBm0aJFXHpp8Lt10KBBDBgwIO5rS0pKuOSSSwC49NJLKSkpAeDVV19lxIgRFISXoR966KEAvPDCC9HDV2ZG9+7d64197Nix0cNl27Zt48ILL2TgwIFMnjyZFStWROu99tpryQsT8KGHHkqbNm342te+xowZM9iyZQtLly6N7rE0t2QOej0O/Bv4L+AOYDyQe5eiRigpSDZqxC/62vak6Lr+zjF72e+++y6/+tWveP311+nWrRsTJkyIe7lk+/bto8/z8vKoqqqKW3eHDh0OKJNsf9eSkhIqKip49NFHAVi/fj3vvfce7h73ip14y9u0aVPj/WrHEhv7D3/4Q/7rv/6Lb37zm5SWljJ69OiE9QJcddVVXHTRRQBccskl0aTR3JI5p/AZd/8RsMvdHwW+BJyY3mZlkJKCSMrs2LGDrl27cvDBB7NhwwbmzZuX8vc4/fTTmTVrFgBvvfVW3D2RlStXsn//ftatW0dZWRllZWXcdNNNzJw5k9NOO40XX3yR998P+udGDh+NGjWKBx54AAg25Fu3bqVNmzZ0796dd999l+rqap555pmE7dq+fTu9evUC4JFHHokuHzVqFFOmTGH//v013q9Pnz707NmTu+66iyuuuKJpH0oTJJMU9oWP28xsIMF9DwrT1qJMU1IQSZni4mL69+/PwIEDufrqqznttNQPYPDtb3+bdevWUVRUxL333svAgQM5pNZgljNmzOCrX/1qjWUXXXQRM2bMID8/nylTpnDBBRcwaNAgxo8fD8BPfvITNm7cyMCBAxk8eDCvvPIKAD//+c8ZPXo0I0eOpHfv3gnbdfPNN3PTTTcdEPM111zDEUccQVFREYMGDYomNICvfe1r9O3bl+MyMGR/VH3DqAITCQbAO4PgUtRNwDXJDMGajimtQ2fv3h0MmX3nnampr5E05G/L1xLiWblyZUrr27FjR0rray779u3zPXv2uLv7f/7zHy8sLPR9+/ZlZTzXXHONP/LIIwnXJxtTvL8Nkhw6u85zCuGgdzs8uMHOy8Axac1QmaaOayJZp7KykpEjR1JVVYW789BDD2VlH4HBgwfTvXt37r///oy2o85PzoNB764nvOdBzlPHNZGs061bN5YuPeCGjVknUd+K5pbMOYW/mdlkM+tjZodGprS3LBOUFESklUtmH+uq8DG2v7mTi4eSlBREpJVLpkdz4wblzkbl5WAW3IpTRKQVSubOa1+Pt9zdH0t9czKsvBx69oRG3rFIRCTbJXNO4XMx0xeA24Hz09imzFEfBZGkDB8+/ICOaPfddx/f/OY363xdly5dgKA38ZgxYxLWvWTJkjrrue+++9i9e3d0/txzz01qbKJkRQbXa43qTQru/u2Y6WrgJKB9fa/LSkoKksOmT4fCQmjTJnhsym0Kxo0bx8yZM2ssmzlzZtIb0qOOOoqnn3660e9fOynMnTu3xuilTbFq1Sqqq6t5+eWX2bVrV0rqjCfRUB6ZlsyeQm27gX6pbkiLoKQgOWr6dJg0Cd5/P7jb7PvvB/ONTQxjxozhueeeY+/evUAwAun69es5/fTTo/0GiouLOfHEE3n22WcPeH1ZWRkDBw4EYM+ePVx66aUUFRVxySWXsGfPnmi56667Ljrs9k9+8hMA7r//ftavX8+IESMYMWIEAIWFhXz00UcA/OIXv2DgwIGcfPLJ0WG3y8rKOOGEE7j66qsZMGAAo0aNqvE+sWbMmMFll13GqFGjmDNnTnR5aWkpZ599NoMGDaK4uJjVq1cDcPfdd3PiiScyaNCg6MiusXs7H330EYWFhUAw3MXYsWM577zzGDVqVJ2f1WOPPRbt9XzZZZexc+dOTjzxRPbtCwaZ2LFjB4WFhdH5lKmvdxvwZ2BOOD1H0Kv5rmR6xqVjSluP5upq9w4d3G+6qel1NVFL6C2barkWU0uIpyE9mgsKgs76taeCgk/LNLQH8LnnnuuzZ892d/ef/exnPnnyZHcPehhv377d3d03b97sxx57rFdXV7u7e+fOnd3d/b333vMBAwa4u/u9997rV155pbu7L1++3PPy8nzx4sXu7l5RUeHu7lVVVX7mmWf68uXLw3gKfPPmzTHxBfNLlizxgQMHemVlpa9fv9779+/vb7zxhr/33nuel5fn//rXv9zdfezYsf7444/Hjatfv35eVlbm8+bN8/POOy+6fNiwYf6nP/3J3d337Nnju3bt8rlz5/opp5ziu3btqtHeM888MxrD5s2bvSD8oKdNm+a9evWKlkv0Wb399tt+3HHHRWOMlB8/frw/88wz7u7+0EMP+Y033hg3hqb0aE5mT+Ee4N5w+hlwhrsfONB5ttu+Hfbu1Z6C5KQPPmjY8mTEHkKKPXTk7tx6660UFRVx9tlns27dOjZGRguI4+WXX2bChAkAFBUVUVRUFF03a9YsiouLOemkk1ixYkXcwe5iLVq0iK9+9at07tyZLl26cOGFF0bHLOrbty+DBw8GEg/PvXjxYg477DAKCgoYOXIkb7zxBlu3bmXnzp2sW7cuOn5Sx44d6dSpEy+88AJXXnklnTp1Aj4ddrsuX/ziF6PlEn1WL774ImPGjKFnz5416r388sujd6xL1z0XkkkKHwCvuftL7v53oMLMClPekkxTHwXJYUcf3bDlyfjKV77C/PnzeeONN9izZ0/05jbTp09n8+bNLF26lGXLlpGfnx93uOxY8YaSfu+997jnnnuYP38+b775Jl/60pfqrcfrGEY7Muw2JB6eu6SkhH//+98UFhZy7LHHsmPHDv74xz8mrNcTDIPdtm1bqqurgbqH1070WSWq9/Of/zxlZWW89NJL7N+/P3oILpWSSQpPAdUx8/vDZblFSUFy2J13QvhjNqpTp2B5Y3Xp0oXhw4dz1VVX1TjBvH37dg4//HDatWvHggULokNSJ3LGGWcwPTy58fbbb/Pmm28CwTHzzp07c8ghh7Bx40b+8pe/RF/TtWtXdu7cGbeu2bNns3v3bnbt2sUzzzzDF77whaTiqa6u5qmnnuLNN9+MDq/97LPPUlJSwsEHH0zv3r2ZPXs2AHv37mX37t2MGjWKhx9+OHrSOzIMdmFhYXTojbpOqCf6rEaOHMmsWbOoqKioUS/A17/+dcaNG5e2O7MlkxTauvsnkZnwee5dfaSkIDls/HiYOhUKCoL+mQUFwXw4SnSjjRs3juXLl0fvfBa813iWLFnC0KFDmT59Oscff3yddVx33XVUVlZSVFTE3XffzbBhw4DgstCTTjqJAQMGcNVVV9UYgnrSpEmcc8450RPNEcXFxVxxxRUMGzaMs846i4kTJ3LSSSclFcvLL79Mr169ovdAgCDJrFy5kg0bNvD4449z//33U1RUxKmnnkp5eTmjR4/m/PPPZ+jQoQwePJh77rkHgMmTJzNlyhROPfXU6AnweBJ9VgMGDOCHP/whZ555JoMGDeKd57juAAAQI0lEQVTGG2+s8ZqtW7em75LZ+k46AH8Dzo+ZvwCYn8wJi3RMaTvR/MtfBmfewhM6mdQSTmKmWq7F1BLi0dDZdcu1eNyDmJ566imfMGFCneXSNnR26Fpgupk9EM6vBeL2cs5q5eVBT+Yk7sMqIpIJkydPZv78+cydOzdt75HM2Eergc+bWRfA3P3AA3m5INJHIc7JHRGRluCee+5JyX2061LvOQUz+18z6+bule6+08y6m9lP09qqTFDHNRGRpE40n+Pu0UFFPLgL27npa1KGKClIFvI6LsGU1qmpfxPJJIU8M4te4GtmBwEd6iifnZQUJMt07NiRiooKJQaJcncqKiro2LFjo+tI5kTzE8B8M5sWzl8JPJpM5WY2GvgVkAf83t3vqrX+l0DkmrJOwOHunppRrRpi/37YvFlJQbJK7969Wbt2LZs3b05JfR9//HGTNiYtTa7FA8nF1LFjR3r37t3o90jmRPPdZvYmcDZgwPNAQX2vM7M84EHgiwRXLC02sznuHu2n7u7fjSn/bYIRWJvf5s1QXa2kIFmlXbt29O2buntgLVy4MOlr+rNBrsUDzRNTsqOklhP0ar4IGAmsSuI1w4BSd1/jQYe3mQR9HBIZB5Qk2Z7UUsc1ERGgjj0FMzsOuJRgY10BPElwSeqIRK+ppRfwYcz8WuDkBO9VAPQFXkywfhIwCSA/P5+FCxfWWF9ZWXnAsoY49PXXKQLeWL+eHU2oJ1WaGk9LlGsx5Vo8kHsx5Vo80EwxJerVRrBn8BLwmZhla5LpEReWHUtwHiEyfxnw6wRlb060rvaUlh7N06YFvZlXr25aPSnSEnrLplquxZRr8bjnXky5Fo9702IiBUNnX0Rw2GiBmf3OzEYSnFNI1lqgT8x8b2B9grKXkqlDR/Dp4aP8/Iw1QUSkJUiYFNz9GXe/BDgeWAh8F8g3sylmNiqJuhcD/cysr5m1J9jwz6ldyMw+C3QHXm1E+1OjvBy6doWYIW1FRFqjZO7RvMvdp7v7lwl+7S8D6r3JjrtXAdcD8whOTM9y9xVmdoeZnR9TdBwwM9y9yQz1URARAZLrpxDl7luAh8IpmfJzgbm1lv241vztDWlDWigpiIgAyV+SmtuUFEREACWFwMaNSgoiIigpwMcfw7ZtSgoiIigpBHsJoKQgIoKSgoa4EBGJoaSgpCAiEqWkoN7MIiJRSgqRpHD44Zlth4hIC6CkUF4OPXtCu3aZbomISMYpKajjmohIlJKCkoKISJSSgpKCiEhU604K7koKIiIxWndS2LEjGOZCSUFEBGjtSUEd10REalBSACUFEZGQkgIoKYiIhJQUQElBRCSkpNCuHXTvnumWiIi0CEoK+fnQpnV/DCIiEa17a6g+CiIiNSgpKCmIiEQpKSgpiIhEtd6ksH8/bNqkpCAiEqP1JoWPPoLqaiUFEZEYrTcpqI+CiMgBlBSUFEREopQUlBRERKKUFPLzM9sOEZEWpHUnhS5dgklERIA0JwUzG21m75hZqZndkqDMxWa20sxWmNmMdLanBvVREBE5QNt0VWxmecCDwBeBtcBiM5vj7itjyvQDfgCc5u5bzezwdLXnAEoKIiIHSOeewjCg1N3XuPsnwEzgglplrgYedPetAO6+KY3tqUlJQUTkAGnbUwB6AR/GzK8FTq5V5jgAM/s7kAfc7u7P167IzCYBkwDy8/NZuHBhjfWVlZUHLKvPaWvXsvGEEyht4OuaQ2PiaelyLaZciwdyL6ZciweaJ6Z0JgWLs8zjvH8/YDjQG3jFzAa6+7YaL3KfCkwFGDp0qA8fPrxGJQsXLqT2sjp9/DFUVtJ7yBB6N+R1zaTB8WSBXIsp1+KB3Isp1+KB5okpnYeP1gJ9YuZ7A+vjlHnW3fe5+3vAOwRJIr02bgwedfhIRKSGdCaFxUA/M+trZu2BS4E5tcrMBkYAmFlPgsNJa9LYpoA6romIxJW2pODuVcD1wDxgFTDL3VeY2R1mdn5YbB5QYWYrgQXATe5eka42RSkpiIjElc5zCrj7XGBurWU/jnnuwI3h1HyUFERE4mqdPZojSeHw5usWISKSDVpnUti4EXr2hHbtMt0SEZEWpXUmBXVcExGJS0lBRESilBRERCSq9SUFdyUFEZEEWl9S2LkT9uxRUhARiaP1JQX1URARSUhJQUREopQUREQkSklBRESiWmdSaNcOunfPdEtERFqc1pkU8vOhTesLXUSkPq1vy6g+CiIiCSkpiIhIlJKCiIhEta6ksH8/bNqkpCAikkDrSgoVFUFiUFIQEYmrdSUF9VEQEamTkoKIiEQpKYiISFTrTAr5+Zlth4hIC9X6kkLnztClS6ZbIiLSIrW+pKBDRyIiCSkpNNL06VBYGAyhVFgYzIuIZLu2mW5Asyovh/79m1zN9OkwaRLs3h3Mv/9+MA8wfnyTqxcRyRjtKTTCD3/4aUKI2L07WB6P9ipEJFu0nqSwdy9s3ZqSpPDBB8kvj+xVvP8+uH+6V9FciUEJSUQaovUkhY0bg8d6kkIyG9Gjj47/2njLG7JXEXnvs846s94NeDLtbGhCSjaBpCPRZDp5NTT2VH1HDSmX7jrri6mltFPfUZr/P9w9q6YhQ4Z4bQsWLDhg2QFee80d3P/854RFnnjCvVOnoFhk6tQpWN6Ycu7uZjXLRSazxteZbNmCgvjvXVDQPLFHyhcUuJtVe0FB0+I5sE5PWGdDy6U6dtWpOpu7zvoASzyJbWzGN/INnRqdFJ59Ngh38eKERRq6EU1mg5NsnQ1572TLJpuQ0tXObElemfyOVKfqTFWd9WkRSQEYDbwDlAK3xFl/BbAZWBZOE+urszFJ4Ykn3AsO3eHGfi/otS/hBrwhG9FkJbtxash7J1u2IX9MydaZjkST6eSVjthVp+ps7jrrk/GkAOQBq4FjgPbAcqB/rTJXAA80pN6GJoV0/GJsqGT2KtLxyyHTv5ZzLXllyy9G1dk666xPS0gKpwDzYuZ/APygVpm0J4V0HJpIh3QdY8zkcfVsSV65dmxZdbbOOuuTbFKwoGzqmdkYYLS7TwznLwNOdvfrY8pcAfwsPIT0H+C77v5hnLomAZMA8vPzh8ycObPG+srKSrokGM/orLPOxN3itM958cWXDlj+wguH8/vfH8OmTR04/PC9TJy4hrPP3pRc0E3UkPdORzuTrbMh5e6557Ps3ZsXXdahw34mT37ngPKprrMh752O2FWn6sxEnXUZMWLEUncfWm/BZDJHYyZgLPD7mPnLgF/XKtMD6BA+vxZ4sb5607mn0FIkdeI8SyRz9VHj60zN1UeNkUvfUUSuxZRr8bg3LSaS3FNIZz+FtUCfmPnewPpaCanC3feGs78DhqS6EXfeCZ061VzWqVOwXNJv/HgoK4MXX3yJsrLUDAMSqbO6mjrrTLaciHwqnUlhMdDPzPqaWXvgUmBObAEzOzJm9nxgVaobMX48TJ0KBQVgFjxOnaoNhIhIPGkbEM/dq8zsemAewZVID7v7CjO7g2A3Zg5wg5mdD1QBWwhOPKfc+PFKAiIiyUjrKKnuPheYW2vZj2Oe/4DgqiQREWkBWs/YRyIiUi8lBRERiVJSEBGRKCUFERGJSluP5nQxs83A+7UW9wQ+ykBz0iXX4oHciynX4oHciynX4oGmxVTg7ofVVyjrkkI8ZrbEk+m+nSVyLR7IvZhyLR7IvZhyLR5onph0+EhERKKUFEREJCpXksLUTDcgxXItHsi9mHItHsi9mHItHmiGmHLinIKIiKRGruwpiIhICigpiIhIVFYnBTMbbWbvmFmpmd2S6fakgpmVmdlbZrbMzJZkuj2NYWYPm9kmM3s7ZtmhZvY3M3s3fOyeyTY2RIJ4bjezdeH3tMzMzs1kGxvCzPqY2QIzW2VmK8zsv8Pl2fwdJYopK78nM+toZq+b2fIwnv8Jl/c1s9fC7+jJ8LYEqX3vbD2nYGZ5BLfw/CLBDX0WA+PcfWVGG9ZEZlYGDHX3rO10Y2ZnAJXAY+4+MFx2N7DF3e8KE3h3d785k+1MVoJ4bgcq3f2eTLatMcL7mBzp7m+YWVdgKfAVgqHrs/U7ShTTxWTh92RmBnR290ozawcsAv4buBH4k7vPNLPfAsvdfUoq3zub9xSGAaXuvsbdPwFmAhdkuE0CuPvLBPfHiHUB8Gj4/FGCf9iskCCerOXuG9z9jfD5ToKbW/Uiu7+jRDFlpfAOmpXhbLtwcuAs4OlweVq+o2xOCr2AD2Pm15LFfwQxHPirmS01s0mZbkwK5bv7Bgj+gYHDM9yeVLjezN4MDy9lzaGWWGZWCJwEvEaOfEe1YoIs/Z7MLM/MlgGbgL8Bq4Ft7l4VFknLNi+bk4LFWZadx8JqOs3di4FzgG+Fhy6k5ZkCHAsMBjYA92a2OQ1nZl2APwLfcfcdmW5PKsSJKWu/J3ff7+6DCe5vPww4IV6xVL9vNieFtUCfmPnewPoMtSVl3H19+LgJeIbgjyEXbIzckzt83JTh9jSJu28M/2mrgd+RZd9TeJz6j8B0d/9TuDirv6N4MWX79wTg7tuAhcDngW5mFrljZlq2edmcFBYD/cKz8e2BS4E5GW5Tk5hZ5/AkGWbWGRgFvF33q7LGHODy8PnlwLMZbEuTRTaeoa+SRd9TeBLzD8Aqd/9FzKqs/Y4SxZSt35OZHWZm3cLnBwFnE5wnWQCMCYul5TvK2quPAMLLy+4D8oCH3f3ODDepSczsGIK9Awjunz0jG2MysxJgOMEwvxuBnwCzgVnA0cAHwFh3z4qTtwniGU5wSMKBMuCayPH4ls7MTgdeAd4CqsPFtxIcg8/W7yhRTOPIwu/JzIoITiTnEfx4n+Xud4TbiJnAocC/gAnuvjel753NSUFERFIrmw8fiYhIiikpiIhIlJKCiIhEKSmIiEiUkoKIiEQpKYiEzGx/zGiay1I58q6ZFcaOsirSUrWtv4hIq7EnHFZApNXSnoJIPcJ7XPw8HN/+dTP7TLi8wMzmh4OtzTezo8Pl+Wb2TDgW/nIzOzWsKs/MfheOj//XsKcqZnaDma0M65mZoTBFACUFkVgH1Tp8dEnMuh3uPgx4gKAXPeHzx9y9CJgO3B8uvx94yd0HAcXAinB5P+BBdx8AbAMuCpffApwU1nNtuoITSYZ6NIuEzKzS3bvEWV4GnOXua8JB18rdvYeZfURwY5d94fIN7t7TzDYDvWOHHwiHc/6bu/cL528G2rn7T83seYKb+MwGZseMoy/S7LSnIJIcT/A8UZl4Yseo2c+n5/S+BDwIDAGWxoyCKdLslBREknNJzOOr4fN/EIzOCzCe4JaJAPOB6yB6o5SDE1VqZm2APu6+APg+0A04YG9FpLnoF4nIpw4K73QV8by7Ry5L7WBmrxH8kBoXLrsBeNjMbgI2A1eGy/8bmGpm3yDYI7iO4AYv8eQBT5jZIQQ3jvplOH6+SEbonIJIPcJzCkPd/aNMt0Uk3XT4SEREorSnICIiUdpTEBGRKCUFERGJUlIQEZEoJQUREYlSUhARkaj/D/4jqQhC91s/AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xt8VNW5//HPA0QChIuCpihCrFhbQIRAQatVsGrxbq1WabRaa1HUaqtYUX9tLdVzbF9qra3FYmuLgkSsN45arZfgpReVeAAFvCAFjaBcPALhJoHn98feGSfJJJlJZmcym+/79dqvzF6zZu31zMA8s/ZlbXN3REREADrkugMiItJ+KCmIiEiCkoKIiCQoKYiISIKSgoiIJCgpiIhIgpKCRMbMOppZtZn1z2bdfGRmF5jZ3PBxk7Em123htv5uZmUtfb3s2pQUJCH8oqpddprZlqT1jL9k3H2Huxe5+3vZrNvWzKyrmW0wsyNSPPdbMyvPpL1sxmpmN5jZX+q1f6y7z2xt2ym2NcPMrs92u9K+KClIQvhFVeTuRcB7wElJZQ2+ZMysU9v3su25+2bgAeA7yeVmVgCcBUzPRb9EoqCkIGkLf5Xeb2azzGwjcLaZHWpm/zazT8xslZndHn5ZYmadzMzNrCRcnxE+/zcz22hm/zKz/TKtGz5/nJm9bWbrw1/r/zCz81L0eV8z22xmPZPKvmxmq8NtfsHMXgjbWWtm9zUS/nTgDDPrklR2HFAD/D1s9/+Z2bKwv4vM7ORG3sf6se5pZo+Fo5F/A/vVq/87M6sKn3/VzL4Slp8I/BgoC0dzlWH5S7XvhZl1MLOfmtmKMOa/mFmP8LmBYT++E7a/xswmNxJ/k8zscDObF76Pr5jZ6KTnvmdmy8P3ZZmZnRWWp/veSxtSUpBMfQO4D+gJ3E/wpXg50Ac4DBgHXNjE678N/ATYg2A08otM65rZXsBs4Kpwu/8BRqVqwN3fB+YBp9Vrd7a71wA3Ao8DuwP9gDsa6cuLwDrglKSyc4CZ7r4jXH+b4D3oGbZ7n5kVNxFfranARuBzwATg/HrPvwwMJXgf/go8YGad3f0x4FdhH4rcfUSKti8AzgbGAPuHcf6mXp2vAAOBrwM/N7MD0uhzgpn1IXgPbwF6A7cDT5jZ7mECuhU4xt27E7w/C8OXpvveSxtSUpBMveTu/+PuO919i7u/6u4vu3uNuy8DpgFHNvH6v7r7PHffDswEhrWg7onAfHd/NHzu18DaJtq5DxgPwS9n4MywDGA7UAL0dfet7v6PVA14MEnYPYS7kMysF3ASSbuO3H22u68K35v7gOXAyCb6VbsL6lTgJ+6+2d0XAvfW2/a97v5xmMR+BfQg+BJPRxlws7v/x903AtcC3w7fh1rXh7G/BiwCDk6z7VonAYvcfVb472AGsAw4oTYEYIiZFYbvz+KwPK33XtqWkoJk6v3kFTP7opk9bmYfmtkGYArBr/fGfJj0eDNQ1IK6eyf3I/zCrmqinQeAr4a/2scCW939n+FzVwIFwDwze93Mzm2inXuAY8zsc8C3gMXu/nrtk2Z2npktCHelfQJ8kabfC4BioCN139cVyRXM7Mdm9qaZrQf+D+iWRru19q7X3gpgN2DP2gJ3z+QzSWcbtdvZx903ECTkS4APw91kXwjrZPLeSxtRUpBM1Z9W9w/AG8BAd+8B/BSwiPuwimB3AwBmZsA+jVV293XAc8AZBLuOZiU9t8rdL3D3vgRfXNOSj13Ua2cZ8K+wjXMIkkRtHz5PsBtoItDb3XsBb9L8e/ERsBPYN6kscaqqmY0FrgC+CfQi2NVSndRuc9McrwQG1Gv7U2BNM6/LRP1t1G7nAwB3/5u7Hw30BZYS/JvJ6L2XtqOkIK3VHVgPbDKzL9H08YRseQwoNbOTLDgD6nKSfvk24j7gXIJjC4kDmmb2LTOrTSifEHzJ7mj48oTp4fZGJ7dD8OvaCb5szcwuIBgpNCnc/fUIwb78LmY2hCDh1OpOcNxmLcGv6usJRgq1PgJKwsSYyizgCjMrMbPuBPvxZ7n7zub61ohOZlaYtOxG8HkMNrMzw4Po3ybYvfWEmfUNP6euBMloE+H724L3XtqAkoK01pUEX7YbCX4B3h/1Bt39I4LjArcSHPzdH/hfYFsTL3sEGAS85+6LkspHA6+a2SbgIeCSZq4feIBg181T7r46qU8LCQ6wvkIwkvkiwQHidEwkGAF8BPwJ+HPSc08AzwDvEByj2BC2X+t+gt1BH5vZKynavius8yLBfv6NBEmtpa4DtiQtf3f3NcDJwNUEn8ePgBPd/WOCXWNXhX1eR3BQ+9KwrUzfe2kDppvsSL4zs44EuzBOd/cXc90fkXymkYLkJTMbZ2Y9zawzwWmrNQS/0kWkFZQUJF8dTrA7ZC3BtRGnuntTu49EJA3afSQiIgkaKYiISELeTWjWp08fLykpqVO2adMmunXrlvoFeShu8UD8YopbPBC/mOIWD7QupsrKyrXu3typ2/mXFEpKSpg3b16dsrlz5zJmzJjcdCgCcYsH4hdT3OKB+MUUt3igdTGZWf2rzlPS7iMREUlQUhARkQQlBRERSVBSEBGRBCUFERFJUFIQicjMmVBSAh06BH9nNrjLdWb1om7zqKOOzIt+pttmc/G0l35m8zPKCnfPq2XEiBFeX0VFRYOyfBa3eNzbf0wzZrgPGOBuFvydMaO5ejubrde1qzt8tnTt2rB+uvXUptpMp25TgHmexndszr/kM12UFPJTe44piv/IAwbUrVe7DBjQsnpqU22mU7cpSgp5LG7xuOcmpnR//UfxH9ksdV2zltVTm2oznbpNSTcp6JiCxNLMmTBhAqxYEfwXWrEiWE+1L/a9Rm7rUr883XoA/fs3LEtVnm49tak20y1vtXQyR3taNFLIT9mMKZ0RQK6H/HHbX60223+bzUG7j/JX3OJxbz6mTA70pvMfJJMhdxT/kTONKZ160bfZ/MHz9tHP7JwM0H76mb3PqClKCnksbvG4Nx1Trg/g1vYh2184+SZu/+7iFo9762JKNylEdkzBzArN7BUzW2Bmi8zs5ynqnGdma8xsfrhcEFV/pP267jrYvLlu2ebNQXl96e7Xv/FG6Nq1blnXrkF5KmVlsHw57NwZ/C0ra7rec88932Q9kXwV5YHmbcBR7n4wMAwYZ2aHpKh3v7sPC5c/RtgfaaeiOIBbVgbTpsGAAWAW/J02TV/iIs2JLCmEI5bqcLUgXDyq7UnbyfaVmJmcXZHJCCDdX/8i8plI79FsZh2BSmAgcIe7X13v+fOA/wbWAG8DP3L391O0MwGYAFBcXDyivLy8zvPV1dUUFRVFEUJOtOd4nnlmL26++UC2beuYKOvceQeTJr3F0UevblHdTNqsrf/HP36e1as7s9de27jggmUp60WpPX9GLRW3mOIWD7QuprFjx1a6+8hmK6Zz4KG1C9ALqACG1CvvDXQOH18EPNdcWzrQnFtRXYmZyZkY7UF7/oxaKm4xxS0e97Y50Nwmt+N090/MbC4wDngjqXxdUrW7gF+2RX+k5TLZ/59J3bIy7d4RaQ+iPPtoTzPrFT7uAhwNvFmvTt+k1ZOBJVH1R7KjXV+JKSKtFuXZR32BCjNbCLwKPO3uj5nZFDM7OaxzWXi66gLgMuC8CPsjWZDJgd5MTwsVkdyLbPeRuy8Ehqco/2nS42uAa6Lqg2Rf7S6e664LdgP17x98yafa9VO3rtO/vzVaV0TahzY5piDxksn+/9q6c+c+z5gxYyLtl4i0nmZJlYRMrj8QkXjSSEGAz6aarp1uonaqadDuHpFdiUYKMZfur/9M5h8SkfjSSCHGMvn1n8k1BSISXxopxFgmv/51TYGIgJJCrGXy61/XFIgIKCnEWia//jXVtIiAkkKsRXWjGRGJLyWFGNOvfxHJlM4+ijnNPioimdBIQUREEpQUREQkQUlBREQSlBRERCRBSUFERBKUFEREJEFJQUREEiJLCmZWaGavmNmC8D7MP09Rp7OZ3W9mS83sZTMriao/IiLSvChHCtuAo9z9YGAYMM7MDqlX53vA/7n7QODXwC8j7I+IiDQjsqTggepwtSBcvF61U4Dp4eO/Al8zM4uqTyIi0jRzr/89ncXGzToClcBA4A53v7re828A49y9Klx/Fxjt7mvr1ZsATAAoLi4eUV5eXmc71dXVFBUVRRZHW4tbPBC/mOIWD8QvprjFA62LaezYsZXuPrLZiu4e+QL0AiqAIfXKFwH9ktbfBXo31daIESO8voqKigZl+SydeGbMcB8wwN0s+DtjRtS9ap1d8TPKN3GLKW7xuLcuJmCep/F93SZnH7n7J8BcYFy9p6qAfQHMrBPQE/i4LfqUz2pvs7liBbh/dpvNxu6/LCKSrijPPtrTzHqFj7sARwNv1qs2Bzg3fHw68FyY0aQJmdxmU0QkE1FOnd0XmB4eV+gAzHb3x8xsCsEwZg7wJ+BeM1tKMEI4K8L+xEYmt9kUEclEZEnB3RcCw1OU/zTp8VbgjKj6EFf9+we7jFKVi4i0hq5ozkOZ3mZTRCRdSgp5SLfZFJGo6HaceUq32RSRKGikICIiCUoKIiKSoKQgIiIJSgoiIpKgpCAiIglKCiIikqCkICIiCUoKIiKSoKQgIiIJSgoiIpKgpCAiIglKCiIikqCkICIiCUoKIiKSoKQgIiIJkSUFM9vXzCrMbImZLTKzy1PUGWNm681sfrj8NFVbIiLSNqK8yU4NcKW7v2Zm3YFKM3va3RfXq/eiu58YYT9ERCRNkY0U3H2Vu78WPt4ILAH2iWp7IiLSeubu0W/ErAR4ARji7huSyscADwJVwEpgkrsvSvH6CcAEgOLi4hHl5eV1nq+urqaoqCii3re9uMUD8YspbvFA/GKKWzzQupjGjh1b6e4jm63o7pEuQBFQCZyW4rkeQFH4+HjgnebaGzFihNdXUVHRoCyfxS0e9/jFFLd43OMXU9zicW9dTMA8T+M7O9Kzj8ysgGAkMNPdH0qRkDa4e3X4+AmgwMz6RNknERFpXJRnHxnwJ2CJu9/aSJ3PhfUws1Fhf9ZF1ScREWlalGcfHQacA7xuZvPDsmuB/gDufidwOjDRzGqALcBZ4TBHRERyILKk4O4vAdZMnd8Bv4uqDyIikhld0SwiIglKCiIikqCkICIiCUoKIiKSoKQgIiIJSgrtyMyZUFICRx11JCUlwbqISFuK8joFycDMmTBhAmzeDGCsWBGsA5SV5bJnIrIr0UihnbjuutqE8JnNm4NyEZG2oqTQTrz3XmblIiJRUFJoJ/r3z6xcRCQKSgrtxI03Qteudcu6dg3KRUTaipJCO1FWBtOmwYABYOYMGBCs6yCziLQlJYV2pKwMli+H5557nuXLlRBEpO0pKYiISIKSgoiIJOjiNRFJafv27VRVVbF169Zcd6VFevbsyZIlS3LdjaxKJ6bCwkL69etHQUFBi7ahpCAiKVVVVdG9e3dKSkoI75qbVzZu3Ej37t1z3Y2sai4md2fdunVUVVWx3377tWgb2n0kIilt3bqV3r1752VC2FWZGb17927V6C6ypGBm+5pZhZktMbNFZnZ5ijpmZreb2VIzW2hmpVH1R0Qyp4SQf1r7mUU5UqgBrnT3LwGHAJeY2aB6dY4DDgiXCcDUCPuTM7Wzn3bogGY/FUnDunXrGDZsGMOGDeNzn/sc++yzT2L9008/TauNiRMn8tZbbzVZ54477mBmlv5DHn744cyfPz8rbeVSZMcU3H0VsCp8vNHMlgD7AIuTqp0C3OPuDvzbzHqZWd/wtbFQd/ZTNPupSBp69+6d+IK9/vrrKSoqYtKkSXXquDvuTocOqX/bTp06tdljCpdcckl2OhwjbXKg2cxKgOHAy/We2gd4P2m9KiyrkxTMbALBSILi4mLmzp1bp5Hq6uoGZe3FlVcewubNhXXKNm+GK6/cyj77/Dvla9pzPC0Vt5jiFg80jKlnz55s3Lgxdx0Kbdu2jYKCAjZu3Mi7777Lt7/9bQ499FDmzZvH7Nmzuemmm1iwYAFbtmzhtNNOY/LkyQAcc8wx3HLLLQwaNIj99tuP888/n6effpouXbpQXl7OnnvuyZQpU+jduzeXXHIJxx57LIceeijPP/88GzZsYOrUqYwePZpNmzZx4YUXsmzZMg488ECWLVvGb3/7W4YOHVqnnzt27GDTpk113rMtW7bwwx/+kAULFlBQUMBNN93EYYcdxqJFi7j44oupqalh586d3HffffTp04dzzz2XDz/8kB07dnDNNddw6qmnNthGOp/J1q1bW/zvM/KkYGZFwIPAD919Q/2nU7zEGxS4TwOmAYwcOdLHjBlT5/m5c+dSv6y9WL26sfLCRvvcnuNpqbjFFLd4oGFMS5Ys+eyX9g9/CNneNTJsGNx2W7PVOnfuTOfOnenevTtFRUW8+eabTJ8+nS9/+csA3HLLLeyxxx7U1NQwduxYysrKGDRoEGZGt27d6N69O+vXr+eYY47h1ltv5YorrmD27NlMnjyZzp07U1hYSPfu3enYsSMFBQVUVlYyZ84cbr75Zp588knuuOMO+vXrx6OPPsqCBQsoLS1NtJusY8eODcp///vf061bNxYtWsSiRYs4/vjjeeedd5g+fTpXX301Z555Jtu2bcPdefTRRxk4cCBPP/00AOvXr2+wjXTPqCosLGT48OHN1kslrWMKZra/mXUOH48xs8vMrFcarysgSAgz3f2hFFWqgH2T1vsBK9PpU77Q7Kci2bX//vsnEgLArFmzKC0tpbS0lCVLlrB48eIGr+nSpQvHHXccACNGjGD58uUp2z7ttNMa1HnppZc466yzADj44IMZPHhw2n196aWXOOeccwAYPHgwe++9N0uXLuUrX/kKN9xwA7/61a94//33KSwsZOjQoTz55JNMnjyZf/zjH/Ts2TPt7WRTuiOFB4GRZjYQ+BMwB7gPOL6xF1hwCPxPwBJ3v7WRanOAS82sHBgNrI/T8QQIZjlNPqYAmv1U8lAav+jbSrdu3RKP33nnHX7zm9/wyiuv0KtXL84+++yUp2PutttuiccdO3akpqYmZdudO3duUCc45Nkyjb32nHPO4dBDD+Xxxx/nmGOOYfr06RxxxBHMmzePJ554gquuuooTTzyRa6+9tsXbbql0zz7a6e41wDeA29z9R0DfZl5zGHAOcJSZzQ+X483sIjO7KKzzBLAMWArcBVyceQjtW93ZT9HspyJZtGHDBrp3706PHj1YtWoVTz31VNa3cfjhhzN79mwAXn/99ZQjkcYcccQRibOblixZwqpVqxg4cCDLli1j4MCBXH755ZxwwgksXLiQDz74gKKiIs455xyuuOIKXnvttazHko50RwrbzWw8cC5wUljW5DXU7v4SqY8ZJNdxIPaH/8vKlAREolBaWsqgQYMYMmQIn//85znssMOyvo0f/OAHfOc732Ho0KGUlpYyZMiQRnftfP3rX09ML/HVr36Vu+++mwsvvJCDDjqIgoIC7rnnHnbbbTfuu+8+Zs2aRUFBAXvvvTc33HAD//znP5k8eTIdOnRgt912484778x6LGmpPa2rqQUYBNwOjA/X9wMmp/PabC8jRozw+ioqKhqU5bO4xeMev5jiFo97w5gWL16cm45kyYYNG7LSzvbt233Lli3u7v722297SUmJb9++PSttZyrdmFJ9dsA8T+M7Nq2RgrsvBi4DMLPdge7uflMUSUpEpD2prq7ma1/7GjU1Nbg7f/jDH+jUKb7TxqUVmZnNBU4O688H1pjZ8+5+RYR9ExHJuV69elFZWZnrbrSZdA809/TgGoPTgD+7+wjg6Oi6JSIiuZBuUuhkZn2BbwGPRdgfERHJoXSTwhTgKeBdd3/VzD4PvBNdt0REJBfSPdD8APBA0voy4JtRdUpERHIj3Wku+pnZw2a22sw+MrMHzaxf1J0TkV3TmDFjGlyIdtttt3HxxU1f31pUVATAypUrE9NLpGp73rx5TbZz2223sTlpGoLjjz+eTz75JJ2uN+n666/n5ptvbnU7UUp399GfCaak2JtgFtP/CctERIDs3jdk/PjxlJeX1ykrLy9n/Pjxab1+77335t57723x9usnhSeeeIJevZqd7i0W0k0Ke7r7n929Jlz+AuwZYb9EJI/U3jdkxQpw/+y+IS1NDKeffjqPPfYY27ZtA2D58uWsXLmSww8/PHHdQGlpKQcddBCPPvpog9cvX76c0aNHA8H01WeddRZDhw7lzDPPZMuWLYl6EydOZOTIkQwePJif/exnANx+++2sXLmSsWPHMnbsWABKSkpYu3YtALfeeitDhgxhyJAh3BbOCbV8+XK+9KUv8f3vf5/Bgwdz7LHH1tlOc1K1uWnTJk444QQOPvhghgwZwv333w/A5MmTGTRoEEOHDm1wj4lsSPcKjLVmdjYwK1wfD6zLem9EJC9dd13dSR8hWL/uupZN8dK7d29GjRrFk08+ySmnnEJ5eTlnnnkmZkZhYSEPP/wwPXr0YO3atRxyyCGcfPLJjd6GcurUqXTt2pWFCxeycOFCSks/u+vvjTfeyB577MGOHTv42te+xsKFC7nsssu49dZbqaiooE+fPnXaqqys5M9//jMvv/wy7s7o0aM58sgj2X333XnnnXeYNWsWd911F9/61rd48MEHOfvss5uNtbE2ly1bxt57783jjz8OBFNpf/zxxzz88MO8+eabmFlWdmnVl+5I4XyC01E/JLgBzunAd7PeGxHJS++9l1l5OpJ3ISXvOnJ3rr32WoYOHcrRRx/NBx98wEcffdRoOy+88ELiy3no0KF1bo4ze/ZsSktLGT58OIsWLWp2sruXXnqJb3zjG3Tr1o2ioiJOO+00XnzxRQD2228/hg0bBjQ9PXe6bR500EE888wzXH311bz44ov07NmTHj16UFhYyAUXXMBDDz1E165d09pGJtJKCu7+nruf7O57uvte7n4qwYVsIiKR3Dfk1FNP5dlnn+W1115jy5YtiV/4M2fOZM2aNVRWVjJ//nyKi4tTTpedLNUo4j//+Q8333wzzz77LAsXLuSEE05oth1vYhrt2mm3oenpudNt8wtf+AKVlZUcdNBBXHPNNUyZMoVOnTrxyiuv8M1vfpNHHnmEcePGpbWNTKQ7UkhFU1yICBDcH6T+j9bW3jekqKiIMWPGcP7559c5wLx+/Xr22msvCgoKqKioYMWKFU22kzx99RtvvMHChQuBYNrtbt260bNnTz766CP+9re/JV7TvXv3lLe9POKII3jkkUfYvHkzmzZt4uGHH+arX/1qy4Nsos2VK1fStWtXzj77bCZNmsRrr71GdXU169ev5/jjj+e2225L3Mc6m1ozq1OT02KLyK6j9rjBddcFu4z69w8SQmunjB8/fjynnXZanTORysrKOOmkkxg5ciTDhg3ji1/8YpNtTJw4ke9+97sMHTqUYcOGMWrUKCC4i9rw4cMZPHhwg2m3J0yYwHHHHUffvn2pqKhIlJeWlnLeeecl2rjgggsYPnx42ruKAG644YbEwWSAqqqqlG0+9dRTXHXVVXTo0IGCggKmTp1KdXU1ZWVlbN26FXfn17/+ddrbTVs6U6mmWoD3Wvra1iyaOjs/xS2muMXjrqmz80HOp842s41Aqh1eBnTJfooSEZFcavKYgrt3d/ceKZbu7h7fCcXTkM0LdURE2ovWHGhukpndHU6L8UYjz48xs/VJ92/+aVR9ybZsX6gjItJeRJYUgL8AzZ0v9aK7DwuXKRH2JauaulBHJE68iVMwpX1q7WcWWVJw9xeAj6NqP5eiuFBHpL0pLCxk3bp1Sgx5xN1Zt24dhYWFLW4j18cFDjWzBcBKYJK7L8pxf9LSv3+wyyhVuUhc9OvXj6qqKtasWZPrrrTI1q1bW/Xl2B6lE1NhYSH9+rV8EmuL8leAmZUAj7n7kBTP9QB2unu1mR0P/MbdD2iknQnABIDi4uIR9WdPrK6uTkyZ2xaeeWYvbr75QLZt65go69x5B5MmvcXRR69udfttHU9biFtMcYsH4hdT3OKB1sU0duzYSncf2WzFdM5bbekClABvpFl3OdCnuXrt5TqFGTPcBwxwNwv+zpiRvbZ3hXPg813c4nGPX0xxi8e9dTGRjesUomRmnwM+cnc3s1EExzfyZubVsrLWX60pItLeRJYUzGwWMAboY2ZVwM+AAgB3v5NgptWJZlYDbAHOCrOZiIjkSGRJwd2bvEWSu/8O+F1U2xcRkcxFeZ2CiIjkGSUFERFJUFIQEZEEJQUREUlQUhARkQQlBRERSVBSEBGRBCUFERFJUFIQEZEEJQUREUlQUhARkQQlBRERSVBSEBGRBCUFERFJUFIQEZEEJQUREUlQUhARkQQlBRERSVBSEBGRhMiSgpndbWarzeyNRp43M7vdzJaa2UIzK42qLyIikp4oRwp/AcY18fxxwAHhMgGYGmFfREQkDZElBXd/Afi4iSqnAPd44N9ALzPrG1V/RESkeebu0TVuVgI85u5DUjz3GHCTu78Urj8LXO3u81LUnUAwmqC4uHhEeXl5neerq6spKirKev9zJW7xQPxiils8EL+Y4hYPtC6msWPHVrr7yObqdWpR69lhKcpSZih3nwZMAxg5cqSPGTOmzvNz586lflk+i1s8EL+Y4hYPxC+muMUDbRNTLs8+qgL2TVrvB6zMUV8SZs6EkhLo0CH4O3NmrnskItJ2cpkU5gDfCc9COgRY7+6rctgfZs6ECRNgxQpwD/5OmKDEICK7jihPSZ0F/As40MyqzOx7ZnaRmV0UVnkCWAYsBe4CLo6qL+m67jrYvLlu2ebNQbmIyK4gsmMK7j6+mecduCSq7bfEe+9lVi4iEje6ojlJ//6ZlYuIxI2SQpIbb4SuXeuWde0alIuI7AqUFJKUlcG0aTBgAJgFf6dNC8pFRHYFubxOoV0qK1MSEJFdl0YKIiKSoKQgIiIJSgoiIpKwSyQFTV0hIpKe2B9orp26ovZK5dqpK0AHlEVE6ov9SEFTV4iIpC/2SUFTV4iIpC/2SUFTV4iIpC/2SSHl1BVs4sZz38pNh0RE2rHYJ4UGU1f028G04p9Q9vvD4d13c909EZF2JfZw3F+PAAALoklEQVRJAYLEsHw57NwJy9/vSNmLE4OVE0+ETz7JdfdERNqNXSIpNHDAAfDQQ8FI4fTTYfv2XPdIRKRd2DWTAsCRR8Jdd8Gzz8LFFwf33xQR2cXF/uK1Jp17Lrz9NvzXf8GBB8KkSbnukYhITkU6UjCzcWb2lpktNbPJKZ4/z8zWmNn8cLkgyv6k9ItfwBlnwI9/DI880uabFxFpTyJLCmbWEbgDOA4YBIw3s0Epqt7v7sPC5Y9R9adRHTrA9OkwalRwRLqyss27ICLSXkQ5UhgFLHX3Ze7+KVAOnBLh9lquSxd49FHo0wdOOgmqqnLdIxGRnDCP6ACrmZ0OjHP3C8L1c4DR7n5pUp3zgP8G1gBvAz9y9/dTtDUBmABQXFw8ory8vM7z1dXVFBUVtbrP3ZYtY/gPfsDWvn3539/+lh1durS6zZbIVjztSdxiils8EL+Y4hYPtC6msWPHVrr7yGYrunskC3AG8Mek9XOA39ar0xvoHD6+CHiuuXZHjBjh9VVUVDQoa7G//c29Qwf3c8/NXpsZymo87UTcYopbPO7xiylu8bi3LiZgnqfx3R3l7qMqYN+k9X7AynoJaZ27bwtX7wJGRNif9IwbB1dcAffeC2++meveiIi0qSiTwqvAAWa2n5ntBpwFzEmuYGZ9k1ZPBpZE2J/0XXUVFBbCDTfkuiciIm0qsqTg7jXApcBTBF/2s919kZlNMbOTw2qXmdkiM1sAXAacF1V/MrLXXnDJJTBrlkYLIrJLifQ6BXd/wt2/4O77u/uNYdlP3X1O+Pgadx/s7ge7+1h3bz/fwBotiMguaNed5qI5e+4Jl16q0YKI7FKUFJoyaVIwWvjFL3LdExGRNqGk0BSNFkRkF6Ok0JxJk4Jbt2m0ICK7ACWF5mi0ICK7ECWFdFx5ZTBamDIl1z0REYmUkkI6akcL5eWwpH1cXyciEgUlhXTVjhZ0bEFEYkxJIV0aLYjILkBJIRM6E0lEYk5JIRN9+sAPfhCMFhYvznVvRESyTkkhUzq2ICIxpqSQqdrRwv33a7QgIrGjpNASGi2ISEwpKbSERgsiElNKCi115ZXQrZtGCyISK0oKLZV8JtJRR8Gdd8Lq1bnulYhIqygptMZPfgI/+xmsXAkTJ0LfvnD00TBtGqxZk+veiYhkLNKkYGbjzOwtM1tqZpNTPN/ZzO4Pn3/ZzEqi7E/WdekC118fXOG8cCFcey289x5ceGGQII49Fv74R1i3Ltc9FRFJS6eoGjazjsAdwDFAFfCqmc1x9+Qjs98D/s/dB5rZWcAvgTOj6lNkzOCgg4JlypQgQcyeHRyI/v734aKLghFEaWlw1lKXLp/9TfG464oVsHQpdOoEBQXB39oleb2DBnoikl2RJQVgFLDU3ZcBmFk5cAqQnBROAa4PH/8V+J2Zmbt7hP2KlhkcfHCw3HADzJ8fJIgHHoBnnoEdO5ptYlS62+rQoeFi1vjj2qW2n42VAbgHS1OPk2NOtSRtd/S2bcGtTZNfk+pxqvV0ZdpmumUpfHnTpuBEgxiJW0xxiweg35FHwpgxkW4jyqSwD/B+0noVMLqxOu5eY2brgd7A2uRKZjYBmABQXFzM3Llz6zRSXV3doKxd+frXgwWwmho6bNtGh23b6Bj+7bBtGx0//ZQOW7fSYds2Pt24ka6dOmE7dgRLTU3Kxx1qamDnzqDdnTvB/bO/7g2fS/HFnqosISlJeP0v3BTJw2ofJ6+HfajZvp1OBQV1t1n7+mRZ+D1g6bTZyu3U7L47mzpF+d+n7cUtprjFA7Cxa9fIv+uifMdS/eSq/z8xnTq4+zRgGsDIkSN9TL1MOXfuXOqX5bO5c+cyIkbxQDw/ozjFA/GLKW7xACxug5ii3CldBeybtN4PWNlYHTPrBPQEPo6wTyIi0oQok8KrwAFmtp+Z7QacBcypV2cOcG74+HTgubw+niAikuci230UHiO4FHgK6Ajc7e6LzGwKMM/d5wB/Au41s6UEI4SzouqPiIg0L9KjMO7+BPBEvbKfJj3eCpwRZR9ERCR9OtFdREQSlBRERCRBSUFERBKUFEREJMHy7QxQM1sDrKhX3Id6V0HnubjFA/GLKW7xQPxiils80LqYBrj7ns1VyrukkIqZzXP3kbnuR7bELR6IX0xxiwfiF1Pc4oG2iUm7j0REJEFJQUREEuKSFKblugNZFrd4IH4xxS0eiF9McYsH2iCmWBxTEBGR7IjLSEFERLJASUFERBLyOimY2Tgze8vMlprZ5Fz3JxvMbLmZvW5m881sXq770xJmdreZrTazN5LK9jCzp83snfDv7rnsYyYaied6M/sg/Jzmm9nxuexjJsxsXzOrMLMlZrbIzC4Py/P5M2osprz8nMys0MxeMbMFYTw/D8v3M7OXw8/o/vC2BNnddr4eUzCzjsDbwDEEN+t5FRjv7oubfGE7Z2bLgZHunrcX3ZjZEUA1cI+7DwnLfgV87O43hQl8d3e/Opf9TFcj8VwPVLv7zbnsW0uYWV+gr7u/ZmbdgUrgVOA88vczaiymb5GHn5OZGdDN3avNrAB4CbgcuAJ4yN3LzexOYIG7T83mtvN5pDAKWOruy9z9U6AcOCXHfRLA3V+g4R30TgGmh4+nE/yHzQuNxJO33H2Vu78WPt4ILCG4X3o+f0aNxZSXPFAdrhaEiwNHAX8NyyP5jPI5KewDvJ+0XkUe/yNI4sDfzazSzCbkujNZVOzuqyD4DwzsleP+ZMOlZrYw3L2UN7takplZCTAceJmYfEb1YoI8/ZzMrKOZzQdWA08D7wKfuHtNWCWS77x8TgqWoiw/94XVdZi7lwLHAZeEuy6k/ZkK7A8MA1YBt+S2O5kzsyLgQeCH7r4h1/3JhhQx5e3n5O473H0Ywf3tRwFfSlUt29vN56RQBeybtN4PWJmjvmSNu68M/64GHib4xxAHH4X7fWv3/67OcX9axd0/Cv/T7gTuIs8+p3A/9YPATHd/KCzO688oVUz5/jkBuPsnwFzgEKCXmdXeMTOS77x8TgqvAgeER+N3I7i/85wc96lVzKxbeJAMM+sGHAu80fSr8sYc4Nzw8bnAoznsS6vVfnmGvkEefU7hQcw/AUvc/dakp/L2M2ospnz9nMxsTzPrFT7uAhxNcJykAjg9rBbJZ5S3Zx8BhKeX3QZ0BO529xtz3KVWMbPPE4wOILh/9n35GJOZzQLGEEzz+xHwM+ARYDbQH3gPOMPd8+LgbSPxjCHYJeHAcuDC2v3x7Z2ZHQ68CLwO7AyLryXYB5+vn1FjMY0nDz8nMxtKcCC5I8GP99nuPiX8jigH9gD+Fzjb3bdlddv5nBRERCS78nn3kYiIZJmSgoiIJCgpiIhIgpKCiIgkKCmIiEiCkoJIyMx2JM2mOT+bM++aWUnyLKsi7VWn5quI7DK2hNMKiOyyNFIQaUZ4j4tfhvPbv2JmA8PyAWb2bDjZ2rNm1j8sLzazh8O58BeY2VfCpjqa2V3h/Ph/D69UxcwuM7PFYTvlOQpTBFBSEEnWpd7uozOTntvg7qOA3xFcRU/4+B53HwrMBG4Py28Hnnf3g4FSYFFYfgBwh7sPBj4BvhmWTwaGh+1cFFVwIunQFc0iITOrdveiFOXLgaPcfVk46dqH7t7bzNYS3Nhle1i+yt37mNkaoF/y9APhdM5Pu/sB4frVQIG732BmTxLcxOcR4JGkefRF2pxGCiLp8UYeN1YnleQ5anbw2TG9E4A7gBFAZdIsmCJtTklBJD1nJv39V/j4nwSz8wKUEdwyEeBZYCIkbpTSo7FGzawDsK+7VwA/BnoBDUYrIm1Fv0hEPtMlvNNVrSfdvfa01M5m9jLBD6nxYdllwN1mdhWwBvhuWH45MM3MvkcwIphIcIOXVDoCM8ysJ8GNo34dzp8vkhM6piDSjPCYwkh3X5vrvohETbuPREQkQSMFERFJ0EhBREQSlBRERCRBSUFERBKUFEREJEFJQUREEv4/435Q4A5HwN4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "acc= history.history['acc']\n",
    "val_acc= history.history['val_acc']\n",
    "loss= history.history['loss']\n",
    "val_loss= history.history['val_loss']\n",
    "\n",
    "epochs= range(1,len(acc)+1)\n",
    "\n",
    "plt.plot(epochs, acc, color='red', label='Training Accuracy')\n",
    "plt.plot(epochs, val_acc, 'bo', color='blue', label='Validation Accuracy')\n",
    "plt.grid()\n",
    "plt.title('Training vs Validation Accuracy')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend()\n",
    "plt.show()\n",
    "\n",
    "\n",
    "plt.plot(epochs, loss, color='red', label='Training Loss')\n",
    "plt.plot(epochs, val_loss, 'bo', color='blue', label='Validation Loss')\n",
    "plt.grid()\n",
    "plt.title('Training vs Validation Loss')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Loss')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Validation accuracy stalls in the low 50s. So in this case, pretrained word embeddings outperform jointly learned embeddings. If we increase the number of training samples, this will quickly stop being the case.\n",
    "\n",
    "Finally, let’s evaluate the model on the test data. First, we need to tokenize the test data.\n",
    "\n",
    "### Tokenize The Data of The Test Set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_dir= os.path.join(imdb_dir,'test')\n",
    "labels=[]\n",
    "texts=[]\n",
    "for label_type in ['pos', 'neg']:\n",
    "    dir_name= os.path.join(test_dir, label_type)\n",
    "    for fname in sorted(os.listdir(dir_name)):\n",
    "        if fname[-4:]=='.txt':\n",
    "            f= open(os.path.join(dir_name, fname), encoding='utf8')\n",
    "            texts.append(f.read())\n",
    "            f.close()\n",
    "            if label_type=='neg':\n",
    "                labels.append(0)\n",
    "            else:\n",
    "                labels.append(1)\n",
    "                \n",
    "sequences= tokenizer.texts_to_sequences(text)\n",
    "x_test= pad_sequences(sequences, maxlen)\n",
    "y_test= np.array(labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluate The Model on The Test Set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25000/25000 [==============================] - 1s 37us/step\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[2.3902314934539794, 0.49404]"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.load_weights('pretrained_glove_model.h5')\n",
    "model.evaluate(x_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We get an appalling test accuracy of 49%. Working with just a handful of training samples is difficult!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}