notebooks/nlp/Text Classification/SentimentClassification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "### Sentiment Classification\n",
    "\n",
    "Jay Urbain, PhD\n",
    "\n",
    "#### Predict Sentiment From Movie Reviews Using Deep Learning\n",
    "\n",
    "We will be training a neural network for text classification. In text classification, documents or segments of text are assigned to different categories. For example, categories could be positive or negative sentiment, different authors, different topics, or different writing styles.\n",
    "\n",
    "In this lab, you will discover how you can predict the sentiment of movie reviews as either positive or negative in Python using the Keras deep learning library.\n",
    "\n",
    "We will build several models and evaluate the results.\n",
    "\n",
    "Credits:  \n",
    "TensorFlow, https://www.tensorflow.org/tutorials/.  \n",
    "Froncois Challet, 'Deep Learning with Python.'\n",
    "\n",
    "Topic:  \n",
    "- NLP terminology  \n",
    "- IMDB dataset description\n",
    "- Dataset preparation  \n",
    "- Language embeddings  \n",
    "- Develop and evaluate a multi-layer perceptron (MLP)  \n",
    "- Develop and evaluate a one-dimensional CNN  \n",
    "- Develop and evaluate a one-dimensional RNN  \n",
    "- Model comparison and analysis  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### NLP terminology for describing input data:\n",
    "\n",
    "- `token` - A unit of text, typically a work, but it could also be a phrase like \"New York\", a sub-word like \"mega\" in \"megabyte\", or a letter like \"m\". Each token can be represented as a distinct number in a process called *tokenizing*. \n",
    "- `document` - A sequence of *tokens.* This could be whole book or a tweet. In this case, we're going to classify each *movie review* as having a positive or a negative sentiment.  \n",
    "- `corpus` - A set of *documents.* You can think of this as your \"dataset\". If you want to learn about language in general, you might use a *corpus* like Wikipedia. In this case, we'll use a *corpus* containing movie reviews since each *document* is paired with a rating indicating *sentiment.*\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Term-document metrics\n",
    "\n",
    "Terms in matrices can be represented by their presence in a document (1/0), term-frequency, `TF-IDF`, a widely used measure of word *importance*, or other application specific metric. In our application, we will use overall word sentiment.\n",
    "\n",
    "`TF` (*Term Frequency*), is the number of times a particular term $w$ appears in a document $d$.  \n",
    "\n",
    "$$TF(w, d) = \\frac{count(w, d)}{\\sum_{v \\in V}count(v, d)}$$  \n",
    "where $w, v$ words, in a document $d$, over the corpus vocabulary $V$.\n",
    "\n",
    "`DF` (*Document Frequency*) is the number of documents a particular word $w$ occurs.\n",
    "\n",
    "`IDF` (*Inversed Document Frequency*) represents how distinctive a word is: \n",
    "\n",
    "$$IDF(w) = log \\frac{|D|}{\\sum_{d \\in D}\\mathbb{1}(w, d)} $$\n",
    "where $D$ is a corpus of documents, $\\mathbb{1}$.\n",
    "\n",
    "`TF-IDF` is the product of $TF*IDF$ and assigns weights to words based on their frequency within a document ($TF$), and their overall distinctiveness ($IDF$) within the corpus.\n",
    "\n",
    "`Sentiment value` for our task will be using the overall sentiment for each individual word."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Tokenization\n",
    "\n",
    "The ```Tokenizer``` class in Keras allows you to vectorize a text corpus, by turning each text/document into either a sequence of integers (each integer being the index of a token in a dictionary) or into a vector where the coefficient for each token could be binary, based on word count, based on tf-idf, etc.\n",
    "\n",
    "Keras provides the `Tokenizer` class for preparing text documents for deep learning. The Tokenizer must be constructed, and then fit on either raw text documents or integer encoded text documents.\n",
    "\n",
    "Once the Tokenizer has been fit on training data, it can be used to encode documents in the train or test datasets.\n",
    "\n",
    "The `texts_to_matrix()` function on the `Tokenizer` can be used to create one vector per document provided per input. The length of the vectors is the total size of the vocabulary.\n",
    "\n",
    "This function provides a suite of standard bag-of-words model text encoding schemes that can be provided via a mode argument to the function.\n",
    "\n",
    "Once fit, the Tokenizer provides 4 attributes that you can use to query what has been learned about your documents:  \n",
    "- word_counts: A dictionary of words and their counts.  \n",
    "- word_docs: A dictionary of words and how many documents each appeared in.  \n",
    "- word_index: A dictionary of words and their uniquely assigned integers.  \n",
    "- document_count:An integer count of the total number of documents that were used to fit the Tokenizer.  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below is a basic of example of using the Tokenizer class where we encode the sample text using binary, count, and tfidf encoding respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OrderedDict([('great', 2), ('acting', 2), ('waste', 1), ('of', 1), ('time', 1), ('to', 1), ('see', 1), ('this', 1), ('movie', 1), ('effort', 1), ('major', 1), ('looser', 1), ('excellent', 1)])\n",
      "5\n",
      "{'great': 1, 'acting': 2, 'waste': 3, 'of': 4, 'time': 5, 'to': 6, 'see': 7, 'this': 8, 'movie': 9, 'effort': 10, 'major': 11, 'looser': 12, 'excellent': 13}\n",
      "defaultdict(<class 'int'>, {'great': 2, 'acting': 2, 'waste': 1, 'to': 1, 'this': 1, 'of': 1, 'time': 1, 'movie': 1, 'see': 1, 'effort': 1, 'looser': 1, 'major': 1, 'excellent': 1})\n",
      "Binary\n",
      "[[0. 1. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      " [0. 0. 0. 1. 1. 1. 1. 1. 1. 1. 0. 0. 0. 0.]\n",
      " [0. 1. 1. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0.]\n",
      " [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 1. 0.]\n",
      " [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1.]]\n",
      "\n",
      "Count\n",
      "[[0. 1. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      " [0. 0. 0. 1. 1. 1. 1. 1. 1. 1. 0. 0. 0. 0.]\n",
      " [0. 1. 1. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0.]\n",
      " [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 1. 0.]\n",
      " [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1.]]\n",
      "\n",
      "tfidf\n",
      "[[0.         0.98082925 0.98082925 0.         0.         0.\n",
      "  0.         0.         0.         0.         0.         0.\n",
      "  0.         0.        ]\n",
      " [0.         0.         0.         1.25276297 1.25276297 1.25276297\n",
      "  1.25276297 1.25276297 1.25276297 1.25276297 0.         0.\n",
      "  0.         0.        ]\n",
      " [0.         0.98082925 0.98082925 0.         0.         0.\n",
      "  0.         0.         0.         0.         1.25276297 0.\n",
      "  0.         0.        ]\n",
      " [0.         0.         0.         0.         0.         0.\n",
      "  0.         0.         0.         0.         0.         1.25276297\n",
      "  1.25276297 0.        ]\n",
      " [0.         0.         0.         0.         0.         0.\n",
      "  0.         0.         0.         0.         0.         0.\n",
      "  0.         1.25276297]]\n"
     ]
    }
   ],
   "source": [
    "from keras.preprocessing.text import Tokenizer\n",
    "# define 5 documents\n",
    "docs = ['Great acting',\n",
    "\t\t'Waste of time to see this movie',\n",
    "\t\t'Great acting effort',\n",
    "\t\t'Major looser',\n",
    "\t\t'Excellent!']\n",
    "# create the tokenizer\n",
    "t = Tokenizer()\n",
    "# fit the tokenizer on the documents\n",
    "t.fit_on_texts(docs)\n",
    "# summarize what was learned\n",
    "print(t.word_counts)\n",
    "print(t.document_count)\n",
    "print(t.word_index)\n",
    "print(t.word_docs)\n",
    "# integer encode documents\n",
    "print(\"Binary\")\n",
    "encoded_docs = t.texts_to_matrix(docs, mode='binary')\n",
    "print(encoded_docs)\n",
    "\n",
    "print(\"\\nCount\")\n",
    "encoded_docs = t.texts_to_matrix(docs, mode='count')\n",
    "print(encoded_docs)\n",
    "\n",
    "print(\"\\ntfidf\")\n",
    "encoded_docs = t.texts_to_matrix(docs, mode='tfidf')\n",
    "print(encoded_docs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### IMDB dataset\n",
    "\n",
    "The dataset we will be using is the `Large Movie Review Dataset` often referred to as the IMDB dataset.\n",
    "\n",
    "The IMDB dataset was constructed from a collection of 50,000 reviews from IMDB, allowing no more than 30 reviews per movie. The constructed dataset contains an even number of positive and negative reviews, so randomly\n",
    "guessing yields 50% accuracy. A negative review has a score ≤ 4 out of 10, and a positive review has a score ≥ 7 out of 10.\n",
    "\n",
    "The data was collected by Stanford researchers and was used in the 2011 paper: <a href=\"http://ai.stanford.edu/~amaas/data/sentiment/\">Learning Word Vectors for Sentiment Analysis</a>, where an accuracy of 88.89% was achieved.\n",
    "\n",
    "Challenges with classifying sentiment can be shown in the following example which contains positive sentiment words like *funny* and *super witty*, but the overal sentiment of the review is clearly negative. \n",
    "\n",
    "<blockquote>\n",
    "\"*This movie was actually neither that funny, nor super witty.*\"\n",
    "</blockquote>\n",
    "\n",
    "To identify the correct sentiment, we will need to develop models that capture word context.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3\n"
     ]
    }
   ],
   "source": [
    "print(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Dataset preparation\n",
    "\n",
    "The `keras.datasets.imdb.load_data()` function allows you to load the dataset in a format that is ready for use in neural network and deep learning models. So we'll be able to skip the data cleansing and tokenization step discussed above. *It's important to understand that in most practical applications, this is a significant amount of work.*\n",
    "\n",
    "Each review in the dataset is labeled by sentiment (positive/negative). Reviews have been preprocessed, and each review is encoded as a sequence of word indexes (integer encoded). For convenience, words are indexed by overall frequency in the dataset, so that for instance the integer \"3\" encodes the 3rd most frequent word in the data. This allows for quick filtering operations such as: \"only consider the top 10,000 most common words, but eliminate the top 20 most common words\".\n",
    "\n",
    "As a convention, \"0\" does not stand for a specific word, but instead is used to encode any unknown word.\n",
    "\n",
    "Calling `imdb.load_data()` the first time will download the IMDB dataset to your computer and store it in your home directory under ~/.keras/datasets/imdb.pkl as a 32 megabyte file.\n",
    "\n",
    "`imdb.load_data()` provides additional arguments including the number of top words to load (where words with a lower integer are marked as zero in the returned data), the number of top words to skip (to avoid the “the”‘s) and the maximum length of reviews to support.\n",
    "\n",
    "Let’s load the dataset and calculate some properties of it. We will start off by loading some libraries and loading the entire IMDB dataset as a training dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy\n",
    "from keras.datasets import imdb\n",
    "from matplotlib import pyplot\n",
    "# load the dataset\n",
    "top_words = 5000\n",
    "(X_train, y_train), (X_test, y_test) = imdb.load_data(num_words=top_words)\n",
    "X = numpy.concatenate((X_train, X_test), axis=0)\n",
    "y = numpy.concatenate((y_train, y_test), axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display the first review. \n",
    "\n",
    "*Note: integers that indicate the absolute popularity of the word in the dataset.* This technique was developed in the Kaggle competition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([list([1, 14, 22, 16, 43, 530, 973, 1622, 1385, 65, 458, 4468, 66, 3941, 4, 173, 36, 256, 5, 25, 100, 43, 838, 112, 50, 670, 2, 9, 35, 480, 284, 5, 150, 4, 172, 112, 167, 2, 336, 385, 39, 4, 172, 4536, 1111, 17, 546, 38, 13, 447, 4, 192, 50, 16, 6, 147, 2025, 19, 14, 22, 4, 1920, 4613, 469, 4, 22, 71, 87, 12, 16, 43, 530, 38, 76, 15, 13, 1247, 4, 22, 17, 515, 17, 12, 16, 626, 18, 2, 5, 62, 386, 12, 8, 316, 8, 106, 5, 4, 2223, 2, 16, 480, 66, 3785, 33, 4, 130, 12, 16, 38, 619, 5, 25, 124, 51, 36, 135, 48, 25, 1415, 33, 6, 22, 12, 215, 28, 77, 52, 5, 14, 407, 16, 82, 2, 8, 4, 107, 117, 2, 15, 256, 4, 2, 7, 3766, 5, 723, 36, 71, 43, 530, 476, 26, 400, 317, 46, 7, 4, 2, 1029, 13, 104, 88, 4, 381, 15, 297, 98, 32, 2071, 56, 26, 141, 6, 194, 2, 18, 4, 226, 22, 21, 134, 476, 26, 480, 5, 144, 30, 2, 18, 51, 36, 28, 224, 92, 25, 104, 4, 226, 65, 16, 38, 1334, 88, 12, 16, 283, 5, 16, 4472, 113, 103, 32, 15, 16, 2, 19, 178, 32])],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X[:1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the IMDB word index and display the text for the first review.\n",
    "\n",
    "*Note: the first 3 indices are reserved for padding, start, and unknown word*\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<START> this film was just brilliant casting location scenery story direction everyone's really suited the part they played and you could just imagine being there robert <UNK> is an amazing actor and now the same being director <UNK> father came from the same scottish island as myself so i loved the fact there was a real connection with this film the witty remarks throughout the film were great it was just brilliant so much that i bought the film as soon as it was released for <UNK> and would recommend it to everyone to watch and the fly <UNK> was amazing really cried at the end it was so sad and you know what they say if you cry at a film it must have been good and this definitely was also <UNK> to the two little <UNK> that played the <UNK> of norman and paul they were just brilliant children are often left out of the <UNK> list i think because the stars that play them all grown up are such a big <UNK> for the whole film but these children are amazing and should be <UNK> for what they have done don't you think the whole story was so lovely because it was true and was someone's life after all that was <UNK> with us all\n"
     ]
    }
   ],
   "source": [
    "import keras\n",
    "NUM_WORDS=1000 # only use top 1000 words\n",
    "INDEX_FROM=3   # word index offset\n",
    "\n",
    "# train,test = keras.datasets.imdb.load_data(num_words=NUM_WORDS, index_from=INDEX_FROM)\n",
    "# train_x,train_y = train\n",
    "# test_x,test_y = test\n",
    "\n",
    "word_to_id = keras.datasets.imdb.get_word_index()\n",
    "word_to_id = {k:(v+INDEX_FROM) for k,v in word_to_id.items()}\n",
    "word_to_id[\"<PAD>\"] = 0\n",
    "word_to_id[\"<START>\"] = 1\n",
    "word_to_id[\"<UNK>\"] = 2\n",
    "\n",
    "id_to_word = {value:key for key,value in word_to_id.items()}\n",
    "print(' '.join(id_to_word[id] for id in X[0] ))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display the sentiment label of the first ten reviews"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 0, 0, 1, 0, 0, 1, 0, 1, 0])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### TODO\n",
    "\n",
    "Display a few other reviews along with their sentiment in the cell(s) below. You can use the following print statment and adjust the index:\n",
    "\n",
    "```\n",
    "    print(' '.join(id_to_word[id] for id in X[0] ))\n",
    "```\n",
    "\n",
    "Record your observations about postive and negative sentiment reviews. What makes reviews positive or negative? Can you find any tricky examples where the true sentiment may be difficult to judge? Please share your results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we can display the shape of the training dataset. We should have 25,000 training and 25,000 testing examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training data: \n",
      "(25000,)\n",
      "(25000,)\n",
      "Test data: \n",
      "(25000,)\n",
      "(25000,)\n"
     ]
    }
   ],
   "source": [
    "# summarize size\n",
    "print(\"Training data: \")\n",
    "print(X_train.shape)\n",
    "print(y_train.shape)\n",
    "\n",
    "print(\"Test data: \")\n",
    "print(X_test.shape)\n",
    "print(y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also print the unique class values (encoding)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classes: \n",
      "[0 1]\n"
     ]
    }
   ],
   "source": [
    "# Summarize number of classes\n",
    "print(\"Classes: \")\n",
    "print(numpy.unique(y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we can get an idea of the total number of unique words in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of words: \n",
      "4998\n"
     ]
    }
   ],
   "source": [
    "# Summarize number of words\n",
    "print(\"Number of words: \")\n",
    "print(len(numpy.unique(numpy.hstack(X))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can get an idea of the average review length.\n",
    "\n",
    "Looking at a box and whisker plot for the review lengths in words, we can probably cover most of the mass of the word distribution with a clipped length of 400 to 500 words.\n",
    "\n",
    "In a box plot, the rectangle includes the center 50 percentile (inter quartile range) of the distribution. The red horizontal bar in the box is the median. The whisker (horizonal bars) are by default +/-1.5 the IQR. Dots drawn outside of the whiskers, are considered outliers.\n",
    "\n",
    "https://matplotlib.org/api/_as_gen/matplotlib.pyplot.boxplot.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Review length: \n",
      "Mean 234.76 words (172.911495)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD8CAYAAAB+UHOxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAFIpJREFUeJzt3X9sVXWe//Hn29Ifoc7yW2KsfDGGbMo2WZ10HJPhj2/nm6j4j+wfk7FOdggQ+ZJIw35R0bV/ON/dQDYky4ZpZmXc0BlJlhqT3WXIqssY0mRCZmfX+h3joN2JZBalgoCCM6aktLSf7x89YBGEnlvoaXuej+Tm3vvu5977vn+0r57zOedzIqWEJKl8bim6AUlSMQwASSopA0CSSsoAkKSSMgAkqaQMAEkqKQNAkkrKAJCkkjIAJKmkZhXdwLUsXLgwLV26tOg2JGlaeeuttz5JKS263rgpHQBLly6lp6en6DYkaVqJiA/GM85dQJJUUgaAJJWUASBJJWUASFJJXTcAIuLOiOiOiN6IeDciNmX1H0TERxHxdnZ7eMxr/jIijkTEbyPiwTH1h7LakYh49uZ8JUnSeIxnC+AC8GRKqRG4H3giIpZnP/u7lNI92e01gOxnjwJ/AjwE/H1EVEVEFfAjYCWwHGgd8z7StNHV1UVTUxNVVVU0NTXR1dVVdEtSRa57GGhK6QRwInv8eUT0Andc4yWPAC+nlM4D/x0RR4D7sp8dSSn9DiAiXs7GvjeB/qVJ1dXVRXt7O7t372bFihUcOnSIdevWAdDa2lpwd1I+ueYAImIpcC/wH1lpY0S8ExGdETEvq90BHBvzsr6s9lV1adrYunUru3fvpqWlherqalpaWti9ezdbt24tujUpt3EHQETcCvwT8BcppT8ALwB3A/cwuoXwtxeHXuXl6Rr1L3/O+ojoiYie06dPj7c9aVL09vayYsWKy2orVqygt7e3oI6kyo0rACKimtE//v+YUvpngJTSyZTScEppBPgHvtjN0wfcOeblDcDxa9Qvk1J6MaXUnFJqXrToumcyS5OqsbGRQ4cOXVY7dOgQjY2NBXUkVW48RwEFsBvoTSntGFO/fcywPwMOZ4/3A49GRG1E3AUsA/4TeBNYFhF3RUQNoxPF+2/M15AmR3t7O+vWraO7u5uhoSG6u7tZt24d7e3tRbcm5TaetYC+Bfw58JuIeDurPcfoUTz3MLob5yjwvwFSSu9GxCuMTu5eAJ5IKQ0DRMRG4ABQBXSmlN69gd9FuukuTvS2tbXR29tLY2MjW7dudQJY01KkdMVu+Cmjubk5uRicJOUTEW+llJqvN84zgSWppAwASSopA0CSSsoAkKSSMgAkqaQMAEkqKQNAysnVQDVTTOmLwktTjauBaibxRDAph6amJjo6OmhpablU6+7upq2tjcOHD1/jldLkGe+JYAaAlENVVRUDAwNUV1dfqg0NDVFXV8fw8HCBnUlf8Exg6SZwNVDNJAaAlIOrgWomcRJYysHVQDWTOAcgSTOMcwCSpGsyACSppAwASSopA0CSSsoAkKSSMgAkqaQMAEkqKQNAkkrKAJBy8noAmikMACmHrq4uNm3aRH9/Pykl+vv72bRpkyGgackAkHLYsmULVVVVdHZ2cv78eTo7O6mqqmLLli1FtyblZgBIOfT19bFnzx5aWlqorq6mpaWFPXv20NfXV3RrUm4GgCSVlAEg5dDQ0MDq1asvux7A6tWraWhoKLo1KTcDQMph+/btXLhwgbVr11JXV8fatWu5cOEC27dvL7o1KTcDQMqhtbWVnTt3Ul9fD0B9fT07d+70gjCalrwgjCTNMDfsgjARcWdEdEdEb0S8GxGbsvr8iHgjIt7P7udl9YiIH0bEkYh4JyK+Pua9Vmfj34+I1RP5gpKkiRnPLqALwJMppUbgfuCJiFgOPAscTCktAw5mzwFWAsuy23rgBRgNDOB54JvAfcDzF0NDkjT5rhsAKaUTKaX/lz3+HOgF7gAeAV7Khr0ErMoePwLsSaN+BcyNiNuBB4E3UkpnUkpngTeAh27ot5EkjVuuSeCIWArcC/wHsDildAJGQwK4LRt2B3BszMv6stpX1SVJBRh3AETErcA/AX+RUvrDtYZepZauUf/y56yPiJ6I6Dl9+vR425Mk5TSuAIiIakb/+P9jSumfs/LJbNcO2f2prN4H3Dnm5Q3A8WvUL5NSejGl1JxSal60aFGe7yJJymE8RwEFsBvoTSntGPOj/cDFI3lWAz8bU/9+djTQ/cDvs11EB4AHImJeNvn7QFaTJBVg1jjGfAv4c+A3EfF2VnsO+BvglYhYB3wIfCf72WvAw8AR4BywBiCldCYi/hp4Mxv3VymlMzfkW0iScvNEMEmaYW7YiWCSpJnJAJCkkjIAJKmkDAApp7a2Nurq6ogI6urqaGtrK7olqSIGgJRDW1sbu3btYtu2bfT397Nt2zZ27dplCGha8iggKYe6ujq2bdvG5s2bL9V27NjBc889x8DAQIGdSV8Y71FABoCUQ0TQ39/P7NmzL9XOnTtHfX09U/l3SeXiYaDSTVBbW8uuXbsuq+3atYva2tqCOpIqN54zgSVlHn/8cZ555hkANmzYwK5du3jmmWfYsGFDwZ1J+RkAUg4dHR0APPfcczz55JPU1tayYcOGS3VpOnEOQJJmGOcAJEnXZABIUkkZAFJOXV1dNDU1UVVVRVNTE11dXUW3JFXESWAph66uLtrb29m9ezcrVqzg0KFDrFu3DoDW1taCu5PycRJYyqGpqYlVq1axb98+ent7aWxsvPT88OHDRbcnAeOfBHYLQMrhvffe49y5c1dsARw9erTo1qTcnAOQcqipqWHjxo20tLRQXV1NS0sLGzdupKampujWpNwMACmHwcFBOjo66O7uZmhoiO7ubjo6OhgcHCy6NSk3dwFJOSxfvpxVq1bR1tZ2aQ7ge9/7Hvv27Su6NSk3twCkHNrb29m7dy8dHR0MDAzQ0dHB3r17aW9vL7o1KTe3AKQcWltb+eUvf8nKlSs5f/48tbW1PP744x4CqmnJLQAph66uLl599VVef/11BgcHef3113n11Vc9GUzTkucBSDk0NTXR0dFBS0vLpVp3dzdtbW2eB6ApwyuCSTdBVVUVAwMDVFdXX6oNDQ1RV1fH8PBwgZ1JX3A1UOkmaGxs5NChQ5fVDh06RGNjY0EdSZVzEljKob29ne9+97vU19fz4YcfsmTJEvr7+9m5c2fRrUm5uQUgVWgq7z6VxsMAkHLYunUr69evp76+noigvr6e9evXs3Xr1qJbk3JzF5CUw3vvvcfJkye59dZbAejv7+fHP/4xn376acGdSfm5BSDlUFVVxcjICJ2dnQwMDNDZ2cnIyAhVVVVFtybldt0AiIjOiDgVEYfH1H4QER9FxNvZ7eExP/vLiDgSEb+NiAfH1B/Kakci4tkb/1Wkm+/ChQtXrPxZU1PDhQsXCupIqtx4tgB+Cjx0lfrfpZTuyW6vAUTEcuBR4E+y1/x9RFRFRBXwI2AlsBxozcZK086aNWtoa2ujrq6OtrY21qxZU3RLUkWuOweQUvpFRCwd5/s9ArycUjoP/HdEHAHuy352JKX0O4CIeDkb+17ujqUCNTQ08JOf/IS9e/deuiDMY489RkNDQ9GtSblNZA5gY0S8k+0impfV7gCOjRnTl9W+qn6FiFgfET0R0XP69OkJtCfdeNu3b2d4eJi1a9dSW1vL2rVrGR4eZvv27UW3JuVWaQC8ANwN3AOcAP42q8dVxqZr1K8spvRiSqk5pdS8aNGiCtuTbo7W1lZ27tx52WGgO3fudDVQTUsVHQaaUjp58XFE/APwr9nTPuDOMUMbgOPZ46+qS9NKa2urf/A1I1S0BRARt495+mfAxSOE9gOPRkRtRNwFLAP+E3gTWBYRd0VEDaMTxfsrb1uSNFHjOQy0C/h34I8joi8i1gHbI+I3EfEO0AL8H4CU0rvAK4xO7v4b8ERKaTildAHYCBwAeoFXsrHStNPV1UVTUxNVVVU0NTV5LQBNW+M5Cuhq27q7rzF+K3DFefHZoaKv5epOmmK6urrYtGkT9fX1pJTo7+9n06ZNAO4W0rTjmcBSDlu2bGFwcPCy2uDgIFu2bCmoI6lyBoCUQ19f36VVQCNGD25LKdHX11dkW1JFDAApp1mzZl22FtCsWa6pqOnJAJBy+vJ1ALwugKYr/3WRchoYGODBBx9kaGiI6upqtwA0bbkFIOUwf/58BgYGWLBgAbfccgsLFixgYGCA+fPnF92alJv/ukg5zJ49m5GREerq6kgpUVdXx5w5c5g9e3bRrUm5uQUg5XD8+HGam5v54IMPSCnxwQcf0NzczPHjrmyi6ccAkHKYO3cuBw8eZPHixdxyyy0sXryYgwcPMnfu3KJbk3IzAKQcPvvsMyKCp59+ms8//5ynn36aiOCzzz4rujUpNwNAymFkZISnnnqKzs5Ovva1r9HZ2clTTz3FyMhI0a1JuRkAUk4LFy7k8OHDDA8Pc/jwYRYuXFh0S1JFYiqfxNLc3Jx6enqKbkO6ZMGCBZw9e5bFixdz6tQpbrvtNk6ePMm8efP49NNPi25PAiAi3kopNV9vnFsAUg6PPfYYAB9//DEjIyN8/PHHl9Wl6cQAkHLYt28fdXV1VFdXA1BdXU1dXR379u0ruDMpPwNAyqGvr485c+Zw4MABBgcHOXDgAHPmzHE1UE1LBoCU0+bNm2lpaaG6upqWlhY2b95cdEtSRQwAKacdO3bQ3d3N0NAQ3d3d7Nixo+iWpIq4FpCUQ0NDAx999BHf/va3L9UigoaGhgK7kirjFoCUQ0RcWgQOuLQo3MWrg0nTiVsAUg7Hjh3j3nvvZXBwkN7eXu6++25qamr49a9/XXRrUm4GgJTTz3/+88vO/v3kk09YtGhRgR1JlTEApJy+8Y1vcOLECc6fP09tbS2333570S1JFTEApBzmz5/P0aNHL+3zHxwc5OjRo14RTNOSk8BSDheXfb64htbFe5eD1nRkAEg5XFz2uaamhoigpqbmsro0nbgLSKrA4ODgZffSdOQWgFSBi3MAHv+v6cwAkCrw5TkAaToyACSppK4bABHRGRGnIuLwmNr8iHgjIt7P7udl9YiIH0bEkYh4JyK+PuY1q7Px70fE6pvzdSRJ4zWeLYCfAg99qfYscDCltAw4mD0HWAksy27rgRdgNDCA54FvAvcBz18MDUlSMa4bACmlXwBnvlR+BHgpe/wSsGpMfU8a9StgbkTcDjwIvJFSOpNSOgu8wZWhIkmaRJXOASxOKZ0AyO5vy+p3AMfGjOvLal9VlyQV5EZPAl/tmLh0jfqVbxCxPiJ6IqLn9OnTN7Q5SdIXKg2Ak9muHbL7U1m9D7hzzLgG4Pg16ldIKb2YUmpOKTW7wqIk3TyVBsB+4OKRPKuBn42pfz87Guh+4PfZLqIDwAMRMS+b/H0gq0mSCnLdpSAiogv4n8DCiOhj9GievwFeiYh1wIfAd7LhrwEPA0eAc8AagJTSmYj4a+DNbNxfpZS+PLEsSZpEMZXPZGxubk49PT1FtyFdcq2lH6by75LKJSLeSik1X2+cZwJLUkkZAJJUUgaAJJWUASBJJWUASFJJGQCSVFIGgCSVlAEgSSVlAEhSSRkAklRSBoAklZQBIEklZQBIUkkZAJJUUgaAJJWUASBJJWUASFJJGQCSVFIGgCSVlAEgSSVlAEhSSRkAklRSBoAklZQBIEklZQBIUkkZAJJUUgaAJJWUASBJJWUASFJJGQCSVFIGgCSV1IQCICKORsRvIuLtiOjJavMj4o2IeD+7n5fVIyJ+GBFHIuKdiPj6jfgCkqTK3IgtgJaU0j0ppebs+bPAwZTSMuBg9hxgJbAsu60HXrgBny1JqtDN2AX0CPBS9vglYNWY+p406lfA3Ii4/SZ8vpRbRIzrNtH3kKaSiQZAAn4eEW9FxPqstjildAIgu78tq98BHBvz2r6sJhUupTSu20TfQ5pKZk3w9d9KKR2PiNuANyLiv64x9mr//lzxG5EFyXqAJUuWTLA9SdJXmdAWQErpeHZ/CvgX4D7g5MVdO9n9qWx4H3DnmJc3AMev8p4vppSaU0rNixYtmkh70g33Vf/F+9+9pqOKAyAi6iPiaxcfAw8Ah4H9wOps2GrgZ9nj/cD3s6OB7gd+f3FXkTSdjN2d464dTWcT2QW0GPiXbGJrFrA3pfRvEfEm8EpErAM+BL6TjX8NeBg4ApwD1kzgsyVJE1RxAKSUfgf86VXqnwL/6yr1BDxR6edJkm4szwSWpJIyACSppAwASSopA0CSSsoAkKSSMgAkqaQMAEkqKQNAkkrKAJCkkjIAJKmkDABJKikDQJJKaqIXhJGmpPnz53P27Nmb/jk3+zKP8+bN48yZMzf1M1ReBoBmpLNnz86Idfq9jrBuJncBSVJJGQCSVFIGgCSVlAEgSSVlAEhSSRkAklRSHgaqGSk9/0fwgzlFtzFh6fk/KroFzWAGgGak+L9/mDHnAaQfFN2FZip3AUlSSRkAklRS7gLSjDUTllGYN29e0S1oBjMANCNNxv7/iJgR8wwqL3cBSVJJGQCSVFIGgCSVlAEgSSVlAEhSSU16AETEQxHx24g4EhHPTvbnS5JGTWoAREQV8CNgJbAcaI2I5ZPZgyRp1GRvAdwHHEkp/S6lNAi8DDwyyT1Ikpj8E8HuAI6Ned4HfHPsgIhYD6wHWLJkyeR1plKr9KzhvK/zxDFNJZO9BXC135bLfiNSSi+mlJpTSs2LFi2apLZUdimlSblJU8lkB0AfcOeY5w3A8UnuQZLE5AfAm8CyiLgrImqAR4H9k9yDJIlJngNIKV2IiI3AAaAK6EwpvTuZPUiSRk36aqAppdeA1yb7cyVJl/NMYEkqKQNAkkrKAJCkkjIAJKmkYiqfnBIRp4EPiu5D+goLgU+KbkK6iv+RUrrumbRTOgCkqSwielJKzUX3IVXKXUCSVFIGgCSVlAEgVe7FohuQJsI5AEkqKbcAJKmkDAApp4jojIhTEXG46F6kiTAApPx+CjxUdBPSRBkAUk4ppV8AZ4ruQ5ooA0CSSsoAkKSSMgAkqaQMAEkqKQNAyikiuoB/B/44IvoiYl3RPUmV8ExgSSoptwAkqaQMAEkqKQNAkkrKAJCkkjIAJKmkDABJKikDQJJKygCQpJL6/8OI2tn3bWPVAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Summarize review length\n",
    "print(\"Review length: \")\n",
    "result = [len(x) for x in X]\n",
    "print(\"Mean %.2f words (%f)\" % (numpy.mean(result), numpy.std(result)))\n",
    "# plot review length\n",
    "pyplot.boxplot(result)\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Neural networks don’t take raw text as input. Only numeric tensors.\n",
    "\n",
    "Vectorizing text is the process of transforming text into numeric tensors. This can be done in multiple ways:   \n",
    "- Segment text into words, and transform each word into a vector.   \n",
    "- Segment text into characters, and transform each character into a vector.   \n",
    "- Extract n-grams of words or characters, and transform each n-gram into a vector.   \n",
    "- Apply categorical transformation: One-hot encoding or word embedding.\n",
    "\n",
    "\n",
    "*One-hot*   \n",
    "Words are categorical, they have no ordinal relationship. So they can’t just be numerically encoded.\n",
    "For each word:  \n",
    "- Numerically encode each word: {red:0, green:1, blue:2}  \n",
    "- Generate vector of zeros the length of all possible words/categories  \n",
    "- Set the index value for the word in the vector  \n",
    "\n",
    "|red|green|blue|\n",
    "|-----|-----|-----|\n",
    "|1|0|0|\n",
    "|0|1|0|\n",
    "|0|0|1|\n",
    "\n",
    "One-hot encoding results in a sparse vector representation (mostly zeros, hi dimensions). Word embeddings provide dense vector representations. Idea is to “embed” word into a lower-dimensionality space.\n",
    "The dimensions of this space are typically defined by word context, i.e., semantically similar words are embedded near each other.  \n",
    "\n",
    "Popular word-embedding algorithms:  \n",
    "- Point-wise mutual information (PMI)  \n",
    "- Word2Vec: Skip-gram or CBOW  \n",
    "- GLoVE - Global Vectors for Word Representation  \n",
    "\n",
    "*Word2Vec*  \n",
    "CBOW model predict missing word (focus word) using context (surrounding words). Skip gram model predicts context based on the word in focus. Context is a fixed number of words to the left and right of the word in focus. Maximize average log probability of context words co-occurring with focus words.\n",
    "\n",
    "![Word2Vec](word2vec.png)\n",
    "\n",
    "One way to understand the concept of embedding is to think of words being embedded in a higher dimensional vector space.\n",
    "\n",
    "![Vector Space](vector_space.png)\n",
    "\n",
    "This comes from the deck of slides: http://www.slideshare.net/ChristopherMoody3/word2vec-lda-and-introducing-a-new-hybrid-algorithm-lda2vec-57135994\n",
    "\n",
    "For our lab, we will use Keras' `Embedding` layer to automatically build SkipGram embeddings for our text collection.\n",
    "\n",
    "The layer takes arguments that define the mapping including the maximum number of expected words also called the vocabulary size (e.g. the largest integer value that will be seen as an integer). The layer also allows you to specify the dimensionality for each word vector, called the output dimension.\n",
    "\n",
    "We would like to use a word embedding representation for the IMDB dataset.\n",
    "\n",
    "If we are only interested in the 5,000 most frequently used words in the dataset, our vocabulary size will be 5,000. We can choose to use a 32-dimension vector to represent each word. Most word vectors are in the 100 to 200 dimension range for general NLP problems.\n",
    "\n",
    "Embeddings will help us match words by meaning, not just surface form, and help capture word context.\n",
    "\n",
    "The word embedding representation is a true innovation. We will demonstrate what would have been considered world class results in 2011 with a relatively simple neural network.\n",
    "\n",
    "The first layer of our model will be an word embedding layer created using the Embedding class as follows:\n",
    "\n",
    "`Embedding(5000, 32, input_length=500)`\n",
    "\n",
    "The output of this first layer would be a matrix with the size 32×500 for a given review training or test pattern in integer format."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Truncation and padding input examples\n",
    "\n",
    "We can use the Keras utility `sequence.pad_sequences()` to truncate or pad the dataset to a length of 500 for each observation using the function. Neural network models expect input examples to be the same dimension\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Keras\n",
    "The core data structure of Keras is a model, a way to organize layers. The simplest type of model is the  Sequential model, a linear stack of layers. For more complex architectures, you should use the Keras functional API, which allows you to build arbitrary graphs of layers.\n",
    "\n",
    "Here is the Sequential model:   \n",
    "```\n",
    "from keras.models import Sequential\n",
    "model = Sequential()\n",
    "```\n",
    "\n",
    "Stacking layers by using .add():  \n",
    "```\n",
    "from keras.layers import Dense\n",
    "model.add(Dense(units=64, activation='relu', input_dim=100))\n",
    "model.add(Dense(units=10, activation='softmax'))\n",
    "```\n",
    "\n",
    "Once your model looks good, configure its learning process with .compile():  \n",
    "```\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='sgd',\n",
    "              metrics=['accuracy'])\n",
    "```\n",
    "                            \n",
    "If you need to, you can further configure your optimizer.   \n",
    "```\n",
    "model.compile(loss=keras.losses.categorical_crossentropy,\n",
    "              optimizer=keras.optimizers.SGD(lr=0.01, momentum=0.9, nesterov=True))\n",
    "```\n",
    "\n",
    "You can iterate on your training data in batches:\n",
    "```\n",
    "model.fit(x_train, y_train, epochs=5, batch_size=32)\n",
    "```\n",
    "\n",
    "More here:   \n",
    "\n",
    "https://keras.io/  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Simple Multi-Layer Perceptron Model \n",
    "\n",
    "We can start off by developing a simple multi-layer perceptron (MLP) model with a single hidden layer. MLP's are also called a fully-connected network (FCN). They're \"traditional\" neural networks.\n",
    "\n",
    "Let’s start off by importing the classes and functions required for this model and initializing the random number generator to a constant value to ensure we can easily reproduce the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# MLP for the IMDB problem\n",
    "import numpy\n",
    "from keras.datasets import imdb\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "from keras.layers import Flatten\n",
    "from keras.layers.embeddings import Embedding\n",
    "from keras.preprocessing import sequence\n",
    "# fix random seed for reproducibility\n",
    "seed = 42\n",
    "numpy.random.seed(seed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will limit our vocabulary to the 5,000 most common terms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the dataset but only keep the top n words, zero the rest\n",
    "top_words = 5000\n",
    "(X_train, y_train), (X_test, y_test) = imdb.load_data(num_words=top_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will limit each review length to 500 words. Longer reviews will be truncated. Shorter reviews will use zero-padding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_length = 500\n",
    "X_train = sequence.pad_sequences(X_train, maxlen=input_length)\n",
    "X_test = sequence.pad_sequences(X_test, maxlen=input_length)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will start with an embedding size of 32.\n",
    "\n",
    "The top_words, max_words, and embedding_size are considered `hyperparameters` which require tuning by conducting several iterations of training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_size = 32"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output of the first embedding layer will be a 32×500 sized tensor.\n",
    "\n",
    "We will flatten the embedding layer's output to one dimension, then use one dense hidden layer of 250 units with a rectified linear unit (ReLU) activation function. \n",
    "\n",
    "ReLU is a common nonlinearity, and is defined by a simple formula:\n",
    "$$ReLU(z) = max(0, z)$$\n",
    "Here is its graphical representation (and next to another activation, sigmoid, for comparison):\n",
    "![](https://cdn-images-1.medium.com/max/1600/1*XxxiA0jJvPrHEJHD4z893g.png)\n",
    "\n",
    "The output layer has one neuron and we will use a sigmoid activation function to output values of 0 and 1 as predictions. If we had to classify more than two values, we would use softmax.\n",
    "\n",
    "The model uses binary cross-entropy loss:\n",
    "\n",
    "$$loss = -[y_t * log_2(y_p) + (1 - y_t)*log_2(1-y_p)]$$\n",
    "\n",
    "and is optimized using the efficient gradient descent based ADAM optimization procedure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_1 (Embedding)      (None, 500, 32)           160000    \n",
      "_________________________________________________________________\n",
      "flatten_1 (Flatten)          (None, 16000)             0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 250)               4000250   \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 1)                 251       \n",
      "=================================================================\n",
      "Total params: 4,160,501\n",
      "Trainable params: 4,160,501\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "# create the model\n",
    "model = Sequential()\n",
    "model.add(Embedding(top_words, embedding_size, input_length=input_length))\n",
    "model.add(Flatten())\n",
    "model.add(Dense(250, activation='relu'))\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "print(model.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition to printing out the model summary, there are other ways to visualize our architecture.\n",
    "\n",
    "Yaml, which stands for \"YAML ain't markup language.\" Seriously. That's what it stands for. YAML will let us read what we've built. It can also be used to automatically define a network.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "backend: tensorflow\n",
      "class_name: Sequential\n",
      "config:\n",
      "  layers:\n",
      "  - class_name: Embedding\n",
      "    config:\n",
      "      activity_regularizer: null\n",
      "      batch_input_shape: !!python/tuple [null, 500]\n",
      "      dtype: float32\n",
      "      embeddings_constraint: null\n",
      "      embeddings_initializer:\n",
      "        class_name: RandomUniform\n",
      "        config: {maxval: 0.05, minval: -0.05, seed: null}\n",
      "      embeddings_regularizer: null\n",
      "      input_dim: 5000\n",
      "      input_length: 500\n",
      "      mask_zero: false\n",
      "      name: embedding_1\n",
      "      output_dim: 32\n",
      "      trainable: true\n",
      "  - class_name: Flatten\n",
      "    config: {data_format: channels_last, name: flatten_1, trainable: true}\n",
      "  - class_name: Dense\n",
      "    config:\n",
      "      activation: relu\n",
      "      activity_regularizer: null\n",
      "      bias_constraint: null\n",
      "      bias_initializer:\n",
      "        class_name: Zeros\n",
      "        config: {}\n",
      "      bias_regularizer: null\n",
      "      kernel_constraint: null\n",
      "      kernel_initializer:\n",
      "        class_name: VarianceScaling\n",
      "        config: {distribution: uniform, mode: fan_avg, scale: 1.0, seed: null}\n",
      "      kernel_regularizer: null\n",
      "      name: dense_1\n",
      "      trainable: true\n",
      "      units: 250\n",
      "      use_bias: true\n",
      "  - class_name: Dense\n",
      "    config:\n",
      "      activation: sigmoid\n",
      "      activity_regularizer: null\n",
      "      bias_constraint: null\n",
      "      bias_initializer:\n",
      "        class_name: Zeros\n",
      "        config: {}\n",
      "      bias_regularizer: null\n",
      "      kernel_constraint: null\n",
      "      kernel_initializer:\n",
      "        class_name: VarianceScaling\n",
      "        config: {distribution: uniform, mode: fan_avg, scale: 1.0, seed: null}\n",
      "      kernel_regularizer: null\n",
      "      name: dense_2\n",
      "      trainable: true\n",
      "      units: 1\n",
      "      use_bias: true\n",
      "  name: sequential_1\n",
      "keras_version: 2.2.4\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(model.to_yaml())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also generate a graphic. Note: Graphviz and pydot need to be installed.\n",
    "\n",
    "$ pip install pydot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<svg height=\"369pt\" viewBox=\"0.00 0.00 332.46 369.00\" width=\"332pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g class=\"graph\" id=\"graph0\" transform=\"scale(1 1) rotate(0) translate(4 365)\">\n",
       "<title>G</title>\n",
       "<polygon fill=\"#ffffff\" points=\"-4,4 -4,-365 328.459,-365 328.459,4 -4,4\" stroke=\"transparent\"/>\n",
       "<!-- 48032489768 -->\n",
       "<g class=\"node\" id=\"node1\">\n",
       "<title>48032489768</title>\n",
       "<polygon fill=\"none\" points=\"0,-243.5 0,-287.5 324.459,-287.5 324.459,-243.5 0,-243.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"82.0708\" y=\"-261.3\">embedding_1: Embedding</text>\n",
       "<polyline fill=\"none\" points=\"164.1416,-243.5 164.1416,-287.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"191.9761\" y=\"-272.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"164.1416,-265.5 219.8105,-265.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"191.9761\" y=\"-250.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"219.8105,-243.5 219.8105,-287.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"272.1348\" y=\"-272.3\">(None, 500)</text>\n",
       "<polyline fill=\"none\" points=\"219.8105,-265.5 324.459,-265.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"272.1348\" y=\"-250.3\">(None, 500, 32)</text>\n",
       "</g>\n",
       "<!-- 48032474224 -->\n",
       "<g class=\"node\" id=\"node2\">\n",
       "<title>48032474224</title>\n",
       "<polygon fill=\"none\" points=\"26.438,-162.5 26.438,-206.5 298.021,-206.5 298.021,-162.5 26.438,-162.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"82.0708\" y=\"-180.3\">flatten_1: Flatten</text>\n",
       "<polyline fill=\"none\" points=\"137.7036,-162.5 137.7036,-206.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"165.5381\" y=\"-191.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"137.7036,-184.5 193.3726,-184.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"165.5381\" y=\"-169.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"193.3726,-162.5 193.3726,-206.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"245.6968\" y=\"-191.3\">(None, 500, 32)</text>\n",
       "<polyline fill=\"none\" points=\"193.3726,-184.5 298.021,-184.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"245.6968\" y=\"-169.3\">(None, 16000)</text>\n",
       "</g>\n",
       "<!-- 48032489768&#45;&gt;48032474224 -->\n",
       "<g class=\"edge\" id=\"edge2\">\n",
       "<title>48032489768-&gt;48032474224</title>\n",
       "<path d=\"M162.2295,-243.3664C162.2295,-235.1516 162.2295,-225.6579 162.2295,-216.7252\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"165.7296,-216.6068 162.2295,-206.6068 158.7296,-216.6069 165.7296,-216.6068\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 48032491616 -->\n",
       "<g class=\"node\" id=\"node3\">\n",
       "<title>48032491616</title>\n",
       "<polygon fill=\"none\" points=\"33.4448,-81.5 33.4448,-125.5 291.0142,-125.5 291.0142,-81.5 33.4448,-81.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"85.5708\" y=\"-99.3\">dense_1: Dense</text>\n",
       "<polyline fill=\"none\" points=\"137.6968,-81.5 137.6968,-125.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"165.5313\" y=\"-110.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"137.6968,-103.5 193.3657,-103.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"165.5313\" y=\"-88.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"193.3657,-81.5 193.3657,-125.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"242.1899\" y=\"-110.3\">(None, 16000)</text>\n",
       "<polyline fill=\"none\" points=\"193.3657,-103.5 291.0142,-103.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"242.1899\" y=\"-88.3\">(None, 250)</text>\n",
       "</g>\n",
       "<!-- 48032474224&#45;&gt;48032491616 -->\n",
       "<g class=\"edge\" id=\"edge3\">\n",
       "<title>48032474224-&gt;48032491616</title>\n",
       "<path d=\"M162.2295,-162.3664C162.2295,-154.1516 162.2295,-144.6579 162.2295,-135.7252\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"165.7296,-135.6068 162.2295,-125.6068 158.7296,-135.6069 165.7296,-135.6068\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 48032492064 -->\n",
       "<g class=\"node\" id=\"node4\">\n",
       "<title>48032492064</title>\n",
       "<polygon fill=\"none\" points=\"40.4448,-.5 40.4448,-44.5 284.0142,-44.5 284.0142,-.5 40.4448,-.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"92.5708\" y=\"-18.3\">dense_2: Dense</text>\n",
       "<polyline fill=\"none\" points=\"144.6968,-.5 144.6968,-44.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"172.5313\" y=\"-29.3\">input:</text>\n",
       "<polyline fill=\"none\" points=\"144.6968,-22.5 200.3657,-22.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"172.5313\" y=\"-7.3\">output:</text>\n",
       "<polyline fill=\"none\" points=\"200.3657,-.5 200.3657,-44.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"242.1899\" y=\"-29.3\">(None, 250)</text>\n",
       "<polyline fill=\"none\" points=\"200.3657,-22.5 284.0142,-22.5 \" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"242.1899\" y=\"-7.3\">(None, 1)</text>\n",
       "</g>\n",
       "<!-- 48032491616&#45;&gt;48032492064 -->\n",
       "<g class=\"edge\" id=\"edge4\">\n",
       "<title>48032491616-&gt;48032492064</title>\n",
       "<path d=\"M162.2295,-81.3664C162.2295,-73.1516 162.2295,-63.6579 162.2295,-54.7252\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"165.7296,-54.6068 162.2295,-44.6068 158.7296,-54.6069 165.7296,-54.6068\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "<!-- 48032490888 -->\n",
       "<g class=\"node\" id=\"node5\">\n",
       "<title>48032490888</title>\n",
       "<polygon fill=\"none\" points=\"115.7295,-324.5 115.7295,-360.5 208.7295,-360.5 208.7295,-324.5 115.7295,-324.5\" stroke=\"#000000\"/>\n",
       "<text fill=\"#000000\" font-family=\"Times,serif\" font-size=\"14.00\" text-anchor=\"middle\" x=\"162.2295\" y=\"-338.3\">48032490888</text>\n",
       "</g>\n",
       "<!-- 48032490888&#45;&gt;48032489768 -->\n",
       "<g class=\"edge\" id=\"edge1\">\n",
       "<title>48032490888-&gt;48032489768</title>\n",
       "<path d=\"M162.2295,-324.2521C162.2295,-316.3888 162.2295,-306.9498 162.2295,-297.9612\" fill=\"none\" stroke=\"#000000\"/>\n",
       "<polygon fill=\"#000000\" points=\"165.7296,-297.7376 162.2295,-287.7377 158.7296,-297.7377 165.7296,-297.7376\" stroke=\"#000000\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.SVG object>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from keras.utils import plot_model\n",
    "from IPython.display import SVG\n",
    "from keras.utils.vis_utils import model_to_dot\n",
    "\n",
    "plot_model(model, to_file='model.png', show_shapes=True)\n",
    "SVG(model_to_dot(model, show_shapes=True).create(prog='dot', format='svg'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TODO:**   \n",
    "\n",
    "Take a moment and try to understand the model you just built and the summary. Note the number of parameters in the model. This relates to the number of weights that have to be learned using back propagation. The larger the number of parameters, the more complex the model. The more complex the model, the longer the training time and a need for larger training sets.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can fit the model. We will hold out 10% of the training data for validation. Validation data is used to \"test\" the model during training to see if our loss decreases and our acccurcy decreases. The model can be re-trained before deployment with all available data.\n",
    "\n",
    "We've specified 3 epochs, or number of iterations across the training data.\n",
    "\n",
    "Batch size defines how many examples we show to the network before updating weights using back propagation.\n",
    "\n",
    "We will perform final evaluation using the held-out test results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 22500 samples, validate on 2500 samples\n",
      "Epoch 1/3\n",
      "22500/22500 [==============================] - 11s 507us/step - loss: 0.6353 - acc: 0.6123 - val_loss: 0.3964 - val_acc: 0.8240\n",
      "Epoch 2/3\n",
      "22500/22500 [==============================] - 12s 526us/step - loss: 0.2537 - acc: 0.8951 - val_loss: 0.2888 - val_acc: 0.8828\n",
      "Epoch 3/3\n",
      "22500/22500 [==============================] - 13s 573us/step - loss: 0.1122 - acc: 0.9615 - val_loss: 0.3471 - val_acc: 0.8728\n",
      "Accuracy: 86.88%\n"
     ]
    }
   ],
   "source": [
    "# Fit the model\n",
    "epochs=3\n",
    "batch_size=128\n",
    "\n",
    "history = model.fit(X_train, y_train,\n",
    "                    batch_size=batch_size,\n",
    "                    epochs=epochs,\n",
    "                    verbose=1,\n",
    "                    validation_split=0.1)\n",
    "\n",
    "# Final evaluation of the model\n",
    "scores = model.evaluate(X_test, y_test, verbose=0)\n",
    "print(\"Accuracy: %.2f%%\" % (scores[1]*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Running this example fits the model and summarizes the estimated performance. We can see that a realtively simple model achieves a score of nearly ~86% which is in the neighborhood of the original paper, with very little effort. And good engineering!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can plot the accuracy and training loss using the history from training the model. Since we only used 3 training epochs, these graphs will not show much. 10's or 100's of epochs is commong in more complex models.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['val_loss', 'val_acc', 'loss', 'acc'])\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0xb2b3103c8>]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xl8VPW9//HXh7DvSQjKFpIoqCjKMiS4tG5FqbXF1mpRUBCQelu91i63ervT3qu3y7W3t9xWBAQExaX+LK1trVa97a0mJJEdRUPYAgiYsC+BJJ/fH3OgQwxmIpPMZOb9fDzy4Mw535P55HDynpMzZ87H3B0REUkNbeJdgIiItByFvohIClHoi4ikEIW+iEgKUeiLiKQQhb6ISApR6IuIpBCFvohIClHoi4ikkLbxLqC+Xr16eU5OTrzLEBFpVUpLS99396zGxiVc6Ofk5FBSUhLvMkREWhUz2xTNOJ3eERFJIQp9EZEUotAXEUkhCn0RkRSi0BcRSSEKfRGRFKLQFxFJIQl3nb6ISCpxd9bvOkjRhkoM49aC7GZ9PoW+iEgLcnfe3XmAovJKCjdUUVRexfsHqgEYnt1ToS8i0prV1TnrduynqLySog1VLN1QReXBowCc2b0jl52dSUFeJgW5GeT26tLs9Sj0RURiqLbOeWv7Poo2VFFUXsnSjVXsOXQMgH49O3H5OVmMzs2kIC+D7IzOmFmL1hdV6JvZWOC/gDRgtrs/VG/5QGAukAVUARPdvSJYVgusCoZudvfPxKh2EZG4q6mtY+32fRSVV1G0oZKlG6rYd6QGgAEZnRhz3hknjuQHZHSOc7VRhL6ZpQEzgTFABVBsZkvcfW3EsJ8CC9x9vpldBTwI3BYsO+zuw2Jct4hIXByrrWP11r0njuSLN+7mQHU45HN7deG6oX0oyMugIDeTvj07xbnaD4rmSD8fKHP3cgAzWwyMAyJDfwhwXzD9KvB8LIsUEYmXozV1rNq6h8LyKoo2VFG6sYqDR2sBOCurC58Z1peC3AxG52VyRveOca62cdGEfj9gS8TjCqCg3pgVwI2ETwF9FuhmZpnuXgl0NLMSoAZ4yN31giAiCau6ppYVW/aeeOO1dNNuDh8Lh/zgM7ryuRH9KcjLID83g97dEj/k64sm9Bt6l8HrPf468Eszmwz8FdhKOOQBst19m5nlAa+Y2Sp3X3/SE5hNB6YDZGc37+VKIiKRjhyrZdnmPRRtqKSovIo3N++muqYOgHPP7MYXRg2gIDcc8pldO8S52tMXTehXAAMiHvcHtkUOcPdtwOcAzKwrcKO7741YhruXm9lrwHBgfb31ZwGzAEKhUP0XFBGRmDl8tJY3N+8+cZ388s17OFpbhxkM6dOdCQUDw0fyORmkd2kf73JjLprQLwYGmVku4SP48cCtkQPMrBdQ5e51wAOEr+TBzNKBQ+5eHYy5FPhxDOsXEflQB6trKN20+8SR/IqKPRyrddoYXNCvB5MuGUhBbiajcjLo0bldvMttdo2GvrvXmNndwIuEL9mc6+5rzGwGUOLuS4ArgAfNzAmf3vlysPp5wCNmVkf4Pj8P1bvqR0QkpvYfOUbJpt0nLqFcVbGXmjonrY0xtF8PplyWy+jcTEbmpNO9Y/KHfH3mnlhnU0KhkKtHrohEa+/hY5RsDF9ZU1heyeqte6lzaJdmXNi/JwW5GRTkZTJyYDpdOyTv51HNrNTdQ42NS94tICJJac+hoyzdEA75og2VrNm2D3don9aGYQN68uUrz6YgN5MRA3vSub0irj5tERFJaFUHj7J0QyWF5eEj+XU79odDvm0bRmT35J+vGsTovEyGZ/ekY7u0eJeb8BT6IpJQdu2vDo7kKyksr+SdHQcA6NiuDSMHpnPf0MGMzsvkogE96NBWId9UCn0Riaud+44EtxgOh/z6XQcB6Nw+jZED0xk3rB+j8zIY2q8n7duq79PpUuiLSIvavvfwiStrCsur2PB+OOS7dmhLKCedm0LhD0Nd0K8H7dIU8rGm0BeRZlWx+9BJIb+56hAA3Tq2JT8ng1vzsynIy2BIn+60Vcg3O4W+iMSMu7Ol6jCFwQehCssr2brnMAA9OrUjPzeDSZfkUJCbwXl9upPWpmXvJS8KfRE5De7OxspDJ25OVlheyfa9RwDI6NKe/JwM7vxYLgV5mZxzRjfaKOTjTqEvIlGLbOJdWB5+83Xn/nB/115d21OQl8no4MNQZ2d1VcgnIIW+iJzSSU28g/vJH2/i3btbB0bnZZ5oGHJWVpcWb/0nTafQF5ETIpt4F5ZXsXRjFVVBE+8+PTrysUG9TtzWICez5fu7yulT6IuksMgm3oXllRTXa+J95Tm9KcjLYHRuJgMyOinkk4BCXySFRDbxLiyvZOnGKvYHTbyzMzpzzZAzKMgNn7Lpnx7/Jt4Sewp9kSQW2cS7sLySkogm3nm9unD9hX1OhHyfHonXxFtiT6EvkkQim3gXlldSumk3h4Im3mf37sq4YX1PXGHTuxU08ZbYU+iLtGLHm3gXlldStCEc8keOhfu7nnNGNz4/sj8FuZnk52aQ1a3193eV06fQF2lFjjfxPh7yyzbvobom3N/13DO7M35UNqPzMsjPzSQjCfu7yulT6IsksONNvAvLw7c1WL4l3MS7jcGQvt2ZOHogBbkZ5Odm0LOzQl4ap9AXSSDHm3gXBrc1WBnRxHtovx5MvjR835pQTgY9OqVef1c5fVGFvpmNBf6LcGP02e7+UL3lA4G5QBZQBUx094pg2STg28HQH7n7/BjVLtLqHW/iffxIfvXWcBPvtm2Mof17MPWyPAryMggNTKdbCjbxlthrNPTNLA2YCYwBKoBiM1vi7msjhv0UWODu883sKuBB4DYzywC+B4QAB0qDdXfH+gcRaQ2ON/E+fiQf2cT7ov49+eLleRTkhpt4d0niJt4SP9HsVflAmbuXA5jZYmAcEBn6Q4D7gulXgeeD6WuBl9y9Klj3JWAs8OTply6S+I438S4M7ie/dntEE+/sntx95dkU5GUyIjudTu3V+k+aXzSh3w/YEvG4AiioN2YFcCPhU0CfBbqZWeYp1u33kasVSXCVB473dz25iXeHtm0YkZ3OvVeHm3gPG6Am3hIf0YR+Qzfb8HqPvw780swmA38FtgI1Ua6LmU0HpgNkZ2dHUZJIYti1v5qioGFI0YZ/NPHu1C7c3/VTQ/sw+qxMLuyvJt6SGKIJ/QpgQMTj/sC2yAHuvg34HICZdQVudPe9ZlYBXFFv3dfqP4G7zwJmAYRCoQ+8KIgkih37jpw4H18U0cS7S/s0RuZkBE28Mxnar4eaeEtCiib0i4FBZpZL+Ah+PHBr5AAz6wVUuXsd8ADhK3kAXgT+3czSg8fXBMtFWpW9h49x54ISlm6oAqBb0MT75tAACvIyuaCv+rtK69Bo6Lt7jZndTTjA04C57r7GzGYAJe6+hPDR/INm5oRP73w5WLfKzH5I+IUDYMbxN3VFWouD1TVMfmwpq7fu5RvXnsPHB2UxpK/6u0rrZO6JdTYlFAp5SUlJvMsQAcK3PbjjsWKWbqxi5q0jGHvBmfEuSaRBZlbq7qHGxunvUZFTOFpTx5cWvUnhhkp+etOFCnxJCgp9kQbU1jn3Pb2cV97eyY9uuIDPDu8f75JEYkKhL1JPXZ3zzd+s5IWV2/nWdecxoWBgvEsSiRmFvkgEd+cHv1vDs6UV3Hv1IO78eF68SxKJKYW+SISfvLiO+W9s4s6P5fKVTwyKdzkiMafQFwnMfLWM/3ltPbcWZPOv152HmS7JlOSj0BcBHvv7Bn7y4jpuGNaXH427QIEvSUuhLynv6eIt/OB3a7n2/DP46U0X0UYfupIkptCXlPa7Fdv45nMr+fjgLH5xy3DdSkGSnvZwSVkvr93BfU8tZ9TADB6ZOFJ3wZSUoNCXlPT3svf50hNvMqRvd+ZMDqmBiaQMhb6knNJNVUybX0JuZhfm35Gv3rOSUhT6klJWb93L5MeKObNHRx6flk96l/bxLkmkRSn0JWW8u2M/t89dSveO7Vg4rYDe3TrGuySRFqfQl5SwqfIgE2YXkdbGWDStgH49O8W7JJG4UOhL0tu25zC3PlrEsdo6Fk4tIKdXl3iXJBI3Cn1Jarv2VzNxdhH7Dh9jwZQCzjmzW7xLEomraHrkirRKew4d5bY5RWzbe5jHpxYwtH+PeJckEnc60pekdKC6hkmPFVO+6yCP3h5iVE5GvEsSSQg60pekc/hoLVPnFbN6615+NWEEHxuUFe+SRBJGVEf6ZjbWzNaZWZmZ3d/A8mwze9XMlpnZSjO7LpifY2aHzWx58PXrWP8AIpGO1tTxT4tKWbqxiv+8+SKuOV99bUUiNXqkb2ZpwExgDFABFJvZEndfGzHs28DT7v4rMxsC/AHICZatd/dhsS1b5INqauu4d/EyXlu3i4c+N5Rxw/rFuySRhBPNkX4+UObu5e5+FFgMjKs3xoHuwXQPYFvsShRpXF2d8y/PruSPq9/jO9cPYXx+drxLEklI0YR+P2BLxOOKYF6k7wMTzayC8FH+PRHLcoPTPv9rZh9r6AnMbLqZlZhZya5du6KvXoRwX9vvLlnNc8u28rUxg5l6WW68SxJJWNGEfkMdJbze41uAee7eH7gOeNzM2gDbgWx3Hw58FXjCzLrXWxd3n+XuIXcPZWXpTTeJnrvz0B/fZmHhZr54eR53X3V2vEsSSWjRhH4FMCDicX8+ePpmKvA0gLu/AXQEerl7tbtXBvNLgfXA4NMtWuS4/36ljEf+Ws5towdy/9hz1eZQpBHRhH4xMMjMcs2sPTAeWFJvzGbgagAzO49w6O8ys6zgjWDMLA8YBJTHqnhJbbP/Vs5/vvQOnxvRjx985nwFvkgUGr16x91rzOxu4EUgDZjr7mvMbAZQ4u5LgK8Bj5rZfYRP/Ux2dzezjwMzzKwGqAXucveqZvtpJGU8uXQzP3rhLT55wZn8+MYL1ddWJErmXv/0fHyFQiEvKSmJdxmSwH67fCtfeWo5lw/OYtZtIdq31QfLRcys1N1DjY3Tb4u0Kn9e8x5ffXoFBbkZ/HriSAW+SBPpN0Zajb+9u4u7n1jG0H49mD1pFB3bqa+tSFMp9KVVKN5YxZ0LSsjLCve17dpBt40S+SgU+pLwVlbsYcpjxfTt2YnHpxbQo7MamYt8VAp9SWjr3gv3te3RuR2LphWQ1a1DvEsSadUU+pKwNrx/kIlziujQtg2LphXQp4f62oqcLoW+JKStew4z4dFCauucRdMKGJipvrYisaDQl4Szc/8RJjxayP7qGhZMyefs3uprKxIrCn1JKLsPHuW22UvZub+aeXeM4oJ+6msrEku67k0Sxv4jx5j02FI2VB5k3uRRjByovrYisaYjfUkI4b62Jazdto9fTRjBJWf3indJIklJoS9xV11Ty/THSyjZVMXDXxjG1eedEe+SRJKWTu9IXB2rreOeJ5bxt3ff58efv5BPX9Q33iWJJDUd6Uvc1NU533hmBX9eu4Pvf3oIN4cGNL6SiJwWhb7EhbvzredX8/zybXzj2nOYfKn62oq0BIW+tDh3599eeIsnl27mS1ecxZevVF9bkZai0JcW9/OX32X2/21g8iU5fOPac+JdjkhKUehLi5r11/X811/e5aaR/fnu9UPU11akhSn0pcUsLNzEv//hbT51YR8eUl9bkbiIKvTNbKyZrTOzMjO7v4Hl2Wb2qpktM7OVZnZdxLIHgvXWmdm1sSxeWo/n3qzgO79dzdXn9ubhm4eRpsAXiYtGr9M3szRgJjAGqACKzWyJu6+NGPZt4Gl3/5WZDQH+AOQE0+OB84G+wMtmNtjda2P9g0ji+tPq7Xz9mRVcnJfJzAkj1NdWJI6i+e3LB8rcvdzdjwKLgXH1xjjQPZjuAWwLpscBi9292t03AGXB95MU8dq6ndzz5DKGDejJo7eH1NdWJM6iCf1+wJaIxxXBvEjfByaaWQXho/x7mrCuJKnC8kq++Hgpg3p347E78umivrYicRdN6Dd08tXrPb4FmOfu/YHrgMfNrE2U62Jm082sxMxKdu3aFUVJkuiWb9nD1HnFDMjozONT8+nRSX1tRRJBNKFfAUR+Pr4//zh9c9xU4GkAd38D6Aj0inJd3H2Wu4fcPZSVlRV99ZKQ3tq+j0lzl5LZtQMLpxaQ2VV9bUUSRTShXwwMMrNcM2tP+I3ZJfXGbAauBjCz8wiH/q5g3Hgz62BmucAgYGmsipfEs37XAW6bU0SndmksmlbAmT06xrskEYnQ6ElWd68xs7uBF4E0YK67rzGzGUCJuy8BvgY8amb3ET59M9ndHVhjZk8Da4Ea4Mu6cid5bak6xMTZRbjDwmkFDMjoHO+SRKQeC2dz4giFQl5SUhLvMqSJduw7ws2PvMHug0dZPP1ihvTt3vhKIhIzZlbq7qHGxumCaTltVQePMnF2Ee/vr2b+lHwFvkgC0zV0clr2HTnG7XOL2Fx1iHl35DM8Oz3eJYnIh9CRvnxkh47WcMdjxax7bz+/njiSi8/KjHdJItIIhb58JEeO1XLnghKWbd7NL8YP58pze8e7JBGJgk7vSJMdq63j7ife5O9llfzspov45NA+8S5JRKKkI31pkto656tPr+Dlt3byw3Hnc+PI/vEuSUSaQKEvUaurc/71uVX8bsU27v/kudx2cU68SxKRJlLoS1TcnRm/X8tTJVv456vO5q7Lz4p3SSLyESj0JSo/+/M7zHt9I1MuzeW+MYPjXY6IfEQKfWnUr15bzy9fLWP8qAF85/rz1NdWpBVT6MuHWvDGRv7jT2/zmYv68m+fHarAF2nlFPpySs+UbOG7v13DmCFn8LObL1JfW5EkoNCXBr2wcjvf/M1KLju7F/99y3DapWlXEUkG+k2WD3jl7R3cu3gZI7LTmXX7SPW1FUkiCn05yevr3+euhW9yXp/uzL1jFJ3b60PbIslEoS8nvLl5N9Pml5CT2Zn5U/Lp3lF9bUWSjUJfAFizbS+T5y6ld7dwX9uMLu3jXZKINAOFvlC28wC3z1lK1w5tWTitgN7d1ddWJFkp9FPc5spDTJhdiJmxcFoB/dPV11YkmUUV+mY21szWmVmZmd3fwPKHzWx58PWOme2JWFYbsWxJLIuX0/Pe3iNMmFNIdU0dC6flk5fVNd4liUgza/TSDDNLA2YCY4AKoNjMlrj72uNj3P2+iPH3AMMjvsVhdx8Wu5IlFt4/UM2E2YXsPniMRdMKOPdM9bUVSQXRHOnnA2XuXu7uR4HFwLgPGX8L8GQsipPmsffQMW6fs5Stew4zZ1KIiwb0jHdJItJCogn9fsCWiMcVwbwPMLOBQC7wSsTsjmZWYmaFZnbDR65UYuJAdQ2T5y2lbOcBHrktREGe+tqKpJJoPnnT0A1X/BRjxwPPunttxLxsd99mZnnAK2a2yt3Xn/QEZtOB6QDZ2dlRlCQfxZFjtdw5v4SVFXuZeesILh+cFe+SRKSFRXOkXwEMiHjcH9h2irHjqXdqx923Bf+WA69x8vn+42NmuXvI3UNZWQqi5nC0po4vLXqTwg3hvrZjLzgz3iWJSBxEE/rFwCAzyzWz9oSD/QNX4ZjZOUA68EbEvHQz6xBM9wIuBdbWX1eaV01tHfc9tZxX3t7Jv90wlBuGN3h2TkRSQKOnd9y9xszuBl4E0oC57r7GzGYAJe5+/AXgFmCxu0ee+jkPeMTM6gi/wDwUedWPNL+6Ouf+51bxwqrtfPtT53FrgU6fiaQyOzmj4y8UCnlJSUm8y0gK7s73lqxhwRub+MonBvGVT6jNoUiyMrNSdw81Nk6fyE1iP35xHQve2MSdH8vl3qsHxbscEUkACv0kNfPVMn712nomFGTzr9epr62IhCn0k9Dc/9vAT15cx2eH9+OH4y5Q4IvICQr9JPN08RZm/H4t155/Bj/5/IW0UV9bEYmg0E8iS1Zs45vPreTjg7P4xS3Daau+tiJSj1IhSby8dgdffWo5o3IyeGTiSDq0VV9bEfkghX4S+HvZ+3zpiTc5v2935kwK0am9Al9EGqbQb+VKNlYxbX4Jeb26MH9KPt3U11ZEPoRCvxVbvXUvdzxWzJk9OvL41AJ6dlZfWxH5cAr9VuqdHfu5bU4R3Tu1Y9G0ArK6dYh3SSLSCij0W6FNlQeZOLuItmltWDStgL49O8W7JBFpJRT6rcy2PYe59dEijtXWsWhaATm9usS7JBFpRRT6rciu/dVMnF3EvsPHWDClgMFndIt3SSLSykTTOUsSwJ5DR7ltThHb9x7h8an5DO3fI94liUgrpCP9VuBAdQ2THiumfNdBHr09RCgnI94liUgrpSP9BHf4aC1T5hWzeutefj1xJJcN6hXvkkSkFdORfgKrrqnlroWlFG+s4j9vvogxQ86Id0ki0sop9BNUTW0d9z65nP99ZxcPfW4o44apr62InD6FfgKqq3P+5dmV/GnNe3z3+iF8YZT62opIbCj0E4y7853frua5ZVv52pjBTLksN94liUgSiSr0zWysma0zszIzu7+B5Q+b2fLg6x0z2xOxbJKZvRt8TYpl8cnG3Xnwj2+zqGgzd11+FndfdXa8SxKRJNPo1TtmlgbMBMYAFUCxmS1x97XHx7j7fRHj7wGGB9MZwPeAEOBAabDu7pj+FEniF38pY9Zfy7n94oF8c+w5anMoIjEXzZF+PlDm7uXufhRYDIz7kPG3AE8G09cCL7l7VRD0LwFjT6fgZDX7b+U8/PI73DiiP9//9PkKfBFpFtGEfj9gS8TjimDeB5jZQCAXeKUp65rZdDMrMbOSXbt2RVN3UnmiaDM/euEtrht6Jv9x41D1tRWRZhNN6DeUQH6KseOBZ929tinruvssdw+5eygrKyuKkpLH88u28q3nV3HlOVn8/AvqaysizSuahKkABkQ87g9sO8XY8fzj1E5T1005L655j689s4KC3Ax+NXEk7dsq8EWkeUWTMsXAIDPLNbP2hIN9Sf1BZnYOkA68ETH7ReAaM0s3s3TgmmBeyvvrO7u454llDO3Xg9mTRtGxnfraikjza/TqHXevMbO7CYd1GjDX3deY2QygxN2PvwDcAix2d49Yt8rMfkj4hQNghrtXxfZHaH2Wbqhi+uMlnNW7K/PvyKdrB90CSURahkVkdEIIhUJeUlIS7zKazcqKPdz6aBG9u3fg6S9eTK+uanMoIqfPzErdPdTYOJ1EbkHr3tvP7XOX0rNzuK+tAl9EWppCv4VseP8gE2YX0aFtG56YNpo+PdTXVkRank4mt4CK3YeY8Gghde4snjaa7MzO8S5JRFKUjvSb2c59R5g4u4j91TUsmJLP2b3V11ZE4keh34x2HzzKxDlF7Nxfzbw78rmgn/raikh86fROM9l35Bi3z13KxspDzJs8ipED0+NdkoiIjvSbw6GjNUydV8xb2/fx64kjuORs9bUVkcSg0I+x6ppavvh4KaWbdvPz8cO46lz1tRWRxKHTOzF0rLaOu59Yxt/efZ+ffP5Crr+wb7xLEhE5iY70Y6S2zvn6Myt4ae0OfvCZ87kpNKDxlUREWphCPwbcnW8/v4rfLt/Gv4w9h0mX5MS7JBGRBin0T5O786MX3uLJpVv48pVn8aUr1NdWRBKXQv80Pfzyu8z5vw1MviSHr19zTrzLERH5UAr90/DI/67nF395l5tD/fnu9UPU11ZEEp5C/yN6vHATD/7xba6/sA8Pfu5C9bUVkVZBof8RPPdmBd95fjWfOK83D39hGGkKfBFpJRT6TfTHVdv5+jMruPTsTH556wjaqZG5iLQiSqwmeHXdTv558TKGZ6cz67aQ+tqKSKuj0I9SYXkldz1eyuAzujF38ii6qK+tiLRCUYW+mY01s3VmVmZm959izM1mttbM1pjZExHza81sefC1pKF1E92yzbuZOq+YARmdWTAlnx6d2sW7JBGRj6TRw1UzSwNmAmOACqDYzJa4+9qIMYOAB4BL3X23mfWO+BaH3X1YjOtuMW9t38fkx4rJ7NqBRdMKyFRfWxFpxaI50s8Hyty93N2PAouBcfXG3AnMdPfdAO6+M7Zlxsf6XQe4bU4RndunsWhaAWd07xjvkkRETks0od8P2BLxuCKYF2kwMNjM/m5mhWY2NmJZRzMrCebfcJr1tpgtVYeYOLsIgIXTChiQob62ItL6RfNuZEMXoXsD32cQcAXQH/ibmV3g7nuAbHffZmZ5wCtmtsrd15/0BGbTgekA2dnZTfwRYm/HviNMmF3EweoaFk+/mLOyusa7JBGRmIjmSL8CiLxPcH9gWwNjfuvux9x9A7CO8IsA7r4t+LcceA0YXv8J3H2Wu4fcPZSVldXkHyKWKg9UM3F2EZUHqpk/JZ8hfbvHtR4RkViKJvSLgUFmlmtm7YHxQP2rcJ4HrgQws16ET/eUm1m6mXWImH8psJYEtfdwuK/t5qpDzJk8iuHZ6msrIsml0dM77l5jZncDLwJpwFx3X2NmM4ASd18SLLvGzNYCtcA33L3SzC4BHjGzOsIvMA9FXvWTSA5W1zBlXjHv7NjPrNtDjM7LjHdJIiIxZ+71T8/HVygU8pKSkhZ9ziPHapkyr5jC8kpm3jqCTw7t06LPLyJyusys1N1DjY1L+Y+Vhvvavsnr6yv52U0XKfBFJKml9G0Yauuc+55azstv7eSHN1zAjSP7x7skEZFmlbKhX1fnPPDcSn6/cjsPfPJcbhs9MN4liYg0u5QMfXdnxu/X8nRJBf981dl88fKz4l2SiEiLSMnQ/9mf32He6xuZelku940ZHO9yRERaTMqF/v+8VsYvXy3jlvwBfPtT56mvrYiklJQK/fmvb+THf1rHuGF9+dENQxX4IpJyUib0nynZwveWrGHMkDP46U0Xqa+tiKSklAj9F1Zu55u/WcnHBvXil7cOV19bEUlZSZ9+r7y9g3sXL2PkwHQeuW0kHdqqr62IpK6kDv3Xy97nroVvcl6f7syZPIrO7VP+A8gikuKSNvRLN+1m2oIScjLDfW27d1RfWxGRpAz91Vv3MvmxpfTu1oGFUwtI79I+3iWJiCSEpAv9sp37uX3uUrp1aMvCaQX0Vl9bEZETkir0N1ceYsLsItqYsejO0fRPV19QodELAAAHGElEQVRbEZFISRP67+09woQ5hVTX1LFwWj65vbrEuyQRkYSTNJezdOmQxuDe3bj3E4M490z1tRURaUjShH63ju2YM3lUvMsQEUloSXN6R0REGhdV6JvZWDNbZ2ZlZnb/KcbcbGZrzWyNmT0RMX+Smb0bfE2KVeEiItJ0jZ7eMbM0YCYwBqgAis1sibuvjRgzCHgAuNTdd5tZ72B+BvA9IAQ4UBqsuzv2P4qIiDQmmiP9fKDM3cvd/SiwGBhXb8ydwMzjYe7uO4P51wIvuXtVsOwlYGxsShcRkaaKJvT7AVsiHlcE8yINBgab2d/NrNDMxjZhXRERaSHRXL3T0I3nvYHvMwi4AugP/M3MLohyXcxsOjAdIDs7O4qSRETko4jmSL8CGBDxuD+wrYExv3X3Y+6+AVhH+EUgmnVx91nuHnL3UFZWVlPqFxGRJogm9IuBQWaWa2btgfHAknpjngeuBDCzXoRP95QDLwLXmFm6maUD1wTzREQkDho9vePuNWZ2N+GwTgPmuvsaM5sBlLj7Ev4R7muBWuAb7l4JYGY/JPzCATDD3as+7PlKS0vfN7NNH/1Hohfw/mms31xUV9OorqZRXU2TjHUNjGaQuX/gFHurZmYl7h6Kdx31qa6mUV1No7qaJpXr0idyRURSiEJfRCSFJGPoz4p3AaeguppGdTWN6mqalK0r6c7pi4jIqSXjkb6IiJxCqwn9xu70aWYdzOypYHmRmeVELHsgmL/OzK5t4bq+Gtx9dKWZ/cXMBkYsqzWz5cFX/c8+NHddk81sV8TzT4tY1mx3Ro2irocjanrHzPZELGvO7TXXzHaa2epTLDcz+0VQ90ozGxGxrDm3V2N1TQjqWWlmr5vZRRHLNprZqmB7lbRwXVeY2d6I/6/vRixr9K69zVjXNyJqWh3sUxnBsubcXgPM7FUze8vCdyK+t4ExLbOPuXvCfxH+fMB6IA9oD6wAhtQb8yXg18H0eOCpYHpIML4DkBt8n7QWrOtKoHMw/U/H6woeH4jj9poM/LKBdTMIf7AuA0gPptNbqq564+8h/LmQZt1ewff+ODACWH2K5dcBfyR8a5HRQFFzb68o67rk+PMBnzxeV/B4I9ArTtvrCuD3p7sPxLquemM/DbzSQturDzAimO4GvNPA72SL7GOt5Ug/mjt9jgPmB9PPAlebmQXzF7t7tYdvEVEWfL8WqcvdX3X3Q8HDQsK3omhu0WyvU2nOO6M2ta5bgCdj9Nwfyt3/CnzYBwfHAQs8rBDoaWZ9aOY7yTZWl7u/7v+4VXlL7V/RbK9TOZ19M9Z1teT+td3d3wym9wNv8cGbT7bIPtZaQj+au3WeGOPuNcBeIDPKdZuzrkhTCb+SH9fRzEosfGfSG2JUU1PqujH4M/JZMzt+j6SE2F7BabBc4JWI2c21vaJxqtoT6U6y9fcvB/5sZqUWvqlhS7vYzFaY2R/N7PxgXkJsLzPrTDg4fxMxu0W2l4VPPQ8HiuotapF9rLX0yI3mbp2nGhPVnT4/oqi/t5lNJNxM5vKI2dnuvs3M8oBXzGyVu69vobp+Bzzp7tVmdhfhv5KuinLd5qzruPHAs+5eGzGvubZXNOKxf0XNzK4kHPqXRcy+NNhevYGXzOzt4Ei4JbwJDHT3A2Z2HeH7cw0iQbYX4VM7f/eTbwvT7NvLzLoSfqH5irvvq7+4gVVivo+1liP9aO/0OQDAzNoCPQj/mRfVnT6bsS7M7BPAt4DPuHv18fnuvi34txx4jfCrf4vU5e6VEbU8CoyMdt3mrCvCeOr96d2M2ysap6q9ObdXVMzsQmA2MM6De17BSdtrJ/D/iN1pzUa5+z53PxBM/wFoZ+GbMcZ9ewU+bP9qlu1lZu0IB/4id3+ugSEts481x5sWsf4i/BdJOeE/94+/+XN+vTFf5uQ3cp8Ops/n5Ddyy4ndG7nR1DWc8BtXg+rNTwc6BNO9gHeJ0RtaUdbVJ2L6s0Ch/+NNow1BfenBdEZL1RWMO4fwm2rWEtsr4jlyOPUbk5/i5DfZljb39oqyrmzC71NdUm9+F6BbxPTrwNgWrOvM4/9/hMNzc7DtotoHmquuYPnxA8IuLbW9gp99AfDzDxnTIvtYzDZ0c38Rfmf7HcIB+q1g3gzCR88AHYFngl+ApUBexLrfCtZbB3yyhet6GdgBLA++lgTzLwFWBTv9KmBqC9f1ILAmeP5XgXMj1p0SbMcy4I6WrCt4/H3goXrrNff2ehLYDhwjfGQ1FbgLuCtYboR7Ra8Pnj/UQtursbpmA7sj9q+SYH5esK1WBP/P32rhuu6O2L8KiXhRamgfaKm6gjGTCV/cEblec2+vywifklkZ8X91XTz2MX0iV0QkhbSWc/oiIhIDCn0RkRSi0BcRSSEKfRGRFKLQFxFJIQp9EZEUotAXEUkhCn0RkRTy/wFBPjzno9qqpAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "# list all data in history\n",
    "print(history.history.keys())\n",
    "\n",
    "# summarize history for accuracy\n",
    "plt.plot(history.history['acc'])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot model loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 0, 'epoch')"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xl8VIW5//HPk4SEfUcgEQQVF0AIEDZttbbuC6kbi0vFqsD96e3tz1vvtddqW9veWrtYtVZAxbXKom3BFXetlS1gQNkXQSCAEVE2SUh47h9zEsc0wARyZjIz3/frlRcz55yZeXKY5DtzzuR5zN0REREByEh0ASIi0nAoFEREpJpCQUREqikURESkmkJBRESqKRRERKSaQkEkRmb2qJn9MsZt15rZGYd7PyLxplAQEZFqCgUREammUJCUEhy2udnMFpnZLjN72Mw6mtlLZrbDzF4zszZR2w8zs8Vm9rmZvWVmJ0at62dmC4LbTQEa13isC8ysOLjte2bW5xBrvt7MVpnZZ2Y2w8xyg+VmZneb2Sdm9kXwPfUO1p1nZkuC2jaa2Y8OaYeJ1KBQkFR0CXAmcBxwIfAS8D9AeyLP+R8AmNlxwNPAD4EOwIvAc2aWbWbZwN+BJ4C2wLTgfglu2x+YBIwF2gETgBlmllOXQs3s28CvgeFAZ2AdMDlYfRZwavB9tAZGAFuDdQ8DY929BdAbeKMujyuyPwoFSUX3ufsWd98I/AOY4+7vu3sZ8DegX7DdCOAFd3/V3fcCvwOaACcDQ4BGwB/dfa+7PwPMi3qM64EJ7j7H3Svd/TGgLLhdXVwBTHL3BUF9PwaGmlk3YC/QAjgBMHdf6u6bgtvtBXqaWUt33+buC+r4uCK1UihIKtoSdfnLWq43Dy7nEnllDoC77wPWA3nBuo3+9Y6R66IuHwX8Z3Do6HMz+xzoEtyuLmrWsJPIu4E8d38D+BNwP7DFzCaaWctg00uA84B1Zva2mQ2t4+OK1EqhIOmshMgvdyByDJ/IL/aNwCYgL1hWpWvU5fXAr9y9ddRXU3d/+jBraEbkcNRGAHe/190HAL2IHEa6OVg+z90LgSOIHOaaWsfHFamVQkHS2VTgfDP7jpk1Av6TyCGg94BZQAXwAzPLMrOLgUFRt30QGGdmg4MTws3M7Hwza1HHGp4CrjGz/OB8xP8SOdy11swGBvffCNgF7AEqg3MeV5hZq+Cw13ag8jD2g0g1hYKkLXdfDlwJ3Ad8SuSk9IXuXu7u5cDFwGhgG5HzD3+Num0RkfMKfwrWrwq2rWsNrwO3Ac8SeXdyDDAyWN2SSPhsI3KIaSuR8x4AVwFrzWw7MC74PkQOm2nIjoiIVNE7BRERqaZQEBGRagoFERGpplAQEZFqWYkuoK7at2/v3bp1S3QZIiJJZf78+Z+6e4eDbZd0odCtWzeKiooSXYaISFIxs3UH30qHj0REJIpCQUREqikURESkmkJBRESqKRRERKSaQkFERKopFEREpFrahMLaT3fxm5eXsW+fusKKiOxP2oTCK0s288Bbq7lpajEVlfsSXY6ISIOUdH/RfKjGnHoMeyud385czu7ySu67vB85WZmJLktEpEFJm3cKADecfiw/u7AnryzZwnWPFbG7vCLRJYmINChpFQoAo0/pzl2X9uGfqz7lew/PZfuevYkuSUSkwUi7UAAYXtCF+0b1p3j951z+4Gw+21We6JJERBqEtAwFgPP7dObB7xWwcstORkyYxZbtexJdkohIwqVtKACcfsIRPHrNIEo+/5LLxs9i/We7E12SiEhCpXUoAAw9ph1PXjeYL77cy2XjZ7Hqk52JLklEJGHSPhQA+nVtw+QxQ6jYt48RE2axpGR7oksSEUkIhULgxM4tmTp2KDlZGYycOIv567YluiQRkbgLNRTM7BwzW25mq8zslv1sM9zMlpjZYjN7Ksx6DuboDs2ZOm4obZplc9XDc3hv1aeJLEdEJO5CCwUzywTuB84FegKjzKxnjW16AD8GTnH3XsAPw6onVke2acq0sUPp0qYpox+dx+tLtyS6JBGRuAnzncIgYJW7r3H3cmAyUFhjm+uB+919G4C7fxJiPTE7omVjJo8ZwgmdWjD2ifk8t7Ak0SWJiMRFmKGQB6yPur4hWBbtOOA4M/unmc02s3NCrKdO2jTL5i/XDaZ/1zb8YPL7TJn3caJLEhEJXZihYLUsq9m3OgvoAXwLGAU8ZGat/+WOzMaYWZGZFZWWltZ7ofvTonEjHvv+IL7ZowP//ewHTHr3o7g9tohIIoQZChuALlHXjwRqHofZAEx3973u/hGwnEhIfI27T3T3Ancv6NChQ2gF16ZJdiYPfm8A5/TqxB3PL+G+11firpkMIpKawgyFeUAPM+tuZtnASGBGjW3+DpwOYGbtiRxOWhNiTYckJyuTP13ej4v75fH7V1dw50vLFAwikpJCm6fg7hVmdiMwE8gEJrn7YjO7Ayhy9xnBurPMbAlQCdzs7lvDqulwZGVm8LvL+tI0J5MJ76xhZ1kFvyjsTUZGbUfJRESSU6hDdtz9ReDFGstuj7rswE3BV4OXkWH8orA3zXMaMf7t1ewur+S3l/YhK1N/AygiqSFtJq/VFzPjlnNPoEXjrGCKWwX3jtIUNxFJDXqJe4huOP1YfnphT2Yujkxx+7K8MtEliYgcNoXCYbjmlO7cdUkwxW3SHE1xE5Gkp1A4TMMHduHeUf14/+PPueLBOZriJiJJTaFQDy7ok8vE7w1g+ZYdjJgwi080xU1EkpRCoZ58+4SOPHrNwMgUtwma4iYiyUmhUI9OPqY9T143mG27yhk+YRarSzXFTUSSi0KhnvXr2oYpY4eyt1JT3EQk+SgUQnBi55ZMGTuURpmRKW4LPtYUNxFJDgqFkBzToTlTx0amuF350BzeW60pbiLS8CkUQtSlbWSK25FtmjD6kXm8sUxT3ESkYVMohCwyxW0ox3dswZjH5/P8Ik1xE5GGS6EQB22bZfPU9cEUt6ffZ+q89Qe/kYhIAigU4qRqitspx7bnv55dxCP/1BQ3EWl4FApx1CQ7k4euLuDsXh35+XNL+NMbmuImIg2LQiHOcrIyuf/y/lzcL4/fvbKC37y8XMEgIg2G5ikkQNUUtybZmYx/ezU7y/ZyxzBNcRORxFMoJEhGhvHL7/ameU4WE95Zw+6ySu7SFDcRSTCFQgJFT3H73Ssr2F1eyT2j8jXFTUQSRi9LE8zMuPHbPbj9gp68vHgz1z8+X1PcRCRhFAoNxPe/EZni9o+VpVw9aS47NMVNRBJAodCADB/YhXtH9mPBx9u44qE5bNMUNxGJM4VCA3Nh31wmXDWAZZt3MGKipriJSHwpFBqg75zYkUdHD2TDtsgUtw3bNMVNROJDodBAnXzsV1PcLhs/izWa4iYicaBQaMD6d23D5DFDKa/Yx/AJs1i6SVPcRCRcCoUGrmduS6aOi0xxGzFhFu9ripuIhEihkASqpri1bhqZ4jZr9dZElyQiKUqhkCS6tG3KtHFDyW3dhNGPzOXNZZ8kuiQRSUGhhoKZnWNmy81slZndUsv60WZWambFwdd1YdaT7Dq2bMyUsUPp0bE51z9exAuLNiW6JBFJMaGFgpllAvcD5wI9gVFm1rOWTae4e37w9VBY9aSKyBS3IfTr2pp/f3oBU4s0xU1E6k+Y7xQGAavcfY27lwOTgcIQHy9ttIye4vbMIh7VFDcRqSdhhkIeEP0ydkOwrKZLzGyRmT1jZl1quyMzG2NmRWZWVFpaGkatSadpdlb1FLefPbeE+99cleiSRCQFhBkKtU2MqTli7Dmgm7v3AV4DHqvtjtx9orsXuHtBhw4d6rnM5FU1xe2ifnn8duZyfvPyMk1xE5HDEuY8hQ1A9Cv/I4GS6A3cPfqzlQ8CvwmxnpSUlZnB7y/rS9PsTB54azW7yir42YW9NMVNRA5JmKEwD+hhZt2BjcBI4PLoDcyss7tXfYRmGLA0xHpSVtUUt2Y5WUx8Zw07yyq46xJNcRORugstFNy9wsxuBGYCmcAkd19sZncARe4+A/iBmQ0DKoDPgNFh1ZPqzIwfn3sCzXOy+MOrK9hdpiluIlJ3lmzHoAsKCryoqCjRZTRoD7/7Eb94fgmnHdeB8VcOoEm2gkEk3ZnZfHcvONh2Or6Qgq79Rnd+c8lJvLOylKsf0RQ3EYmdQiFFjRjYNTLFbd02rtQUNxGJkUIhhV3YN5fxVw5g6eYdjJw4m092aIqbiByYQiHFndGzI4+MHsj6bbsZPl5T3ETkwBQKaeCUY9vzxLWD2bqrnOGa4iYiB6BQSBMDjmrD5DFDKKvYx/AJszXFTURqpVBII71yWzFl7FCyMoyRE2dTvP7zRJckIg2MQiHNHHtEc6aNG0qrJo244sHZzF6jKW4i8hWFQhqKnuJ29aS5vLlcU9xEJEKhkKaip7iNebyIFz/QFDcRUSiktaopbn2PbM2NTy1gmqa4iaQ9hUKaa9m4EY9fG5nidvMzi3jsvbWJLklEEkihINVT3M7q2ZGfzlisKW4iaUyhIEAwxe2K/nw3P1dT3ETSWJhDdiTJNMrM4A/D82mak8UDb61md1kFP9UUN5G0olCQr8nIMH713d40y87kwX98xK7ySu68+CRNcRNJEwoF+Rdmxv+cdyLNcxpx92sr2F1ewR9H9CM7S8EgkuoUClIrM+M/zuhBs5xMfvnCUnaVFWmKm0ga0Es/OaDrvnk0v75YU9xE0oVCQQ5q1KCu3BM1xe3z3ZriJpKqFAoSk2FRU9xGTNAUN5FUpVCQmFVNcfv4s92MmDCbjZ9/meiSRKSeKRSkTk45tj1PXjeIT3eWMXz8LD76dFeiSxKReqRQkDobcFRbnr5+CF/ureSy8bNYtllT3ERShUJBDknvvFZMHTuErAxjxARNcRNJFQoFOWTHHtGCaeOG0rJJlqa4iaQIhYIcli5tmzJt7Ml0Dqa4vaUpbiJJTaEgh61Tq8ZMGTOEY49ozvWPF/GSpriJJC2FgtSLds1zeOr6IfQ5sjU3PLWAZ+ZvSHRJInIIQg0FMzvHzJab2Sozu+UA211qZm5mBWHWI+Fq1aQRT1w7iKHHtONH0xby+Ky1iS5JROootFAws0zgfuBcoCcwysx61rJdC+AHwJywapH4aZqdxcNXD+TMnh25ffpi/vyWpriJJJMw3ykMAla5+xp3LwcmA4W1bPcL4C5AfRNSRONGmfz5iv4U5udy18vLuUtT3ESSRpihkAesj7q+IVhWzcz6AV3c/fkD3ZGZjTGzIjMrKi0trf9Kpd5VTXEbNagrf35rNT9/bgn79ikYRBq6MOcp1DbDsfq3gpllAHcDow92R+4+EZgIUFBQoN8sSSIzw/jfi3rTPCcyxW1nWQW/uaQPmRrvKdJghRkKG4AuUdePBEqirrcAegNvmRlAJ2CGmQ1z96IQ65I4qpri1iwniz++tpIvyyu5e0S+priJNFBhhsI8oIeZdQc2AiOBy6tWuvsXQPuq62b2FvAjBULqMTN+eMZxNM/J4pcvLGV3eQUPXDmAxo00xU2koYnp5ZqZ/YeZtbSIh81sgZmddaDbuHsFcCMwE1gKTHX3xWZ2h5kNO/zSJdlc982j+d+LTuKtFaVcPWkuO8sqEl2SiNRgsXwqxMwWuntfMzsbuAG4DXjE3fuHXWBNBQUFXlSkNxPJbHrxRm6aupDeea147JqBtG6aneiSRFKemc1394P+LVisB3arzgyeRyQMFlL7iWSRgyrMz4tMcSvZzsiJsyndUZbokkQkEGsozDezV4iEwszgD872hVeWpLoze3Zk0uiBrNu6m+ETZmmKm0gDEWsoXAvcAgx0991AI+Ca0KqStPCNHl+f4rZWU9xEEi7WUBgKLHf3z83sSuAnwBfhlSXp4mtT3CbMYvnmHYkuSSStxRoKDwC7zawv8F/AOuDx0KqStNI7rxVTxgwhw2DExFks1BQ3kYSJNRQqPPIxpULgHne/h8gfn4nUix4dWzBt7Mm0aJzFFQ/NYY6muIkkRKyhsMPMfgxcBbwQdEBtFF5Zko66totMcevYMoerH9EUN5FEiDUURgBlwPfdfTORxna/Da0qSVudWjVm6tihHN1eU9xEEiGmUAiC4C9AKzO7ANjj7jqnIKFo1zyHp8cM4aS8Vtzw1AKe1RQ3kbiJtc3FcGAucBkwHJhjZpeGWZikt8gUt8EMPaYd/zltIU/MWpvokkTSQqwN8W4l8jcKnwCYWQfgNeCZsAoTaZYTmeJ241MLuG36YnaVVzLutGMSXZZISov1nEJGVSAEttbhtiKHrHGjTB64cgDD+uZy50vL+N3M5ZriJhKiWN8pvGxmM4Gng+sjgBfDKUnk6xplZnD3iHya5WTypzdXsbOsgtsv6EmGhvWI1LuYQsHdbzazS4BTiDTCm+jufwu1MpEokSluJ9EsO4uH3v2IXWUV3KkpbiL1LuYhO+7+LPBsiLWIHJCZcev5kSlu97y+kt2a4iZS7w4YCma2g6i5ytGrAHf3lqFUJbIfZsb/PzMyxe1XLy7ly72V/PmK/priJlJPDvgSy91buHvLWr5aKBAkka4/NTLF7c3ln3DNI/M0xU2knuh9tyStywd35e7h+cxd+xlXPjSHz3eXJ7okkaSnUJCk9t1+eTxwRX+WaIqbSL1QKEjSO6tXJx4eXcC6rbsZMWEWJZriJnLIFAqSEr7ZowNPXDuI0h1lXKYpbiKHTKEgKaOgW1ueHjOE3eUVmuImcogUCpJSeue1YurYoRiRKW6LNmiKm0hdKBQk5fTo2IJnxp1M85wsLn9wDnM/+izRJYkkDYWCpKSu7ZoybdxQOrbM4XuT5vD2itJElySSFBQKkrI6t2rClGCK23WPzePlDzXFTeRgFAqS0tp/bYrb+/x1gaa4iRyIQkFSXtUUt8Hd23LT1IU8MXtdoksSabAUCpIWmuVkMWn0QM448Qhu+/uHTHh7daJLEmmQQg0FMzvHzJab2Sozu6WW9ePM7AMzKzazd82sZ5j1SHqrmuJ2QZ/O/PqlZfz+FU1xE6kp5nkKdWVmmcD9wJnABmCemc1w9yVRmz3l7uOD7YcBfwDOCasmkUaZGdwzsh/Nc7K4743IFLfbztcUN5EqoYUCMAhY5e5rAMxsMlAIVIeCu2+P2r4Ztc9uEKlXmRnGry8+iWY5WTwcTHH79cWa4iYC4YZCHrA+6voGYHDNjczsBuAmIBv4dm13ZGZjgDEAXbt2rfdCJf2YGT8Jprjd+/pKdpVXcvdwTXETCfMnoLaXXf/yTsDd73f3Y4D/Bn5S2x25+0R3L3D3gg4dOtRzmZKuzIybzjyO/znvBF5YtIlxT85nz97KRJclklBhhsIGoEvU9SOBkgNsPxn4boj1iNRqzKnH8KuLeldPcdulKW6SxsIMhXlADzPrbmbZwEhgRvQGZtYj6ur5wMoQ6xHZrysGH8UfhveNTHF7eA5f7N6b6JJEEiK0UHD3CuBGYCawFJjq7ovN7I7gk0YAN5rZYjMrJnJe4eqw6hE5mIv6Hcn9l/dn8cbtjHxwNp/u1BQ3ST+WbJ/TLigo8KKiokSXISnsnRWljHmiiNxWTXjyusHktm6S6JJEDpuZzXf3goNtp49aiNRw6nEdeOLawdVT3NZt1RQ3SR8KBZFaDOzWlqeuD6a4jZ/Fii2a4ibpQaEgsh8nHdmKKWOHAjBiwiw+2PBFgisSCZ9CQeQAjuvYgmnjhtI0O4vLH5zNvLWa4iapTaEgchBHtWvGM/82lA4tc7jq4Tn8Y6WmuEnqUiiIxKBzqyZMHTuU7u2bc+2jRbz84eZElyQSCoWCSIzaN89h8vVD6JXXkhueWsDf3tcUN0k9CgWROmjVtBFPRk1xe1JT3CTFKBRE6qhqitu3jz+Cn2iKm6QYhYLIIWjcKJPxV301xe0PmuImKSLMeQoiKa1qilvT7EzufWMVO8oq+K+zT6BJdmaiSxM5ZAoFkcOQmWHceXEfmuVk8cg/1zJ13nrO7tWJYfm5nHJsexpl6s24JBeFgshhysgwbr+gJ2f36sT04o28sGgTf31/I+2aZXN+n84U5ufSv2sbzDTuUxo+dUkVqWdlFZW8s+JTphdv5LWlW9izdx95rZswLD+XwvxcTujUMtElShqKtUuqQkEkRDvLKnh1yWamF5fwj5WfUrnPOb5jC4bl5zKsby5d2jZNdImSJhQKIg3M1p1lvPjBJqYXl1C0bhsAA45qQ2F+Lued1Jn2zXMSXKGkMoWCSAO2/rPdPLeohBnFJSzbvIPMDOMbx7anMD+Xs3p1onmOTvdJ/VIoiCSJ5Zt3MGPhRqYXl7Bh25fkZGVwRs+OFPbN5bTjO5CTpY+4yuFTKIgkGXdnwcefM6N4I88v2sTWXeW0bJzFeSd1ZljfXAYf3Y7MDH2CSQ6NQkEkiVVU7uOfq7cyvXgjMz/czK7ySo5okcOFfSOfYDopr5U+4ip1olAQSRF79lby+tJPmF68kbeWl1JeuY/u7ZsxLAiIozs0T3SJkgQUCiIp6Ivde3l5ceQTTLPWbMUdTsprRWF+Lhf0yaVTq8aJLlEaKIWCSIrbsn0Pzy0sYcbCEhZt+AIzGNK9HYX5uZzbuzOtmjZKdInSgCgURNLIR5/uYkZxCdOLN7Lm0100yjS+dfwRFObn8p0TOqpJnygURNKRu7O4ZDvTizcyY2EJW7aX0Sw7k7N7deLC/Fy+oSZ9aUuhIJLmKvc5cz/6jBkLI036tu+poG2zbM4/6asmfRn6iGvaUCiISDU16ROFgojUSk360pNCQUQOSk360keDCAUzOwe4B8gEHnL3O2usvwm4DqgASoHvu/u6A92nQkEkHBu27ea5hZuYXrxRTfpSUMJDwcwygRXAmcAGYB4wyt2XRG1zOjDH3Xeb2b8B33L3EQe6X4WCSPj216RvWN9cvqUmfUkp1lAIM/oHAavcfU1Q0GSgEKgOBXd/M2r72cCVIdYjIjE6vlMLbu50Aj866/ivNel7YdEmWjbO4tzekU8wqUlf6gkzFPKA9VHXNwCDD7D9tcBLta0wszHAGICuXbvWV30ichBmxoCj2jDgqDbcdkHP6iZ9zy8qYUrRejXpS0FhhkJtz45aj1WZ2ZVAAXBabevdfSIwESKHj+qrQBGJXVZmBqcd14HTjuvAnou+atL3xKx1PPzuR9VN+obl53KMmvQlrTBDYQPQJer6kUBJzY3M7AzgVuA0dy8LsR4RqSeNG2Vyfp/OnN+nc3WTvhkLS7j3jZXc8/pKNelLYmGeaM4icqL5O8BGIieaL3f3xVHb9AOeAc5x95Wx3K9ONIs0XFu27+H5RZuYUbyRhWrS16Ak/NNHQRHnAX8k8pHUSe7+KzO7Ayhy9xlm9hpwErApuMnH7j7sQPepUBBJDvtr0jesby5nnKgmffHWIEIhDAoFkeSyvyZ9Z/XqxDA16YsbhYKINDjRTfpe/GAzX3y5V0364kShICINmpr0xZdCQUSSRlWTvhnFJbyjJn2hUCiISFLaurOMFz/czIzijcxbG2nS179rawrz8zi/j5r0HSqFgogkvdqa9J1ybHsK++ZyVq+OtGisj7jGSqEgIiml1iZ9J3ZkWL6a9MVCoSAiKcndv9akb+uuclo0zuI8Nek7IIWCiKS8isp91U36Zn64mV3llWrStx8KBRFJK3v2Rpr0zVi4kTeXlVJeuU9N+qIoFEQkbX3x5V5mfriZ6Qs38t7qrbhD77yWFPbN48K+6dmkT6EgIkLtTfoGd29LYX4e5/buROum2YkuMS4UCiIiNVQ36Vu4kTWlkSZ9px13BIX5qd+kT6EgIrIftTXpa5qdydkp3KRPoSAiEoN0adKnUBARqaOqJn0zFpbw6pLNKdWkT6EgInIYdpVV8OqSLUwv3ljdpO+4js0pzM9LyiZ9CgURkXqSCk36FAoiIiFI1iZ9CgURkZAlU5M+hYKISJwkQ5M+hYKISAJUVO7jvdVbmV5cwszFm9lZVlHdpG9Y31z6HJmYJn0KBRGRBNuzt5I3ln3C9OKvmvR1a9eUYfl5FMa5SZ9CQUSkAUl0kz6FgohIA5WIJn0KBRGRJBCvJn0KBRGRJBLdpO+5hZvYvH1PvTbpUyiIiCSpffucuWs/Y3pxCS9+sKm6Sd9PL+xJYX7eId1nrKGQdUj3LiIiocnIMIYc3Y4hR7fj58N68c6KUqYvLCG3dZPwHzvMOzezc8xsuZmtMrNball/qpktMLMKM7s0zFpERJJRdlYGZ/TsyH2j+jGwW9vQHy+0UDCzTOB+4FygJzDKzHrW2OxjYDTwVFh1iIhI7MI8fDQIWOXuawDMbDJQCCyp2sDd1wbr9oVYh4iIxCjMw0d5wPqo6xuCZSIi0kCFGQq1Nfc4pI86mdkYMysys6LS0tLDLEtERPYnzFDYAHSJun4kUHIod+TuE929wN0LOnToUC/FiYjIvwozFOYBPcysu5llAyOBGSE+noiIHKbQQsHdK4AbgZnAUmCquy82szvMbBiAmQ00sw3AZcAEM1scVj0iInJwof7xmru/CLxYY9ntUZfnETmsJCIiDUDStbkws1Jg3SHevD3waT2WU19UV92orrprqLWprro5nLqOcveDnpRNulA4HGZWFEvvj3hTXXWjuuquodamuuomHnWF2uZCRESSi0JBRESqpVsoTEx0AfuhuupGddVdQ61NddVN6HWl1TkFERE5sHR7pyAiIgegUBARkWopEwoxDPTJMbMpwfo5ZtYtat2Pg+XLzezsONd1k5ktMbNFZva6mR0Vta7SzIqDr3ptERJDXaPNrDTq8a+LWne1ma0Mvq6Oc113R9W0wsw+j1oX5v6aZGafmNmH+1lvZnZvUPciM+sftS6U/RVDTVcEtSwys/fMrG/UurVm9kGwr+p9vm0MtX3LzL6I+v+6PWrdAZ8DIdd1c1RNHwbPqbbBulD2mZl1MbM3zWypmS02s/+oZZv4Pb/cPem/gExgNXA0kA0sBHrW2Ob/AeODyyOBKcHlnsH2OUD34H4y41jX6UDT4PK/VdUVXN+ZwP01GvhTLbdtC6wJ/m0TXG4Tr7pqbP/vwKSw91dw36cC/YEP97P+POAlIt2BhwBz4rC/DlbTyVXnFboNAAAFmklEQVSPRWTY1ZyodWuB9gncX98Cnj/c50B911Vj2wuBN8LeZ0BnoH9wuQWwopafx7g9v1LlnUL1QB93LweqBvpEKwQeCy4/A3zHzCxYPtndy9z9I2BVcH9xqcvd33T33cHV2cSn7Ucs+2t/zgZedffP3H0b8CpwToLqGgU8XU+PfUDu/g7w2QE2KQQe94jZQGsz60yI++tgNbn7e8FjQvyeW1WPfbD9tT+H89ys77ri8vxy903uviC4vINIr7ias2fi9vxKlVCIZaBP9TYeadb3BdAuxtuGWVe0a4m8GqjS2CJzJGab2Xfrqaa61HVJ8Fb1GTOraoPeIPZXcJitO/BG1OKw9lcs9ld7Qxk2VfO55cArZjbfzMYkoB6AoWa20MxeMrNewbIGsb/MrCmRX67PRi0OfZ9Z5LB2P2BOjVVxe36F2hAvjmIZ6LO/beptGFAtYr5vM7sSKABOi1rc1d1LzOxo4A0z+8DdV8eprueAp929zMzGEXmX9e0YbxtmXVVGAs+4e2XUsrD2VywS8fyKiZmdTiQUvhG1+JRgXx0BvGpmy4JX0fGygEgvnp1mdh7wd6AHDWB/BS4E/unu0e8qQt1nZtacSAj90N2311xdy01CeX6lyjuFWAb6VG9jZllAKyJvI+ttGNAh1oWZnQHcCgxz97Kq5e5eEvy7BniLyCuIuNTl7lujankQGBDrbcOsK8pIary1D3F/xWJ/tYe5vw7KzPoADwGF7r61annUvvoE+Bv1d8g0Ju6+3d13BpdfBBqZWXsSvL+iHOj5Ve/7zMwaEQmEv7j7X2vZJH7Pr/o+aZKILyLveNYQOZxQdXKqV41tbuDrJ5qnBpd78fUTzWuovxPNsdTVj8iJtR41lrcBcoLL7YGV1NMJtxjr6hx1+SJgtn91YuujoL42weW28aor2O54Iif9LB77K+oxurH/E6fn8/UTgXPD3l8x1NSVyDmyk2ssbwa0iLr8HnBOfe6rGGrrVPX/R+SX68fBvovpORBWXcH6qheMzeKxz4Lv+3HgjwfYJm7Pr3p9EiTyi8jZ+RVEfsHeGiy7g8irb4DGwLTgh2QucHTUbW8NbrccODfOdb0GbAGKg68ZwfKTgQ+CH4oPgGvjXNevgcXB478JnBB12+8H+3EVcE086wqu/wy4s8btwt5fTwObgL1EXp1dC4wDxgXrDbg/qPsDoCDs/RVDTQ8B26KeW0XB8qOD/bQw+D++tT73VYy13Rj1/JpNVHDV9hyIV13BNqOJfPgk+nah7TMih/UcWBT1f3Veop5fanMhIiLVUuWcgoiI1AOFgoiIVFMoiIhINYWCiIhUUyiIiEg1hYJIHAXdQZ9PdB0i+6NQEBGRagoFkVqY2ZVmNjfonT/BzDLNbKeZ/d7MFlhk9kWHYNv8oAnfIjP7m5m1CZYfa2avBU3fFpjZMcHdNw+aDC4zs78E3XpFGgSFgkgNZnYiMIJIA7R8oBK4gkh7gwXu3h94G/hpcJPHgf929z5E/tq0avlfgPvdvS+Rv7jeFCzvB/yQyCyPo4FTQv+mRGKUKl1SRerTd4g0AJwXvIhvAnwC7AOmBNs8CfzVzFoBrd397WD5Y8A0M2sB5Ln73wDcfQ9AcH9z3X1DcL2YSC+ed8P/tkQOTqEg8q8MeMzdf/y1hWa31djuQD1iDnRIqCzqciX6OZQGRIePRP7V68ClQd98zKxtMNQnA7g02OZy4F13/wLYZmbfDJZfBbztkX74G6qG/VhkRnjTuH4XIodAr1BEanD3JWb2EyJTtjKIdNS8AdgF9DKz+UQm940IbnI1MD74pb8GuCZYfhUwwczuCO7jsjh+GyKHRF1SRWJkZjvdvXmi6xAJkw4fiYhINb1TEBGRanqnICIi1RQKIiJSTaEgIiLVFAoiIlJNoSAiItX+D3y4U4Oq33KKAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# summarize history for loss\n",
    "plt.plot(history.history['loss'])\n",
    "plt.title('model loss')\n",
    "plt.ylabel('loss')\n",
    "plt.xlabel('epoch')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### TODO:\n",
    "\n",
    "Generate plots of validation accuracy and loss. Are we underfitting or overfitting?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### TODO:\n",
    "\n",
    "I’m sure we can do better. Take a few minutes and experiment with the model and see if you can improve its performance on test accuracy.\n",
    "\n",
    "You can try adusting any of the hyperparameters: epochs, batch_size, embedding_size, input_length, top_words, the number of nodes (currently 250) in the dense layer, adding another hidden layer, or adding dropout or regularization.\n",
    "\n",
    "```\n",
    "from keras import regularizers\n",
    "model.add(Dense(64, input_dim=64,\n",
    "                kernel_regularizer=regularizers.l2(0.01),\n",
    "                activity_regularizer=regularizers.l1(0.01)))\n",
    "```\n",
    "\n",
    "https://keras.io/regularizers/\n",
    "\n",
    "https://keras.io/layers/core/\n",
    "\n",
    "https://chrisalbon.com/deep_learning/keras/adding_dropout/\n",
    "\n",
    "```\n",
    "model.add(layers.Dropout(0.2))\n",
    "```\n",
    "\n",
    "Goldstar to the best model. Report your results below."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## STOP, take a deep breath and wait for instruction before proceeding."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### One-Dimensional Convolutional Neural Network Model \n",
    "\n",
    "Convolutional neural networks were designed to honor the spatial structure in image data while being robust to the position and orientation of learned objects in the scene.\n",
    "\n",
    "This same principle can be used on sequences, such as the one-dimensional sequence of words in a movie review. The same properties that make the CNN model attractive for learning to recognize objects in images can help to learn structure in paragraphs of words, namely the techniques invariance to the specific position of features.\n",
    "\n",
    "Keras supports one-dimensional convolutions and pooling by the Conv1D and MaxPooling1D classes respectively.\n",
    "\n",
    "Again, let’s import the classes and functions needed for this example and initialize our random number generator to a constant value so that we can easily reproduce results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# CNN for the IMDB problem\n",
    "import numpy\n",
    "from keras.datasets import imdb\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "from keras.layers import Flatten\n",
    "from keras.layers.convolutional import Conv1D\n",
    "from keras.layers.convolutional import MaxPooling1D\n",
    "from keras.layers.embeddings import Embedding\n",
    "from keras.preprocessing import sequence\n",
    "from keras import layers\n",
    "# fix random seed for reproducibility\n",
    "seed = 42\n",
    "numpy.random.seed(seed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also load and prepare our IMDB dataset as we did before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the dataset but only keep the top n words, zero the rest\n",
    "top_words = 5000\n",
    "(X_train, y_train), (X_test, y_test) = imdb.load_data(num_words=top_words)\n",
    "# pad dataset to a maximum review length in words\n",
    "input_length = 500\n",
    "X_train = sequence.pad_sequences(X_train, maxlen=input_length)\n",
    "X_test = sequence.pad_sequences(X_test, maxlen=input_length)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now define our convolutional neural network model. This time, after the Embedding input layer, we insert a `Conv1D` layer. This convolutional layer has `32` feature maps and reads embedded word representations with a filter size (kernel_size) equal to 3. I.e., 3 vector elements of the word embedding at a time.\n",
    "\n",
    "The convolutional layer is followed by a 1D max pooling layer with a size of 2 that halves the size of the feature maps from the convolutional layer. The rest of the network is the same as the neural network above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_2 (Embedding)      (None, 500, 32)           160000    \n",
      "_________________________________________________________________\n",
      "conv1d_1 (Conv1D)            (None, 500, 32)           3104      \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 500, 32)           0         \n",
      "_________________________________________________________________\n",
      "max_pooling1d_1 (MaxPooling1 (None, 250, 32)           0         \n",
      "_________________________________________________________________\n",
      "flatten_2 (Flatten)          (None, 8000)              0         \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 250)               2000250   \n",
      "_________________________________________________________________\n",
      "dense_4 (Dense)              (None, 1)                 251       \n",
      "=================================================================\n",
      "Total params: 2,163,605\n",
      "Trainable params: 2,163,605\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "# create the model\n",
    "model = Sequential()\n",
    "model.add(Embedding(top_words, 32, input_length=input_length))\n",
    "model.add(Conv1D(filters=32, kernel_size=3, padding='same', activation='relu'))\n",
    "# Add a dropout layer for previous hidden layer\n",
    "model.add(layers.Dropout(0.2))\n",
    "model.add(MaxPooling1D(pool_size=2))\n",
    "# Add additional convolution layer\n",
    "#model.add(Conv1D(filters=64, kernel_size=3, padding='same', activation='relu'))\n",
    "# model.add(layers.Dropout(0.2))\n",
    "# model.add(MaxPooling1D(pool_size=2))\n",
    "model.add(Flatten())\n",
    "model.add(Dense(250, activation='relu'))\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "print(model.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also fit the network the same as before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 22500 samples, validate on 2500 samples\n",
      "Epoch 1/3\n",
      "22500/22500 [==============================] - 17s 760us/step - loss: 0.5361 - acc: 0.6932 - val_loss: 0.3044 - val_acc: 0.8760\n",
      "Epoch 2/3\n",
      "22500/22500 [==============================] - 16s 692us/step - loss: 0.2351 - acc: 0.9069 - val_loss: 0.2693 - val_acc: 0.8888\n",
      "Epoch 3/3\n",
      "22500/22500 [==============================] - 15s 676us/step - loss: 0.1796 - acc: 0.9312 - val_loss: 0.2849 - val_acc: 0.8840\n",
      "Accuracy: 88.48%\n"
     ]
    }
   ],
   "source": [
    "# Fit the model\n",
    "history = model.fit(X_train, y_train,\n",
    "                    batch_size=batch_size,\n",
    "                    epochs=epochs,\n",
    "                    verbose=1,\n",
    "                    validation_split=0.1)\n",
    "# Final evaluation of the model\n",
    "scores = model.evaluate(X_test, y_test, verbose=0)\n",
    "print(\"Accuracy: %.2f%%\" % (scores[1]*100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['val_loss', 'val_acc', 'loss', 'acc'])\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x121277400>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3XmcFPWd//HXZ4aBmeGcAVROQYMGTQzgiBo10TVRwHht9qcoZMEkojGHySbZaIzRuJe7mxg3iVc0RI0KEhOPGBPBiGYTLwZFPCKCrDIDHtz3zMDM5/dHVUN1z9E9MNXd0/1+Ph79oLrqW9WfLnrq09+qT33b3B0REZGOlOQ6ABERyX9KFiIikpaShYiIpKVkISIiaSlZiIhIWkoWIiKSlpKFCGBmd5rZv2bY9m0z+1TcMYnkEyULERFJS8lCpICYWY9cxyCFSclCuo3w9M+3zWypmW03s1+Y2YFm9gcz22pmT5hZVaT9WWb2mpltMrOnzGxsZNl4M3sxXO9+oDzltT5jZkvCdZ8xs6MyjPEMM3vJzLaYWZ2ZXZuy/MRwe5vC5TPD+RVm9iMze8fMNpvZX8J5J5tZfRv74VPh9LVm9oCZ3WNmW4CZZjbRzJ4NX+NdM/uZmfWMrH+kmS0wsw1m9r6ZfdfMDjKzHWY2MNLuaDNba2Zlmbx3KWxKFtLdfBb4NHAYcCbwB+C7wCCCz/PXAMzsMGAO8HVgMPAY8Dsz6xkeOB8CfgVUA78Ot0u47gRgNnAJMBC4DXjEzHplEN924B+BAcAZwJfM7JxwuyPDeH8axjQOWBKu90PgaODjYUz/DLRkuE/OBh4IX/NeoBn4RrhPjgdOBS4LY+gLPAH8ERgKfAj4k7u/BzwFnBfZ7nRgrrvvyjAOKWBKFtLd/NTd33f31cD/As+7+0vu3gg8CIwP250P/N7dF4QHux8CFQQH4+OAMuBGd9/l7g8AiyKvcTFwm7s/7+7N7n4X0Biu1yF3f8rdX3H3FndfSpCwPhkungY84e5zwtdd7+5LzKwE+DxwubuvDl/zmfA9ZeJZd38ofM2d7r7Y3Z9z993u/jZBskvE8BngPXf/kbs3uPtWd38+XHYXQYLAzEqBCwgSqoiShXQ770emd7bxvE84PRR4J7HA3VuAOmBYuGy1J4+i+U5k+mDgm+FpnE1mtgkYEa7XITM71swWhqdvNgOXEnzDJ9zGW22sNojgNFhbyzJRlxLDYWb2qJm9F56a+vcMYgB4GDjCzA4h6L1tdvcX9jEmKTBKFlKo1hAc9AEwMyM4UK4G3gWGhfMSRkam64B/c/cBkUelu8/J4HXvAx4BRrh7f+BWIPE6dcChbayzDmhoZ9l2oDLyPkoJTmFFpQ4dfQvwBjDG3fsRnKZLFwPu3gDMI+gBfQ71KiRCyUIK1TzgDDM7NbxA+02CU0nPAM8Cu4GvmVkPM/t7YGJk3duBS8NegplZ7/DCdd8MXrcvsMHdG8xsInBhZNm9wKfM7LzwdQea2biw1zMbuMHMhppZqZkdH14jeRMoD1+/DPgekO7aSV9gC7DNzD4MfCmy7FHgIDP7upn1MrO+ZnZsZPndwEzgLOCeDN6vFAklCylI7r6M4Pz7Twm+uZ8JnOnuTe7eBPw9wUFxI8H1jd9G1q0luG7xs3D5irBtJi4DrjOzrcD3CZJWYrurgCkEiWsDwcXtj4WLvwW8QnDtZAPwn0CJu28Ot3kHQa9oO5BUHdWGbxEkqa0Eie/+SAxbCU4xnQm8BywHToks/yvBhfUXw+sdIgCYfvxIRKLM7EngPne/I9exSP5QshCRPczsGGABwTWXrbmOR/KHTkOJCABmdhfBPRhfV6KQVOpZiIhIWupZiIhIWgUz6NigQYN81KhRuQ5DRKRbWbx48Tp3T713p5WCSRajRo2itrY212GIiHQrZvZO+lY6DSUiIhlQshARkbSULEREJK2CuWbRll27dlFfX09DQ0OuQ4ldeXk5w4cPp6xMv1MjIl2voJNFfX09ffv2ZdSoUSQPMFpY3J3169dTX1/P6NGjcx2OiBSggj4N1dDQwMCBAws6UQCYGQMHDiyKHpSI5EZBJwug4BNFQrG8TxHJjYI+DSUiUoi2Ne6mbsMOVm3YQd2GHVT27MGFx45Mv+J+ULKI2aZNm7jvvvu47LLLOrXelClTuO+++xgwYEBMkYlIvmpucd7dvJNVG3ZQvyH4d1UkOazf3pTUfsLIAUoW3d2mTZu4+eabWyWL5uZmSktL213vscceizs0EcmhLQ27WLV+x54eQjQZrN60k13Newd57VFiDB1QwcjqSk478iBGVlcmPfpXxl8FqWQRsyuuuIK33nqLcePGUVZWRp8+fRgyZAhLlizh9ddf55xzzqGuro6GhgYuv/xyZs2aBewdvmTbtm1MnjyZE088kWeeeYZhw4bx8MMPU1FRkeN3JiId2dXcwrubGlolglUbdlC3cQebduxKal9VWcbI6ko+Mqw/Uz46ZE8iGFFdyZD+5fQoze0l5qJJFj/43Wu8vmZLl27ziKH9uObMIztsc/311/Pqq6+yZMkSnnrqKc444wxeffXVPSWus2fPprq6mp07d3LMMcfw2c9+loEDByZtY/ny5cyZM4fbb7+d8847j9/85jdMnz69S9+LiHSOu7Npx649B/9oMli1YQdrNjXQ3LK3d1BWaoyoCg7+HxvRPykZjKiupF95ft8jVTTJIl9MnDgx6V6In/zkJzz44IMA1NXVsXz58lbJYvTo0YwbNw6Ao48+mrfffjtr8YoUs6bdLazetDO5Z7B+7/TWxt1J7Qf16cmI6komjKzinHFBEkgkhQP7lVNa0n2rFosmWaTrAWRL796990w/9dRTPPHEEzz77LNUVlZy8sknt3mvRK9evfZMl5aWsnPnzqzEKlLo3J3125taJYJVG3ZQv3EnazbvJPr7cL16lOxJABNHVyclg+FVFfTuVbiH1MJ9Z3mib9++bN3a9i9Ubt68maqqKiorK3njjTd47rnnshydSOFr2NVM/cYd1LVRVbRqww52NDUntT+wXy9GVFVybDQZDAz+HdynFyXduHewP5QsYjZw4EBOOOEEPvKRj1BRUcGBBx64Z9mkSZO49dZbOeqoozj88MM57rjjchipSPfk7qzd2piUCKLJ4P0tjUntK8pK91wr+PihgxhZXbEnGQyvqqS8rP0qxWJWML/BXVNT46k/fvS3v/2NsWPH5iii7Cu29yvFY0fT7qSeQfSGtLqNO2jY1bKnrRkM6VeedIoocRF5ZHUlg/r01IgHEWa22N1r0rVTz0JEcq6lxXlvS8PeBJDUS9jJum3JvYM+vXoworqSQwb35uTDB+9JCCOrKxlWVUGvHuoddDUlCxHJiq0Nu/b0DupSThfVb9xJU/Pe3kGJsecmtE+NPSCplzCyupIBlWXqHWSZkoWIdIndzS28u7mh9R3JG3dSt2EHG1KGqOhfEdyENnZIP0478iBGVFfsSQZDB1RQluOb0CSZkoWIZGzzjl17bkBLvX6weuNOdrckD1ExrCpIAJM+kjxExYiq7AxRIV1HyUJE9tjV3MKaTW2XmK5av4MtDck3oVX3Dm5CO2r4AD5z1JCkawcH9cv9EBXSdZQsRIqIu7MxHKKi1R3JG3ewZtNOIp0DepaWMLy6ghFVlYwfUZWUDEZUV9A3z4eokK6jZBGzfR2iHODGG29k1qxZVFZWxhCZFKrG3c3Uh9cJUquK6jbsYFurISp6MbK6gpqDqxg5fljSjWgH9i0v2pvQJJmSRczaG6I8EzfeeCPTp09XspAk7s66bU2tqooSz9/b0tBqiIrEtYJjR1en3HtQQWVPHQYkPX1KYhYdovzTn/40BxxwAPPmzaOxsZFzzz2XH/zgB2zfvp3zzjuP+vp6mpubufrqq3n//fdZs2YNp5xyCoMGDWLhwoW5fiuSRQ27mttMBInS0527Wg9RMbK6kuMPHdjqtw4G9+2lMlPZb8WTLP5wBbz3Stdu86CPwuTrO2wSHaJ8/vz5PPDAA7zwwgu4O2eddRZ//vOfWbt2LUOHDuX3v/89EIwZ1b9/f2644QYWLlzIoEGDujZuybmWFueDcIiKutQLyRt28MHW5JvQKnuW7jk1dOKYQUm9g+FVFRqiQmJXPMkiD8yfP5/58+czfvx4ALZt28by5cs56aST+Na3vsV3vvMdPvOZz3DSSSflOFLpSq+u3sxzK9cn34i2cSdNu5OHqBjav4IR1RVJdyQnrh8M7K0hKiS3iidZpOkBZIO7c+WVV3LJJZe0WrZ48WIee+wxrrzySk477TS+//3v5yBC6WrPvrWef5z9PLuanb7hEBVjDujLqWMPTLoreeiAcg1RIXmteJJFjkSHKD/99NO5+uqrmTZtGn369GH16tWUlZWxe/duqqurmT59On369OHOO+9MWlenobqnt9Zu49J7FnPwwN7c+8VjOUDXDqQbU7KIWXSI8smTJ3PhhRdy/PHHA9CnTx/uueceVqxYwbe//W1KSkooKyvjlltuAWDWrFlMnjyZIUOG6AJ3N7N+WyMX/XIRPUqMX848hgP7lec6JJH9EusQ5WY2CfgfoBS4w92vT1l+MDAbGAxsAKa7e324bAbwvbDpv7r7XR29loYoL773m68adjUz7Y7neXX1ZubMOo4JI6tyHZJIuzIdojy2e/HNrBS4CZgMHAFcYGZHpDT7IXC3ux8FXAf8R7huNXANcCwwEbjGzPQXJ3mvpcX59gNLWfzORm44b5wShRSMOAdumQiscPeV7t4EzAXOTmlzBPCncHphZPnpwAJ33+DuG4EFwKQYYxXpEj9+4k1+9/Ia/nnS4Zxx1JBchyPSZeJMFsOAusjz+nBe1MvAZ8Ppc4G+ZjYww3Uxs1lmVmtmtWvXrm0ziEL5JcB0iuV95rMHFtfz0ydXcH7NCL70yUNzHY5Il4ozWbRV9pF6RPsW8Ekzewn4JLAa2J3hurj7z929xt1rBg8e3GqF8vJy1q9fX/AHUndn/fr1lJfrImquPPvWeq787VJO+NBA/vXcj6jqSQpOnNVQ9cCIyPPhwJpoA3dfA/w9gJn1AT7r7pvNrB44OWXdpzobwPDhw6mvr6e9XkchKS8vZ/jw4bkOoyit+GAbl/yqloMH9ubmaUfrR3ukIMWZLBYBY8xsNEGPYSpwYbSBmQ0CNrh7C3AlQWUUwOPAv0cuap8WLu+UsrIyRo8evY/hi6S3flsjn79zEWWlJfxy5jH0r9CQ3VKYYvsK5O67ga8QHPj/Bsxz99fM7DozOytsdjKwzMzeBA4E/i1cdwPwLwQJZxFwXThPJG807Gpm1q8W8/6WBm6fUcOIao0OLIUr1vsssqmt+yxE4tLS4lx+/xJ+9/IabrpwgiqfpNvK+X0WIoVMJbJSbJQsRDpJJbJSjDQ2lEgnZFQi6w6NW6FhE+zcGDxamqFHLyjtBaVl4XTPvf9Gp0s0+qzkHyULkfa0NEPD5j0H/DXvruHh37/A5X0b+eLBAyib/9DeZBB9NGyClt3pt98eK207iZT2hB49g4TTI0w6pb3CeT0j09GElJKcottL2kbqdMrrlfaEUh0uipn+96Xw7W5s46C+qe0DfXR54+akzQwFrgdoAP4K9OoPFQOgoip49B+e/DzxKB8Q9BZ2N0JzU/Boa3p3IzTvgubGlOmw3Z7pcFnDlr3T7W2j9b2s+85K2klO0STUQXLqVLLr2Xbia/V6PaGkR/DrURIrJQvpHtyhaVtmB/pIb4CdG2HXjva3a6XhQT08yPc5AAYfnnSwb+rZn/96+n2WrjOuOf8EjjxkFJT3z/9v2u5B72hP8kkkp0TCiUy3SkhtJKfdja2TU1vJrmkb7FifkuxSXs9b0sefMUufnJIS1f4ku070ykrLCiqJ5fmnXQpO0qmdNg76DR0kgo5O7ZT2gsrqvQf5qlEwdNzeb/ap3/YTj159O/yDbmlxvnn/En73Xl9unjaBIz/ajSqfzIKEVtoDevbOdTTJWpr3Jp89yamtHlcGySnTnljTDmje2EZPLLJdb+7a99mpRJVuuoNeWe/BMPK4ro09hZKF7Jvdja0P9h0d6BPJoWEzHZ4a6dVv77f88gFwwBHtH+ijPYKyiljeZrREdkp3ShT5rqQUelYCeXYjY0tz+6cJk3pi6ZJTU/pe2Z5/G4K/i456YumugQ2rgYv/1HGb/aRkUczcoWl7Bgf5RC8g0m7X9va3ayXJ3+YrB8HAMe0f6Pec2+8ffGPKE7+urVOJbLEpKYWSiti+fOyzlpYOThM2Bb2LmClZFIKWluBibLsXcDs6tbOr/e2W9oSKyKmdASNhyMfCg3wHp3Z69oWS7n0LzzNvreO7D76iUWQlP5SUQEk5lOVuZGkli3yyu6mNb/npLuaG3/g7OrXTs294IO8f/HvAh9s/0Ed7BGUVBXWBLlMrPtjGpb9arFFkRSKULLqae1B9k7Y0M3qOP/y3aVsHG7bkb/OV1TDw0LYP8qmnevLo1E6+0yiyIm1TsmhPSws0bungQm0HyaC5qf3tlpQlV+30Gw4HfrT98/iJeb36d/tTO/kuOorsnFnHaRRZkQglix0b4A//3HYy6KgWvKx38sF80GHpL+BWVEFZZVGe2sl3LS3Ot379Movf2cjN0yYwYWRV+pVEioiSRUkPqK+N1OeP7vjibeKUT4/4qw8ke25Y8CaPLn2X70z6sEpkRdqgZFHeDy5fkusoJId+XVvHzxYGJbKXfvKQXIcjkpd0ElyK2jNvrePK36pEViQdJQspWokS2dGDVCIrko7+OqQord/WyEV3vkDPHiXMVomsSFq6ZiFFJ1Ei+8GWRuaqRFYkI0oWUlRSS2THq0RWJCM6DSVFRSWyIvtGyUKKRqJEduoxKpEV6SwlCykKiRLZEz80iH85RyWyIp2lZCEFL1oie9O0CSqRFdkH+quRgqYSWZGuoWooKVgNu5q5+O5alciKdAElCylIiRLZF1dtUomsSBfQaSgpSCqRFelaShZScOapRFakyylZSEF5ZsU6vqsSWZEup2QhBWPFB9u49B6VyIrEQX9NUhBUIisSr1iThZlNMrNlZrbCzK5oY/lIM1toZi+Z2VIzmxLOH2VmO81sSfi4Nc44pXuLlsje/o81KpEViUFspbNmVgrcBHwaqAcWmdkj7v56pNn3gHnufouZHQE8BowKl73l7uPiik8Kg0pkRbIjzp7FRGCFu6909yZgLnB2ShsH+oXT/YE1McYjBehHC5bx6NJ3uWKySmRF4hRnshgG1EWe14fzoq4FpptZPUGv4quRZaPD01NPm9lJbb2Amc0ys1ozq127dm0Xhi7dwbzaOm5a+BZTjxnBJZ9QiaxInOJMFm3VLHrK8wuAO919ODAF+JWZlQDvAiPdfTzwT8B9ZtYvZV3c/efuXuPuNYMHD+7i8CWfqURWJLviTBb1wIjI8+G0Ps30BWAegLs/C5QDg9y90d3Xh/MXA28Bh8UYq3Qj0RLZm6erRFYkG+L8K1sEjDGz0WbWE5gKPJLSZhVwKoCZjSVIFmvNbHB4gRwzOwQYA6yMMVbpJlJLZPuVq0RWJBtiq4Zy991m9hXgcaAUmO3ur5nZdUCtuz8CfBO43cy+QXCKaqa7u5l9ArjOzHYDzcCl7r4hrlile9AosiK5Y+6plxG6p5qaGq+trc11GBKTlhbna3Nf4tGl73LLtAlMVuWTSJcws8XuXpOunU72SrcQLZFVohDJPiULyXsqkRXJPSULyWsqkRXJD0oWkrdWfLBVJbIieUJ/fZKX1m1r5KI7F6lEViRPZJQszOw3ZnZGeHe1SKwadjUzS6PIiuSVTA/+twAXAsvN7Hoz+3CMMUkRa2lxvhmOInvj+eM0iqxInsgoWbj7E+4+DZgAvA0sMLNnzOwiM9P5AekyP1qwjN+rRFYk72R8WsnMBgIzgS8CLwH/Q5A8FsQSmRSdRInsBRNVIiuSbzIa7sPMfgt8GPgVcKa7vxsuut/MdNu07LdEiexJYwZx3dkqkRXJN5mODfUzd3+yrQWZ3CYu0pEVH2zlkrBE9qZpKpEVyUeZ/lWONbMBiSdmVmVml8UUkxSRRIlsL5XIiuS1TJPFxe6+KfHE3TcCF8cTkhSL6Ciyd8w4RiWyInks02RRYpGTyOFvTfSMJyQpBokS2ZfCEtlxIwakX0lEcibTaxaPA/PM7FaC3524FPhjbFFJwUuUyF6pElmRbiHTZPEd4BLgSwS/rT0fuCOuoKSwRUtkZ6lEVqRbyChZuHsLwV3ct8QbjhQ6lciKdE+Z3mcxBvgP4AiC38kGwN31tVAyliiRPWSwSmRFuptM/1p/SdCr2A2cAtxNcIOeSEaiJbK/mKESWZHuJtNkUeHufyL4ze533P1a4O/iC0sKiUpkRbq/TC9wN4TDky83s68Aq4ED4gtLCkW0RPaWaRNUIivSTWXas/g6UAl8DTgamA7MiCsoKRw/nK8SWZFCkLZnEd6Ad567fxvYBlwUe1RSEOYtquPmp1QiK1II0vYs3L0ZONpU4yid8NcV6/jugyqRFSkUmV6zeAl42Mx+DWxPzHT338YSlXRrKz7YyqUqkRUpKJkmi2pgPckVUA4oWUiSddsamfnLRfTqUaoSWZECkukd3LpOIWklSmTXbWtk7qzjVSIrUkAyvYP7lwQ9iSTu/vkuj0i6pZYW55vzXmZJ3SZuvlAlsiKFJtPTUI9GpsuBc4E1XR+OdFc/nL+M37+iElmRQpXpaajfRJ+b2RzgiVgikm5nb4nsSJXIihSofS1TGQOM7MpApHtKLpE9UiWyIgUq02sWW0m+ZvEewW9cSBFTiaxI8cjor9vd+7p7v8jjsNRTU20xs0lmtszMVpjZFW0sH2lmC83sJTNbamZTIsuuDNdbZmand+5tSdyiJbKzZ6pEVqTQZZQszOxcM+sfeT7AzM5Js04pcBMwmeB3MC4wsyNSmn0PmOfu44GpwM3hukeEz48EJgE3h9uTPBAtkb1jRg3Dq1QiK1LoMj1vcI27b048cfdNwDVp1pkIrHD3le7eBMwFzk5p40C/cLo/eyuszgbmunuju/8fsCLcnuRYtET2xvPHqURWpEhkmizaapfuescwoC7yvD6cF3UtMN3M6oHHgK92Yl3MbJaZ1ZpZ7dq1a9OEI10hUSJ7xaQPM+kjKpEVKRaZJotaM7vBzA41s0PM7MfA4jTrtFUWk3pj3wXAne4+HJgC/Cr83YxM1sXdf+7uNe5eM3jw4AzehuwPlciKFK9Mk8VXgSbgfmAesBP4cpp16oERkefDaX0j3xfC7eHuzxLc8Dcow3Uli1QiK1LcMr0pbzvQqpopjUXAGDMbTfDLelOBC1ParAJOBe40s7EEyWIt8Ahwn5ndAAwluK/jhU6+vnSR5e+rRFak2GVaDbXAzAZEnleZ2eMdrePuu4GvAI8DfyOoenrNzK4zs7PCZt8ELjazl4E5wEwPvEbQ43gd+CPw5fB3NSTL1m5t5KI7VSIrUuzMvdWlgNaNzF4Ky1s7nJdLNTU1Xltbm+swCkrDrmam/vw53nhvC3NnHa/KJ5ECZGaL3b0mXbtMzye0mNme4T3MbBRtXHCWwpEokX25XiWyIpL5qLNXAX8xs6fD558AZsUTkuSD/w5LZL87RSWyIpL5Be4/mlkNQYJYAjxMUBElBej+Rau4JSyRvfgklciKSOYDCX4RuJyghHUJcBzwLMk/syoF4K8r1nHVg6+qRFZEkmR6zeJy4BjgHXc/BRhPUOIqBUQlsiLSnkyPBg3u3gBgZr3c/Q3g8PjCkmxTiayIdCTTC9z14X0WDwELzGwjuqO6YERHkb1/1vEaRVZEWsn0Ave54eS1ZraQYITYP8YWlWRNtET2lmlH8zGVyIpIGzLtWezh7k+nbyXdRXKJ7EG5DkdE8pSuYBaxRInshceqRFZEOqZkUaSiJbI/OEslsiLSMSWLIpQokT10cB+VyIpIRnSUKDLREtlfzKxRiayIZETJoohES2R/MaNGJbIikrFOV0NJ99TS4vzTvCUqkRWRfaKeRZH47/nLeOyV97hyskpkRaTzlCyKwNwXVCIrIvtHyaLA/WX5Or73kEpkRWT/KFkUsOXvb+VL96pEVkT2n44eBUolsiLSlZQsCpBKZEWkq6l0tsCoRFZE4qCeRYH5r8eDEtnvTh6rElkR6TJKFgVk7guruPXpoET2iyeNznU4IlJAlCwKxF+Wr+Oqh17lE4cN5jqVyIpIF1OyKACJEtkPDe7DTReOp4dKZEWki+mo0s0lSmTLy0qZfdEx9FWJrIjEQMmiG0stkR02oCLXIYlIgVLpbDeVWiJ71HCVyIpIfNSz6KZUIisi2aRk0Q2pRFZEsk3JoptRiayI5IKSRTfy5vtb+dI9KpEVkeyL9WhjZpPMbJmZrTCzK9pY/mMzWxI+3jSzTZFlzZFlj8QZZ3ewdmsjF/1yEeU9VSIrItkXWzWUmZUCNwGfBuqBRWb2iLu/nmjj7t+ItP8qMD6yiZ3uPi6u+LqTnU3NfPHuWtZvb2TeJcerRFZEsi7OnsVEYIW7r3T3JmAucHYH7S8A5sQYT7eUKJFdWr+JG88frxJZEcmJOJPFMKAu8rw+nNeKmR0MjAaejMwuN7NaM3vOzM5pZ71ZYZvatWvXdlXceeW/Hl/GH15ViayI5FacyaKtMh1vp+1U4AF3b47MG+nuNcCFwI1mdmirjbn/3N1r3L1m8ODB+x9xnpkTlshOU4msiORYnMmiHhgReT4cWNNO26mknIJy9zXhvyuBp0i+nlHw/nf5Wr4Xlsj+QCWyIpJjcSaLRcAYMxttZj0JEkKrqiYzOxyoAp6NzKsys17h9CDgBOD11HUL1Zvvb+Wye15kzAEqkRWR/BBbNZS77zazrwCPA6XAbHd/zcyuA2rdPZE4LgDmunv0FNVY4DYzayFIaNdHq6gKWbRE9hczVSIrIvnBko/R3VdNTY3X1tbmOoz9srOpmam3P8ey97Yw75LjVfkkIrEzs8Xh9eEOadTZPBEtkb11ukaRFZH8opPheSJRInvVlLGcfqRKZEUkvyhZ5IFoiewXTlSJrIjkHyWLHEuUyH5SJbIikseULHIoWiL7M5XIikjNQrl2AAAKjklEQVQe09EpR1QiKyLdiZJFDkRHkf3FjBqNIisieU+ls1mmElkR6Y7Us8iy/3z8DZXIiki3o2SRRXNeWMVtT69UiayIdDtKFlmiElkR6c6ULLJAJbIi0t3pqBUzlciKSCFQsoiRSmRFpFCodDYmKpEVkUKinkVMVCIrIoVEySIGiRLZ6cepRFZECoOSRReLlshee6ZKZEWkMChZdKFl76lEVkQKk45mXWTt1kY+f6dKZEWkMClZdIFEieyG7U3MnnGMSmRFpOCodHY/RUtkb5t+NB8d3j/XIYmIdDn1LPZTtET2NJXIikiBUrLYDyqRFZFioWSxj1QiKyLFRMliH6hEVkSKjY5ynfTB1gaVyIpI0VGy6ISdTc1cfJdKZEWk+Kh0NkMtLc437l/C0tWbVSIrIkVHPYsM/ecf3+CPr6lEVkSKk5JFBu57fhW3/VklsiJSvJQs0vjzm2u5+mGVyIpIcYs1WZjZJDNbZmYrzOyKNpb/2MyWhI83zWxTZNkMM1sePmbEGWd7lr23lS/fqxJZEZHYLnCbWSlwE/BpoB5YZGaPuPvriTbu/o1I+68C48PpauAaoAZwYHG47sa44k2VKJGt6FnKbJXIikiRi/Or8kRghbuvdPcmYC5wdgftLwDmhNOnAwvcfUOYIBYAk2KMNUm0RPYXM45hqEpkRaTIxZkshgF1kef14bxWzOxgYDTwZGfWNbNZZlZrZrVr167tkqCjJbL/M3WcSmRFRIg3WbR1JdjbaTsVeMDdmzuzrrv/3N1r3L1m8ODB+xhmMpXIioi0FmeyqAdGRJ4PB9a003Yqe09BdXbdLpMokf3ccQerRFZEJCLOZLEIGGNmo82sJ0FCeCS1kZkdDlQBz0ZmPw6cZmZVZlYFnBbOi02iRPbkwwdzzZlHqERWRCQitmood99tZl8hOMiXArPd/TUzuw6odfdE4rgAmOvuHll3g5n9C0HCAbjO3TfEFWu0RPanF6hEVkQklUWO0d1aTU2N19bWdnq9D7Y2cO5Nz7CruYWHvnyCKp9EpKiY2WJ3r0nXrugHEuxVWsrYIX25/NTDlChERNpR9Mmif2UZd8w4JtdhiIjkNZ2cFxGRtJQsREQkLSULERFJS8lCRETSUrIQEZG0lCxERCQtJQsREUlLyUJERNIqmOE+zGwt8M5+bGIQsK6LwulKiqtzFFfnKK7OKcS4Dnb3tL/xUDDJYn+ZWW0m46Nkm+LqHMXVOYqrc4o5Lp2GEhGRtJQsREQkLSWLvX6e6wDaobg6R3F1juLqnKKNS9csREQkLfUsREQkLSULERFJq+CThZlNMrNlZrbCzK5oY3kvM7s/XP68mY2KLLsynL/MzE7Pclz/ZGavm9lSM/uTmR0cWdZsZkvCxyOp68Yc10wzWxt5/S9Gls0ws+XhY0aW4/pxJKY3zWxTZFmc+2u2mX1gZq+2s9zM7Cdh3EvNbEJkWZz7K11c08J4lprZM2b2sciyt83slXB/df63ivcvrpPNbHPk/+v7kWUdfgZijuvbkZheDT9T1eGyOPfXCDNbaGZ/M7PXzOzyNtpk5zPm7gX7AEqBt4BDgJ7Ay8ARKW0uA24Np6cC94fTR4TtewGjw+2UZjGuU4DKcPpLibjC59tyuL9mAj9rY91qYGX4b1U4XZWtuFLafxWYHff+Crf9CWAC8Go7y6cAfwAMOA54Pu79lWFcH0+8HjA5EVf4/G1gUI7218nAo/v7GejquFLangk8maX9NQSYEE73Bd5s428yK5+xQu9ZTARWuPtKd28C5gJnp7Q5G7grnH4AONXMLJw/190b3f3/gBXh9rISl7svdPcd4dPngOFd9Nr7FVcHTgcWuPsGd98ILAAm5SiuC4A5XfTaHXL3PwMbOmhyNnC3B54DBpjZEOLdX2njcvdnwteF7H2+Mtlf7dmfz2ZXx5XNz9e77v5iOL0V+BswLKVZVj5jhZ4shgF1kef1tN7Re9q4+25gMzAww3XjjCvqCwTfHBLKzazWzJ4zs3O6KKbOxPXZsLv7gJmN6OS6ccZFeLpuNPBkZHZc+ysT7cUe5/7qrNTPlwPzzWyxmc3KQTzHm9nLZvYHMzsynJcX+8vMKgkOuL+JzM7K/rLgFPl44PmURVn5jPXY1xW7CWtjXmqtcHttMll3X2W8bTObDtQAn4zMHunua8zsEOBJM3vF3d/KUly/A+a4e6OZXUrQK/u7DNeNM66EqcAD7t4cmRfX/spELj5fGTOzUwiSxYmR2SeE++sAYIGZvRF+886GFwnGKtpmZlOAh4Ax5Mn+IjgF9Vd3j/ZCYt9fZtaHIEF93d23pC5uY5Uu/4wVes+iHhgReT4cWNNeGzPrAfQn6I5msm6ccWFmnwKuAs5y98bEfHdfE/67EniK4NtGVuJy9/WRWG4Hjs503TjjiphKyimCGPdXJtqLPc79lREzOwq4Azjb3dcn5kf21wfAg3Td6de03H2Lu28Lpx8DysxsEHmwv0Idfb5i2V9mVkaQKO5199+20SQ7n7E4Lsrky4Og57SS4LRE4qLYkSltvkzyBe554fSRJF/gXknXXeDOJK7xBBf0xqTMrwJ6hdODgOV00YW+DOMaEpk+F3jO915M+78wvqpwujpbcYXtDie42GjZ2F+R1xhF+xdszyD54uMLce+vDOMaSXAd7uMp83sDfSPTzwCTshjXQYn/P4KD7qpw32X0GYgrrnB54otk72ztr/C93w3c2EGbrHzGumxH5+uDoFLgTYID71XhvOsIvq0DlAO/Dv9wXgAOiax7VbjeMmByluN6AngfWBI+Hgnnfxx4JfxjeQX4Qpbj+g/gtfD1FwIfjqz7+XA/rgAuymZc4fNrgetT1ot7f80B3gV2EXyT+wJwKXBpuNyAm8K4XwFqsrS/0sV1B7Ax8vmqDecfEu6rl8P/56uyHNdXIp+v54gks7Y+A9mKK2wzk6DoJbpe3PvrRIJTR0sj/1dTcvEZ03AfIiKSVqFfsxARkS6gZCEiImkpWYiISFpKFiIikpaShYiIpKVkIZIHwtFWH811HCLtUbIQEZG0lCxEOsHMppvZC+FvF9xmZqVmts3MfmRmL1rw2yODw7bjwsELl5rZg2ZWFc7/kJk9EQ6W96KZHRpuvk84OOMbZnZvOPqxSF5QshDJkJmNBc4nGDhuHNAMTCMY5uFFd58APA1cE65yN/Addz+K4M7axPx7gZvc/WMEd5i/G84fD3yd4LdUDgFOiP1NiWSo0EedFelKpxIMnLgo/NJfAXwAtAD3h23uAX5rZv2BAe7+dDj/LuDXZtYXGObuDwK4ewNAuL0X3L0+fL6EYKyiv8T/tkTSU7IQyZwBd7n7lUkzza5OadfRGDodnVpqjEw3o79PySM6DSWSuT8B/xD+bgFmVh3+2FIJ8A9hmwuBv7j7ZmCjmZ0Uzv8c8LQHv0VQn/gRJgt+A74yq+9CZB/om4tIhtz9dTP7HsGvopUQjFD6ZWA7cKSZLSb4pcXzw1VmALeGyWAlcFE4/3PAbWZ2XbiN/5fFtyGyTzTqrMh+MrNt7t4n13GIxEmnoUREJC31LEREJC31LEREJC0lCxERSUvJQkRE0lKyEBGRtJQsREQkrf8P6ymoI0ErygkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(history.history.keys())\n",
    "# summarize history for accuracy\n",
    "plt.plot(history.history['acc'])\n",
    "plt.plot(history.history['val_acc'])\n",
    "plt.title('model accuracy')\n",
    "plt.ylabel('accuracy')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'test'], loc='upper left')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x121204f98>"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xl8VOW9+PHPN8skZCcLWwIEEFQ2WQJubdXWBbUFq1atGwgVaWttb21v9Wrtre29tfumVVGjUqtWbe2PtlqXKrW9FiEgsimyCJKAQBKy78n398c5CZOQZCbLyUwm3/frNa+cOed5Zr5zGM53zvM85zmiqhhjjDHdiQp1AMYYY8KfJQtjjDEBWbIwxhgTkCULY4wxAVmyMMYYE5AlC2OMMQFZsjCmH4jIYyLy/SDL7hWRc/v6OsYMJEsWxhhjArJkYYwxJiBLFmbIcJt/vikim0WkWkQeEZGRIvKiiFSKyKsiMtyv/EIR2SYiZSKyRkRO9ts2W0Q2uvV+D8R3eK9Pi8gmt+6bIjKzlzHfKCK7RKRURFaLyBh3vYjIz0XksIiUu59purvtIhHZ7sZWJCLf6NUOM8aPJQsz1FwGnAdMAT4DvAj8F5CJ8//hFgARmQI8BXwNyAJeAP4sIj4R8QF/An4LpAPPuq+LW3cOkA/cBGQADwKrRSSuJ4GKyCeBHwBXAKOBfcDT7ubzgU+4nyMNuBIocbc9AtykqsnAdOC1nryvMZ2xZGGGml+r6iFVLQL+Cbylqm+raj3wPDDbLXcl8FdVfUVVG4GfAMOAM4DTgFjgF6raqKrPAev93uNG4EFVfUtVm1X1caDerdcT1wD5qrrRje924HQRyQUagWTgJEBU9V1VPejWawSmikiKqh5V1Y09fF9jjmPJwgw1h/yWazt5nuQuj8H5JQ+AqrYA+4Fsd1uRtp+Fc5/f8njgVrcJqkxEyoCxbr2e6BhDFc7ZQ7aqvgbcC9wHHBKRlSKS4ha9DLgI2Cci/xCR03v4vsYcx5KFMZ07gHPQB5w+ApwDfhFwEMh217Ua57e8H/gfVU3zeySo6lN9jCERp1mrCEBVf6Wqc4FpOM1R33TXr1fVRcAInOayZ3r4vsYcx5KFMZ17BrhYRD4lIrHArThNSW8C/waagFtEJEZELgXm+9V9CFghIqe6HdGJInKxiCT3MIYngRtEZJbb3/G/OM1me0Vknvv6sUA1UAc0u30q14hIqtt8VgE092E/GANYsjCmU6q6A7gW+DVQjNMZ/hlVbVDVBuBSYAlwFKd/449+dQtw+i3udbfvcsv2NIa/A98G/oBzNjMJuMrdnIKTlI7iNFWV4PSrAFwH7BWRCmCF+zmM6ROxmx8ZY4wJxM4sjDHGBGTJwhhjTECWLIwxxgTkabIQkQUissOdruC2TrYvEZEj7rQIm0TkC37bmv3Wr/YyTmOMMd3zrINbRKKB93GmVijEucL186q63a/MEiBPVW/upH6VqiZ1XN+VzMxMzc3N7WvYxhgzpGzYsKFYVbMClYvxMIb5wC5V3QMgIk8Di4Dt3dbqpdzcXAoKCrx4aWOMiVgisi9wKW+bobJxrmRtVeiu6+gyd8bM50RkrN/6eBEpEJG1InKJh3EaY4wJwMtkIZ2s69jm9WcgV1VnAq8Cj/ttG6eqecDVwC9EZNJxbyCy3E0oBUeOHOmvuI0xxnTgZbIoxJlLp1UOzlw3bVS1xJ1NE5yrUef6bTvg/t0DrOHYbKD+9Veqap6q5mVlBWxyM8YY00te9lmsByaLyAScic+uwjlLaCMio/2mVV4IvOuuHw7UqGq9iGQCZwI/6mkAjY2NFBYWUldX14ePMTjEx8eTk5NDbGxsqEMxxkQgz5KFqjaJyM3AS0A0zrz820TkbqBAVVfjTMS2EGdStlKOzZ9zMvCgiLTgnP3c4z+KKliFhYUkJyeTm5tL+wlCI4uqUlJSQmFhIRMmTAh1OMaYCOTlmQWq+gLOHcb8193lt3w7zg1dOtZ7E5jR1/evq6uL+EQBICJkZGRg/TbGGK9E/BXckZ4oWg2Vz2mMCY2ITxaBqCoHy2tpaGoJdSjGGBO2hnyyaGhqobS6gQ+Kq2lq7v+EUVZWxm9+85se17vooosoKyvr93iMMaY3hnyyiIuNJjcjkcbmFvaW1NDc0r/Tn3SVLJqbu7952QsvvEBaWlq/xmKMMb015JMFQGJcDOPSE6htaGJfSTUt/Thf1m233cbu3buZNWsW8+bN45xzzuHqq69mxgyn//6SSy5h7ty5TJs2jZUrV7bVy83Npbi4mL1793LyySdz4403Mm3aNM4//3xqa2v7LT5jjAmGp6Ohwsl3/7yN7Qcqui3T1NJCfWMLMdFRxMUEzqNTx6Twnc9M67bMPffcw9atW9m0aRNr1qzh4osvZuvWrW1DXPPz80lPT6e2tpZ58+Zx2WWXkZGR0e41du7cyVNPPcVDDz3EFVdcwR/+8AeuvdbulGmMGThDJlkEIyYqCo1x+jEE8AWRMHpq/vz57a6F+NWvfsXzzz8PwP79+9m5c+dxyWLChAnMmjULgLlz57J3795+j8sYY7ozZJJFoDOAVs7oqDqKq+oZlRLPiJT4fo0jMTGxbXnNmjW8+uqr/Pvf/yYhIYGzzz6706vN4+Li2pajo6OtGcoYM+CGTLIIlogwOjWephblo4o6YqKF9MS4wBW7kJycTGVlZafbysvLGT58OAkJCbz33nusXbu21+9jjDFesmTRCREhZ/gwmluUoqO1REdFkTqsd3MuZWRkcOaZZzJ9+nSGDRvGyJEj27YtWLCABx54gJkzZ3LiiSdy2mmn9ddHMMaYfuXZnfIGWl5enna8+dG7777LySef3OvXbG5RPiiupraxmQmZiSTFhXdu7evnNcYMPSKywb0dRLds6Gw3oqOE3IwEfNFR7Cuuprah+2sjjDEmUlmyCCAmOooJmYlERQkfFFdT32QJwxgz9FiyCIIvxkkYitMs1ejBtCDGGBPOLFkEKd6dFqSpWdlbXE1ziyUMY8zQYcmiBxLjYhiXkUBdYwv7Smpo6ed5pIwxJlxZsuihlPhYctKHUVXfxP6jNUTKaDJjjOmOJYteGJ7gY3TqMMprGykqq+02YfR2inKAX/ziF9TU1PQ2TGOM6TeWLHopKzmOrOQ4SqsbOFRZ32U5SxbGmEjg6VVmIrIA+CUQDTysqvd02L4E+DFQ5K66V1UfdrctBu50139fVR/3MtbeGJUST3OzcriijpgoITPp+GlB/KcoP++88xgxYgTPPPMM9fX1fPazn+W73/0u1dXVXHHFFRQWFtLc3My3v/1tDh06xIEDBzjnnHPIzMzk9ddfD8EnNMYYh2fJQkSigfuA84BCYL2IrFbV7R2K/l5Vb+5QNx34DpAHKLDBrXu01wG9eBt8tKXX1Tsjo2aQveAHNLUoB8pqiYkS0hJ87cr4T1H+8ssv89xzz7Fu3TpUlYULF/LGG29w5MgRxowZw1//+lfAmTMqNTWVn/3sZ7z++utkZmb2a9zGGNNTXjZDzQd2qeoeVW0AngYWBVn3AuAVVS11E8QrwAKP4uwTEWFcegKJvhj2H62lsq6xy7Ivv/wyL7/8MrNnz2bOnDm899577Ny5kxkzZvDqq6/yrW99i3/+85+kpqYO4CcwxpjAvGyGygb2+z0vBE7tpNxlIvIJ4H3gP1R1fxd1sztWFJHlwHKAcePGdR/Nhfd0v70PoqKE8RkJ7CmuZl9JDROzEknwHb9rVZXbb7+dm2666bhtGzZs4IUXXuD222/n/PPP56677vIsXmOM6Skvzyykk3Udhw39GchV1ZnAq0Brv0QwdVHVlaqap6p5WVlZfQq2r1qnBYmJEvYW11Df6EwL4j9F+QUXXEB+fj5VVVUAFBUVcfjwYQ4cOEBCQgLXXnst3/jGN9i4ceNxdY0xJpS8PLMoBMb6Pc8BDvgXUNUSv6cPAT/0q3t2h7pr+j3CfhbrJozdR6r5oLiaSSOS2k1RfuGFF3L11Vdz+umnA5CUlMQTTzzBrl27+OY3v0lUVBSxsbHcf//9ACxfvpwLL7yQ0aNHWwe3MSakPJuiXERicJqWPoUz2mk9cLWqbvMrM1pVD7rLnwW+paqnuR3cG4A5btGNwFxVLe3q/byYory3ahqa2HOkGl9MFBMzE4mJHpgRyjZFuTGmp4KdotyzMwtVbRKRm4GXcIbO5qvqNhG5GyhQ1dXALSKyEGgCSoElbt1SEfkeToIBuLu7RBFuEnwxjM9IYG9JDftKatpmrTXGmMHK0+ssVPUF4IUO6+7yW74duL2LuvlAvpfxeSk5Ppaxw4fxYWkNH5bWMD4jARFLGMaYwSnir+AO5dxNaQk+stOGUVHXSOHR7qcF6Subo8oY46WIThbx8fGUlJSE9ECakRTHiJR4jtY08FFFnSfvoaqUlJQQHx/vyesbY0x431S6j3JycigsLOTIkSOhDoXqmgYOfdjM/mGxJMf3/26Pj48nJyen31/XGGMgwpNFbGwsEyZMCHUYADS3KF95aiMvbPmQn11xCpfOsQO7MWbwiOhmqHASHSX8/MpZnDEpg/98bjOvv3c41CEZY0zQLFkMoLiYaB68bi4njU7mi7/bwIZ9vZ8X0RhjBpIliwGWHB/Lo0vmMzIlnqWPrWfnIZvOwxgT/ixZhEBWchy/XXoqvpgors9fR1FZbahDMsaYblmyCJFxGQk8fsN8quqauP6Rtyitbgh1SMYY0yVLFiE0dUwKDy/OY//RWpY+tp6ahqZQh2SMMZ2yZBFip07M4N7Pz2ZzYRkrnthIQ1NLqEMyxpjjWLIIA+dPG8X/fnYGb7x/hG8+9w4tLTZ1hzEmvET0RXmDyVXzx1FS3cCPX9pBeqKPuz491SYeNMaEDUsWYeRLZ0+iuKqeR/9vL5lJcXz5nBNCHZIxxgCWLMKKiPDti6dS6p5hZCb5uHJegHuLG2PMALBkEWaiooQfX34KR2sauf2PW0hL8HHBtFGhDssYM8RZB3cY8sVEcf81c5iRk8ZXnnqbt/aUBK5kjDEesmQRphLjYnh0yTzGDh/GF1YV8O7BilCHZIwZwixZhLH0RB+rlp1KUlwM1+evY39pTahDMsYMUZ4mCxFZICI7RGSXiNzWTbnLRURFJM99nisitSKyyX084GWc4Sw7bRirls6noamF6x55i+Kq+lCHZIwZgjxLFiISDdwHXAhMBT4vIlM7KZcM3AK81WHTblWd5T5WeBXnYDB5ZDL5S+bxUUUdSx5dR2VdY6hDMsYMMV6eWcwHdqnqHlVtAJ4GFnVS7nvAjwBvblAdIeaOH87918zl3YOV3PTbDdQ3NYc6JGPMEOJlssgG9vs9L3TXtRGR2cBYVf1LJ/UniMjbIvIPEfl4Z28gIstFpEBECsLhPtteO+ekEfz48pm8ubuE//j9JpptWhBjzADx8jqLzuaqaDu6iUgU8HNgSSflDgLjVLVEROYCfxKRaarabkiQqq4EVgLk5eUNiSPnpXNyKK1u4Pt/fZfhCVv5/iXTbVoQY4znvEwWhcBYv+c5wAG/58nAdGCNe7AbBawWkYWqWgDUA6jqBhHZDUwBCjyMd9D4wscncqSqngf/sYfMpDj+47wpoQ7JGBPhvEwW64HJIjIBKAKuAq5u3aiq5UBm63MRWQN8Q1ULRCQLKFXVZhGZCEwG9ngY66Bz24KTKK1q4Jd/30lmchzXnTY+1CEZYyKYZ8lCVZtE5GbgJSAayFfVbSJyN1Cgqqu7qf4J4G4RaQKagRWqWupVrIORiPCDS2dwtKaBu/7fVtITfFw8c3SowzLGRChRjYym/ry8PC0oGHqtVLUNzVz3yFu8U1jGYzfM58wTMgNXMsYYl4hsUNW8QOXsCu5BbpgvmkcWz2NiZhLLVxWwpbA81CEZYyKQJYsIkJoQy6pl80lL8LHk0XV8UFwd6pCMMRHGkkWEGJkSz2+XzUeB6x55i0MVdo2jMab/WLKIIBOzknjshnkcrW5gcf46ymttWhBjTP+wZBFhZuak8eB1eew+UsUXHl9PXaNNC2KM6TtLFhHoY5Mz+fmVsyjYd5Sbn3ybpuaWUIdkjBnkLFlEqE/PHMN3F07j1XcP8V/PbyFShkgbY0LD7sEdwa4/PZfiynp+9douMpLi+NaCk0IdkjFmkLJkEeH+47wpFFc3cP+a3WQmxbHsYxNCHZIxZhCyZBHhRITvLZrO0eoGvveX7WQk+rhkdnbgisYY48f6LIaA6Cjh51fO4rSJ6Xzj2Xd4fcfhUIdkjBlkLFkMEfGx0Tx0fR4njkrmS09sZOOHR0MdkjFmELFkMYQkx8fy2A3zGZESx9LH1rPrcGWoQzLGDBKWLIaYrOQ4frv0VGKjo7jukXUcKKsNdUjGmEHAksUQNC4jgcdvmE9VXRPX56/jaHVDqEMyxoQ5SxZD1NQxKTy0OI8PS2u44bH11DQ0hTokY0wYs2QxhJ02MYNfXTWbzYVlfOl3G2m0aUGMMV2wZDHELZg+iv/97AzW7DjCfz63mZYWmxbEGHM8uyjPcNX8cRRX1fOTl98nPdHHnRefjIiEOixjTBjx9MxCRBaIyA4R2SUit3VT7nIRURHJ81t3u1tvh4hc4GWcBr58zgksOSOXR/71AQ/8Y0+owzHGhBnPzixEJBq4DzgPKATWi8hqVd3eoVwycAvwlt+6qcBVwDRgDPCqiExRVbs5g0dEhLs+PZXS6gZ++Lf3yEj0ccW8saEOyxgTJrw8s5gP7FLVParaADwNLOqk3PeAHwH+9wFdBDytqvWq+gGwy30946GoKOEnnzuFj0/O5LY/buaV7YdCHZIxJkx4mSyygf1+zwvddW1EZDYwVlX/0tO6bv3lIlIgIgVHjhzpn6iHOF9MFA9cO5cZOWnc/ORG1n1QGuqQjDFhwMtk0VkPadtQGxGJAn4O3NrTum0rVFeqap6q5mVlZfU6UNNeYlwMjy6ZR/bwYSx7fD3vHqwIdUjGmBDzMlkUAv6N3jnAAb/nycB0YI2I7AVOA1a7ndyB6hqPpSf6WLV0Pom+GBbnr2N/aU2oQzLGhJCXyWI9MFlEJoiID6fDenXrRlUtV9VMVc1V1VxgLbBQVQvccleJSJyITAAmA+s8jNV0Imd4AquWzae+qYXr89dRXFUf6pCMMSHiWbJQ1SbgZuAl4F3gGVXdJiJ3i8jCAHW3Ac8A24G/AV+2kVChMWVkMvlL8jhYXssNj66nqt6mBTFmKBLVyLhiNy8vTwsKCkIdRsR67b1D3LhqA6dNTCd/yTziYqJDHZIxph+IyAZVzQtUzqb7MEH55Ekj+dFlM/m/XSV8/ffv0GzTghgzpNh0HyZol83NobS6gf954V3SE33cvWiaTQtizBBhycL0yI2fmEhxVT0PvrGHzKQ4vnru5FCHZIwZAJYsTI/dduFJFFc18PNX3ycjyce1p40PdUjGGI9ZsjA9JiLcc9kMymoa+Pb/20p6oo+LZowOdVjGGA9ZB7fpldjoKO69eg5zxw3na09v4s1dxaEOyRjjIUsWpteG+aJ5ZPE8JmQmcuOqArYWlYc6JGOMRyxZmD5JTYjl8aXzSUvwseTRdewtrg51SMYYD1iyMH02KjWeVcvm06JwXf5bHK6oC1zJGDOoWLIw/WJSVhKPLplHSVUD1+evo7y2MdQhGWP6kSUL029OGZvGg9fNZfeRKm5cVUBdo03nZUykCCpZiMhXRSRFHI+IyEYROd/r4Mzg8/HJWfz0ilms31vKV556m6bmllCHZIzpB8GeWSxV1QrgfCALuAG4x7OozKC28JQxfOfTU3ll+yHueH4rkTJZpTFDWbAX5bVOAHQR8KiqviM2KZDpxpIzJ1BS3cCvX9tFZrKPb15wUqhDMsb0QbDJYoOIvAxMAG4XkWTA2hdMt75+3hSKqxq47/XdZCTGsfRjE0IdkjGml4JNFsuAWcAeVa0RkXScpihjuiQifP+S6RytbuDuv2wnI8nHolnZoQ7LGNMLwfZZnA7sUNUyEbkWuBOwy3VNQNFRwi+umsVpE9O59Zl3WLPjcKhDMsb0QrDJ4n6gRkROAf4T2Aes8iwqE1HiY6NZeX0eU0Ym88UnNvL2h0dDHZIxpoeCTRZN6gxpWQT8UlV/CSQHqiQiC0Rkh4jsEpHbOtm+QkS2iMgmEfmXiEx11+eKSK27fpOIPNCTD2XCT0p8LI8tnUdWchxLH1vPrsOVoQ7JGNMDwSaLShG5HbgO+KuIRAOx3VVwy9wHXAhMBT7fmgz8PKmqM1R1FvAj4Gd+23ar6iz3sSLIOE0YG5Ecz2+XzSc6KorrH1nHgbLaUIdkjAlSsMniSqAe53qLj4Bs4McB6swHdqnqHlVtAJ7GOTNp41670SoRsAH5EW58RiKP3TCPyromFuevo6ymIdQhGWOCEFSycBPE74BUEfk0UKeqgfossoH9fs8L3XXtiMiXRWQ3zpnFLX6bJojI2yLyDxH5eGdvICLLRaRARAqOHDkSzEcxYWB6diorr89jX2kNSx9bT01DU6hDMsYEEOx0H1cA64DPAVcAb4nI5YGqdbLuuDMHVb1PVScB38IZZQVwEBinqrOBrwNPikhKJ3VXqmqequZlZWUF81FMmDh9Uga/umoWm/aX8aXfbaTRpgUxJqwF2wx1BzBPVRer6vU4TUzfDlCnEBjr9zwHONBN+aeBSwBUtV5VS9zlDcBuYEqQsZpBYsH00fzPZ2ewZscRvvXcZlparBXSmHAVbLKIUlX/AfIlQdRdD0wWkQki4gOuAlb7FxCRyX5PLwZ2uuuz3A5yRGQiMBnYE2SsZhD5/Pxx3HreFP74dhH/+8K7No+UMWEq2Cu4/yYiLwFPuc+vBF7oroKqNonIzcBLQDSQr6rbRORuoEBVVwM3i8i5QCNwFFjsVv8EcLeINAHNwApVLe3JBzODx82fPIGS6gYe/tcHZCbHseKsSaEOyRjTgQT7S05ELgPOxOmLeENVn/cysJ7Ky8vTgoKCUIdheqmlRfnq7zfx53cO8OPLZ/K5vLGBKxlj+kxENqhqXqBywZ5ZoKp/AP7Qp6iM6UJUlPDTz51CWU0Dt/1xC8MTfJw7dWSowzLGuLrtdxCRShGp6ORRKSIV3dU1pqd8MVHcf+1cpo9J4ctPbmT9Xmt5NCZcdJssVDVZVVM6eSSr6nFDWY3pq6S4GPKXzCM7bRjLHlvPex/ZbxJjwoHdg9uEnYykOFYtm88wXzTXP7KO/aU1oQ7JmCHPkoUJSznDE1i19FTqGptZnL+Okqr6UIdkzJBmycKErRNHJZO/ZB5FZbXc8Nh6quptWhBjQsWShQlrebnp/OaaOWw7UMGK326gvqk51CEZMyRZsjBh71Mnj+SHl83kX7uKufWZd2xaEGNCIOjrLIwJpcvn5lBSVc8PXnyPjEQf/71wGiKdzVVpjPGCJQszaNx01iRKqhtY+cYeMpLiuOVTkwNXMsb0C0sWZlC5bcFJFFfV87NX3icjycc1p44PdUjGDAmWLMygEhUl/PCymZTVNHLnn7aSnuDjwhmjQx2WMRHPOrjNoBMbHcV9V89hzrjhfPXpTby5uzjUIRkT8SxZmEFpmC+aRxbnkZuZwPJVG9haVB7qkIyJaJYszKCVluDj8aXzSR0Wy5JH17G3uDrUIRkTsSxZmEFtdOowHl86n+YW5fr8dRyurAt1SMZEJEsWZtA7YUQSj94wn+Kqehbnr6eirjHUIRkTcSxZmIgwa2waD1w7l12HK7nx8QLqGm1aEGP6kyULgDrrHI0En5iSxU8+dwpvfVDKLU+9TVNzS6hDMiZieHqdhYgsAH4JRAMPq+o9HbavAL4MNANVwHJV3e5uux1Y5m67RVVf8iTI2jL44XiIT4W0cZA2Hobn+i2Pd5Z9iZ68velfi2ZlU1rdwHf/vJ07/7SVH1w6w6YFMaYfeJYsRCQauA84DygE1ovI6tZk4HpSVR9wyy8EfgYsEJGpwFXANGAM8KqITFHV/m9bEIHzvgdl+6DsQyjeCbv+Dk217cslZLqJw00ercvDcyE1B2Li+j000zs3nDmBkqoG7n19F5lJcXzjghNDHZIxg56XZxbzgV2qugdARJ4GFgFtyUJV/e+ZmQi0Tie6CHhaVeuBD0Rkl/t6/+73KONT4cxb2q9TheojcHSfk0SO7nUSSdk+OPA2vPtnaPHvRBVIHn3sLKTtjMR9npIN0Xax/EC69fwplFTXc+/ru8hI8nHDmRNCHZIxg5qXR7BsYL/f80Lg1I6FROTLwNcBH/BJv7prO9TN7qTucmA5wLhx4/olaPeFIWmE8xg77/jtLc1QefBYMin78Njyvjdhy7Ogfu3lEu2cfbSdkeS2TyxJIyHKuo/6k4jwvUXT25qk0hN9LJp13FfIGBMkL5NFZw3Fx92IQFXvA+4TkauBO4HFPai7ElgJkJeXN3A3OYhyD/6pOcCZx29vaoCKwvZJ5KibVHa+AlWH2pePjoO0se37SNqWcyEh3UlgpkdioqP45VWzWZy/jm88+w7DE3x8YkpWqMMyZlDyMlkUAmP9nucAB7op/zRwfy/rhpcYH6RPdB6daayFsv1+TVx+ZycHNkLt0fblfUnHEki7/hL3eXyq5x9psIqPjeahxXlc+eBaVjyxgSdvPI1ZY9NCHZYxg46oevODXERigPeBTwFFwHrgalXd5ldmsqrudJc/A3xHVfNEZBrwJE4/xRjg78Dk7jq48/LytKCgwJPPMuDqKo71kbSekbQt74OGqvbl49M6nJHkHkssaePAlxCSjxFODlfWcfn9/6ayrpFnV5zBCSOSQh2SMWFBRDaoal7Acl4lCzeIi4Bf4AydzVfV/xGRu4ECVV0tIr8EzgUagaPAza3JRETuAJYCTcDXVPXF7t4ropJFd1SdMw//Tnf/vpOyD6Gpw5QXiSM6PyNJGw+pY50zoSFgX0k1l93/JnEx0Tz3xdMZnTos1CEZE3JhkSwG0pBJFoG0tED1Yb/+kr3tz07KC6Glya+CQMqY40dwtS6njHH6aCLE1qJyrlq5ltGp8Ty74nTSEoZGojSmK5YsTOeam5yRXB3PSFqXKw7QbixBVIw7kmt856O5kkYOus73N3cXsyR/PTNyUnli2alPn/nQAAAVuUlEQVQM80VOMjSmpyxZmN5paoDy/V30l3zonLX4i4k/1jdy3NlJLgwbHpbJ5MUtB/nSkxs558QRPHjdXGKjbeiyGURaWqChEuornT5OiYIRJ/XqpSxZGG801BzrG+lsNFddWfvyvuSu+0uGj4e45JB8DIDfvbWPO57fyqVzsvnJ5acQFRV+Sc1EGFVnNGR9xbEDfX1Fh+eVx9a1Pa9sX6ahsv3rZufBjX/vVUjBJgu7rNj0jC/B+QXT1a+YuvLOry85uhf2/AMaO9ygaNjwzs9I0sY7157EetcJfc2p4ympauBnr7xPZlIc/3XRyZ69l4kAzY09OLh33F5+7KDfrs+wC7EJzg+puBTnb3wKJI+EuNRjz/23J3t/H3pLFqZ/xafCqBnOoyNVqCn163T3OyM5tA12/A2a69vXSRrZxcWK45yRXNGxfQr3K588gZKqela+sYeMRB83nTWpT69nwlBrk81xB/LyLg7sXfyq7zjKsDNRsR0O5CnOj564qe0P/HHJnRz4/Q7+ffxee8GShRk4IpCY4Tyy5x6/vaXFubq9Xaf7Xufv/nWw9Y/gf6mNRDnzbnUcwdW6nDw64EguEeE7n5lGSXUDP3jxPTKS4rh8bk7/fm7TO6rQWHPsgN3bX/Udm2w6Jc6B2v9An5gFGZPaH/g7+1Ufn3rseUxcWPbR9QdLFiZ8REVBymjnMe6047c3N0FFUefXl+xZ44zyajeSK9YZydWuv8RvOTELRIiKEn56xSmU1TTyrT9sZnhCLJ86eeRAferI1NTQvvmlt7/qg5loOjbx+F/oyaOPP/h396velxSxB/n+YsnCDB7RMc5Bfvh44OPHb2+qd64jae109x/N9d5foaa4ffmYYW1nIXFp43lkSg4/K6/nvif3kn7tAmZPmTD0DiAtzZ13qPr/og/mV32Pmmz8fqGnjeviF3wXB35fss3oPEBsNJQZOhqq/Zq3OhnN1eGOic2+ZKKH53beX5I2HuLCaMoQ/yabjp2qPflV33Eqmc5IVBBNMyndbHcP/HYPmLBgo6GM6ciXCCNOdh6dqS2Dsn0UF+7kib+9wZimw3xmWCPDSnbD7tecg7G/hIxO+kvchJI6FmLjg4urqSFAu3tX7fSVvWuy6XhgTxnTRYdrxzZ5d50vceidcRk7szCmM+99VMEVD/ybzKQ4nl1xOhmJPqguds9I9h4/mqt8PzQ3tH+RpFHHkkhUdNe/6juOAOtMdFxwTTMdO1zbtctbk405nl2UZ0wfrd9byrUPv8VJo5J58sbTSIzr5kDb0gJVH3U+hUrZPqffvdMDe1fDKDv8qrcmG+MRSxbG9INXtx/ipic2cMakDB5ZPA9fjE0LYiJLsMnCvvnGdOPcqSO559IZ/HNnMbc++w4tLZHx48qYnrIGTGMC+FzeWEqqG7jnxffISPTxnc9MRayD1wwxliyMCcJNn5hISVU9D/3zAzKTfNz8ycmhDsmYAWXJwpggiAi3X3gyJVUN/OTl90lPjOPqU8eFOixjBowlC2OCFBUl/PDymRytaeDOP20hPTGWBdO9n+3TmHBgHdzG9EBsdBT3XTOHWWPTuOWpTfx7d0moQzJmQHiaLERkgYjsEJFdInJbJ9u/LiLbRWSziPxdRMb7bWsWkU3uY7WXcRrTEwm+GPKXzGN8RgLLVxWw7UB54ErGDHKeJQsRiQbuAy4EpgKfF5GpHYq9DeSp6kzgOeBHfttqVXWW+1joVZzG9EZago9Vy+aTHB/D4vz17CupDlzJmEHMyzOL+cAuVd2jqg3A08Ai/wKq+rqqtk64sxawGwmYQWN06jBWLTuV5pYWrntkHYcrg5hp1ZhBystkkQ3s93te6K7ryjLgRb/n8SJSICJrReSSziqIyHK3TMGRI0f6HrExPXTCiCTyl8zjSGU9S/LXU1HXGOqQjPGEl8mis6uWOr38VUSuBfKAH/utHudegn418AsROe5+l6q6UlXzVDUvKyurP2I2psdmjxvOA9fN5f1DlSxfVUBdYxCzvxozyHiZLAqBsX7Pc4ADHQuJyLnAHcBCVW2bflNVD7h/9wBrgNkexmpMn5w1JYufXnEKa/eU8rWnN9Fs04KYCONlslgPTBaRCSLiA64C2o1qEpHZwIM4ieKw3/rhIhLnLmcCZwLbPYzVmD5bNCubuz49lb9t+4g7/7SVSJmk0xjw8KI8VW0SkZuBl4BoIF9Vt4nI3UCBqq7GaXZKAp5159r50B35dDLwoIi04CS0e1TVkoUJe0s/NoGS6nrue303mUk+bj3/xFCHZEy/8PQKblV9AXihw7q7/JbP7aLem8AML2MzxivfOP9ESqoa+PVru8hI9LHkzAmhDsmYPrPpPozpZyLC9y+ZTml1A9/9y3bSk+JYeMqYUIdlTJ/YdB/GeCAmOopffX4283LTufWZTfz67zt5c3exDa01g5adWRjjkfjYaB5enMeyx9bz01feb1s/ITORGdmpziMnlWljUkiOjw1hpMYEZsnCGA+lxMfy7IozKK1uYEtROVuLytlcWEbB3lJWv+OMJBeBia0JJCeNmTmpTB2d0v09v40ZYPZtNGYApCf6OGtKFmdNOXbxaHFVPVuKytlSWM7mwnLW7inlT5uOJZATspKYkZPKTPcMZOroVIb5okP1EcwQJ5EyFjwvL08LCgpCHYYxfXK4os5JIK1JpKicI5XOtapRAlNGJjM9O5WZOU4z1smjU4iPtQRiek9ENrizZXRfzpKFMeFLVTlU0XoGUsZmN4mUVDcAEBMlTB6Z3Hb2MSM7lZNGJxMXYwnEBCfYZGHNUMaEMRFhVGo8o1LjOW/qSMBJIAfL69hcWM6WojK2FFXw8vaP+H2BM29nbLRw4qhktxPd6QOZMjIZX4wNfjS9Z2cWxkQAVaXwaK3Tge6efWwpKqe81hmq64uO4qTRTgKZmZPK9GwngcRGWwIZ6qwZypghTlXZX1rL5qKytuSxpaicyromAHwxUUwdndI2hHdmTionZCURYwlkSLFkYYw5TkuLsq+05lgfSKEznLe6wZlWPT7WSSAzc9LaksikrCSiozq744CJBJYsjDFBaWlRPiipbhvCu7WonK0HyqlxE0iCL5ppY1KYkZ3GjBzn78TMRKIsgUQESxbGmF5rblH2HKlyO9Gdx7YD5dQ1tgCQFBfD1DEp7UZh5WZYAhmMbDSUMabXot0huZNHJnPZ3BwAmppb2H2kms2FZWwpcs5Cfrt2H/VNTgJJjotpuwak9e+49ATc2w+YQc7OLIwxvdbY3MLOQ1XuEF5nFNa7BytpaHYSSEp8jHvmkdZ2IWHO8GGWQMKINUMZY0KioamF9w9Vtp19bCkqY8dHlTQ2O8eatITYtiG8rfNhjUmNtwQSIpYsjDFho76pmR0fVbZ1oG8uLOf9Q5U0ufcqz0j0tZvGZEZOKqNSLIEMBOuzMMaEjbiYaGbmpDEzJ61tXV1jM+8erGhLHluKyvnNmmKa3QSSmRR3LHm4iWRESnyoPsKQ52myEJEFwC9x7sH9sKre02H714EvAE3AEWCpqu5zty0G7nSLfl9VH/cyVmPMwIqPjWb2uOHMHje8bV1tQzPbD1awpdCZxmRLURlrdhzGzR+MTIlrN43J9OxUspLjQvQJhhbPmqFEJBp4HzgPKATWA59X1e1+Zc4B3lLVGhH5InC2ql4pIulAAZAHKLABmKuqR7t6P2uGMiYyVdc3uQmk3O0HKWNPcTWth67RqfHtpjGZkZ1KRpIlkGCFQzPUfGCXqu5xA3oaWAS0JQtVfd2v/FrgWnf5AuAVVS11674CLACe8jBeY0wYSoyLYV5uOvNy09vWVdU3sa2ovK0TfWtROS9vP9S2PTttWLshvDOyU0lL8IUi/IjhZbLIBvb7PS8ETu2m/DLgxW7qZvdrdMaYQSspLoZTJ2Zw6sSMtnUVdY3O1ed+fSAvbv2obfvY9GHMzE5ru4hwenYqqcPsdrbB8jJZdDaModM2LxG5FqfJ6aye1BWR5cBygHHjxvUuSmNMREiJj+WMSZmcMSmzbV15TSNbDxwbwru5qIy/bjnYtj03I4EZOWnMyHamMZmebfdD74qXyaIQGOv3PAc40LGQiJwL3AGcpar1fnXP7lB3Tce6qroSWAlOn0V/BG2MiRypCbGceUImZ55wLIEcde+H3noR4cZ9R/nzO8cOTRMzE9vOPmbmpDFtjN0PHbzt4I7B6eD+FFCE08F9tapu8yszG3gOWKCqO/3Wp+N0as9xV23E6eAu7er9rIPbGNNbJf73Q3f/flRRBzj3Q5+UlcTM7GN9IFPHpJDgi4wEEvIOblVtEpGbgZdwhs7mq+o2EbkbKFDV1cCPgSTgWffimw9VdaGqlorI93ASDMDd3SUKY4zpi4ykOM4+cQRnnziibd3hyrq2/o+tReX8c1cxf3y7CHDuh37CiKR2Q3injk5hmC9yb2drV3AbY0yQDlXU+Z19OPNhFVc590OPjhImj0hqN43JSaOSiY8N7wRi030YY4zHVJWPKtz7oftN515a7SSQmChhyshkJ4G4SeTEUcnExYRPArFkYYwxIaCqFJXVthvCu7nw2P3QY6OFk0altCWP1gQSqvuhW7IwxpgwoaoUHq1lc2F5u3ui+98P/eRRyc690N1rQSaPGJj7oVuyMMaYMNbSonxYWsPmtgsJy9haVEFVvZNA4mKi2u5GON0dxjspK7HfE4glC2OMGWRaWpS9JdXH7gVS2P5+6MNinfuhtw7hnZmTyoTMJKL7cDtbSxbGGBMBmluUD4r97odeWM62AxXUNjoJJMEXzSdPGsG9V88J8EqdC/l1FsYYY/ouOko4YUQyJ4xI5tI5zv3Qm1uU3UfcBFJYNiBXmFuyMMaYQSbaHZI7ZWQyl8/NGZD3DM1YLWOMMYOKJQtjjDEBWbIwxhgTkCULY4wxAVmyMMYYE5AlC2OMMQFZsjDGGBOQJQtjjDEBRcx0HyJyBNjXh5fIBIr7KZz+ZHH1jMXVMxZXz0RiXONVNStQoYhJFn0lIgXBzI8y0CyunrG4esbi6pmhHJc1QxljjAnIkoUxxpiALFkcszLUAXTB4uoZi6tnLK6eGbJxWZ+FMcaYgOzMwhhjTECWLIwxxgQU8clCRBaIyA4R2SUit3WyPU5Efu9uf0tEcv223e6u3yEiFwxwXF8Xke0isllE/i4i4/22NYvIJvexeoDjWiIiR/ze/wt+2xaLyE73sXiA4/q5X0zvi0iZ3zYv91e+iBwWka1dbBcR+ZUb92YRmeO3zcv9FSiua9x4NovImyJyit+2vSKyxd1f/Xqv4iDiOltEyv3+ve7y29btd8DjuL7pF9NW9zuV7m7zcn+NFZHXReRdEdkmIl/tpMzAfMdUNWIfQDSwG5gI+IB3gKkdynwJeMBdvgr4vbs81S0fB0xwXyd6AOM6B0hwl7/YGpf7vCqE+2sJcG8nddOBPe7f4e7y8IGKq0P5rwD5Xu8v97U/AcwBtnax/SLgRUCA04C3vN5fQcZ1Ruv7ARe2xuU+3wtkhmh/nQ38pa/fgf6Oq0PZzwCvDdD+Gg3McZeTgfc7+T85IN+xSD+zmA/sUtU9qtoAPA0s6lBmEfC4u/wc8CkREXf906par6ofALvc1xuQuFT1dVWtcZ+uBQbi3onB7K+uXAC8oqqlqnoUeAVYEKK4Pg881U/v3S1VfQMo7abIImCVOtYCaSIyGm/3V8C4VPVN931h4L5fweyvrvTlu9nfcQ3k9+ugqm50lyuBd4HsDsUG5DsW6ckiG9jv97yQ43d0WxlVbQLKgYwg63oZl79lOL8cWsWLSIGIrBWRS/oppp7EdZl7uvuciIztYV0v48JtrpsAvOa32qv9FYyuYvdyf/VUx++XAi+LyAYRWR6CeE4XkXdE5EURmeauC4v9JSIJOAfcP/itHpD9JU4T+WzgrQ6bBuQ7FtPbioOEdLKu41jhrsoEU7e3gn5tEbkWyAPO8ls9TlUPiMhE4DUR2aKquwcorj8DT6lqvYiswDkr+2SQdb2Mq9VVwHOq2uy3zqv9FYxQfL+CJiLn4CSLj/mtPtPdXyOAV0TkPfeX90DYiDNXUZWIXAT8CZhMmOwvnCao/1NV/7MQz/eXiCThJKivqWpFx82dVOn371ikn1kUAmP9nucAB7oqIyIxQCrO6Wgwdb2MCxE5F7gDWKiq9a3rVfWA+3cPsAbn18aAxKWqJX6xPATMDbaul3H5uYoOTQQe7q9gdBW7l/srKCIyE3gYWKSqJa3r/fbXYeB5+q/5NSBVrVDVKnf5BSBWRDIJg/3l6u775cn+EpFYnETxO1X9YydFBuY75kWnTLg8cM6c9uA0S7R2ik3rUObLtO/gfsZdnkb7Du499F8HdzBxzcbp0JvcYf1wIM5dzgR20k8dfUHGNdpv+bPAWj3WmfaBG99wdzl9oOJyy52I09koA7G//N4jl647bC+mfefjOq/3V5BxjcPphzujw/pEINlv+U1gwQDGNar13w/noPuhu++C+g54FZe7vfWHZOJA7S/3s68CftFNmQH5jvXbjg7XB85IgfdxDrx3uOvuxvm1DhAPPOv+x1kHTPSre4dbbwdw4QDH9SpwCNjkPla7688Atrj/WbYAywY4rh8A29z3fx04ya/uUnc/7gJuGMi43Of/DdzToZ7X++sp4CDQiPNLbhmwAljhbhfgPjfuLUDeAO2vQHE9DBz1+34VuOsnuvvqHfff+Y4Bjutmv+/XWvySWWffgYGKyy2zBGfQi389r/fXx3Cajjb7/VtdFIrvmE33YYwxJqBI77MwxhjTDyxZGGOMCciShTHGmIAsWRhjjAnIkoUxxpiALFkYEwbc2Vb/Euo4jOmKJQtjjDEBWbIwpgdE5FoRWefeu+BBEYkWkSoR+amIbBTn3iNZbtlZ7uSFm0XkeREZ7q4/QURedSfL2ygik9yXT3InZ3xPRH7nzn5sTFiwZGFMkETkZOBKnInjZgHNwDU40zxsVNU5wD+A77hVVgHfUtWZOFfWtq7/HXCfqp6Cc4X5QXf9bOBrOPdSmQic6fmHMiZIkT7rrDH96VM4Eyeud3/0DwMOAy3A790yTwB/FJFUIE1V/+Gufxx4VkSSgWxVfR5AVesA3Ndbp6qF7vNNOHMV/cv7j2VMYJYsjAmeAI+r6u3tVop8u0O57ubQ6a5pqd5vuRn7/2nCiDVDGRO8vwOXu/ctQETS3ZstRQGXu2WuBv6lquXAURH5uLv+OuAf6tyLoLD1Jkzi3AM+YUA/hTG9YL9cjAmSqm4XkTtx7ooWhTND6ZeBamCaiGzAudPilW6VxcADbjLYA9zgrr8OeFBE7nZf43MD+DGM6RWbddaYPhKRKlVNCnUcxnjJmqGMMcYEZGcWxhhjArIzC2OMMQFZsjDGGBOQJQtjjDEBWbIwxhgTkCULY4wxAf1/TIQX28dfTIQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# summarize history for loss\n",
    "plt.plot(history.history['loss'])\n",
    "plt.plot(history.history['val_loss'])\n",
    "plt.title('model loss')\n",
    "plt.ylabel('loss')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'test'], loc='upper left')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Running the example, we are first presented with a summary of the network structure. We can see our convolutional layer preserves the dimensionality of our Embedding input layer of 32-dimensional input with a maximum of 500 words. The pooling layer compresses this representation by halving it.\n",
    "\n",
    "Running the example offers a small but welcome improvement over the neural network model above with an accuracy of ~ 88%!\n",
    "\n",
    "**TODO:**\n",
    "\n",
    "It's your turn. See if you can improve the performance of the network by adjusting the hyperparameters, adding additional convolution layers, and playing with dropout.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your work here:\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## STOP, take a deep breath and wait for instruction before proceeding."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### LSTM\n",
    "\n",
    "Below is an example of training an LSTM network.\n",
    "\n",
    "Notes:  \n",
    "- RNNs are tricky. Choice of batch size is important,\n",
    "choice of loss and optimizer is critical, etc.   \n",
    "- Some configurations won't converge.  \n",
    "- LSTM loss decrease patterns during training can be quite different\n",
    "from what you see with CNNs/MLPs/etc.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading data...\n",
      "25000 train sequences\n",
      "25000 test sequences\n",
      "Pad sequences (samples x time)\n",
      "x_train shape: (25000, 80)\n",
      "x_test shape: (25000, 80)\n",
      "Build model...\n",
      "Train...\n",
      "Train on 25000 samples, validate on 25000 samples\n",
      "Epoch 1/15\n",
      "25000/25000 [==============================] - 96s 4ms/step - loss: 0.4626 - acc: 0.7803 - val_loss: 0.3982 - val_acc: 0.8264\n",
      "Epoch 2/15\n",
      "25000/25000 [==============================] - 94s 4ms/step - loss: 0.2985 - acc: 0.8789 - val_loss: 0.3789 - val_acc: 0.8334\n",
      "Epoch 3/15\n",
      "25000/25000 [==============================] - 94s 4ms/step - loss: 0.2172 - acc: 0.9148 - val_loss: 0.4344 - val_acc: 0.8106\n",
      "Epoch 4/15\n",
      "25000/25000 [==============================] - 94s 4ms/step - loss: 0.1541 - acc: 0.9439 - val_loss: 0.4674 - val_acc: 0.8320\n",
      "Epoch 5/15\n",
      "25000/25000 [==============================] - 94s 4ms/step - loss: 0.1091 - acc: 0.9604 - val_loss: 0.5986 - val_acc: 0.8240\n",
      "Epoch 6/15\n",
      "25000/25000 [==============================] - 95s 4ms/step - loss: 0.0800 - acc: 0.9719 - val_loss: 0.6347 - val_acc: 0.8266\n",
      "Epoch 7/15\n",
      "25000/25000 [==============================] - 95s 4ms/step - loss: 0.0596 - acc: 0.9792 - val_loss: 0.7131 - val_acc: 0.8218\n",
      "Epoch 8/15\n",
      "25000/25000 [==============================] - 95s 4ms/step - loss: 0.0426 - acc: 0.9862 - val_loss: 0.7804 - val_acc: 0.8156\n",
      "Epoch 9/15\n",
      "25000/25000 [==============================] - 95s 4ms/step - loss: 0.0376 - acc: 0.9874 - val_loss: 0.8342 - val_acc: 0.8135\n",
      "Epoch 10/15\n",
      "25000/25000 [==============================] - 96s 4ms/step - loss: 0.0249 - acc: 0.9925 - val_loss: 0.9107 - val_acc: 0.8154\n",
      "Epoch 11/15\n",
      "25000/25000 [==============================] - 96s 4ms/step - loss: 0.0191 - acc: 0.9942 - val_loss: 0.9798 - val_acc: 0.8144\n",
      "Epoch 12/15\n",
      "25000/25000 [==============================] - 96s 4ms/step - loss: 0.0179 - acc: 0.9943 - val_loss: 1.0156 - val_acc: 0.8137\n",
      "Epoch 13/15\n",
      "25000/25000 [==============================] - 96s 4ms/step - loss: 0.0151 - acc: 0.9951 - val_loss: 1.1029 - val_acc: 0.8076\n",
      "Epoch 14/15\n",
      "25000/25000 [==============================] - 97s 4ms/step - loss: 0.0175 - acc: 0.9943 - val_loss: 0.9683 - val_acc: 0.8108\n",
      "Epoch 15/15\n",
      "25000/25000 [==============================] - 96s 4ms/step - loss: 0.0091 - acc: 0.9970 - val_loss: 1.2153 - val_acc: 0.8070\n",
      "25000/25000 [==============================] - 11s 448us/step\n",
      "Test score: 1.2152831412661076\n",
      "Test accuracy: 0.80704\n"
     ]
    }
   ],
   "source": [
    "from keras.preprocessing import sequence\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Embedding\n",
    "from keras.layers import LSTM\n",
    "from keras.datasets import imdb\n",
    "\n",
    "max_features = 20000\n",
    "maxlen = 80  # cut texts after this number of words (among top max_features most common words)\n",
    "batch_size = 32\n",
    "\n",
    "print('Loading data...')\n",
    "(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=max_features)\n",
    "print(len(x_train), 'train sequences')\n",
    "print(len(x_test), 'test sequences')\n",
    "\n",
    "print('Pad sequences (samples x time)')\n",
    "x_train = sequence.pad_sequences(x_train, maxlen=maxlen)\n",
    "x_test = sequence.pad_sequences(x_test, maxlen=maxlen)\n",
    "print('x_train shape:', x_train.shape)\n",
    "print('x_test shape:', x_test.shape)\n",
    "\n",
    "print('Build model...')\n",
    "model = Sequential()\n",
    "model.add(Embedding(max_features, 128))\n",
    "model.add(LSTM(128, dropout=0.2, recurrent_dropout=0.2))\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "\n",
    "# try using different optimizers and different optimizer configs\n",
    "model.compile(loss='binary_crossentropy',\n",
    "              optimizer='adam',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "print('Train...')\n",
    "history = model.fit(x_train, y_train,\n",
    "          batch_size=batch_size,\n",
    "          epochs=15,\n",
    "          validation_data=(x_test, y_test))\n",
    "score, acc = model.evaluate(x_test, y_test,\n",
    "                            batch_size=batch_size)\n",
    "print('Test score:', score)\n",
    "print('Test accuracy:', acc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['val_loss', 'val_acc', 'loss', 'acc'])\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0xb26d82160>"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xd8lfXd//HXJ3uSyQ6QgEwVQQLuiQM3jjpx9W6xddR619lWbb3bX73tuG1r66xbUcTZioosqXUEkCVLRhghjJCQANnJ+fz+uK6QkxCSA+Tkyjn5PB+P8zjnXOt8wjjvXN/v9f1eoqoYY4wxrYnwugBjjDGdn4WFMcaYNllYGGOMaZOFhTHGmDZZWBhjjGmThYUxxpg2WVgYA4jIiyLymwC33SAiZwW7JmM6EwsLY4wxbbKwMCaMiEiU1zWY8GRhYUKG2/xzj4gsFZFyEfmHiPQUkY9EZI+IzBSRNL/tLxaR5SJSKiJzRWS437rRIvKNu9+bQFyzz7pQRBa7+34hIiMDrPECEVkkIrtFZLOI/KrZ+pPd45W6629yl8eLyB9FZKOIlInI5+6y00WkoIU/h7Pc178SkWki8qqI7AZuEpFxIvKl+xlbReQJEYnx2/9IEflUREpEZLuI/FxEeolIhYhk+G03RkSKRCQ6kJ/dhDcLCxNqLgfOBoYAFwEfAT8HMnH+Pf8EQESGAFOAnwLdgenAP0Ukxv3ifA94BUgH3nKPi7vvscDzwC1ABvA08IGIxAZQXzlwA5AKXAD8WEQmusft79b7V7emUcBid78/AGOAE92a7gV8Af6ZXAJMcz/zNaAeuMv9MzkBGA/c6taQDMwEPgb6AEcAs1R1GzAXuNLvuJOAN1S1NsA6TBizsDCh5q+qul1VtwD/Br5W1UWqWg28C4x2t7sK+FBVP3W/7P4AxON8GR8PRAOPq2qtqk4D5vt9xg+Bp1X1a1WtV9WXgGp3v1ap6lxVXaaqPlVdihNYp7mrrwNmquoU93OLVXWxiEQA3wfuVNUt7md+4f5MgfhSVd9zP7NSVReq6leqWqeqG3DCrqGGC4FtqvpHVa1S1T2q+rW77iWcgEBEIoFrcALVGAsLE3K2+72ubOF9kvu6D7CxYYWq+oDNQF933RZtOovmRr/XA4Cfuc04pSJSCvRz92uViBwnInPc5psy4Ec4v+HjHmNdC7tl4jSDtbQuEJub1TBERP4lItvcpqn/F0ANAO8DI0RkIM7ZW5mq5h1iTSbMWFiYcFWI86UPgIgIzhflFmAr0Ndd1qC/3+vNwG9VNdXvkaCqUwL43NeBD4B+qpoCPAU0fM5mYFAL++wEqg6wrhxI8Ps5InGasPw1nzr6SWAVMFhVu+E007VVA6paBUzFOQO6HjurMH4sLEy4mgpcICLj3Q7an+E0JX0BfAnUAT8RkSgRuQwY57fvs8CP3LMEEZFEt+M6OYDPTQZKVLVKRMYB1/qtew04S0SudD83Q0RGuWc9zwN/EpE+IhIpIie4fSTfAXHu50cDvwTa6jtJBnYDe0VkGPBjv3X/AnqJyE9FJFZEkkXkOL/1LwM3ARcDrwbw85ouwsLChCVVXY3T/v5XnN/cLwIuUtUaVa0BLsP5UtyF07/xjt++C3D6LZ5w1691tw3ErcAjIrIHeAgntBqOuwk4Hye4SnA6t49xV98NLMPpOykB/heIUNUy95jP4ZwVlQNNro5qwd04IbUHJ/je9KthD04T00XANmANcIbf+v/gdKx/4/Z3GAOA2M2PjDH+RGQ28LqqPud1LabzsLAwxuwjImOBT3H6XPZ4XY/pPKwZyhgDgIi8hDMG46cWFKY5O7MwxhjTJjuzMMYY06awmXQsMzNTs7OzvS7DGGNCysKFC3eqavOxO/sJm7DIzs5mwYIFXpdhjDEhRUQ2tr2VNUMZY4wJgIWFMcaYNgUtLETkeRHZISLfHmC9iMhfRGStOPcnONZv3Y0issZ93BisGo0xxgQmmH0WL+JMl/DyAdafBwx2H8fhTH52nIikAw8DuTgTpC0UkQ9UddfBFlBbW0tBQQFVVVWHUH5oiYuLIysri+hou0+NMab9BS0sVHWeiGS3ssklwMvuNNFfiUiqiPQGTgc+VdUSABH5FJiAc1+Ag1JQUEBycjLZ2dk0nWA0vKgqxcXFFBQUkJOT43U5xpgw5GWfRV+azsNf4C470PKDVlVVRUZGRlgHBYCIkJGR0SXOoIwx3vAyLFr6BtdWlu9/AJHJIrJARBYUFRW1/CFhHhQNusrPaYzxhpfjLApwbkbTIAvnhjUFOE1R/svntnQAVX0GeAYgNzfX5i0xxoS9unofO/ZUs7WsksLSKraWVZIUG821x/Vve+fD4GVYfADcLiJv4HRwl6nqVhH5BPh/IpLmbncO8IBXRR6u0tJSXn/9dW699daD2u/888/n9ddfJzU1NUiVGWM6G1WluLyGraVVFJZVsrW0ksKyKgpLK9laVsXW0kq276mm3tf0d+PR/VNDNyxEZArOGUKmiBTgXOEUDaCqTwHTcW4EsxaoAG5215WIyP/g3AQG4JGGzu5QVFpayt///vf9wqK+vp7IyMgD7jd9+vRgl2ZMSFJVfNr4XFvvo6q2nuo6n/uop7rWed24/ADL6nzu8nqqahuX1dT5iI6MIDY6gtioCOKiI4mNiiA2yn2Odl7HRfsti4ogNrrx9b59/Jb5FLaVNQSBEwINr7eWOcFQU+dr8vPGREXQJyWO3inxHD8og76p8fROiad3ahx93OduccG/CjKYV0Nd08Z6BW47wLrncW4zGfLuv/9+1q1bx6hRo4iOjiYpKYnevXuzePFiVqxYwcSJE9m8eTNVVVXceeedTJ48GWicvmTv3r2cd955nHzyyXzxxRf07duX999/n/j4eI9/MtOV+XzKnqo6SitrKK2opbSyltKKGsoqaymtqGVXRQ1lfstLK2upqqnHp+Bzv+RB971Xv+eGENj3vtl27SkmMqLJl3/Dc0ykUFuvfiHSNEjaS2SE0DM5lt6p8RzVN4Vzj+xF75Q4eqfG7wuCjMSYTtEnGTZzQ7Xl1/9czorC3e16zBF9uvHwRUe2us2jjz7Kt99+y+LFi5k7dy4XXHAB33777b5LXJ9//nnS09OprKxk7NixXH755WRkZDQ5xpo1a5gyZQrPPvssV155JW+//TaTJk1q15/FmNp6H6u37WHNjj3sKne+6MvcL/qGQGh4X1ZZ2+oXd1JsFCnx0aQlRpMaH0Pv1HjioyOJFCEiAkCIEIgQ51lEkNbe07g8Yt86532M3xnAfr/tN/nN329ddAQxkRFERBz8l7DPp9TUN56RNARJk1Cp3f/MpdoNmZ7d4uiT6pwp9EiOJSoyNCbS6DJh0VmMGzeuyViIv/zlL7z77rsAbN68mTVr1uwXFjk5OYwaNQqAMWPGsGHDhg6r14QnVaVgVyWLN5eyZHMpizeX8m1hGVW1jb81i0C3uGhSE6JJjY8mJSGGAekJTd6nxrvrE6JJiY9xn6OJDpEvwEMRESHERUQSFx2J27LeJXSZsGjrDKCjJCYm7ns9d+5cZs6cyZdffklCQgKnn356i2MlYmNj972OjIyksrKyQ2o14aOsopYlBaX7wmFJQSk799YATpv4UX26ce24ARzTL4Uj+6SQmRRDclw0kYfwm7cJT10mLLySnJzMnj0t36GyrKyMtLQ0EhISWLVqFV999VUHV2fCUU2dj5Vbdzc5a1i/s3zf+kHdEzltSA9G9UthVL80hvZKJiYqfM8ETPuwsAiyjIwMTjrpJI466iji4+Pp2bPnvnUTJkzgqaeeYuTIkQwdOpTjjz/ew0pNKFJVNhZXsNgNhcWbS1lRuJuaeqc5KTMpllH9Url8TBbHZKUysl9Kh1w5Y8JP2NyDOzc3V5vf/GjlypUMHz7co4o6Xlf7ecOBqlJd56O8uo6Kmnr2VtdRUVPH3up6Kqrr3PeNy8ur6/dtW1pZw/LC3ZRW1AIQHx3J0VkpjOqXyqh+qRzTL5U+KXGd4koa03mJyEJVzW1rOzuzMCZIqmrr+XJdMXNW76CwtMr9km8eAPX7DbA6kAiBxNgokmKjSIiJJCkumnNH9GJUfyccBvdICpkra0zosbAwph1tK6ti9qodzF61nc/X7qSq1kdCTCQDMhJJjIkkNSGGvmnxJMZEkRgbRWJsJAkxfgEQG0VCbBSJMZHO+hhnm8TYKGKjIuwswXjGwsKYw+DzKUu3lDF75XZmrdrBcncsT1ZaPFfl9mP88J4cNzCd2KgDj9Y3JhRYWBhzkPZW1/H5miJmrdzBnNU72Lm3hgiBMQPSuG/CMMYP78HgHkl2FmDCioWFMQHYVFzBrFXbmb1qB1+tL6a2XukWF8VpQ3swflgPThvSnbTEGK/LNCZoLCyMaUFdvY+FG3cxe9UOZq3awdodewFnjMLNJ+Vw5rAejBmQFtYjlY3xZ2ERZIc6RTnA448/zuTJk0lISAhCZaa5sopa5n63g1krd/DZd0WUVdYSHSkcl5PBteP6c+awHmRnJrZ9IGPCkIVFkB1oivJAPP7440yaNMnCIog27Cxn5srtzFy5nfkbdlHvUzISYzh7RE/GD+vByYMzSbZBbMZYWASb/xTlZ599Nj169GDq1KlUV1dz6aWX8utf/5ry8nKuvPJKCgoKqK+v58EHH2T79u0UFhZyxhlnkJmZyZw5c7z+UcJCvU9ZvHkXn67YwcyV2/c1Lw3rlcyPThvI+OE9GZWVekizkRoTzrpOWHx0P2xb1r7H7HU0nPdoq5v4T1E+Y8YMpk2bRl5eHqrKxRdfzLx58ygqKqJPnz58+OGHgDNnVEpKCn/605+YM2cOmZmZ7Vt3F1NeXce/1+xk5kqng7qkvIaoCOG4gelcd1x/zhrek37pdvZmTGu6Tlh0AjNmzGDGjBmMHj0agL1797JmzRpOOeUU7r77bu677z4uvPBCTjnlFI8rDX3byqr2NS99sa6Ymjof3eKiOGNYD84a3pPThna3OZKMOQhdJyzaOAPoCKrKAw88wC233LLfuoULFzJ9+nQeeOABzjnnHB566CEPKgxdqsrywt3MWuk0Ly3bUgZA//QErj9+AOOH92BsdrpdvWTMIeo6YeER/ynKzz33XB588EGuu+46kpKS2LJlC9HR0dTV1ZGens6kSZNISkrixRdfbLKvNUO1rLqunq/WlzBzxXZmrdxOYVkVInBs/zTunTCUs4f35AgbHGdMu7CwCDL/KcrPO+88rr32Wk444QQAkpKSePXVV1m7di333HMPERERREdH8+STTwIwefJkzjvvPHr37m0d3C5V5ct1xbzy1UbmfVdEeU098dGRnDI4k5+ePYQzh/UgMym27QMZYw6KTVEeRsL551VV5q4u4q+z1/DNplIyk2I458henD28JycMynBvcWmMOVg2RbkJCz6fMmPFNv46ey3LC3fTNzWe30w8iivGZFlAGNOBLCxMp1RX7+PDZVt5YvZa1uzYS3ZGAo9dMZJLR/e1TmpjPBD2YaGqXaKDM1yaE2vrfbz7zRb+PnctG4orGNIziT9fPYoLR/Yh0gbKGeOZsA6LuLg4iouLycjICOvAUFWKi4uJi4vzupRDVlVbz1sLC3hq7jq2lFZyZJ9uPDVpDOeM6GmjqY3pBMI6LLKysigoKKCoqMjrUoIuLi6OrKwsr8s4aBU1dbz+9SaembeeHXuqGd0/ld9MPIrTh3YP64A3JtSEdVhER0eTk5PjdRmmBXuqannlq43849/5FJfXcMLADB6/ahQnDArvs0BjQlVYh4XpfEoranjhPxt44T/57K6q4/Sh3bn9jCPIzU73ujRjTCssLEyH2Lm3muf+nc8rX26gvKaec0b05I4zB3N0VorXpRljAmBhYYJqW1kVT89bx5S8TVTX+bhwZB9uO2MQw3p187o0Y8xBsLAwQeHzKS98sYHHPl5FnU+5dHRffnz6IAZ1T/K6NGPMIbCwMO1uS2kl97y1hC/WFTN+WA9+dfGRdr8IY0KchYVpN6rKu4u28PD7y/Gp8uhlR3PV2H52dZMxYcDCwrSLkvIafvHuMj76dhu5A9L405Wj6J9hZxPGhAsLC3PYZq/azn1vL6O0oob7Jgxj8qkDbWoOY8KMhYU5ZOXVdfzmw5VMydvEsF7JvHTzOEb0saucjAlHFhbmkCzcWMJ/T13CppIKbjl1IP99zhBio2zKcGPCVVDDQkQmAH8GIoHnVPXRZusHAM8D3YESYJKqFrjr6oFl7qabVPXiYNZqAlNT5+Pxmd/x1Gfr6JMazxs/PJ7jBmZ4XZYxJsiCFhYiEgn8DTgbKADmi8gHqrrCb7M/AC+r6ksicibwO+B6d12lqo4KVn3m4K3etoe73lzMiq27uTI3iwcvHEFyXLTXZRljOkAwzyzGAWtVdT2AiLwBXAL4h8UI4C739RzgvSDWYw6Rz6f84/N8fv/JapLjonjm+jGcc2Qvr8syxnSgYN5yrC+w2e99gbvM3xLgcvf1pUCyiDS0acSJyAIR+UpEJrb0ASIy2d1mQVeYhtwLBbsquObZr/jt9JWcNrQ7n9x1qgWFMV1QMM8sWrp2svnt3O4GnhCRm4B5wBagzl3XX1ULRWQgMFtElqnquiYHU30GeAYgNzc3PG4V10moKtMWFvDrfzongo9dMZLvjcmyAXbGdFHBDIsCoJ/f+yyg0H8DVS0ELgMQkSTgclUt81uHqq4XkbnAaKBJWJjgKN5bzQPvLGPGiu2My0nnj987xqbrMKaLC2ZYzAcGi0gOzhnD1cC1/huISCZQoqo+4AGcK6MQkTSgQlWr3W1OAh4LYq3GNXPFdu5/Zym7K+v4+fnD+K+TbYCdMSaIYaGqdSJyO/AJzqWzz6vqchF5BFigqh8ApwO/ExHFaYa6zd19OPC0iPhw+lUebXYVlWlne6vr+J9/ruDNBZsZ3rsbr/7gGJtG3Bizj6iGR1N/bm6uLliwwOsyQtLuqlquePIL1uzYyy2nDuKuswfbADtjuggRWaiquW1tZyO4u7h6n/LTNxazvqicF28ex2lDuntdkjGmEwrmpbMmBPz+k9XMXrWDhy8+0oLCGHNAFhZd2HuLtvDUZ+u47rj+XH/8AK/LMcZ0YhYWXdSSzaXc+/ZSjstJ5+GLjvS6HGNMJ2dh0QXt2F3F5FcW0CM5lr9fdywxUfbPwBjTOuvg7mKqauuZ/MpC9lTV8faPTyQjKdbrkowxIcDCogtRVX7+zjIWby7lqUljGN7bxlEYYwJj7Q9dyHP/zuedRVu466whTDjKJgM0xgTOwqKLmLN6B7/7aCXnH92LO848wutyjDEhxsKiC1hXtJefTFnE0F7d+MP3jiHC5noyxhwkC4swV1ZZyw9fWkBMZATP3jCGhBjrpjLGHDz75ghj9T7ljimL2Lyrgtd/eDxZaTbNuDHm0FhYhLFHP1rJvO+K+N1lRzM2O93rcowxIcyaocLUtIUFPPvvfG48YQDXjOvvdTnGmBBnYRGGvtm0i5+/s4wTB2XwywtHeF2OMSYMWFiEmW1lVdzyykJ6pcTxt2uPJTrS/oqNMYfP+izCiDOVxwIqqut47QfHkZYY43VJxpgwYWERJlSV+95eyrItZTxzfS5DeiZ7XZIxJoxYG0WYeOqz9by/uJC7zxnK2SN6el2OMSbMWFiEgVkrt/PYJ6u4cGRvbj19kNflGGPCkIVFiFuzfQ93vrGYI/t04/dXHIOITeVhjGl/FhYhrLSihh++vIC46EieuT6X+JhIr0syxoQpC4sQVVfv4/bXF1FYWsXT14+hT2q81yUZY8KYXQ0Von47fSWfr93JY1eMZMyANK/LMcaEOTuzCEFvzt/EC//ZwPdPyuHK3H5el2OM6QIsLELMwo0l/PK9bzllcCY/P3+Y1+UYY7oIC4sQsre6jp9MWUzvlHieuOZYomwqD2NMB7E+ixDyvx+torCskqm3nEBKQrTX5RhjuhD71TREfLmumFe+2sjNJ+bYvSmMMR3OwiIEVNTUce/bSxiQkcA95w71uhxjTBcUUFiIyNsicoGIWLh44LGPV7O5pJLHLh9pA++MMZ4I9Mv/SeBaYI2IPCoidhlOB/lqfTEvfrGBm07M5riBGV6XY4zpogIKC1WdqarXAccCG4BPReQLEblZRKynNUgqauq47+2l9E9P4N4J1vxkjPFOwM1KIpIB3AT8AFgE/BknPD4NSmWG33+ymo3FFfzv5SNJiLEL14wx3gnoG0hE3gGGAa8AF6nqVnfVmyKyIFjFdWXzN5Tw4hcbuOGEAZwwyJqfjDHeCvTX1SdUdXZLK1Q1tx3rMUBlTT33TltK39R47ptg3UPGGO8F2gw1XERSG96ISJqI3Bqkmrq8P85YTf7Och67fCSJsdb8ZIzxXqBh8UNVLW14o6q7gB+2tZOITBCR1SKyVkTub2H9ABGZJSJLRWSuiGT5rbtRRNa4jxsDrDPkLdxYwj/+k8+k4/tz4hGZXpdjjDFA4GERIX63YBORSCCmtR3cbf4GnAeMAK4RkRHNNvsD8LKqjgQeAX7n7psOPAwcB4wDHhaRsJ+Hu6q2nnveWkqflHjuP2+41+UYY8w+gYbFJ8BUERkvImcCU4CP29hnHLBWVderag3wBnBJs21GALPc13P81p8LfKqqJe5ZzKfAhABrDVl/+vQ71u8s57ErRpJkzU/GmE4k0LC4D5gN/Bi4DecL/t429ukLbPZ7X+Au87cEuNx9fSmQ7F6iG8i+iMhkEVkgIguKiooC/FE6p2827eK5f6/nmnH9Ocman4wxnUxAv76qqg9nFPeTB3FsaWGZNnt/N/CEiNwEzAO2AHUB7ouqPgM8A5Cbm7vf+lDhND8toVe3OLtHhTGmUwp0nMVgnP6EEUBcw3JVHdjKbgWA/23csoBC/w1UtRC4zP2MJOByVS0TkQLg9Gb7zg2k1lD0+Mw1rCsq5+XvjyM5zgbEG2M6n0CboV7AOauoA84AXsYZoNea+cBgEckRkRjgauAD/w1EJNNvcsIHgOfd158A57iX6KYB57jLws7izaU8M28dV4/tx6lDuntdjjHGtCjQsIhX1VmAqOpGVf0VcGZrO6hqHXA7zpf8SmCqqi4XkUdE5GJ3s9OB1SLyHdAT+K27bwnwPziBMx94xF0WVhqan3p2i+PnF9jVT8aYzivQS26q3DOANSJyO07fQo+2dlLV6cD0Zsse8ns9DZh2gH2fp/FMIyz9ZdYa1uzYy4s3j6WbNT8ZYzqxQM8sfgokAD8BxgCTgC4zUC4YlhaU8vS89VyZm8XpQ9vMXWOM8VSbZxbu4LorVfUeYC9wc9CrCnPVdc7gu+5JsfzigubjFI0xpvNp88xCVeuBMf4juM3heWL2WlZv38PvLjualHhrfjLGdH6B9lksAt4XkbeA8oaFqvpOUKoKY99uKePvc9dx+bFZnDHMmp+MMaEh0LBIB4ppegWUAhYWB6Gmzsfdby0hIzGGhy605idjTOgIdAS39VO0gyfmrGXVtj08d0MuKQnW/GSMCR2BjuB+gZan2/h+u1cUppYXlvH3OWu5bHRfzhrR0+tyjDHmoATaDPUvv9dxOJP+FR5gW9NMbb2Pu99aSlpiDA9dZM1PxpjQE2gz1Nv+70VkCjAzKBWFob/PWcfKrbt59oZcUhNavQ2IMcZ0SoEOymtuMNC/PQsJVysKd/PX2WuYOKoPZ1vzkzEmRAXaZ7GHpn0W23DucWFaUVvv455pS0hNiOHhi470uhxjjDlkgTZDJQe7kHD01Nx1LC/czdPXjyEt0ZqfjDGhK6BmKBG5VERS/N6nisjE4JUV+tYV7eUvs9dw0TF9OPfIXl6XY4wxhyXQPouHVbWs4Y2qlgIPB6ek8PDRsq3U1isPXmhTjxtjQl+gYdHSdoFedtslfZ1fwrBeyfRIjmt7Y2OM6eQCDYsFIvInERkkIgNF5P+AhcEsLJTV1ftYuHEX43LSvS7FGGPaRaBhcQdQA7wJTAUqgduCVVSoW164m4qaegsLY0zYCPRqqHLg/iDXEjby8p07wI7LtrAwxoSHQK+G+lREUv3ep4nIJ8ErK7R9nV9CTmYiPbpZf4UxJjwE2gyV6V4BBYCq7iKAe3B3RT6fMn9DiZ1VGGPCSqBh4RORfdN7iEg2LcxCa+C7HXsoq6y1/gpjTFgJ9PLXXwCfi8hn7vtTgcnBKSm0zW/or7CwMMaEkUA7uD8WkVycgFgMvI9zRZRp5uv8EnqnxJGVFu91KcYY024CnUjwB8CdQBZOWBwPfEnT26x2eapKXn4JJwzKQES8LscYY9pNoH0WdwJjgY2qegYwGigKWlUhamNxBTv2VFsTlDEm7AQaFlWqWgUgIrGqugoYGryyQlPD+IrjLCyMMWEm0A7uAnecxXvApyKyC7ut6n6+zi8hPTGGQd2TvC7FGGPaVaAd3Je6L38lInOAFODjoFUVovI2FDMuO936K4wxYeegZ45V1c/a3qrrKSytZHNJJTefmON1KcYY0+4O9R7cppn5G2x8hTEmfFlYtJOv80tIjo1ieO9uXpdijDHtzsKineTll5CbnUZkhPVXGGPCj4VFOyjeW83aHXsZa01QxpgwZWHRDuZv2AXY+ApjTPiysGgHefklxEZFcHTf1LY3NsaYEGRh0Q7yNhRzbP80YqLsj9MYE56C+u0mIhNEZLWIrBWR/W7LKiL9RWSOiCwSkaUicr67PFtEKkVksft4Kph1Ho7dVbWsKNxtl8waY8LaQQ/KC5SIRAJ/A84GCoD5IvKBqq7w2+yXwFRVfVJERgDTgWx33TpVHRWs+trLwo278Kn1VxhjwlswzyzGAWtVdb2q1gBvAJc020aBhoEJKYTgfFN5+SVERQij+6d5XYoxxgRNMMOiL7DZ732Bu8zfr4BJIlKAc1Zxh9+6HLd56jMROaWlDxCRySKyQEQWFBV5M2N6Xn4JI7NSiI+J9OTzjTGmIwQzLFoandb8vt3XAC+qahZwPvCKiEQAW4H+qjoa+G/gdRHZb2i0qj6jqrmqmtu9e/d2Lr9tlTX1LC0oZVxORod/tjHGdKRghkUB0M/vfRb7NzP9FzAVQFW/BOKATFWtVtVid/lCYB0wJIi1HpJFm3f2xxcwAAAVJUlEQVRRW6/WX2GMCXvBDIv5wGARyRGRGOBq4INm22wCxgOIyHCcsCgSke5uBzkiMhAYDKwPYq2HJC+/BBEYk239FcaY8Ba0q6FUtU5Ebgc+ASKB51V1uYg8AixQ1Q+AnwHPishdOE1UN6mqisipwCMiUgfUAz9S1ZJg1Xqo8vJLGN6rG93ior0uxRhjgipoYQGgqtNxOq79lz3k93oFcFIL+70NvB3M2g5XTZ2Pbzbt4uqx/b0uxRhjgs6GHB+ibwvLqKr1WX+FMaZLsLA4RHn5TquYzTRrjOkKLCwOUV5+CYO6J5KZFOt1KcYYE3QWFoeg3qfM31Bi4yuMMV2GhcUhWLVtN3uq6qy/whjTZVhYHIKG/gqbadYY01VYWByCvPwSstLi6ZMa73UpxhjTISwsDpKqkpdfYmcVxpguxcLiIK0rKqe4vIZx2W2ERUUJbF8B2nzuRGOMCT1BHcEdjg7YX6EKO1bCdx/Dmhmw+WtQH5x8F4x/GKSlSXiNMSY0WFgcpLz8YjKTYsnJTITaSsifB9994gREmXv7jl5Hw8n/DXu2wef/BzUVMOFRiOikJ3JlW+DtH0DNHhgxEY68FDIGeV2VMaYTsbA4SJvyv+PutBXI6087QVFXBdEJMPAMOPVuGHwOdOvjbKwK8anw5RNQWw4X/QUiOtlNknasglcvg6rd0GMYzP4f59HraCc0Rky04DDGWFi0qb4OCvLgu0+oXfUx71SvgiKgLhuOvRGGnAMDTobouP33FYFzfuOEybzHoLYKLn0KIjvJLLUbv4QpV0NkDNz8IfQ+BsoKYMX7sPw9mPWI8+g1Eo6caMFhTBcmGiYdsLm5ubpgwYL2OVhFCayd6TQvrZ0JVaUQEcWO9GN5ZutgrrnuBwwaPvrg+iE+/z+Y+SsYegF87wWI8niakJX/dJqeuvWF69+BtOz9t9kXHO9CwXxnWa+RzhnHkRMhfWCHlmyMaX8islBVc9vczsICp7lo+/LGzumC+U7ndEKm06w05BwYdCb3f7iR6cu2svihc4iIOIQO66+fho/uhUHj4apXISbh0Oo9XPP/AdPvhj7HwrVTITGAaUtKNzvBseK9xuDofYzbx9HBweGrd/qLYpM67jONCVMWFoEq3QTPT4DdW5z3vY+BIRNg8LnQZ3STTukz/zCXgd0Tee7GsYde6DevwAd3wIAT4do3ITb50I91sFRhzm9h3u+dEPzeixCTePDHaQiO5e/CFvfPvPcxjX0c6TmHXmNdDezdBrsLnb+T3YX7v96zDbQeUvs7f0e9RznPfUZBvN210JiDYWERKF89vHcrZJ8ER5wN3Xq3uNmOPVWM++0sfn7+MCafepjt9sumwTuTnS+3SW93zBdcfR38605Y9CqMngQX/hki26HLqnRTYx/HvuAY1djH4R8ctVWwp7DlAGh4vXcHzk0T/UQnOhcNdOvjNJt16wPR8bD9WyhcDLvyG7dNy/YLj9FOiMWnHv7PaUyYsrBoZx8u3cptr3/De7edxKh+7fDls+pDeOsmyBwK178LSd0P/5gHUlMOb90Maz6BU++BM34RnHEf+4LjXdiy0FnW82jnefcWqGzhzrhxKY0B4B8G/q9ju7Veb0UJbF0CWxdD4SInQEo3Nq5PH9j07KP3Mc7nGmMsLNrbw+9/y9QFBSz91TlER7bTeIm1M+GN65zmlBveb7zktj2VF8PrVzpf3hf8Acb+oP0/oyW7NjrBsfZT98ygd7Mw6AvJvYPX71BR4hcei6BwCZRtalyfcUTTAOk1EuK6BacWYzoxC4t2NuHxeWQmxfLqD45r3wNv+BxevwoSM+GGDyBtQPsde9cGePVyp4/hin/A8Iva79ihqLwYti5qPPsoXAy7C9yV4gRI1ljIORUGnhac8Damkwk0LGycRQBKK2pYvX0P5x/dcn/GYck+2TmrePUyeOF853XmEYd/3K1L4LXvOYMGb3gfBpxw+McMdYkZcMRZzqPB3iL3DGQxFH7jNNUted1ZlzHYCY2c0yDnFOs8N12ahUUAFmzYhWoQ71+RlQs3/gtemQgvnOd8ufcccejHWz8X3pjktMt//wNnZLZpWVJ3GHy28wDw+ZyO8/zPnBH6i6fA/OcAcfo6GsKj/wneXfpsjAcsLAKQt6GEmMiI9unYPpDeI+Hmj+Cli+HFC5yBcn1GH/xxlk2Dd38EmYPhummQ0rf9aw1nERHO30XvkXDiHVBf6/T3rP/MCZAv/w7/+bMz6j1rXGN49D2284zMNyYIrM8iABP/9h+iI4W3fnRiUI7fRMl6eOkSZ9T4dW9B/+MD3/eLJ2DGL2DASXD163bJaDDUlMOmLxvDY+tSQCEm2Rk70xAePUZ03okjjfFjfRbtpLy6jm+3lHHLaR00Qjl9INw8HV6+BF65FK55w/kCao3PB58+6ExYOPxiuOzZlueqMocvJrFpv0dFCWz4t9P0t/4zp88DnNH/DR3lOac54z9smnoTwiws2rBoUyl1PmVcTgBTYrSX1H5Ok9QrE51O6qtegSHntrxtXQ2892P4dhqM/SGc97+db2bbcJaQDiMucR7gzKeVP6/xzGP5O87yqHhI7O70kSR2d65+S+zhvvZf3sM5Znv/HapC9W6oKHauCqto/tjpBF9FMVTvdQY9xiQ4lz3HJDiTYcYkOo+G102eG7ZNbLpfVLydYYUJC4s25OUXEyEwZkAHXwmT3BNu+tAJjDeucy59bfhCalC1G6Ze7/xWO/4h5x4a9turt1KyYNS1zkMVdq5xQmPXBijfCeVFzgDFrUuc1766Fg4ikJABST3cUHFDpOF1khsy8WlQs9f9si9xjr9fCPg9WvwsICLaOXZChhNUCZlQV+nch6Vyl/NcW+E0wdWUO1OtHIzohMZHVAxExrrP7iMq1m+Z/3Os3zbNlu17joPuQ50zcvu3H1QWFm34Or+Eo/qmkBTrwR9VQjrc+E/n7OKtm2Dik3DM1c66PdvhtcudW7dOfNL5cjKdiwh0H+I8WqLqfBmX74TyHU54lO90pjwpL2p8FC5yllfvDuRD3S/8DOeRPtC52i4hs3HZvke6ExIxSYF/0apCfY0TGrUVTpDU7G18XVu+f7g0vK6rgrpqZ/+6aqivdi4gqChvuqyupunzgULOX3Ifp9kv51TnMufU/oH9PCZgFhatqK6rZ9HmUm44vh0Hyh2suBSY9A68cY1zlVNtBWSfCq9e6nyBXPtm42WfJrRIwxd7+oEDxV9tZePZSXmRczYRm9w0AOJTg9sMKeL8Vh8VCwTpUvLmfD4nTJoEiRs6tRXOWVr+PGdGhKVvOPukZTvBke2GR3Kvjqk1jFlYtGJpQRk1dT7GBmt8RaBik5ypxKfeAP+6y7nyJioWbvoX9B3jbW2m40THO/1Zqf28rqRjRURARNyBL9roOwZyv++c9exY6QRH/jxnuplvXna2yRzqhEbOqZB9ihPQwVZf69xqedcGKMl3nqt3O2d7mUOdy9vTskOmj9HCohV5+c7Ed2OzPQ4LcL4ornoN3r/VuVzzmil21zpj/Ik4g1l7joDjf+TMKL1taWN4+A+w7HWUe9ZxqnPJ86HOC1ZZ6oTALjcM/IOhrKBp/05kjNPk5z+hZmSMM81M5mA3QNxmy4wjDu32AUFk4yxaccPzeWwrq2TGXW1cutrRVK0zz5iDVV8LW75xw+Mz2JznNGlJpDOZZEOfR7/jG0fn++qdCxIaAqB5KFSVNv2MhAznbCEtx3lOd5/Tsp1+lYgIp/mweC0UrYad3zkXQexc7RxTfY3HSunnhEfmECdMurthkti9Xf//20SCh6mu3scxv57Bpcf25TcTj2634xpjOonaKijIazzz2LLQ6UyPiIZeRztBULoZfLWN+0REOZ3nDQGQ5hcGadmHN3NxXTUUr2saIA2vaysat4tLbQyR7g3Pww75pmM2KO8wrdi6m/Ka+o4dX2GM6TjRcY1nE+CML9n0lXPWUbjICYURlzQNhW592+emYS2Jim1sRvPn8zlnN01CZI0z/f/iV51teo+CWz4LTl0N5QX16CGsob9iXGforzDGBF9sEgw+y3l0JhERjRc2HDG+6brKXbBzrXNlWJBZWBxAXn4JAzIS6JVi02YYYzqp+DToN7ZDPsrG4bfA51PmbyixswpjjHEFNSxEZIKIrBaRtSJyfwvr+4vIHBFZJCJLReR8v3UPuPutFpEDTIwUHGuL9rKrojZ4968wxpgQE7RmKBGJBP4GnA0UAPNF5ANVXeG32S+Bqar6pIiMAKYD2e7rq4EjgT7ATBEZonqwk9Icmq/d/orjrHPbGGOA4J5ZjAPWqup6Va0B3gCazYSHAg3XmqUAhe7rS4A3VLVaVfOBte7xOkRefgk9u8XSLz2+oz7SGGM6tWCGRV9gs9/7AneZv18Bk0SkAOes4o6D2BcRmSwiC0RkQVFRUbsUrark5RczLicDsYFvxhgDBDcsWvqmbT4C8BrgRVXNAs4HXhGRiAD3RVWfUdVcVc3t3r37YRcMsKmkgu27q62/whhj/ATz0tkCwH/Gsywam5ka/BcwAUBVvxSROCAzwH2DorG/wsLCGGMaBPPMYj4wWERyRCQGp8P6g2bbbALGA4jIcCAOKHK3u1pEYkUkBxgM5AWx1n3y8ktIS4jmiO5JHfFxxhgTEoJ2ZqGqdSJyO/AJEAk8r6rLReQRYIGqfgD8DHhWRO7CaWa6SZ3JqpaLyFRgBVAH3NZRV0Ll5ZcwNjudiAjrrzDGmAZBHcGtqtNxOq79lz3k93oFcNIB9v0t8Ntg1tfc1rJKNpVUcMMJHt7syBhjOiEbwe0nz8ZXGGNMiyws/MzfUEJSbBTDeyd7XYoxxnQqFhZ+8vJLGDMgjahI+2Mxxhh/9q3oKimv4bvte218hTHGtMDCwjV/g3v/CgsLY4zZj4WFKy+/hJioCEZmpXhdijHGdDoWFq68/BJG90slNirS61KMMabTsbAA9lTVsrywzKb4MMaYA7CwABZu3IVPYZyNrzDGmBZZWOA0QUVFCMcOSPW6FGOM6ZQsLHDC4qi+KSTEBHX2E2OMCVldPiyqautZUlBq/RXGGNOKLh8Wu6tqOf/o3pw2pH1unmSMMeGoy7e79EiO489Xj/a6DGOM6dS6/JmFMcaYtllYGGOMaZOFhTHGmDZZWBhjjGmThYUxxpg2WVgYY4xpk4WFMcaYNllYGGOMaZOoqtc1tAsRKQI2HsYhMoGd7VROsIVSrRBa9YZSrRBa9YZSrRBa9R5OrQNUtc0pLMImLA6XiCxQ1Vyv6whEKNUKoVVvKNUKoVVvKNUKoVVvR9RqzVDGGGPaZGFhjDGmTRYWjZ7xuoCDEEq1QmjVG0q1QmjVG0q1QmjVG/Rarc/CGGNMm+zMwhhjTJssLIwxxrSpy4eFiEwQkdUislZE7ve6ntaISD8RmSMiK0VkuYjc6XVNbRGRSBFZJCL/8rqWtohIqohME5FV7p/xCV7XdCAicpf7b+BbEZkiInFe1+RPRJ4XkR0i8q3fsnQR+VRE1rjPaV7W2OAAtf7e/XewVETeFZFUL2v011K9fuvuFhEVkcz2/twuHRYiEgn8DTgPGAFcIyIjvK2qVXXAz1R1OHA8cFsnrxfgTmCl10UE6M/Ax6o6DDiGTlq3iPQFfgLkqupRQCRwtbdV7edFYEKzZfcDs1R1MDDLfd8ZvMj+tX4KHKWqI4HvgAc6uqhWvMj+9SIi/YCzgU3B+NAuHRbAOGCtqq5X1RrgDeASj2s6IFXdqqrfuK/34HyZ9fW2qgMTkSzgAuA5r2tpi4h0A04F/gGgqjWqWuptVa2KAuJFJApIAAo9rqcJVZ0HlDRbfAnwkvv6JWBihxZ1AC3VqqozVLXOffsVkNXhhR3AAf5sAf4PuBcIylVLXT0s+gKb/d4X0Im/fP2JSDYwGvja20pa9TjOP16f14UEYCBQBLzgNps9JyKJXhfVElXdAvwB5zfIrUCZqs7wtqqA9FTVreD84gP08LieQH0f+MjrIlojIhcDW1R1SbA+o6uHhbSwrNNfSywiScDbwE9VdbfX9bRERC4EdqjqQq9rCVAUcCzwpKqOBsrpPM0kTbht/ZcAOUAfIFFEJnlbVXgSkV/gNP++5nUtByIiCcAvgIeC+TldPSwKgH5+77PoZKfzzYlINE5QvKaq73hdTytOAi4WkQ04zXtnisir3pbUqgKgQFUbztSm4YRHZ3QWkK+qRapaC7wDnOhxTYHYLiK9AdznHR7X0yoRuRG4ELhOO/eAtEE4vzgscf+/ZQHfiEiv9vyQrh4W84HBIpIjIjE4nYQfeFzTAYmI4LSpr1TVP3ldT2tU9QFVzVLVbJw/19mq2ml/+1XVbcBmERnqLhoPrPCwpNZsAo4XkQT338R4OmlnfDMfADe6r28E3vewllaJyATgPuBiVa3wup7WqOoyVe2hqtnu/7cC4Fj333S76dJh4XZg3Q58gvOfbaqqLve2qladBFyP81v6YvdxvtdFhZE7gNdEZCkwCvh/HtfTIvfsZxrwDbAM5/9xp5qaQkSmAF8CQ0WkQET+C3gUOFtE1uBctfOolzU2OECtTwDJwKfu/7OnPC3SzwHqDf7ndu6zK2OMMZ1Blz6zMMYYExgLC2OMMW2ysDDGGNMmCwtjjDFtsrAwxhjTJgsLYzoBETk9FGbmNV2XhYUxxpg2WVgYcxBEZJKI5LkDtZ5279exV0T+KCLfiMgsEenubjtKRL7yuydCmrv8CBGZKSJL3H0GuYdP8rufxmvu6GxjOgULC2MCJCLDgauAk1R1FFAPXAckAt+o6rHAZ8DD7i4vA/e590RY5rf8NeBvqnoMzpxOW93lo4Gf4txbZSDOiH1jOoUorwswJoSMB8YA891f+uNxJsPzAW+627wKvCMiKUCqqn7mLn8JeEtEkoG+qvougKpWAbjHy1PVAvf9YiAb+Dz4P5YxbbOwMCZwArykqk3umiYiDzbbrrU5dFprWqr2e12P/f80nYg1QxkTuFnAFSLSA/bdU3oAzv+jK9xtrgU+V9UyYJeInOIuvx74zL3/SIGITHSPEevej8CYTs1+czEmQKq6QkR+CcwQkQigFrgN50ZJR4rIQqAMp18DnGm4n3LDYD1ws7v8euBpEXnEPcb3OvDHMOaQ2KyzxhwmEdmrqkle12FMMFkzlDHGmDbZmYUxxpg22ZmFMcaYNllYGGOMaZOFhTHGmDZZWBhjjGmThYUxxpg2/X/dw+KZhxnWRgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(history.history.keys())\n",
    "# summarize history for accuracy\n",
    "plt.plot(history.history['acc'])\n",
    "plt.plot(history.history['val_acc'])\n",
    "plt.title('model accuracy')\n",
    "plt.ylabel('accuracy')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'test'], loc='upper left')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0xb29a21cf8>"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xl8VNX5x/HPM5OVrJCENYTVDRABI6K4gCsoiq3WBXGrLbZaa39trdJFra2tta1VWzdUilu1blVUVERA3EABkU2URSBhC2tIQtaZ5/fHGeIQSAiQyZ1JnvfrldfcmXvn3ica5jvnnnvPEVXFGGOMAfB5XYAxxpjoYaFgjDGmloWCMcaYWhYKxhhjalkoGGOMqWWhYIwxppaFgjGNJCKTROSPjdx2tYiccaj7Maa5WSgYY4ypZaFgjDGmloWCaVFCp21uFpGFIlImIk+ISAcReUtESkRkmoi0Ddv+fBFZIiI7RGSmiBwVtm6giMwPve+/QFKdY40SkQWh934sIv0PsuYfisgKEdkmIpNFpHPodRGRf4hIkYgUh36nfqF154jI0lBt60Tklwf1H8yYOiwUTEt0IXAmcDhwHvAW8GsgG/c3/1MAETkceA74GZADTAFeF5EEEUkAXgWeBtoBL4b2S+i9g4CJwHVAFvAoMFlEEg+kUBE5DfgzcDHQCVgDPB9afRZwSuj3yAQuAbaG1j0BXKeqaUA/YPqBHNeY+lgomJbon6q6SVXXAR8Ac1T1c1WtBP4HDAxtdwnwpqq+q6rVwN+AZOBEYAgQD9ynqtWq+hLwWdgxfgg8qqpzVDWgqk8ClaH3HYjLgYmqOj9U33jgBBHpDlQDacCRgKjql6q6IfS+aqCPiKSr6nZVnX+AxzVmnywUTEu0KWy5fB/PU0PLnXHfzAFQ1SBQAHQJrVune44YuSZsuRvwi9Cpox0isgPoGnrfgahbQymuNdBFVacD/wIeBDaJyAQRSQ9teiFwDrBGRN4XkRMO8LjG7JOFgmnN1uM+3AF3Dh/3wb4O2AB0Cb22W17YcgFwl6pmhv20UdXnDrGGFNzpqHUAqvqAqh4L9MWdRro59PpnqjoaaI87zfXCAR7XmH2yUDCt2QvAuSJyuojEA7/AnQL6GPgEqAF+KiJxIvJdYHDYex8DfiQix4c6hFNE5FwRSTvAGv4DXCMiA0L9EX/Cne5aLSLHhfYfD5QBFUAg1OdxuYhkhE577QQCh/DfwZhaFgqm1VLVr4CxwD+BLbhO6fNUtUpVq4DvAlcD23H9D6+EvXcurl/hX6H1K0LbHmgN7wG/A17GtU56AZeGVqfjwmc77hTTVly/B8AVwGoR2Qn8KPR7GHPIxCbZMcYYs5u1FIwxxtSyUDDGGFPLQsEYY0wtCwVjjDG14rwu4EBlZ2dr9+7dvS7DGGNiyrx587aoas7+tou5UOjevTtz5871ugxjjIkpIrJm/1vZ6SNjjDFhLBSMMcbUslAwxhhTK+b6FPalurqawsJCKioqvC4l4pKSksjNzSU+Pt7rUowxLVCLCIXCwkLS0tLo3r07ew5q2bKoKlu3bqWwsJAePXp4XY4xpgVqEaePKioqyMrKatGBACAiZGVltYoWkTHGGy0iFIAWHwi7tZbf0xjjjRYTCsYY06LNvBsKPo34YSIWCiIyUUSKRGRxPesvF5GFoZ+PReSYSNUSaTt27OChhx464Pedc8457NixIwIVGWNalMK5MPPPsHJGxA8VyZbCJGBEA+u/AU5V1f7AH4AJEawlouoLhUCg4cmwpkyZQmZmZqTKMsa0BKrw7u2QkgMn3BDxw0Xs6iNVnSUi3RtY/3HY09lAbqRqibRbb72VlStXMmDAAOLj40lNTaVTp04sWLCApUuXcsEFF1BQUEBFRQU33XQT48aNA74dsqO0tJSRI0dy0kkn8fHHH9OlSxdee+01kpOTPf7NjDGeWzEN1nwI5/wNElMjfrhouST1WuCt+laKyDhgHEBeXl59mwHw+9eXsHT9ziYtrk/ndG4/r2+96++++24WL17MggULmDlzJueeey6LFy+uvWx04sSJtGvXjvLyco477jguvPBCsrKy9tjH8uXLee6553jssce4+OKLefnllxk71mZYNKZVCwZh2h3QtjsMuqpZDul5R7OIDMeFwi31baOqE1Q1X1Xzc3L2O8if5wYPHrzHfQQPPPAAxxxzDEOGDKGgoIDly5fv9Z4ePXowYMAAAI499lhWr17dXOUaY6LVohdh02I47XcQl9Ash/S0pSAi/YHHgZGqurUp9tnQN/rmkpKSUrs8c+ZMpk2bxieffEKbNm0YNmzYPu8zSExMrF32+/2Ul5c3S63GmChVUwkz/ggd+0Pf7zbbYT1rKYhIHvAKcIWqfu1VHU0hLS2NkpKSfa4rLi6mbdu2tGnThmXLljF79uxmrs4YE5PmToQda+GMO8DXfB/VEWspiMhzwDAgW0QKgduBeABVfQS4DcgCHgrdkFWjqvmRqieSsrKyGDp0KP369SM5OZkOHTrUrhsxYgSPPPII/fv354gjjmDIkCEeVmqMiQkVO2HWX6HHqdDrtGY9tKhqsx7wUOXn52vdSXa+/PJLjjrqKI8qan6t7fc1ptWZfhfMugd+OAO6DGqSXYrIvMZ88fa8o9kYY0yYkk3wyYPQ9ztNFggHwkLBGGOiyax7IFDprjjygIWCMcZEi60rYd4kd09CVi9PSrBQMMaYaDH9j+BPgFPrvW0r4iwUjDEmGqz/HJa84sY3Suuw/+0jxELBGGOiwbQ7ILkdnPhTT8uwUGgCBzt0NsB9993Hrl27mrgiY0xMWTkDVs2EU26GpHRPS7FQaAIWCsaYg7Z70LuMPDjuWq+riZpRUmNa+NDZZ555Ju3bt+eFF16gsrKS73znO/z+97+nrKyMiy++mMLCQgKBAL/73e/YtGkT69evZ/jw4WRnZzNjRuQn0DDGhKmparaB5uq19H+wYQF851GIS9z/9hHW8kLhrVth46Km3WfHo2Hk3fWuDh86e+rUqbz00kt8+umnqCrnn38+s2bNYvPmzXTu3Jk333wTcGMiZWRkcO+99zJjxgyys7ObtmZjTP2KC+H1m2DtbBjzAnQf6k0dgWp47w/Qvi8c/T1vaqjDTh81salTpzJ16lQGDhzIoEGDWLZsGcuXL+foo49m2rRp3HLLLXzwwQdkZGR4XaoxrY8qzP03PDgE1nwMbdrBs9+DNZ94U8+8SbD9m9Cgd35vaqij5bUUGvhG3xxUlfHjx3PdddfttW7evHlMmTKF8ePHc9ZZZ3Hbbbd5UKExrdT21TD5RvhmFvQ4Bc57AOKTYdK58OxFcMX/oOvg5qunshTevwe6DYXDzmy+4+6HtRSaQPjQ2WeffTYTJ06ktLQUgHXr1lFUVMT69etp06YNY8eO5Ze//CXz58/f673GmAgIBmHOBHjoRFj3OYy6D66cDO16QFpHuOp1SG0PT38XCufuf39NZfZDUFYEZ/we3EjRUaHltRQ8ED509siRIxkzZgwnnHACAKmpqTzzzDOsWLGCm2++GZ/PR3x8PA8//DAA48aNY+TIkXTq1Mk6mo1paltXwms/gbUfQ6/T4bz7IbPrntukd4ar3oBJ57hguPLVyA9EV7YFPnoAjhwFXY+L7LEOkA2dHYNa2+9rzAELBtw38el/dFf0nP1nGDCm4W/kOwpcMFQUu5ZE5wGRq++tW+HTR+H6OZBzeOSOE8aGzjbGtE5Fy+CJs2Dqb90ENdfPgYGX7/8UTWZX12JITIenRsOGhZGpb/tq+OxxGDi22QLhQFgoGGNahkANfPB3ePRk2LYKLnwCLv0PpHdq/D7adnN9DAmpLhg2LWn6Omf8yV1pNGx80++7CbSYUIi102AHq7X8nsYckI2L4fHT4L074Yhz4IY5cPRFB9eB264HXDUZ4pLgyfOh6MsmrHMRLHwBjv+R68uIQi0iFJKSkti6dWuL/8BUVbZu3UpSUpLXpRgTHWqqYObdMGEY7FwPFz8FFz/priY6FFm9XIvBFwdPngebv2qScpn2e0jKgJN+1jT7i4AWcfVRbm4uhYWFbN682etSIi4pKYnc3FyvyzDGe+s/d1cWbVoMR18MI//ibkZrKtm9XTBMOtcFw9VvQvZhB7+/bz6AFe/CmXdCctumq7OJtYirj4wxrUh1Bbz/F/jofkjJgVH/gCPPidzxipbBk6NA/HDNlIObEU0VHj8dSjbCjfPcTXPNzK4+Msa0PIVz4dFT4MN74ZjLXN9BJAMBoP2R7hLVYDVMGuU6sQ/Ul6/Dunmuc9mDQDgQFgrGmOhXXQ7v/AaeOBOqymDsy3DBg5Cc2TzH79DHBUNNOUw6z11W2liBGtcBnn2EC7IoF7FQEJGJIlIkIovrWS8i8oCIrBCRhSIS4VsIjTExpabSnYd/7054aAh88i849mq4/hPofUbz19OxH1z5GlSVumDYsbZx71vwDGxdDmfcDv7o78aNZIWTgH8BT9WzfiRwWOjneODh0KMxpjVShaKloVnIZrhRTKt3uXP5XQe7Aex6nuptjZ2OccNgPDXanUq6ZgpkNHDhR9UumPFn6Hq8u1Q2BkQsFFR1loh0b2CT0cBT6nq6Z4tIpoh0UtUNkarJGBNldm5w01CuCk1HWbrJvZ51mLvjt+dw6H6S51NU7qHzQDei6lMXfBsM9d1zMOcRKN0I3/t3VA161xAv2zJdgIKw54Wh1ywUjGmpKkthzUcuAFbOgM2hG8PaZEPPYdBruHts6Nt3NOhyLIx9BZ7+jguGq9/c+87pXdvgw/vg8BHQ7URv6jwIXobCvmJzn9fHisg4YBxAXl5eJGsyxjSlYMDdT7D7lFDBp+4qnrgkyDsBBlzmWgMd+oEvxq576Xqc6/B+5rvf3seQ1uHb9R/eC5U74fTbvavxIHgZCoVA+Bi2ucD6fW2oqhOACeDuU4h8acaYg7Zt1bch8M0sN+ooQMf+cML1LgTyhkT9pZmNknc8XP4iPHPRt8GQmuNGXJ0zwV1t1KGP11UeEC9DYTLwExF5HtfBXGz9CcbEqGAQlr7qZhLbfUooPReOOs+FQM9hkNJC5yHvdiJc/oILhqfOd3dBzwzNADn8197WdhAiFgoi8hwwDMgWkULgdiAeQFUfAaYA5wArgF3ANZGqxRgTIaqwfKqbfH7TIsg5Ekbe44aszuodM52rh6z7STDmv/Cfi2HiCNi2EoZcv/eEPjEgklcfNXiXRuiqoxsidXxjTITtvoeg8FNo2x2+M8GNTBolE9A3u56nwmXPwX8udUNvn/wLrys6KNF/J4UxJroUzoPpd7oriNI6u7GHBl4B/nivK/Ner9Pg2qkQqG7awfmakYWCMaZxNi2B6XfBV29Cmyw46y447tqW0WHclCI5jWczsFAwxjRs60qY+WdY9BIkpsHw38CQH7tl0+JYKBhj9q240F1N9Pkz4E+AoTe5nxg9LWIax0LBGLOn0s1uruO5T7iri477ges0Db8xy7RYFgrGGKd8O3z8T5j9iBsiesAYOPUWyLRRBFoTCwVjWrvKUjdw28cPuLuP+37X3XR1KFNPmphloWBMa1VdAfP+7U4VlW12A7cN/w106u91ZcZDFgrGtDbBIHzxHMz4E+wshO4nw6X/cXMWmFbPQsGY1mTtHHj7FjdyaedBbkrLnsO8rspEEQsFY1qD4nUw7XZY9CKkdYLvPgb9Loq94apNxFkoGNOSVe1yVxR9dJ+b2+CUm2HozyAx1evKTJSyUDCmJVKFJa/Au7dDcQH0uQDOvBPadvO6MhPlLBSMaWnWL4C3b4W1n0CHo+E7j7ihnY1pBAsFY1qK0iKY/geY/7QbsO68+93opa11KGtzUCwUjIl1NVXu5rNZf4XqXXDCDa7vIDnT68pMDLJQMCZWqcLX78A7v3YzfR12Npx9l92JbA6JhYIxsahoGbwzHlZOh+zD4fKX4bAzvK7KtAAWCsbEkvLtblL4Tx9zUz6OuNuNYmqznpkmYqFgTCwI1Lhximb8CSp2wLFXu3GKUrK9rsy0MBYKxkS7Ve+7S0yLlrpxikbcDR37eV2VaaEsFIyJRtXlsPQ1mDsRCuZAZje4+Gk46jwQ8bo604JZKBgTTbYsh7n/hi/+4/oPsnrD2X+G/O9DfJLX1ZlWwELBGK/VVMGy110YrP4AfHGuRZD/fXe6yFoGphlFNBREZARwP+AHHlfVu+uszwOeBDJD29yqqlMiWZMxUWPbNzBvEix41k1yk5kHp98OA8dCanuvqzOtVMRCQUT8wIPAmUAh8JmITFbVpWGb/RZ4QVUfFpE+wBSge6RqMsZzgRr4+m3XV7DyPRAfHHEOHHsN9DrNhrI2notkS2EwsEJVVwGIyPPAaCA8FBRIDy1nAOsjWI8x3ikuhPlPuZ+SDZDWGYaNd2MTZXTxujpjakUyFLoABWHPC4Hj62xzBzBVRG4EUoB93pIpIuOAcQB5eXlNXqgxEREMwIr3XKtg+TtuWIreZ8C598JhZ4HfuvRM9InkX+W+ese0zvPLgEmq+ncROQF4WkT6qWpwjzepTgAmAOTn59fdhzHRpWQTfP4UzHsKitdCSns46f9g0FU2n4GJepEMhUKga9jzXPY+PXQtMAJAVT8RkSQgGyiKYF3GND1V+GYWzH0Clr0JwRrocSqcdScccS7EJXhdoTGNEslQ+Aw4TER6AOuAS4ExdbZZC5wOTBKRo4AkYHMEazKmaam6Qelm/hkKP4PkdnD8j1zHcXZvr6sz5oBFLBRUtUZEfgK8g7vcdKKqLhGRO4G5qjoZ+AXwmIj8H+7U0tWqaqeHTGxY9b4bi6hgNqTnwqj74JjL7CYzE9Mi2tMVuudgSp3XbgtbXgoMjWQNxjS51R+5MFjzobuK6Ny/u6uI4hK9rsyYQ2aXPxjTWGtnuzD45n1I7QAj73Gdx9YyMC2IhYIx+1M414XByvcgJQfO/lNoLKJkryszpslZKBhTn3XzXQfy8qnQJgvOvNNNaJOQ4nVlxkSMhYIxdW1Y6MLgqymQ3NaNRzR4HCSmel2ZMRFnoWDMbpuWuDD48nVIyoDhv4Xjr4Ok9P2/15gWwkLBmKJl8P7dsOR/kJgOp94KQ34MyZleV2ZMs7NQMK3XluXw/l9g0Uuun+CUm2HI9dCmndeVGeMZCwXT+mxdCe/fA4tegLgkOOlncMKNkJLldWXGeM5CwbQelSUw66/wyUNudrMTboATb4LUHK8rMyZqWCiYlk8VFr0I797m5jIYcDmcfhukdfS6MmOijoWCadk2LoIpv4K1H0PngXDx09D1OK+rMiZqWSiYlmnXNphxl5vgJrktnPeAG5/Iprs0pkEWCqZlCQbclJfv3QkVO+C4H8Lw8S4YjDH7ZaFgWo6CT2HKL2HDF9BtqBuwrmM/r6syJqZYKJjYV7IJpt0BX/zHDWV94RPQ70KQfc0Ia4xpiIWCiV2BapjzKMy8G2oq3DzIJ//Sxigy5hBYKJjYtHIGvHULbPkKep8JI/8CWb28rsqYmGehYGLLjrXwzq/doHVtu8Nlz8PhI+xUkTFNxELBxIbqcvjoAfjwXkDgtN+6oSls1jNjmlSjQkFEbgL+DZQAjwMDgVtVdWoEazPG3Y381RR4+1bXSuhzAZz1R8js6nVlxrRIjW0pfF9V7xeRs4Ec4BpcSFgomMgIVLtpMGf91U2DmXMUXDkZep7qdWXGtGiNDYXdJ2zPAf6tql+I2Elc04RUYdsqWDkdVs2Eb2ZB5U43v8GIu900mP54r6s0psVrbCjME5GpQA9gvIikAcHIlWVahfId7sN/5XT3s2ONez0jD/p9F3oOh57DbLIbY5pRY0PhWmAAsEpVd4lIO9wpJGMaL1AN6+Z9GwLr5oEGISEVepwCJ94IvU6Ddj3taiJjPNLYUDgBWKCqZSIyFhgE3L+/N4nIiNB2fuBxVb17H9tcDNwBKPCFqo5pZE0m2tV3Skh80HmQu9Gs12mQm2+nhoyJEo0NhYeBY0TkGOBXwBPAU0C9vX4i4gceBM4ECoHPRGSyqi4N2+YwYDwwVFW3i0j7g/s1TNRozCmhHqfYlJfGRKnGhkKNqqqIjAbuV9UnROSq/bxnMLBCVVcBiMjzwGhgadg2PwQeVNXtAKpadGDlm6hQvh1mP+KuErJTQsbEtMaGQomIjAeuAE4OtQL2197vAhSEPS8Ejq+zzeEAIvIR7hTTHar6dt0dicg4YBxAXl5eI0s2zWLXNnjqfNi4GLoca6eEjIlxjQ2FS4AxuPsVNopIHvDX/bxnX18LdR/HPwwYBuQCH4hIP1XdscebVCcAEwDy8/Pr7sN4pWwLPDUatiyHsS9B7zO8rsgYc4gaNQ2Vqm4EngUyRGQUUKGqT+3nbYVA+G2nucD6fWzzmqpWq+o3wFe4kDDRrnQzPHkebF0BY563QDCmhWhUKISuEPoU+B5wMTBHRC7az9s+Aw4TkR4ikgBcCkyus82rwPDQMbJxp5NWNb5844nSInhyFGz7Bsb8150uMsa0CI09ffQb4LjdHcEikgNMA16q7w2qWiMiPwHewfUXTFTVJSJyJzBXVSeH1p0lIkuBAHCzqm49+F/HRFzJJtdCKC6Ay19wncnGmBajsaHgq3Nl0FYa0cpQ1SnAlDqv3Ra2rMDPQz8m2u3c4AJh53q4/EXofpLXFRljmlhjQ+FtEXkHeC70/BLqfNibFm7nepg0Cko3uU7lbid6XZExJgIaFQqqerOIXAgMxV1VNEFV/xfRykz0KF7n+hBKN8PYlyFviNcVGWMipNGT7Kjqy8DLEazFRKPiQtdCKNsCV7wCXQd7XZExJoIaDAURKWHvewvAtRZUVdMjUpWJDjvWukAo3w5XvupuSDPGtGgNhoKqpjVXISbKbF/jThlVFLtA6HKs1xUZY5qBzdFs9rbtG3eVUWUJXPkadB7odUXGmGZioWD2tG0VTDoPqkpDgTDA64qMMc3IQsF8a+tK14dQUwFXvQ6d+ntdkTGmmVkoGGfLcnfKKFDlAqFjP68rMsZ4wELBwOavXadyMABXvQEd+nhdkTHGIxYKrV3RMtdCQOHqN6D9UV5XZIzxkIVCa1b0pQsE8bkWQs4RXldkjPFYo4bONi3QpiWuU1n8cPWbFgjGGMBCoXXauMi1EPzxLhCybV4jY4xjodDabFjoAiEuKRQIvb2uyBgTRVpNn4KqsqBgBwPz2npdSvMKBmHTYlj9Iaz5CFbOgOS2cPXr0K6n19UZY6JMqwmFF+YWcMvLi3j0imM5u29Hr8uJnGDg2xBY/ZELgoodbl3b7tDvu3DKzdC2m6dlGmOiU6sJhdEDuvDsnLX8/L8L+N8NQzm8QwsZ6y8YcH0Eu1sCaz5yg9gBtO0BR50H3U+G7kMhI9fbWo0xUU/cjJixIz8/X+fOnXtQ791QXM55//yI1EQ/r91wEhlt4pu4umYQDMDGhWEtgY+hMhQC7Xq6KTK7nwzdhkJGF29rNcZEDRGZp6r7Hf++1bQUADplJPPI2EFc9thsfvLcfCZdMxi/T7wuq2GBmm9DYM3uENjp1rXrBX0v+LYlkN7Z21qNMTGvVYUCQH73dtw5uh/jX1nEPW8vY/w5UXgHb0UxLHwBlr8Laz/5NgSyDnN9ArtbAumdvK3TGNPitLpQALhscB5L1hfz6KxV9OmczugBUXKaZdMS+PQxFwjVZaEQuDB0SugkSGvBHeTGmKjQKkMB4LZRffl6Yym/emkhvXJS6dclw5tCAtXw5WT49HFY+7G7f6DfRTD4Bza5jTGm2bWqjua6tpRWcv4/PwRg8o0nkZ2a2CT7bZSdG2Dev2HeJCjd5C4Xzb8WBo6FNu2arw5jTKvQ2I7miN7RLCIjROQrEVkhIrc2sN1FIqIi0qwzw2enJvLoFflsLavi+mfnUx0IRvaAqq7D+IWr4L5+8P490LE/jHkRbvwchv7UAsEY46mInT4SET/wIHAmUAh8JiKTVXVpne3SgJ8CcyJVS0OOzs3gnov6c9PzC7jz9aX84YIITC5TWQIL/+tOEW3+EpIy4fgfwXHX2l3FxpioEsk+hcHAClVdBSAizwOjgaV1tvsDcA/wywjW4sb8+XQCZPWCrN7ucs52PSA+mdEDurBk/U4mzFpF387pXDo4r2mOufkr+OxxWPAcVJVAp2Pg/H+5zuOENk1zDGOMaUKRDIUuQEHY80Lg+PANRGQg0FVV3xCRekNBRMYB4wDy8g7yA7u4AL5+G8o2h+/Z3eXbrie3ZvWmW8c43pv8Of2SRtCv7zFuFNEDFaiBr99yAfTNLPAnQJ8LYPA4yM0HifL7IowxrVrEOppF5HvA2ar6g9DzK4DBqnpj6LkPmA5craqrRWQm8EtVbbAX+ZA7miuK3QT121a5x60rYFvocffwEICKH2nbzbUoalsXPd1jRi74/Hvut7QI5j8JcyfBzkJIz4X8a2DQVZCac/D1GmNME4iGO5oLga5hz3OB9WHP04B+wExx3547ApNF5Pz9BcMhScqALoPcTzhV2LWNtSsW8cjL7zAgZSsXdqjAv32Vu4u4uuzbbf0JblyhrN6Q1RNKNsKSVyFYDT2Hwci/wOEjwN9qr/g1xsSoSH5qfQYcJiI9gHXApcCY3StVtRjI3v28sS2FiBGBlCzyjhnGqXFHct3T85jTI5e/XdcfAffBv7tFUdvSWAErpkFcous0Pu4HNmGNMSamRSwUVLVGRH4CvAP4gYmqukRE7gTmqurkSB37UJ3dtyM3nX4Y97+3nL6d0/n+ST3ckBLpndydxeGCAdDgwfU/GGNMlIno+Q1VnQJMqfPabfVsOyyStRyom04/jC837OSuKV9yRMc0hvbO3veGPj8u84wxJvbZdJz18PmEey8ZQM/sFG74z3wKtu3yuiRjjIk4C4UGpCbG8diV+QSDyg+fmsuuqhqvSzLGmIiyUNiP7tkp/HPMIL7eVMLNLy4k1saKMsaYA2Gh0AinHp7DLSOO5M1FG3ho5kqvyzHGmIixUGikcaf05PxjOvO3qV8xfdkmr8sxxpiIsFBoJBHhLxf2p0+ndG56bgErikq9LskYY5qchcIBSE7wM+HKfBLifIx7ei47K6q9LskYY5qUhcIB6pKZzEOXD2KE0J4vAAASeElEQVTt1l387PkFBILW8WyMaTksFA7C8T2zuP28PkxfVsS9737ldTnGGNNkbMS2gzR2SDeWrN/JgzNWclSndEb17+x1ScYYc8ispXCQRITfj+7Lsd3acuNzn3Pba4spLrc+BmNMbLNQOASJcX4mXXMcV53QnWdmr+H0v7/PawvW2Q1uxpiYZaFwiNKS4rnj/L68dsNJdMlM4qbnFzD2iTms3GyXrBpjYo+FQhM5OjeDV64fyh8u6MfCwmJG3vcBf5/6FRXVAa9LM8aYRrNQaEJ+n3DFkG5M/8Uwzu3fiX9OX8FZ/5jFjK+KvC7NGGMaxUIhAnLSEvnHJQP4zw+PJ94vXPPvz/jxM/PYUFzudWnGGNMgC4UIOrFXNm/ddAo3n30E05cVccbf3+fxD1ZREwh6XZoxxuyThUKEJcT5uGF4b6b9/FSO75nFH9/8klH//JB5a7Z5XZoxxuzFQqGZdG3XhieuyueRscdSXF7NhQ9/wq0vL2R7WZXXpRljTC0LhWYkIozo15FpPz+Vcaf05MV5hZz295m8MLeAoI2hZIyJAhYKHkhJjOPX5xzFmz89iV45qfzqpYVcMuETvtpY4nVpxphWzkLBQ0d2TOeF607gnov6s6KolHMe+IA/T/mSskqbC9oY4w0LBY/5fMLF+V2Z/othfO/YXB6dtYoz732ftxdvtOEyjDHNLqKhICIjROQrEVkhIrfuY/3PRWSpiCwUkfdEpFsk64lmbVMSuPvC/rz84xNIT47nR8/M4/LH5/Dxii0WDsaYZhOxUBARP/AgMBLoA1wmIn3qbPY5kK+q/YGXgHsiVU+sOLZbO9648SRuG9WHFUWljHl8Dhc89DFTl2y0zmhjTMRFsqUwGFihqqtUtQp4HhgdvoGqzlDVXaGns4HcCNYTM+L8Pr5/Ug9m/Wo4d32nH9vLqhj39DxG3D+LVz9fZze/GWMiJpKh0AUoCHteGHqtPtcCb0WwnpiTFO/n8uO7Mf0Xp3L/pQMQhJ/9dwHD/z6TZ2avscH2jDFNLpKhIPt4bZ/nP0RkLJAP/LWe9eNEZK6IzN28eXMTlhgb4vw+Rg/owls3ncxjV+aTlZLIb19dzMn3zODR91dSalcrGWOaiESqE1NETgDuUNWzQ8/HA6jqn+tsdwbwT+BUVd3vcKL5+fk6d+7cCFQcO1SVT1Zt5eGZK/lg+RYykuO56sTuXHNid9qmJHhdnjEmConIPFXN3+92EQyFOOBr4HRgHfAZMEZVl4RtMxDXwTxCVZc3Zr8WCnv6omAHD81cwTtLNpEc72fM8Xn88OSedMxI8ro0Y0wU8TwUQkWcA9wH+IGJqnqXiNwJzFXVySIyDTga2BB6y1pVPb+hfVoo7NvyTSU8/P5KXluwHp/AhYNy+dGpveieneJ1acaYKBAVoRAJFgoNK9i2iwmzVvHfuQXUBIKc278zPz61F306p3tdmjHGQxYKrVxRSQUTP1zNM7PXUFpZw2lHtuf6Yb3I797O69KMMR6wUDAAFJdX8/Qnq5n40Wq2lVUxuEc7LhvclTOO6kBaUrzX5RljmomFgtnDrqoa/vtZAY9/8A3rdpSTEOdj+BE5jOrfmdOPak+bhDivSzTGRJCFgtmnYFCZv3Y7byzcwJuLNrC5pJLkeD+nHdWe8/p3YtgR7UmK93tdpjGmiVkomP0KBJVPv9nGGwvX8/bijWwtqyIlwc+ZfTpwbv/OnHJ4NolxFhDGtAQWCuaA1ASCfLJqK28u3MDbSzayY1c1aUlxnNWnI6OO6cRJvbOJ99tI68bEKgsFc9CqA0E+XLGFN77YwNQlGymprCGzTTwj+nZkVP/ODOnZjjgLCGNiioWCaRKVNQFmfb2FNxauZ9rSTZRVBchKSWDk0R059+jODO7RDr9vX8NcGWOiiYWCaXIV1QFmLCvijYUbeG/ZJiqqg7RPS+Scozsxsl9HBuRlWh+EMVHKQsFEVFllDe8tK+KNL9Yz8+vNVNUESfD76NM5nYF5mQzMa8ugvEy6ZCYjYi0JY7xmoWCaTUlFNR+t2MLnBTv4fM0OFq7bQUW1mwgoJy2RgV1dSAzMy6R/bobdE2GMBxobCvav0xyytKR4RvTrxIh+nQDXUf3VxhI+X7udz9fu4POCHUxdugkAv084okNabWtiYF4mPbNTrDVhTJSwloJpFtvKqviiYIcLioIdLFi7g5LQ5EAZyfEuJLq6kDimayYZyTYEhzFNyVoKJqq0S0lg+JHtGX5ke8DdWb1yc2moJeFaFPd9/TW7v6P0bp/KwK6ZHJ2bQa+cVHrlpNIhPdFaFMZEmLUUTNQoqahmYWHxHqedtpVV1a5PSfDTMyeVnjkp9Ap77JGdYkNzGLMf1lIwMSctKZ6hvbMZ2jsbcNOOFpVUsrKolJVbylhZVMqqLWXMXb2dyV+sr21ViECXzGR65qTSKyel9rFXTirt06x1YcyBsFAwUUtE6JCeRIf0JE4MBcVu5VUBvtlSxqotpawsCj1uLmXu6m3sqgrUbpeaGEfPnBR6Zu9uXaTSq30K3dqlkJxgrQtj6rJQMDEpOcFPn87pe80op6ps3FnBqs1lrNxcWvv42ertvLpg/R7bZqcmkNu2DV3btSG3bTJd27aha7tkctu2oUtmMglxNpSHaX0sFEyLIiJ0ykimU0Zy7Wmo3XZV1fDNljJWbi6jYNsu97N9FwsLd/DWog3UBDVsP9AxPak2LHLbtaFr2+RQiLj92/AepiWyUDCtRpuEOPp2zqBv54y91gWCroVRuG0XBdvLKdi2i8Lt5RRs38XsVVvZsGAd4ddkxPmETplJrnXR1rU0ctslk5oYT3K8n+QEH4lxfpIT/CTF+91r8X4S43z4LExMFLNQMAZ3U12XzGS6ZCZz/D7WV9UE2VBcTsE2FxSF23fVLk//qojNJZWNPlZinM+FRSg0dj9Pjv82QJLi/STF+0iO99MmMY6c1ARy0pLISUukfVoiOWmJdsWViQgLBWMaISHOR7esFLplpexzfXlVgPXF5ZRV1lBRHaS8OkB5VYDKGvdYUR2gPPR6ZXWA8uqw18K2Ky6vpqI6QEV1MLQ+sEfHebj0pLhQSOwZFu3TE8lJTQo9JpLZJt6uwDKNZqFgTBNITvDTKyc1IvuuCQTZVlZFUUklm0sqKSqpCD1W1j4uKNhBUUlF7ZhT4eL9Qk6qC4zw1kbbNvG0SYyjTYI/9LP3cnKCnwS/z0KlFbFQMCbKxfl9tE9Pon16UoPbqSqllTV7hEV4iGwuqaRw+y4+X7udrWE3Be73+D4hORQWKQlxtcvhwZESthzv9+ETwe8Dnwgigl/A59u9LPjErfP56lkO28bvE0Qg3u8j3u8jzi/E+3zExwlxPh/xfql9PcHvI87vI87nlpuq/yYQVGqCQWoC6n6CQWqCSnUgSCCoVAfC1geVOJ+QkRxPelI8qUlxMXVRQkRDQURGAPcDfuBxVb27zvpE4CngWGArcImqro5kTca0VCJCWlI8aUnx+221VAeC7CyvZldVIPRTQ3louSxsubw6QFlljVuuCrCrOsCu0PMd5dVsKC6nrHL3aa6afbZUvOSrGyZ+H/E+Ic7vwsTvE2qCoQ/6QJDqoIY+5N0HfCCoVAeDHMrADyLufpn0pHjSk+PJSA5fjg8tu9cykt3r6clxtevaJPibtaUWsVAQET/wIHAmUAh8JiKTVXVp2GbXAttVtbeIXAr8BbgkUjUZY5x4v4+s1ESymni/gdCHalDdj1t2rZg9ltUtB2u3de+tXRekdh+719UE3LfzqkCw9kN893J16AO9JhB0y6Fv9O71INU17pt8dWD3B75bDgSVOL8QFxYUcT4ffp+45VCrI87n2+d27nn4di54qkKhu7Oihp3l1RSXV7Ozopqd5e752m27ateXhgaGrI/fJ6QnuZAYO6QbPzi5ZxP/X9tTJFsKg4EVqroKQESeB0YD4aEwGrgjtPwS8C8REY21AZmMMYD7AIulUyXRoCYQpKSihp0VofAorwkFyJ5hUlxeTXZqYsTriWQodAEKwp4Xwl5X+9Vuo6o1IlIMZAFbwjcSkXHAOIC8vLxI1WuMMc0uzu+jbUoCbVMSvC4FgEjex7+vrwt1WwCN2QZVnaCq+aqan5OT0yTFGWOM2VskQ6EQ6Br2PBdYX982IhIHZADbIliTMcaYBkQyFD4DDhORHiKSAFwKTK6zzWTgqtDyRcB0608wxhjvRKxPIdRH8BPgHdwlqRNVdYmI3AnMVdXJwBPA0yKyAtdCuDRS9RhjjNm/iN6noKpTgCl1XrstbLkC+F4kazDGGNN4NmC8McaYWhYKxhhjalkoGGOMqSWxdrGPiGwG1hzk27Opc2NclIulemOpVoitemOpVoitemOpVji0erup6n5v9Iq5UDgUIjJXVfO9rqOxYqneWKoVYqveWKoVYqveWKoVmqdeO31kjDGmloWCMcaYWq0tFCZ4XcABiqV6Y6lWiK16Y6lWiK16Y6lWaIZ6W1WfgjHGmIa1tpaCMcaYBlgoGGOMqdVqQkFERojIVyKyQkRu9bqe+ohIVxGZISJfisgSEbnJ65oaQ0T8IvK5iLzhdS0NEZFMEXlJRJaF/huf4HVNDRGR/wv9HSwWkedEJMnrmsKJyEQRKRKRxWGvtRORd0VkeeixrZc17lZPrX8N/S0sFJH/iUimlzWG21e9Yet+KSIqItlNfdxWEQph80WPBPoAl4lIH2+rqlcN8AtVPQoYAtwQxbWGuwn40usiGuF+4G1VPRI4hiiuWUS6AD8F8lW1H2604WgbSXgSMKLOa7cC76nqYcB7oefRYBJ71/ou0E9V+wNfA+Obu6gGTGLvehGRrsCZwNpIHLRVhAJh80WrahWwe77oqKOqG1R1fmi5BPeh1cXbqhomIrnAucDjXtfSEBFJB07BDdmOqlap6g5vq9qvOCA5NAlVG/aeqMpTqjqLvSfGGg08GVp+ErigWYuqx75qVdWpqloTejobNxlYVKjnvy3AP4BfsY9ZKptCawmFfc0XHdUftAAi0h0YCMzxtpL9ug/3Rxr0upD96AlsBv4dOtX1uIikeF1UfVR1HfA33DfCDUCxqk71tqpG6aCqG8B9yQHae1xPY30feMvrIhoiIucD61T1i0gdo7WEQqPmgo4mIpIKvAz8TFV3el1PfURkFFCkqvO8rqUR4oBBwMOqOhAoI3pObewldC5+NNAD6AykiMhYb6tqmUTkN7hTt896XUt9RKQN8Bvgtv1teyhaSyg0Zr7oqCEi8bhAeFZVX/G6nv0YCpwvIqtxp+VOE5FnvC2pXoVAoarubnm9hAuJaHUG8I2qblbVauAV4ESPa2qMTSLSCSD0WORxPQ0SkauAUcDlUT4dcC/cF4QvQv/ecoH5ItKxKQ/SWkKhMfNFRwUREdw57y9V9V6v69kfVR2vqrmq2h3333W6qkblt1lV3QgUiMgRoZdOB5Z6WNL+rAWGiEib0N/F6URxx3iY8LnXrwJe87CWBonICOAW4HxV3eV1PQ1R1UWq2l5Vu4f+vRUCg0J/102mVYRCqCNp93zRXwIvqOoSb6uq11DgCtw37gWhn3O8LqoFuRF4VkQWAgOAP3lcT71CLZqXgPnAIty/16galkFEngM+AY4QkUIRuRa4GzhTRJbjrpK528sad6un1n8BacC7oX9rj3haZJh66o38caO7tWSMMaY5tYqWgjHGmMaxUDDGGFPLQsEYY0wtCwVjjDG1LBSMMcbUslAwphmJyLBoH0nWtG4WCsYYY2pZKBizDyIyVkQ+Dd3Q9GhovohSEfm7iMwXkfdEJCe07QARmR02Jn/b0Ou9RWSaiHwRek+v0O5Tw+Z0eDZ0t7IxUcFCwZg6ROQo4BJgqKoOAALA5UAKMF9VBwHvA7eH3vIUcEtoTP5FYa8/CzyoqsfgxizaEHp9IPAz3NwePXF3sRsTFeK8LsCYKHQ6cCzwWehLfDJuULcg8N/QNs8Ar4hIBpCpqu+HXn8SeFFE0oAuqvo/AFWtAAjt71NVLQw9XwB0Bz6M/K9lzP5ZKBizNwGeVNU9ZuESkd/V2a6hMWIaOiVUGbYcwP4dmihip4+M2dt7wEUi0h5q5xzuhvv3clFomzHAh6paDGwXkZNDr18BvB+aA6NQRC4I7SMxNB6+MVHNvqEYU4eqLhWR3wJTRcQHVAM34Cbl6Ssi84BiXL8DuOGhHwl96K8Crgm9fgXwqIjcGdrH95rx1zDmoNgoqcY0koiUqmqq13UYE0l2+sgYY0wtaykYY4ypZS0FY4wxtSwUjDHG1LJQMMYYU8tCwRhjTC0LBWOMMbX+H56nw6VFrpwWAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# summarize history for loss\n",
    "plt.plot(history.history['loss'])\n",
    "plt.plot(history.history['val_loss'])\n",
    "plt.title('model loss')\n",
    "plt.ylabel('loss')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'test'], loc='upper left')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TODO:   \n",
    "Record your results.  \n",
    "\n",
    "Is the model underfitting or overfitting the data? \n",
    "\n",
    "Based on your observations what steps would you take to improve the model?   \n",
    "\n",
    "Extra credit: Try to actually improve the recurrent model.  \n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# SKIP - will come back to this if we have time\n",
    "# Log to tensorboard\n",
    "# tensorBoardCallback = TensorBoard(log_dir='./logs', write_graph=True)\n",
    "# model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (tf)",
   "language": "python",
   "name": "tf"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}