first_edition/3.5-classifying-movie-reviews.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2.6.0'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import keras\n",
    "import tensorflow as tf\n",
    "keras.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Classifying movie reviews: a binary classification example\n",
    "\n",
    "This notebook contains the code samples found in Chapter 3, Section 5 of [Deep Learning with Python](https://www.manning.com/books/deep-learning-with-python?a_aid=keras&a_bid=76564dff). Note that the original text features far more content, in particular further explanations and figures: in this notebook, you will only find source code and related comments.\n",
    "\n",
    "----\n",
    "\n",
    "\n",
    "Two-class classification, or binary classification, may be the most widely applied kind of machine learning problem. In this example, we \n",
    "will learn to classify movie reviews into \"positive\" reviews and \"negative\" reviews, just based on the text content of the reviews."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The IMDB dataset\n",
    "\n",
    "\n",
    "We'll be working with \"IMDB dataset\", a set of 50,000 highly-polarized reviews from the Internet Movie Database. They are split into 25,000 \n",
    "reviews for training and 25,000 reviews for testing, each set consisting in 50% negative and 50% positive reviews.\n",
    "\n",
    "Why do we have these two separate training and test sets? You should never test a machine learning model on the same data that you used to \n",
    "train it! Just because a model performs well on its training data doesn't mean that it will perform well on data it has never seen, and \n",
    "what you actually care about is your model's performance on new data (since you already know the labels of your training data -- obviously \n",
    "you don't need your model to predict those). For instance, it is possible that your model could end up merely _memorizing_ a mapping between \n",
    "your training samples and their targets -- which would be completely useless for the task of predicting targets for data never seen before. \n",
    "We will go over this point in much more detail in the next chapter.\n",
    "\n",
    "Just like the MNIST dataset, the IMDB dataset comes packaged with Keras. It has already been preprocessed: the reviews (sequences of words) \n",
    "have been turned into sequences of integers, where each integer stands for a specific word in a dictionary.\n",
    "\n",
    "The following code will load the dataset (when you run it for the first time, about 80MB of data will be downloaded to your machine):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from keras.datasets import imdb\n",
    "\n",
    "(train_data, train_labels), (test_data, test_labels) = imdb.load_data(num_words=10000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "The argument `num_words=10000` means that we will only keep the top 10,000 most frequently occurring words in the training data. Rare words \n",
    "will be discarded. This allows us to work with vector data of manageable size.\n",
    "\n",
    "The variables `train_data` and `test_data` are lists of reviews, each review being a list of word indices (encoding a sequence of words). \n",
    "`train_labels` and `test_labels` are lists of 0s and 1s, where 0 stands for \"negative\" and 1 stands for \"positive\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1,\n",
       " 14,\n",
       " 22,\n",
       " 16,\n",
       " 43,\n",
       " 530,\n",
       " 973,\n",
       " 1622,\n",
       " 1385,\n",
       " 65,\n",
       " 458,\n",
       " 4468,\n",
       " 66,\n",
       " 3941,\n",
       " 4,\n",
       " 173,\n",
       " 36,\n",
       " 256,\n",
       " 5,\n",
       " 25,\n",
       " 100,\n",
       " 43,\n",
       " 838,\n",
       " 112,\n",
       " 50,\n",
       " 670,\n",
       " 2,\n",
       " 9,\n",
       " 35,\n",
       " 480,\n",
       " 284,\n",
       " 5,\n",
       " 150,\n",
       " 4,\n",
       " 172,\n",
       " 112,\n",
       " 167,\n",
       " 2,\n",
       " 336,\n",
       " 385,\n",
       " 39,\n",
       " 4,\n",
       " 172,\n",
       " 4536,\n",
       " 1111,\n",
       " 17,\n",
       " 546,\n",
       " 38,\n",
       " 13,\n",
       " 447,\n",
       " 4,\n",
       " 192,\n",
       " 50,\n",
       " 16,\n",
       " 6,\n",
       " 147,\n",
       " 2025,\n",
       " 19,\n",
       " 14,\n",
       " 22,\n",
       " 4,\n",
       " 1920,\n",
       " 4613,\n",
       " 469,\n",
       " 4,\n",
       " 22,\n",
       " 71,\n",
       " 87,\n",
       " 12,\n",
       " 16,\n",
       " 43,\n",
       " 530,\n",
       " 38,\n",
       " 76,\n",
       " 15,\n",
       " 13,\n",
       " 1247,\n",
       " 4,\n",
       " 22,\n",
       " 17,\n",
       " 515,\n",
       " 17,\n",
       " 12,\n",
       " 16,\n",
       " 626,\n",
       " 18,\n",
       " 2,\n",
       " 5,\n",
       " 62,\n",
       " 386,\n",
       " 12,\n",
       " 8,\n",
       " 316,\n",
       " 8,\n",
       " 106,\n",
       " 5,\n",
       " 4,\n",
       " 2223,\n",
       " 5244,\n",
       " 16,\n",
       " 480,\n",
       " 66,\n",
       " 3785,\n",
       " 33,\n",
       " 4,\n",
       " 130,\n",
       " 12,\n",
       " 16,\n",
       " 38,\n",
       " 619,\n",
       " 5,\n",
       " 25,\n",
       " 124,\n",
       " 51,\n",
       " 36,\n",
       " 135,\n",
       " 48,\n",
       " 25,\n",
       " 1415,\n",
       " 33,\n",
       " 6,\n",
       " 22,\n",
       " 12,\n",
       " 215,\n",
       " 28,\n",
       " 77,\n",
       " 52,\n",
       " 5,\n",
       " 14,\n",
       " 407,\n",
       " 16,\n",
       " 82,\n",
       " 2,\n",
       " 8,\n",
       " 4,\n",
       " 107,\n",
       " 117,\n",
       " 5952,\n",
       " 15,\n",
       " 256,\n",
       " 4,\n",
       " 2,\n",
       " 7,\n",
       " 3766,\n",
       " 5,\n",
       " 723,\n",
       " 36,\n",
       " 71,\n",
       " 43,\n",
       " 530,\n",
       " 476,\n",
       " 26,\n",
       " 400,\n",
       " 317,\n",
       " 46,\n",
       " 7,\n",
       " 4,\n",
       " 2,\n",
       " 1029,\n",
       " 13,\n",
       " 104,\n",
       " 88,\n",
       " 4,\n",
       " 381,\n",
       " 15,\n",
       " 297,\n",
       " 98,\n",
       " 32,\n",
       " 2071,\n",
       " 56,\n",
       " 26,\n",
       " 141,\n",
       " 6,\n",
       " 194,\n",
       " 7486,\n",
       " 18,\n",
       " 4,\n",
       " 226,\n",
       " 22,\n",
       " 21,\n",
       " 134,\n",
       " 476,\n",
       " 26,\n",
       " 480,\n",
       " 5,\n",
       " 144,\n",
       " 30,\n",
       " 5535,\n",
       " 18,\n",
       " 51,\n",
       " 36,\n",
       " 28,\n",
       " 224,\n",
       " 92,\n",
       " 25,\n",
       " 104,\n",
       " 4,\n",
       " 226,\n",
       " 65,\n",
       " 16,\n",
       " 38,\n",
       " 1334,\n",
       " 88,\n",
       " 12,\n",
       " 16,\n",
       " 283,\n",
       " 5,\n",
       " 16,\n",
       " 4472,\n",
       " 113,\n",
       " 103,\n",
       " 32,\n",
       " 15,\n",
       " 16,\n",
       " 5345,\n",
       " 19,\n",
       " 178,\n",
       " 32]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_labels[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we restricted ourselves to the top 10,000 most frequent words, no word index will exceed 10,000:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9999"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max([max(sequence) for sequence in train_data])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For kicks, here's how you can quickly decode one of these reviews back to English words:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# word_index is a dictionary mapping words to an integer index\n",
    "word_index = imdb.get_word_index()\n",
    "# We reverse it, mapping integer indices to words\n",
    "reverse_word_index = dict([(value, key) for (key, value) in word_index.items()])\n",
    "# We decode the review; note that our indices were offset by 3\n",
    "# because 0, 1 and 2 are reserved indices for \"padding\", \"start of sequence\", and \"unknown\".\n",
    "decoded_review = ' '.join([reverse_word_index.get(i - 3, '?') for i in train_data[0]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"? this film was just brilliant casting location scenery story direction everyone's really suited the part they played and you could just imagine being there robert ? is an amazing actor and now the same being director ? father came from the same scottish island as myself so i loved the fact there was a real connection with this film the witty remarks throughout the film were great it was just brilliant so much that i bought the film as soon as it was released for ? and would recommend it to everyone to watch and the fly fishing was amazing really cried at the end it was so sad and you know what they say if you cry at a film it must have been good and this definitely was also ? to the two little boy's that played the ? of norman and paul they were just brilliant children are often left out of the ? list i think because the stars that play them all grown up are such a big profile for the whole film but these children are amazing and should be praised for what they have done don't you think the whole story was so lovely because it was true and was someone's life after all that was shared with us all\""
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "decoded_review"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing the data\n",
    "\n",
    "\n",
    "We cannot feed lists of integers into a neural network. We have to turn our lists into tensors. There are two ways we could do that:\n",
    "\n",
    "* We could pad our lists so that they all have the same length, and turn them into an integer tensor of shape `(samples, word_indices)`, \n",
    "then use as first layer in our network a layer capable of handling such integer tensors (the `Embedding` layer, which we will cover in \n",
    "detail later in the book).\n",
    "* We could one-hot-encode our lists to turn them into vectors of 0s and 1s. Concretely, this would mean for instance turning the sequence \n",
    "`[3, 5]` into a 10,000-dimensional vector that would be all-zeros except for indices 3 and 5, which would be ones. Then we could use as \n",
    "first layer in our network a `Dense` layer, capable of handling floating point vector data.\n",
    "\n",
    "We will go with the latter solution. Let's vectorize our data, which we will do manually for maximum clarity:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "def vectorize_sequences(sequences, dimension=10000):\n",
    "    # Create an all-zero matrix of shape (len(sequences), dimension)\n",
    "    results = np.zeros((len(sequences), dimension))\n",
    "    for i, sequence in enumerate(sequences):\n",
    "        results[i, sequence] = 1.  # set specific indices of results[i] to 1s\n",
    "    return results\n",
    "\n",
    "# Our vectorized training data\n",
    "x_train = vectorize_sequences(train_data)\n",
    "# Our vectorized test data\n",
    "x_test = vectorize_sequences(test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's what our samples look like now:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 1., 1., ..., 0., 0., 0.])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We should also vectorize our labels, which is straightforward:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Our vectorized labels\n",
    "y_train = np.asarray(train_labels).astype('float32')\n",
    "y_test = np.asarray(test_labels).astype('float32')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now our data is ready to be fed into a neural network."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building our network\n",
    "\n",
    "\n",
    "Our input data is simply vectors, and our labels are scalars (1s and 0s): this is the easiest setup you will ever encounter. A type of \n",
    "network that performs well on such a problem would be a simple stack of fully-connected (`Dense`) layers with `relu` activations: `Dense(16, \n",
    "activation='relu')`\n",
    "\n",
    "The argument being passed to each `Dense` layer (16) is the number of \"hidden units\" of the layer. What's a hidden unit? It's a dimension \n",
    "in the representation space of the layer. You may remember from the previous chapter that each such `Dense` layer with a `relu` activation implements \n",
    "the following chain of tensor operations:\n",
    "\n",
    "`output = relu(dot(W, input) + b)`\n",
    "\n",
    "Having 16 hidden units means that the weight matrix `W` will have shape `(input_dimension, 16)`, i.e. the dot product with `W` will project the \n",
    "input data onto a 16-dimensional representation space (and then we would add the bias vector `b` and apply the `relu` operation). You can \n",
    "intuitively understand the dimensionality of your representation space as \"how much freedom you are allowing the network to have when \n",
    "learning internal representations\". Having more hidden units (a higher-dimensional representation space) allows your network to learn more \n",
    "complex representations, but it makes your network more computationally expensive and may lead to learning unwanted patterns (patterns that \n",
    "will improve performance on the training data but not on the test data).\n",
    "\n",
    "There are two key architecture decisions to be made about such stack of dense layers:\n",
    "\n",
    "* How many layers to use.\n",
    "* How many \"hidden units\" to chose for each layer.\n",
    "\n",
    "In the next chapter, you will learn formal principles to guide you in making these choices. \n",
    "For the time being, you will have to trust us with the following architecture choice: \n",
    "two intermediate layers with 16 hidden units each, \n",
    "and a third layer which will output the scalar prediction regarding the sentiment of the current review. \n",
    "The intermediate layers will use `relu` as their \"activation function\", \n",
    "and the final layer will use a sigmoid activation so as to output a probability \n",
    "(a score between 0 and 1, indicating how likely the sample is to have the target \"1\", i.e. how likely the review is to be positive). \n",
    "A `relu` (rectified linear unit) is a function meant to zero-out negative values, \n",
    "while a sigmoid \"squashes\" arbitrary values into the `[0, 1]` interval, thus outputting something that can be interpreted as a probability."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's what our network looks like:\n",
    "\n",
    "![3-layer network](https://s3.amazonaws.com/book.keras.io/img/ch3/3_layer_network.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And here's the Keras implementation, very similar to the MNIST example you saw previously:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-02-18 19:02:07.356901: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/R/4.0.5/lib/R/lib::/lib:/usr/local/lib:/usr/lib/x86_64-linux-gnu:/usr/lib/jvm/java-11-openjdk-amd64/lib/server\n",
      "2022-02-18 19:02:07.356938: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)\n",
      "2022-02-18 19:02:07.356957: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (taylor): /proc/driver/nvidia/version does not exist\n",
      "2022-02-18 19:02:07.357226: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n"
     ]
    }
   ],
   "source": [
    "from keras import models\n",
    "from keras import layers\n",
    "\n",
    "model = models.Sequential()\n",
    "model.add(layers.Dense(16, activation='relu', input_shape=(10000,)))\n",
    "model.add(layers.Dense(16, activation='relu'))\n",
    "model.add(layers.Dense(1, activation='sigmoid'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Lastly, we need to pick a loss function and an optimizer. Since we are facing a binary classification problem and the output of our network \n",
    "is a probability (we end our network with a single-unit layer with a sigmoid activation), is it best to use the `binary_crossentropy` loss. \n",
    "It isn't the only viable choice: you could use, for instance, `mean_squared_error`. But crossentropy is usually the best choice when you \n",
    "are dealing with models that output probabilities. Crossentropy is a quantity from the field of Information Theory, that measures the \"distance\" \n",
    "between probability distributions, or in our case, between the ground-truth distribution and our predictions.\n",
    "\n",
    "Here's the step where we configure our model with the `rmsprop` optimizer and the `binary_crossentropy` loss function. Note that we will \n",
    "also monitor accuracy during training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "model.compile(optimizer='rmsprop',\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are passing our optimizer, loss function and metrics as strings, which is possible because `rmsprop`, `binary_crossentropy` and \n",
    "`accuracy` are packaged as part of Keras. Sometimes you may want to configure the parameters of your optimizer, or pass a custom loss \n",
    "function or metric function. This former can be done by passing an optimizer class instance as the `optimizer` argument:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from keras import optimizers\n",
    "\n",
    "model.compile(optimizer=tf.keras.optimizers.RMSprop(learning_rate=0.001),\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The latter can be done by passing function objects as the `loss` or `metrics` arguments:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from keras import losses\n",
    "from keras import metrics\n",
    "\n",
    "model.compile(optimizer=tf.keras.optimizers.RMSprop(lr=0.001),\n",
    "              loss=losses.binary_crossentropy,\n",
    "              metrics=[metrics.binary_accuracy])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Validating our approach\n",
    "\n",
    "In order to monitor during training the accuracy of the model on data that it has never seen before, we will create a \"validation set\" by \n",
    "setting apart 10,000 samples from the original training data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "x_val = x_train[:10000]\n",
    "partial_x_train = x_train[10000:]\n",
    "\n",
    "y_val = y_train[:10000]\n",
    "partial_y_train = y_train[10000:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now train our model for 20 epochs (20 iterations over all samples in the `x_train` and `y_train` tensors), in mini-batches of 512 \n",
    "samples. At this same time we will monitor loss and accuracy on the 10,000 samples that we set apart. This is done by passing the \n",
    "validation data as the `validation_data` argument:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-02-18 19:04:31.062373: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "30/30 [==============================] - 1s 33ms/step - loss: 0.5257 - binary_accuracy: 0.7943 - val_loss: 0.4023 - val_binary_accuracy: 0.8680\n",
      "Epoch 2/20\n",
      "30/30 [==============================] - 0s 12ms/step - loss: 0.3219 - binary_accuracy: 0.8991 - val_loss: 0.3248 - val_binary_accuracy: 0.8745\n",
      "Epoch 3/20\n",
      "30/30 [==============================] - 0s 10ms/step - loss: 0.2337 - binary_accuracy: 0.9242 - val_loss: 0.2818 - val_binary_accuracy: 0.8903\n",
      "Epoch 4/20\n",
      "30/30 [==============================] - 0s 9ms/step - loss: 0.1866 - binary_accuracy: 0.9407 - val_loss: 0.2797 - val_binary_accuracy: 0.8890\n",
      "Epoch 5/20\n",
      "30/30 [==============================] - 0s 10ms/step - loss: 0.1482 - binary_accuracy: 0.9539 - val_loss: 0.2818 - val_binary_accuracy: 0.8884\n",
      "Epoch 6/20\n",
      "30/30 [==============================] - 0s 9ms/step - loss: 0.1245 - binary_accuracy: 0.9614 - val_loss: 0.2929 - val_binary_accuracy: 0.8859\n",
      "Epoch 7/20\n",
      "30/30 [==============================] - 0s 9ms/step - loss: 0.0995 - binary_accuracy: 0.9709 - val_loss: 0.3074 - val_binary_accuracy: 0.8850\n",
      "Epoch 8/20\n",
      "30/30 [==============================] - 0s 7ms/step - loss: 0.0866 - binary_accuracy: 0.9747 - val_loss: 0.3307 - val_binary_accuracy: 0.8794\n",
      "Epoch 9/20\n",
      "30/30 [==============================] - 0s 7ms/step - loss: 0.0683 - binary_accuracy: 0.9818 - val_loss: 0.3521 - val_binary_accuracy: 0.8782\n",
      "Epoch 10/20\n",
      "30/30 [==============================] - 0s 7ms/step - loss: 0.0568 - binary_accuracy: 0.9851 - val_loss: 0.3883 - val_binary_accuracy: 0.8722\n",
      "Epoch 11/20\n",
      "30/30 [==============================] - 0s 7ms/step - loss: 0.0453 - binary_accuracy: 0.9889 - val_loss: 0.4071 - val_binary_accuracy: 0.8771\n",
      "Epoch 12/20\n",
      "30/30 [==============================] - 0s 6ms/step - loss: 0.0356 - binary_accuracy: 0.9925 - val_loss: 0.4462 - val_binary_accuracy: 0.8754\n",
      "Epoch 13/20\n",
      "30/30 [==============================] - 0s 7ms/step - loss: 0.0299 - binary_accuracy: 0.9939 - val_loss: 0.4671 - val_binary_accuracy: 0.8747\n",
      "Epoch 14/20\n",
      "30/30 [==============================] - 0s 8ms/step - loss: 0.0217 - binary_accuracy: 0.9965 - val_loss: 0.4978 - val_binary_accuracy: 0.8747\n",
      "Epoch 15/20\n",
      "30/30 [==============================] - 0s 8ms/step - loss: 0.0179 - binary_accuracy: 0.9978 - val_loss: 0.5357 - val_binary_accuracy: 0.8715\n",
      "Epoch 16/20\n",
      "30/30 [==============================] - 0s 8ms/step - loss: 0.0135 - binary_accuracy: 0.9983 - val_loss: 0.5691 - val_binary_accuracy: 0.8674\n",
      "Epoch 17/20\n",
      "30/30 [==============================] - 0s 8ms/step - loss: 0.0130 - binary_accuracy: 0.9978 - val_loss: 0.5995 - val_binary_accuracy: 0.8697\n",
      "Epoch 18/20\n",
      "30/30 [==============================] - 0s 6ms/step - loss: 0.0075 - binary_accuracy: 0.9995 - val_loss: 0.6689 - val_binary_accuracy: 0.8667\n",
      "Epoch 19/20\n",
      "30/30 [==============================] - 0s 6ms/step - loss: 0.0059 - binary_accuracy: 0.9993 - val_loss: 0.7637 - val_binary_accuracy: 0.8499\n",
      "Epoch 20/20\n",
      "30/30 [==============================] - 0s 8ms/step - loss: 0.0041 - binary_accuracy: 0.9999 - val_loss: 0.7028 - val_binary_accuracy: 0.8648\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(partial_x_train,\n",
    "                    partial_y_train,\n",
    "                    epochs=20,\n",
    "                    batch_size=512,\n",
    "                    validation_data=(x_val, y_val))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On CPU, this will take less than two seconds per epoch -- training is over in 20 seconds. At the end of every epoch, there is a slight pause \n",
    "as the model computes its loss and accuracy on the 10,000 samples of the validation data.\n",
    "\n",
    "Note that the call to `model.fit()` returns a `History` object. This object has a member `history`, which is a dictionary containing data \n",
    "about everything that happened during training. Let's take a look at it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['loss', 'binary_accuracy', 'val_loss', 'val_binary_accuracy'])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "history_dict = history.history\n",
    "history_dict.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It contains 4 entries: one per metric that was being monitored, during training and during validation. Let's use Matplotlib to plot the \n",
    "training and validation loss side by side, as well as the training and validation accuracy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAzI0lEQVR4nO3dedyVc/7H8ddHi6Qila0dJUtpuStUlLXwK7uaBtkSsmQGmQYxmhnDGFuM7EuUYSSjJiNF9u5IKkVSREi0aa/P74/vded0d6/d93XOuc95Px+P+3Gfc13Xuc7nvjpdn/Pdzd0REZHstUOqAxARkdRSIhARyXJKBCIiWU6JQEQkyykRiIhkOSUCEZEsp0Qg5crMxpvZueV9bCqZ2QIzOyaG87qZ7Rc9/qeZ3VCSY7fjffqa2avbG2cR5+1qZovK+7ySfJVTHYCknpmtSnhaHVgHbIqeX+zuI0t6LnfvEcexmc7dB5THecysCfAlUMXdN0bnHgmU+N9Qso8SgeDuNfIem9kC4EJ3fy3/cWZWOe/mIiKZQ1VDUqi8or+ZXWdm3wGPmVltM/uPmS0xs5+jxw0SXjPZzC6MHvczs7fM7I7o2C/NrMd2HtvUzN40s5Vm9pqZDTezpwuJuyQx/snM3o7O96qZ1U3Yf7aZLTSzpWY2pIjr09HMvjOzSgnbTjGzGdHjDmb2rpktM7PFZnafmVUt5FyPm9mtCc+viV7zrZmdn+/YE83sIzNbYWZfm9nQhN1vRr+XmdkqMzss79omvP5wM5tqZsuj34eX9NoUxcwOiF6/zMxmmVnPhH0nmNns6JzfmNnvo+11o3+fZWb2k5lNMTPdl5JMF1yKsyewG9AY6E/4zDwWPW8ErAHuK+L1HYG5QF3gb8AjZmbbcewzwAdAHWAocHYR71mSGH8DnAfsDlQF8m5MBwIPROffO3q/BhTA3d8HfgGOynfeZ6LHm4BB0d9zGHA0cGkRcRPF0D2K51igGZC/feIX4BxgV+BE4BIzOznad0T0e1d3r+Hu7+Y7927AK8A90d92J/CKmdXJ9zdsc22KibkK8DLwavS6y4GRZrZ/dMgjhGrGmsDBwOvR9t8Bi4B6wB7AHwDNe5NkSgRSnM3ATe6+zt3XuPtSd3/B3Ve7+0pgGHBkEa9f6O4Pufsm4AlgL8J/+BIfa2aNgPbAje6+3t3fAsYW9oYljPExd//M3dcAzwGto+2nA/9x9zfdfR1wQ3QNCvMs0AfAzGoCJ0TbcPdp7v6eu2909wXAgwXEUZAzo/hmuvsvhMSX+PdNdvdP3H2zu8+I3q8k54WQOD5396eiuJ4F5gD/l3BMYdemKIcCNYC/Rv9GrwP/Ibo2wAbgQDOr5e4/u/uHCdv3Ahq7+wZ3n+KaAC3plAikOEvcfW3eEzOrbmYPRlUnKwhVEbsmVo/k813eA3dfHT2sUcpj9wZ+StgG8HVhAZcwxu8SHq9OiGnvxHNHN+Klhb0X4dv/qWa2I3Aq8KG7L4ziaB5Ve3wXxfFnQumgOFvFACzM9/d1NLNJUdXXcmBACc+bd+6F+bYtBOonPC/s2hQbs7snJs3E855GSJILzewNMzss2n47MA941czmm9ngkv0ZUp6UCKQ4+b+d/Q7YH+jo7rX4tSqisOqe8rAY2M3Mqidsa1jE8WWJcXHiuaP3rFPYwe4+m3DD68HW1UIQqpjmAM2iOP6wPTEQqrcSPUMoETV0912Afyact7hv098SqswSNQK+KUFcxZ23Yb76/S3ndfep7t6LUG00hlDSwN1Xuvvv3H0foCdwtZkdXcZYpJSUCKS0ahLq3JdF9c03xf2G0TfsXGComVWNvk3+XxEvKUuMzwMnmVnnqGH3For/f/IMcCUh4fwrXxwrgFVm1gK4pIQxPAf0M7MDo0SUP/6ahBLSWjPrQEhAeZYQqrL2KeTc44DmZvYbM6tsZmcBBxKqccrifULp4Vozq2JmXQn/RqOif7O+ZraLu28gXJPNAGZ2kpntF7UFLSe0qxRVFScxUCKQ0roL2An4EXgP+G+S3rcvocF1KXArMJow3qEgd7GdMbr7LOAyws19MfAzoTGzKHl19K+7+48J239PuEmvBB6KYi5JDOOjv+F1QrXJ6/kOuRS4xcxWAjcSfbuOXrua0CbydtQT59B8514KnEQoNS0FrgVOyhd3qbn7esKNvwfhut8PnOPuc6JDzgYWRFVkAwj/nhAaw18DVgHvAve7+6SyxCKlZ2qXkYrIzEYDc9w99hKJSKZTiUAqBDNrb2b7mtkOUffKXoS6ZhEpI40slopiT+DfhIbbRcAl7v5RakMSyQyxVg1F39zuBioBD7v7X/Ptb0ToL75rdMxgdx8XW0AiIrKN2BJB1Gf7M8LoyEXAVKBP1N0u75gRwEfu/kA0onOcuzeJJSARESlQnFVDHYB57j4fwMxGEep1Zycc40Ct6PEuhL7IRapbt643adKkfCMVEclw06ZN+9Hd6xW0L85EUJ+tR0cuIswlk2goYUTh5cDObDunCgBm1p8wzw2NGjUiNze33IMVEclkZpZ/RPkWqe411Ad43N0bEIafP1XQzIPuPsLdc9w9p169AhOaiIhspzgTwTdsPUy+AdsOY7+AX4eavwtUo+RzpoiISDmIMxFMBZpZmEe+KtCbbWeM/IowNS9mdgAhESyJMSYREckntjYCd99oZgOBCYSuoY+6+ywzuwXIdfexhGHuD5nZIELDcb/tmYJ2w4YNLFq0iLVr1xZ/sKRUtWrVaNCgAVWqVEl1KCISqXBTTOTk5Hj+xuIvv/ySmjVrUqdOHQpf80RSzd1ZunQpK1eupGnTpqkORySrmNk0d88paF+qG4vLxdq1a5UEKgAzo06dOiq5iaSZjEgEgJJABaF/J5H0kzGJQEQkXW3eDI8/DosXpzqSgikRlIOlS5fSunVrWrduzZ577kn9+vW3PF+/fn2Rr83NzeWKK64o9j0OP/zwcol18uTJnHTSSeVyLhEpmeHD4bzz4IgjYFFxq1ukQFYmgpEjoUkT2GGH8HvkyLKdr06dOkyfPp3p06czYMAABg0atOV51apV2bhxY6GvzcnJ4Z577in2Pd55552yBSkiKbFgAVx/PXToAD/8AF27wteFrridGlmXCEaOhP79YeFCcA+/+/cvezLIr1+/fgwYMICOHTty7bXX8sEHH3DYYYfRpk0bDj/8cObOnQts/Q196NChnH/++XTt2pV99tlnqwRRo0aNLcd37dqV008/nRYtWtC3b1/yen6NGzeOFi1a0K5dO6644opiv/n/9NNPnHzyybRq1YpDDz2UGTNmAPDGG29sKdG0adOGlStXsnjxYo444ghat27NwQcfzJQpU8r3golkIHe4+GIwg+eeg1dfhSVL4Mgjw70nXWTdegRDhsDq1VtvW706bO/bt+DXbK9FixbxzjvvUKlSJVasWMGUKVOoXLkyr732Gn/4wx944YUXtnnNnDlzmDRpEitXrmT//ffnkksu2abP/UcffcSsWbPYe++96dSpE2+//TY5OTlcfPHFvPnmmzRt2pQ+ffoUG99NN91EmzZtGDNmDK+//jrnnHMO06dP54477mD48OF06tSJVatWUa1aNUaMGMHxxx/PkCFD2LRpE6vzX0QR2cYTT4Sb/333QePG4ee11+C440LJYNKkUCuRallXIvjqq9JtL4szzjiDSpUqAbB8+XLOOOMMDj74YAYNGsSsWbMKfM2JJ57IjjvuSN26ddl99935/vvvtzmmQ4cONGjQgB122IHWrVuzYMEC5syZwz777LOlf35JEsFbb73F2WefDcBRRx3F0qVLWbFiBZ06deLqq6/mnnvuYdmyZVSuXJn27dvz2GOPMXToUD755BNq1qy5vZdFJCt89x0MGgSdO8Mll/y6vX37kAyWLw8lg/nzUxdjnqxLBI0alW57Wey8885bHt9www1069aNmTNn8vLLLxfal37HHXfc8rhSpUoFti+U5JiyGDx4MA8//DBr1qyhU6dOzJkzhyOOOII333yT+vXr069fP5588slyfU+RTDNwIKxZAw8/HNojE7VrBxMnwqpVoWTwxRcpCXGLrEsEw4ZB9epbb6tePWyP0/Lly6lfvz4Ajz/+eLmff//992f+/PksWLAAgNGjRxf7mi5dujAyahyZPHkydevWpVatWnzxxRe0bNmS6667jvbt2zNnzhwWLlzIHnvswUUXXcSFF17Ihx9+WO5/g0imeOGF8DN0KOy/f8HHtGkDr78eqqaPPBI+/zypIW4l6xJB374wYkSoqzMLv0eMKP/2gfyuvfZarr/+etq0aVPu3+ABdtppJ+6//366d+9Ou3btqFmzJrvsskuRrxk6dCjTpk2jVatWDB48mCeeeAKAu+66i4MPPphWrVpRpUoVevToweTJkznkkENo06YNo0eP5sorryz3v0EkE/z8M1x2WbjR/+53RR97yCEhGaxbF0oGUR+SpMuIuYY+/fRTDjjggBRFlD5WrVpFjRo1cHcuu+wymjVrxqBBg1Id1jb07yWZ7Lzz4KmnYOrUkAxKYuZMOOooqFQpNCC3aFH+cWX8XEMSPPTQQ7Ru3ZqDDjqI5cuXc/HFF6c6JJGs8uqrYQTxddeVPAkAHHwwTJ4cupt27QqzZxf3ivKlEoEknf69JBOtWhVu6NWqwfTp4XdpzZkD3bqFKSkmTgznKy8qEYiIxGzIkNAN/ZFHti8JQKgSmjw5VBF16waffFKuIRZKiUBEpIzefhvuvTc0EnfqVLZz7b9/SAZVq4Zk8PHH5RJikZQIRETKYO1auPBCaNgQ/vKX8jln8+bwxhuw006hEXn69PI5b2FiTQRm1t3M5prZPDMbXMD+f5jZ9OjnMzNbFmc8IiLl7dZbQ93+iBEQTQlWLvbbL5QMdt45JIM4h+7ElgjMrBIwHOgBHAj0MbMDE49x90Hu3trdWwP3Av+OK544devWjQkTJmy17a677uKSxHHl+XTt2pW8Ru8TTjiBZcuWbXPM0KFDueOOO4p87zFjxjA7oYvBjTfeyGuvvVaK6Aum6apFijd9Otx2G5x7Lhx/fPmff999Q8mgVi04+mjI10+m3MRZIugAzHP3+e6+HhgF9Cri+D7AszHGE5s+ffowatSorbaNGjWqRPP9QJg1dNddd92u986fCG655RaOOeaY7TqXiJTcxo1wwQVQpw7ceWd879O0aSgZ1KsX3/TVcSaC+kBi2Iuibdsws8ZAU+D1Qvb3N7NcM8tdsmRJuQdaVqeffjqvvPLKlkVoFixYwLfffkuXLl245JJLyMnJ4aCDDuKmm24q8PVNmjThxx9/BGDYsGE0b96czp07b5mqGsIYgfbt23PIIYdw2mmnsXr1at555x3Gjh3LNddcQ+vWrfniiy/o168fzz//PAATJ06kTZs2tGzZkvPPP59169Zteb+bbrqJtm3b0rJlS+bMmVPk36fpqkW29fe/h+qa4cNht93ifa8mTcKgs1NOief86TINdW/geXffVNBOdx8BjIAwjqCoE111Vfk3rLRuDXfdVfj+3XbbjQ4dOjB+/Hh69erFqFGjOPPMMzEzhg0bxm677camTZs4+uijmTFjBq1atSrwPNOmTWPUqFFMnz6djRs30rZtW9q1awfAqaeeykUXXQTAH//4Rx555BEuv/xyevbsyUknncTpp5++1bnWrl1Lv379mDhxIs2bN+ecc87hgQce4KqrrgKgbt26fPjhh9x///3ccccdPPzww4X+fZquWmRrn30GN90Ep54Kp52WnPesWjW+c8dZIvgGaJjwvEG0rSC9qaDVQnkSq4cSq4Wee+452rZtS5s2bZg1a9ZW1Tj5TZkyhVNOOYXq1atTq1YtevbsuWXfzJkz6dKlCy1btmTkyJGFTmOdZ+7cuTRt2pTmzZsDcO655/Lmm29u2X/qqacC0K5duy0T1RVG01WL/Grz5tBLaKedwjoDmSDOEsFUoJmZNSUkgN7Ab/IfZGYtgNrAu+XxpkV9c49Tr169GDRoEB9++CGrV6+mXbt2fPnll9xxxx1MnTqV2rVr069fv0Knny5Ov379GDNmDIcccgiPP/44kydPLlO8eVNZl2Ua68GDB3PiiScybtw4OnXqxIQJE7ZMV/3KK6/Qr18/rr76as4555wyxSqSTh58EKZMgUcfhb32SnU05SO2EoG7bwQGAhOAT4Hn3H2Wmd1iZj0TDu0NjPKKNtdFPjVq1KBbt26cf/75W0oDK1asYOedd2aXXXbh+++/Z/z48UWe44gjjmDMmDGsWbOGlStX8vLLL2/Zt3LlSvbaay82bNiwZepogJo1a7Jy5cptzrX//vuzYMEC5s2bB8BTTz3FkUceuV1/m6arFgm++gquvRaOPRb69Ut1NOUn1jYCdx8HjMu37cZ8z4fGGUMy9enTh1NOOWVLFVHetM0tWrSgYcOGdCpmyGHbtm0566yzOOSQQ9h9991p3779ln1/+tOf6NixI/Xq1aNjx45bbv69e/fmoosu4p577tnSSAxQrVo1HnvsMc444ww2btxI+/btGTBgwHb9XXlrKbdq1Yrq1atvNV31pEmT2GGHHTjooIPo0aMHo0aN4vbbb6dKlSrUqFFDC9hIxnCHAQPC7xEjwjT2mUKTzknS6d9LKqKnn4azz4a774Yrrkh1NKWnSedERMpg4UK48ko47LAwn1CmUSIQESnCTz9B9+6ht9Bjj4WZQTNNuowjKDN3xzKp0i5DVbSqSMlua9dCr14wf35YdKaw9YcruowoEVSrVo2lS5fqJpPm3J2lS5dSbXsnaxdJok2b4Le/hbfeCktPbmenuwohI0oEDRo0YNGiRaTj9BOytWrVqtGgQYNUhyFSJHe4+mp44YUwj9CZZ6Y6onhlRCKoUqUKTZs2TXUYIpIh/v53uOceGDQo/GS6jKgaEhEpL88+C9dcE0oBxcwCnzGUCEREIpMmhbUFjjgCnngCdsiSO2SW/JkiIkX75BM4+eSwTOSYMdu/AH1FpEQgIlnv66+hR4+w1OT48VC7dqojSq6MaCwWEdley5aFJLByZZhVtGHDYl+ScZQIRCRrrVsXqoM++wz++18oZM2ojKdEICJZafPm0DD8xhswciQcdVSqI0odtRGISFa69loYPRpuuw1+s82SWdlFiUBEss7dd4dBYwMHhjED2U6JQESyyvPPh9HCp5wSlrbVXJUxJwIz625mc81snpkNLuSYM81stpnNMrNn4oxHRLLblClhIrnDDw/tApk4pfT2iK2x2MwqAcOBY4FFwFQzG+vusxOOaQZcD3Ry95/NbPe44hGR7DZ7NvTsCU2awEsvwU47pTqi9BFniaADMM/d57v7emAU0CvfMRcBw939ZwB3/yHGeEQkC23eDC++CMcfH0YL//e/UKdOqqNKL3EmgvrA1wnPF0XbEjUHmpvZ22b2npl1L+hEZtbfzHLNLFdTTYtISWzeDP/6F7RuDaeeGkoA48eHEoFsLdWNxZWBZkBXoA/wkJntmv8gdx/h7jnunlOvXr3kRigiFcqmTWEG0ZYtwwyiGzaEhednzw5JQbYVZyL4BkgcrN0g2pZoETDW3Te4+5fAZ4TEICJSKhs3hpXEDjwwjAswg1GjYOZM6NsXKmv4bKHiTARTgWZm1tTMqgK9gbH5jhlDKA1gZnUJVUXzY4xJRDLMhg1hUfkWLeCcc0I7wPPPw4wZcNZZ6hlUErElAnffCAwEJgCfAs+5+ywzu8XMekaHTQCWmtlsYBJwjbsvjSsmEckc69fDQw+FaaPPPx922SVMH/3RR3DaadmzlkB5sIq24HtOTo7n5uamOgwRSZF160IJ4C9/ga++gvbt4aab4IQTNDisKGY2zd1zCtqnWjMRqRDWroWHH4a//hW++QYOOwxGjIDjjlMCKCslAhFJaxs3wiOPwM03w+LF0LkzPP44HH20EkB5USIQkbTkHur8r78e5s6FTp3gmWfgyCOVAMqbmlNEJO28/Xb45n/qqeGmP2ZMmCeoa1clgTgoEYhI2pgzJ8wK2rkzfPllaAP45BPo1UsJIE5KBCKScosXw8UXw8EHw8SJcOut8PnncNFFGgiWDLrEIpIyK1bA7bfDnXeGgWGXXgo33ACaSSa5lAhEJOnWr4cHH4Q//QmWLAkjgIcNg333TXVk2UlVQyKSNO7w3HNhPqArrghVQR98EOYEUhJIHSUCEUmKyZOhY8fw7b96dRg3LrQHtG+f6sgkKxLByJFhDvIddgi/R45MdUQi2ePbb6F3b+jWDb77LgwG++gj6NFDPYHSRca3EYwcCf37w+rV4fnCheE5hKlpRSQemzbB/ffDkCGhTeDmm+Gaa7REZDrK+BLBkCG/JoE8q1eH7SISj9zcUA10xRVhTqCZM+HGG5UE0lXGJ4KvvirddhHZfsuWwcCB0KFDqBIaPTqsEbzffqmOTIqS8YmgUaPSbReR0nMPy0O2aAEPPACXXx5GCZ95ptoBKoKMTwTDhoUeComqVw/bRaTsPvsMjj02LA/ZsGHoDnr33VCrVqojk5LK+ETQt2+Yr6Rx4/DNpHHj8FwNxSJls3ZtWBCmZUuYOhWGD4f33oN27VIdmZRWrInAzLqb2Vwzm2dmgwvY38/MlpjZ9Ojnwjji6NsXFiyAzZvDbyUBkbJ59dWQAG65BU4/PUwTfemlWh+4ooqt+6iZVQKGA8cCi4CpZjbW3WfnO3S0uw+MKw4RKT/ffgtXXx0agZs1g//9D445JtVRSVnFWSLoAMxz9/nuvh4YBfSK8f1EJCYbNsC998IBB4S1AW6+GWbMUBLIFHEmgvrA1wnPF0Xb8jvNzGaY2fNm1rCgE5lZfzPLNbPcJUuWxBGriBRgw4awTnDz5mFMwKGH/jomoFq1VEcn5SXVjcUvA03cvRXwP+CJgg5y9xHunuPuOfU0P61I7BITwEUXhWmh//MfjQnIVHEmgm+AxG/4DaJtW7j7UndfFz19GFB/A5EUWr9+2wTwyivw/vtw4okaE5Cp4kwEU4FmZtbUzKoCvYGxiQeY2V4JT3sCn8YYj4gUYv16eOihXxPA7ruH2UHffx9OOEEJINPF1mvI3Tea2UBgAlAJeNTdZ5nZLUCuu48FrjCznsBG4CegX1zxiMi21q+HJ54IAywXLgxTQzzwAHTvrpt/NjF3T3UMpZKTk+O5ubmpDkOkQlu/PkwHPWxYmHerY0cYOhSOP14JIFOZ2TR3zyloX6obi0UkidavDyPrmzULi8XvtReMHw/vvqtSQDbL+PUIRCQsEv/MM/CXv/xaAhgxAo47Tjd/USIQyVirVoUun6NHh2/969aFcQBKAJKfEoFIBlm9OvT2GT06dPtcsyZU/1x8cVgr+LDDlABkW0oEIhXc2rVhoNfo0fDyy/DLL6H753nnhZt/585hvW6RwigRiFRA69aFCd9Gj4aXXoKVK6FOnTCz7llnwZFHaiZQKTklApEKYsMGmDgx3PxffBGWL4fatcMqYGeeCd26QZUqqY5SKiIlApE0t3lzGPX7xz/Cjz/CLrvAySeHm/8xx0DVqqmOUCo6JQKRNPbpp9C/P7z1VvjGf9VVYdDXjjumOjLJJEoEImlo3Tr461/hz3+GGjXgscfg3HPV40fioUQgkmbeeiuUAj79NCwI/49/hF5AInFRpzKRNLF8OVxyCXTp8ut4gJEjlQQkfkoEImng3/8Oy0COGBHWBJ41C3r0SHVUki1UNSSSQt98AwMHhnWA27QJA8LaaXkmSTKVCERSYPNmuP/+UAqYMAH+9jf44AMlAUkNlQhEkmzWrLAK2LvvwrHHwj//Cfvsk+qoJJuVqERgZjub2Q7R4+Zm1tPMNIZRpBTWroUbbghVQJ99Bk8+GUoDSgKSaiWtGnoTqGZm9YFXgbOBx4t7kZl1N7O5ZjbPzAYXcdxpZuZmVuDqOSIV2dy5cPvt0KoV3Hor9O4Nc+bA2WdrXICkh5ImAnP31cCpwP3ufgZwUJEvMKsEDAd6AAcCfczswAKOqwlcCbxfmsBLa+HCMDingq3MKRXQpk1hLMA118D++0OLFnDttVCzZigBPPkk1K2b6ihFflXSNgIzs8OAvsAF0bbi5jbsAMxz9/nRCUYBvYDZ+Y77E3AbcE0JY9kuI0fCkCGw665w6aVxvpNko1WrwmygL70U1gH48ccwAVy3bnDFFfB//weNGqU6SpGClTQRXAVcD7zo7rPMbB9gUjGvqQ98nfB8EdAx8QAzaws0dPdXzKzQRGBm/YH+AI2283/T4MHw9ttw5ZWhiN6583adRmSLb78NK4CNHQuvvRamhdh1VzjxROjZM6wBXKtWqqMUKV6JEoG7vwG8ARA1Gv/o7leU5Y2j89wJ9CvB+48ARgDk5ORsV+XODjuEUkH79nDGGTBtGuy99/acSbKVO8ycGW78L70EU6eG7U2bhhHBvXpBp06aCloqnpL2GnrGzGqZ2c7ATGB2Ud/gI98ADROeN4i25akJHAxMNrMFwKHA2DgbjHfdNczjvnIlnH56+AYnUpzNm8Okb/vtF0qTf/xj+GLx5z+HxPDFF2E+oK5dlQSkYippY/GB7r4COBkYDzQl9BwqylSgmZk1NbOqQG9gbN5Od1/u7nXdvYm7NwHeA3q6e24p/4ZSOfjg8J/63XdDNZFIUaZOhcMPh/PPh3r1wroAixfDe+/B9dfDQQep549UfCVNBFWicQMnA2PdfQNQZBWNu28EBgITgE+B56L2hVvMrGcZYi6zM86A666DBx+Ehx9OZSSSrr7/Hi64ADp0CD3OnnwyfHm48ELYc89URydSvkraWPwgsAD4GHjTzBoDK4p7kbuPA8bl23ZjIcd2LWEs5WLYMPjwQ7jsMmjZEjp2LP41kvk2bIDhw+Gmm2DNmtAF9IYbQtdPkUxlvp0d682scvStP6lycnI8N7d8ao+WLoWcnPCff9o02GOPcjmtVFATJ4aunrNnh1XA7r47jAMQyQRmNs3dC2yDLWlj8S5mdqeZ5UY/fwd2LtcoU6BOndB4/NNPobpow4ZURySpsHBh6DxwzDFhGoiXXoLx45UEJHuUtI3gUWAlcGb0swJ4LK6gkql169BOMGUK/P73qY5GkmnNGrj55jDyd/z4MP3DrFlhDIAagCWblLSNYF93Py3h+c1mNj2GeFLiN7+B3NzQBTAnJ8wBI5nLPZQEr746lAbOOivMBdSwYfGvFclEJS0RrDGzLWNxzawTsCaekFLjb38L/cD79w+NyJKZZs+G446D004Lo34nTYJRo5QEJLuVNBEMAIab2YJo8Nd9wMWxRZUClSvD6NGhr/gpp4S5YiRzLFwYSgCHHBJKf/feGxJ+166pjkwk9UqUCNz9Y3c/BGgFtHL3NsBRsUaWArvvHtaO/f77MFXwxqT3iZLy9P33cN99YdqHJk3grrvgvPPCWgADB4bkLyKlXKrS3VdEI4wBro4hnpTLyQkrRk2cGEaOSsWybBk8+mhY+WvvveHyy8PMoH/+c5gKYsSIUOoTkV+V5TtRxvar6NcvTC1wxx0hMZx1VqojkqKsXh0WfX/22dD7Z/162Hdf+MMfoE8fOHCbVTBEJFFZEkFGL/Hyj3/Axx+HOWYOOCBMNibpY/36sMjLs8+G2UB/+SWUAC67LNz8c3LUBVSkpIpMBGa2koJv+AbsFEtEaaJqVXj+eWjbNjQeT50Ku+2W6qiy26ZN8MYb4eb/wgvw88/h36Rv33Dz79IFKhW3XJKIbKPIRODuWT3Dyp57hhtOly5Qv34Yddq4cZinqG/fVEeXPWbPDjPGjhwZZv7ceWc4+eRw8z/22JC0RWT7qd9EMebPD3PPr10bni9cGMYagJJBnJYtC9/8H38cPvgg9PA58cRwzU88EapXT3WEIpljuyedS5XynHSuJJo0CTf//Bo3hgULkhZGVti0KfTWeuyxMPJ33bowM+x554UEsPvuqY5QpOIqatI5lQiK8dVXBW8vKDnI9pk3L3zzf+IJWLQIatcO8/6fd15oo1Gjr0i8SjWOIBs1alT4vmuu0XKX22vlytDfv0sXaNYM/vKXsHrc6NFhUfj77oN27ZQERJJBiaAYw4ZtWx+9005w9NFhnEGHDvDJJ6mJraLZvDn0+unXD/baK6wA9sMPIQl89VUYA3DmmVCtWqojFckusSYCM+tuZnPNbJ6ZDS5g/wAz+8TMppvZW2aWdkN/+vYNo1EbNw7fThs3DuvWvvYa/Oc/8N13oc/6nXeGG51s7auv4Jln4NJLw+LvXbuGaTz69IG334Y5c2Dw4NArS0RSI7bGYjOrBHwGHAssIixm38fdZyccUytvyopoHeNL3b17UedNdmNxcZYsgYsuCouZdOsW6rmzdSbLzZtDV88pU+Ctt8JPXhtLzZrQuXNIAKeeGrqAikjypKqxuAMwz93nR0GMAnoBWxJBwrxFEFY8q1hdmAjz1rz4YqjvvvLK0Mvl/vvDGgeZbt26MJNn3o3/7bdDt08IVT9duoTFfjp3DtdFk7yJpKc4/2vWB75OeL4I2GaJeDO7jDCBXVUKmdHUzPoD/QEaFdV6myJmob67a1c455xQnfTyyyEh1K6d6ujKz7Jl8M474aY/ZUoYbZ3XWN6iRVjusUuXcONv2lQNvSIVRZxVQ6cD3d39wuj52UBHdx9YyPG/AY5393OLOm+6VQ3lt3Ej3HYbDB0aRiY//nhoWK5INm+GL7+EGTPCz8cfh99ffBH2V64cevR07hx+OnXSjJ4i6S5VVUPfAIm15Q2ibYUZBTwQYzxJUbkyDBkCxx8Pv/1tWBB90KAwDXI69oZZvjz0esq72c+YEZ7/8kvYbxa6d7ZpE3r7dOoUekqpjl8kc8SZCKYCzcysKSEB9Aa2qjk3s2bu/nn09ETgczJETk5YAevaa8NMpq++Cg8/HG6qO+0UksIOMXfedQ8llDVrws9PP4WbfN4Nf8aMrQfG1a4dVvC64IIw22qrVnDQQZrOQSTTxZYI3H2jmQ0EJgCVgEfdfZaZ3QLkuvtYYKCZHQNsAH4GiqwWqmiqVw8Do046KYySPeywrfdXrRqSQl5iKO5xlSphzqO8G3vi44Ker1lTcJfWSpVCnf7hh8OAAb/e9OvXV72+SDbSXENJsnRp6GK6atW2N+7Cbu75H69fv3ViyJ8oCtuW97xWrfAN/4AD0rOaSkTio7mG0kCdOmGRGxGRdKMpJkREspwSgYhIllMiEBHJckoEIiJZTolARCTLKRGIiGQ5JQIRkSynRJAEI0dCkyZhSokmTcJzEZF0oQFlMRs5Evr3h9Wrw/OFC8NzCNNVi4ikmkoEMRsy5NckkGf16rBdRCQdKBHELG+pxpJuFxFJNiWCmBW2oFoaLrQmIllKiSBmw4ZtO59/9ephu4hIOlAiiFnfvjBiBDRuHOb6b9w4PFdDsYikC/UaSoK+fXXjF5H0pRKBiEiWizURmFl3M5trZvPMbHAB+682s9lmNsPMJppZ4zjjERGRbcWWCMysEjAc6AEcCPQxswPzHfYRkOPurYDngb/FFY+IiBQszhJBB2Ceu8939/XAKKBX4gHuPsnd84ZbvQc0iDEeEREpQJyJoD7wdcLzRdG2wlwAjC9oh5n1N7NcM8tdsmRJOYZYMWiuIhGJU1r0GjKz3wI5wJEF7Xf3EcAIgJycHE9iaCmnuYpEJG5xlgi+ARomPG8QbduKmR0DDAF6uvu6GOOpkDRXkYjELc5EMBVoZmZNzawq0BsYm3iAmbUBHiQkgR9ijKXC0lxFIhK32BKBu28EBgITgE+B59x9lpndYmY9o8NuB2oA/zKz6WY2tpDTZS3NVSQicYu1jcDdxwHj8m27MeHxMXG+fyYYNmzrNgLQXEUiUr40sjjNaa4iEYlbWvQakqJpriIRiZNKBCIiWU6JQEQkyykRZAGNTBaRoqiNIMNpZLKIFEclggynkckiUhwlggynkckiUhwlggynkckiUhwlggw3bFgYiZxII5NFJJESQYbTyGQRKY4SQRbo2xcWLIDNm8Pv0iYBdT8VyWzqPipFUvdTkcynEoEUSd1PRTKfEoEUSd1PRTKfEoEUSd1PRTKfEoEUSd1PRTJfrInAzLqb2Vwzm2dmgwvYf4SZfWhmG83s9Dhjke1THt1P1etIJL3F1mvIzCoBw4FjgUXAVDMb6+6zEw77CugH/D6uOKTsyrIwjnodiaS/OEsEHYB57j7f3dcDo4BeiQe4+wJ3nwFsjjEOSSH1OhJJf3EmgvrA1wnPF0XbSs3M+ptZrpnlLlmypFyCk+Qoj15HqloSiVeFaCx29xHunuPuOfXq1Ut1OFIKZe11lFe1tHAhuP9ataRkIFJ+4kwE3wANE543iLZJFilrryNVLYnEL85EMBVoZmZNzawq0BsYG+P7SRoqa68jDWgTiV9sicDdNwIDgQnAp8Bz7j7LzG4xs54AZtbezBYBZwAPmtmsuOKR1CnLpHca0CYSv1jbCNx9nLs3d/d93X1YtO1Gdx8bPZ7q7g3cfWd3r+PuB8UZj1Q85TGgTY3NIkWrEI3Fkr3KWrWkxmaR4pm7pzqGUsnJyfHc3NxUhyEVRJMm4eafX+PGoZpKJFuY2TR3zylon0oEktE0jkGkeEoEktE0jkGkeEoEktE0jkGkeEoEktHSYRyDqpYk3WnNYsl4ZZk9tVGjghubS1u1pNlXJZ2pRCBSBFUtSTZQIhApQjpULYGqlyReSgQixUj1FBnl0XNJiUSKokQgEqPymCKjrNVL6gIrxVEiEIlReaz5XNbqpfJop1CJIrMpEYjErCxVS1D26qWyJhJVTWU+JQKRNFfW6qWyJpJ0qJpSIomXEoFImitr9VJZE0mqq6aUSJLA3SvUT7t27VxESufpp90bN3Y3C7+ffrrkr23c2D3cgrf+ady4ZK83K/j1Zsl5/6efdq9efevXVq9eumtQlutXHq8vD0CuF3JfTfmNvbQ/SgQiyVXWG2m2J5J0SETuKUwEQHdgLjAPGFzA/h2B0dH+94EmxZ1TiUAk+cpyI8r2RJLqRJQnJYkAqAR8AewDVAU+Bg7Md8ylwD+jx72B0cWdV4lApOLJ5kSS6kSUp6hEEGdjcQdgnrvPd/f1wCigV75jegFPRI+fB442M4sxJhFJgbJ0oU11Y3lZe12luvtvScSZCOoDXyc8XxRtK/AYd98ILAfq5D+RmfU3s1wzy12yZElM4YpIuqrIiSTViagkKkT3UXcf4e457p5Tr169VIcjIhVMKhNJqhNRScS5HsE3QMOE5w2ibQUds8jMKgO7AEtjjElEpNTKsqZFWV+f97ohQ0J1UKNGIQmU53oWcSaCqUAzM2tKuOH3Bn6T75ixwLnAu8DpwOtRo4aIiETKmoiKE1sicPeNZjYQmEDoQfSou88ys1sIrddjgUeAp8xsHvATIVmIiEgSxbpUpbuPA8bl23ZjwuO1wBlxxiAiIkWrEI3FIiISHyUCEZEsp0QgIpLlrKJ10jGzJcDCVMdRiLrAj6kOogiKr2zSPT5I/xgVX9mUJb7G7l7gQKwKlwjSmZnluntOquMojOIrm3SPD9I/RsVXNnHFp6ohEZEsp0QgIpLllAjK14hUB1AMxVc26R4fpH+Miq9sYolPbQQiIllOJQIRkSynRCAikuWUCErJzBqa2SQzm21ms8zsygKO6Wpmy81sevRzY0HnijHGBWb2SfTeuQXsNzO7x8zmmdkMM2ubxNj2T7gu081shZldle+YpF8/M3vUzH4ws5kJ23Yzs/+Z2efR79qFvPbc6JjPzezcJMV2u5nNif79XjSzXQt5bZGfhZhjHGpm3yT8O55QyGu7m9nc6PM4OInxjU6IbYGZTS/ktbFew8LuKUn9/BW2hqV+Cl2LeS+gbfS4JvAZ267F3BX4TwpjXADULWL/CcB4wIBDgfdTFGcl4DvCQJeUXj/gCKAtMDNh29+AwdHjwcBtBbxuN2B+9Lt29Lh2EmI7DqgcPb6toNhK8lmIOcahwO9L8Bkocm3zuOLLt//vwI2puIaF3VOS+flTiaCU3H2xu38YPV4JfMq2S3Cmu17Akx68B+xqZnulII6jgS/cPeUjxd39TcJU6IkS19R+Aji5gJceD/zP3X9y95+B/wHd447N3V/1sLwrwHuEhZ9SppDrVxIlWdu8zIqKL1on/Uzg2fJ+35Io4p6StM+fEkEZmFkToA3wfgG7DzOzj81svJkdlNzIcOBVM5tmZv0L2F+S9aSToTeF/+dL5fXLs4e7L44efwfsUcAx6XAtzyeU8ApS3GchbgOj6qtHC6naSIfr1wX43t0/L2R/0q5hvntK0j5/SgTbycxqAC8AV7n7iny7PyRUdxwC3AuMSXJ4nd29LdADuMzMjkjy+xfLzKoCPYF/FbA71ddvGx7K4WnX19rMhgAbgZGFHJLKz8IDwL5Aa2AxofolHfWh6NJAUq5hUfeUuD9/SgTbwcyqEP7BRrr7v/Pvd/cV7r4qejwOqGJmdZMVn7t/E/3+AXiRUPxOVJL1pOPWA/jQ3b/PvyPV1y/B93lVZtHvHwo4JmXX0sz6AScBfaMbxTZK8FmIjbt/7+6b3H0z8FAh753Sz6KFtdJPBUYXdkwyrmEh95Skff6UCEopqk98BPjU3e8s5Jg9o+Mwsw6E67w0SfHtbGY18x4TGhVn5jtsLHCOBYcCyxOKoMlS6LewVF6/fPLW1Cb6/VIBx0wAjjOz2lHVx3HRtliZWXfgWqCnu68u5JiSfBbijDGx3emUQt57y9rmUSmxN+G6J8sxwBx3X1TQzmRcwyLuKcn7/MXVEp6pP0BnQhFtBjA9+jkBGAAMiI4ZCMwi9IB4Dzg8ifHtE73vx1EMQ6LtifEZMJzQW+MTICfJ13Bnwo19l4RtKb1+hKS0GNhAqGe9AKgDTAQ+B14DdouOzQEeTnjt+cC86Oe8JMU2j1A3nPcZ/Gd07N7AuKI+C0m8fk9Fn68ZhJvaXvljjJ6fQOgp80VcMRYUX7T98bzPXcKxSb2GRdxTkvb50xQTIiJZTlVDIiJZTolARCTLKRGIiGQ5JQIRkSynRCAikuWUCEQiZrbJtp4ZtdxmwjSzJokzX4qkk8qpDkAkjaxx99apDkIk2VQiEClGNB/936I56T8ws/2i7U3M7PVoUrWJZtYo2r6HhTUCPo5+Do9OVcnMHormnH/VzHaKjr8imot+hpmNStGfKVlMiUDkVzvlqxo6K2HfcndvCdwH3BVtuxd4wt1bESZ9uyfafg/whodJ89oSRqQCNAOGu/tBwDLgtGj7YKBNdJ4B8fxpIoXTyGKRiJmtcvcaBWxfABzl7vOjycG+c/c6ZvYjYdqEDdH2xe5e18yWAA3cfV3COZoQ5o1vFj2/Dqji7rea2X+BVYRZVsd4NOGeSLKoRCBSMl7I49JYl/B4E7+20Z1ImPupLTA1mhFTJGmUCERK5qyE3+9Gj98hzJYJ0BeYEj2eCFwCYGaVzGyXwk5qZjsADd19EnAdsAuwTalEJE765iHyq51s6wXM/+vueV1Ia5vZDMK3+j7RtsuBx8zsGmAJcF60/UpghJldQPjmfwlh5suCVAKejpKFAfe4+7Jy+ntESkRtBCLFiNoIctz9x1THIhIHVQ2JiGQ5lQhERLKcSgQiIllOiUBEJMspEYiIZDklAhGRLKdEICKS5f4fnzB/vnW17MwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "acc = history.history['binary_accuracy']\n",
    "val_acc = history.history['val_binary_accuracy']\n",
    "loss = history.history['loss']\n",
    "val_loss = history.history['val_loss']\n",
    "\n",
    "epochs = range(1, len(acc) + 1)\n",
    "\n",
    "# \"bo\" is for \"blue dot\"\n",
    "plt.plot(epochs, loss, 'bo', label='Training loss')\n",
    "# b is for \"solid blue line\"\n",
    "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
    "plt.title('Training and validation loss')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAtCklEQVR4nO3deZxU1Zn/8c9Do2ADIjSoSAMNiij8ZG0xYlRMjKI4EA0akRhQMwhqMvobx5holGiYidEkjnHJkLhLBvCnIa5xQUUTN1oEXFHQVkFUBNlBtuf3x7kN1c2t7qJr7e7v+/W6r7p7PXW7+j51zrn3XHN3REREamqW7wBERKQwKUGIiEgsJQgREYmlBCEiIrGUIEREJJYShIiIxFKCkJSZ2eNmNjbT6+aTmVWa2fFZ2K+b2UHR+B/N7BeprFuP9xljZk/WN06R2pjug2jczGxdwmQx8DWwLZo+392n5j6qwmFmlcCP3P3pDO/XgZ7uvihT65pZGfAhsIe7b81IoCK1aJ7vACS73L111XhtJ0Mza66TjhQKfR8Lg6qYmigzG2pmS8zsp2b2GXCnmbUzs0fMbLmZfRWNlyZs85yZ/SgaH2dm/zCzG6J1PzSzk+q5bncze97M1prZ02Z2i5ndlyTuVGK81sz+Ge3vSTPrkLD8bDP7yMxWmNkVtRyfI8zsMzMrSph3qpktiMYHm9lLZrbKzJaZ2c1mtmeSfd1lZr9KmP6PaJtPzezcGusON7PXzWyNmX1iZpMSFj8fva4ys3VmdmTVsU3YfoiZzTGz1dHrkFSPzW4e5/Zmdmf0Gb4ys5kJy0aa2bzoMyw2s2HR/GrVeWY2qervbGZlUVXbeWb2MfBMNP/+6O+wOvqO9EnYfi8z+23091wdfcf2MrNHzezHNT7PAjM7Ne6zSnJKEE3b/kB7oBswnvB9uDOa7gpsBG6uZfsjgIVAB+A3wO1mZvVY9y/Aq0AJMAk4u5b3TCXGs4BzgH2BPYFLAcysN3BbtP8DovcrJYa7vwKsB75VY79/ica3AZdEn+dI4NvABbXETRTDsCie7wA9gZrtH+uBHwL7AMOBiWb23WjZMdHrPu7e2t1fqrHv9sCjwE3RZ/sd8KiZldT4DLscmxh1Hed7CVWWfaJ9/T6KYTBwD/Af0Wc4BqhM8h5xjgUOBU6Mph8nHKd9gblAYpXoDcAgYAjhe3wZsB24G/hB1Upm1g/oTDg2sjvcXUMTGQj/qMdH40OBzUDLWtbvD3yVMP0coYoKYBywKGFZMeDA/ruzLuHksxUoTlh+H3Bfip8pLsYrE6YvAP4ejV8FTEtY1io6Bscn2fevgDui8TaEk3e3JOteDPw1YdqBg6Lxu4BfReN3AL9OWO/gxHVj9nsj8PtovCxat3nC8nHAP6Lxs4FXa2z/EjCurmOzO8cZ6EQ4EbeLWe9/quKt7fsXTU+q+jsnfLYetcSwT7ROW0IC2wj0i1mvJfAVoV0HQiK5NRv/U419UAmiaVvu7puqJsys2Mz+JyqyryFUaeyTWM1Sw2dVI+6+IRptvZvrHgCsTJgH8EmygFOM8bOE8Q0JMR2QuG93Xw+sSPZehNLCaWbWAjgNmOvuH0VxHBxVu3wWxfGfhNJEXarFAHxU4/MdYWbPRlU7q4EJKe63at8f1Zj3EeHXc5Vkx6aaOo5zF8Lf7KuYTbsAi1OMN86OY2NmRWb266iaag07SyIdoqFl3HtF3+npwA/MrBkwmlDikd2kBNG01byE7d+BXsAR7r43O6s0klUbZcIyoL2ZFSfM61LL+unEuCxx39F7liRb2d3fJpxgT6J69RKEqqp3Cb9S9wZ+Xp8YCCWoRH8BHgK6uHtb4I8J+63rksNPCVVCiboCS1OIq6bajvMnhL/ZPjHbfQIcmGSf6wmlxyr7x6yT+BnPAkYSquHaEkoZVTF8CWyq5b3uBsYQqv42eI3qOEmNEoQkakMotq+K6rOvzvYbRr/IK4BJZranmR0J/EuWYvx/wClm9s2oQfka6v4f+Avwb4QT5P014lgDrDOzQ4CJKcYwAxhnZr2jBFUz/jaEX+ebovr8sxKWLSdU7fRIsu/HgIPN7Cwza25m3wd6A4+kGFvNOGKPs7svI7QN3Bo1Zu9hZlUJ5HbgHDP7tpk1M7PO0fEBmAecGa1fDoxKIYavCaW8YkIprSqG7YTqut+Z2QFRaePIqLRHlBC2A79FpYd6U4KQRDcCexF+nb0M/D1H7zuG0NC7glDvP51wYohzI/WM0d3fAi4knPSXEeqpl9Sx2f8SGk6fcfcvE+ZfSjh5rwX+FMWcSgyPR5/hGWBR9JroAuAaM1tLaDOZkbDtBmAy8E8LV099o8a+VwCnEH79ryA02p5SI+5U3Ujtx/lsYAuhFPUFoQ0Gd3+V0Aj+e2A1MJudpZpfEH7xfwX8kuolsjj3EEpwS4G3ozgSXQq8AcwBVgLXUf2cdg9wGKFNS+pBN8pJwTGz6cC77p71Eow0Xmb2Q2C8u38z37E0VCpBSN6Z2eFmdmBUJTGMUO88M89hSQMWVd9dAEzJdywNmRKEFIL9CZdgriNcwz/R3V/Pa0TSYJnZiYT2ms+puxpLaqEqJhERiaUShIiIxGo0nfV16NDBy8rK8h2GiEiD8tprr33p7h3jljWaBFFWVkZFRUW+wxARaVDMrObd9zuoiklERGIpQYiISCwlCBERiaUEISIisZQgREQkVtYShJndYWZfmNmbSZabmd1kZouixwEOTFg21szej4ax2YpRRKQhmzoVysqgWbPwOnVqXVvsnmyWIO4ChtWy/CTCowR7Eh53eRvseGzi1YRHVA4GrjazdlmMU0SkXtI9Qaez/dSpMH48fPQRuIfX8eMzmySyliDc/XlCF7zJjATu8eBlwtOqOhGeRfuUu1c9seopak80ItJA5fMEm+726Z6g093+iitgw4bq8zZsCPMzJpvPMyU8AerNJMseAb6ZMD0LKCf08Z743NxfAJcm2cd4wsNmKrp27eoi0nDcd597cbF7OD2Gobg4zG8I23frVn3bqqFbt9xsbxa/vVlq21cBKrwxPpPa3ae4e7m7l3fsGHunuIhkUTq/wNP9BZzv7T/+ePfmZ3r7rjUfVlvH/PrIZ4JYSvVn85ZG85LNF5ECkm4VSb5PsPk+Qae7/eTJUFxcfV5xcZifKflMEA8BP4yuZvoGsNrDs26fAE6InnXbDjghmiciGZbPEkC+T7D5PkGnu/2YMTBlCnTrBmbhdcqUMD9jktU9pTsQnuW7jPDc2iXAecAEYEK03IBbgMWE58qWJ2x7LuF5vYuAc1J5v0GDBu1exZtII3DffaHO2iy8plp/XrVtOnXw6daB57sNId3tq/ZR3+Ofie0zgVraILLaSJ3LQQlCmpqG3sha9RnyeYIthBN0vtWWIBrNE+XKy8td3X1LU1JWFur9a+rWDSor696+WbNwSq/JDLZvr3v7qjaIxGqm4uIsVHNIVpnZa+5eHresQV/FJNKU5buRNSd14JJXShAieZROI3G+G1khJIPKylDiqKxUcmhslCBE8iTdy0QbxFUw0qCpDUIkT9JtQ4CQTK64IlQrde0akoNO8LI71AYhkiXpVBGl24YAquKR7FKCEKmndKuIctFVgkg6lCBE6indO4lz0VWCSDqUIETqKd0qIjUSS6FTgpAmLZ+XmYLaEKSwKUFIk5Xvy0xFCp0ShDRZ6bYhqIpIGjvdByFNVrp9EYk0BroPQiSGLjMVqZ0ShDRZakMQqZ0ShDRo6VyFpDYEkdo1z3cAIvVV83kEVVchQeon+TFjlBBEklEJQhqsdK9CEpHaKUFIg5WJzu5EJDklCMmrfN/JLCLJKUFI3uhOZpHCpgQheaM7mUUKm+6klrzRncwi+ac7qaUgqQ1BpLApQUjeqA1BpLApQUjeqA1BpLDpTmrJK93JLFK4VIIQEZFYShCSlnRudBORwqYqJqm3THSWJyKFSyUIqTd1lifSuClBSL2pszyRxk0JQupNN7qJNG5KEFJvutFNpHFTgpB6041uIo2bEkQTl+5lqmPGQGVl6FyvslLJQaQx0WWuTZguUxWR2qgE0YTpMlURqY0SRBOmy1RFpDZZTRBmNszMFprZIjO7PGZ5NzObZWYLzOw5MytNWLbNzOZFw0PZjLOp0mWqIlKbrCUIMysCbgFOAnoDo82sd43VbgDucfe+wDXAfyUs2+ju/aNhRLbibMp0maqI1CabJYjBwCJ3/8DdNwPTgJE11ukNPBONPxuzXLJIl6mKSG2ymSA6A58kTC+J5iWaD5wWjZ8KtDGzkmi6pZlVmNnLZvbduDcws/HROhXLly/PYOhNhy5TFZFk8t1IfSlwrJm9DhwLLAW2Rcu6RQ/SPgu40cwOrLmxu09x93J3L+/YsWPOghYRaQqyeR/EUqBLwnRpNG8Hd/+UqARhZq2B77n7qmjZ0uj1AzN7DhgALM5ivCIikiCbJYg5QE8z625mewJnAtWuRjKzDmZWFcPPgDui+e3MrEXVOsBRwNtZjFVERGrIWoJw963ARcATwDvADHd/y8yuMbOqq5KGAgvN7D1gP6Dq+plDgQozm09ovP61uytBiIjkkLl7vmPIiPLycq+oqMh3GCIiDYqZvRa19+4i343UkiY9E1pEskWd9TVg6mxPRLJJJYgGTJ3tiUg2KUE0YOpsT0SySQmiAVNneyKSTUoQDZg62xORbFKCaMDU2Z6IZJOuYmrgxoxRQhCR7FAJQkREYilBiIhILCUIERGJpQQhIiKxlCBERCSWEoSIiMRSghARkVhKECIiEksJIs/0PAcRKVS6kzqP9DwHESlkKkHkkZ7nICKFTAkij/Q8BxEpZEoQeaTnOYhIIVOCyCM9z0FECpkSRB7peQ4iUsh0FVOe6XkOIlKoVIIQEZFYShAiIhJLCUJERGIpQYiISCwlCBERiaUEISIisZQgREQklhKEiIjEUoIQEZFYShAiIhJLCUJERGIpQYiISCwlCBERiZVSgjCzVmbWLBo/2MxGmNke2Q1NRETyKdUSxPNASzPrDDwJnA3cla2gREQk/1JNEObuG4DTgFvd/XSgT50bmQ0zs4VmtsjMLo9Z3s3MZpnZAjN7zsxKE5aNNbP3o2Fsqh9IREQyI+UEYWZHAmOAR6N5RXVsUATcApwE9AZGm1nvGqvdANzj7n2Ba4D/irZtD1wNHAEMBq42s3YpxppTU6dCWRk0axZep07Nd0QiIpmRaoK4GPgZ8Fd3f8vMegDP1rHNYGCRu3/g7puBacDIGuv0Bp6Jxp9NWH4i8JS7r3T3r4CngGEpxpozU6fC+PHw0UfgHl7Hj1eSEJHGIaUE4e6z3X2Eu18XNVZ/6e4/qWOzzsAnCdNLonmJ5hOqrQBOBdqYWUmK22Jm482swswqli9fnspHyagrroANG6rP27AhzBcRaehSvYrpL2a2t5m1At4E3jaz/8jA+18KHGtmrwPHAkuBbalu7O5T3L3c3cs7duyYgXB2z8cf7958EZGGJNUqpt7uvgb4LvA40J1wJVNtlgJdEqZLo3k7uPun7n6auw8ArojmrUpl20LQtevuzRcRaUhSTRB7RPc9fBd4yN23AF7HNnOAnmbW3cz2BM4EHkpcwcw6VN1fQWjjuCMafwI4wczaRY3TJ0TzCsrkyVBcXH1ecXGYLyLS0KWaIP4HqARaAc+bWTdgTW0buPtW4CLCif0dYEbUwH2NmY2IVhsKLDSz94D9gMnRtiuBawlJZg5wTTSvoIwZA1OmQLduYBZep0wJ80VEGjpzr6sgkGRDs+ZREigI5eXlXlFRke8wREQaFDN7zd3L45al2kjd1sx+V3XFkJn9llCaEBGRRirVKqY7gLXAGdGwBrgzW0GJiEj+NU9xvQPd/XsJ0780s3lZiKdJWrECNm2CDh2gRYt8RyMiEqSaIDaa2Tfd/R8AZnYUsDF7YTUdDz4Io0fD5s1hunXrkCiSDSUlu07voX51RSQLUk0QE4B7zKxtNP0VoA700nTPPXDOOXDEETBuHHz55a7DwoXhde3a5Ptp2zYklr32CkNx8c7xVKe7dIFevULSMcvZIRCRApZSgnD3+UA/M9s7ml5jZhcDC7IYW6N2661w4YXw7W/DzJnhBF+br78OVVFxSeTLL2H9+tDNx8aNO4fly3edt3HjztJKnH32gYMPDski8bVnz13v+RCRxi2dy1w/dveCuWe4IV3m+pvfwE9/Cv/yLzBjBrRsmdv337atesJYvz50NLhwIbz33s7XTz6pvl2XLtWTRtV4t25QVGvfviJSqGq7zDXVKqbY/aaxbZPkDr/4RbjT+swzQxVTPtoPiopCiSWx1HLooTCsRn+569fDokXVk8bChaG32tWrd663556haqpNG9h77+pD3Ly4+R07qmpLpNCkkyDqV/RoorZvh0sugZtugh/9CP74x8L/1d2qFfTrF4ZE7qH66r33dg4rVsCaNTuHzz8P7SZV09vq6IKxrAzOOCMMAwcqWYgUglqrmMxsLfGJwIC93D2dBJNRhVzFtG1beE7EHXeEJPHb3zatE6B7qMqqShaJiWPNmpBcnngCnn4atm6FHj12Jov+/ZvWsRLJtdqqmOrdBlFoCjVBbN4MZ58d2hquugomTdIJL5kVK0KD/YwZMGtWSKwHHbQzWfTtq2MnkmlKEHmycSOcfjo8+ihcfz1cemm+I2o4vvwS/vrXkCyeeSZU0fXqtTNZ9OmjZCGSCUoQebBuHYwYAc89B7fdBuefn++IGq4vvtiZLJ57LiSLQw/dmSx613zSuYikTAkix776Ck4+GebMgbvugh/8IN8RNR6ffx7uPp8xA2bPDu0bffqEtoqSEmjfPvlr27YqdYjUpASRQ198ASecAO+8A9Omwamn5juixmvZspAsHnwQPvwQVq6sfvltTUVF0K7dzoSRmDyOPDLcl7LXXrmLX6QQKEHkyJIlcPzx4ZnUM2eGRCG5tXVrKMGtXBkavVN5rbrjvE0bGDUqlPiGDoVmqfZ1nIYlS8IVXF9/DQccEIbOnWG//aB5wVwjKI1Ztm6UkwSLF4fksHIlPPkkfPOb+Y6oaWrePNx017Fj6tts2xbaNu67D+6/H+68E0pL4ayzQrI47LDMxbdtG7zySrhw4dFHYf78+PXMQpKoShqJySNxukOH3CQyaZpUgsiAt98OyWHz5vBrcNCgvIQhGbBhAzz8MNx7L/z97+GE3rdvSBRnnRVO0Lvrq6/C9+LRR+Hxx0PJpagIjjoKTjkltFeVlMDSpfDpp8mHL77Ydd/Nm0OnTqHqrFWrnUNx8e5Np3tH//77h6o6aXhUxZQlCxbA7beHhuji4nCjV58+OQ1Bsmj5cpg+PZQsXnkl/Kr/1rdCsjjttNBFSBz38KOhqpTwz3+GRFNSEpLB8OGh+rFdu92LZ/Nm+OyzXRPH0qWh7WX9+p2dNlaNVw113cmeCV27woABYRg4MLx27qwLAwqdEkQGrVkTGp///OdwldKee4aG6P/8z3AHsDRO778fEsV998EHH4TG7JEjQ7I44YTQ9vHccyEhPPJI6PwQwtVVw4eHYfDg/HSv4g5btlRPGDWTyNY0ni7vHj7v66/D3LnhWFWdVjp02DVpHHSQqsUKiRJEmtzhpZdCUpg+Pfxz/Z//E/pU+sEPwi9DaRrc4eWXQ6KYNi20OZWU7OxWvbg4VDcOHx5KC6Wl+Y4499atC20rr7++c3jzzZCkIHQS2a9f9aTRp48efJUvShD1tHx5qIv+85/DZautW4env513Xvg1qKJz07Z5c2inuP/+8ByN4cPD1U+57r69Idi8Gd56q3rSmDcvlF4gJNYhQ+DYY8MweLAev5srShC7Ydu20Jbw5z/D3/4WfvUceWQoLZxxRt0P9hGR1GzfHrqTnzs3tNPMng1vvBGWtWwJ3/jGzoTxjW9k/h6VjRvD/TOLF4eq4r59Q2N7U/vhpwSRgo8/Dpc33nFHGC8pgR/+MJQW1PAskhsrVsALL4RkMXt2KGW4hxP44ME7E8aQIeHqq7qsWhUSwOLFIRklji9duuv6HTqERFE19OsXunJpzKVCJYhaLFkC//qv4TJEgO98J5QWRoxQEVck31atgn/8Y2fCmDs3lPKbN4fy8p0Jo02b+ESwYkX1/e23X2gkP/DAMFSNb9wYrkqsGt58M8yD0KDeq1f1xNG3b3jCYmMobShB1GLTplCFNGIEnHNOeHCNiBSmtWt3VkfNnh2uJEy8AqtZs3C5beLJv2q8R4/Uq4i3bQsJJjFpLFgQqqSqtG27s5Rxxhlw9NGZ/ay5ogQhIo3S+vXhqrLNm0MiKCsL1VHZsmZNKF1UJYz588Pr11/DAw+E/rwaGiUIEZEsWbUqVE3Pnx86jjzllNy+/3//d7hR8qqr6rd9bQlCt6uIiKRhn31C/2t9+8L3vgePPZab992+HS67DC6+OCSnbNwtrwQhIpKmdu3gqafCDbSnnRbuj8mmzZvDVZbXXw8XXBCej5KNu/SVIEREMqAqSRx6KHz3u6FUkQ1r14abMqdOhcmT4eabs9eFixKEiEiGtG8fbrQ95JDQV9dTT2V2/599Fi7rffbZcN/Wz3+e3UttlSBERDKopCQkiYMPDpfPz5qVmf2+9164JP+990KHkOPGZWa/tVGCEBHJsA4dQmLo2TNc+vrMM+nt75VXwt3j69eH0sOwYZmJsy5KECIiWVCVJHr0CJe+Pvdc/fbzyCNw3HHhxrwXX4TDD89omLVSghARyZKOHUPpoXv30LD8/PO7t/3tt4cG7969Q3I46KCshJmUEoSISBbtu29IEt26hWeEvPBC3du4wzXXhH7hjj8+lD722y/roe5CCUJEJMv22y8kiS5d4KSTQgeEyWzdChMmwNVXh3sdHn44f48ZUIIQEcmB/fcPSaJz55AkXnxx13U2bAh3Y0+ZAj/7WXjefT6ftJfVBGFmw8xsoZktMrPLY5Z3NbNnzex1M1tgZidH88vMbKOZzYuGP2YzThGRXOjUKVyF1KlTuBLppZd2LluxIlQnPfww/OEP4Tn3+e5OvHm2dmxmRcAtwHeAJcAcM3vI3d9OWO1KYIa732ZmvYHHgLJo2WJ375+t+ERE8uGAA0KSGDoUTjwx3Ey3//4hYXz4Yeg2Y9SofEcZZC1BAIOBRe7+AYCZTQNGAokJwoG9o/G2wKdZjEdEpCB07rwzSZxwQng63saNoXuOY47Jd3Q7ZbOKqTPwScL0kmheoknAD8xsCaH08OOEZd2jqqfZZhb7KA4zG29mFWZWsXz58gyGLiKSXaWlIUl06BD6UvrHPworOUD+G6lHA3e5eylwMnCvmTUDlgFd3X0A8H+Bv5jZ3jU3dvcp7l7u7uUdO3bMaeAiIunq0gXeeAPefRf69Ml3NLvKZoJYCnRJmC6N5iU6D5gB4O4vAS2BDu7+tbuviOa/BiwGDs5irCIieVFcHKqYClE2E8QcoKeZdTezPYEzgYdqrPMx8G0AMzuUkCCWm1nHqJEbM+sB9AQ+yGKsIiJSQ9Yaqd19q5ldBDwBFAF3uPtbZnYNUOHuDwH/DvzJzC4hNFiPc3c3s2OAa8xsC7AdmODuK7MVq4iI7ErPpBYRacL0TGoREdltShAiIhJLCUJERGIpQYiISCwlCBERiaUEISIisZQgREQklhKEiIjEUoIQEZFYShAiIhJLCUJERGIpQYiISCwlCBERiaUEISIisZQgREQklhKEiIjEUoIQEZFYShAiIhJLCUJERGIpQYiISKzm+Q5ARBq+LVu2sGTJEjZt2pTvUCSJli1bUlpayh577JHyNkoQIpK2JUuW0KZNG8rKyjCzfIcjNbg7K1asYMmSJXTv3j3l7VTFJCJp27RpEyUlJUoOBcrMKCkp2e0SnhKEiGSEkkNhq8/fRwlCRERiKUGISM5NnQplZdCsWXidOjW9/a1YsYL+/fvTv39/9t9/fzp37rxjevPmzbVuW1FRwU9+8pM632PIkCHpBdkAqZFaRHJq6lQYPx42bAjTH30UpgHGjKnfPktKSpg3bx4AkyZNonXr1lx66aU7lm/dupXmzeNPd+Xl5ZSXl9f5Hi+++GL9gmvAVIIQkZy64oqdyaHKhg1hfiaNGzeOCRMmcMQRR3DZZZfx6quvcuSRRzJgwACGDBnCwoULAXjuuec45ZRTgJBczj33XIYOHUqPHj246aabduyvdevWO9YfOnQoo0aN4pBDDmHMmDG4OwCPPfYYhxxyCIMGDeInP/nJjv0mqqys5Oijj2bgwIEMHDiwWuK57rrrOOyww+jXrx+XX345AIsWLeL444+nX79+DBw4kMWLF2f2QNVCJQgRyamPP969+elYsmQJL774IkVFRaxZs4YXXniB5s2b8/TTT/Pzn/+cBx54YJdt3n33XZ599lnWrl1Lr169mDhx4i73Drz++uu89dZbHHDAARx11FH885//pLy8nPPPP5/nn3+e7t27M3r06NiY9t13X5566ilatmzJ+++/z+jRo6moqODxxx/nb3/7G6+88grFxcWsXLkSgDFjxnD55Zdz6qmnsmnTJrZv3575A5WEEoSI5FTXrqFaKW5+pp1++ukUFRUBsHr1asaOHcv777+PmbFly5bYbYYPH06LFi1o0aIF++67L59//jmlpaXV1hk8ePCOef3796eyspLWrVvTo0ePHfcZjB49milTpuyy/y1btnDRRRcxb948ioqKeO+99wB4+umnOeeccyguLgagffv2rF27lqVLl3LqqacC4Wa3XFIVk4jk1OTJEJ0DdyguDvMzrVWrVjvGf/GLX3Dcccfx5ptv8vDDDye9J6BFixY7xouKiti6dWu91knm97//Pfvttx/z58+noqKizkb0fFKCEJGcGjMGpkyBbt3ALLxOmVL/BupUrV69ms6dOwNw1113ZXz/vXr14oMPPqCyshKA6dOnJ42jU6dONGvWjHvvvZdt27YB8J3vfIc777yTDVEDzcqVK2nTpg2lpaXMnDkTgK+//nrH8lxQghCRnBszBiorYfv28Jrt5ABw2WWX8bOf/YwBAwbs1i/+VO21117ceuutDBs2jEGDBtGmTRvatm27y3oXXHABd999N/369ePdd9/dUcoZNmwYI0aMoLy8nP79+3PDDTcAcO+993LTTTfRt29fhgwZwmeffZbx2JOxqtb3hq68vNwrKiryHYZIk/TOO+9w6KGH5juMvFu3bh2tW7fG3bnwwgvp2bMnl1xySb7D2iHu72Rmr7l77HW+KkGIiGTIn/70J/r370+fPn1YvXo1559/fr5DSouuYhIRyZBLLrmkoEoM6VIJQkREYilBiIhILCUIERGJldUEYWbDzGyhmS0ys8tjlnc1s2fN7HUzW2BmJycs+1m03UIzOzGbcYqIyK6yliDMrAi4BTgJ6A2MNrPeNVa7Epjh7gOAM4Fbo217R9N9gGHArdH+RER2cdxxx/HEE09Um3fjjTcyceLEpNsMHTqUqkvjTz75ZFatWrXLOpMmTdpxP0IyM2fO5O23394xfdVVV/H000/vRvSFK5sliMHAInf/wN03A9OAkTXWcWDvaLwt8Gk0PhKY5u5fu/uHwKJofyIiuxg9ejTTpk2rNm/atGlJO8yr6bHHHmOfffap13vXTBDXXHMNxx9/fL32VWiyeZlrZ+CThOklwBE11pkEPGlmPwZaAVVHtTPwco1tO9d8AzMbD4wH6JqNnr5EZLddfDFEj2bImP794cYbky8fNWoUV155JZs3b2bPPfeksrKSTz/9lKOPPpqJEycyZ84cNm7cyKhRo/jlL3+5y/ZlZWVUVFTQoUMHJk+ezN13382+++5Lly5dGDRoEBDucZgyZQqbN2/moIMO4t5772XevHk89NBDzJ49m1/96lc88MADXHvttZxyyimMGjWKWbNmcemll7J161YOP/xwbrvtNlq0aEFZWRljx47l4YcfZsuWLdx///0ccsgh1WKqrKzk7LPPZv369QDcfPPNOx5adN1113HffffRrFkzTjrpJH7961+zaNEiJkyYwPLlyykqKuL+++/nwAMPTOu457uRejRwl7uXAicD95pZyjG5+xR3L3f38o4dO2YtSBEpbO3bt2fw4ME8/vjjQCg9nHHGGZgZkydPpqKiggULFjB79mwWLFiQdD+vvfYa06ZNY968eTz22GPMmTNnx7LTTjuNOXPmMH/+fA499FBuv/12hgwZwogRI7j++uuZN29etRPypk2bGDduHNOnT+eNN95g69at3HbbbTuWd+jQgblz5zJx4sTYaqyqbsHnzp3L9OnTdzz1LrFb8Pnz53PZZZcBoVvwCy+8kPnz5/Piiy/SqVOn9A4q2S1BLAW6JEyXRvMSnUdoY8DdXzKzlkCHFLcVkQJU2y/9bKqqZho5ciTTpk3j9ttvB2DGjBlMmTKFrVu3smzZMt5++2369u0bu48XXniBU089dUeX2yNGjNix7M033+TKK69k1apVrFu3jhNPrP3amYULF9K9e3cOPvhgAMaOHcstt9zCxRdfDISEAzBo0CAefPDBXbYvhG7Bs1mCmAP0NLPuZrYnodH5oRrrfAx8G8DMDgVaAsuj9c40sxZm1h3oCbyajSAz/WxcEcmPkSNHMmvWLObOncuGDRsYNGgQH374ITfccAOzZs1iwYIFDB8+PGk333UZN24cN998M2+88QZXX311vfdTparL8GTdhRdCt+BZSxDuvhW4CHgCeIdwtdJbZnaNmVWl5X8H/tXM5gP/C4zz4C1gBvA28HfgQnfflukYq56N+9FH4L7z2bhKEiINT+vWrTnuuOM499xzdzROr1mzhlatWtG2bVs+//zzHVVQyRxzzDHMnDmTjRs3snbtWh5++OEdy9auXUunTp3YsmULUxNOEm3atGHt2rW77KtXr15UVlayaNEiIPTKeuyxx6b8eQqhW/CstkG4+2PufrC7H+juk6N5V7n7Q9H42+5+lLv3c/f+7v5kwraTo+16uXvtf9V6ytWzcUUkN0aPHs38+fN3JIh+/foxYMAADjnkEM466yyOOuqoWrcfOHAg3//+9+nXrx8nnXQShx9++I5l1157LUcccQRHHXVUtQblM888k+uvv54BAwZUe150y5YtufPOOzn99NM57LDDaNasGRMmTEj5sxRCt+BNurvvZs1CyaEms9BPvYikRt19Nwzq7ns3JLsyVlfMiog08QSRy2fjiog0NE06QeTr2bgijVFjqa5urOrz92nyDwwaM0YJQSRdLVu2ZMWKFZSUlGBm+Q5HanB3VqxYsdv3RzT5BCEi6SstLWXJkiUsX74836FIEi1btqS0tHS3tlGCEJG07bHHHnTv3j3fYUiGNek2CBERSU4JQkREYilBiIhIrEZzJ7WZLQc+yncctegAfJnvIGqh+NKj+NKj+NKTTnzd3D32eQmNJkEUOjOrSHY7eyFQfOlRfOlRfOnJVnyqYhIRkVhKECIiEksJInem5DuAOii+9Ci+9Ci+9GQlPrVBiIhILJUgREQklhKEiIjEUoLIEDPrYmbPmtnbZvaWmf1bzDpDzWy1mc2LhqvyEGelmb0Rvf8uj+Cz4CYzW2RmC8xsYA5j65VwbOaZ2Rozu7jGOjk9hmZ2h5l9YWZvJsxrb2ZPmdn70Wu7JNuOjdZ538zG5jC+683s3ejv91cz2yfJtrV+F7IY3yQzW5rwNzw5ybbDzGxh9F28PIfxTU+IrdLM5iXZNhfHL/a8krPvoLtryMAAdAIGRuNtgPeA3jXWGQo8kuc4K4EOtSw/GXgcMOAbwCt5irMI+IxwE0/ejiFwDDAQeDNh3m+Ay6Pxy4HrYrZrD3wQvbaLxtvlKL4TgObR+HVx8aXyXchifJOAS1P4+y8GegB7AvNr/j9lK74ay38LXJXH4xd7XsnVd1AliAxx92XuPjcaXwu8A3TOb1T1MhK4x4OXgX3MrFMe4vg2sNjd83p3vLs/D6ysMXskcHc0fjfw3ZhNTwSecveV7v4V8BQwLBfxufuT7r41mnwZ2L0+njMoyfFLxWBgkbt/4O6bgWmE455RtcVn4cEWZwD/m+n3TVUt55WcfAeVILLAzMqAAcArMYuPNLP5Zva4mfXJbWQAOPCkmb1mZuNjlncGPkmYXkJ+Et2ZJP/HzPcx3M/dl0XjnwH7xaxTKMfxXEKJME5d34VsuiiqArsjSfVIIRy/o4HP3f39JMtzevxqnFdy8h1UgsgwM2sNPABc7O5raiyeS6gy6Qf8AZiZ4/AAvunuA4GTgAvN7Jg8xFArM9sTGAHcH7O4EI7hDh7K8gV5rbiZXQFsBaYmWSVf34XbgAOB/sAyQjVOIRpN7aWHnB2/2s4r2fwOKkFkkJntQfgjTnX3B2sud/c17r4uGn8M2MPMOuQyRndfGr1+AfyVUJRPtBTokjBdGs3LpZOAue7+ec0FhXAMgc+rqt2i1y9i1snrcTSzccApwJjoBLKLFL4LWeHun7v7NnffDvwpyfvm+/g1B04DpidbJ1fHL8l5JSffQSWIDInqK28H3nH33yVZZ/9oPcxsMOH4r8hhjK3MrE3VOKEx880aqz0E/NCCbwCrE4qyuZL0l1u+j2HkIaDqipCxwN9i1nkCOMHM2kVVKCdE87LOzIYBlwEj3H1DknVS+S5kK77ENq1Tk7zvHKCnmXWPSpRnEo57rhwPvOvuS+IW5ur41XJeyc13MJst8E1pAL5JKOYtAOZFw8nABGBCtM5FwFuEKzJeBobkOMYe0XvPj+K4IpqfGKMBtxCuIHkDKM9xjK0IJ/y2CfPydgwJiWoZsIVQh3seUALMAt4HngbaR+uWA39O2PZcYFE0nJPD+BYR6p6rvod/jNY9AHistu9CjuK7N/puLSCc6DrVjC+aPplw1c7iXMYXzb+r6juXsG4+jl+y80pOvoPqakNERGKpiklERGIpQYiISCwlCBERiaUEISIisZQgREQklhKESB3MbJtV72U2Yz2LmllZYk+iIoWkeb4DEGkANrp7/3wHIZJrKkGI1FP0PIDfRM8EeNXMDorml5nZM1FndLPMrGs0fz8Lz2eYHw1Dol0Vmdmfov7+nzSzvaL1fxI9B2CBmU3L08eUJkwJQqRue9WoYvp+wrLV7n4YcDNwYzTvD8Dd7t6X0FHeTdH8m4DZHjoaHEi4AxegJ3CLu/cBVgHfi+ZfDgyI9jMhOx9NJDndSS1SBzNb5+6tY+ZXAt9y9w+iDtU+c/cSM/uS0H3Elmj+MnfvYGbLgVJ3/zphH2WEPvt7RtM/BfZw91+Z2d+BdYQea2d61EmhSK6oBCGSHk8yvju+Thjfxs62weGEfrEGAnOiHkZFckYJQiQ93094fSkaf5HQ+yjAGOCFaHwWMBHAzIrMrG2ynZpZM6CLuz8L/BRoC+xSihHJJv0iEanbXlb9wfV/d/eqS13bmdkCQilgdDTvx8CdZvYfwHLgnGj+vwFTzOw8QklhIqEn0ThFwH1REjHgJndflaHPI5IStUGI1FPUBlHu7l/mOxaRbFAVk4iIxFIJQkREYqkEISIisZQgREQklhKEiIjEUoIQEZFYShAiIhLr/wOQUB4xy9OL4AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.clf()   # clear figure\n",
    "acc_values = history_dict['binary_accuracy']\n",
    "val_acc_values = history_dict['val_binary_accuracy']\n",
    "\n",
    "plt.plot(epochs, acc, 'bo', label='Training acc')\n",
    "plt.plot(epochs, val_acc, 'b', label='Validation acc')\n",
    "plt.title('Training and validation accuracy')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "The dots are the training loss and accuracy, while the solid lines are the validation loss and accuracy. Note that your own results may vary \n",
    "slightly due to a different random initialization of your network.\n",
    "\n",
    "As you can see, the training loss decreases with every epoch and the training accuracy increases with every epoch. That's what you would \n",
    "expect when running gradient descent optimization -- the quantity you are trying to minimize should get lower with every iteration. But that \n",
    "isn't the case for the validation loss and accuracy: they seem to peak at the fourth epoch. This is an example of what we were warning \n",
    "against earlier: a model that performs better on the training data isn't necessarily a model that will do better on data it has never seen \n",
    "before. In precise terms, what you are seeing is \"overfitting\": after the second epoch, we are over-optimizing on the training data, and we \n",
    "ended up learning representations that are specific to the training data and do not generalize to data outside of the training set.\n",
    "\n",
    "In this case, to prevent overfitting, we could simply stop training after three epochs. In general, there is a range of techniques you can \n",
    "leverage to mitigate overfitting, which we will cover in the next chapter.\n",
    "\n",
    "Let's train a new network from scratch for four epochs, then evaluate it on our test data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/4\n",
      "49/49 [==============================] - 1s 8ms/step - loss: 0.4577 - accuracy: 0.8232\n",
      "Epoch 2/4\n",
      "49/49 [==============================] - 0s 6ms/step - loss: 0.2581 - accuracy: 0.9088\n",
      "Epoch 3/4\n",
      "49/49 [==============================] - 0s 5ms/step - loss: 0.1993 - accuracy: 0.9295\n",
      "Epoch 4/4\n",
      "49/49 [==============================] - 0s 7ms/step - loss: 0.1680 - accuracy: 0.9399\n",
      "782/782 [==============================] - 1s 1ms/step - loss: 0.2952 - accuracy: 0.8830\n"
     ]
    }
   ],
   "source": [
    "model = models.Sequential()\n",
    "model.add(layers.Dense(16, activation='relu', input_shape=(10000,)))\n",
    "model.add(layers.Dense(16, activation='relu'))\n",
    "model.add(layers.Dense(1, activation='sigmoid'))\n",
    "\n",
    "model.compile(optimizer='rmsprop',\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.fit(x_train, y_train, epochs=4, batch_size=512)\n",
    "results = model.evaluate(x_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.29524675011634827, 0.8830400109291077]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our fairly naive approach achieves an accuracy of 88%. With state-of-the-art approaches, one should be able to get close to 95%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using a trained network to generate predictions on new data\n",
    "\n",
    "After having trained a network, you will want to use it in a practical setting. You can generate the likelihood of reviews being positive \n",
    "by using the `predict` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.2911505 ],\n",
       "       [0.99929786],\n",
       "       [0.94573534],\n",
       "       ...,\n",
       "       [0.18926457],\n",
       "       [0.10064304],\n",
       "       [0.6434837 ]], dtype=float32)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.predict(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, the network is very confident for some samples (0.99 or more, or 0.01 or less) but less confident for others (0.6, 0.4). \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Further experiments\n",
    "\n",
    "\n",
    "* We were using 2 hidden layers. Try to use 1 or 3 hidden layers and see how it affects validation and test accuracy.\n",
    "* Try to use layers with more hidden units or less hidden units: 32 units, 64 units...\n",
    "* Try to use the `mse` loss function instead of `binary_crossentropy`.\n",
    "* Try to use the `tanh` activation (an activation that was popular in the early days of neural networks) instead of `relu`.\n",
    "\n",
    "These experiments will help convince you that the architecture choices we have made are all fairly reasonable, although they can still be \n",
    "improved!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusions\n",
    "\n",
    "\n",
    "Here's what you should take away from this example:\n",
    "\n",
    "* There's usually quite a bit of preprocessing you need to do on your raw data in order to be able to feed it -- as tensors -- into a neural \n",
    "network. In the case of sequences of words, they can be encoded as binary vectors -- but there are other encoding options too.\n",
    "* Stacks of `Dense` layers with `relu` activations can solve a wide range of problems (including sentiment classification), and you will \n",
    "likely use them frequently.\n",
    "* In a binary classification problem (two output classes), your network should end with a `Dense` layer with 1 unit and a `sigmoid` activation, \n",
    "i.e. the output of your network should be a scalar between 0 and 1, encoding a probability.\n",
    "* With such a scalar sigmoid output, on a binary classification problem, the loss function you should use is `binary_crossentropy`.\n",
    "* The `rmsprop` optimizer is generally a good enough choice of optimizer, whatever your problem. That's one less thing for you to worry \n",
    "about.\n",
    "* As they get better on their training data, neural networks eventually start _overfitting_ and end up obtaining increasingly worse results on data \n",
    "never-seen-before. Make sure to always monitor performance on data that is outside of the training set.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}