TensorFlow/TextClassification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the second lesson on TensorFlow. It is around [text classification](https://www.tensorflow.org/tutorials/keras/basic_text_classification). This will be an example of a binary or two-class classification. It will classify movie reviews as either positive or negative based on the text of the review.  \n",
    "  \n",
    "We will use the [IMDB dataset](https://www.tensorflow.org/api_docs/python/tf/keras/datasets/imdb) that contains 50,000 movie reviews. We will split them into 25,000 for training and 25,000 for testing. We will pass in the parameter 'num_words' that will keep the 10,000 most frequently occuring words. If we brought in every word our dataset would be massive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Import keras and helper libraries\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/imdb.npz\n",
      "17465344/17464789 [==============================] - 2s 0us/step\n"
     ]
    }
   ],
   "source": [
    "#Download the built-in dataset\n",
    "imdb = keras.datasets.imdb\n",
    "(train_data, train_labels), (test_data, test_labels) = imdb.load_data(num_words=10000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exploring the Data**  \n",
    "LIke any dataset you are going to use we need to explore it to get a feel for it. This will help us preprocess the data as well as build a model. The first steps are to see the size of your data. Next you would see what the actual content of the data. There are many more steps and each dataset will reveal when you need to do when you start training.  \n",
    "  \n",
    "In the following code you can see that we have 25,000 reviews and those reviews are broken down by replacing the words with integer values representing the specific word in the dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training entries: 25000, labels 25000\n",
      "[1, 14, 22, 16, 43, 530, 973, 1622, 1385, 65, 458, 4468, 66, 3941, 4, 173, 36, 256, 5, 25, 100, 43, 838, 112, 50, 670, 2, 9, 35, 480, 284, 5, 150, 4, 172, 112, 167, 2, 336, 385, 39, 4, 172, 4536, 1111, 17, 546, 38, 13, 447, 4, 192, 50, 16, 6, 147, 2025, 19, 14, 22, 4, 1920, 4613, 469, 4, 22, 71, 87, 12, 16, 43, 530, 38, 76, 15, 13, 1247, 4, 22, 17, 515, 17, 12, 16, 626, 18, 2, 5, 62, 386, 12, 8, 316, 8, 106, 5, 4, 2223, 5244, 16, 480, 66, 3785, 33, 4, 130, 12, 16, 38, 619, 5, 25, 124, 51, 36, 135, 48, 25, 1415, 33, 6, 22, 12, 215, 28, 77, 52, 5, 14, 407, 16, 82, 2, 8, 4, 107, 117, 5952, 15, 256, 4, 2, 7, 3766, 5, 723, 36, 71, 43, 530, 476, 26, 400, 317, 46, 7, 4, 2, 1029, 13, 104, 88, 4, 381, 15, 297, 98, 32, 2071, 56, 26, 141, 6, 194, 7486, 18, 4, 226, 22, 21, 134, 476, 26, 480, 5, 144, 30, 5535, 18, 51, 36, 28, 224, 92, 25, 104, 4, 226, 65, 16, 38, 1334, 88, 12, 16, 283, 5, 16, 4472, 113, 103, 32, 15, 16, 5345, 19, 178, 32]\n"
     ]
    }
   ],
   "source": [
    "print('Training entries: {}, labels {}'.format(len(train_data), len(train_labels)))\n",
    "print(train_data[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the previous notebook, every image was the same size so we knew what side our first layer of the neural network needed to be. In this dataset, each review is different lengths so we will need to handle that later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First review length: 218, Second review length: 189\n"
     ]
    }
   ],
   "source": [
    "print('First review length: {}, Second review length: {}'.format(len(train_data[0]),len(train_data[1])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Convert Ints back to Words**  \n",
    "Here is the code to convert the integers back to their actual words. This will be nice to have to validate your work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/imdb_word_index.json\n",
      "1646592/1641221 [==============================] - 0s 0us/step\n"
     ]
    }
   ],
   "source": [
    "# A dictionary mapping words to an integer index\n",
    "word_index = imdb.get_word_index()\n",
    "\n",
    "# The first indices are reserved\n",
    "word_index = {k:(v+3) for k,v in word_index.items()} \n",
    "word_index[\"<PAD>\"] = 0\n",
    "word_index[\"<START>\"] = 1\n",
    "word_index[\"<UNK>\"] = 2  # unknown\n",
    "word_index[\"<UNUSED>\"] = 3\n",
    "\n",
    "reverse_word_index = dict([(value, key) for (key, value) in word_index.items()])\n",
    "\n",
    "def decode_review(text):\n",
    "    return ' '.join([reverse_word_index.get(i, '?') for i in text])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"<START> this film was just brilliant casting location scenery story direction everyone's really suited the part they played and you could just imagine being there robert <UNK> is an amazing actor and now the same being director <UNK> father came from the same scottish island as myself so i loved the fact there was a real connection with this film the witty remarks throughout the film were great it was just brilliant so much that i bought the film as soon as it was released for <UNK> and would recommend it to everyone to watch and the fly fishing was amazing really cried at the end it was so sad and you know what they say if you cry at a film it must have been good and this definitely was also <UNK> to the two little boy's that played the <UNK> of norman and paul they were just brilliant children are often left out of the <UNK> list i think because the stars that play them all grown up are such a big profile for the whole film but these children are amazing and should be praised for what they have done don't you think the whole story was so lovely because it was true and was someone's life after all that was shared with us all\""
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "decode_review(train_data[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Preprocessing the Data**  \n",
    "First, we need to convert the words to tensors so that we can feed them into the neural network.  \n",
    "  \n",
    "We can do this 2 ways. First, we can do what they call one-hot encoding. I have done this in most of my assignments that revolved around text. What you do is created a column for each of the words and if that word is in the text you set it to 1. If it isn't, you set it to 0. The obvious downside is that we would have 10k extra columns. Second, we can pad the reviews so that they are all the same length. This is the approach the lesson decided to use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First review length: 256, Second review length: 256\n"
     ]
    }
   ],
   "source": [
    "train_data = keras.preprocessing.sequence.pad_sequences(train_data,\n",
    "                                                      value=word_index[\"<PAD>\"],\n",
    "                                                      padding='post',\n",
    "                                                      maxlen=256)\n",
    "\n",
    "test_data = keras.preprocessing.sequence.pad_sequences(test_data,\n",
    "                                                      value=word_index[\"<PAD>\"],\n",
    "                                                      padding='post',\n",
    "                                                      maxlen=256)\n",
    "\n",
    "#Validate\n",
    "print('First review length: {}, Second review length: {}'.format(len(train_data[0]),len(train_data[1])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Build the Model**  \n",
    "We need to make 2 decisions. First, how many layers. Second, how many units/neurons for each layer.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding (Embedding)        (None, None, 16)          160000    \n",
      "_________________________________________________________________\n",
      "global_average_pooling1d (Gl (None, 16)                0         \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 16)                272       \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 1)                 17        \n",
      "=================================================================\n",
      "Total params: 160,289\n",
      "Trainable params: 160,289\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# input shape is the vocabulary count used for the movie reviews (10,000 words)\n",
    "vocab_size = 10000\n",
    "\n",
    "model = keras.Sequential()\n",
    "model.add(keras.layers.Embedding(vocab_size, 16))\n",
    "model.add(keras.layers.GlobalAveragePooling1D())\n",
    "model.add(keras.layers.Dense(16, activation=tf.nn.relu))\n",
    "model.add(keras.layers.Dense(1, activation=tf.nn.sigmoid))\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a breakdown of the layers  \n",
    "  \n",
    "**vocab_size**: We need the input layer to be the same size as our text we are passing in.  \n",
    "**Embedding**: This takes in the int encoding and looks up the embedding vector for each word-index. These vectors are learned as the model trains. This adds a dimension to the output array (batch, sequence, embedding).  \n",
    "**GlobalAveragePooling1D**: This layer returns a fixed-length output vector for each example by averaging over the sequence dimension. This allows the model to handle input of variable length.  \n",
    "**Dense(16, activation=tf.nn.relu))**: This fixed-length output vector is piped through a fully-connected (Dense) layer with 16 hidden units.  \n",
    "**Dense(1, activation=tf.nn.sigmoid))**: Densely connected with a single output node (Positive or negative review). Using sigmoid makes this value between 0 and 1 representing a probability level."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Hidden Units**  \n",
    "Our neural network has 2 hidden layers between the input and output layers. The number of hidden layers and units allows the network a much larger freedom to learn. But, the downside is that it is much more computationally expensive."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Loss function and optimizer**  \n",
    "Every model needs a loss function and an optimizer for training. This model is a binary classification with a single probability output we will use *binary_crossentropy* loss function.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.compile(optimizer=tf.train.AdamOptimizer(),\n",
    "              loss='binary_crossentropy',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Validation Set**  \n",
    "In the previous notebook we were just given the validation set. This time we will create out own. The need for a validation set is that we need to check out work with data that the network has NEVER seen. This is different than the *testing* set that we will use to test our model. The validation set is used during training to tune our model. For our validation we will set aside 10k rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x_val = train_data[:10000]\n",
    "partial_x_train = train_data[10000:]\n",
    "\n",
    "y_val = train_labels[:10000]\n",
    "partial_y_train = train_labels[10000:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Train the model**  \n",
    "We will train the model using 40 epochs and mini-batches of 512 samples. This means that we iteration 40 times over all the samples in the training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 15000 samples, validate on 10000 samples\n",
      "Epoch 1/40\n",
      "15000/15000 [==============================] - 1s 65us/step - loss: 0.6912 - acc: 0.5649 - val_loss: 0.6888 - val_acc: 0.5882\n",
      "Epoch 2/40\n",
      "15000/15000 [==============================] - 1s 50us/step - loss: 0.6845 - acc: 0.6528 - val_loss: 0.6806 - val_acc: 0.7152\n",
      "Epoch 3/40\n",
      "15000/15000 [==============================] - 1s 48us/step - loss: 0.6722 - acc: 0.7279 - val_loss: 0.6658 - val_acc: 0.7295\n",
      "Epoch 4/40\n",
      "15000/15000 [==============================] - 1s 49us/step - loss: 0.6513 - acc: 0.7651 - val_loss: 0.6428 - val_acc: 0.7469\n",
      "Epoch 5/40\n",
      "15000/15000 [==============================] - 1s 47us/step - loss: 0.6209 - acc: 0.7867 - val_loss: 0.6102 - val_acc: 0.7803\n",
      "Epoch 6/40\n",
      "15000/15000 [==============================] - 1s 47us/step - loss: 0.5821 - acc: 0.8075 - val_loss: 0.5722 - val_acc: 0.7984\n",
      "Epoch 7/40\n",
      "15000/15000 [==============================] - 1s 52us/step - loss: 0.5379 - acc: 0.8274 - val_loss: 0.5314 - val_acc: 0.8135\n",
      "Epoch 8/40\n",
      "15000/15000 [==============================] - 1s 51us/step - loss: 0.4926 - acc: 0.8417 - val_loss: 0.4913 - val_acc: 0.8294\n",
      "Epoch 9/40\n",
      "15000/15000 [==============================] - 1s 53us/step - loss: 0.4500 - acc: 0.8556 - val_loss: 0.4549 - val_acc: 0.8395\n",
      "Epoch 10/40\n",
      "15000/15000 [==============================] - 1s 54us/step - loss: 0.4113 - acc: 0.8679 - val_loss: 0.4237 - val_acc: 0.8471\n",
      "Epoch 11/40\n",
      "15000/15000 [==============================] - 1s 55us/step - loss: 0.3783 - acc: 0.8777 - val_loss: 0.4000 - val_acc: 0.8526\n",
      "Epoch 12/40\n",
      "15000/15000 [==============================] - 1s 52us/step - loss: 0.3505 - acc: 0.8845 - val_loss: 0.3765 - val_acc: 0.8597\n",
      "Epoch 13/40\n",
      "15000/15000 [==============================] - 1s 50us/step - loss: 0.3255 - acc: 0.8917 - val_loss: 0.3600 - val_acc: 0.8652\n",
      "Epoch 14/40\n",
      "15000/15000 [==============================] - 1s 50us/step - loss: 0.3049 - acc: 0.8977 - val_loss: 0.3452 - val_acc: 0.8694\n",
      "Epoch 15/40\n",
      "15000/15000 [==============================] - 1s 50us/step - loss: 0.2871 - acc: 0.9025 - val_loss: 0.3337 - val_acc: 0.8724\n",
      "Epoch 16/40\n",
      "15000/15000 [==============================] - 1s 49us/step - loss: 0.2718 - acc: 0.9053 - val_loss: 0.3243 - val_acc: 0.8742\n",
      "Epoch 17/40\n",
      "15000/15000 [==============================] - 1s 50us/step - loss: 0.2571 - acc: 0.9123 - val_loss: 0.3165 - val_acc: 0.8776\n",
      "Epoch 18/40\n",
      "15000/15000 [==============================] - 1s 50us/step - loss: 0.2444 - acc: 0.9166 - val_loss: 0.3098 - val_acc: 0.8800\n",
      "Epoch 19/40\n",
      "15000/15000 [==============================] - 1s 51us/step - loss: 0.2327 - acc: 0.9210 - val_loss: 0.3044 - val_acc: 0.8803\n",
      "Epoch 20/40\n",
      "15000/15000 [==============================] - 1s 49us/step - loss: 0.2224 - acc: 0.9230 - val_loss: 0.3000 - val_acc: 0.8809\n",
      "Epoch 21/40\n",
      "15000/15000 [==============================] - 1s 48us/step - loss: 0.2124 - acc: 0.9274 - val_loss: 0.2962 - val_acc: 0.8823\n",
      "Epoch 22/40\n",
      "15000/15000 [==============================] - 1s 48us/step - loss: 0.2032 - acc: 0.9307 - val_loss: 0.2935 - val_acc: 0.8834\n",
      "Epoch 23/40\n",
      "15000/15000 [==============================] - 1s 53us/step - loss: 0.1948 - acc: 0.9338 - val_loss: 0.2914 - val_acc: 0.8835\n",
      "Epoch 24/40\n",
      "15000/15000 [==============================] - 1s 53us/step - loss: 0.1865 - acc: 0.9380 - val_loss: 0.2888 - val_acc: 0.8837\n",
      "Epoch 25/40\n",
      "15000/15000 [==============================] - 1s 50us/step - loss: 0.1791 - acc: 0.9417 - val_loss: 0.2873 - val_acc: 0.8848\n",
      "Epoch 26/40\n",
      "15000/15000 [==============================] - 1s 49us/step - loss: 0.1718 - acc: 0.9439 - val_loss: 0.2866 - val_acc: 0.8845\n",
      "Epoch 27/40\n",
      "15000/15000 [==============================] - 1s 48us/step - loss: 0.1656 - acc: 0.9469 - val_loss: 0.2862 - val_acc: 0.8850\n",
      "Epoch 28/40\n",
      "15000/15000 [==============================] - 1s 48us/step - loss: 0.1590 - acc: 0.9496 - val_loss: 0.2854 - val_acc: 0.8857\n",
      "Epoch 29/40\n",
      "15000/15000 [==============================] - 1s 51us/step - loss: 0.1530 - acc: 0.9514 - val_loss: 0.2853 - val_acc: 0.8857\n",
      "Epoch 30/40\n",
      "15000/15000 [==============================] - 1s 49us/step - loss: 0.1478 - acc: 0.9533 - val_loss: 0.2859 - val_acc: 0.8859\n",
      "Epoch 31/40\n",
      "15000/15000 [==============================] - 1s 51us/step - loss: 0.1416 - acc: 0.9567 - val_loss: 0.2867 - val_acc: 0.8866\n",
      "Epoch 32/40\n",
      "15000/15000 [==============================] - 1s 49us/step - loss: 0.1367 - acc: 0.9593 - val_loss: 0.2879 - val_acc: 0.8869\n",
      "Epoch 33/40\n",
      "15000/15000 [==============================] - 1s 49us/step - loss: 0.1309 - acc: 0.9613 - val_loss: 0.2893 - val_acc: 0.8860\n",
      "Epoch 34/40\n",
      "15000/15000 [==============================] - 1s 48us/step - loss: 0.1260 - acc: 0.9636 - val_loss: 0.2911 - val_acc: 0.8854\n",
      "Epoch 35/40\n",
      "15000/15000 [==============================] - 1s 49us/step - loss: 0.1215 - acc: 0.9645 - val_loss: 0.2927 - val_acc: 0.8862\n",
      "Epoch 36/40\n",
      "15000/15000 [==============================] - 1s 48us/step - loss: 0.1161 - acc: 0.9675 - val_loss: 0.2954 - val_acc: 0.8855\n",
      "Epoch 37/40\n",
      "15000/15000 [==============================] - 1s 50us/step - loss: 0.1117 - acc: 0.9681 - val_loss: 0.2981 - val_acc: 0.8858\n",
      "Epoch 38/40\n",
      "15000/15000 [==============================] - 1s 48us/step - loss: 0.1079 - acc: 0.9691 - val_loss: 0.3004 - val_acc: 0.8841\n",
      "Epoch 39/40\n",
      "15000/15000 [==============================] - 1s 50us/step - loss: 0.1031 - acc: 0.9710 - val_loss: 0.3030 - val_acc: 0.8841\n",
      "Epoch 40/40\n",
      "15000/15000 [==============================] - 1s 49us/step - loss: 0.0990 - acc: 0.9723 - val_loss: 0.3062 - val_acc: 0.8842\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(partial_x_train,\n",
    "                    partial_y_train,\n",
    "                    epochs=40,\n",
    "                    batch_size=512,\n",
    "                    validation_data=(x_val, y_val),\n",
    "                    verbose=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Evaluate the Model**  \n",
    "Now, we will evaluate how well we did in our training. We will get two return values (same as the last notebook). The first is the loss (error) and accuracty."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25000/25000 [==============================] - 0s 19us/step\n",
      "[0.32605546572208405, 0.8732]\n"
     ]
    }
   ],
   "source": [
    "results = model.evaluate(test_data, test_labels)\n",
    "\n",
    "print(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Graph Accuracy and Loss Over Time**  \n",
    "We will use the *History* object that is returned from *model.fit()*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['val_loss', 'val_acc', 'loss', 'acc'])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "history_dict = history.history\n",
    "history_dict.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The history dictionary has 4 entries. One for each monitored metric during training and validation. We can plot these to give us some visibility into the training of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xt8VNW5//HPQwggcgly8UKEAHJU\nwABpxGsFrfWAVvBWBYOKR0ux9VJte6RqvdDSUrWKWI6tWtEKSqn+tNS2cmyl9XIqAgooIAURJYLc\n5CqoBJ7fH2snDGGSTC6TmUy+79drv2b2nj17ntnJzDNrrb3WMndHREQEoEmqAxARkfShpCAiImWU\nFEREpIySgoiIlFFSEBGRMkoKIiJSRklB6pSZZZnZDjPrUpf7ppKZHWVmdX7ttpmdaWarYtaXmdlX\nE9m3Bq/1qJndUtPnV3Lcn5rZ43V9XEmdpqkOQFLLzHbErLYEvgD2ROvfdvdp1Tmeu+8BWtX1vo2B\nux9dF8cxs6uBke4+KObYV9fFsSXzKSk0cu5e9qUc/RK92t3/VtH+ZtbU3UvqIzYRqX+qPpJKRdUD\nvzezp81sOzDSzE4yszfMbIuZrTWzSWaWHe3f1MzczPKi9anR4381s+1m9i8z61bdfaPHh5jZv81s\nq5k9aGavm9moCuJOJMZvm9kKM9tsZpNinptlZveb2SYzex8YXMn5uc3MppfbNtnM7ovuX21mS6P3\n8370K76iYxWb2aDofkszezKKbTHwlTivuzI67mIzGxptPw74FfDVqGpuY8y5vTPm+WOi977JzJ43\ns8MTOTdVMbPzoni2mNnLZnZ0zGO3mNkaM9tmZu/FvNcTzeytaPs6M7sn0deTJHB3LVpwd4BVwJnl\ntv0U+BI4l/Aj4iDgeOAEQkmzO/Bv4Npo/6aAA3nR+lRgI1AIZAO/B6bWYN9OwHZgWPTYTcBuYFQF\n7yWRGP8ItAXygE9L3ztwLbAYyAXaA6+Ej0rc1+kO7AAOjjn2eqAwWj832seAM4BdQH702JnAqphj\nFQODovv3Av8A2gFdgSXl9r0YODz6m1waxXBo9NjVwD/KxTkVuDO6f1YUYz+gBfA/wMuJnJs47/+n\nwOPR/WOjOM6I/ka3ROc9G+gNfAgcFu3bDege3Z8LjIjutwZOSPVnoTEvKilIIl5z9z+5+1533+Xu\nc919jruXuPtK4GFgYCXPf8bd57n7bmAa4cuouvt+A1jg7n+MHrufkEDiSjDGn7v7VndfRfgCLn2t\ni4H73b3Y3TcBEyp5nZXAu4RkBfB1YIu7z4se/5O7r/TgZeDvQNzG5HIuBn7q7pvd/UPCr//Y153h\n7mujv8lThIRemMBxAYqAR919gbt/DowFBppZbsw+FZ2bygwHZrr7y9HfaALQhpCcSwgJqHdUBflB\ndO4gJPeeZtbe3be7+5wE34ckgZKCJGJ17IqZHWNmfzazT8xsGzAO6FDJ8z+Jub+TyhuXK9r3iNg4\n3N0Jv6zjSjDGhF6L8Au3Mk8BI6L7lxKSWWkc3zCzOWb2qZltIfxKr+xclTq8shjMbJSZLYyqabYA\nxyR4XAjvr+x47r4N2Ax0jtmnOn+zio67l/A36uzuy4DvE/4O66PqyMOiXa8EegHLzOxNMzs7wfch\nSaCkIIkofznmbwi/jo9y9zbA7YTqkWRaS6jOAcDMjP2/xMqrTYxrgSNj1qu6ZPb3wJnRL+1hhCSB\nmR0EPAP8nFC1kwP8b4JxfFJRDGbWHXgIuAZoHx33vZjjVnX57BpClVTp8VoTqqk+TiCu6hy3CeFv\n9jGAu09191MIVUdZhPOCuy9z9+GEKsJfAs+aWYtaxiI1pKQgNdEa2Ap8ZmbHAt+uh9d8ASgws3PN\nrClwA9AxSTHOAL5nZp3NrD1wc2U7u/s64DVgCrDM3ZdHDzUHmgEbgD1m9g3ga9WI4RYzy7HQj+Pa\nmMdaEb74NxDy49WEkkKpdUBuacN6HE8DV5lZvpk1J3w5v+ruFZa8qhHzUDMbFL32DwntQHPM7Fgz\nOz16vV3RsofwBi4zsw5RyWJr9N721jIWqSElBamJ7wNXED7wvyH8Uk6q6Iv3EuA+YBPQA3ib0K+i\nrmN8iFD3/w6hEfSZBJ7zFKHh+KmYmLcANwLPERprLyIkt0TcQSixrAL+Cvwu5riLgEnAm9E+xwCx\n9fAvAcuBdWYWWw1U+vwXCdU4z0XP70JoZ6gVd19MOOcPERLWYGBo1L7QHLib0A70CaFkclv01LOB\npRaubrsXuMTdv6xtPFIzFqpmRRoWM8siVFdc5O6vpjoekUyhkoI0GGY22MzaRlUQPyZc0fJmisMS\nyShKCtKQnAqsJFRBDAbOc/eKqo9EpAZUfSQiImVUUhARkTINbkC8Dh06eF5eXqrDEBFpUObPn7/R\n3Su7jBtogEkhLy+PefPmpToMEZEGxcyq6pkPqPpIRERiKCmIiEiZpCaF6LryZdG47GPjPH6/mS2I\nln9HA3uJiEiKJK1NIepxOpkwlHAxMNfMZrr7ktJ93P3GmP2vA/onKx4RqZndu3dTXFzM559/nupQ\nJAEtWrQgNzeX7OyKhr6qXDIbmgcAK0rHTI9mpxpGmCwknhGE8V5EJI0UFxfTunVr8vLyCIPTSrpy\ndzZt2kRxcTHdunWr+glxJLP6qDP7jwdfTAVDHZtZV8Jwui9X8PhoM5tnZvM2bNhQ7UCmTYO8PGjS\nJNxOq9ZU9CKN2+eff0779u2VEBoAM6N9+/a1KtUlMynE+w+qqPv0cMKMW3viPejuD7t7obsXduxY\n5WW2+5k2DUaPhg8/BPdwO3q0EoNIdSghNBy1/VslMykUs/8kIbmEUS3jGU4Y473O3Xor7Ny5/7ad\nO8P2UipJiIgEyUwKcwnzrnYzs2ZE87eW38nMjiaMrf6vZATx0UeVb1dJQiS9bdq0iX79+tGvXz8O\nO+wwOnfuXLb+5ZeJTbtw5ZVXsmzZskr3mTx5MtPq6IN/6qmnsmDBgjo5Vn1LWlJw9xLCbFGzgKXA\nDHdfbGbjzGxozK4jgOmepJH5ulQwkWLLlvDSS3DLLVWXJEQkcXVd8m7fvj0LFixgwYIFjBkzhhtv\nvLFsvVmzZkBoYN27t+LJ2qZMmcLRRx9d6et897vfpaio1nMNNXhJ7afg7n9x9/9w9x7uPj7adru7\nz4zZ5053P6APQ10ZPz4kgFhZWbB3L5x1VtUlCRFJXH2WvFesWEGfPn0YM2YMBQUFrF27ltGjR1NY\nWEjv3r0ZN25c2b6lv9xLSkrIyclh7Nix9O3bl5NOOon169cDcNtttzFx4sSy/ceOHcuAAQM4+uij\n+b//+z8APvvsMy688EL69u3LiBEjKCwsrLJEMHXqVI477jj69OnDLbfcAkBJSQmXXXZZ2fZJkyYB\ncP/999OrVy/69u3LyJEj6/ycJSLjezQXFcHDD0PXrmAWbp94Aj79FJ55Bg46KP7zKiphiEjFEmnD\nq0tLlizhqquu4u2336Zz585MmDCBefPmsXDhQl566SWWLDnwCvitW7cycOBAFi5cyEknncRjjz0W\n99juzptvvsk999xTlmAefPBBDjvsMBYuXMjYsWN5++23K42vuLiY2267jdmzZ/P222/z+uuv88IL\nLzB//nw2btzIO++8w7vvvsvll18OwN13382CBQtYuHAhv/rVr2p5dmom45MChMSwalUoHaxaFdZb\ntIALL4RHHjkwMTRtCrfdtm9dDdEiianvknePHj04/vjjy9affvppCgoKKCgoYOnSpXGTwkEHHcSQ\nIUMA+MpXvsKqVaviHvuCCy44YJ/XXnuN4cOHA9C3b1969+5daXxz5szhjDPOoEOHDmRnZ3PppZfy\nyiuvcNRRR7Fs2TJuuOEGZs2aRdu2bQHo3bs3I0eOZNq0aTXufFZbjSIpVKaoKCSGrl3DeqtWsGcP\n3HUX/OlPaogWqY6KStjJKnkffPDBZfeXL1/OAw88wMsvv8yiRYsYPHhw3Ov1S9shALKysigpKYl7\n7ObNmx+wT3WbPivav3379ixatIhTTz2VSZMm8e1vfxuAWbNmMWbMGN58800KCwvZsyfuVfpJ1eiT\nAuwrSbjD9u3w2mvQti0MHRoSgBqiRRITrw2vZcuwPdm2bdtG69atadOmDWvXrmXWrFl1/hqnnnoq\nM2bMAOCdd96JWxKJdeKJJzJ79mw2bdpESUkJ06dPZ+DAgWzYsAF355vf/CZ33XUXb731Fnv27KG4\nuJgzzjiDe+65hw0bNrCz/JdPPWhw8ynUh5NPhrfegnvu2b8aKZYaokUOVHrxzq23hs9Ily4hIdTH\nRT0FBQX06tWLPn360L17d0455ZQ6f43rrruOyy+/nPz8fAoKCujTp09Z1U88ubm5jBs3jkGDBuHu\nnHvuuZxzzjm89dZbXHXVVbg7ZsYvfvELSkpKuPTSS9m+fTt79+7l5ptvpnXr1nX+HqrS4OZoLiws\n9PqcZKdzZ1gTp8td166hdCGS6ZYuXcqxxx6b6jDSQklJCSUlJbRo0YLly5dz1llnsXz5cpo2Ta/f\n1/H+ZmY2390Lq3puer2TNHT33fCtb8GuXfu2HXRQ/RSHRSS97Nixg6997WuUlJTg7vzmN79Ju4RQ\nW5n1bpIgtjj8YTSZXY8ecP75qYtJRFIjJyeH+fPnpzqMpFJDcwJiG6KnTIHFi+Eb34DPPguP65JV\nEckUKilU06hRoR/DFVfA2WfDZZfBDTfsu0Kp9JJVqJ/GNRGRuqSkUAMjR4bEMHIkzJkDX3yx/+Ol\nl6wqKYhIQ6OkUEPDh0N2Nlx0UfzHdcmqiDREalOohQsvhIrm/NHYSSJ1Y9CgQQd0RJs4cSLf+c53\nKn1eq1atAFizZg0XVfDrbdCgQVR1ifvEiRP360R29tlns2XLlkRCr9Sdd97JvffeW+vj1DUlhVq6\n/36IesOXqa8enCKNwYgRI5g+ffp+26ZPn86IESMSev4RRxzBM888U+PXL58U/vKXv5CTk1Pj46U7\nJYVaKiqC3/4W2rUL64ccEkZlVXuCSN246KKLeOGFF/giarxbtWoVa9as4dRTTy3rN1BQUMBxxx3H\nH//4xwOev2rVKvr06QPArl27GD58OPn5+VxyySXsiumAdM0115QNu33HHXcAMGnSJNasWcPpp5/O\n6aefDkBeXh4bN24E4L777qNPnz706dOnbNjtVatWceyxx/Ktb32L3r17c9ZZZ+33OvEsWLCAE088\nkfz8fM4//3w2b95c9vq9evUiPz+/bCC+f/7zn2WTDPXv35/t27fX+NzGozaFOlBUBJdeCuecAy+/\nDH37pjoikeT43vegricU69cPou/TuNq3b8+AAQN48cUXGTZsGNOnT+eSSy7BzGjRogXPPfccbdq0\nYePGjZx44okMHTq0wnmKH3roIVq2bMmiRYtYtGgRBQUFZY+NHz+eQw45hD179vC1r32NRYsWcf31\n13Pfffcxe/ZsOnTosN+x5s+fz5QpU5gzZw7uzgknnMDAgQNp164dy5cv5+mnn+aRRx7h4osv5tln\nn610foTLL7+cBx98kIEDB3L77bdz1113MXHiRCZMmMAHH3xA8+bNy6qs7r33XiZPnswpp5zCjh07\naNGiRTXOdtVUUqgjZvD445CTAyNG7N8DWkRqJ7YKKbbqyN255ZZbyM/P58wzz+Tjjz9m3bp1FR7n\nlVdeKftyzs/PJz8/v+yxGTNmUFBQQP/+/Vm8eHGVg9299tprnH/++Rx88MG0atWKCy64gFdffRWA\nbt260a9fP6Dy4bkhzO+wZcsWBg4cCMAVV1zBK6+8UhZjUVERU6dOLes5fcopp3DTTTcxadIktmzZ\nUuc9qlVSqEOdOoXEMGQI3HwzRJMpiWSMyn7RJ9N5553HTTfdxFtvvcWuXbvKfuFPmzaNDRs2MH/+\nfLKzs8nLy4s7XHaseKWIDz74gHvvvZe5c+fSrl07Ro0aVeVxKhs3rnlMQ2NWVlaV1UcV+fOf/8wr\nr7zCzJkz+clPfsLixYsZO3Ys55xzDn/5y1848cQT+dvf/sYxxxxTo+PHo5JCHRs8OBSxH3wQ/vzn\nsE09nkVqp1WrVgwaNIj/+q//2q+BeevWrXTq1Ins7Gxmz57Nh6Vj0VTgtNNOY1r0AXz33XdZtGgR\nEIbdPvjgg2nbti3r1q3jr3/9a9lzWrduHbfe/rTTTuP5559n586dfPbZZzz33HN89atfrfZ7a9u2\nLe3atSsrZTz55JMMHDiQvXv3snr1ak4//XTuvvtutmzZwo4dO3j//fc57rjjuPnmmyksLOS9996r\n9mtWRiWFJJgwAWbPDr2f77gjlBrU41mkdkaMGMEFF1yw35VIRUVFnHvuuRQWFtKvX78qfzFfc801\nXHnlleTn59OvXz8GDBgAhFnU+vfvT+/evQ8Ydnv06NEMGTKEww8/nNmzZ5dtLygoYNSoUWXHuPrq\nq+nfv3+lVUUVeeKJJxgzZgw7d+6ke/fuTJkyhT179jBy5Ei2bt2Ku3PjjTeSk5PDj3/8Y2bPnk1W\nVha9evUqm0Wurmjo7CRZsgQKC8N4SfFKoRp6WxoKDZ3d8NRm6GxVHyVJr16hD0NF1ZLq8Swi6UhJ\nIYlGjw5zL8SjHs8iko6UFJLIDO67L9zGUo9naWgaWjVzY1bbv5WSQpKNGRMamkt17aoez9KwtGjR\ngk2bNikxNADuzqZNm2rVoS2pVx+Z2WDgASALeNTdJ8TZ52LgTsCBhe5+aTJjSoWf/xzWrYMnnwyX\nqfbuneqIRBKXm5tLcXExGzZsSHUokoAWLVqQm5tb4+cn7eojM8sC/g18HSgG5gIj3H1JzD49gRnA\nGe6+2cw6ufv6yo7bUK4+Km/jRjj66JAQ/vnPA6uURESSKR2uPhoArHD3le7+JTAdGFZun28Bk919\nM0BVCaEh69AB7r4bXn0Vnngi1dGIiMSXzKTQGVgds14cbYv1H8B/mNnrZvZGVN10ADMbbWbzzGxe\nQy7CXnklnHwy/OAHsGlTqqMRETlQMpNCvAqS8nVVTYGewCBgBPComR0wULm7P+zuhe5e2LGiWW0a\ngCZN4Ne/hi1bYOzYVEcjInKgZCaFYuDImPVcYE2cff7o7rvd/QNgGSFJZKzjjoMbb4RHH4XXX091\nNCIi+0tmUpgL9DSzbmbWDBgOzCy3z/PA6QBm1oFQnbQyiTGlhTvugCOPhGuuCe0LGixPRNJF0pKC\nu5cA1wKzgKXADHdfbGbjzGxotNssYJOZLQFmAz9094yvbW/VKgyr/c478K1vhUHy3PcNlqfEICKp\nogHxUsQdDj44/mQ8GixPROpaOlySKpUwq3h2Ng2WJyKpoqSQQl27xt+uwfJEJFWUFFJo/PgDR1HV\nYHkikkpKCilUVASPPAKHHhrW27bVYHkiklpKCilWVASffAIXXgi7d8OgQamOSEQaMyWFNHHPPVBS\nop7OIpJaSgppols3uOkmmDoV3ngj1dGISGOlpJBGbrkFDjsMvve90I9BRKS+KSmkkdat4Wc/gzlz\n4KmnUh2NiDRGSgpp5ooroKAgTOH52WepjkZEGhslhTTTpAlMnAgffxwm5RERqU9KCmnoq1+Fiy8O\nSeGBBzSKqojUHw2Il6Y+/BB69oS9e2HPnn3bW7ZUBzcRqT4NiNfAde0aEkBsQgDYuRNuvTU1MYlI\n5lNSSGNbt8bfrlFURSRZlBTSmEZRFZH6pqSQxjSKqojUNyWFNFY6iuphh4X1Nm3UyCwiyaWkkOaK\nimDtWrjsMvj8czjhhFRHJCKZTEmhgZgwAbKz4fvfT3UkIpLJlBQaiCOOgNtug5kzYdasVEcjIplK\nSaEBufFG6NEjjKK6e3eqoxGRTKSk0IA0bw733QfvvQe/+lWqoxGRTKSk0MCcey6cdRbceSesX5/q\naEQk0yQ1KZjZYDNbZmYrzOyAiSbNbJSZbTCzBdFydTLjyQRmYRRVDXchIsmQtKRgZlnAZGAI0AsY\nYWa94uz6e3fvFy2PJiueTHLssXDddfDb38L8+amORkQySTJLCgOAFe6+0t2/BKYDw5L4eo3K7bdD\nhw5www2aulNE6k4yk0JnYHXMenG0rbwLzWyRmT1jZkfGO5CZjTazeWY2b8OGDcmItcHJyQlTd77+\nOnTqpPkWRKRuJDMpWJxt5X/T/gnIc/d84G/AE/EO5O4Pu3uhuxd27NixjsNsuJo3D20MGzeG0sKH\nH8Lo0UoMIlJzyUwKxUDsL/9cYE3sDu6+yd2/iFYfAb6SxHgyzo9/fGDVkRqgRaQ2kpkU5gI9zayb\nmTUDhgMzY3cws8NjVocCS5MYT8apaF4FzbcgIjXVNFkHdvcSM7sWmAVkAY+5+2IzGwfMc/eZwPVm\nNhQoAT4FRiUrnkzUpUuoMoq3XUSkJjRHcwM2bVpoQ9i5c9+27GyYMkXDa4vI/jRHcyNQVBTmVyid\noe2gg8JVSKecktq4RKThUlJo4IqKYNWq0OD83nuhpDBmjPouiEjNKClkkC5d4Oc/D0NrT52a6mhE\npCFSUsgw11wDJ50UhtfWgHkiUl1KChkmKyvM67x9e0gMIiLVoaSQgXr3Dh3Ynn4a/vznVEcjIg2J\nkkKGGjsWevUK1Unbt6c6GhFpKJQUMlTz5vDoo1BcDLfckupoRKShUFLIYCedBF//epi600yjqIpI\n1ZQUMti0afDqq/vWNYqqiFRFSSGD3Xor7Nq1/zaNoioilVFSyGAaRVVEqktJIYNVNFrqoYfWbxwi\n0nAoKWSw8eOhZcv9t5nB7t3w6aepiUlE0puSQgaLHUXVLNzeeSds2wajRmnQPBE5UNIm2ZH0UFR0\n4NwKOTlwww1w333w/e+nJi4RSU8qKTRC110H558fej3/61+pjkZE0omSQiNkBo89Brm5cMklal8Q\nkX2UFBqpnByYMQM++QSuuELtCyISKCk0YscfD/feCy+8ADfeqMQgImpobvSuuw7efx8eeACaNoV7\n7gnVSyLSOCVUUjCzHmbWPLo/yMyuN7Oc5IYm9eGpp+D558P9X/4Shg1TiUGkMUu0+uhZYI+ZHQX8\nFugGPJW0qKReTJsWBsiLHfbiT3+CCy5IXUwiklqJJoW97l4CnA9MdPcbgcOTF5bUh1tvDQPklff8\n8zBuXP3HIyKpl2hS2G1mI4ArgBeibdlVPcnMBpvZMjNbYWZjK9nvIjNzMytMMB6pA5UNjHfHHWGY\nDBFpXBJNClcCJwHj3f0DM+sGTK3sCWaWBUwGhgC9gBFm1ivOfq2B64E51Qlcaq+iAfO6dIGRI+G2\n2+AXv6jfmEQktRJKCu6+xN2vd/enzawd0NrdJ1TxtAHACndf6e5fAtOBYXH2+wlwN/B5dQKX2os3\nYF7LlvCzn8Hjj8OIEaHX8y9/mZLwRCQFEr366B9m1sbMDgEWAlPM7L4qntYZWB2zXhxtiz1uf+BI\nd3+BSpjZaDObZ2bzNmzYkEjIkoB4A+Y9/HDYnpUFv/sdXHwx/OAHcP/9qY5WROpDov0U2rr7NjO7\nGpji7neY2aIqnhPvaveyix3NrAlwPzCqqhd394eBhwEKCwt1wWQdijdgXqmmTWHqVNi7F266KWy7\n8cb6i01E6l+iSaGpmR0OXAwkOpljMXBkzHousCZmvTXQB/iHhd5ShwEzzWyou89L8DUkybKzQ18G\n95AYzOB730t1VCKSLIk2NI8DZgHvu/tcM+sOLK/iOXOBnmbWzcyaAcOBmaUPuvtWd+/g7nnunge8\nASghpKHsbHj6abjwwlBSeOCBVEckIsmSUEnB3f8A/CFmfSVwYRXPKTGzawnJJAt4zN0Xm9k4YJ67\nz6zs+ZJeShPD8OGhpGAG11+f6qhEpK4l2tCca2bPmdl6M1tnZs+aWW5Vz3P3v7j7f7h7D3cfH227\nPV5CcPdBKiWkl2nTIC8PmjQJtzNmwPTpYS6GG26ABx9MdYQiUtcSbVOYQhjW4pvR+sho29eTEZSk\nXukQGKU9nj/8MKxDSAyXXBJKCmZw7bWpi1NE6laibQod3X2Ku5dEy+NAxyTGJSkWbwiMnTvD9mbN\n4Pe/D4PnXXcdTJyoQfREMkWiSWGjmY00s6xoGQlsSmZgkloVDYFRur1Zs1CddN55ofH5m9+ETfqP\nEGnwEk0K/0W4HPUTYC1wEWHoC8lQlQ2BUapZM3jmmTAUxsyZkJ8Pf/tb/cQnIsmR6DAXH7n7UHfv\n6O6d3P08QAMsZ7CKhsAoP0heVhb893/DG29Amzbw9a+HHtBffFF/sYpI3anNdJw31VkUknYqGwIj\nnoICmD8fvvOdMFbSCSfA4sX1G7OI1F5tkoImbcxwRUWwalUY5mLVqooTQqmWLWHy5DBRz5o1UFgY\nLltVI7RIw1GbpKCPusT1jW/AO+/AGWeEy1bPOivMAy0i6a/SpGBm281sW5xlO3BEPcUoaap857Zp\n0/Y9duih8MIL8NBDMGcO9OkDEybA7t2pilZEElFpUnD31u7eJs7S2t0T7fgmGai0c9uHH4bqodLO\nbbGJwQzGjIGlS+Hss+FHP4KvfCU0SotIeqpN9ZE0YpV1biuvc2d49tkw9/PmzXDyyaEX9LZt9ROr\niCROSUFqpKrObfEMGwZLloRe0P/zP3DssfDcc8mJT0RqRklBaiSRzm3xtG4dht5+4w3o2BEuuCA0\nSP/jH7pKSSQdKClIjSTaua0iAwbA3LkhQSxdCqefDgMHhh7RSg4iqaOkIDVS3c5t8WRnh0tWV64M\n/Rk++CD0iD75ZPjrX5UcRFLBvIF98goLC33ePE27kIm++AIefxx+9rPQNlFYCD/+cej30EQ/X0Rq\nxczmu3thVfvpoyZJU1k/hniaN4dvfxuWL4dHHw2jrg4bBj17hkH31q+vj6hFGjclBUmKRPoxVKRZ\nM7jqKli2DJ56Co48EsaOhdzcMLnPyy+rakkkWVR9JEmRlxcSQXldu4ZxlKpr6dLQZvHEE6GvQ8+e\nIcmMGgUdOtQyWJFGQNVHklI16cdQmWOPhfvvh48/ht/9Djp1gh/+EI44As45J2zbsqXm8YpIoKQg\nSVHTfgxVOegguOwyeO21MOje9dfDu+/CFVeE8ZaGDoWpU9VbWqSmlBQkKWrbjyERffrAvfeG6qg3\n3ghDZ7z9dkganTqFqUKffFKIWTtnAAAQIElEQVQN1CLVoTYFSZpp08JYSB99FEoI48dXrx9DTezd\nG0ZlnTED/vCHUN1kFgbiGzIEBg8OEwBlZSU3DpF0k2ibgpKCpEyyk8bevfDWW/Dii6Ez3BtvhG3t\n2oVOckOGwJlnhquaRDKdkoKktdJLVmNHWm3Zsvq9oqtj82Z46aWQIF58ET75JGzv2hVOPRVOOSXc\n9u6tznKSfvbsCUuzZjV7flokBTMbDDwAZAGPuvuEco+PAb4L7AF2AKPdfUllx1RSyAx1fclqdbnD\nwoXwz3+GRuvXXtuXJHJy4KSTQoI4/njo2ze0UYgk286dYbiX998Py8qV+24/+AB+/Wu48sqaHTvl\nScHMsoB/A18HioG5wIjYL30za+Pu26L7Q4HvuPvgyo6rpJAZmjSJ3wHNLFTx1Df38KErTRCvvx6G\n+S51+OHQr19Y+vYNt0cdpbYJSdyuXWHu8uLi+Mvq1bBu3f7PadMGevSA7t3D7UUXhR8qNZFoUkjm\n7GkDgBXuvjIKaDowDCj7qJUmhMjBaN7nRqNLl/glhdpeslpTZuGD1707XH552Pbpp7BgQVgWLgy3\nL70EJSXh8ZYtQ/+JXr3C0rt3uM3LU7JoTNzDFW6rV4cv948/Dl/+a9bsf3/z5gOf27ZtaNPKzYX8\n/PC/06PHvqV9+/C/WZ+SmRQ6A6tj1ouBE8rvZGbfBW4CmgFnxDuQmY0GRgN0SdW3htSp8ePjtynU\n5SWrtXXIIWGuhzNi/iu/+CL0ri5NFEuWwOzZ4dLXUi1awDHHhKVr15DounQJw3V06RKqp+r7gy41\n4x76vKxevf/y0Uf7bouLw/9FrKysULo84ojQ+37QoHD/iCPC/0FubpiRsFWrlLytSiWz+uibwH+6\n+9XR+mXAAHe/roL9L432v6Ky46r6KHNUdfVRKi5pramtW0OyWLJk37JsWfji2L17/30PPji8n86d\nQ4e70qVTpwPvZ2en5v1kspKS0Pt98+ZQGoxdNm3aV5VTuuzYsf/zs7LCl3tpoi9N9rFf9h07pl9p\nMR3aFE4C7nT3/4zWfwTg7j+vYP8mwGZ3b1vZcZUUGodUXJ2UDHv3hnri0l+Vsb8wP/44PLZuXahv\nLs8MDjts35dN+duOHUOpIyen5lekNFTu4Zxt3Rq+4GNvS7/cN22CjRv33S9d37q18mMfeui+L/t4\ny+GHQ9Nk1rEkSTokhaaEhuavAR8TGpovdffFMfv0dPfl0f1zgTuqClpJoXFI9dVJ9W3HjpAc1q/f\nlyjWrt1XT13Rr9ZSLVuG/hc5OeG2bduwrXnzUJ0Ve1u6ZGdXvDRrFvYvXUqfX7pkZ4eLBczCbfn7\nu3eHKpXS5fPPD1wvXXbt2n99507Yvn3fsm3b/ve3bQtf7OVLYOW1bh3q5OMthxwSf8nJaZhf+IlI\neUOzu5eY2bXALMIlqY+5+2IzGwfMc/eZwLVmdiawG9gMVFp1JI1HXQ+ol+5atQpLjx6V77d1676r\nVTZu3FcNUnpben/NmvBlW/4L+fPPw7Xu6a5ly3DlTevW+26PPDLcb9MmJL2cnPi37dqFL/7GVnqq\nK+q8JmmpsZUU6lNJSUgQJSXh1/bu3fvf370bvvzywF/0sb/sd+8OVTh79+67jb3ftOn+pZLypZQW\nLcLghrGlj9JtzZunX318Jkh5SUGkNqq6OqkhNUKnm6ZNM7eKRGpPnfklLRUVhUblrl1DXXXXrvsa\nmWszq5uIVE7VR9LgqGpJpPo085pkrMbWCC1Sn5QUpMFJ1qxuIqKkIA1QIrO6TZsWqpmaNAm3am8Q\nSYySgjQ4lTVCgxqiRWpDDc2ScdQQLXIgNTRLo6WGaJGaU1KQjJNIQ7TaHETiU1KQjFNVQ7TaHEQq\npqQgGaeqhuhbb91/+AwI67feWv+xiqQbNTRLo5Nu80OL1Ac1NItUQG0OIhVTUpBGR20OIhVTUpBG\nR20OIhVTm4JIOWpzkEykNgWRGlKbgzRmSgoi5ajNQRozJQWRctTmII2ZkoJIHEVFYfC8vXvDbez8\nz1WNraSqJWnIlBREqqmyNgdVLUlDp6QgUk2VtTmoakkaOiUFkWqqrM0hkWG7Vb0k6SypScHMBpvZ\nMjNbYWZj4zx+k5ktMbNFZvZ3M+uazHhE6kpFbQ5VXc6q6iVJd0lLCmaWBUwGhgC9gBFm1qvcbm8D\nhe6eDzwD3J2seETqQ1WXs6p6SdJdMksKA4AV7r7S3b8EpgPDYndw99nuXvoReQPITWI8IklX1eWs\nql6SdNc0icfuDKyOWS8GTqhk/6uAv8Z7wMxGA6MBulRUPhdJE0VF+1/CGqtLl/jzR5evXiotTZRW\nL5UeVyTZkllSsDjb4g60ZGYjgULgnniPu/vD7l7o7oUdO3aswxBF6peqlyTdJTMpFANHxqznAmvK\n72RmZwK3AkPd/YskxiOScqpeknSXzOqjuUBPM+sGfAwMBy6N3cHM+gO/AQa7+/okxiKSNlS9JOks\naSUFdy8BrgVmAUuBGe6+2MzGmdnQaLd7gFbAH8xsgZnNTFY8Ig1BXVQvqSQhtaH5FETSzLRp4Uv+\no49CCWH8+H2lgKrmeihfkoCQVGKrqKRxSnQ+BSUFkQYkLy9+9VLXrqETXVWPS+OlSXZEMlBV1Usa\nwVVqS0lBpAGp6uoljeAqtaWkINLAVDbXQ21HcFVJQpQURDJIbUZwVUlCQElBJOPUdARXlSQElBRE\nGo26aKRWSSLzKSmINBK1aaQGlSQaCyUFkUakpo3UoJJEY6GkICKAShISKCmISBmVJERJQUQSkuyS\nhEoR6UFJQUQSlqySRCKlCCWN+qGkICJ1ojYliURKEap6qh9KCiJSZ2pakqiqPUKN2PVHSUFE6kVl\nJYmq2iPUiF1/lBREpN5UVJKoqj1Cl8PWHyUFEUm5qtoj6uNyWCWNQDOviUiDUNk0pbWdka4xTGOq\nmddEJKMks2Od+lDso6QgIg1ebTvW1bYPRSZRUhCRjFCbkkRt+lBAZpUklBREJOPVpiG7sTViq6FZ\nRISKG7IzpRE7LRqazWywmS0zsxVmNjbO46eZ2VtmVmJmFyUzFhGRytS0D0Wm9cZOWlIwsyxgMjAE\n6AWMMLNe5Xb7CBgFPJWsOEREaiOZjdiQftVPySwpDABWuPtKd/8SmA4Mi93B3Ve5+yJgbxLjEBGp\nlWQ1YkP6DQaYzKTQGVgds14cbRMRyRjJ7o2dSPVTXUpmUrA422rUqm1mo81snpnN27BhQy3DEhGp\nW5WVJJJd/VTXkpkUioEjY9ZzgTU1OZC7P+zuhe5e2LFjxzoJTkSkviSz+qmuJTMpzAV6mlk3M2sG\nDAdmJvH1REQanNpWP9W1pCUFdy8BrgVmAUuBGe6+2MzGmdlQADM73syKgW8CvzGzxcmKR0QkXdWm\n+qmuqfOaiEgjkBad10REpGFRUhARkTJKCiIiUkZJQUREyigpiIhImQZ39ZGZbQDiDFQLQAdgYz2G\nU13pHJ9iqxnFVjOKrWZqE1tXd6+y92+DSwqVMbN5iVxylSrpHJ9iqxnFVjOKrWbqIzZVH4mISBkl\nBRERKZNpSeHhVAdQhXSOT7HVjGKrGcVWM0mPLaPaFEREpHYyraQgIiK1oKQgIiJlMiYpmNlgM1tm\nZivMbGyq44llZqvM7B0zW2BmKR3i1cweM7P1ZvZuzLZDzOwlM1se3bZLo9juNLOPo3O3wMzOTlFs\nR5rZbDNbamaLzeyGaHvKz10lsaX83JlZCzN708wWRrHdFW3vZmZzovP2+2jOlXSJ7XEz+yDmvPWr\n79hiYswys7fN7IVoPfnnzd0b/AJkAe8D3YFmwEKgV6rjiolvFdAh1XFEsZwGFADvxmy7Gxgb3R8L\n/CKNYrsT+EEanLfDgYLofmvg30CvdDh3lcSW8nNHmJa3VXQ/G5gDnAjMAIZH238NXJNGsT0OXJTq\n/7korpuAp4AXovWkn7dMKSkMAFa4+0p3/xKYDgxLcUxpyd1fAT4tt3kY8ER0/wngvHoNKlJBbGnB\n3de6+1vR/e2EiaM6kwbnrpLYUs6DHdFqdrQ4cAbwTLQ9VeetotjSgpnlAucAj0brRj2ct0xJCp2B\n1THrxaTJhyLiwP+a2XwzG53qYOI41N3XQviCATqlOJ7yrjWzRVH1UkqqtmKZWR7Qn/DLMq3OXbnY\nIA3OXVQFsgBYD7xEKNVv8TA7I6Tw81o+NncvPW/jo/N2v5k1T0VswETgv4G90Xp76uG8ZUpSsDjb\n0ibjA6e4ewEwBPiumZ2W6oAakIeAHkA/YC3wy1QGY2atgGeB77n7tlTGUl6c2NLi3Ln7HnfvB+QS\nSvXHxtutfqOKXrRcbGbWB/gRcAxwPHAIcHN9x2Vm3wDWu/v82M1xdq3z85YpSaEYODJmPRdYk6JY\nDuDua6Lb9cBzhA9GOllnZocDRLfrUxxPGXdfF31w9wKPkMJzZ2bZhC/dae7+/6LNaXHu4sWWTucu\nimcL8A9CvX2OmTWNHkr55zUmtsFRdZy7+xfAFFJz3k4BhprZKkJ1+BmEkkPSz1umJIW5QM+oZb4Z\nMByYmeKYADCzg82sdel94Czg3cqfVe9mAldE968A/pjCWPZT+oUbOZ8UnbuoPve3wFJ3vy/moZSf\nu4piS4dzZ2YdzSwnun8QcCahzWM2cFG0W6rOW7zY3otJ8kaos6/38+buP3L3XHfPI3yfvezuRdTH\neUt163pdLcDZhKsu3gduTXU8MXF1J1wNtRBYnOrYgKcJVQm7CSWsqwh1lX8Hlke3h6RRbE8C7wCL\nCF/Ah6cotlMJRfVFwIJoOTsdzl0lsaX83AH5wNtRDO8Ct0fbuwNvAiuAPwDN0yi2l6Pz9i4wlegK\npVQtwCD2XX2U9POmYS5ERKRMplQfiYhIHVBSEBGRMkoKIiJSRklBRETKKCmIiEgZJQWRiJntiRkZ\nc4HV4Wi7ZpYXO/qrSLpqWvUuIo3GLg9DHog0WiopiFTBwnwYv4jG3n/TzI6Ktnc1s79HA6f93cy6\nRNsPNbPnonH6F5rZydGhsszskWjs/v+NetFiZteb2ZLoONNT9DZFACUFkVgHlas+uiTmsW3uPgD4\nFWEMGqL7v3P3fGAaMCnaPgn4p7v3JcwPsTja3hOY7O69gS3AhdH2sUD/6DhjkvXmRBKhHs0iETPb\n4e6t4mxfBZzh7iujgec+cff2ZraRMHTE7mj7WnfvYGYbgFwPA6qVHiOPMDRzz2j9ZiDb3X9qZi8C\nO4Dnged93xj/IvVOJQWRxHgF9yvaJ54vYu7vYV+b3jnAZOArwPyYUTBF6p2SgkhiLom5/Vd0//8I\nI1gCFAGvRff/DlwDZZO4tKnooGbWBDjS3WcTJlTJAQ4orYjUF/0iEdnnoGgWrlIvunvpZanNzWwO\n4YfUiGjb9cBjZvZDYANwZbT9BuBhM7uKUCK4hjD6azxZwFQza0uYROV+D2P7i6SE2hREqhC1KRS6\n+8ZUxyKSbKo+EhGRMiopiIhIGZUURESkjJKCiIiUUVIQEZEySgoiIlJGSUFERMr8f/kQXiVIowhR\nAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "acc = history.history['acc']\n",
    "val_acc = history.history['val_acc']\n",
    "loss = history.history['loss']\n",
    "val_loss = history.history['val_loss']\n",
    "\n",
    "epochs = range(1, len(acc) + 1)\n",
    "\n",
    "# \"bo\" is for \"blue dot\"\n",
    "plt.plot(epochs, loss, 'bo', label='Training loss')\n",
    "# b is for \"solid blue line\"\n",
    "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
    "plt.title('Training and validation loss')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3XmclXX5//HXm30VEdCUXcWFHZww\nf2ISbmjumkLYTzQjLcxv5rc0Lf35TS3Lr1pZimWLUriUSqmRImZZJoOCigYiM8oAyrCvCsNcvz8+\n95m553A2hjlzzsxcz8fjfpx7P9e5B851Psv9uWVmOOecc5m0KnQAzjnnip8nC+ecc1l5snDOOZeV\nJwvnnHNZebJwzjmXlScL55xzWXmycDmT1FrSFkn9GnLfQpJ0qKQG7z8u6URJ5bHlxZKOy2XferzX\nLyR9u77HO5eLNoUOwOWPpC2xxU7Ax8CuaPnLZjZjT85nZruALg29b0tgZoc3xHkkXQZcZGbjYue+\nrCHO7VwmniyaMTOr+bKOfrleZmbPpdtfUhszq2qM2JzLxv89FhevhmrBJH1P0sOSfi9pM3CRpGMk\nvSxpg6RVkn4sqW20fxtJJmlAtPxQtP0ZSZsl/UvSwD3dN9p+qqQlkjZK+omklyRNSRN3LjF+WdJS\nSesl/Th2bGtJd0paK+ldYEKG63ODpJlJ6+6R9L/R/GWS3o4+z7vRr/5056qQNC6a7yTpwSi2RcBR\nKd53WXTeRZLOjNYPA34KHBdV8a2JXdubYsdfHn32tZKekHRgLtdmT65zIh5Jz0laJ+kDSd+Mvc93\nomuySVKppINSVflJ+kfi7xxdzxej91kH3CBpkKS50WdZE123brHj+0efsTLafrekDlHMR8b2O1DS\nNkk90n1el4WZ+dQCJqAcODFp3feAHcAZhB8OHYFPAkcTSp0HA0uAadH+bQADBkTLDwFrgBKgLfAw\n8FA99t0f2AycFW27GtgJTEnzWXKJ8UmgGzAAWJf47MA0YBHQB+gBvBj+G6R8n4OBLUDn2LlXAyXR\n8hnRPgLGA9uB4dG2E4Hy2LkqgHHR/I+AF4DuQH/graR9LwAOjP4mn49iOCDadhnwQlKcDwE3RfMn\nRzGOBDoAPwOez+Xa7OF17gZ8CFwFtAf2AcZE264DFgKDos8wEtgPODT5WgP/SPydo89WBVwBtCb8\nezwMOAFoF/07eQn4UezzvBldz87R/sdG26YDt8Te5xvA44X+f9iUp4IH4FMj/aHTJ4vnsxx3DfBo\nNJ8qAdwb2/dM4M167Hsp8PfYNgGrSJMscozxU7HtfwSuieZfJFTHJbadlvwFlnTul4HPR/OnAksy\n7Ptn4KvRfKZk8X78bwF8Jb5vivO+CXw2ms+WLH4D3Brbtg+hnapPtmuzh9f5C0Bpmv3eTcSbtD6X\nZLEsSwznA/Oi+eOAD4DWKfY7FigDFC0vAM5t6P9XLWnyaii3PL4g6QhJT0XVCpuAm4GeGY7/IDa/\njcyN2un2PSgeh4X/3RXpTpJjjDm9F/BehngBfgdMiuY/D9R0CpB0uqR/R9UwGwi/6jNdq4QDM8Ug\naYqkhVFVygbgiBzPC+Hz1ZzPzDYB64HesX1y+ptluc59gaVpYuhLSBj1kfzv8ROSHpG0Iorh10kx\nlFvoTFGHmb1EKKWMlTQU6Ac8Vc+YHN5m4cIvzbj7CL9kDzWzfYDvEn7p59Mqwi9fACSJul9uyfYm\nxlWEL5mEbF17HwZOlNSHUE32uyjGjsBjwG2EKqJ9gb/mGMcH6WKQdDDwc0JVTI/ovP+JnTdbN9+V\nhKqtxPm6Eqq7VuQQV7JM13k5cEia49Jt2xrF1Cm27hNJ+yR/vh8QevENi2KYkhRDf0mt08TxW+Ai\nQinoETP7OM1+LgeeLFyyrsBGYGvUQPjlRnjPPwOjJZ0hqQ2hHrxXnmJ8BPgvSb2jxs5vZdrZzD4k\nVJX8ClhsZu9Em9oT6tErgV2STifUrecaw7cl7atwH8q02LYuhC/MSkLevIxQskj4EOgTb2hO8nvg\ni5KGS2pPSGZ/N7O0JbUMMl3nWUA/SdMktZO0j6Qx0bZfAN+TdIiCkZL2IyTJDwgdKVpLmkossWWI\nYSuwUVJfQlVYwr+AtcCtCp0GOko6Nrb9QUK11ecJicPtBU8WLtk3gIsJDc73EX5Z51X0hXwh8L+E\n//yHAK8RflE2dIw/B+YAbwDzCKWDbH5HaIP4XSzmDcDXgccJjcTnE5JeLm4klHDKgWeIfZGZ2evA\nj4FXon2OAP4dO/ZZ4B3gQ0nx6qTE8X8hVBc9Hh3fD5icY1zJ0l5nM9sInAScR2hQXwIcH23+IfAE\n4TpvIjQ2d4iqF78EfJvQ2eHQpM+Wyo3AGELSmgX8IRZDFXA6cCShlPE+4e+Q2F5O+DvvMLN/7uFn\nd0kSjT/OFY2oWmElcL6Z/b3Q8bimS9JvCY3mNxU6lqbOb8pzRUHSBEK1wkeErpdVhF/XztVL1P5z\nFjCs0LE0B14N5YrFWGAZoXpiAnC2N0i6+pJ0G+Fej1vN7P1Cx9MceDWUc865rLxk4ZxzLqtm02bR\ns2dPGzBgQKHDcM65JmX+/PlrzCxTV3WgGSWLAQMGUFpaWugwnHOuSZGUbRQDwKuhnHPO5cCThXPO\nuaw8WTjnnMuq2bRZpLJz504qKir46KOPCh2Ky6BDhw706dOHtm3TDXfknCu0Zp0sKioq6Nq1KwMG\nDCAMZOqKjZmxdu1aKioqGDhwYPYDnHMF0ayroT766CN69OjhiaKISaJHjx5e+nOuHmbMgAEDoFWr\n8DpjRrYj6q9ZJwvAE0UT4H8j51LLlAxmzICpU+G998AsvE6dmr+E0eyThXPOFVK2L/z6JoPrr4dt\n2+q+17ZtYX0+eLLIo7Vr1zJy5EhGjhzJJz7xCXr37l2zvGPHjpzOcckll7B48eKM+9xzzz3MyGf5\n0zmXVn2/8Pc2GbyfZnjEdOv3WqEfAt5Q01FHHWXJ3nrrrd3WZfLQQ2b9+5tJ4fWhh/bo8IxuvPFG\n++EPf7jb+urqatu1a1fDvVETtad/K+caSrb/95m2P/SQWadOZuHrPkydOtXu079/3W2JqX//zNvM\nwvul2i5lP/eeAEoth+9YL1lEGrP+b+nSpQwdOpTLL7+c0aNHs2rVKqZOnUpJSQlDhgzh5ptvrtl3\n7NixLFiwgKqqKvbdd1+uvfZaRowYwTHHHMPq1asBuOGGG7jrrrtq9r/22msZM2YMhx9+OP/8Z3hA\n2NatWznvvPMYMWIEkyZNoqSkhAULFuwW24033sgnP/nJmvgsGpV4yZIljB8/nhEjRjB69GjKy8sB\nuPXWWxk2bBgjRozg+nyVf53Lk2z/7/P56z9byaBfmqfDJ9bfcgt06lR3W6dOYX1e5JJRmsK0tyWL\nhsrS6cRLFu+8845JsldeeaVm+9q1a83MbOfOnTZ27FhbtGiRmZkde+yx9tprr9nOnTsNsKefftrM\nzL7+9a/bbbfdZmZm119/vd155501+3/zm980M7Mnn3zSTjnlFDMzu+222+wrX/mKmZktWLDAWrVq\nZa+99tpucSbiqK6utokTJ9a83+jRo23WrFlmZrZ9+3bbunWrzZo1y8aOHWvbtm2rc2x9eMnC5Uum\nkkG2//f5/PWf7dzZSi3ZPluu8JLFnmns+r9DDjmET37ykzXLv//97xk9ejSjR4/m7bff5q233trt\nmI4dO3LqqacCcNRRR9X8uk927rnn7rbPP/7xDyZOnAjAiBEjGDJkSMpj58yZw5gxYxgxYgR/+9vf\nWLRoEevXr2fNmjWcccYZQLiJrlOnTjz33HNceumldOzYEYD99ttvzy+Ecw0gXbtBtpJBtv/3+fz1\nn61kMHkyTJ8O/fuDFF6nTw/rEyZPhvJyqK4Or5Pr+7T1HHiyiGT7oze0zp0718y/88473H333Tz/\n/PO8/vrrTJgwIeV9B+3atauZb926NVVVVSnP3b59+932Mcv+kKtt27Yxbdo0Hn/8cV5//XUuvfTS\nmjhSdW81M+/26hpFfRuRs1UTZft/v7dVQZm+8IstGWTjySLS6PV/MZs2baJr167ss88+rFq1itmz\nZzf4e4wdO5ZHHnkEgDfeeCNlyWX79u20atWKnj17snnzZv7whz8A0L17d3r27Mmf/vQnINzsuG3b\nNk4++WR++ctfsn37dgDWrVvX4HG7liFfXUizlQyy/b/P96//YkoG2XiyiOTyR8+X0aNHM3jwYIYO\nHcqXvvQljj322AZ/jyuvvJIVK1YwfPhw7rjjDoYOHUq3bt3q7NOjRw8uvvhihg4dyjnnnMPRRx9d\ns23GjBnccccdDB8+nLFjx1JZWcnpp5/OhAkTKCkpYeTIkdx5550NHrdrHvJ5P0GmhJCtZJDt/31T\n+/WfV7k0bNR3AiYAi4GlwLUptvcH5gCvAy8AfWLbdgELomlWtvdqiK6zzdnOnTtt+/btZma2ZMkS\nGzBggO3cubPAUdXyv1XTlq/upWZ714icSyNxS0eODdz5TBStgXeBg4F2wEJgcNI+jwIXR/PjgQdj\n27bsyft5sshs/fr1Nnr0aBs+fLgNGzbMZs+eXeiQ6vC/VdNVyGSQy/vn8/6p5qAYksUxwOzY8nXA\ndUn7LEqUJgABm2LbPFm0IP63Kn7pvnQLnQwyxeayyzVZ5LPNojewPLZcEa2LWwicF82fA3SV1CNa\n7iCpVNLLks5O9QaSpkb7lFZWVjZk7M61OPVtV8j3zWXeblAkcsko9ZmAzwG/iC1/AfhJ0j4HAX8E\nXgPuJiSUbolt0evBQDlwSKb385JF0+Z/q/wr1LAVXjIobjSFaqik/bsAFWm2/Ro4P9P7ebJo2vxv\nlV/5bFfwZNC05Zos8lkNNQ8YJGmgpHbARGBWfAdJPSUlYrgOeCBa311S+8Q+wLHA7jcGOOdqZKpG\n2tsRTDNVJXk1UcuQt2RhZlXANGA28DbwiJktknSzpDOj3cYBiyUtAQ4AErfAHQmUSloIzAW+b2ZN\nLlmMGzdutxvs7rrrLr7yla9kPK5Lly4ArFy5kvPPPz/tuUtLSzOe56677mJb7BvitNNOY8OGDbmE\n7pqYvR3WoiHaFTwZNHO5FD+awlSM1VD33nuvTZkypc66o48+2l588cWMx3Xu3DnruY8//nibN29e\nxn369+9vlZWV2QMtAoX+WzUF+RwQz6uSWi4K3WbR2FMxJos1a9ZYz5497aOPPjIzs7KyMuvbt69V\nV1fb5s2bbfz48TZq1CgbOnSoPfHEEzXHJZJFWVmZDRkyxMzMtm3bZhdeeKENGzbMLrjgAhszZkxN\nsrj88svtqKOOssGDB9t3v/tdMzO7++67rW3btjZ06FAbN26cmdVNHnfccYcNGTLEhgwZUjNibVlZ\nmR1xxBF22WWX2eDBg+2kk06qGVE2btasWTZmzBgbOXKknXDCCfbBBx+YmdnmzZttypQpNnToUBs2\nbJg99thjZmb2zDPP2KhRo2z48OE2fvz4lNeq0H+rYpHuCznbl3m27qmeDFw6niys7hfQVVeZHX98\nw05XXZXxb2BmZqeddlpNIrjtttvsmmuuMbNwR/XGjRvNzKyystIOOeQQq66uNrPUyeKOO+6wSy65\nxMzMFi5caK1bt65JFomhwauqquz444+3hQsXmtnuJYvEcmlpqQ0dOtS2bNlimzdvtsGDB9urr75q\nZWVl1rp165qhyz/3uc/Zgw8+uNtnWrduXU2s999/v1199dVmZvbNb37TropdlHXr1tnq1autT58+\ntmzZsjqxJvNkkfkLfW9LDonzezJwyXJNFj42VJ5NmjSJmTNnAjBz5kwmTZoEhCT97W9/m+HDh3Pi\niSeyYsUKPvzww7TnefHFF7nooosAGD58OMOHD6/Z9sgjjzB69GhGjRrFokWLUg4SGPePf/yDc845\nh86dO9OlSxfOPfdc/v73vwMwcOBARo4cCaQfBr2iooJTTjmFYcOG8cMf/pBFixYB8Nxzz/HVr361\nZr/u3bvz8ssv8+lPf5qBAwcCPox5fRuh93ZAPPB2Bbd32hQ6gMYSPUiu0Z199tlcffXVvPrqq2zf\nvp3Ro0cDYWC+yspK5s+fT9u2bRkwYEDKYcnjUg0HXlZWxo9+9CPmzZtH9+7dmTJlStbzhB8TqSWG\nN4cwxHliRNm4K6+8kquvvpozzzyTF154gZtuuqnmvMkxplrXUiUaoRMJIdEIDeGLO9uAeO+9t/u2\n+IB4UJtY+vULicITgmsoXrLIsy5dujBu3DguvfTSmlIFwMaNG9l///1p27Ytc+fO5b1U3wQxn/70\np5kR/Qx98803ef3114EwvHnnzp3p1q0bH374Ic8880zNMV27dmXz5s0pz/XEE0+wbds2tm7dyuOP\nP85xxx2X82fauHEjvXuHm/F/85vf1Kw/+eST+elPf1qzvH79eo455hj+9re/UVZWBjT/Ycz3pvtq\nph5JXnJwhebJohFMmjSJhQsX1jypDmDy5MmUlpZSUlLCjBkzOOKIIzKe44orrmDLli0MHz6c22+/\nnTFjxgDhqXejRo1iyJAhXHrppXWGN586dSqnnnoqn/nMZ+qca/To0UyZMoUxY8Zw9NFHc9lllzFq\n1KicP89NN93E5z73OY477jh69uxZs/6GG25g/fr1DB06lBEjRjB37lx69erF9OnTOffccxkxYgQX\nXnhhzu9TjPZmqO29qUoq5BD6zgEtp4HbFbem8Lfa27ugvRHaFSO8gdu5PZfPu6C9Ksk1ZZ4snIvk\n+y5or0pyTVmzTxaWoeePKw6N/TdKV3rYmwZo8JKDa96adbLo0KEDa9eu9YRRxMyMtWvX0qFDh0Z5\nv715LkNDPHfBuaZKzeWLtKSkxJIH1tu5cycVFRVZ7ztwhdWhQwf69OlD27Zt8/5eAwakvl+hf//w\nmm5b4t7EGTP8XgbXvEiab2YlWfdrzsnCuWStWoUSRTIJHnyw7k1zEEoOXjpwzVmuyaJZV0O5lilT\nj6a9fS6Dcy2VJwvX5OzNjXH+XAbn6ievyULSBEmLJS2VdG2K7f0lzZH0uqQXJPWJbbtY0jvRdHE+\n43RNR7ZkkK1Hk5cenKufvLVZSGoNLAFOAioIj1mdZLEn3kl6FPizmf1G0njgEjP7gqT9gFKgBDBg\nPnCUma1P937eZtF8ZGpEztRAXV6euU2iujqfUTvXNBVDm8UYYKmZLTOzHcBM4KykfQYDc6L5ubHt\npwDPmtm6KEE8C0zIY6yuSOT7xjjnXP3kM1n0BpbHliuidXELgfOi+XOArpJ65HgskqZKKpVUWllZ\n2WCBu8JpjBvjnHN7Lp/JItVDDJIrCK4Bjpf0GnA8sAKoyvFYzGy6mZWYWUmvXr32Nl7XSDI1UPuN\ncc4Vp3w+/KgC6Btb7gOsjO9gZiuBcwEkdQHOM7ONkiqAcUnHvpDHWF0jyfYAoIZ4yM/kyZ4cnGto\n+WzgbkNo4D6BUGKYB3zezBbF9ukJrDOzakm3ALvM7LtRA/d8YHS066uEBu60T87xBu6mIVsDdXIy\nAb8xzrl8KngDt5lVAdOA2cDbwCNmtkjSzZLOjHYbByyWtAQ4ALglOnYd8D+EBDMPuDlTonBNR7Zq\nJq9Gcq44+XAfLi/SdX/NVrJwLhdmoSv0zp21065d0LkzdOwYfmi43ORasshnm4VroTK1S9xyS+pq\nJu+tlH9msH17+GJt0wbatg1Tui/WXbvg44/ho49qp+3bYevW9NP27ZljqK4O5921C6qqaucT00cf\n7X7OLVtq53fsqE0O6bRpA/vsUzt16xZeu3YN/9Y6dQoJJXm+VavwXps3107xZbOQjLp0Ca/Jk1Qb\nW1VV3URWXQ09e0Lv3nWnzp3r//dsbJ4sXIPL1P01UXrwkVvDl8/WrbB+PaxbV/u6cWP4sqmuDlPi\nV3R8Pt2XbeILN36++OvHH+8eR+vWdZNHVVU4R6Yv5PqSwvsl3jMxn5jat6/7ZXzggXW/kNu3r42z\nbdu6cbduHa7npk3hGm7aVDu/ahUsXhyS2bZt4TXTYNRt24bk0qVLeO3aNaz/8MPdk1mmyplWrcK5\nWrVKnUi7dQtJ44ADwmdJJ/F3T/VvoboahgyBX/wit79BfXmycPWS6S7rXNolmlNy2LWr7pfQtm3h\ni3n16vDlsnr17vNr14Z9GvILOfGF26EDdO8epv32gyOPDK+Jde3b1/3Vm/wruG3bcI5UU/v2db+8\nk39ld+jQdKqAqqtDwkj83Xbtqk0Q7dvndg6zcI4tW8JyciJrFWsV3rIFVqxIPa1enTl5QThXYmrd\nunZeapwSiicLt8f2tvtrsdm+HZYuhSVLwq/PJUtg+fK6VR7JX6o7dtR+yezYkf09uneH/fcP0+DB\n0KNH7Rd48uu++9b+GpXqfikkXpN/lSfWu9y1alVbFVVfUqjG6tgx+75dusDhh4epKfJk4fZYpmqm\nyZOLt12iuhr+8x945RWYPz/ML1mye0nooINCg3uHDiHuVFUe7drtXucdf40nh169wv7ONWWeLNwe\ny6WaCQrfLrFiRUgMiWnevNBQCeFX3pFHwnHHwWGHhV97hx0Ghx5aWz/tnKvlycLtsVyqmfLdLlFV\nFZLB8uVhqqio+/ree5AYLqxtWxgxAr7wBRgzJkyHH163Ptk5l5knC5dSpgbsxq5m2rgRFi4M04IF\nYXrzzd3bCvbZB/r2hT59YORIGD48JIYRI0KVknOu/jxZuN1ka8DOZzXTrl0hGcyZAy+/HObLymq3\n9+oVEsFVV4Vqo759axPEPvvs/fs751LzO7jdbhrzLmuz0ANpzpwwvfBC6FIKMGgQjBoVkkNi+sQn\nvNePcw3J7+B29ZatAXtvbd8OTz8NTz4ZEsTKaCzi/v3hnHPghBNg/PiQGJxzxcGThdtNPu6T2LED\n/vpXmDkzJIktW8LwB4nEcMIJcPDBXmpwrlh5snC7aagG7KoqmDs3JIg//hE2bAj3H0ycGKbjj888\nxIFzrnh458EWKtPT6vZ2mPBVq+A73wlj3px8Mjz6KJxxBjz1FHzwAdx/fyhJeKJwrunw/64tULbe\nTonXPe3d9NprcOedoSRRVRUSxJQpcOqp3nXVuaYuryULSRMkLZa0VNK1Kbb3kzRX0muSXpd0WrR+\ngKTtkhZE0735jLOlyTRcx57atSu0QYwbB6NHh+qmyy8Pw2g8+WRosPZE4VzTl7eShaTWwD3ASYTn\ncc+TNMvM3ortdgPhCXo/lzQYeBoYEG1718xG5iu+lqwhejt9+GEoQfzkJ/Duu6Hx+4c/hMsuCwPh\nOeeal3xWQ40BlprZMgBJM4GzgHiyMCBxK1U3YGUe43GR+vZ2Wrs2lBwefjg0XFdXw6c+BbfeCuee\n620QzjVn+ayG6g0sjy1XROvibgIuklRBKFVcGds2MKqe+puk4/IYZ4tzyy27D8ucrrfTxo3w29/C\nZz8b7nuYOjWUQL79bXjjDfjXv+CCCzxRONfc5fO/eKoe88m3i08Cfm1md0g6BnhQ0lBgFdDPzNZK\nOgp4QtIQM9tU5w2kqcBUgH7F+rCEIpTLcB2vvgq33w5PPBGertavH1x9NVx4Ybir2u+HcK5lyWfJ\nogLoG1vuw+7VTF8EHgEws38BHYCeZvaxma2N1s8H3gUOS34DM5tuZiVmVtKrV688fISmK1PXWAiJ\nobw8VCWVl4dlszDcximnwFFHwTPPwJe/DP/8Z9jnBz8IjdieKJxrefJZspgHDJI0EFgBTAQ+n7TP\n+8AJwK8lHUlIFpWSegHrzGyXpIOBQcCyPMbarOTSNTauuhr+/Ge47bYweN/++4f5K64Izwh2zrm8\nlSzMrAqYBswG3ib0elok6WZJZ0a7fQP4kqSFwO+BKRZGNvw08Hq0/jHgcjNbl69Ym5tcu8ZWVcFD\nD4WhvM86K9ww97OfhVLEtdd6onDO1fJRZ5uhVq1ClVIyKZQiAF56KVQxLVoEQ4eG5HDhhd5Q7VxL\nk+uosz7cRzOUrq2/X78wPtPll8PYseERo3/4Q3io0OTJniicc+l5smiGUnWN7dgxDL9x5JFhbKar\nrw6linPP9ceLOuey89+SzVBy19iDDgrDgf/0p6GX01NPhV5NzjmXK/9N2UTl0jV26dJwr8T69WH+\nzjtDbydPFM65PeUliyYol66xO3aEO6uffDJUP/30p3v38CLnXMvmJYsmKFvX2I8/hvPPD4ni7rvD\nqycK59ze8JJFE5Rp1NiPPgqJ4qmnwj0TV1zRuLE555onL1k0QelKCX37hudHPPUU3HefJwrnXMPx\nZNEEpesa260bzJ4dusYm2jCcc64heDVUE5TcNbZPn5Ao3nwTfvlLuOSSwsbnnGt+vGTRRCVGjd28\nGQYNCjfY/epXniicc/nhJYsmbMuW0C32xRfDA4ouuqjQETnnmisvWRSxTDferVsHJ50Ef/87PPig\nJwrnXH55yaJIZbrx7oQT4OSTYfFiePTR0APKOefyKWuykDQNmGFm6xshHhdJd+Pdt74Vej6tXBm6\nyJ54YmHic861LLlUQ30CmCfpEUkTJH+oZmNId+PdihWwZg0895wnCudc48maLMzsBsJjTX8JTAHe\nkXSrpEOyHRsll8WSlkq6NsX2fpLmSnpN0uuSTottuy46brGkU/boUzUD6W68a9UqPCf7mGMaNRzn\nXAuXUwN39KjTD6KpCugOPCbp9nTHSGoN3AOcCgwGJkkanLTbDYTHrY4iPKP7Z9Gxg6PlIcAE4GfR\n+VqMVDfeSfCDH8CIEYWJyTnXcmVNFpK+Jmk+cDvwEjDMzK4AjgLOy3DoGGCpmS0zsx3ATOCspH0M\n2Cea7wasjObPAmaa2cdmVgYsjc7XYkyeDNOnw/77h+U2bcIQ49dcU9i4nHMtUy69oXoC55rZe/GV\nZlYt6fQMx/UGlseWK4Cjk/a5CfirpCuBzkCiFr438HLSsb2T30DSVGAqQL9mOKzqQQeFx6COHBmG\n8UgkDueca2y5VEM9DaxLLEjqKuloADN7O8NxqRrCLWl5EvBrM+sDnAY8KKlVjsdiZtPNrMTMSnr1\n6pXlYzQtb7wBZ58Nhx4Kzz/vicI5V1i5JIufA1tiy1ujddlUAH1jy32orWZK+CLwCICZ/QvoQCjJ\n5HJss7V8OZx6KnTpAn/5C3TQx0dzAAAV6ElEQVTvXuiInHMtXS7JQlEDNxCqn8it+moeMEjSQEnt\nCA3Ws5L2eR84AUDSkYRkURntN1FSe0kDCb2xXsnhPZuUVHdob9gQEsXmzfDMM2HYceecK7RcvvSX\nSfoataWJrwDLsh1kZlXRDX2zgdbAA2a2SNLNQKmZzQK+Adwv6euEaqYpUWJaJOkR4C1C76uvmtmu\nPf1wxSzVHdpf+hL07w/vvhtKFMOHFzZG55xLUKzQkHoHaX/gx8B4whf6HOC/zGx1/sPLXUlJiZWW\nlhY6jJwNGBASRCq/+x1MmtSo4TjnWihJ882sJNt+WUsWUVKY2CBRuRrp7tAGTxTOueKTy9hQHQgN\n0UMIbQoAmNmleYyr2evXL3XJohn2AHbONQO5NHA/SBgf6hTgb4SeSZvzGVRLkO7RqLfeWph4nHMu\nk1ySxaFm9h1gq5n9BvgsMCy/YTV/kyeHO7ITwzL26xeenZ14ZKpzzhWTXHpD7YxeN0gaShgfakDe\nImpBVq8GM3jttXCXtnPOFatcksV0Sd0Jg/7NAroA38lrVC3Ali1w111w+umeKJxzxS9jsoiG3tgU\nPfjoReDgRomqBbjvPli7NjzkyDnnil3GNovobu1pjRRLi/HRR/CjH8H48fCpTxU6Guecyy6XBu5n\nJV0jqa+k/RJT3iNrBlIN5wHwq1/BBx/ADTcUMjrnnMtdLm0WifspvhpbZ3iVVEaphvOYOhWqqsID\njI45BsaNK2iIzjmXs1zu4B7YGIE0N9dfX5soErZtg298I7RV3HNPbbdZ55wrdrncwf1/U603s982\nfDjNR7rhPNauDb2fTjst9XbnnCtGuVRDfTI234EwpPirgCeLDNIN5wGh1OGlCudcU5JLNdSV8WVJ\n3QhDgLgMbrmlbpsFhARx4IFw7rmFi8s55+ojl95QybYRHkbkMpg8GaZPD8+nkKBXr3C39m23hd5R\nzjnXlOTSZvEnap9/3QoYTPQo1ByOnQDcTXj40S/M7PtJ2+8EPhMtdgL2N7N9o227gDeibe+b2Zm5\nvGcxmTw5TGbhforOnX34cedc05RLm8WPYvNVwHtmVpHtIEmtgXuAkwjP1J4naZaZvZXYx8y+Htv/\nSmBU7BTbzaxZDIQxZw688grcey+0bVvoaJxzbs/lkizeB1aZ2UcAkjpKGmBm5VmOGwMsNbNl0XEz\ngbMIj0pNZRJwY05RNzG33AIHHQRTphQ6Euecq59cas8fBapjy7uiddn0BpbHliuidbuR1B8YCDwf\nW91BUqmklyWdnea4qdE+pZWVlTmE1PheegleeAGuuQbaty90NM45Vz+5JIs2ZrYjsRDNt8vhuFSd\nQ9M98Hsi8JiZ7Yqt6xc9F/bzwF2SDtntZGbTzazEzEp69eqVQ0iNa9MmmDYNevYMPaOcc66pyiVZ\nVEqqaVyWdBawJofjKoC+seU+wMo0+04Efh9fYWYro9dlwAvUbc8oetu3w5lnwptvwm9/Gxq3nXOu\nqcolWVwOfFvS+5LeB74FfDmH4+YBgyQNlNSOkBBmJe8k6XCgO/Cv2LruktpH8z2BY0nf1lF0qqpg\n4kR48cWQKE49tdAROefc3snlprx3gU9J6gLIzHJ6/raZVUmaBswmdJ19wMwWSboZKDWzROKYBMw0\ns3gV1ZHAfZKqCQnt+/FeVMWsuhouuwxmzQrjP3lXWedcc6C639EpdpBuBW43sw3RcnfgG2ZWVANs\nl5SUWGlpaUFjMAsDBd55J9x8M3zHnyfonCtykuZH7cMZ5VINdWoiUQBET83zYfBSuPXWkCi+9jV/\nVoVzrnnJJVm0TrQfQLjPAvBOoEnuvTckiIsuCgnDBwp0zjUnudyU9xAwR9KvouVLgN/kL6SmY8aM\nMIJsYnTZUaPggQd87CfnXPOT9WvNzG4HvkdodB4M/AXon+e4il7iSXjxYcj/8x94JKdRs5xzrmnJ\n9TfwB4S7uM8jPM/i7bxF1ESkehLe9u1hvXPONTdpq6EkHUa4N2ISsBZ4mNB76jPpjmkpzNI/2Cjd\nE/Kcc64py9Rm8R/g78AZZrYUQNLXM+zfIuzcCZdfnn57v36NF4tzzjWWTNVQ5xGqn+ZKul/SCaQe\n76nF2LIFzjorNGKffTZ07Fh3e6dOYYRZ55xrbtImCzN73MwuBI4gjM30deAAST+XdHIjxVc0Vq+G\nz3wGZs+G++6Dxx+H+++vfRJe//7hyXiTJxc6Uueca3hZ7+Cus7O0H/A54EIzG5+3qOohn3dwL10K\nEybAypXw8MNwxhl5eRvnnGt0ud7Bnct9FjXMbB1wXzS1CCtWwDHHhPm5c+Hoowsbj3POFcIeJYuW\n6IUXYM0a+Oc/PVE451ouv9c4i/Ly8DqyWTwN3Dnn6seTRRZlZXDAAbv3fHLOuZbEk0UW5eUwYECh\no3DOucLKa7KQNEHSYklLJV2bYvudkhZE0xJJG2LbLpb0TjRdnM84Mykrg4EDC/XuzjlXHPLWwC2p\nNXAPcBLhedzzJM2KP/HOzL4e2/9KoudsR110bwRKAAPmR8euz1e8qezaFYbvuOCCxnxX55wrPvks\nWYwBlprZMjPbAcwEzsqw/yTg99H8KcCzZrYuShDPAhPyGGtKK1eG52l7ycI519LlM1n0BpbHliui\ndbuR1B8YCDy/J8dKmiqpVFJpZWVlgwQdV1YWXr3NwjnX0uUzWaQaRyrd7eITgcfMbNeeHGtm082s\nxMxKevXqVc8w00t0m/WShXOupctnsqgA+saW+wAr0+w7kdoqqD09Nm8SJQsfSdY519LlM1nMAwZJ\nGiipHSEhzEreSdLhQHfgX7HVs4GTJXWX1B04OVrXqMrL4aCDoL0/cdw518LlrTeUmVVJmkb4km8N\nPGBmiyTdDJSaWSJxTAJmWmxEQzNbJ+l/CAkH4OZoXKpGVV7uVVDOOQd7OOpsMcvHqLMDBsDYsfDQ\nQw16WuecKxq5jjrrd3CnUVUFFRVesnDOOfBkkdby5eGmPO8265xznizS8m6zzjlXy5NFGn5DnnPO\n1fJkkUZ5ObRqBX37Zt3VOeeaPU8WaZSXQ58+0LZtoSNxzrnC82SRRlmZV0E551yCJ4s0/IY855yr\n5ckihY8/hhUrvGThnHMJnixSWL4czLxk4ZxzCZ4sUvBus845V5cnixQSN+R5snDOucCTRQrl5dCm\nDfRO+Vw/55xreTxZpFBWFm7Ga5O3Adydc65p8WSRQqLb7IwZoSqqVavwOmNGgQNzzrkCyWuykDRB\n0mJJSyVdm2afCyS9JWmRpN/F1u+StCCadnvCXj6VlYURZ6dOhffeCz2j3nsvLHvCcM61RHmraJHU\nGrgHOInwTO15kmaZ2VuxfQYB1wHHmtl6SfvHTrHdzEbmK750tm+HDz4Ir9u21d22bRtcfz1MntzY\nUTnnXGHls2QxBlhqZsvMbAcwEzgraZ8vAfeY2XoAM1udx3hy8v774XXjxszbnXOuJclnsugNLI8t\nV0Tr4g4DDpP0kqSXJU2IbesgqTRaf3Ye46wjcY/FAQek3t6vX2NF4pxzxSOfyUIp1iU/8LsNMAgY\nB0wCfiFp32hbv+i5sJ8H7pJ0yG5vIE2NEkppZWVlgwSduMfi+uuhU6e62zp1gltuaZC3cc65JiWf\nyaICiD8Nog+wMsU+T5rZTjMrAxYTkgdmtjJ6XQa8AIxKfgMzm25mJWZW0qtXrwYJuqwM2rWDr34V\npk+H/v1BCq/Tp3t7hXOuZcpnspgHDJI0UFI7YCKQ3KvpCeAzAJJ6EqqllknqLql9bP2xwFs0gvLy\nkBhatQqJobwcqqvDqycK51xLlbfeUGZWJWkaMBtoDTxgZosk3QyUmtmsaNvJkt4CdgH/bWZrJf0f\n4D5J1YSE9v14L6p88udYOOfc7mSW3IzQNJWUlFhpaelen2f//eHss0OVk3PONXeS5kftwxn5Hdwx\nW7dCZaUPTe6cc8k8WcT4aLPOOZeaJ4uYRLLwkoVzztXlySLGH3rknHOpebKIKS+HDh3S373tnHMt\nlSeLmES3WaW699w551owTxYx5eVeBeWcc6l4sohJPPTIOedcXZ4sIps2wbp1XrJwzrlUPFlEvNus\nc86l58ki4t1mnXMuPU8WES9ZOOdcep4sIuXl0Lkz9OhR6Eicc674eLKI+D0WzjmXnieLiHebdc65\n9DxZAGb+0CPnnMskr8lC0gRJiyUtlXRtmn0ukPSWpEWSfhdbf7Gkd6Lp4nzGuWFDuM/CSxbOOZda\n3h6rKqk1cA9wElABzJM0K/54VEmDgOuAY81svaT9o/X7ATcCJYAB86Nj1+cjVu8265xzmeWzZDEG\nWGpmy8xsBzATOCtpny8B9ySSgJmtjtafAjxrZuuibc8CE/IVqD/0yDnnMstnsugNLI8tV0Tr4g4D\nDpP0kqSXJU3Yg2ORNFVSqaTSysrKegfq91g451xm+UwWqTqhWtJyG2AQMA6YBPxC0r45HouZTTez\nEjMr6dWrV70DLSuDffaBffet9ymcc65Zy2eyqAD6xpb7ACtT7POkme00szJgMSF55HJsg0l0m/V7\nLJxzLrV8Jot5wCBJAyW1AyYCs5L2eQL4DICknoRqqWXAbOBkSd0ldQdOjtblhXebdc65zPKWLMys\nCphG+JJ/G3jEzBZJulnSmdFus4G1kt4C5gL/bWZrzWwd8D+EhDMPuDlal4c4/YY855zLJm9dZwHM\n7Gng6aR1343NG3B1NCUf+wDwQD7jA1izBrZu9ZKFc85lktdk0RR06gSPPgojRhQ6EuecK14tPll0\n7gznn1/oKJxzrrj52FDOOeey8mThnHMuK08WzjnnsvJk4ZxzLitPFs4557LyZOGccy6rFp8sZswI\nN+S1ahVeZ8wodETOOVd8WvR9FjNmwNSpsG1bWH7vvbAMMHly4eJyzrli06JLFtdfX5soErZtC+ud\nc87VatHJ4v3392y9c861VC06WfTrt2frnXOupWrRyeKWW8JAgnGdOoX1zjnnarXoZDF5MkyfDv37\nh6fk9e8flr1x2znn6sprspA0QdJiSUslXZti+xRJlZIWRNNlsW27YuuTn7DXYCZPDg8/qq4Or54o\nnHNud3nrOiupNXAPcBLhmdrzJM0ys7eSdn3YzKalOMV2MxuZr/icc87lLp8lizHAUjNbZmY7gJnA\nWXl8P+ecc3mSz2TRG1geW66I1iU7T9Lrkh6T1De2voOkUkkvSzo71RtImhrtU1pZWdmAoTvnnIvL\nZ7JQinWWtPwnYICZDQeeA34T29bPzEqAzwN3STpkt5OZTTezEjMr6dWrV0PF7ZxzLkk+k0UFEC8p\n9AFWxncws7Vm9nG0eD9wVGzbyuh1GfACMCqPsTrnnMsgn2NDzQMGSRoIrAAmEkoJNSQdaGarosUz\ngbej9d2BbWb2saSewLHA7ZnebP78+WskvZdhl57Amnp9kvzz2OrHY6sfj61+mmts/XPZKW/Jwsyq\nJE0DZgOtgQfMbJGkm4FSM5sFfE3SmUAVsA6YEh1+JHCfpGpC6ef7KXpRJb9fxnooSaVRtVbR8djq\nx2OrH4+tflp6bHkdddbMngaeTlr33dj8dcB1KY77JzAsn7E555zLXYu+g9s551xuWlKymF7oADLw\n2OrHY6sfj61+WnRsMkvuzeqcc87V1ZJKFs455+rJk4Vzzrmsmn2yyDbybaFJKpf0RjS6bmmBY3lA\n0mpJb8bW7SfpWUnvRK/diyi2myStiI1OfFoB4uoraa6ktyUtknRVtL7g1y1DbMVw3TpIekXSwii2\n/xetHyjp39F1e1hSuyKK7deSymLXrWADnUpqLek1SX+OlvN/3cys2U6E+zveBQ4G2gELgcGFjisp\nxnKgZ6HjiGL5NDAaeDO27nbg2mj+WuAHRRTbTcA1Bb5mBwKjo/muwBJgcDFctwyxFcN1E9Almm8L\n/Bv4FPAIMDFafy9wRRHF9mvg/EJet1iMVwO/A/4cLef9ujX3koWPfLsHzOxFws2RcWdRO2bXb4CU\ngzrmW5rYCs7MVpnZq9H8ZsIoBL0pguuWIbaCs2BLtNg2mgwYDzwWrS/UdUsXW1GQ1Af4LPCLaFk0\nwnVr7ski15FvC8mAv0qaL2lqoYNJ4QCLhmSJXvcvcDzJpkWjFj9QqCqyBEkDCGOY/Zsiu25JsUER\nXLeoKmUBsBp4llALsMHMqqJdCvb/NTk2M0tct1ui63anpPaFiA24C/gmUB0t96ARrltzTxa5jHxb\naMea2WjgVOCrkj5d6ICakJ8DhwAjgVXAHYUKRFIX4A/Af5nZpkLFkUqK2IriupnZLgsPOOtDqAU4\nMtVujRtV9KZJsUkaShht4gjgk8B+wLcaOy5JpwOrzWx+fHWKXRv8ujX3ZJF15NtCs9rRdVcDjxP+\n0xSTDyUdCGHgR8IvraJgZh9G/6mrCaMWF+TaSWpL+DKeYWZ/jFYXxXVLFVuxXLcEM9tAGFn6U8C+\nkhLDEBX8/2sstglRtZ5ZGCn7VxTmuh0LnCmpnFCtPp5Q0sj7dWvuyaJm5Nuod8BEIG/P895TkjpL\n6pqYB04G3sx8VKObBVwczV8MPFnAWOpIfBlHzqEA1y6qL/4l8LaZ/W9sU8GvW7rYiuS69ZK0bzTf\nETiR0KYyFzg/2q1Q1y1VbP+JJX8R2gQa/bqZ2XVm1sfMBhC+z543s8k0xnUrdKt+vifgNEIvkHeB\n6wsdT1JsBxN6aC0EFhU6PuD3hGqJnYRS2RcJ9aFzgHei1/2KKLYHgTeA1wlfzgcWIK6xhCL/68CC\naDqtGK5bhtiK4boNB16LYngT+G60/mDgFWAp8CjQvohiez66bm8CDxH1mCrUBIyjtjdU3q+bD/fh\nnHMuq+ZeDeWcc64BeLJwzjmXlScL55xzWXmycM45l5UnC+ecc1l5snAuC0m7YiONLlADjl4saUB8\nJF3nilWb7Ls41+JttzD0g3MtlpcsnKsnhWeR/CB69sErkg6N1veXNCcacG6OpH7R+gMkPR49J2Gh\npP8Tnaq1pPujZyf8NbprGElfk/RWdJ6ZBfqYzgGeLJzLRcekaqgLY9s2mdkY4KeEMXqI5n9rZsOB\nGcCPo/U/Bv5mZiMIz+ZYFK0fBNxjZkOADcB50fprgVHReS7P14dzLhd+B7dzWUjaYmZdUqwvB8ab\n2bJowL4PzKyHpDWEITR2RutXmVlPSZVAHwsD0SXOMYAwBPagaPlbQFsz+56kvwBbgCeAJ6z2GQvO\nNTovWTi3dyzNfLp9Uvk4Nr+L2rbEzwL3AEcB82OjijrX6DxZOLd3Loy9/iua/ydhRFCAycA/ovk5\nwBVQ83CdfdKdVFIroK+ZzSU86GZfYLfSjXONxX+pOJddx+ipaQl/MbNE99n2kv5N+OE1KVr3NeAB\nSf8NVAKXROuvAqZL+iKhBHEFYSTdVFoDD0nqRni4zZ0Wnq3gXEF4m4Vz9RS1WZSY2ZpCx+Jcvnk1\nlHPOuay8ZOGccy4rL1k455zLypOFc865rDxZOOecy8qThXPOuaw8WTjnnMvq/wP2iVkCl9sKAwAA\nAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.clf()   # clear figure\n",
    "acc_values = history_dict['acc']\n",
    "val_acc_values = history_dict['val_acc']\n",
    "\n",
    "plt.plot(epochs, acc, 'bo', label='Training acc')\n",
    "plt.plot(epochs, val_acc, 'b', label='Validation acc')\n",
    "plt.title('Training and validation accuracy')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These two plots show you the loss values and the accuracy for each dataset over each of the 40 epochs.  \n",
    "  \n",
    "If you notice that each graph shows that we peak on the validation set around 20 epochs. This is when overfitting starts to show up. The model preforms better on the training set but the validation set doesn't get better. At this point, the model learns the specifics of the training set instead of learning the generalized information."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " @title MIT License\n",
    "\n",
    " Copyright (c) 2017 François Chollet\n",
    "\n",
    " Permission is hereby granted, free of charge, to any person obtaining a\n",
    " copy of this software and associated documentation files (the \"Software\"),\n",
    " to deal in the Software without restriction, including without limitation\n",
    " the rights to use, copy, modify, merge, publish, distribute, sublicense,\n",
    " and/or sell copies of the Software, and to permit persons to whom the\n",
    " Software is furnished to do so, subject to the following conditions:\n",
    "\n",
    " The above copyright notice and this permission notice shall be included in\n",
    " all copies or substantial portions of the Software.\n",
    "\n",
    " THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
    " IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n",
    " FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL\n",
    " THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n",
    " LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING\n",
    " FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER\n",
    " DEALINGS IN THE SOFTWARE."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:tf_current]",
   "language": "python",
   "name": "conda-env-tf_current-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}