deep_learning_basics/deep_learning_basics.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "deep_learning_basics.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/truongkhanhduy95/MIT-Deep-Learning/blob/master/deep_learning_basics/deep_learning_basics.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "metadata": {
        "id": "lUspw0bxhCfn",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Deep Learning Basics\n",
        "\n",
        "This tutorial accompanies the [lecture on Deep Learning Basics](https://www.youtube.com/watch?list=PLrAXtmErZgOeiKm4sgNOknGvNjby9efdf&v=O5xeyoRL95U) given as part of [MIT Deep Learning](https://deeplearning.mit.edu). Acknowledgement to amazing people involved is provided throughout the tutorial and at the end. You can watch the video on YouTube:\n",
        "\n",
        "[![Deep Learning Basics](https://i.imgur.com/FfQVV8q.png)](https://www.youtube.com/watch?list=PLrAXtmErZgOeiKm4sgNOknGvNjby9efdf&v=O5xeyoRL95U)\n",
        "\n",
        "In this tutorial, we mention seven important types/concepts/approaches in deep learning, introducing the first 2 and providing pointers to tutorials on the others. Here is a visual representation of the seven:\n",
        "\n",
        "![Deep learning concepts](https://i.imgur.com/EAl47rp.png)\n",
        "\n",
        "At a high-level, neural networks are either encoders, decoders, or a combination of both. Encoders find patterns in raw data to form compact, useful representations. Decoders generate new data or high-resolution useful infomation from those representations. As the lecture describes, deep learning discovers ways to **represent** the world so that we can reason about it. The rest is clever methods that help use deal effectively with visual information, language, sound (#1-6) and even act in a world based on this information and occasional rewards (#7).\n",
        "\n",
        "1. **Feed Forward Neural Networks (FFNNs)** - classification and regression based on features. See [Part 1](#Part-1:-Boston-Housing-Price-Prediction-with-Feed-Forward-Neural-Networks) of this tutorial for an example.\n",
        "2. **Convolutional Neural Networks (CNNs)** - image classification, object detection, video action recognition, etc. See [Part 2](#Part-2:-Classification-of-MNIST-Dreams-with-Convolution-Neural-Networks) of this tutorial for an example.\n",
        "3. **Recurrent Neural Networks (RNNs)** - language modeling, speech recognition/generation, etc. See [this TF tutorial on text generation](https://www.tensorflow.org/tutorials/sequences/text_generation) for an example.\n",
        "4. **Encoder Decoder Architectures** - semantic segmentation, machine translation, etc. See [our tutorial on semantic segmentation](https://github.com/lexfridman/mit-deep-learning/blob/master/tutorial_driving_scene_segmentation/tutorial_driving_scene_segmentation.ipynb) for an example.\n",
        "5. **Autoencoder** - unsupervised embeddings, denoising, etc.\n",
        "6. **Generative Adversarial Networks (GANs)** - unsupervised generation of realistic images, etc. See [this TF tutorial on DCGANs](https://github.com/tensorflow/tensorflow/blob/r1.11/tensorflow/contrib/eager/python/examples/generative_examples/dcgan.ipynb) for an example.\n",
        "7. **Deep Reinforcement Learning** - game playing, robotics in simulation, self-play, neural arhitecture search, etc. We'll be releasing notebooks on this soon and will link them here.\n",
        "\n",
        "There are selective omissions and simplifications throughout these tutorials, hopefully without losing the essence of the underlying ideas. See Einstein quote...\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "uSSHrjknhMm_",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Part 0: Prerequisites:\n",
        "\n",
        "We recommend that you run this this notebook in the cloud on Google Colab (see link with icon at the top) if you're not already doing so. It's the simplest way to get started. You can also [install TensorFlow locally](https://www.tensorflow.org/install/). But, again, simple is best (with caveats):\n",
        "\n",
        "![Einstein](https://i.imgur.com/vfPDHGN.png)\n",
        "\n",
        "[tf.keras](https://www.tensorflow.org/guide/keras) is the simplest way to build and train neural network models in TensorFlow. So, that's what we'll stick with in this tutorial, unless the models neccessitate a lower-level API.\n",
        "\n",
        "Note that there's [tf.keras](https://www.tensorflow.org/guide/keras) (comes with TensorFlow) and there's [Keras](https://keras.io/) (standalone). You should be using [tf.keras](https://www.tensorflow.org/guide/keras) because (1) it comes with TensorFlow so you don't need to install anything extra and (2) it comes with powerful TensorFlow-specific features."
      ]
    },
    {
      "metadata": {
        "id": "KXw9oSM7gZSb",
        "colab_type": "code",
        "outputId": "1613a15f-5b41-478c-e2a1-d747fcb18d3f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "cell_type": "code",
      "source": [
        "# TensorFlow and tf.keras\n",
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "\n",
        "# Commonly used modules\n",
        "import numpy as np\n",
        "import os\n",
        "import sys\n",
        "\n",
        "# Images, plots, display, and visualization\n",
        "import matplotlib.pyplot as plt\n",
        "import pandas as pd\n",
        "import seaborn as sns\n",
        "import cv2\n",
        "import IPython\n",
        "\n",
        "# Scikit-learn \n",
        "from sklearn.preprocessing import StandardScaler, MinMaxScaler\n",
        "\n",
        "print(tf.__version__)"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "1.12.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "AOuGWs1og1v_",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Part 1: Boston Housing Price Prediction with Feed Forward Neural Networks\n",
        "\n",
        "Let's start with using a fully-connected neural network to do predict housing prices. The following image highlights the difference between regression and classification (see part 2). Given an observation as input, **regression** outputs a continuous value (e.g., exact temperature) and classificaiton outputs a class/category that the observation belongs to.\n",
        "\n",
        "<img src=\"https://i.imgur.com/vvSoAzg.jpg\" alt=\"classification_regression\" width=\"400\"/>\n",
        "\n",
        "For the Boston housing dataset, we get 506 rows of data, with 13 features in each. Our task is to build a regression model that takes these 13 features as input and output a single value prediction of the \"median value of owner-occupied homes (in $1000).\"\n",
        "\n",
        "Now, we load the dataset. Loading the dataset returns four NumPy arrays:\n",
        "\n",
        "* The `train_images` and `train_labels` arrays are the *training set*—the data the model uses to learn.\n",
        "* The model is tested against the *test set*, the `test_images`, and `test_labels` arrays."
      ]
    },
    {
      "metadata": {
        "id": "MQ9N3PEFg5KG",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "(X_train, y_train), (X_test, y_test) = keras.datasets.boston_housing.load_data()\n",
        "\n",
        "#Zscore normalization\n",
        "scaler = StandardScaler()\n",
        "X_train = scaler.fit_transform(X_train)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "wBswDHVKilp-",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Build the model\n",
        "\n",
        "Building the neural network requires configuring the layers of the model, then compiling the model. First we stack a few layers together using `keras.Sequential`. Next we configure the loss function, optimizer, and metrics to monitor. These are added during the model's compile step:\n",
        "\n",
        "* *Loss function* - measures how accurate the model is during training, we want to minimize this with the optimizer.\n",
        "* *Optimizer* - how the model is updated based on the data it sees and its loss function.\n",
        "* *Metrics* - used to monitor the training and testing steps.\n",
        "\n",
        "Let's build a network with 1 hidden layer of 20 neurons, and use mean squared error (MSE) as the loss function (most common one for regression problems):"
      ]
    },
    {
      "metadata": {
        "id": "HmWOVsrpiYYm",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def build_model():\n",
        "  model = keras.Sequential([\n",
        "      keras.layers.Dense(20, activation=tf.nn.relu, input_shape=[len(X_train[0])]),\n",
        "      keras.layers.Dense(1)\n",
        "  ])\n",
        "  \n",
        "  model.compile(optimizer = tf.train.AdamOptimizer(),\n",
        "               loss = 'mse',\n",
        "               metrics = ['mae','mse'])\n",
        "  return model"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "a7bHxEayjOxc",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Train the model\n",
        "\n",
        "Training the neural network model requires the following steps:\n",
        "\n",
        "1. Feed the training data to the model—in this example, the `train_features` and `train_labels` arrays.\n",
        "2. The model learns to associate features and labels.\n",
        "3. We ask the model to make predictions about a test set—in this example, the `test_features` array. We verify that the predictions match the labels from the `test_labels` array. \n",
        "\n",
        "To start training,  call the `model.fit` method—the model is \"fit\" to the training data:"
      ]
    },
    {
      "metadata": {
        "id": "_dC7qWF6jNPM",
        "colab_type": "code",
        "outputId": "52ce82e0-cd79-4b6d-900e-202acab180ba",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 199
        }
      },
      "cell_type": "code",
      "source": [
        "# this helps makes our output less verbose but still shows progress\n",
        "class PrintDot(keras.callbacks.Callback):\n",
        "    def on_epoch_end(self, epoch, logs):\n",
        "        if epoch % 100 == 0: print('')\n",
        "        print('.', end='')\n",
        "\n",
        "model = build_model()\n",
        "\n",
        "early_stop = keras.callbacks.EarlyStopping(monitor='val_loss', patience=50)\n",
        "history = model.fit(X_train, y_train, epochs=1000, verbose=0, validation_split = 0.1, callbacks=[early_stop, PrintDot()])\n",
        "\n",
        "hist = pd.DataFrame(history.history)\n",
        "hist['epoch'] = history.epoch\n",
        "\n",
        "# Show RMSE measure\n",
        "rmse_final = np.sqrt(float(hist['val_mean_squared_error'].tail(1)))\n",
        "print()\n",
        "print('Final Root Mean Square Error on validation set: {}'.format(round(rmse_final, 3)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\n",
            "....................................................................................................\n",
            "....................................................................................................\n",
            "....................................................................................................\n",
            "....................................................................................................\n",
            "....................................................................................................\n",
            "....................................................................................................\n",
            "....................................................................................................\n",
            ".............................\n",
            "Final Root Mean Square Error on validation set: 2.392\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "FcNKOCk-o0oE",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Now, let's plot the loss function measure on the training and validation sets. The validation set is used to prevent overfitting ([learn more about it here](https://www.tensorflow.org/tutorials/keras/overfit_and_underfit)). However, because our network is small, the training convergence without noticeably overfitting the data as the plot shows."
      ]
    },
    {
      "metadata": {
        "id": "ZWA1stHgm0jk",
        "colab_type": "code",
        "outputId": "522cce2a-1a5c-4e2c-9650-3c1c5d4a3541",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 365
        }
      },
      "cell_type": "code",
      "source": [
        "def plot_history(hist):\n",
        "  plt.figure()\n",
        "  plt.xlabel('Epoch')\n",
        "  plt.ylabel('Mean Squared Error [Thoursand Dollart$^2$]')\n",
        "  plt.plot(hist['epoch'], hist['mean_squared_error'], label='Train Error')\n",
        "  plt.plot(hist['epoch'], hist['val_mean_squared_error'], label='Val Error')\n",
        "  plt.legend()\n",
        "  plt.ylim([0,50])\n",
        "  \n",
        "plot_history(hist)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfUAAAFcCAYAAADRWyc3AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzs3Xl4VNX9x/H3nS0zkz0hCSQQEggQ\nICwxEAVEtoqAiEhdaFzBipWi0moRqj9Fi1KLotVad7AqCApItYqsooAYloR9XxMSsu8kk5nMzO+P\nSCTsM7nJZPm+nofnSWbu3PkcJ+abc+655yhOp9OJEEIIIZo8jacDCCGEEEIdUtSFEEKIZkKKuhBC\nCNFMSFEXQgghmgkp6kIIIUQzIUVdCCGEaCZ0nnrj5ORkHn/8cTp16gRA586d+f3vf8+0adOw2+2E\nhIQwZ84cDAaDpyIKIYQQTYrHijpAYmIib7zxRs33M2bMICkpiZEjRzJ37lyWLFlCUlKSBxMKIYQQ\nTUejGn5PTk5m2LBhAAwZMoTNmzd7OJEQQgjRdHi0p37kyBH+8Ic/UFxczJQpU6ioqKgZbg8ODiY3\nN9eT8YQQQogmxWNFPSoqiilTpjBy5EjS09O57777sNvtNc9f7eq1VVV2dDptfcWsM4fDwfgv/kj3\n0M4M8B7HPxfvYOr4eIb1jfR0NCGEEM2Mx4p6WFgYo0aNAiAyMpJWrVqxe/duLBYLRqOR7OxsQkND\nr3iewsJyVXOFhPiSm1uq6jk1ioaKykoqNFYA8gvLVX+Pq1Uf7WtMpH1NV3NuG0j7mrrG1L6QEN9L\nPuexa+pfffUVH374IQC5ubnk5+czbtw4Vq5cCcCqVasYOHCgp+KpSq/RYXNUYfhlRMFW5fBwIiGE\nEM2Rx3rqQ4cO5cknn2Tt2rXYbDZmzpxJ165deeqpp1i8eDHh4eGMHTvWU/FUpdfosTmq0Ouq/4ay\nVdmv8AohhBDCdR4r6j4+PrzzzjsXPD5//nwPpKlfOo2OKrsNwy9F3WqTnroQQgj1Napb2por3S/D\n77qzPXW7FHUhhBDqk6LeAPQaHVXnXlOXnroQQoh6IEW9AVRPlDtn+F2uqQshhKgHUtQbwNmJcjqt\nAsjwuxBCiPrh0RXlWgqD1oATJ4q2upjL8LsQQtSfN998jYMH91NQkI/FYiE8PAI/P39eemnOFV/7\n7bdf4+3tw6BBQ6547JQpk2rWVjlrzJhxDB8+ok7560KKegPQa/TVX2iqi7lV7lMXQoh68+ijfwKq\nC/SxY0eZMmXqVb921KhbXHqvv/71WTp0iHHpNfVJinoDMGh/KepK9bV0uU9dCCEaXkrKNhYt+pTy\n8nKmTPkTqanbWb9+LQ6Hg379BjBx4iQ+/PBdAgICiI7uyLJln6MoGk6ePM7NN4/irrvud+t9Zs78\nK507x5KYeC1du8Yxd+7LKIqC2ezNM8/M5MiRw7WOj43t6nYbpag3AMMvPfUqpw29TkOlTYq6EKJl\n+HzdEbYeyFH1nH1jQ7lzqHu946NHj/DZZ8swGAykpm7n3//+AI1Gw5133spdd9Xe6nvfvr0sXLgU\nh8PBnXeOueqifv77ZGZm8NJLr9ChQ0cee+wPTJ78ON27x7Fw4Sd88cUi4uMTah1fF1LUG4D+l566\nzVGFl15LpVxTF0IIj4iJ6VRTOI1GI1OmTEKr1VJUVERJSUmtY7t0ia11vfxiXnrphVrH/PWvz13k\nfUx06NARgBMnjtO9exwA11zTh/nz3yM+PqHW8XUhRb0BGDTVH5TVbq0u6tYqDycSQoiGcefQGLd7\n1fVBr6/uZGVlnWbx4gXMm7cAs9nMvffeecGxWu2VdwC92DX1rKzTNe9T/Z4XL7VVVTY0Gk2tXHUl\nt7Q1AL2m+gO1OWwYDVosVhl+F0IITyoqKiIwMBCz2czBgwfIysrCZrPV+/tGR3dkz55dAKSmptCl\ni/vXzy9GeuoNwKA921O3YZDhdyGE8LhOnTpjMpl55JGJ9OjRm1tvHcerr75Mz569XDrP+cPvCQl9\n6dUr/pLHT536ZM1EOV9fX/761+c4ePCA2+04n+J0Op2qnc0D1N7ftj72zF2fvokvDv+XB+PuYe1a\nO/tPFvLeXwaj0zb8QElj2hO4Pkj7mq7m3DaQ9jV1jal9jXI/9Zbk7C1tNrsNL331NRqrzIAXQgih\nMinqDeDs4jNWhxUvQ3VRlyF4IYQQapOi3gAu1lO3yAx4IYQQKpOi3gBqbmlznDv8Lj11IYQQ6pKi\n3gD05/bUDdJTF0IIUT+kqDcAQ8019er71AFZKlYIIYTq5D71BnC2p2512AjQy0Q5IYSoTw8/PIE/\n/WlarY1R3nnnX/j7B/C7391zwfEpKdtYtuxzZs36R63HBw26lh49qu9bNxh0WK1VPPHEdKKjO9Rv\nA+pAinoDONtTl4lyQghR/2688SbWrVtdq6ivX7+ON998x6Xz+Pj48K9/vQc0rvvUL0eKegM4t6d+\n9pq6TJQTQoj6MWzYcB555EEmT34MgAMH9hMSEkJISChbtybzwQfvoNfr8fX15YUX/u7y+T/88F0y\nMzM4fTqTiRMn8fnnC2u2Tc3IOMXixQvQarV06dKVqVOfrHX8m2++e1VryrtLinoD+LWnbsXLJD11\nIUTLsezI/0jN2a3qOeNDezAuZvQlnw8MDCI8PIJ9+/bQrVsc69at5sYbRwBQWlrKc8/NIjw8gr/9\n7VmSkzdjNptdzlBVZePf//6AlJRtNdumVlVV8dxzM5g/fyFms5lp0/5ESsq2WsfXNynqDaBm8Rm7\nTJQTQoiGcOONI1i7djXdusWxadOPvP32PAACAgJ4+eVZ2O12MjMzSEjoe8miXlZWxpQpk4Dqa+oG\ng5G//30uAF27dq857uy2qcePH6Nt28ia88XHJ3Do0IELjq9PUtQbgFajRatosZ1zn3qlVYbfhRDN\n37iY0ZftVdeXQYOG8PHH87jxxpto1y4SPz8/AGbP/htz5rxOVFQ0c+e+fNlzXO6aeu2tVau/VhQ4\ndzuVqiobXl5eFxxfn+SWtgZi0OprXVOvtMnwuxBC1Bez2ZuOHTvx8cfza4beAc6cKSMsrDWlpaWk\npGxXdbvVdu3ac+pUGuXlZ4CzW6t2U+38V0N66g1Er9HXmv0ut7QJIUT9uvHGEcya9RzPPfe3msfG\njbuDRx55kHbtIrn77vuYN+89Jk2afNHXnz/8brVWMX783Zd8P5PJxB//+DhPPPEoiqKhZ8/e9OrV\nm23bktVt2GXI1qvnqa/bFp776e9UOe08nfAUU17/kd4xrXjs9p6qv8+VNJXbMtwl7Wu6mnPbQNrX\n1DWm9snWq42AXvtLT91Q/Z9cZr8LIYRQmxT1BmLQGLA6rGg1GnRajQy/CyGEUJ0U9Qai1+qwOapw\nOB0YDVq5pU0IIYTqpKg3kLPbr1Y5qvDSa6m0SlEXQgihLinqDcTwy1KxlXYrXtJTF0IIUQ+kqDcQ\nL231AgSV9kq89Fos0lMXQgihMinqDcSoqy7qlqpKjAYtVXYHdodMlhNCCKEeKeoN5GxP3fJLTx1k\nqVghhBDqkqLeQIw1w+/Wc5aKlSF4IYQQ6pGi3kC8dLWvqYMUdSGEEOqSot5AzvbULVXnDr9LURdC\nCKEeKeoNxHju7HcZfhdCCFEPpKg3EC/duT31s+u/S1EXQgihHinqDeTcnrrRUL3jrVV66kIIIVQk\nRb2BXOyWNumpCyGEUJMU9QZy/uIzINuvCiGEUJcU9QZy7jKxRq/qol5RKUVdCCGEenSuHLx169ar\nOq5v375uhWnOjOcMv5u8qv+zV1TK8LsQQgj1uFTU//SnPzFw4MDLHrNhwwY2btxYp1DNkVajRafR\nUVlVifmXol4uPXUhhBAqcqmoJyYmMnv27Mse8+c//7lOgZozo9arVk9drqkLIYRQ01VdU6+srMRq\ntTJ37tyax3Jyci567LnHiNq8tF5UnlPUpacuhBBCTVfsqX/00UesW7cOnU5Hu3btmDFjBkajkSef\nfJKPP/64ITI2G0adF/kVhRh0GrQaRSbKCSGEUNUVi/rKlSv57LPPANi0aRN/+MMfmDlzZn3napbO\n9tQBTF46mSgnhBBCVVcs6g6Hg6qqKnQ6HQMGDKBjx47MmDGDEydOqBLAYrEwevRoJk+eTL9+/Zg2\nbRp2u52QkBDmzJmDwWBQ5X0aA6PWCydOrA4bJi+t9NSFEEKo6orX1KdNm0ZhYWHN961bt+bdd99l\n6tSptY6zWCxuBXj77bfx9/cH4I033iApKYmFCxfSvn17lixZ4tY5G6tz1383eenkmroQQghVXbGo\nJyQkEBISUuuxRx55hHHjxtV67O6773b5zY8ePcqRI0cYPHgwAMnJyQwbNgyAIUOGsHnzZpfP2Zj9\nuv67BbOXjkqrHYfD6eFUQgghmguXbmn76quveOutt8jMzKwpxABVVVUEBwe7/OYvv/wy//d//8fy\n5csBqKioqBluDw4OJjc31+VzNmbG83rqABXWKryNek/GEkII0Uy4VNTHjBnDzTffzNNPP82jjz5a\n87hGoyE0NNSlN16+fDm9e/emXbt2F33e6by6HmxgoBmdTuvSe19JSIivquerOW9OAKSDwUch0N8E\ngMnbSEiQuV7e75I56ql9jYW0r+lqzm0DaV9T1xTa51JRB9BqtfTt25eIiIg6vfH69etJT09n/fr1\nZGVlYTAYMJvNWCwWjEYj2dnZV/WHQmFheZ1ynC8kxJfc3FJVz3mWs7L6j4/MvHwUpxGAU5lFaOwN\nNwu+PtvXGEj7mq7m3DaQ9jV1jal9l/vjwuWiDrB69WqGDx+Or6/7f7W8/vrrNV+/+eabREREkJqa\nysqVK7n11ltZtWrVFZekbWrM+ureeYWtApOXT/XXMllOCCGEStwq6haLhaFDhxIdHY1e/+v14AUL\nFtQpzKOPPspTTz3F4sWLCQ8PZ+zYsXU6X2Nj1lUX9fKqipr13+VedSGEEGpxq6hPnjz5gsdKSkrc\nDnHu9fn58+e7fZ7G7mxPvbyqggDZflUIIYTK3NpPPTExkaCgIBRFQVEUbDYbr776qtrZmp2anrqt\nXNZ/F0IIoTq3euovvvgiGzduJC8vj8jISNLT05k4caLa2Zodk656lnt5VQVm89nhdynqQggh1OFW\nT33Xrl2sWLGC2NhYli5dyrx586ioqFA7W7NTM/xuq5CeuhBCCNW5VdTPLhBjs9lwOp3ExcWRkpKi\narDmSK/RodfoKa8qx8dUPcGwrMLm4VRCCCGaC7eG36Ojo1mwYAF9+vRhwoQJREdHU1raOO7fa+zM\nOhPltgp8zNVF/YwUdSGEECpxq6g///zzFBcX4+fnx//+9z8KCgp4+OGH1c7WLJn1JoorSzB56VAU\nKJWiLoQQQiVuFXVFUQgICACql44VV8+sM5F1Jgdw4mPSS09dCCGEalwq6klJSSiKcsnn67r4TEtg\n1ptw4sRSVYmPSU9puRR1IYQQ6nCpqJ+/h/q5Llfsxa/M59zW5m3Sk1VQjsPpRCP//YQQQtSRS0X9\n9ddfv2jxdjqdKIoiPfWr8OtSseX4mvQ4nVBuqaqZDS+EEEK4S7Weurg6pnPuVfc2/ToDXoq6EEKI\nunKpqCcmJgJgt9v5+uuv2bNnDwC9e/dm9OjR6qdrhrz11cPvZ2xn8DVV79RWVmEjzJOhhBBCNAtu\nzX6fNWsW+fn5XHvttTidTlasWMGOHTt45pln1M7X7PgZqrerLbGW4WMKBOS2NiGEEOpwq6gfPnyY\nTz/9tOb7e+65h6SkJNVCNWdni3qptYxAkyxAI4QQQj1uLRNrs9lwOBw139vtdux22Rf8avgZqofc\nS62lNdfR5bY2IYQQanCrpz5o0CBuv/12+vbtC0BycjKjRo1SNVhz5Vsz/F6Kd9AvPXWLFHUhhBB1\n51ZRnzx5Mv3792fnzp0oisILL7xAz5491c7WLBm1Xug1OkqsZfiaZVMXIYQQ6nGrqB89epTTp0/T\npk0bunXrRtu2bdXO1WwpioKfwZcSaym+5urd7krOWD2cSgghRHPgUlG3WCw88cQT7N+/n7i4OMrK\nyti/fz/XX389L774Ys2WrOLyfA2+nCrNwOSlRatRpKgLIYRQhUsT5f79738TFhbGqlWreOONN5g3\nbx7r1q3Dy8uL1157rb4yNju+Bh+qnHYq7Rb8vA0US1EXQgihApeK+rZt25g+fTo63a8dfJPJxHPP\nPcfGjRtVD9dc+Z0zWS7Ap7qoO51OD6cSQgjR1LlU1LVa7UWH2PV6PX5+fqqFau7O3tZWYi3D39sL\nW5WDisoqD6cSQgjR1LlU1C+3E5tWq61zmJbCt2YBmlL8vKv/SJIheCGEEHXl0kS51NRUBg8efMHj\nTqeTwsJCtTI1e/6/FPXiyhL8vatHOIrLrLQJ9vZkLCGEEE2cS0X9u+++q68cLUqgMQCAAksRQT4d\nAOmpCyGEqDuXinpERER95WhRgozVG7kUWAqJ9vYCoLis0pORhBBCNANurf0u6sZH741eo6fAUoi/\nj1xTF0IIoQ4p6h6gKApBxkDyLYX4y0Q5IYQQKnFp+H358uWXfX7s2LF1CtOShJiCyC7PQWeoXvdd\nht+FEELUlUtFfdOmTQAUFhZy4MABevXqhd1uZ9euXcTHx0tRd0GYdyh78g9QYC3Ax6SnoFSKuhBC\niLpxqajPmTMHgMcee4w1a9ZgNBoBKCsr45lnnlE/XTPW2hwGQFZ5NsH+RjLzzuB0Oi+7FoAQQghx\nOW5dU8/MzKwp6AA+Pj5kZmaqFqolaO0dAkDWmRxC/I3YqhxyXV0IIUSduLX1aqdOnRg/fjzx8fFo\nNBp27NhBZGSk2tmatTBzKADZ5bm08u8KQF6xhQAfL0/GEkII0YS5VdRfeuklfvrpJw4dOoTT6WTS\npEkMHDhQ7WzNmrfejK/eh6wzOcQGVI965BVVEBPh7+FkQgghmiq3ht+tVisVFRX4+fnh5+dHbm4u\ny5YtUztbs9faO5QCSyEBvtXr5ucWWzycSAghRFPmVk/9wQcfRKPRXLDC3O23365KqJYizBzC4aJj\nKMZyAPKLKzycSAghRFPmVlGvqqpi0aJFamdpcVp7V8+Ar9SWAJBbJD11IYQQ7nNr+D0mJkZ2ZVNB\nmLl6Bny+JRc/bwP5MvwuhBCiDtzqqWdlZTF8+HA6duxYax/1BQsWqBasJWjzS08980w2If5dOZFV\nisPhRKORe9WFEEK4zq2iPmnSpAseKykpqXOYlibAyx8fvTcnS9Jp6x/P0cwSCksrCfY3XvnFQggh\nxHncGn5PTEwkKCgIRVFQFAWbzcarr76qdrZmT1EUovzaUVhZhJ+/E4A8mSwnhBDCTW711F988UU2\nbtxIXl4ekZGRpKenM3HiRLWztQhRfpHsyT8A5iKgegGaLh7OJIQQomlyq6e+a9cuVqxYQWxsLEuX\nLmXevHlUVEgP0x3t/doBUKnLByCroNyTcYQQQjRhbhV1g6F6D3CbzYbT6SQuLo6UlBRVg7UUZ4t6\noSMbgNP5UtSFEEK4x63h9+joaBYsWECfPn2YMGEC0dHRlJaWqp2tRfDWm2ltDiX9TDomry6czj/j\n6UhCCCGaKLeK+vPPP09xcTF+fn7873//o6CggIcffljtbC1GTGAHNmb8TKtwCxknnFTZHei0bg2i\nCCGEaMHcKuqKohAQEADAmDFjcDgcaDRShNzVs1U3Nmb8jD3oBI7jnckurCCilbenYwkhhGhi3KrE\ny5YtY8GCBVRVVfG73/2OYcOGsXDhQrWztRjdgroQam5FiTYNFAen82QIXgghhOvcKuqLFy/mjjvu\nYM2aNXTq1Im1a9eyYsUKtbO1GIqi0CmgI3aqUMylZMp1dSGEEG5wq6h7eXlhMBj44YcfGDlypAy9\nq6CjfxQAWt8C0rPLPBtGCCFEk+R2NX7++edJSUkhMTGR1NRUrFarmrlanNigTgAYgvI5kSVL7goh\nhHCdWxPlXnnlFb799lvuvfdetFotGRkZPP/88y6do6KigunTp5Ofn09lZSWTJ08mNjaWadOmYbfb\nCQkJYc6cOTX3xDd3/l5+RPq2Jc2ZQX55CSXlVvzMLaPtQggh1OFWUd+wYQMPPPBAzfejR492+Rzf\nf/89cXFxPPTQQ2RkZDBx4kSuueYakpKSGDlyJHPnzmXJkiUkJSW5E7FJurZNAmmlp9CFZHAyq5Qe\nHYI9HUkIIUQT4tbw++rVq+u82MyoUaN46KGHADh9+jRhYWEkJyczbNgwAIYMGcLmzZvr9B5NTWLY\nNWjQoA06zYnTMgQvhBDCNW711C0WC0OHDiU6Ohq9Xl/zuDv7qY8fP56srCzeeecdJkyYUDPcHhwc\nTG5u7hVfHxhoRqfTXvE4V4SE+Kp6vqvnS7eQWPawj4PFh5kY0rNe3sVz7WsY0r6mqzm3DaR9TV1T\naJ9bRX3y5MmqBVi0aBH79+/nL3/5C06ns+bxc7++nMJCdddKDwnxJTfXc0vejo4azp6c/ZxQfiY7\n5zdoFHXvLPB0++qbtK/pas5tA2lfU9eY2ne5Py7cKuppaWkXfTwxMfGqz7Fnzx6Cg4Np06YNXbt2\nxW634+3tjcViwWg0kp2dTWhoqDvxmrR2vuH4VXagxHiUDSe3MSjq6v+bCiGEaNnc6gZu37695t/m\nzZt599132bp1q0vn2LZtG/PmzQMgLy+P8vJy+vfvz8qVKwFYtWoVAwcOdCdek9fL7zqcToXlx78m\n60y2p+MIIYRoItzqqc+ePbvW9xUVFcyYMcOlc4wfP56nn36apKQkLBYLzz77LHFxcTz11FMsXryY\n8PBwxo4d6068Ju+a9u1Zu6orSvQ+vjr6HZN63u/pSEIIIZoAt4r6+Uwm0yWH5C/FaDTy6quvXvD4\n/Pnz1YjUpEW38UNb2B5dmyx25u0lreQUkX5tPR1LCCFEI+dWUU9KSkJRlJrvs7Oz6dKli2qhWjq9\nTkNM2wAOnOiAV2wB3xxfxSO9Jno6lhBCiEbOraI+derUmq8VRcHHx4fY2FjVQgmIjQxk34lgwgxt\n2ZN/gOPFaUT7R3o6lhBCiEbMrYlyCQkJZGZmsmrVKlauXMnRo0dr9dxF3cW2DwQUgsur71X/5vgq\nzwYSQgjR6LlV1GfNmsW6deuIjo4mKiqKFStWMGvWLLWztWhRrX3x0ms5dcJIl8AY9hcc4sXkuZyx\nqXtfvhBCiObDreH3w4cP8+mnn9Z8f88997SoNdobgk6roXt0ECmHchkbdAMHC4+QeSaLLw59xQPd\nx3s6nhBCiEbIrZ66zWbD4XDUfG+327Hb7aqFEtX6dAkBIDvdi0d6TkCn0bEtO5UTJa7daSCEEKJl\ncKunPmjQIG6//Xb69u0LQHJyMqNGjVI1mIBeMa3QaRW2HcxlzPWJ/D7uHt7Z9RGvbHuLWzuO5Mb2\ngz0dUQghRCPi9trv/fv3Z+fOnSiKwgsvvEDPnvWz+UhLZvLSERcdzI4jeWQVlNOjVTcm9biPxQeX\ns/zotxh1RgZGXOfpmEIIIRoJt4bfKyoqyM3NxcfHB7PZzKFDh1iyZIna2QSQ8MsQ/NYDOQD0Conj\nzwmTMelM/PfoCspsZzwZTwghRCPiVlGfNGkSCxcuZNu2bbXWgRfqi+/UCp1Ww+Y9WTU717UyBTEq\nahgVVRUsOfQ1DqfjCmcRQgjRErg1/G61Wvnkk0/UziIuwmzUc03nVmzZn8OxzBI6RvgDcEPb/mzJ\nTmVrdgpdAjvSL7yvh5MKIYTwNLd66t26daOgoEDtLOISru/RBoAfdmbWPKbT6Hi4x/3oNXq+OPxf\nMsuyPBVPCCFEI+FST/3smu92u50RI0bQoUMHtFptzfMLFixQPaCAblFBhAWZ2bwnizEDomjlbwIg\n0BjA77qM4+P9i1mXvoF7ut7h4aRCCCE8yaWifu6a76LhaDQKo66LZP63B/hxZybjbuhY81zf1vF8\nc3w127JTGd1hOAFe/h5MKoQQwpNcKupZWVmMGTOmvrKIy0jsGsaitUfYuOs0t14fjVZTfeVEo2i4\nqf0QFh5cyvfpG7kt5mYPJxVCCOEpLl1Tl9vWPMdLr6Vf9zCKyqykHsqr9VximwS89WbWpP3A/oJD\nHkoohBDC09yaKCc8Y1hCWxTgm59P1tzeBqDX6BjbsbqH/p99iyixlnoooRBCCE9yafg9NTWVwYMH\nX/C40+lEURTWr1+vUixxMW2CvUmIDWXbgRz2HC+gR4fgmuf6h/elvKqcL498w1s7PmTqNQ9j0pk8\nmFYIIURDc6mod+vWjblz59ZXFnEVRvdrz7YDOXy96QRx0UG19rEf1u4Gcsvz2JiZzCf7PmdSz/s9\nmFQIIURDc6moGwwGIiIi6iuLuAqRYb70jmnFjiN5HEovoktkYM1ziqJwV5fbSC/NZFfePooqi2U2\nvBBCtCAuXVOXTVsah1H92gOw5IejOM65tg7Vs+H7hffBiZO1aT96Ip4QQggPcamoZ2VdedWyJ554\nwu0w4urERPiT0CWEoxklbN5z4Wdybes+BBuDWJe+gdUn19eaVCeEEKL5cmn4PTk5mRkzZlzxGFH/\n7hoaw+6j+Xyx/ijxnUIwG3/9KA1aPZN63MdbOz9k+dFvsTsdjIga6sG0QgghGoJLRf1qJsnddttt\nbocRV6+Vv4mb+7Xnyw3H+WrTccYP61Tr+ba+4TzV9zFe2fYWXx/7DpPOyA0R/TyUVgghRENwqagn\nJibWVw7hhhHXRrJx92nWbDvF9T3b0DbEp9bzAV7+TO41kTnb3uTzQ8sx60yMCr3BQ2mFEELUN1l8\npgnT67Qk/aYzDqeTj787eMGkOYBwn9Y8Fv8wGkXDdyfWyt7rQgjRjElRb+J6xbSiT5cQjmQU8+OO\nzIseE+0fSWLra8gqz2HLqR0NnFAIIURDcauoL126VO0cog6SbuyMyUvHF+uPUFhaedFjhrcfgoLC\n21s/4VjxiYYNKIQQokG4VdRXr15NaamsL95YBPh4ccfgjlRU2vlszcU3dAkzhzAiaigVNgtv75xP\nua28gVMKIYSoby5NlDvLYrEwdOhQoqOj0ev1NY8vWLBAtWDCNTf0DuenvVlsO5jLjzszuaFX+AXH\njO5wE/6+3iza/RUbM5IZHjWmH5usAAAgAElEQVTEA0mFEELUF7eK+uTJk9XOIepIoyg8NLobz8/f\nyqK1h+nRIZhAX68LjrspZhD/3b+aVWnf0z88ER+DtwfSCiGEqA9uDb8nJCSQmZnJqlWrWLVqFTk5\nOXK7WyMQEmDijiEdsVjtvPvVXuyOC2e6exvMjIwaRkWVhXXpGzyQUgghRH1xq6jPmjWLdevWER0d\nTVRUFCtWrGDWrFlqZxNuuKFXOAldQjiUXsR/Nx6/6DEDwhPx1ptZeXIdW7JSGjihEEKI+uLW8Pvh\nw4f59NNPa76/5557SEpKUi2UcJ+iKEwYGcvJrFK++ekkXSID6R4VVOsYo87I4/EP81rK2yw6uIwu\ngZ3w9/L1UGIhhBBqcaunbrPZcJwztGu327Hb7aqFEnVjNup5ZGwcGo3C+1/vo7jswtvcInzaMKbD\nCCrtVtaf2uiBlEIIIdTmVlEfPHgwt99+O7Nnz2b27Nn89re/ZdiwYWpnE3UQ3caPO4bEUHLGesnr\n69e16YtJZ+KnzC3klOd5IKUQQgg1uVXUQ0NDefbZZwkPDyciIoIXXniBSZMmqZ1N1NGNfdqS0DmE\nA2lFfPH90QueN2j1jIgaSpntDC9tmcu2rFQPpBRCCKEWt66pr169muHDh9O7d2+18wgVKYrCxJu7\nkpl/hlVb04lq7cstg2tfOx/W7gZ89T58cfgrPtn/OVH+kbQyBXsosRBCiLqQxWeaOZOXjinjevC3\n/2zjoxUHiOscio/+1wEaRVG4tk0CdqeDBQe+YGtWKiOjf+PBxEIIIdwli8+0AG2CvXlodDfeXLab\nF+dv4el7E/Ax6WsdEx/ag88PfcnPp7dxU9RQNIrs9SOEEE2NW7+509PTSUxMvOCfaLziO4cwun8U\n2QXlvPfVXhyO2tu0mnRGElsnkGcpYE3aDx5KKYQQoi5kQ5cWZOz10STEhrLneAFfrD+C87z910dF\n/wZ/gx9fH1tJ9pkcD6UUQgjhLreK+tlr6nfeeSd33313zT/RuGk0Ck/enUDrIDMrt6Tz7c8naz0f\n4OXPnZ1vxeF0sOjQcmx2m4eSCiGEcIdcU29hfMwGnhzfm9mfbmfpD8cwG/UMiY+oeb5XSBzdgrqw\nr+AgS4/8j/FdbvNgWiGEEK5wqac+b948gJpr6CaTqebr5cuX10tAob4gPyNPjo/H16zn05UH2Xrg\n16F2RVF4qMe9hJlD2JjxM5sykz2YVAghhCtcKurr16+v9f2cOXNqvj516pQqgUTDCAsy8+c7e+Nl\n0PLeV3vZe6Kg5jmD1sC9Xe8EYOGBpRwuPOapmEIIIVzgUlE/f2LV+d+LpqV9a18e/W1PFAX+tXQ3\nx0+X1DwX7d++prDvzt/nqYhCCCFc4FJRVxTlks9JgW+aurYP5OEx3bFW2Xnt852czj9T81x8aE8M\nGj0bTm3mVGmmB1MKIYS4GnVaYeTcIn+5gi8at4Quodx3UxfKKmy8ungHBSUWoHpt+Hu63oHVYeM/\n+xbJbHghhGjkXJr9npqayuDBg2u+z8/PZ/DgwTidTgoLC9XOJhrQoN4RlFXYWPrDMV5dvIMZ91Sv\nOpcQ1pvDRcfZkLGZ5Ue/5fZOY+QPOCGEaKRcKurfffddfeUQjcCo69pTWm5j1dZ0Xvt8J3++qxfe\nRj23xdzMwYLDrD+1CR+9DyOjZZtdIYRojFwq6hEREVc+yAX/+Mc/2L59O1VVVTz88MP06NGDadOm\nYbfbCQkJYc6cORgMBlXfU1yaoijcOTSGMxU2Nu3J4l9Ld/OnO3vhpTcwpfdD/GPbG6xO+56EsJ6E\nmkM8HVcIIcR5PLZrx88//8zhw4dZvHgxH3zwAS+99BJvvPEGSUlJLFy4kPbt27NkyRJPxWuxNIrC\nhJu7ktAlhIPpRcxZlEppuZVgUyDjYkZTabfyz9T3OFR4xNNRhRBCnMdjRb1v377885//BMDPz4+K\nigqSk5MZNqx6aHfIkCFs3rzZU/FaNI2i8PCY7lzXPYyjGSW8umgH5ZYqrm2TwK0dRlJUWcx7uz/h\njK3c01GFEEKcQ3G6cC/alVaNGzt2rFshFi9ezLZt29i4cWNNIU9LS2PatGksWrTosq+tqrKj02nd\nel9xeQ6Hk399sYPVW9Jo39qXVx6/AaNBx/L9K1m4azmju/yG+3r/1tMxhRBC/MKla+qbNm0CoLCw\nkAMHDtCrVy/sdju7du0iPj7eraK+Zs0alixZwrx58xg+fHjN41f7t0Zhobq9xZAQX3Jzm+8OdK62\n745BHSivsLJpdxYz3/2JP47rQWJgX1Z4ree7Q9+TGNSHYFNQPSZ2jXx+TVdzbhtI+5q6xtS+kBDf\nSz7n0vD7nDlzmDNnDmazmTVr1vDWW2/xzjvvsGrVKvR6vcvBNmzYwDvvvMP777+Pr68vZrMZi6X6\nHuns7GxCQ0NdPqdQl06r4b6butA7phV7TxQy57MdWCqd3NLhJqqcdl5PfZcjRcc9HVMIIQRuXlPP\nzMzEaDTWfO/j40NmpmsrjpWWlvKPf/yDd999l4CAAAD69+/PypUrAVi1ahUDBw50J55QmV6nZfJt\ncQyIa83x0yX8fUEKHU3dGBn1GwotRby3+z/syN0jqwoKIYSHubX1aqdOnRg/fjzx8fFoNBp27txJ\n+/btXTrHt99+S2FhIVOnTq157O9//zvPPPMMixcvJjw83O1r9EJ9Oq2GCTd3xcesZ+WWdGYvSOGJ\nuwYQHBvIpwe+4P3dH3NDRH/u7HyrLE4jhBAe4tJEubOcTic//fQThw4dwul00rFjRwYOHIhG0/CT\n6dW+xtGYrpvUBzXatyL5JF98fxRvo46pd/ZC8S7iswNLyTyTxYNx93BNaE+V0rpOPr+mqzm3DaR9\nTV1jap9q19TPUhSFAQMGMGHCBCZOnMigQYM8UtCFZ4y8tj0TRsVSXlnFK4t2YLAG81CP+1BQ+O7E\nWhxOh6cjCiFEi+RWJbZarSxYsIBXXnkFgJ07d1JZWalqMNG4DewZzkO3dKPSaucfC1MpLdLTJ6w3\nGWWn2ZqV6ul4QgjRIrlV1GfOnElaWhrJyckA7N27l+nTp6saTDR+13Vrzf0junDGYmPOZ6nEaBNR\nUPj+1EaZNCeEEB7gVlE/duwYM2bMqJkBn5SURE5OjqrBRNMwqHcEj47rCU746L9pRBg6kl6awZas\nFE9HE0KIFsetoq7VVq/gdnaWc3l5ec395aLl6d2pFU/+Lh6zUceRLRFo0fPx/sX8lLnF09GEEKJF\ncauojxw5kvvvv59Tp04xa9Ysxo4dyy233KJ2NtGExET4M+OeawgyBlJ+uDsAiw8tJ6+iwMPJhBCi\n5XCrqJtMJp544gmSkpKIjIxk7ty5PPDAAypHE01Nm2Bvnr4vgShTZ6xHe1LlqGLe7oVyfV0IIRqI\nW4vPrF69muHDh9Ozp+fuRxaNU4CPF9PvvoZFa33ZWJjFSdLYnnmAPhFdPR1NCCGaPbeKusViYejQ\noURHR9da833BggWqBRNNl06r4e4bO2P5vj+pLOc/KV9jtIYSFx3s6WhCCNGsuVXUJ0+efMFjJSUl\ndQ4jmg9FUXhwSD9m/rCNPO9TvP7t99yVeC039m3n6WhCCNFsuXVNPTExkaCgIBRFQVEUbDYbr776\nqtrZRBOnKAr39BoNgFfnFBZt2MWnqw5SabN7OJkQQjRPbvXUX3zxRTZu3EheXh6RkZGkp6czceJE\ntbOJZqBTYAfu7DyWzw8txzvmAOtSjKTnlPHY7T3xNrq+Xa8QQohLc6unvmvXLlasWEFsbCxLly5l\n3rx5VFRUqJ1NNBM3RPQjNrATdp9susXZOXyqmOfnb+VkVuPYHEEIIZoLt4q6wWAAwGaz4XQ6iYuL\nIyVFVhATF6coCmM6jgDA2DaNW/pHkVdsYdbH2/hszWFsVbIBjBBCqMGt4ffo6GgWLFhAnz59mDBh\nAtHR0ZSWSq9LXFp7v3bEBESzv+AQ/br35c9te/HpqkOs3pZOZv4ZptzWAy+D1tMxhRCiSdPOnDlz\npqsvGjx4MJGRkQwePBidTofBYGDq1Kn4+PjUQ8TLKy+3qno+b28v1c/ZmHiyfSGmVqRk72Rbzg4C\n/HQ8PGQQ6Tll7D5WwL4ThXRu54+v2VCn95DPr+lqzm0DaV9T15ja5+3tdcnn3Bp+X7p0KWvWrGHZ\nsmVUVlbi4+PDxo0b3Q4oWoaYgGim9X2MMHMIa9J+4Kfsn5l8Wxz941pz/HQJT7+fzNrtpzwdUwgh\nmiy3ht+3b99e87XVamXXrl1cc8013H777aoFE81TG+8wJvW4j1e2v8UXh/5LkaWYB28eSe+YVny6\n6iCfrTnMyexS7hwSg49JZscLIYQr3Crqs2fPrvV9RUUFM2bMUCWQaP5ae4fxdOKfeS3lHdalb2BA\n+LX0iQ3F38fAf747yMZdpzl+uoS/jI/Hz7tuw/FCCNGSuDX8fj6TyURaWpoapxItRKAxgFs7jsDu\ntLP40JdU2q10ahvACxMTGRIfQUbuGWbO38Kh9CJPRxVCiCbDrZ56UlJSzV7qANnZ2XTp0kW1UKJl\niA/tSdfT29hfcIh/pr7LXxKmoNEo3D28M8H+Rpb9cIx/LEzljiEdGd63Xa2fOSGEEBdyq6hPnTq1\n5mtFUfDx8SE2Nla1UKJl0CgaJvW4n3d3fcSBwsN8efQbxsWMRqMojLquPTER/rz93z0sXneE3cfy\neWBkLK38TZ6OLYQQjZbba7+f/de3b1+6du0qvSjhFoNWz11dxuKr92Ft2o9sz95R81zndgE8e39f\nYiMD2HeikGfeT2bviQIPphVCiMbNrZ76P//5z8s+//jjj7sVRrRMoeYQ/pzwCLO3vM78vZ9hc1Rx\nXZs+AAT6evGX38Xz485MPll5iFcX7WB433bcPrgjOq0qU0KEEKLZcOu3YlZWFj/88AMWiwWr1cq6\ndevIyMhAq9Wi1cqqYMJ1oeYQ7us2Ho2iYfGh5RRVFtc8pygKg3pH8PCt3QkNNLFqazqvf7GTorJK\nDyYWQojGx62eemFhIZ9//jk6XfXLH3/8cR599FGmTJmiajjRssSH9qCiahwLDnzBu7v+wyO9JuBn\n8K15vm9sKHHRQbz31V52Hs3nmfeTGTeoAzf0CpdeuxBC4GZPPScnp6agQ/UGL7m5uaqFEi3XdW0S\n6BUSR1rpKf614wNyyvNqPW/y0vHo7T25d3hn7E4nn646xKz/bONoRvElziiEEC2HWz317t27c+ed\nd5KQkABASkqK3NImVKFRNDwUdy8f7l1Aas4uZm99neeu+wsBXv7nHKMw5Jq2XNM5hGU/HmPDrtO8\n+Ml2RiRG8se74j2YXgghPMutDV2GDh1K27ZtqaiowNvbm5EjRzJhwgSPzICXDV1c0xTapygKccGx\nZJfnklF2msOFR+kcGIO33lzrOKNBR3ynEDq19edgWiG7jxVwLKOYDm18MXm59fdqo9cUPj93Nee2\ngbSvqWtM7VNtQ5eysjI++ugjAPr164fRaOTLL7/k888/Jz8/v04hhTiXQWvgwbi7GRCeSHpZJi9t\nmUt6acZFj+0WFcS0pGvo0i6A5L1ZPP1BMqu3puNwOBs4tRBCeJZLRf3ZZ5+tKd7Hjx/ntddeY/r0\n6QwYMIAXX3yxXgKKlkujaPhdl9/y2063YHNU8VrK2xwrPnnRY0MCTExLiufxu3qj0yh8tvYwz3yQ\nzI7DeRc9XgghmiOXinp6ejpPPPEEACtXrmTEiBH079+fu+66i7w8+eUp1KcoCkPaXk9ccCyVdiuv\nbn+LhQeW4HRe2AtXFIXfJLbnxUnXMSCuNVkF5byxdBfvfrWXisoqD6QXQoiG5VJRN5t/vaa5ZcsW\nrrvuuprvZUU5UV8UReHebncxvP0QQk2t2JS5heMlF++xA/iZDTw4uht/vTeBtiHeJO/L5ql3NrN8\nwzEsVinuQojmy6Wibrfbyc/PJy0tjdTUVAYMGADAmTNnqKioqJeAQgD46L25teNIfhf7WwAWH1xO\ngaXwsq+JifDn/+7vw203dMDpdPLVphM8/X4yP+/NwnGRnr4QQjR1LhX1hx56iFGjRnHLLbcwefJk\n/P39sVgsJCUlMXbs2PrKKESNTgEd6N+mL6fKMnlzx/tYqi6/qpxep+WW/lG88scBjO4fRWm5lfe+\n3sffPtom68gLIZodxXmxi5OXYbPZqKysxMfHp+axjRs3cv3116se7mrk5paqer6QEF/Vz9mYNIf2\nOZ1Ovjj8FT+c2kTfsHju63YXGqX679MrtS+3qIIvfzzGz/uyAegeHcQdgzsSGeZ7ydc0Js3h87uU\n5tw2kPY1dY2pfSEhl/595fLNvHq9Hr1eX+sxTxV00TIpisK4mJs5WZLO1uxUtmanck/sHfQL73vF\n14YEmJg0pjs3JUbyxfoj7D1ewN7jBUS38eOuoTF0bhfQAC0QQoj6IQtmiyZJp9Hxh54P4P/L2vCf\nHviCb4+vvuis+Itp39qXJ8fH88RdvenU1p/jp0t4eUEKsz7exsG0y1+rF0KIxqp5LrslWgRfgw9P\n9Z1K5pnTfHZgKd8cX41NW8kt7UbVDMdfSffoILpHB3H4VBELVx/mWGYJcz/fyaDe4QyIa0NkmI/c\n2SGEaDJcvqYOUFlZyYYNGyguLq7VM7r99ttVDXc15Jq6a5pr+4orS3hr54dklJ1mUNv+3NnZvYmb\nu4/l8+H/9lFSbgMgLjqI3/2mE22CvdWM67bm+vlB824bSPuausbUPlWvqQP8/ve/R1EUIiIiaj3u\niaIuBIC/lx+PxU/irV0f8MOpn+gcGEPvkDiXz9OjQzCv/HEAu4/ms3JLGnuOF/B/H2xhyDURjLw2\nkiA/Yz2kF0IIdbhV1G02G4sWLVI7ixB14qP3Zmr/B3lq5Ut8sm8xhjg93YJd3z1Qp9UQ3zmEnjHB\nbN2fw5cbjrF2+ynWp2ZwbbcwRiRG0jbU58onEkKIBubWRLmYmBgKC2UykWh82vq14YFuv6PKaeff\nO+cxf+9CbHabW+fSajRc1701s35/HRNGxhIaaOKnPVk8O28Lr32+kwMnC696Yp4QQjQEt3rqWVlZ\nDB8+nI4dO6LVamseX7BggWrBhHBX79AePO7ly9ztb7MtewdtvFszvP3gq548dz69TsPAXuEM6NmG\nXUfyWZF8kt3H8tl9LJ/oNr6MuLY9CZ1D0GhkQp0QwrPcmii3ZcuWCx4rKSnhN7/5jSqhXCET5VzT\nktpXaCniuc0vY3faARjfZRwDI6673Muv2pGMYr5LTiP1UC5OIDTAxE2J7RjQow0GvfaKr3dXc/78\nmnPbQNrX1DWm9l1uopxbXZfExESCgoJQFAVFUbDZbLz66qtuBxSiPgQaA3gsfhJtfcIBWHRwGWvS\nfsDq5nD8uWIi/JkyrgcvTrqOQb3DKSit5JNVh3jy3z/x1cbjlFXU/T2EEMJVbg2/v/jii2zcuJG8\nvDwiIyNJT09n4sSJamcTos5iAqKZ1udRVpxYy4oTa/jyyDfszN3DmA4jiAnoUOd70FsHmbl/RCxj\nB3Zg7fZ01m3PYPnG43z780kG9gxneGI7Wvkb5V53IUSDcKuo79q1ixUrVnDvvffyySefsGfPHlav\nXq12NiFUodVoGd1hODe07ceiA8vYmbeX11PfJT6kB/d1uwuD1lDn9/D3NjDuho6Muq49G3aeZtXW\nNNamnGJtyikCfb24uV97rq/noXkhhHBr+N1gqP4laLPZcDqdxMXFkZKSomowIdTmZ/Dl9z3u5b6u\nd9HON4LU3N18sv9zVWewGw06buzbjtkP92PSLd2IjQyg5IyVT1cdYsrrG1i09jA5heUya14IUS/c\n6qlHR0ezYMEC+vTpw4QJE4iOjqa0tHFMIBDicjSKhmvbJJAQ1os3Ut8jJWcXZbZyovzaMSr6RvQa\ndVZO1mmrb4e7rntris9YWfHzSbYdzGHV1nRWbU2nc1t/OoT7M+LaSPy86z5SIIQQANqZM2fOdPVF\ngwcPJjIyksGDB6PT6TAYDEydOrXWdqxX69ChQ9x1111oNBp69uzJ6dOnmTx5MkuWLOHHH39k2LBh\ntW6bO195udXl97wcb28v1c/ZmEj7qmkUDd2CYzlecpIjRcc5WnyCAC8/2vu1Uz2T0aAlrkMwQ+Lb\n4u9toLTcytHMEo5kFLNySxp7jxdQUm4lqrUvWu3lB8+a8+fXnNsG0r6mrjG1z9vb65LPuVXUbTYb\ny5cvZ/Xq1UyYMAGAdu3aodO51sspLy/nL3/5Cz169KBVq1b07NmTl156idGjRzN9+nT2799PWloa\nPXr0uMw5pKi7Qtr3K6POi/7hiXQKiCY5azt78w+SdSYHjaLlaNFxvPXemPUm1bJpNQodwv24oVc4\n/eNaY/LSYXc4OZJRzP6ThWzcfZqDaUUUlFgA8DHp0Z1X5Jvz59ec2wbSvqauMbXvckXdrWvqM2fO\nJC0tjeTkZAD27dvH9OnTXT6PwWDg/fffJzQ0tOax5ORkhg0bBsCQIUPYvHmzOxGFuGqdA2N4PP5h\n2vqGsz1nJ+/t/g8LDy7lH9veIK8iX/X3UxSF0EAzt93Qgb/em8Ark/szIjESq83BjiN5fLH+KLM/\nTeGv7/3M9oM52KrsqmcQQjRPbl1APHbsGIsWLeLee+8FICkpiW+++cb1N9fpLujdV1RU1EzECw4O\nJjc3152IQrikc2BHnurzGLvz9nH6TDb5lgI2ZW7ho72f8eeEyW6vRnc1gvyM3Dk0htsHdySroJxN\ne06TlV/OjsN5vPXlHhQgOtyP6DZ+DLu2Pa28L+zBCyEEuFnUzxbis/felpeXY7FY1Ev1i6uZIRwY\naEanU/c2ocut1tMcSPsubVjoryvO2TdV8fOpFN7Y+Q4Do66lT3hPqpx2WpmD0Gnq59a0sDA/enVt\nDUBaVgkrfz7JwZOFHEwr5FhmCWu3nyIsyMzA3hEM6BVOxwj/ZnUPvPxsNm3SPs9zq6iPGDGC+++/\nn1OnTjFr1ix+/PFHkpKSVAlkNpuxWCwYjUays7NrDc1fTGFhuSrve1ZjWgqwPkj7rt7YqNFUVFrZ\nmbuHwwUnmJeyGIDW5lAm93qQYFOgKu9zKSatwtgBUTAgijMWG0czSth7spDvt59iybrDLFl3mCA/\nL7q0C+CGXwp8U+7By89m0ybtaziX++PCrYlyPXv2pGPHjgQFBREUFMQDDzzA8OHD3Q64ZcsWTCYT\nPXv25MiRI1RUVBAbG8v8+fO55ppr6N69+yVfKxPlXCPtu3peWgMJYb3oE9qLVqYgcspz8daZyanI\nIzV3N5G+ESiKglbRoq2nnvtZBp2WsCAzQxPb0y82hI7h/jgcTnKLKjiaWcKm3Vms2Z5O8r4cjmeV\n4HCAyUuHl0HbZHry8rPZtEn7Gs7lJsq5tKHL1q1bL/t83759rz4VsGfPHl5++WUyMjLQ6XSEhYXx\nyiuvMH36dCorKwkPD2f27Nno9fpLnkM2dHGNtK/uvj2+mm+O/7qCYqRvBI/2fgiz3lyv7wsXts/p\ndLLjcB57TxSw/2QhecUWbFWOmuf9vQ3Etg8kLNBEny6hhAWZqLDa8TM3vnvj5WezaZP2NZzL9dRd\nKuqxsbF06NCBnj17XvSv/9mzZ7uXsA6kqLtG2ld3TqeTA4WH2ZDxMztz9wAQ6BXA/d3G0zEgql4n\n1V2pfVV2BzsO55GZd4aT2aUcPlV80c1lvAxaYsL9GNUvCqOhepShfWtfNB7s1cvPZtMm7Ws4lyvq\nLl1T//TTT1m2bBnbt29n8ODBjBkz5rJD40I0R4qi0DWoM12DOuNwOvjuxFpWnFjL66nvYNIZmdA9\nie7BsR7JptNq6BP76zwUu8PBqZxfCnx6EXnFFpzAofQi9p4oZO+Jwppjw4LMdG7rT0SIDx3D/QgJ\nMOFr1jeZ4XshhJv7qVssFlauXMmXX35JXl4eo0eP5pZbbiEiIqI+Ml6W9NRdI+2rH3vy9vP2rvk1\n3/cO6UG34M70DYtXZcOYs9RqX1mFjaMZxRxMK8LhdFJabmXL/hzsjtq/Dgw6DYqi0DHCjzbB3rQL\n9UFRoE2wN1GtfVWdmCc/m02btK/hqDb8fj673c6SJUuYO3cuQM1iNA1JirprpH31x+F0kFZ6ind2\nfkSprazmcZPOxJgON3FD2/51fo/6bF+l1U5ecQUns0s5mVVGfomF/GILWYXlVFovXABHq1EICzIT\nGeqDQa8lPNhMaKCZkAAjrQJM6HUa7HYnet3VFX752WzapH0NR7Xh97OOHj3KkiVL+O677+jWrRsv\nvPACQ4YMcTugEM2BRtEQ5RfJs9c9yZasVPbmH6Cospis8hwWH1rO3vwDRPq2pVtwLNH+kZ6OewEv\ng5aIEB8iQnzoH/fr406nk4KSSkrKrRw5VQzA6YJyTpwu4VRuGZl5Zy56Pp1Wg0Gn4bruYWg1GjpG\n+BHg44W/j4FW/ka0mqZ7+50QjZVLPfXFixezbNkyFEVhzJgxjBo1ioCAgPrMd0XSU3eNtK/hnSrN\n5OP9i8koO13zWNegzvRr05c23mGEmUOu+pa4xtY+p9NJek4ZlTY7haWV5BZVkFdsIa+ogqyCCvJL\nLr4olVajEBJgIjTQhNVmx8dsID42FLNOQ2igiSA/I17NbO/5xvbZqU3a13BUnf3evn37mgVhzp9A\n8/HHH7sZ0X1S1F0j7fMMp9NJbkU+69I3sCGj9n4GUX6RTI1/GL320rduntVY23cp5ZYqsgrKsdrs\nHD9dQpnFRnGZleyCcrIKyjljqbrka01eOvy9Dfh7G/A26TEbdbRt5Y2ft4G2oT74exuwVTnQ6TSN\n8ha98zW1z85V0r6Go1pRz8jIuOzzMlGu8ZP2eV6l3Up6aQYnS9LZm3+Ag4VHaGUMIiawA10DO2HS\nm+ka1Omit8Y1hfa5orTcilajUFhmpbTSzpGTBeQUVVBYYqH4jJWSM1ZKy21c7peUAkS29sXPbCDY\nz4tgfyM+Jj2hASYMBsKUGNEAABjYSURBVC1Gg47wYDMOp9OjQ/7N7bM7n7Sv4ah2Td0TRVuI5sZL\nayAmIJqYgGgGRvTjo70L2Zm3l7zTBfx8ehsAOo2O7kFd8NZ7U1hZRKm1jFamIKYPfsTD6dXl+0sP\n22zUExLiS2yE3wXH2B0OzliqyC2qoKjUSmGphVO5ZZSW2/DSa8kpquBYZsll38eg12C1OWgdVF3c\nfc16Kq12+nVvTZtW3ui1GlAgwMeL0AAjOq1GbuUTTVKdZr83BtJTd420r3GyO+zsKzjI6bJsfszY\nTGFl0UWPC/NuhcVWyQ1tBzAiamgDp6xfdfnsquwObFUO8ostZOafoai0kvLKKkrKbZRbbKTnlKHX\naUjPLrtsr/9cft4GNApEtfbDy6ClbYg3Wo0Ga5Wd8GBvQgNNaBSFAF8vjAbtFW/va6o/m1dL2tdw\n6u2WtsZAirprpH2Nn91h51DhUXQaHU6cFFqKsDpsrE/fSFZ5Ts1xrUzB9A2LZ3QH9/ddaEwa4rOr\nsjtwOJzkFVvQ6zSczColu7AchxMcDifHT5eQXVAOioIC1X8YnLnyet8K4OdjQKMohPgbq+cAeOkI\n8PUi0NeLnMIKuseEUFpaQUiAifBW3ngbrzyHoilpDv/vXU5jap8UdRc0pg+uPkj7mi6r3UqWI5OM\n3FwWHVxGlbP63nFfgw9jOozk2tbXoFE0HCk6TpR/JHqNW3esekxj/OwcTif5xRayC8rJL7Fgsdpp\n5W8kPad6+N9md5BbWFF9219pJQ6nk8LSSq7mt6rRoMXHpK/+Z9bjY9RT9f/t3X90FPW9//HnzM7u\nJrvZTUjIL0L4DQEK8kOkUrHQY0v90fZ71crVXuptb7G1HGz79ahFCoKn5yhY66G17akVvKeHoxTF\n++2ht5Z69JZef4QoYKNEBAOSEAjJZpPsZn/M7s7M5/tHIAUFCihNMrwf/3B2Nmw+L2aZ98znM/P5\n2A5jqwrJWQ7NbT18enJ57wlAvhevR8fv86BrGkqpATc8MBD33ydpIOWTon4eBtKOuxgk3+B2Ip9p\nZYiko/xnw9O0pzpQKIJGgKTVuxTxrPLp/NvEr54ym91ALAQnc8u+y1k2SdMinbHo7slwNJqivStN\nOOTHsR3MjEVzWw89qRwJM9d7cnDSIjxn4/d5KAz46EpkKAnnMaK8gECeF79Xx2vo9KRyVBYHGFEe\nQtc1du2LcNnYEsYPL8R3kR8RdMv+O5OBlE+K+nkYSDvuYpB8g9vp8nWZ3Wxr+h9eO1KH+tCIcXmg\njGHBctrTHaRyaf5l7HXUtu4kkUtSFhjKyHA1Vw2bTb6R/8+McVqX4r47IZOzSaRyJM0cRzuSvUv6\n6hp7m7owsxZJ0yKZztHZkyGbs/F5PcQSWZzzOHwXBn0UFvgI+A2U6j1BKA758RoeEukcBflexlaF\n8egaXkOnqKB3ec8hod4/83wevEbviYHjKHT91BPES3n//bNJUT8PA2nHXQySb3A7W77DPUeIZeIE\nvAH+FnmHvdH9HEu146hzuwocHR7JpOLxXFk5i5SVxqsbNPccoS3ZTmlgKJcNnXxRl5e9lPfdhTCz\nvTcC5iyHRCpLOmvTk+p9BDCezJJI54h0pykvDhCNmUS60/Skc6ed8vdc6JpGIM8g3++ho9ukpDCP\noYV5FAR8KKUwcw5VJQGS6RxVpQUUFvgwPDr5fk/fPAJKgeHR8Ps8xJJZqksL8Hg0PLpOV0+GogLf\ngO1NGkjfz098mlghxMBTHaqiOtT72OmYwpEwDrJ2jqPJVoYFK6iPNPBy818ZVlDJtNIpWI7FkUQr\nLzb9BYXig3gTH8SbeOHQS2f8Hf9nzHVMKB7LwVgTrx6pY4i/kFkVM9jf1ci84Z9hVHgESimiZicl\necXndIB2lHNRl6t1qzyfQZ7vxCE8eM5/z3YclIKc5dDZkyGRyuLRdY50JMhkbXK2g2Ur4qksKOiM\nm2iaRjRukrMcunoylBblE4n1zh54soaD0QvKomm9BX9keYh8vwe/19M334DtKErCvY8ZOkqhaTCk\nwE9+noHXo6PrGl5P7/BDONh7gnGiR+FMTtfT4BZypf4hA+ls7GKQfIPbxcjXnopgWhkOxZtpTbax\nv/sgFYEy/B4fUbOTg7Gmc7ra19AYER5OU/wwcPzEAkhZJqX5xXSa3VSHqjjYfQiP7mHa0E9xIHaI\nQ/Fmvjr+K9w4/QvnnU0phaXsQXFToNu+m0opzKxN1nJQSlFZHmZnQyvdPRmyloNlOViOg5mxSWUs\ncpYN9J4cZHI2XfEMgTyDYJ7B4fbEWWcXPF9FBT4K8r3kbIWZsSgpzKPk+NTD7V0pmtoTzBw/lOGl\nBXg8OgX5Bjmr92TG59UpCedReHz4wW/oBPO9VFcV0dGR+MjvcpRC46MzrF5M0v1+Htz2H+/DJN/g\n1l/5snaOg7FDxLM97G6vZ1iwkjmVV/BBvInWZBuJbJIDsUO0pyJ4dA+Wc34HaK9uMKW8hvc7DjGp\nuIaQL4iu6QwLVtDUc5gCb5BYJk7aMin0hynyF1IeKOX/HXiBLrOLr0/6V5K5JBk7y6crL8ere9kb\n3UehP8zowpGkcmnyjbxTDryWY7En+h4jQlUU5w05p3bmHIucnf3IMIRSiqyTw3+WZXblu3lmJ8pQ\nOmPj8WikMxbtXWks2yGdsUmZOTRNQ9N6exjiqSzpTO/NiDnLQdc1bFvRETfRNY2OWJp0xu5dIVAp\n4qncx86n6xqGruHzetB1jZRpUZBvkDIt/D4PlcUB/D4Dv89DOODF8OhYtoMCLp9QyuRRxR+7DSdI\nUT8P8h9vcJN8/UcphaOcvsJ5MNaE3+NH1zT+5/ArXFkxi5yTI+wLEUlHaUkcZXrpFBqi+9h26GVy\nzsc/8H6YoXmYVjqFtyLvUOQvRAOiZhfFeUPoNLv6fm5KySQmDBlLRbCMD2LNdJpdmJbJpysvJ55N\n8JfDr3Lz+C/zx4Mv0tzTwuyKmYwtHEUylyLsD/F2x7vs63yf+dVzKc0vIXH8BKMiUArA6MKR7Oh4\ng2PdHfz75Fv7FvBxlINpZQh48zEtE13TT3lioS3ZjqZplB3/nLM53dMNnWYXRf7Cf8rwRsnQINEz\nrNjX3zI5GzNrk8la+Ly9jxJ2xEwOtyfQgHgqi65r5Pk8JNMWXT0ZEukcugaZnEPKzJG1FWkzh5m1\nSaRzGJ7emxlztkLXIJbInnFio3y/h1/+33mfWB4p6udhIB80PwmSb3Bza76sncUocMj2aLSl2snY\nGY4m2oianUwsnkB7KkJ1qIq2ZDs9uSSJXAJHOQS9QUrzS3g3uh9H2fg9fqJmZ29RtTJ0mJ39He20\nKoLloBSJXBLLsfnaxJv5r8b/pjsTI+QrIN+TR1VoGO9EGjB0g/+Ysgif7mXboZepLChnfNEYjiRa\nKQuU4tO9/KXlNRq7D+Ioh9kVM+nJJoiko3Sko0woGssVFTMxdA+6pqOhUZpfQsbOUBms4EDsEKAY\nGa7G0A3ejrzLrPLpaEAilyToDRBJR7Eci72d7xPLxJhWOoWh+cXsaqunIbqP4aFh7I7Uc9uEm8nY\nGSaX1IACv+HHqxvk7Bxb3t9Kp9nN7ZP/lZCvANMy+c+GZ1DA50fMY2S4mvrIHiaX1ODTfdQd20Wh\nL8S4otHkGXloaDzz3vMoFP828atn7e5OWya2slFK0RQ/zOSSGpRS/C3yDqAxIjScdzv3cdWw2Rgn\nDd385fCreDQPnx0+h7SV5nDPUSqD5YR8BZSWhmhrjxHP9hD0BnmjdRczyy+jy4xR5A9j6F4yORtl\n671DDFmbfL9BR7KH4kABoys/OgXyhZKifh7cetA8QfINbm7OdzGyvd91gFgmzoyyy8jYGbweH/u7\nDjAqXM2heDMjQsPZ27mftmQ7b7a9RWWwgtkVM2hPdXAk0crezvepDg1jetlU6iMNVATKuG70NTTF\nD/N+90ECRj5dZjfJXIqpQyfTlorQ2P0BHekolrKYUjKJ7kyMfV2Np7Qr38gjbZ1+WdqwL0Q82z/7\nWEP7yGORF8Ln8eE4NpayqQ5VcbjnyCnvzSidikfTeb31zdP+/YCRT8pK970O+0JUBMrY330AgHFF\no/HqXmzlkLbSTBwynoboe6Qtk09XzOSVoztwlEPACBA1Oyn0hbGURTKX+sjvqg5VUV1QRdpK81bk\nHQDmD7+K+kjDKdM1jxkygoNdzWhoBL0BErlkXzav7qXIHyaWiTO7YibDQ8N4sWl7X2/Ql8d8kWtH\nXfOx/11PkKJ+Htx80ATJN9i5OV9/Zztd9/WF3pl/8mfZjs3BWBNXjpvKvpZmCn1hvB4vqVyatJVm\nb+d+usxuyoNlTC6uIegN8LfIHl5u/isLa/6FnmyS14/W0RRvYcHI+eQci/9q/G/yjXyml06htvVN\nKgJlfHfaf7C3cz85J8eocDUtPUdJWyZvtr1FeaCUsYWjSFsmB2NNNMY+wHIsSvKG4Pf4OZo8dkr7\nA0Y+wwuGkWfk0Z6KcCzVTkneEL4wcj71kQb2du4/5eenDp1M9ZByapvfOu26BV7dS9AbYHLxhI8U\n8g8PhWhoeD1exhaO4miilVi2B13TcZRzvJDb5/yYJkBVQSVHEq3n/PMnt0PXdPI8/r5JnS6Urun8\nfP7Dn9jNdFLUz0N/H1guNsk3uLk5n5uzwSebz3bsU8blNbTzKhiOcsjaWby6t+9zerIJjiaOMX7I\nGIBTTmZyjoUGfV3VqVyaQ/FmujMxrqycha7plJaGaG3rwnJsWhJHGeIv5L2u9/F7/FxeNg3ovUM8\nmUvxvy2vcyzVzueq5zIqPILao29yLNXOF0bOJ9+T19emZC5F2jIp8ofpNLsYml+Coxyi6U4M3Yut\nLPKNfN48tpuwL8Se6Htk7Sw3jf8S3Zk47akIcyqvYH/XAXweHyPDw/lry+v86YOX+PLYaxniLyTg\nDeDRdNKWiVKKkvxiWhJHGV80hpCvgIydZVdbPZePnoSd8NCWaqchuo+R4eFE010U+IK8HWlgbtWV\njC0cxeutb/bdy1ARKMPQDbJ2lkklEz72fj9Bivp5kAPL4Cb5Bi83ZwPJN5BcSA/MQMp3tqIuMz4I\nIYS4pLh5siP3JhNCCCEuMVLUhRBCCJeQoi6EEEK4hBR1IYQQwiWkqAshhBAuIUVdCCGEcAkp6kII\nIYRLSFEXQgghXEKKuhBCCOESUtSFEEIIl5CiLoQQQriEFHUhhBDCJaSoCyGEEC4hRV0IIYRwCSnq\nQgghhEtIURdCCCFcQoq6EEII4RJS1IUQQgiXkKIuhBBCuIQUdSGEEMIlpKgLIYQQLiFFXQghhHAJ\nKepCCCGES0hRF0IIIVxCiroQQgjhElLUhRBCCJeQoi6EEEK4hNHfDTidhx56iPr6ejRNY/ny5Vx2\n2WX93SQhhBBiwBtwRf2NN96gqamJzZs3c+DAAZYvX87mzZv7u1lCCCHEgDfgut9ra2v5/Oc/D8DY\nsWOJxWIkEol+bpUQQggx8A24ot7R0cGQIUP6XhcXFxOJRPqxRUIIIcTgMOC63z9MKXXW90tLQ5/4\n77wYnzmQSL7Bzc353JwNJN9gNxjyDbgr9bKyMjo6Ovpet7e3U1pa2o8tEkIIIQaHAVfUr7rqKv78\n5z8D0NDQQFlZGQUFBf3cKiGEEGLgG3Dd7zNnzuRTn/oUt956K5qmsWrVqv5ukhBCCDEoaOofDVoL\nIYQQYlAYcN3vQgghhLgwUtSFEEIIlxhwY+r9yS3T0+7fv58lS5bwjW98g0WLFtHa2sp9992HbduU\nlpbyk5/8BJ/Px9atW/ntb3+LrussXLiQW265pb+bfk4eeeQRdu3ahWVZfOc732Hq1KmuyJdOp1m2\nbBnRaJRMJsOSJUuYOHGiK7KdzDRNvvSlL7FkyRLmzJnjmnx1dXV8//vfZ/z48QBMmDCBxYsXuyYf\nwNatW1m/fj2GYfC9732Pmpoa1+R77rnn2Lp1a9/rPXv2sGnTJlavXg1ATU0NDz74IADr169n27Zt\naJrG0qVLmTdvXn80+fSUUEopVVdXp7797W8rpZRqbGxUCxcu7OcWXZhkMqkWLVqkVqxYoTZu3KiU\nUmrZsmXqhRdeUEop9dOf/lQ9/fTTKplMqgULFqh4PK7S6bS64YYbVFdXV382/ZzU1taqxYsXK6WU\n6uzsVPPmzXNNvj/+8Y/qN7/5jVJKqZaWFrVgwQLXZDvZY489pm666Sb1/PPPuyrfjh071F133XXK\nNjfl6+zsVAsWLFA9PT2qra1NrVixwlX5TlZXV6dWr16tFi1apOrr65VSSt19991q+/btqrm5Wd14\n440qk8moaDSqvvjFLyrLsvq5xX8n3e/HuWV6Wp/Px5NPPklZWVnftrq6Oq655hoAPve5z1FbW0t9\nfT1Tp04lFAqRl5fHzJkz2b17d381+5xdccUV/OxnPwMgHA6TTqddk+/666/njjvuAKC1tZXy8nLX\nZDvhwIEDNDY2Mn/+fMBd383TcVO+2tpa5syZQ0FBAWVlZfz4xz92Vb6T/fKXv+SOO+7gyJEjfT22\nJ/LV1dVx9dVX4/P5KC4upqqqisbGxn5u8d9JUT/OLdPTGoZBXl7eKdvS6TQ+nw+AkpISIpEIHR0d\nFBcX9/3MYMnr8XgIBAIAbNmyhc9+9rOuygdw6623cs8997B8+XLXZVu7di3Lli3re+22fI2Njdx5\n553cdtttvPbaa67K19LSgmma3HnnnXzta1+jtrbWVflOePvtt6msrMTj8RAOh/u2D5Z8MqZ+Bsql\nT/qdKddgy/vSSy+xZcsWnnrqKRYsWNC33Q35fve737F3717uvffeU9o92LP9/ve/Z/r06VRXV5/2\n/cGeb9SoUSxdupTrrruOw4cPc/vtt2Pbdt/7gz0fQHd3N7/4xS84evQot99+u6u+nyds2bKFG2+8\n8SPbB0s+uVI/zs3T0wYCAUzTBKCtrY2ysrLT5j25y34ge+WVV/j1r3/Nk08+SSgUck2+PXv20Nra\nCsCkSZOwbZtgMOiKbADbt2/n5ZdfZuHChTz33HP86le/cs2+AygvL+f6669H0zRGjBjB0KFDicVi\nrslXUlLCjBkzMAyDESNGEAwGXfX9PKGuro4ZM2ZQXFxMd3d33/Yz5TuxfaCQon6cm6en/cxnPtOX\n7cUXX+Tqq69m2rRpvPPOO8TjcZLJJLt372bWrFn93NJ/rKenh0ceeYQnnniCoqIiwD35du7cyVNP\nPQX0DgelUinXZANYt24dzz//PM8++yy33HILS5YscVW+rVu3smHDBgAikQjRaJSbbrrJNfnmzp3L\njh07cByHrq4u130/obdAB4NBfD4fXq+XMWPGsHPnTuDv+a688kq2b99ONpulra2N9vZ2xo0b188t\n/zuZUe4kjz76KDt37uybnnbixIn93aTztmfPHtauXcuRI0cwDIPy8nIeffRRli1bRiaTYdiwYTz8\n8MN4vV62bdvGhg0b0DSNRYsW8ZWvfKW/m/8Pbd68mccff5zRo0f3bVuzZg0rVqwY9PlM0+RHP/oR\nra2tmKbJ0qVLmTJlCj/84Q8HfbYPe/zxx6mqqmLu3LmuyZdIJLjnnnuIx+PkcjmWLl3KpEmTXJMP\neoeGtmzZAsB3v/tdpk6d6qp8e/bsYd26daxfvx7ovUfigQcewHEcpk2bxv333w/Axo0b+cMf/oCm\nafzgBz9gzpw5/dnsU0hRF0IIIVxCut+FEEIIl5CiLoQQQriEFHUhhBDCJaSoCyGEEC4hRV0IIYRw\nCZlRTohLXEtLC9deey0zZsw4Zfu8efNYvHjxx/78uro61q1bx6ZNmz72Zwkhzk6KuhCC4uJiNm7c\n2N/NEEJ8TFLUhRBnNHnyZJYsWUJdXR3JZJI1a9YwYcIE6uvrWbNmDYZhoGkaDzzwAOPGjePQoUOs\nXLkSx3Hw+/08/PDDADiOw6pVq9i7dy8+n48nnniCYDDYz+mEcB8ZUxdCnJFt24wfP56NGzdy2223\n8fOf/xyA++67j/vvv5+NGzfyzW9+kwcffBCAVatW8a1vfYunn36am2++mT/96U9A75Krd911F88+\n+yyGYfDqq6/2WyYh3Eyu1IUQdHZ28vWvf/2Ubffeey/QO+c3wMyZM9mwYQPxeJxoNNq3zvTs2bO5\n++67gd5lK2fPng3ADTfcAPSOqY8ZM4ahQ4cCUFFRQTwev/ihhLgESVEXQpx1TP3kmaQ1TUPTtDO+\nD71d7R/m8Xg+gVYKIf4R6X4XQpzVjh07ANi1axc1NTWEQiFKS0upr68HoLa2lunTpwO9V/OvvPIK\nAC+88AKPPfZY/zRaiEuUXKkLIU7b/T58+HAA3n33XTZt2kQsFmPt2rUArF27ljVr1uDxeNB1ndWr\nVwOwcuVKVq5cyTPPPINhGDz00EM0Nzf/U7MIcSmTVdqEEGdUU1NDQ0MDhiHn/0IMBtL9LoQQQriE\nXKkLIYQQLiFX6kIIIYRLSFEXQgghXEKKuhBCCOESUtSFEEIIl5CiLoQQQriEFHUhhBDCJf4/iHdv\n5pDG13QAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<matplotlib.figure.Figure at 0x7f50528432e8>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "Vyf9RwjUqQgD",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Next, compare how the model performs on the test dataset:"
      ]
    },
    {
      "metadata": {
        "id": "NQdmDB_fpy6B",
        "colab_type": "code",
        "outputId": "124d544b-3194-413d-b797-2b19d8d172ad",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "cell_type": "code",
      "source": [
        "X_test = scaler.transform(X_test)\n",
        "\n",
        "mse, _, _ = model.evaluate(X_test, y_test)\n",
        "rmse = np.sqrt(mse)\n",
        "print('Root Mean Square Error on test set: {}'.format(round(rmse, 3)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "102/102 [==============================] - 0s 146us/step\n",
            "Root Mean Square Error on test set: 4.17\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "N31VRe_rXZou",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Part 2: Classification of MNIST Dreams with Convolutional Neural Networks\n",
        "\n",
        "Next, let's build a convolutional neural network (CNN) classifier to classify images of handwritten digits in the MNIST dataset with a twist where we test our classifier on high-resolution hand-written digits from outside the dataset."
      ]
    },
    {
      "metadata": {
        "id": "MbO9twklXipV",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The MNIST dataset containss 70,000 grayscale images of handwritten digits at a resolution of 28 by 28 pixels. The task is to take one of these images as input and predict the most likely digit contained in the image (along with a relative confidence in this prediction):\n",
        "\n",
        "<img src=\"https://i.imgur.com/ITrm9x4.png\" width=\"500px\">\n",
        "\n",
        "Now, we load the dataset. The images are 28x28 NumPy arrays, with pixel values ranging between 0 and 255. The *labels* are an array of integers, ranging from 0 to 9."
      ]
    },
    {
      "metadata": {
        "id": "XJDfFbM31NHp",
        "colab_type": "code",
        "outputId": "b2319fa0-6541-471b-9316-ca55375594ea",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "cell_type": "code",
      "source": [
        "(X_train, y_train), (X_test, y_test) = keras.datasets.mnist.load_data()"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\n",
            "11493376/11490434 [==============================] - 0s 0us/step\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "o4aaY-N-X0nF",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "We scale these values to a range of 0 to 1 before feeding to the neural network model. For this, we divide the values by 255. It's important that the *training set* and the *testing set* are preprocessed in the same way:"
      ]
    },
    {
      "metadata": {
        "id": "OjxLOlEAXk1b",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def preprocess_images(imgs): # should work for both single and multiple images\n",
        "  sample_img = imgs if len(imgs.shape) == 2 else imgs[0]\n",
        "  assert sample_img.shape == (28,28), sample_img.shape # make sure images are 28x28\n",
        "  return imgs / 255.0\n",
        "\n",
        "X_train = preprocess_images(X_train)\n",
        "X_test = preprocess_images(X_test)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "HLtfvbY6Y-eU",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Display the first 5 images from the *training set* and display the class name below each image. Verify that the data is in the correct format and we're ready to build and train the network."
      ]
    },
    {
      "metadata": {
        "id": "rNzYLctWY7iT",
        "colab_type": "code",
        "outputId": "5cdc5539-29e7-48b2-f8e8-0cebf27d678a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 141
        }
      },
      "cell_type": "code",
      "source": [
        "plt.figure(figsize=(10,2))\n",
        "for i in range(5):\n",
        "    plt.subplot(1,5,i+1)\n",
        "    plt.xticks([])\n",
        "    plt.yticks([])\n",
        "    plt.grid(False)\n",
        "    plt.imshow(X_train[i], cmap=plt.cm.binary)\n",
        "    plt.xlabel(y_train[i])"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjwAAAB8CAYAAABtyGrfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAEcZJREFUeJzt3X10z/X/x/HnTHOZSdOQq0OOi+Ui\nQk1Lri8OucooZC5ykatz0FxWjpbDpA6NSnWIou0oOnTBOAy5GoW1hBxkR2xoVmhin+8/v9/r+3q9\nvz5rts8+7+3lfvvr8e75+Xz2rMXndd6v9+v1CvB4PB4BAACwWAm3GwAAAChsDHgAAID1GPAAAADr\nMeABAADWY8ADAACsx4AHAABYjwEPAACwHgMeAABgPQY8AADAegx4AACA9RjwAAAA6zHgAQAA1mPA\nAwAArMeABwAAWI8BDwAAsB4DHgAAYD0GPAAAwHoMeAAAgPUY8AAAAOuVdLsBIL8OHTqkclxcnFH7\n5JNPVB46dKhRmzBhgsrNmzcvpO4AAEUJd3gAAID1GPAAAADrBXg8Ho/bTRTU7du3Vb569Wqe3uOc\nArl+/brKx48fN2pLly5VeerUqUZt7dq1KpcuXdqoTZ8+XeXXX389T33Bu8OHDxvX7dq1UzkrKyvP\nnxMcHKzylStXCt4YioRt27YZ14MGDVI5KSnJqNWvX98vPSF3MTExxvVrr72msvOraceOHSq3bdu2\nUPuCnbjDAwAArMeABwAAWI8BDwAAsF6RWpb+22+/qXzz5k2jtmfPHpV3795t1DIzM1Vet25dgfuo\nUaOGca0vY16/fr1Ru//++1Vu2rSpUWOeueAOHDigcr9+/Yya/rxWQECAUatQoYLKQUFBRu3SpUsq\n792716i1aNHC6/tssHPnTpUvX75s1Pr06ePvdnwqOTnZuH788cdd6gS5Wblypcrz5883aoGBgSrr\nz2aK/O+fceBucYcHAABYjwEPAACwnqtTWj/++KNx3b59e5XzurzcV/Rbqc6lkuXKlVNZX+oqIlKt\nWjWVH3jgAaPG0te80bcEEBH54YcfVB48eLDK58+fz/Nn1qtXT+Xo6GijNmDAAJXbtGlj1PTf/cyZ\nM/P884oLfWnvyZMnjVpxnNLKyclR+fTp00ZNnyK3YPcNa5w9e1bl7OxsFzu5t+3fv1/l1atXq6xP\ne4uI/PTTT14/Y9GiRSrr34UiIrt27VJ5yJAhRq1169Z316yPcIcHAABYjwEPAACwHgMeAABgPVef\n4alVq5ZxHRISorIvnuFxzhPqz9hs377dqOlLkJ3zjShco0ePNq7XrFlT4M/UT1L/66+/jJq+XYD+\nTIuISEpKSoF/dlGmnyIfHh7uYie+8fvvv6u8fPlyo6b/OW7QoIHfeoJp69atxvWSJUu8vlb/PW3a\ntMmohYaG+raxe0x8fLxxPWnSJJUzMjJUdj7v9swzz6isb+kh8r9HLen0z3G+7/PPP//3hgsBd3gA\nAID1GPAAAADruTqlValSJeN64cKFKm/cuNGoPfbYYypPnDjR62c2a9ZMZeetVH15uXOpXW63WeF7\n+pST89a1tyXE+q1VEZEePXqo7Ly1qi+R1P/fEcl9atP25cv6Mm4bjBw50mtN35oA/qXvhh8VFWXU\nsrKyvL7vlVdeUdn5yAP+3a1bt4xrfffxl156yahdu3ZNZX2a/9VXXzVe99RTT6ns3EYgMjJS5c2b\nN3vtq6jses4dHgAAYD0GPAAAwHoMeAAAgPWK1GnpvXv3Vlk/ZkLEPJX86NGjRu2jjz5SWX+WQ39m\nx+nRRx81rp1LWuFbhw8fNq47duyosnNOXz8VuXv37iqvXbvWeJ2+pPzNN980avqzHZUrVzZq+qn2\nzhOYv/76a5X1Iy5ERJo3by7FjfPPysWLF13qpHBkZmZ6rXXq1MmPnUCnb3+Q25EwzufyXnzxxcJq\n6Z7w6aefGtcjRozw+trOnTurrC9Zr1Chgtf3OJe25/bcTo0aNVQeOnSo19f5E3d4AACA9RjwAAAA\n6xWpKS1dbrfVgoODvdb06a2BAwcatRIlGN/504kTJ1SOjY01avpO2s4pp6pVq6qs3wotX7688Tp9\nWbqeC0I/uf2tt94yar7YAdrfvvnmG+P6xo0bLnXiG84puTNnznh97cMPP1zI3eD/OXfS/fjjj1UO\nDAw0ahUrVlR59uzZhdvYPUD/bzhv3jyjpk/Zjxs3zqjFxMSonNv3rc756EBu9K1enH/Hu4URAAAA\nsB4DHgAAYD0GPAAAwHpF9hme3MyZM8e41o8p0JcqO4+W0Jfhwfec247rWwToy71FzDnjVatWGTV9\nG3I3nzk5d+6caz/bV44fP+61FhYW5sdOfMN5hMiFCxdUrl+/vlHTt7KA7+nPT/Xt2zfP75swYYLK\nzu1H8O/mzp1rXOvP7ZQqVcqodenSReUFCxYYtTJlytzx8//++2/jesuWLSqfPXvWqOlH8TiPpOjV\nq9cdP99N3OEBAADWY8ADAACsVyyntJw7KH/44Ycq67vhOk+HbdeuncrO01v1JXvO3XeRN86diZ3T\nWLqvvvpKZf2kXvhPy5Yt3W5B0Xfb/u6774yavnusfnvdybnEWV/+DN/Tf08pKSleX9ehQwfjetKk\nSYXWk630HcWXLVtm1PTvK30KS0Rkw4YNefr8X3/9VeVBgwYZtYMHD3p9X//+/VWOjo7O089yE3d4\nAACA9RjwAAAA6xXLKS2nunXrqrxy5UqVhw0bZrxOXw3kXBl07do1lZ0H2Ok7/8K7yZMnG9f6E/zO\nQwKLyjSW3uPd1Gxw5cqVfL3vyJEjxnVOTo7K27ZtM2ppaWkq37x5U+XPPvvM62c4V4+0bt1aZecq\nlH/++Udl5zQ1fE+fIpk+fbrX10VERKisHyQqkvtO+bgz/c9ORkaG19fpuxuLiKSnp6u8YsUKo6Y/\nVpCamqryn3/+abxOnzJznlYwePBglXM7rLuo4A4PAACwHgMeAABgPQY8AADAelY8w6Pr06ePyo88\n8ohRmzJlisrOXZhnzJihsnM3yVmzZqnMCcymTZs2qXz48GGjps/9Pvvss37r6W44tyDQr5s1a+bv\ndnzO+TyM/u83evRoo+Y8adkb5zM8+rNO9913n1ErW7asyg0bNlR5+PDhxutatGihsvN5r9DQUJWr\nV69u1PSduBs0aPBvreMuOU+jz+uOynXq1FFZ//0hf4KCglR+6KGHjJr+nE7t2rWNWl63WNG/15wn\np58/f17lkJAQo9azZ888fX5RwR0eAABgPQY8AADAetZNaekaN25sXCckJKi8ceNGoxYVFaXy+++/\nb9ROnjypcmJiog87LP70KQV96aSIeet1wIABfuvJyXmoqfPwWZ2+K+z8+fMLqyW/ce7KWqtWLZX3\n7NmTr8+sWbOmca0fEtioUSOj9sQTT+TrZ+iWL1+usn77XsScOoHvOQ+cDAwMzNP7cluyjrun7xru\n3D25R48eKl++fNmo6Y91OA/z1L/zKlWqpPLAgQON1+lTWs5accMdHgAAYD0GPAAAwHoMeAAAgPWs\nfobHSZ8HHTJkiFEbOXKkyvp29SIiO3fuVHnHjh1GzbmEFv9VunRplf19PIf+3E5MTIxRi42NVblG\njRpGTd+6oHz58oXUnXumTZvmdgt3zXlche65557zYyf3Bn17ic2bN+fpPc5tJ+rXr+/TnvBf+lEr\nIrkfNZFX+ndcUlKSUdOXthf3Z+a4wwMAAKzHgAcAAFjP6imto0ePGtfr1q1TOTk52ag5p7F0+lLb\np59+2kfd2c+fuys7d3nWp63i4+ONmr4888svvyzcxlCoevfu7XYL1uncubPKf/zxh9fX6VMrzhPR\nUbzo24vktvs8y9IBAACKOAY8AADAegx4AACA9ax4huf48eMqv/vuuyo7n8+4cOFCnj6vZEnzP4u+\npLpECcaIOv2kbD2LmFugL1682Oc/++2331b5jTfeMGpXr15VefDgwUZt1apVPu8FsMWlS5dUzu0o\niXHjxqls4xYO95IuXbq43YJf8O0NAACsx4AHAABYr9hMaenTUWvWrDFqcXFxKp85cyZfn9+yZUuV\nZ82aZdT8uby6uNGXLDqXM+q/s4kTJxq14cOHq/zggw8atX379qm8evVqlY8cOWK87ty5cyrrp4CL\niHTt2lXll19+2fu/AIq1kydPqvzkk0+62EnxNWzYMONan5q+ffu21/eFh4cXWk/wr7zuqF3ccYcH\nAABYjwEPAACwHgMeAABgvSL1DM/FixdVTk1NNWrjx49X+ZdffsnX5+tboUdHRxs1/bgBlp77xq1b\nt1ReunSpUdOP+QgODjZqJ06cyNPn688QtG/f3qjNnTs3z32i+MrJyXG7hWJJP4olMTHRqOnP4pUq\nVcqo6c/DhYaGFlJ38LdTp0653YJf8M0OAACsx4AHAABYz+9TWleuXFF59OjRRk2/zZrfW2xt2rRR\necqUKUZN302yTJky+fp8mPSlwK1atTJqBw4c8Po+fcm6PpXpFBISorLzpN7C2L0ZxcvevXtVjoqK\ncq+RYiYzM1Pl3P78VatWzbhetGhRofUE90RERKjs3DHfJtzhAQAA1mPAAwAArMeABwAAWK9QnuHZ\nv3+/yrGxsUYtOTlZ5bS0tHx9ftmyZY1r/dgC/ViIcuXK5evzkXfVq1dX2Xk6/QcffKCy8zTz3Eya\nNEnlsWPHqlyvXr38tAgAyEXjxo1Vdv49qz9P63y2tnLlyoXbmI9xhwcAAFiPAQ8AALBeoUxprV+/\n/o753zRq1Ejlnj17GrXAwECVp06datQqVqx4ty2iEFStWtW4njNnzh0zcDe6deumckJCgoud2KNB\ngwYqO08937Vrl7/bQREyc+ZM43rEiBFea3FxcSrr399FFXd4AACA9RjwAAAA6zHgAQAA1gvw2LyP\nNAAAyLOsrCzjOjIyUuXExESj1q9fP5VXrFhh1IritjDc4QEAANZjwAMAAKzHlBYAALgjfYpLP8lA\nRGTZsmUqp6SkGLWiuEydOzwAAMB6DHgAAID1GPAAAADr8QwPAACwHnd4AACA9RjwAAAA6zHgAQAA\n1mPAAwAArMeABwAAWI8BDwAAsF5JtxsoDPv375fJkydLnTp11D+LiYmRWrVqudgV8mvp0qWSlJQk\nHo9H2rZtK+PHj3e7JRTQvHnz5NixY7J69Wq3W0EBZGRkyNSpU+XmzZuydu1at9tBAXg8HlmwYIEc\nPHhQSpYsKZGRkdK3b1+32/IpKwc8IiIREREyf/58t9tAAR05ckQSExMlISFBRESef/55CQ8Pl+bN\nm7vcGfIrOTlZUlNTpUQJbjAXd5MnT5aIiAjZvn27262ggLZt2yYpKSkSHx8vt2/flhdeeEHatGkj\noaGhbrfmM/yNgyJt586d0qFDBwkKCpKgoCDp0KGDJCUlud0W8un69euycOFCmTZtmtutwAfee+89\nadq0qdttwAdOnz4tTZo0kcDAQAkKCpJWrVrJ7t273W7Lp6wd8Pz8888yduxYGTBggCxevFhycnLc\nbgn5kJ6eLiEhIeq6cuXKkp6e7mJHKIgFCxZIVFSUVKpUye1W4APly5d3uwX4SFhYmOzZs0du3Lgh\n169fl0OHDklGRobbbfmUlVNatWvXljFjxkjXrl0lOztbRo0aJV988YX079/f7dZQQB6PRwICAtxu\nA/nw/fffS2ZmpnTv3l3S0tLcbgeAJjw8XLp16yZRUVESGhoq9erVk1KlSrndlk9ZeYcnNDRUunfv\nLiVKlJAyZcpIx44d5dixY263hXyoUqWKcUcnPT1dqlSp4mJHyK9vv/1Wzp49K5GRkTJ+/HhJTU2V\n6Ohot9sC8H/GjBkj8fHxsmTJEvF4PNb9XWvlgGfDhg3yzjvviIhITk6O7Nu3Txo2bOhyV8iPZ555\nRrZu3SrZ2dmSnZ0tW7ZskXbt2rndFvIhJiZGNmzYIAkJCRIXFydhYWESGxvrdlsAROTUqVMyatQo\n8Xg8cunSJdm3b5+Eh4e73ZZPWTml1alTJ5kxY4ZERkaKiEiTJk2sW153rwgLC5NevXrJoEGDJCAg\nQHr16iWNGzd2uy3gnnf+/HmZNm2aZGVlSVpamgwZMkTatm0rI0eOdLs15EPdunWlZs2a0rdvX/F4\nPDJ79mwJDg52uy2fCvB4PB63mwAAAChMVk5pAQAA6BjwAAAA6zHgAQAA1mPAAwAArMeABwAAWI8B\nDwAAsB4DHgAAYD0GPAAAwHr/AToVzKGPtTMJAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 720x144 with 5 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "lkdhPKXxZdZR",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def build_model():\n",
        "  model = keras.Sequential([\n",
        "      keras.layers.Flatten(input_shape(28, 28)), # flatten image to a 1d-array\n",
        "      keras.layers.Dense(128, activation=tf.nn.relu), # fully connected layer (128 nodes)\n",
        "      keras.layers.Dense(10, activation=tf.nn.softmax)# softmax layer returns an array of 10 probability scores that sum to 1\n",
        "  ])\n",
        "  return model"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "-ZT7tKVyaSBb",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Before the model is ready for training, it needs a few more settings. These are added during the model's *compile* step:\n",
        "\n",
        "* *Loss function* - measures how accurate the model is during training, we want to minimize this with the optimizer.\n",
        "* *Optimizer* - how the model is updated based on the data it sees and its loss function.\n",
        "* *Metrics* - used to monitor the training and testing steps. \"accuracy\" is the fraction of images that are correctly classified."
      ]
    },
    {
      "metadata": {
        "id": "X-UM9sRhaTZx",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "As the model trains, the loss and accuracy metrics are displayed. This model reaches an accuracy of about 98.6% on the training data."
      ]
    },
    {
      "metadata": {
        "id": "nivT5lZEGH8U",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Make predictions\n",
        "\n",
        "With the model trained, we can use it to make predictions about some images. Let's step outside the MNIST dataset for that and go with the beautiful high-resolution images generated by a mixture of CPPN, GAN, VAE. See [great blog post by hardmaru](http://blog.otoro.net/2016/04/01/generating-large-images-from-latent-vectors/) for the source data and a description of how these morphed animations are generated:\n",
        "\n",
        "![MNIST dream](https://i.imgur.com/OrUJs9V.gif)"
      ]
    },
    {
      "metadata": {
        "id": "hjWllA3paOLx",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "9266868d-9719-475b-92eb-aa15d26bb118"
      },
      "cell_type": "code",
      "source": [
        "mnist_dream_path = 'images/mnist_dream.mp4'\n",
        "mnist_prediction_path = 'images/mnist_dream_predicted.mp4'\n",
        "\n",
        "# download the video if running in Colab\n",
        "if not os.path.isfile(mnist_dream_path): \n",
        "    print('downloading the sample video...')\n",
        "    #mnist_dream_path = urllib.request.urlretrieve(os.path.join(deep_repo_url, 'mit_driveseg_sample.mp4')[0]\n",
        "                                                \n",
        "def cv2_imshow(img):\n",
        "    _, ret = cv2.imencode('.png', img) \n",
        "    img_ip = IPython.display.Image(data=ret)\n",
        "    IPython.display.display(img_ip)\n",
        "\n",
        "cap = cv2.VideoCapture(mnist_dream_path) \n",
        "vw = None\n",
        "frame = -1 # counter for debugging (mostly), 0-indexed\n",
        "\n",
        "# go through all the frames and run our classifier on the high res MNIST images as they morph from number to number\n",
        "while True: # should 481 frames\n",
        "    frame += 1\n",
        "    ret, img = cap.read()\n",
        "    if not ret: break\n",
        "               \n",
        "    assert img.shape[0] == img.shape[1] # should be a square\n",
        "    if img.shape[0] != 720:\n",
        "        img = cv2.resize(img, (720, 720))\n",
        "       \n",
        "    #preprocess the image for prediction\n",
        "    img_proc = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)\n",
        "    img_proc = cv2.resize(img_proc, (28, 28))\n",
        "    img_proc = preprocess_images(img_proc)\n",
        "    img_proc = 1 - img_proc # inverse since training dataset is white text with black background\n",
        "    net_in = np.expand_dims(img_proc, axis=0) # expand dimension to specify batch size of 1\n",
        "    \n",
        "    model = build_model()\n",
        "    \n",
        "    preds = model.predict(net_in)[0]\n",
        "    guess = np.argmax(preds)\n",
        "    perc = np.rint(preds * 100).astype(int)\n",
        "    \n",
        "    img = 255 - img\n",
        "    pad_color = 0\n",
        "    img = np.pad(img, ((0,0), (0,1280-720), (0,0)), mode='constant', constant_values=(pad_color))  \n",
        "    \n",
        "    line_type = cv2.LINE_AA\n",
        "    font_face = cv2.FONT_HERSHEY_SIMPLEX\n",
        "    font_scale = 1.3        \n",
        "    thickness = 2\n",
        "    x, y = 740, 60\n",
        "    color = (255, 255, 255)\n",
        "    \n",
        "    text = \"Neural Network Output:\"\n",
        "    cv2.putText(img, text=text, org=(x, y), fontScale=font_scale, fontFace=font_face, thickness=thickness,\n",
        "                    color=color, lineType=line_type)\n",
        "    \n",
        "    text = \"Input:\"\n",
        "    cv2.putText(img, text=text, org=(30, y), fontScale=font_scale, fontFace=font_face, thickness=thickness,\n",
        "                    color=color, lineType=line_type)   \n",
        "        \n",
        "    y = 130\n",
        "    for i, p in enumerate(perc):\n",
        "        if i == guess: color = (255, 218, 158)\n",
        "        else: color = (100, 100, 100)\n",
        "            \n",
        "        rect_width = 0\n",
        "        if p > 0: rect_width = int(p * 3.3)\n",
        "        \n",
        "        rect_start = 180\n",
        "        cv2.rectangle(img, (x+rect_start, y-5), (x+rect_start+rect_width, y-20), color, -1)\n",
        "\n",
        "        text = '{}: {:>3}%'.format(i, int(p))\n",
        "        cv2.putText(img, text=text, org=(x, y), fontScale=font_scale, fontFace=font_face, thickness=thickness,\n",
        "                    color=color, lineType=line_type)\n",
        "        y += 60\n",
        "    \n",
        "    # if you don't want to save the output as a video, set this to False\n",
        "    save_video = True\n",
        "    \n",
        "    if save_video:\n",
        "        if vw is None:\n",
        "            codec = cv2.VideoWriter_fourcc(*'DIVX')\n",
        "            vid_width_height = img.shape[1], img.shape[0]\n",
        "            vw = cv2.VideoWriter(mnist_prediction_path, codec, 30, vid_width_height)\n",
        "        # 15 fps above doesn't work robustly so we right frame twice at 30 fps\n",
        "        vw.write(img)\n",
        "        vw.write(img)\n",
        "    \n",
        "    # scale down image for display\n",
        "    img_disp = cv2.resize(img, (0,0), fx=0.5, fy=0.5)\n",
        "    cv2_imshow(img_disp)\n",
        "    IPython.display.clear_output(wait=True)\n",
        "        \n",
        "cap.release()\n",
        "if vw is not None:\n",
        "    vw.release()"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "downloading the sample video...\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "3vst9mHRGSB7",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The above shows the prediction of the network by choosing the neuron with the highest output. While the output layer values add 1 to one, these do not reflect well-calibrated measures of \"uncertainty\". Often, the network is overly confident about the top choice that does not reflect a learned measure of probability. If everything ran correctly you should get an animation like this:\n",
        "\n",
        "![MNIST dream predictions](https://i.imgur.com/eMF9FOG.gif)"
      ]
    },
    {
      "metadata": {
        "id": "Cb6C3h1PGXIr",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Acknowledgements\n",
        "\n",
        "The contents of this tutorial is based on and inspired by the work of [TensorFlow team](https://www.tensorflow.org) (see their [Colab notebooks](https://www.tensorflow.org/tutorials/)), our [MIT Human-Centered AI team](https://hcai.mit.edu), and individual pieces referenced in the [MIT Deep Learning](https://deeplearning.mit.edu) course slides."
      ]
    },
    {
      "metadata": {
        "id": "2F-ygZqMGPDE",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}