chapter_02/nb_ch02_02a.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tbTYFRhJoaBu"
   },
   "source": [
    "# MNIST digit classification with a fully connected network (fcNN)\n",
    "\n",
    "\n",
    "**Goal:** In this notebook you will see how to use a fully connected networks (fcNN) in an classification task for images.\n",
    "\n",
    "**Usage:** The idea of the notebook is that you try to understand the provided code by running it, checking the output and playing with it by slightly changing the code and rerunning it. \n",
    "\n",
    "**Dataset:** You work with the MNIST dataset. We have 60'000 28x28 pixel greyscale images of digits and want to classify them into the right label (0-9).\n",
    "\n",
    "**Content:**\n",
    "* load the MNIST data \n",
    "* transform the labels into the one hot encoding\n",
    "* visualize samples of the data\n",
    "* flatten the 2D images into a 1D vector\n",
    "* use keras to train a fcNN and look at the perfomance on new unseen test data\n",
    "* use different activation functions and more hidden layers\n",
    "\n",
    "| [open in colab](https://colab.research.google.com/github/tensorchiefs/dl_book/blob/master/chapter_02/nb_ch02_02a.ipynb)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "dpZJ7RaO4io_"
   },
   "source": [
    "#### Install correct TF version (colab only)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "PEIS4WvpsT5t"
   },
   "source": [
    "#### Imports\n",
    "\n",
    "In the next two cells, we load all the required libraries and functions. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "2sz4dJr01FGF"
   },
   "outputs": [],
   "source": [
    "try: #If running in colab \n",
    "    import google.colab\n",
    "    IN_COLAB = True \n",
    "    %tensorflow_version 2.x\n",
    "except:\n",
    "    IN_COLAB = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "JjmQOOt-1FGL",
    "outputId": "f42a19cb-c100-481d-f936-a1d4bf9da7e8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensorflow version:  2.0.0  running in colab?:  False\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "if (not tf.__version__.startswith('2')): #Checking if tf 2.0 is installed\n",
    "    print('Please install tensorflow 2.0 to run this notebook')\n",
    "print('Tensorflow version: ',tf.__version__, ' running in colab?: ', IN_COLAB)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Y6S_hQX5oaBw",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# load required libraries:\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "plt.style.use('default')\n",
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "import tensorflow.keras as keras\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Convolution2D, MaxPooling2D, Flatten , Activation\n",
    "from tensorflow.keras.utils import to_categorical \n",
    "from tensorflow.keras import optimizers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4h_3TS0CtJJb"
   },
   "source": [
    "#### Loading and preparing the MNIST data and transfering the labels into the one hot encoding\n",
    "\n",
    "Here we load the MNIST dataset form keras. The 8-bit greyscale images have values form 0 to 255, we divide all values with 255 so that the values are in a range between 0 and 1. In additon we transform the true labels, which are the numbers from 0 to 9 (the digit on the image) into the one hot encoding. We do this to make use of linear algebra in the calculation of the crossentropy loss.  \n",
    "  \n",
    "The one hot encoding transforms the labels into a vector with the same length as we have labels (in our case 10). The resulting vector in the one hot encodig is zero everywhere except for the position of the true label, there it is 1. Let's look at some examples to make it more clear:  \n",
    "\n",
    "0 becomes [1,0,0,0,0,0,0,0,0,0]  \n",
    "1 becomes [0,1,0,0,0,0,0,0,0,0]  \n",
    "2 becomes [0,0,1,0,0,0,0,0,0,0]  \n",
    "...  \n",
    "9 becomes [0,0,0,0,0,0,0,0,0,1]  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "AwZaqsqG1Lba"
   },
   "source": [
    "#### Listing 2.3 Loading the MNIST data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 153
    },
    "colab_type": "code",
    "id": "4sZ8lqFfoaB2",
    "outputId": "1617996c-6da2-4e1c-a037-3cc04f6c1485"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(50000, 28, 28, 1)\n",
      "(10000, 28, 28, 1)\n",
      "(10000, 28, 28, 1)\n",
      "(50000, 10)\n",
      "(10000, 10)\n",
      "(10000, 10)\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.datasets import mnist\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
    "\n",
    "# separate x_train in X_train and X_val, same for y_train\n",
    "X_train=x_train[0:50000] / 255 #divide by 255 so that they are in range 0 to 1\n",
    "Y_train=keras.utils.to_categorical(y_train[0:50000],10) # one-hot encoding\n",
    "\n",
    "X_val=x_train[50000:60000] / 255\n",
    "Y_val=keras.utils.to_categorical(y_train[50000:60000],10)\n",
    "\n",
    "X_test=x_test / 255\n",
    "Y_test=keras.utils.to_categorical(y_test,10)\n",
    "\n",
    "del x_train, y_train, x_test, y_test\n",
    "\n",
    "X_train=np.reshape(X_train, (X_train.shape[0],28,28,1))\n",
    "X_val=np.reshape(X_val, (X_val.shape[0],28,28,1))\n",
    "X_test=np.reshape(X_test, (X_test.shape[0],28,28,1))\n",
    "\n",
    "print(X_train.shape)\n",
    "print(X_val.shape)\n",
    "print(X_test.shape)\n",
    "print(Y_train.shape)\n",
    "print(Y_val.shape)\n",
    "print(Y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "3ly7CrHtLUP9"
   },
   "source": [
    "Let's visualize the first 4 mnist images. It is very easy to recognise the true label of the digits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 285
    },
    "colab_type": "code",
    "id": "c4gUgwGUeftF",
    "outputId": "5127a9ce-80b9-45ce-8620-4e5776bbd895"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAr8AAADACAYAAAAN8gvTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAdE0lEQVR4nO3de7SU1Znn8d8jihptVNQmxAtoD5LGDByjEmMzSgI6tiEL0MQ0y3AZbXEtJUNnjKOxSZqsRCXx0i1KoojclIn2GkSIaSc4oLJMlOESNAp4XUIgR/DGVSNBnvmjXpIDex9Onao6VbXr/X7WehfnPPVedlX9TrHPe979bnN3AQAAAHlwUK0bAAAAAFQLnV8AAADkBp1fAAAA5AadXwAAAOQGnV8AAADkBp1fAAAA5Aad3yozs4FmtqHIdceY2bMlHqfkbYEYsosUkVukiux2nNx3fs3sLTMbXOt21JqZuZntNLMd2TKt1m3CgZHdAjNrMrMVZvZh9m9TrduE1pHbfZnZqOzz9x9r3RYcGNktMLOpZvaKme0xszG1bk8pct/5bYuZHVzrNlRRP3c/Mlv4IE5cHrJrZp0lzZf0kKRjJM2SND+rI0F5yO1eZnaMpJskvVzrtqB8OcruC5KukbSy1g0pVa47v2b2oKSTJf0iO9v5P82sZ/Zb+JVmtl7S4tifHlr+BmhmB5nZjWb2hpm9Z2b/bmZdi2zD3u22m9lqMxsermL3mNlWM1trZoNaPHCUmT1gZs1mttHMfmRmncp7VZACsvtnAyUdLOnf3P1jd58sySR9uYR9oYOR28CtkiZLereMfaAKyO5fuPsUd18k6Y+lbF8Pct35dfeRktZL+mp2tvMnLR4+X9LfSvqvRezqW5KGZdt8RtIHkqYU2Yw3JP0XSUdJ+oGkh8yse4vHv5Ctc5ykf5H0aIsflJmSdkv6T5LOkHShpOgZWzN73MxubKMtS8zsbTN71Mx6Ftl+1ADZ/bPTJb3o+87T/mJWR50ht/s83l/SWZLuLbLdqCGy22DcPdeLpLckDW7xfU9JLunUFrWBkja0tp2kNZIGtXisu6Q/STo4crxgX/s9vkrS0OzrMZL+IMlaPP7/JI2U1E3Sx5IOb/HYCElPtdj22Xa8DudJ6izpaEn3SHop1n6W+lnIrkvS9yQ9vF9tjqSJtX5/WMjtAY7ZSdJySedk3z8t6R9r/d6wkN12vh7PShpT6/ellCUv16eU4vftWLeHpHlmtqdF7RMVArfxQBua2ShJ/0OFHyJJOlKF39r22uhZyjLrVPhtsYekQyQ1m9nexw5qZ7v/zN2XZF/uMrPxkrap8Jvs70rZH2oqT9ndIanLfrUukraXsC/UVp5ye40Kf7F4voRtUX/ylN2GQOe38FtbW/Wdkj6195vsOpnjWzz+e0lXuPuv23NgM+sh6X5JgyQ95+6fmNkqFa5Z3OsEM7MWgT5Z0oLsmB9LOs7dd7fnuEXy/dqB+kN2CwOFrtvvOH1V/J8RUX3ktnD8883s4uz7rpLOMLMmdx9X5r7Rcchug8j1Nb+ZTZJObWOdVyUdZmZfMbNDJE2QdGiLx++VdHMWTpnZ8WY2tIhjH6HCD8072Xb/TdLn9lvnryX9dzM7xMy+rsLZ2P9w92ZJCyXdYWZdsovo/8bMzi/iuPsws9OtcLuoTmZ2pKQ7VPgNdE1794Wqyn12Vfhz8SfZcQ41s70dh8Ul7AvVQW4Lf2b+W0lN2bJchWs4/7mEfaF6yG7h2J3N7DAVOt6HmNlhZpZUfzKpxnaQWyVNMLMtZvad2AruvlWFP1NNU6FTuFNSy9Gcd6nw29VCM9su6XkVLjw/IHdfrUJH8zkVfqj+s6T9fxtcKqmXCqOBb5b0NXd/L3tslArX6a5W4aL5/63C9UMBM3vCzG5qpSndJD2iwqUOb6rwJ5Uh7v6ntp4Dair32XX3XSoMHhklaYukKyQNy+qoT+TWfYu7v713kbRL0rbseaN+5T67mYWSPpJ0rqSp2dfntfUc6onte3kIAAAA0Lg48wsAAIDcoPMLAACA3KDzCwAAgNyg8wsAAIDcKOs+v2Z2kQojFztJmubuk9pYn9F1qIR33f34tldrHdlFLbh72ffObk92yS0qhM9cpCqa3ZLP/GY3bp4i6e8l9ZE0wsz6lN4+oGjrytmY7CJVZBc1wmcuUhXNbjmXPfSX9Lq7v5ndU/NhScXcqBmoNbKLVJFdpIjcoq6U0/k9QfvOC70hq+3DzMaa2XIzW17GsYBKIrtIVZvZJbeoQ3zmoq6Udc1vMdx9qgozgHAND5JCdpEicotUkV1USzlnfjdKOqnF9ydmNaDekV2kiuwiReQWdaWczu8ySb3M7BQz6yzpH1SYrxqod2QXqSK7SBG5RV0p+bIHd99tZuMk/UqFW5dMd/eXK9YyoIOQXaSK7CJF5Bb1xtyrd1kN1/CgQla4+1nVPCDZRSVU4j6/7UFuUSF85iJV0ewywxsAAAByg84vAAAAcoPOLwAAAHKDzi8AAAByg84vAAAAcoPOLwAAAHKDzi8AAAByg84vAAAAcoPOLwAAAHKDzi8AAAByg84vAAAAcoPOLwAAAHLj4Fo3AEDjOvPMM4PauHHjouuOGjUqqM2ePTuo3X333UFt5cqVJbQOAJBHnPkFAABAbtD5BQAAQG7Q+QUAAEBu0PkFAABAbpi7l76x2VuStkv6RNJudz+rjfVLP1jiOnXqFNSOOuqokvfX2qChT33qU0Gtd+/eQe3aa68NarfffntQGzFiRPQ4f/zjH4PapEmTgtoPfvCD6PZlWtFW1tpCdiuvqakpqC1evDiodenSpazjbN26Nagde+yxZe2zWtzdyt1He7JLbuvboEGDgtqcOXOi655//vlB7ZVXXql4m1rBZ25OTJgwIajF/h8/6KDw3OnAgQOj+3zmmWfKblcZotmtxN0evuTu71ZgP0C1kV2kiuwiReQWdYHLHgAAAJAb5XZ+XdJCM1thZmNjK5jZWDNbbmbLyzwWUElkF6k6YHbJLeoUn7moG+Ve9jDA3Tea2V9LetLM1rr7kpYruPtUSVMlruFBXSG7SNUBs0tuUaf4zEXdKKvz6+4bs383m9k8Sf0lLTnwVvXv5JNPDmqdO3cOaueee25QGzBgQHSfRx99dFC79NJLS2hd+23YsCGoTZ48OagNHz48qG3fvj26zxdeeCGo1fii9nZp1OxWS//+/YPa3Llzg1psUGdrg2xjWdu1a1dQiw1uO+ecc4Jaa7O+xfaZklpn97zzzgtqsfdk3rx51WhO0s4+++ygtmzZshq0pOPVOrcIjRkzJqjdcMMNQW3Pnj1F7a+cGyhUW8mXPZjZEWb2V3u/lnShpJcq1TCgo5BdpIrsIkXkFvWmnDO/3STNM7O9+/lf7v5/KtIqoGORXaSK7CJF5BZ1peTOr7u/KalfBdsCVAXZRarILlJEblFvuNUZAAAAcqMSk1wkKzYjlRSflaqc2diqKXZhemzGlh07dgS12MxCzc3N0eN88MEHQa2Ksw2hA8RmB5Skz3/+80HtoYceCmrdu3cv6/ivvfZaUPvJT34S1B5++OGg9utf/zqoxXIvSbfeemsJrcNesVmcevXqFdQY8Lav2IxYp5xySlDr0aNHdPvskgGgYmJZO+yww2rQkurjzC8AAAByg84vAAAAcoPOLwAAAHKDzi8AAAByg84vAAAAciPXd3tYv359tP7ee+8FtWrd7WHp0qVBbcuWLUHtS1/6UnT72NStDz74YPkNQ8O77777ovURI0ZU5fixu0oceeSRQS02jXbsDgR9+/atSLuwr1GjRgW15557rgYtSUvsbihXXXVVUIvdSUWS1q5dW/E2IR8GDx4crX/rW98qavtY9oYMGRLUNm3a1L6G1RBnfgEAAJAbdH4BAACQG3R+AQAAkBt0fgEAAJAbuR7w9v7770fr119/fVCLXdz929/+NqhNnjy56OOvWrUqqF1wwQVBbefOnUHt9NNPj+5z/PjxRR8f+XXmmWcGta985SvRdYudVjU2EO0Xv/hFULv99tuj2//hD38IarGfsdjU2l/+8peDGtPBdozYNL1o27Rp04paLzbNN1CsAQMGBLUZM2ZE1y12IP9tt90W1NatW9e+htUZPsUAAACQG3R+AQAAkBt0fgEAAJAbdH4BAACQG20OeDOz6ZKGSNrs7p/Lal0lPSKpp6S3JF3m7uEolEQ99thjQW3x4sVBbfv27UGtX79+0X1eeeWVQS028Cc2uC3m5ZdfjtbHjh1b1PZ5kMfsxjQ1NQW1J598Mqh16dIlur27B7UnnngiqMVmgjv//POD2oQJE6LHiQ0Ieuedd4LaCy+8ENT27NkT1FobwBebSW7lypXRdWulXrIbmyWvW7duHXnIhlXs4KLYz2Yq6iW3eTZ69Oig9pnPfKbo7Z9++umgNnv27HKaVJeKOfM7U9JF+9VulLTI3XtJWpR9D9SbmSK7SNNMkV2kZ6bILRLQZufX3ZdI2v+eYEMlzcq+niVpWIXbBZSN7CJVZBcpIrdIRan3+e3m7s3Z129LavXvYGY2VhJ/i0e9ILtIVVHZJbeoM3zmou6UPcmFu7uZhRcF/uXxqZKmStKB1gOqjewiVQfKLrlFveIzF/Wi1M7vJjPr7u7NZtZd0uZKNqoebdu2raj1tm7dWvQ+r7rqqqD2yCOPBLXYYB6UrKGze9pppwW12IyFscE37777bnSfzc3NQW3WrFlBbceOHUHtl7/8ZVG1jnD44YdH69ddd11Qu/zyyzu6OZVQ9exefPHFQa211xV/ERsUeMoppxS17caNGyvdnFpr6M/cWjruuOOC2hVXXBHUWutDbNmyJaj96Ec/Kr9hCSj1VmcLJO0dUjha0vzKNAfocGQXqSK7SBG5Rd1ps/NrZj+X9Jyk3ma2wcyulDRJ0gVm9pqkwdn3QF0hu0gV2UWKyC1S0eZlD+4e3ryzYFCF2wJUFNlFqsguUkRukQpmeAMAAEBu0PkFAABAbpR9qzPsa+LEidH6mWeeGdRiU78OHjw4qC1cuLDsdqGxHHroodF6bMrs2Ij92NTco0aNiu5z+fLlQS3lEf8nn3xyrZuQjN69exe1XmvTredV7OcwdgeIV199NajFfjaBnj17BrW5c+eWtc+77747qD311FNl7TMVnPkFAABAbtD5BQAAQG7Q+QUAAEBu0PkFAABAbjDgrcJ27twZrcemMl65cmVQu//++4Na7AL02CAkSZoyZUpQc2eK9EZzxhlnROuxwW0xQ4cODWrPPPNMWW1Cfi1btqzWTaioLl26BLWLLroouu43v/nNoHbhhRcWdZwf/vCHQS025SwQy1/fvn2L2nbRokXR+l133VVWm1LGmV8AAADkBp1fAAAA5AadXwAAAOQGnV8AAADkBgPequSNN94IamPGjAlqM2bMCGojR44sqiZJRxxxRFCbPXt2UGtubo5ujzTceeed0bqZBbXYQLZGG9x20EHh7/F79uypQUvyqWvXrhXfZ79+/YJaLN+xWTEl6cQTTwxqnTt3DmqXX355UIvl6aOPPooeZ+nSpUHt448/DmoHHxz+d7tixYroPpFvw4YNC2qTJk0qattnn302qI0ePTq67tatW9vXsAbCmV8AAADkBp1fAAAA5AadXwAAAOQGnV8AAADkRpsD3sxsuqQhkja7++ey2kRJV0l6J1vtJnf/j45qZKOaN29eUHvttdeCWmxw06BBg6L7vOWWW4Jajx49gtrNN98c1DZu3BjdZ6oaJbtDhgwJak1NTdF1Y7P5LViwoOJtqjexwW2tzWy4atWqjm5O2eolu7FBXrHX9d577w1qN910U1nHjs1eFRvwtnv37uj2H374YVBbvXp1UJs+fXpQi82g2dog0U2bNgW1DRs2BLXDDz88qK1duza6z1TVS25T0bNnz2h97ty5Je/zzTffDGqxjOZdMWd+Z0qKzev4r+7elC0EGfVopsgu0jRTZBfpmSlyiwS02fl19yWS3q9CW4CKIrtIFdlFisgtUlHONb/jzOxFM5tuZse0tpKZjTWz5WYW/h0JqA2yi1S1mV1yizrEZy7qSqmd359J+htJTZKaJd3R2oruPtXdz3L3s0o8FlBJZBepKiq75BZ1hs9c1J2SZnhz9z9fPW1m90t6vGItyrmXXnopqF122WVB7atf/Wp0+9gMcVdffXVQ69WrV1C74IILimli0lLMbmygTGymKknavHlzUHvkkUcq3qZqOfTQQ4PaxIkTi9p28eLF0fp3v/vdcppUM7XI7jXXXBPU1q1bF9TOPffcih97/fr1Qe2xxx4LamvWrIlu//zzz1e8TTFjx44Nascff3xQiw1EyoMUP3Or5YYbbojWy5mdstiZ4PKupDO/Zta9xbfDJYU9NqAOkV2kiuwiReQW9aiYW539XNJASceZ2QZJ/yJpoJk1SXJJb0kKTy0CNUZ2kSqyixSRW6Sizc6vu4+IlB/ogLYAFUV2kSqyixSRW6SCGd4AAACQG3R+AQAAkBsl3e0B1bVly5ag9uCDD0bXnTZtWlA7+ODwbT7vvPOC2sCBA4Pa008/3XYDUTc+/vjjoNbc3FyDlrRf7M4OEyZMCGrXX399UItNJ3vHHfE7Ku3YsaOE1mGvH//4x7VuQl1pbar5/ZUzZS3SF5uS/sILLyxrn/Pnzw9qr7zySln7zAvO/AIAACA36PwCAAAgN+j8AgAAIDfo/AIAACA3GPBWZ/r27RvUvva1rwW1s88+O7p9bHBbzOrVq4PakiVLitoW9WvBggW1bkKbYgM/pPhAtm984xtBLTbI49JLLy2/YUAHmjdvXq2bgBpauHBhUDvmmGOK3j42XfeYMWPKaVKuceYXAAAAuUHnFwAAALlB5xcAAAC5QecXAAAAucGAtyrp3bt3UBs3blxQu+SSS4Lapz/96bKO/cknnwS12Kxfe/bsKes46BhmVlRNkoYNGxbUxo8fX/E2Fevb3/52UPve974XXfeoo44KanPmzAlqo0aNKr9hAFBFxx57bFBrz/+5P/3pT4Mas1WWjjO/AAAAyA06vwAAAMgNOr8AAADIDTq/AAAAyI02B7yZ2UmSZkvqJsklTXX3u8ysq6RHJPWU9Jaky9z9g45rav2JDUQbMWJEdN3Y4LaePXtWuklavnx5ULv55puDWgozgZWrUbLr7kXVpHgmJ0+eHNSmT58e1N57773oPs8555ygNnLkyKDWr1+/oHbiiScGtfXr10eP86tf/SqoxQZ5NLpGyW2exQaknnbaaUEtNmtXyshuwYwZM4LaQQeVd67xN7/5TVnbY1/FvBu7JV3n7n0knSPpWjPrI+lGSYvcvZekRdn3QD0hu0gRuUWqyC6S0Gbn192b3X1l9vV2SWsknSBpqKRZ2WqzJIX3WAJqiOwiReQWqSK7SEW77vNrZj0lnSFpqaRu7r73ZrFvq/Bnjtg2YyWNLb2JQPnILlJEbpEqsot6VvRFKGZ2pKS5kv7J3be1fMwLFyBGL0J096nufpa7n1VWS4ESkV2kiNwiVWQX9a6oM79mdogKQZ7j7o9m5U1m1t3dm82su6TNHdXIauvWLfyltE+fPkHtnnvuCWqf/exnK96epUuXBrXbbrstuu78+fODWp5nbstbdjt16hTUrrnmmqB26aWXBrVt27YFNUnq1atXye2JDdJ46qmnout+//vfL/k4jSZvuW00sQGp5Q54SkXestvU1BTUBg8eHNRi/w/v2rUrus8pU6YEtU2bNpXQOrSmzZ9GKwxbfUDSGne/s8VDCySNzr4eLSnsdQE1RHaRInKLVJFdpKKYM79/J2mkpN+Z2aqsdpOkSZL+3cyulLRO0mUd00SgZGQXKSK3SBXZRRLa7Py6+7OSwpsWFgyqbHOAyiG7SBG5RarILlKRj4uQAAAAANH5BQAAQI606z6/KevatWtQu++++6LrxkZvnnrqqRVvU2wk/B133BHUYtO+fvTRRxVvD+rTc889F9SWLVsWXffss88uap+xaZBjdzlpTWwq5IcffjiojR8/vuh9Ao3si1/8YlCbOXNm9RuCijr66KODWuzzNWbjxo3R+ne+852y2oS2ceYXAAAAuUHnFwAAALlB5xcAAAC5QecXAAAAuZH8gLcvfOELQe36668Pav379w9qJ5xwQsXb8+GHH0brkydPDmq33HJLUNu5c2fF24S0bdiwIahdcskl0XWvvvrqoDZhwoSyjn/XXXcFtZ/97GdB7fXXXy/rOECjKEx0BqBeceYXAAAAuUHnFwAAALlB5xcAAAC5QecXAAAAuZH8gLfhw4cXVWuP1atXB7XHH388qO3evTuoxWZok6QtW7aU1Sagpebm5mh94sSJRdUAVMYTTzwR1L7+9a/XoCWohbVr1wa12OytAwYMqEZzUCTO/AIAACA36PwCAAAgN+j8AgAAIDfo/AIAACA/3P2Ai6STJD0labWklyWNz+oTJW2UtCpbLi5iX87CUoFleVtZI7ss9biQW5ZEFz5zWVJdotkt5m4PuyVd5+4rzeyvJK0wsyezx/7V3W8vYh9ALZBdpIjcIlVkF0los/Pr7s2SmrOvt5vZGkkndHTDgHKRXaSI3CJVZBepaNc1v2bWU9IZkpZmpXFm9qKZTTezY1rZZqyZLTez5WW1FCgD2UWKyC1SRXZR14q5jie7/uZISSskXZJ9301SJxU60DdLms41PCxVWoq6/ozsstTbQm5ZEl34zGVJdYlmt6gzv2Z2iKS5kua4+6OS5O6b3P0Td98j6X5J/YvZF1BNZBcpIrdIFdlFCtrs/JqZSXpA0hp3v7NFvXuL1YZLeqnyzQNKR3aRInKLVJFdpKKYuz38naSRkn5nZquy2k2SRphZkwqnld+SdHWHtBAoHdlFisgtUkV2kQTLrq2pzsHMqncwNLIV7n5WNQ9IdlEJ7m7VPB65RYXwmYtURbPLDG8AAADIDTq/AAAAyA06vwAAAMgNOr8AAADIDTq/AAAAyA06vwAAAMgNOr8AAADIjWImuaikdyWty74+Lvu+UfB8qqdHDY65N7v1/LqUgudTPbXMrVTfr00peD7VQ3Yri+dTPdHsVnWSi30ObLa82jfN7kg8n3xotNeF55Mfjfba8Hzyo9FeG55P7XHZAwAAAHKDzi8AAAByo5ad36k1PHZH4PnkQ6O9Ljyf/Gi014bnkx+N9trwfGqsZtf8AgAAANXGZQ8AAADIDTq/AAAAyI2qd37N7CIze8XMXjezG6t9/HKZ2XQz22xmL7WodTWzJ83stezfY2rZxvYws5PM7CkzW21mL5vZ+Kye7HPqKGS3vpDd4qSeW6mxsktui5d6dhspt1JjZbeqnV8z6yRpiqS/l9RH0ggz61PNNlTATEkX7Ve7UdIid+8laVH2fSp2S7rO3ftIOkfStdl7kvJzqjiyW5fIbhsaJLdSY2WX3BahQbI7U42TW6mBslvtM7/9Jb3u7m+6+y5JD0saWuU2lMXdl0h6f7/yUEmzsq9nSRpW1UaVwd2b3X1l9vV2SWsknaCEn1MHIbt1huwWJfncSo2VXXJbtOSz20i5lRoru9Xu/J4g6fctvt+Q1VLXzd2bs6/fltStlo0plZn1lHSGpKVqkOdUQWS3jpHdVjVqbqUGeJ/J7QE1anYb4n1OPbsMeKswL9w7Lrn7x5nZkZLmSvond9/W8rFUnxPaJ9X3mewixfeZ3CLV97kRslvtzu9GSSe1+P7ErJa6TWbWXZKyfzfXuD3tYmaHqBDkOe7+aFZO+jl1ALJbh8humxo1t1LC7zO5LUqjZjfp97lRslvtzu8ySb3M7BQz6yzpHyQtqHIbOsICSaOzr0dLml/DtrSLmZmkByStcfc7WzyU7HPqIGS3zpDdojRqbqVE32dyW7RGzW6y73NDZdfdq7pIuljSq5LekPTP1T5+Bdr/c0nNkv6kwjVIV0o6VoURjq9J+r+Suta6ne14PgNU+BPFi5JWZcvFKT+nDnytyG4dLWS36Ncp6dxmz6Fhsktu2/VaJZ3dRspt9nwaJrtMbwwAAIDcYMAbAAAAcoPOLwAAAHKDzi8AAAByg84vAAAAcoPOLwAAAHKDzi8AAAByg84vAAAAcuP/AzBFXZqmNDqtAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 864x864 with 4 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# visualize the 4 first mnist images before shuffling the pixels\n",
    "plt.figure(figsize=(12,12))\n",
    "for i in range(0,4):\n",
    "    plt.subplot(1,4,(i+1))\n",
    "    plt.imshow((X_train[i,:,:,0]),cmap=\"gray\")\n",
    "    plt.title('true label: '+np.str(np.argmax(Y_train,axis=1)[i]))\n",
    "    #plt.axis('off')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "trSndjWX4HO7"
   },
   "source": [
    "## fcNN as classification model for MNIST data\n",
    "Now we want to train a fcNN to classify the MNIST data.\n",
    "* we use a fcNN with 2 hidden layers and use the sigmoid activation function\n",
    "* train it on train data and check the performance on the test data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LB_m8HJaOMsY"
   },
   "source": [
    "\n",
    "#### Flatten the the images into vectors\n",
    "\n",
    "Because we will use fcNN our input cannot be matrices or tensors. We need to flatten our inuput into a 1d vector. We do this in the next cell with reshap and look at the resulting shape of the flattened data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 85
    },
    "colab_type": "code",
    "id": "CoSUhNzRfbj4",
    "outputId": "1f88f5ba-8c4d-449d-a7df-0bde6d747fbf"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(50000, 784)\n",
      "(50000, 10)\n",
      "(10000, 784)\n",
      "(10000, 10)\n"
     ]
    }
   ],
   "source": [
    "# prepare data for fcNN - we need a vector as input\n",
    "\n",
    "X_train_flat = X_train.reshape([X_train.shape[0], 784])\n",
    "X_val_flat = X_val.reshape([X_val.shape[0], 784])\n",
    "X_test_flat = X_test.reshape([X_test.shape[0], 784])\n",
    "\n",
    "# check the shape\n",
    "print(X_train_flat.shape)\n",
    "print(Y_train.shape)\n",
    "print(X_val_flat.shape)\n",
    "print(Y_val.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "uOYYJFOh62qC"
   },
   "source": [
    "### Train the fcNN "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "bulk0OpFOpeO"
   },
   "source": [
    "Here we define the nework, we use two hidden layers with 100 and 50 nodes. In the output we predict the probability for the 10 digits with the softmax actication function, in the hidden layers we use the sigmoid activation function and our loss is the categroical crossentropy loss."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "l_eV6OGV1RaC"
   },
   "source": [
    "#### Listing 2.4 Definition of an fcNN  for the MNIST data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "EAispBe0oaCH"
   },
   "outputs": [],
   "source": [
    "# define fcNN with 2 hidden layers\n",
    "model = Sequential()\n",
    "\n",
    "model.add(Dense(100, batch_input_shape=(None, 784)))\n",
    "model.add(Activation('sigmoid'))\n",
    "model.add(Dense(50))\n",
    "model.add(Activation('sigmoid'))\n",
    "model.add(Dense(10))\n",
    "model.add(Activation('softmax'))\n",
    "\n",
    "# compile model and intitialize weights\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='adam',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 357
    },
    "colab_type": "code",
    "id": "7xfW6yzqoaCK",
    "outputId": "9269228e-0a5f-4245-c79b-6b3d6e6e3885"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense (Dense)                (None, 100)               78500     \n",
      "_________________________________________________________________\n",
      "activation (Activation)      (None, 100)               0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 50)                5050      \n",
      "_________________________________________________________________\n",
      "activation_1 (Activation)    (None, 50)                0         \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 10)                510       \n",
      "_________________________________________________________________\n",
      "activation_2 (Activation)    (None, 10)                0         \n",
      "=================================================================\n",
      "Total params: 84,060\n",
      "Trainable params: 84,060\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# summarize model along with number of model weights\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 374
    },
    "colab_type": "code",
    "id": "mhvTpCtfoaCQ",
    "outputId": "45afbc77-c753-4d47-a0bb-e35cdc470e62"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 50000 samples, validate on 10000 samples\n",
      "Epoch 1/10\n",
      "50000/50000 - 2s - loss: 1.0284 - accuracy: 0.7669 - val_loss: 0.4042 - val_accuracy: 0.9065\n",
      "Epoch 2/10\n",
      "50000/50000 - 1s - loss: 0.3373 - accuracy: 0.9115 - val_loss: 0.2546 - val_accuracy: 0.9312\n",
      "Epoch 3/10\n",
      "50000/50000 - 1s - loss: 0.2428 - accuracy: 0.9320 - val_loss: 0.2057 - val_accuracy: 0.9410\n",
      "Epoch 4/10\n",
      "50000/50000 - 1s - loss: 0.1957 - accuracy: 0.9442 - val_loss: 0.1731 - val_accuracy: 0.9515\n",
      "Epoch 5/10\n",
      "50000/50000 - 1s - loss: 0.1644 - accuracy: 0.9535 - val_loss: 0.1533 - val_accuracy: 0.9560\n",
      "Epoch 6/10\n",
      "50000/50000 - 1s - loss: 0.1407 - accuracy: 0.9591 - val_loss: 0.1379 - val_accuracy: 0.9610\n",
      "Epoch 7/10\n",
      "50000/50000 - 1s - loss: 0.1215 - accuracy: 0.9654 - val_loss: 0.1273 - val_accuracy: 0.9633\n",
      "Epoch 8/10\n",
      "50000/50000 - 1s - loss: 0.1056 - accuracy: 0.9702 - val_loss: 0.1145 - val_accuracy: 0.9679\n",
      "Epoch 9/10\n",
      "50000/50000 - 1s - loss: 0.0923 - accuracy: 0.9743 - val_loss: 0.1080 - val_accuracy: 0.9684\n",
      "Epoch 10/10\n",
      "50000/50000 - 1s - loss: 0.0818 - accuracy: 0.9771 - val_loss: 0.1013 - val_accuracy: 0.9697\n"
     ]
    }
   ],
   "source": [
    "# train the model\n",
    "history=model.fit(X_train_flat, Y_train, \n",
    "                  batch_size=128, \n",
    "                  epochs=10,\n",
    "                  verbose=2, \n",
    "                  validation_data=(X_val_flat, Y_val)\n",
    "                 )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HvOuOKc01bqS"
   },
   "source": [
    "In the next cell we plot the accuray and loss of the train and validation vs the number of train eprochs to see how the development"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 427
    },
    "colab_type": "code",
    "id": "t39HSiHcoaCT",
    "outputId": "90e35d4f-e8db-40c7-fd78-80149d2864b9"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f6edc3a3908>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtcAAAEWCAYAAACt0rvRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdd3zV5d3/8dcne28gkEHCkA0BIm7FDS7q3lbv1lmr3dXe/bXe3h222va2rdWqta2tqBSrpS2KoiBaUSEMGcomJMwAmZCQca7fH98DBkQ4CefkZLyfj0cenPNd1+fQ+s2b61zf6zLnHCIiIiIicuwiwl2AiIiIiEh3oXAtIiIiIhIkCtciIiIiIkGicC0iIiIiEiQK1yIiIiIiQaJwLSIiIiISJArX0m2Y2Z/M7EcBHrvRzM4JdU0iIhIawbrnt+U6IoFQuBYRERERCRKFa5FOxsyiwl2DiIiItI/CtXQo/1dz3zazj8xsj5n9wcz6mNmrZlZrZrPNLL3V8ZeY2QozqzKzuWY2rNW+sWa2yH/ei0DcIW1dZGZL/Oe+Z2ajA6zxQjNbbGY1ZlZmZg8csv9U//Wq/Ptv9m+PN7NfmFmpmVWb2bv+bRPNrPwwfw/n+F8/YGbTzeyvZlYD3GxmE8xsvr+NrWb2WzOLaXX+CDN7w8x2m9l2M/uemWWb2V4zy2x13DgzqzCz6EA+u4hIMHWFe/5har7VzNb6768zzKyff7uZ2a/MbIf/98MyMxvp33eBma3017bZzL7Vrr8w6RYUriUcLgfOBY4DLgZeBb4H9ML7/+Q9AGZ2HPA88DX/vpnAP80sxh80XwH+AmQAf/NfF/+5Y4FngNuBTOD3wAwziw2gvj3ATUAacCFwp5l9wX/d/v56f+OvqQhY4j/vEWA8cLK/pu8AvgD/TqYA0/1tPge0AF8HsoCTgLOBu/w1JAOzgdeAfsAg4E3n3DZgLnBVq+veCLzgnGsKsA4RkWDr7Pf8A8zsLOCnePfRvkAp8IJ/93nA6f7Pkeo/Zpd/3x+A251zycBI4K22tCvdi8K1hMNvnHPbnXObgXeAD5xzi51zDcDLwFj/cVcD/3bOveEPh48A8Xjh9UQgGvg/51yTc246sKBVG7cBv3fOfeCca3HO/RnY5z/viJxzc51zy5xzPufcR3g3+zP8u68DZjvnnve3u8s5t8TMIoD/Au51zm32t/mec25fgH8n851zr/jbrHfOlTjn3nfONTvnNuL9othfw0XANufcL5xzDc65WufcB/59fwZuADCzSOBavF9GIiLh0qnv+Ye4HnjGObfIf/++HzjJzAqAJiAZGAqYc+5j59xW/3lNwHAzS3HOVTrnFrWxXelGFK4lHLa3el1/mPdJ/tf98HoNAHDO+YAyIMe/b7NzzrU6t7TV6/7AN/1fD1aZWRWQ5z/viMzsBDOb4x9OUQ3cgdeDjP8a6w5zWhbeV5SH2xeIskNqOM7M/mVm2/xDRX4SQA0A/8C7wRfi9RRVO+c+bGdNIiLB0Knv+Yc4tIY6vN7pHOfcW8BvgceAHWb2pJml+A+9HLgAKDWzt83spDa2K92IwrV0ZlvwbpiAN94N72a5GdgK5Pi37Zff6nUZ8GPnXFqrnwTn3PMBtDsVmAHkOedSgSeA/e2UAQMPc85OoOFz9u0BElp9jki8rzxbc4e8fxz4BBjsnEvB+wq1dQ0DDle4vydoGl7v9Y2o11pEuo5w3fOPVEMi3jCTzQDOuV8758YDw/GGh3zbv32Bc24K0Btv+Mq0NrYr3YjCtXRm04ALzexs/wN538T7mu89YD7QDNxjZtFmdhkwodW5TwF3+HuhzcwSzXtQMTmAdpOB3c65BjObgDcUZL/ngHPM7CozizKzTDMr8vewPAP80sz6mVmkmZ3kH++3Gojztx8NfB842jjAZKAGqDOzocCdrfb9C+hrZl8zs1gzSzazE1rtfxa4GbgEhWsR6TrCdc9v7XngFjMr8t+/f4I3jGWjmR3vv340XqdJA+Dzjwm/3sxS/cNZagj8eRvphhSupdNyzq3C64H9DV7P8MXAxc65RudcI3AZXojcjTdW7++tzl0I3Ir3FV4lsNZ/bCDuAh40s1rgB7TqgXDObcL76u+b/naXAGP8u78FLMMbB7gb+BkQ4Zyr9l/zabzejz3AQbOHHMa38EJ9Ld4vjRdb1VCLN+TjYmAbsAY4s9X+/+Dd2Bc551p/bSoi0mmF8Z7fuobZwP8DXsLrLR8IXOPfnYJ3P67EGzqyC3jYv+9GYKN/GN8deGO3pYeyg4cviUh3YGZvAVOdc0+HuxYREZGeROFapJsxs+OBN/DGjNeGux4REZGeRMNCRLoRM/sz3hzYX1OwFhER6XjquRYRERERCRL1XIuIiIiIBElUuAsIlqysLFdQUBDuMkRE2qWkpGSnc+7Q+c+7Nd23RaSrOtI9u9uE64KCAhYuXBjuMkRE2sXMety0ibpvi0hXdaR7toaFiIiIiIgEicK1iIiIiEiQKFyLiIiIiARJtxlzLSIiIiKh19TURHl5OQ0NDeEuJeTi4uLIzc0lOjo64HMUrkVEREQkYOXl5SQnJ1NQUICZhbuckHHOsWvXLsrLyyksLAz4PA0LEREREZGANTQ0kJmZ2a2DNYCZkZmZ2eYeeoVrEREREWmT7h6s92vP51S4FhFph4amFhqbfQBsqapn2sIyqvc2hbkqEREJN425FhHBG1vncxAZYeyq28eHG3azs24fO+sa2Vm3j13+P/e/rt3XzFM3FXPu8D6s2lbLd6Z/xKC7khiXnx7uj9Jt7ahp4O6pi/mvUwuZNDI73OWISJhUVVUxdepU7rrrrjadd8EFFzB16lTS0tJCVJlH4VpEuq3mFh+79zYeCMb7/6zwvz5/RDbnDu/Dhp17OP9X83jkqjFcMqYfa3bUcedziwAwg/SEGLKSYshMjGVkTipZSbFkJcVQmJUIwAkDMnjnO2eSnRoXzo/b7aUnxrC0vIoFG3crXIv0YFVVVfzud7/7TLhubm4mKurzo+3MmTNDXRqgcC0iXZTP52hobiEhJooWn+P389YxKieV0wb3YkdNA5MffYfdextx7rPnRkcaWUmxjMpJBSArKYZbTi1ggD8sj8pJ5dV7TyMzKYaMhBiiIo88gi4hJoqEDN1OQy06MoJROaksLasKdykiEkb33Xcf69ato6ioiOjoaOLi4khPT+eTTz5h9erVfOELX6CsrIyGhgbuvfdebrvtNgAKCgpYuHAhdXV1TJ48mVNPPZX33nuPnJwc/vGPfxAfHx+U+vTbQEQ6peYWH9tqGthcWc/mqnrKK+vZXFlPedVeNlfWs6Wqgcmjsnn0mrFERhiPz13HdRPyOW1wL1Liozl/ZDZZiTFkJceSlRRL5v7XibGkxEcd9JBKclw0908eduB9YmwUw/qmhONjd0pm9gxwEbDDOTfyMPsNeBS4ANgL3OycWxSKWsbkpfHX90tpavERfZR/9IhIx7j69/OPeszZw3pz2+kDDxx/xfhcrizOY/eeRu78a8lBx754+0lHvNZDDz3E8uXLWbJkCXPnzuXCCy9k+fLlB6bLe+aZZ8jIyKC+vp7jjz+eyy+/nMzMzIOusWbNGp5//nmeeuoprrrqKl566SVuuOGGtnzsz6VwLSJh0djso2pvI71TvKEUf5m/kRaf4+ZTvJvjqT+bw7aag6c/ykqKJTc9nhE5qZw/MpuxeZ+Ob17w3+cQFx0JQFx0JD+5dFTHfJCe4U/Ab4FnP2f/ZGCw/+cE4HH/n0FXlJfGH97dwKpttYz0f/MgIj3bhAkTDpqH+te//jUvv/wyAGVlZaxZs+Yz4bqwsJCioiIAxo8fz8aNG4NWj8K1iIREQ1MLm6v8vc2V9Wyu2vtp73NlPdtrG8hJi+fd754FwJxVFQeF66+cOZDIiAhy0+PJSY8nJy3+QHg+nCPtk2PjnJtnZgVHOGQK8KxzzgHvm1mamfV1zm0Ndi1Fed6DSEvKqhSuRTqJo/U0H+n4jMSYNp9/qMTExAOv586dy+zZs5k/fz4JCQlMnDjxsPNUx8bGHngdGRlJfX39MdXQmsK1iLRbi88RYd48oPNWVzB//S6+O2koAF99fjFvrNx+4NjICKNvahy56fGcMiiL3PR4+mcmHNj/hy8WHzRU48aTCjrsc3yulmZo2gNN9dC01/+n/3Xj3lbbWu1LzYWx14e78o6WA5S1el/u3/aZcG1mtwG3AeTn57e5odz0eDITY1haVsUNJ/ZvX7Ui0qUlJydTW1t72H3V1dWkp6eTkJDAJ598wvvvv9/B1Slci8hROOfYWdfIhp172LCzjvU797C+Yg8bdu5h0669vHvfmfROjmNpWRXPvV/K184ZTGxUJF88qYDJI7PJSYsnNyOBPsmxR3ww0ABamsDX3Oqnxf9z6LajvG9p/GwYPvDnoYHY/+dBYdm/3deOeasHntUTw3XAnHNPAk8CFBcXH+Zx0yMzM8bkpbFEDzWK9FiZmZmccsopjBw5kvj4ePr06XNg36RJk3jiiScYNmwYQ4YM4cQTT+zw+hSuRQTwhnGs3VFHYVYiibFRzF65nd+8tYb1O/dQ29B84LiYyAj6ZyYwICuRc4b1wXw+qCzlzvwy7r6wHHvzB7B7A6dWb4LmRnAtgYVi5wv9h4yKh+h4iE7w/oxJ8F7HJkNSH/++/fsTPj3uoHMSP7ut9bERPXJ4ymYgr9X7XP+2kCjKS2POqh3UNjSRHBcdqmZEpBObOnXqYbfHxsby6quvHnbf/nHVWVlZLF++/MD2b33rW0GtTeFapAdpavFRtnuvvxd6D+sq9nDp2BwmFGawqLSS657+gKlfPoGTB2URGx1BSnw0l43NYUBGDEMTqhlg28ls3ERE5QbYvR7WroeFpdDS+OnNJCoOMgZAWr73OiKq1U/kIe8jjrK/1TaLPPoxrd8fCMH+4BsV57UnoTADuNvMXsB7kLE6FOOt9zt5YCYbduawZ1+LwrWIdDoK1yLdVHV9EzOXbWXdjjo27NzD+p172LR7Ly2+T7+JT0+Iprh/OhMKMxjRJ45nL05j1N73Yf4mTtu9ntNi1sPG9bBkk9cDvV90oheg+wyHYRd5r/f/JGUrxHYzZvY8MBHIMrNy4IdANIBz7glgJt40fGvxpuK7JZT1FBdkUFyQEcomRETaTeFapItrbvERFRlBc4uP70z/iBMHZnJVcR77mlu4/+/LiIuOoCAzkeF9U5gyPJ0R8bsZELWdvs1bSagrheW/hXkbSK0u43RaDYGNTfHCcr+xMPLyQwJ0b2/pQukRnHPXHmW/A77SQeXsb5PKvU1kJMZ0ZLMiIkelcC3SRTjn2FbTwIrNNazYUsPKrdWs2FLD8Oxknrx6CFH1lURsW0pcdDTExdG7vpIlZ24ntWErtns9bFsPa7YcfNH4DC8s558IGdcdHKATMhSgpdP6/ivLeWPldj743tkHzTIjIhJuCtcinYWvBRqqob4S6qugvpLl6zayZesWaisr2Fe7i9jmatLYw6lWx6WRe0mP2EPixlp4yBuy8QhAJfCRd8k0gMTeXlgeMNEfnAs//TM+/XCViHR6k0f2ZWh2Mj4HkcrWItKJKFyLhIpzULMFdq2Bmq3+0PzZH1dfidVXesGag2cmG+n/AaiPSKQ5MZWIxAxik/sRlZjhheODfg7dlgZRsYdWJtLlnTo4i1MHZ4W7DBGRz1C4FjlW++pg11rvZ+caL0zvXAO71nkLkLTiMJpjUtkbmUyVS2JHczxb9vXj/OKziEvJYv4WHyU7HHdOLiYyMZPdvkSS0noRk5ROfKRmRRBprXTXHur2NTOin1ZqFJHPl5SURF1dHVu2bOGee+5h+vTpnzlm4sSJPPLIIxQXFx9zewrXIoHw+aC6zB+c17YK0GuhpvV0vuZNQZc1GPqfwo7YPN6tTGdxdRILK+CTSsM1eDNp9EqOZUROCiP6pXDKKYXEJcVyEtB6EVjNhyDy+e55YQlxURHHvHSyiPQM/fr1O2ywDjaFa5HWGmoOH6B3rYXmhk+Pi02FrEG4glOpSyqErMEk5wxjdVMvvvnyah44bTjj+2dQsmwr33h9EQWZCYzITeWiCSkM7+cF6t7JceH7nCLdwNi8NKYtLKPF54iM0MBrkZ7ivvvuIy8vj698xZuk6IEHHiAqKoo5c+ZQWVlJU1MTP/rRj5gyZcpB523cuJGLLrqI5cuXU19fzy233MLSpUsZOnQo9fX1QatP4Vp6Hl8LVJUeHKD3D+eo2/7pcRYJ6f0hczAMmMje5EJKLYfljX1YujuK1dv3sGpZLdX1Tdw/eSi39xlIak0DaQnR7J9K+syhvVnxP+eTGKv/1ESCbUxeKn96byNrdtQyNDsl3OWI9Eyv3gfblgX3mtmjYPJDn7v76quv5mtf+9qBcD1t2jRmzZrFPffcQ0pKCjt37uTEE0/kkksu+dzZhB5//HESEhL4+OOP+eijjxg3blzQytdvfOmefC3ecI3d62H3Bqjc4I2B3rXW29bS+Omx8elegB50DmQNZl/qQPamFJKeM4S6lgjuem4Rq0tq2VbTAPiArSTHRjEkO5kLR/dlSJ9kTh6YCUCflDj+8qUTDlw6LrpHLoUt0iGK8rzZbpaWVSlci/QgY8eOZceOHWzZsoWKigrS09PJzs7m61//OvPmzSMiIoLNmzezfft2srOzD3uNefPmcc899wAwevRoRo8eHbT6FK6l62pq8Hqg94fn3f4luSs3QNWmgwN0ZAyk9ffGQh93PmQOpjljIJssh4+rY/A5x8Vj+gEw8advcvJAH7+4KobESEdDYwsnD8zkuOxkhmQnM6RPMn1T4zS3rkiYFWQmkBofzZKyKq4+Pj/c5Yj0TEfoYQ6lK6+8kunTp7Nt2zauvvpqnnvuOSoqKigpKSE6OpqCggIaGhqOfqEQULiWzq2h+jDheaP3umYzB01dF5MMGQXQezgMvcibxzm9EDIKqYrqRUlZDau217J6Wy2frKhlfUU1jS2VAAzunXQgXH/7/CFkp3jjoc2MaXfoYSmRzsjMGJOXxpKy6nCXIiId7Oqrr+bWW29l586dvP3220ybNo3evXsTHR3NnDlzKC0tPeL5p59+OlOnTuWss85i+fLlfPTRR0GrTeFawss5qNvhD8/rPxuk63cffHxiL28BlIJTW4Vn/4IoCZkHVhR0zh3oWb7/78t4ccGyA+Og+6XGMSQ7mTOG9GJIH683emCvpANNXDYut0M+uogcu6LcVH47Zy17G5tJiNGvNJGeYsSIEdTW1pKTk0Pfvn25/vrrufjiixk1ahTFxcUMHTr0iOffeeed3HLLLQwbNoxhw4Yxfvz4oNUW0juRmU0CHgUigaedcw8dsr8/8AzQC9gN3OCcK/fvawH2j5Df5Jy7JJS1SgeoLoe1s/3jnv0BunLjwXNBWwSk5nqhefglB4fn9AKITT5qM68s3sxDr37CW986g4SYKMbmp9E7OZZTBmUxtG8yKXGaL1qkuyjKT8PnYPnmGiYUavJKkZ5k2bJPH6TMyspi/vz5hz2urq4OgIKCApYvXw5AfHw8L7zwQkjqClm4NrNI4DHgXKAcWGBmM5xzK1sd9gjwrHPuz2Z2FvBT4Eb/vnrnXFGo6pMOsnMtfDwDPv4nbFnkbYuM9YJyRiEMOOPA0A3SC705oqNiArp0U4uPxZuqeHv1DuauquCHF49gQmEGOenxnDwok7oGryfrquK80H0+EQmrMblpACwpq1S4FpFOIZQ91xOAtc659QBm9gIwBWgdrocD3/C/ngO8EsJ6pCM4B9uXe2H643/CDv//3Dnj4ZwHYMiFkDkIIiLadfmt1fW8vaqCt1dX8O6andTuayYywhifn06Lf9zH8QUZHF+gX7IiPUFmUizT7ziJoX01W4iIdA6hDNc5QFmr9+XACYccsxS4DG/oyKVAspllOud2AXFmthBoBh5yzn0meJvZbcBtAPn5elI8bHw+2FzyaQ915QZveEf+yTD55zD0Qm+oRzs1NLXwyzdW8/aqClZtrwWgb2ocF43pyxnH9eLkQVka6iHSgxXrH9MiHa71s03dmXPu6AcdItxPf3wL+K2Z3QzMAzYDLf59/Z1zm81sAPCWmS1zzq1rfbJz7kngSYDi4uK2f3ppv5Zm2PQerJwBn/wLardCRDQMmAinfh2GXABJvdp9+ekl5dQ2NHHLKYXERkUwa8U28tITuGJ8LmcM6cXg3kk94j9qETm6Tbv2Mr2kjJtOLiArKTbc5Yh0e3FxcezatYvMzMxu/bvYOceuXbuIi2vbisqhDNebgdaDXXP92w5wzm3B67nGzJKAy51zVf59m/1/rjezucBY4KBwLR2seR+sn+v1UH8y05vJIyoeBp8Dwy6BwedBfFqbL1vf2ML763expKyKr597HABzV+1gW3UDt5xSiJnx1jcnanljETmsiroGfjtnLScMyCRrkMK1SKjl5uZSXl5ORUVFuEsJubi4OHJz2/bteyjD9QJgsJkV4oXqa4DrWh9gZlnAbuecD7gfb+YQzCwd2Ouc2+c/5hTg5yGsVT7Pvjpvho+P/wmrZ0FjLcSmwpBJMOxiGHg2xCS06ZLOOdbuqOPt1d7Y6Q827Kax2UdcdAQ3nNifXsmxPHzFGOJjPl3dUMFaRD7PmNw0lj1wPomx4f4yVqRniI6OprCwMNxldFohuxM555rN7G5gFt5UfM8451aY2YPAQufcDGAi8FMzc3jDQr7iP30Y8Hsz8wEReGOuV36mEQmN+kovSK+cAevehOYGSMiCkZd5PdSFpwc8o8d+Pp/jw427+ceSLcxbXcHmqnoABvVO4qYT+3PGkF4cX5BxYLnw1sFaRORIoiIjiIps30PSIiLBFtJ/5jvnZgIzD9n2g1avpwPTD3Pee8CoUNYmh6jdDqv+7fVQb5gHvmZIyYHxN3s91PknQUTbA+/+Bx7eXlPBLX9cQEJMJKcNzuIrZw7i9OOyyE1vW6+3iMjhvLFyOy8u2MSTNxYToW+6RCSM9B1aT1a1CT7+lzeGetP7gIOMgXDyV71A3W/cgRUP26q5xcdNz3xIcUEG3zj3OE4dlMWj1xRx7vA+WkVNRIKuam8jsz/ewfqdexjUO+noJ4iIhIhSTk/inDfv9KpXvR7qrUu87X1GwcT7vUDde1i7ArXP55i/fhfLN1dz+xkDiYqMoDArkT4p3sNF0ZERTCnKCeanERE5oChv/2IyVQrXIhJWCtfdXfM+2PgOrHrNG0ddvcnbnns8nPsgDL0IMge2+/Ibdu7hpZJyXl68mc1V9aQlRHPDif1JjI3ix5dqZI+IdIyBvZJIio1iaVkVV4xv/7z6IiLHSuG6O6qrgDWzvB7qdXOgaQ9EJ8CAM+GMb3tT5iVnt/vyNQ1N/PujrUwvKaektJIIg9MG9+K7k4dy3vA+Bx5KFBHpKBERxujcVJaUVYW7FBHp4RSuuwPnYPsKWP2q10O9uQRw3gOJY66B4yZB4WkQHX9MzWyuqudnr37CrBXb2NfsY1DvJO6bPJRLx+bQJ6VtE6yLiARbUV4aT85bT0NTi/6RLyJho3DdVTU1wMZ3vUC9ehZU+1eazxkPZ37PC9TZo9r9QOJ+a3fUUdPQxLj8dJJionh//S6uKs7jivG5jM5N7dYrM4lI1zImL41mn2PFlhrG908Pdzki0kMpXHcldTu8IL36tcMM9/gODD4fkvscczP7mluIjfJ6fb7y3CKS4qJ46c6TSU2IZv79Z2tBFxHplMb6H2pcWlalcC0iYaNw3Zk5B9uX+x9GfNU/3ANIyfWGewyZDAWnQfSxD8lobvHxzpqdTC8pZ/76Xbz73TNJiIni4StHk5366fUVrEWks+qdEkff1DiNuxaRsFK47myaGvyze/iHe9SUe9tzxsOZ3/eWHe8z8piHe+y3alstLy3yZvuoqN1HekI0U4pyaGjykRADo3PTgtKOiEhHOG94H63WKCJhpXDdGdRu98/u8RqsnwNNe73hHgPPgon3+Wf3OPbhHvtV7mlkxtItTC8pZ9nmaqIijDOH9uaK8bmcOaQ3MVH6xSQiXdP/TBkZ7hJEpIdTuA6XylL46EVv/HTr4R5F18Fxk6Hg1KAM9zjUtuoGzv7FXPY0tjCiXwo/uGg4U4r6kZkUG/S2RETCpbnFpx5sEQkLhetwWDYd/nkvNO7xhnuc9X1vdo8gDvc4VEXtPnolx5KdGsc3zhvCSQMyGd4vJSRtiYiES1OLj4kPz+WK8bl8/dzjwl2OiPRACtcdqakBZt0PC5+B/JPgsichLT/kzf5z6Ra++belzLj7FIZmp/ClUwtD3qaISDhER0Zw0ei+jFDngYiEicJ1R9m1Dv52M2z7CE65F876fxAZHdIm6/Y1kxQbxamDsrjpxP70TT22RWREpOcys0nAo0Ak8LRz7qFD9ucDfwbS/Mfc55yb2eGFAvdfMCwczYqIAArXHWPlP+Afd4NFwLUvejN+hNC26gYemLGCLdX1vHzXKaQnxvD9i4aHtE0R6b7MLBJ4DDgXKAcWmNkM59zKVod9H5jmnHvczIYDM4GCDi8WcM6xtbqB5LgokuNC24khInIoPe0RSs37YOZ3YNpN0GsI3PFuSIN1i8/x7PyNnPPLt5mzagfnj8jG51zI2hORHmMCsNY5t9451wi8AEw55BgH7B+LkQps6cD6DrJqey0nP/QWb32yI1wliEgPpp7rUKncCH+7BbYsghO/Auc8AFExIWvu46013P/3ZSwpq+K0wVn86Asj6Z+ZGLL2RKRHyQHKWr0vB0445JgHgNfN7KtAInDO4S5kZrcBtwHk54fmmZNBvZKIj45k8aYqphTlhKQNEZHPo3AdCp/8G1650+vHufo5GHZRyJqqb2zh/95czdPvbCAtPpr/u7qIKUX9sBDNOiIi8jmuBf7knPuFmZ0E/MXMRjrnfK0Pcs49CTwJUFxcHJKv1qIiIxiVm8rScq3UKCIdT+E6mFqaYPYDMP+30LcIrvwTZIRuZo531+zk/pc/omx3PVcX53H/BUNJSwhd77iI9FibgbxW73P921r7EjAJwDk338zigCwgLGMzivLS+NN7G2ls9mlhLBHpUArXwVJd7g0DKR/Z7AgAACAASURBVP8Qjr8Vzv8xRIV2YZayyr3EREbw4m0ncsKAzJC2JSI92gJgsJkV4oXqa4DrDjlmE3A28CczGwbEARUdWmUrRXlpNDb7+GRbDaNz08JVhoj0QArXwbD6dXj5Nmhphiv+CCMvC0kzPp/jhQVlxEZFcPn4XK4uzuOycTnERkWGpD0REQDnXLOZ3Q3Mwptm7xnn3AozexBY6JybAXwTeMrMvo43KO5m58L3RPWYPC9QLymrUrgWkQ6lcH0sWpphzo/g3V9Bn1Fw1Z8hc2DImjPzFoRJiovi8vG5REQYsREK1iISev45q2cesu0HrV6vBE7p6Lo+T7/UOHolx7KkrIqbTgp3NSLSkyhct1fNVnjpS1D6Hxh/M0x6CKKDv0hLQ1MLj89dxzUT8uibGs8TN44nJU7/s4mIHImZMSY3jSVleqhRRDqWUlp7rHsLXroVmurhsqdg9FUhaeY/a3fy3y8vY+OuvWQkxvDFkwtIjdeCCCIigRibn8bsj7dTXd+ke6eIdBiF67bwtcDbP4O3fw69hsJVz0Kv44LezK66ffx45sf8fdFmCjITeO7LJ3DKoKygtyMi0p1dNLovI3NSiYvWbCEi0nEUrgNVux3+/mXYMA+KrocLHoaY4C7S4pxjekk5P5n5MXX7mrn7zEHcfdYg4qI1rlpEpK36ZyZqMS0R6XAK14HYMA9e+jI01MCUx2DsDUFvYl1FHf/98jLeX7+b4v7p/OSyURzXJzno7YiI9CQlpbvZXNXAJWP6hbsUEekhFK6PxOeDd34Bc38CGQPhxlegz/CgN1O9t4lLfvMukRHGTy8bxdXFeUREaIVFEZFjNfWDMuatqeDi0X21cq2IdAiF68+zZyf8/Vbv4cVRV8JF/wexSUFtYu2OWgb1TiY1IZqfXzGG4wvT6Z0cF9Q2RER6sm+fP4QfXjJcwVpEOoye8jic0vnwxGmw8T9eqL7sqaAH6yfnreOcX87jnTXeAmYXju6rYC0iEmTZqXGkxGmmEBHpOOq5bs3ng/cehTf/F9L7w5dnQ9/RIWnq+hP6M7hPMsX9M0JyfRER8Tw+dx2ZiTFcdXxeuEsRkR5A4Xq/vbvh5TtgzSwY/gW45DcQlxKy5hJjozhzSO+QXV9ERDxvrNxGZIQpXItIh9CwEICyBd4wkHVvweSH4co/hTRYL9i4m1++voqahqaQtSEiIp6ivHSWba6mucUX7lJEpAfo2eHaOZj/GPxxEkREwJdehxNugxA/+PL6im08MW89sVE9+69fRKQjjMlLpaHJx6rtteEuRUR6gJ6d7ub+FGZ9DwafD7fPg5xxHdJsSWklo3NSiY3S4jAiIqE2Ni8dgKVl1WGuRER6gp4drsd9ESb/HK55DuLTO6TJhqYWlm+uYVz/jmlPRKSny8uIJyMxhiVlleEuRUR6gJ79QGNqDpxwe4c2uWJLNY0tPsblK1yLiHQEM2NMbqp6rkWkQ/TsnuswWFRaBcC4/mlhrkREpOcYk5fG6h211O1rDncpItLNhTRcm9kkM1tlZmvN7L7D7O9vZm+a2UdmNtfMclvt+6KZrfH/fDGUdXakktJK8jMStGCMiEgHKspLwzlYVq7eaxEJrZCFazOLBB4DJgPDgWvNbPghhz0CPOucGw08CPzUf24G8EPgBGAC8EMz6/LjKJxzlGyqZLzGW4uIdKgxuWlkp8RRXd8Y7lJEpJsLZc/1BGCtc269c64ReAGYcsgxw4G3/K/ntNp/PvCGc263c64SeAOYFMJaO0R5ZT0Vtfv0MKOISAdLT4zh/e+dzaSRfcNdioh0c6EM1zlAWav35f5trS0FLvO/vhRINrPMAM/FzG4zs4VmtrCioiJohYfK1uoG+qbGMV4PM4qIiIh0S+F+oPFbwBlmthg4A9gMtAR6snPuSedcsXOuuFevXqGqMWgmFGYw//6zGdY3OdyliIj0OHNW7eC0n7/FtuqGcJciIt1YKKfi2wzktXqf6992gHNuC/6eazNLAi53zlWZ2WZg4iHnzg1hrR3KQrwCpIiIfFavpFhG5aTS0BRwH46ISJuFsud6ATDYzArNLAa4BpjR+gAzyzKz/TXcDzzjfz0LOM/M0v0PMp7n39Zl7dnXzKk/e4t/fbQl3KWIiPRII3NS+d314ynISgx3KSLSjYUsXDvnmoG78ULxx8A059wKM3vQzC7xHzYRWGVmq4E+wI/95+4G/hcvoC8AHvRv67JqG5opykujV1JsuEsREenRquubwl2CiHRj5pwLdw1BUVxc7BYuXBjuMkRE2sXMSpxzxeGuoyOF47798KxP+NN/NvLRA+cTGaEheiLSPke6ZwfUc21mfzezC1sN4ZA2qtqruVVFRMJtYK8k9jS2sK6iLtyliEg3FWhY/h1wHbDGzB4ysyEhrKnb8fkcp/98Dj/+98pwlyIi0qONyUsDYMmmqjBXIiLdVUDh2jk32zl3PTAO2AjMNrP3zOwWM4sOZYHdwbqKOmoamjmuj6bgExEJp8LMRFLiolhcpnAtIqER8DAP/+IuNwNfBhYDj+KF7TdCUlk3UlJaCaBlz0VEwiwiwhiTl8ZShWsRCZFAx1y/DLwDJAAXO+cucc696Jz7KpAUygK7g5LSStIToinU9E8iImFXlJfGqu211DdqvmsRCb5AF5H5tXNuzuF29LSn29ujZFMl4/una/EYEZFOYExuGi0+x7LN1UwozAh3OSLSzQQ6LGS4maXtf+Nf3OWuENXUreze08j6ij2M05AQEZFOYf9DjRoaIiKhEGi4vtU5d+Au5JyrBG4NTUndy+JN3njrcfkK1yIinUGv5Fhy0uJZonAtIiEQaLiOtFZjGswsEogJTUndS0lpJZERxpjctKMfLCLSCZnZJDNbZWZrzey+zznmKjNbaWYrzGxqR9fYVvdNHsr1J+aHuwwR6YYCHXP9GvCimf3e//52/zY5ipLSSkb0SyE+JjLcpYiItJm/M+Ux4FygHFhgZjOccytbHTMYuB84xTlXaWa9w1Nt4C4e0y/cJYhINxVouP4uXqC+0//+DeDpkFTUzdxz9mAam33hLkNEpL0mAGudc+sBzOwFYArQelWsW4HH/EMGcc7t6PAq26ipxcfCjZVkp8ZpJicRCapAF5HxOeced85d4f/5vXNOcxgF4JRBWZw5tNN34oiIfJ4coKzV+3L/ttaOA44zs/+Y2ftmNunzLmZmt5nZQjNbWFFREYJyA9Pc4rjhDx/wUkl52GoQke4poJ5r/1d+PwWGA3H7tzvnBoSorm5haVkVe/Y1c9LATE3DJyLdWRQwGJgI5ALzzGxU6wfh93POPQk8CVBcXOw6ssjW4mMimfrlE7RyrogEXaAPNP4ReBxoBs4EngX+Gqqiuoun393AN/+2VMFaRLqyzUBeq/e5/m2tlQMznHNNzrkNwGq8sN2pnTAgk/REPZsvIsEVaLiOd869CZhzrtQ59wBwYejK6h5+NGUkT39Ra+yISOdhZveaWYp5/mBmi8zsvCOcsgAYbGaFZhYDXAPMOOSYV/B6rTGzLLxhIutDUH5Qbatu4LE5aymv3BvuUkSkGwk0XO8zswhgjZndbWaXomXPjyo1IZoR/VLDXYaISGv/5ZyrAc4D0oEbgYc+72DnXDNwNzAL+BiY5pxbYWYPmtkl/sNmAbvMbCUwB/i2c25XKD9EMFTXN/HwrFV8sH53uEsRkW4k0HB9L5AA3AOMB24AvhiqorqDBRt38+s311C3rzncpYiItLZ/nNoFwF+ccytabTss59xM59xxzrmBzrkf+7f9wDk3w//aOee+4Zwb7pwb5Zx7IaSfIEgG9U4iMSZSi8mISFAdNVz75zi92jlX55wrd87d4py73Dn3fgfU12W9tnwbv5u7ltioQP/9IiLSIUrM7HW8cD3LzJKBHjlfaGSEMTo3jaXlCtciEjxHTX7+KfdO7YBaupWS0kpG56QRHalwLSKdypeA+4DjnXN7gWjglvCWFD5j8tL4eGsNDU2aXVZEgiPQ5LfYzGaY2Y1mdtn+n5BW1oU1NLWwYks14/qnh7sUEZFDnQSscs5VmdkNwPeB6jDXFDZFeWk0tThWbq0Jdyki0k0EGq7jgF3AWcDF/p+LQlVUV7dsczVNLY7xCtci0vk8Duw1szHAN4F1eNOr9khFeWkALNmkoSEiEhwBLSLjnOuxXxm2x6LSSgDG5aeFuRIRkc9ods45M5sC/NY59wcz+1K4iwqX7NQ4slPiNO5aRIIm0BUa/wh8ZiUt59x/Bb2ibqCktJLCrEQyk2LDXYqIyKFqzex+vCn4TvNPsxod5prCakxeqmYMEZGgCXRYyL+Af/t/3gRSgLpQFdWVOedYtKmScfkaEiIindLVwD68+a634a24+HB4Swqvorx06hqa2aOpU0UkCAIdFvJS6/dm9jzwbkgq6uI27d7LzrpGjbcWkU7JObfNzJ4Djjezi4APnXM9dsw1wJdOLeSOMwZgdsTpvkVEAtLeeeIGA72DWUh3saWqgaykWIVrEemUzOwq4EPgSuAq4AMzuyK8VYVXTFSEgrWIBE2gY65rOXjM9TbguyGpqIs7aWAmC/777HCXISLyef4bb47rHQBm1guYDUwPa1Vh9ovXV7F7TyM/vnRUuEsRkS4u0GEhyaEupDtRD4iIdGIR+4O13y7a/y1mt9HY7GNfc49cqFJEgiygG6qZXWpmqa3ep5nZF0JXVtdU29DExIfn8NryreEuRUTk87xmZrPM7GYzuxnvQfWZYa4p7O6/YBiPXDkm3GWISDcQaG/FD51zB1bwcs5VAT8MTUldV01DM0OzU8hI1BR8ItI5Oee+DTwJjPb/POmc0zA/P5/vM7POioi0SUDDQjh8CA/03B4jJy2eJ24cH+4yRESOyD8D1EtHPbAHcc4x+dF3OHlgFj+4eHi4yxGRLizQgLzQzH4JPOZ//xWgJDQldV01DU2kxPXotRhEpJM6zIPpB3YBzjmX0sEldSpmRnJcFIvLKsNdioh0cYEOC/kq0Ai8CLwANOAFbPHz+Ryn/PQtfvbaJ+EuRUTkM5xzyc65lMP8JPf0YL3fmNw0VmypoVEPNorIMQgoXDvn9jjn7nPOFTvnjnfOfc85tyfUxXUla3bUUbuvmUG9ksJdioiItENRfhqNzT4+2VYT7lJEpAsLdLaQN8wsrdX7dDObFbqyup6SUu+rRC0eIyLSNRXleb/mlpZVhbkSEenKAh0WkuWfIQQA51wlWqHxICWllWQmxtA/MyHcpYiISDvkpMWTlRTDYoVrETkGgYZrn5nl739jZgUc/sGYHmvRpkrG9U/XAjIiIl2UmVGUl6aeaxE5JoGG6/8G3jWzv5jZX4G3gftDV1bXsqtuHxt27tGQEBGRLm5MbhrrKvZQXd8U7lJEpIsK9IHG14BiYBXwPPBNoP5o55nZJDNbZWZrzey+w+zPN7M5ZrbYzD4yswv82wvMrN7Mlvh/nmjTp+pgizZ5vRwK1yIiXVtRvjfuell59VGOFBE5vIDmuTazLwP3ArnAEuBEYD5w1hHOicSbF/tcoBxYYGYznHMrWx32fWCac+5xMxuOtwRvgX/fOudcUds+TniUlFYSFWGMykk9+sEiItJpjc5N4/YzBpCdqpV2RaR9Ah0Wci9wPFDqnDsTGAscbVDaBGCtc269c64Rb37sKYcc44D986umAlsCrKdTWVRayYicVOKiI8NdioiIHIPU+GjunzyMQb2Tw12KiHRRga7Q2OCcazAzzCzWOfeJmQ05yjk5QFmr9+XACYcc8wDwupl9FUgEzmm1r9DMFgM1wPedc+8c2oCZ3QbcBpCfn3/o7g5z15kDafHp+U4Rke6goamFFVtqGJefpofURaTNAu25LvfPc/0K8IaZ/QMoDUL71wJ/cs7lAhcAfzGzCGArkO+cGwt8A5hqZp9ZQcw596R/YZviXr16BaGc9pk4pDdnD+sTtvZFRCR4pi0s4/LH32NLdUO4SxGRLiignmvn3KX+lw+Y2Ry8IRyvHeW0zUBeq/e5/m2tfQmY5G9jvpnF4c2pvQPY599eYmbrgOOAhYHU25E+Kq+ivrGFCYUZ6uEQEekGzh7Wh+yUONITosNdioh0QYH2XB/gnHvbOTfDP476SBYAg82s0MxigGuAGYccswk4G8DMhgFxQIWZ9fI/EImZDQAGA+vbWmtH+P289Xxj2lIFaxGRbiInLZ7zRmSTEBPoyEkRkU+F7M7hnGs2s7uBWUAk8IxzboWZPQgsdM7NwJvS7ykz+zrew403O+ecmZ0OPGhmTYAPuMM5tztUtR6L/50yks2VR52VUEREupAVW6pZu6OOKUU54S5FRLqYkP6z3Dk3E296vdbbftDq9UrglMOc9xLwUihrC5aMxBgyEmPCXYaIiATR3xaW8+KCMi4c1ZeoyDZ/ySsiPZjuGMfgww27+d3ctextbA53KSIiEkRFeWnUN7WwentduEsRkS5G4foYzFy2ld+8uZZo9WqIiHQrRXneSo1Lyo62pIOIyMGUCo/Bok2VjMlLVbgWkW7PzCaZ2SozW2tm9x3huMvNzJlZcUfWF2z9MxNIS4hmqcK1iLSRUmE77W1s9i8ykB7uUkREQso/e9NjwGRgOHCtmQ0/zHHJeCv6fhCyYso+DNmlWzMzxuSmqedaRNpM4bqdPiqvpsXnGN9f4VpEur0JwFrn3Hr/NKwvAFMOc9z/Aj8DQrP6ypo34A/nwts/D8nlD1WUl8bqHbXU7dNzNSISOIXrdlq0qRKAseq5FpHuLwcoa/W+3L/tADMbB+Q55/59pAuZ2W1mttDMFlZUVLStioFnwZhrYc6P4Z1ftO3cdijKS8M5WFZeHfK2RKT7ULhup0WllQzolahp+ESkxzOzCOCXeGsXHJFz7knnXLFzrrhXr15taygiEqY8BqOuhDcfhP882r6CAzTG/1Dj0nINDRGRwGn5qXZwzlFSWsk5w/qEuxQRkY6wGchr9T7Xv22/ZGAkMNe/Wm02MMPMLnHOLQxqJRGR8IUnwNcCb/wAIqLgpK8EtYn9MhJjyM9I4OOtNSG5voh0TwrX7bBh5x4q9zZpvLWI9BQLgMFmVogXqq8Brtu/0zlXDWTtf29mc4FvBT1Y7xcZBZc9Ba4FZn3PC9gn3B6SpqbfeRJZibEhubaIdE8K1+2wpaqBtIRohWsR6RGcc81mdjcwC4gEnnHOrTCzB4GFzrkZHV5UZBRc/gevB/vV74BFwIRbg95M7+S4oF9TRLo3het2OHVwFov/37nhLkNEpMM452YCMw/Z9oPPOXZiR9REZDRc8UeYdhPM/JbXg118S1Cb2FW3j4dnreKSon6cPDDr6CeISI+nBxrbyczwjy0UEZFwiYqBq/4Mg8+Df30NFv0lqJdPjI3izU92UL67PqjXFZHuS+G6jarrmzjnl28ze+X2cJciIiIAUbFw1V9g4Nkw46uwZGrQLh0XHcmH3zubq47PO/rBIiIoXLdZTX0TBZkJpCZEh7sUERHZLzoOrnkOBpwBr9wFS18M2qX1LaWItIXCdRvlZSTw9BeP5/iCjHCXIiIirUXHwzXPQ8Gp8ModsGx6UC67cONuzvnl26zZXhuU64lI96Zw3UZ7tAyuiEjnFZMA170I+SfB32+DFS8f8yXTEqJZu6OOxWVaTEZEjk7hug1afI4Tf/Imv3xjdbhLERGRzxOTCNdNg9zjYfqXYOWxzRQ4ICuJ5Lgolipci0gAFK7bYPX2Wmr3NTMgKzHcpYiIyJHEJsEN0yFnPEy/BT75d7svFRFhjMlNY4nCtYgEQOG6DUpKKwG0eIyISFcQm+wF7L5jYNoXYdVr7b7U+P7prNxaw09nfkx9Y0sQixSR7kbhug0WlVbSKzmW3PT4cJciIiKBiEuFG/4OfUbAtBthzex2XebW0wdwdXEev5+3nkmPzuO9tTuDXKiIdBcK121QsqmS8fnpmpZJRKQriU+DG1+GXkPhhetg3VttvkRSbBQPXT6aqbeeAMB1T3/A/3tlebArFZFuQOE6QBW1+yjdtVdDQkREuqKEDLjpH5A1GJ6/Fta/3a7LnDwwi1lfO507zhhIdmpckIsUke5A4TpAizZ5463HKVyLiHRN+wN2xgCYejVsfLddl4mLjuS+yUP5ypmDAHhj5XZue3Yh1XubglmtiHRRCtcBWlRaSUxkBCNzUsJdioiItFdiFtw0A9L7w3NXQel7x3zJnXX72FG7j4TYyCAUKCJdncJ1gBZtqmRkTgqxUbp5ioh0aUm9vICdmgPPXQmbPjimy107IZ+/33ky0ZER1DQ08ZXnFrG+oi5IxYpIV6NwHaDbTx/IXRMHhbsMEREJhuQ+8MV/QnI2/PVyKF94TJeLiPAedF+1rZZ31lQw6dF3eGzOWppafMGoVkS6EIXrAJ0zvA/nDO8T7jJERCRYkrO9gJ2YBX+5FDaXHPMljy/IYPY3z+DcYX14eNYqLv7Nu1rZUaSHUbgOwLLyahZtqsQ5F+5SREQkmFL6wc3/gvh0L2BvWXLMl+ydHMdj14/jyRvHU7m3kUt/9x9+9K+V7G1sDkLBItLZKVwH4Hdz13LP84s1v7WISHeUmusF7NhUeHYKbP0oKJc9b0Q2b3zjDK6dkM/T727gvF/NY97qiqBcW0Q6L4XrAPzPJSN47Lpx4S5DRERCJS0fbv4nxCR5AXv7iqBcNiUumh9fOoppt59ETGQENz3zIWt31Abl2iLSOSlcB6B3Shxj8tLCXYaIiIRSeoEXsKPi4M+XwI6Pg3bpCYUZzLz3NJ64YTyDeicDsHJLjYYbinRDCtdHsWDjbp6at576xpZwlyIiIqGWMcAbIhIRBX++GCpWBe3ScdGRTBqZDcAn22q46Dfv8Oz80qBdX0Q6B4Xro/jX0i38avZqoiM13lpEpEfIHOgFbMwL2DvXBL2Jwb2T+Z9LRvCFsTkA7KhtwOdTL7ZId6BwfRQlmyopyksjKlJ/VSIiPUbWYG+aPufzAvaudUG9fGSEceNJBaTGR9PU4uOmP3zIlb+fz5rtGo8t0tUpMR7Bnn3NfLy1lnH56eEuRUREOlrvod5Kji2NXsDevSEkzURFGLedPoB1FXVc+Ot3eXT2GhqbtfiMSFelcH0ES8uraPE5xvdXuBYR6ZH6DPcCdtNeePpsePW7sOl98AUv/JoZl43LZfY3zmDSyGx+NXs1F/3mHUpKK4PWhoh0HIXrI1i8yVtVa2y+ZgoREemxskfCzf+GvBNh4R/hmfPhVyO8oF06P2hBOyspll9fO5Znbi6mrqGZK554jwdmrKBunxafEelKosJdQGdWUlrJoN5JpCXEhLsUEREJpz4j4Nqp0FADq2fByle8oP3BE5DcF4ZdAiO+4AXwiGPrtzpraB9e/0Ymj8xaxZ/nb+T1Fdt45MoxnDwoKzifRURCKqQ912Y2ycxWmdlaM7vvMPvzzWyOmS02s4/M7IJW++73n7fKzM4PZZ2H4/M5Fm2qZLzGW4uIyH5xKTD6SrjmOfj2WrjsacgZDyV/gj9Ohl8Og5nfho3/AV/7p3BNio3igUtGMP2Ok0mKi2JLdQMA22samLagjJqGpiB9IBEJtpD1XJtZJPAYcC5QDiwwsxnOuZWtDvs+MM0597iZDQdmAgX+19cAI4B+wGwzO84512GTTa/fuYeqvU0aby0iIoe3P2iPvhL21Xo92itehkXPwodPQlKfT3u080+CiMg2NzG+fzqv3ns6Lf5p+t78eAffe3kZxQXppMRFs2HnHuKiI+ibGh/sTyci7RTKYSETgLXOufUAZvYCMAVoHa4dkOJ/nQps8b+eArzgnNsHbDCztf7rzQ9hvQcpr9xLcmwU4xSuRUTkaGKTYdQV3s/+oL3yFVj8F1jwlD9oXwwjLm1z0I6MMCIjvLUWrp2Qx7j+aQzolQTAL15fxb8+2kpRXhqTRmZz/ohsCrMSQ/IRRSQwFqqlV83sCmCSc+7L/vc3Aic49//bu/fwuKrz3uPfdzQzGo2uo5t1t4RtbPmGbMsGY+NwO6kNBEgIMSmQBlpIiYmB5jktKc05PCk9padJ25AQLiG0nAbTgAOB5gFMHBsTws2y8UWyBcZXyZJsSZZGo/tIs84fe6yb5SszGkv7/TzPfjR7z95ba9ny8k9r1l7L3DvknFzgLcAHJAJXG2O2iMhPgQ+MMb8Mn/cL4A1jzNoR3+Nu4G6AoqKiBQcPRnalq1DIIGI9ya2UUtEkIluMMeWxLsdYKi8vNxUVFbEuRnT1tMOedVD1G9jzO+jrgsRsmHk9zLwRJl96Tj3ax+1tbOfNygbWVTWwo9YPwIycZL44K4fls3IozU3W/8OUioJTtdmxfqDx68B/GGN+JCKLgf8UkdlnerEx5mngabAa6UgXzuHQBkkppdTnEJ8Es2+ytp522PNWuEf7edj8jBW0S79kDR2ZvOSsg/aUrCRWXTGVVVdMpbalk7eqjvBmVQM/3bCHx36/h6J0L3ddVsLti4ujUz+l1AmiGa4PA4VD9gvCx4b6c2A5gDHmfRHxAJlneG3UtHb2csvTH/A3y2dwxYzssfq2SimlJrL4JJj9FWvr7bCCdtVvYPsLUPELSMyygvbMcNCOO7v/ogt8Xu5cWsKdS0toau9h/S4raPeEF6Rp7+nj/75ZzTcWT2ZqdnI0aqiUIrrhejMwTURKsILxLcCfjjjnEHAV8B8iUgp4gEbgNWCNiPwL1gON04CPoljWYfxdQSaleEj2xLpjXyl7CAaD1NbW0t3dHeuiRJ3H46GgoACXyxXropwxEVkO/BiIA54xxjw64v2/Av4C6MNqw+80xkR2nN5E4060xl/P+nI4aP/Oehhy+39BxbPgzRzSo730rIN2ZlI8tywq4pZFRQPHquvbeKmilmvn5DI1Gz47GmB/UyeXTcvE4zr3oSlKqeGiNuYaIDy13r9hNcjPGmP+QUR+AFQYY14LzwrycyAJ6+HG17JeoQAAF2JJREFUvzbGvBW+9iHgTqzG+n5jzBun+l62GLun1AS1f/9+kpOTycjImNDjQ40xNDc3EwgEKCkpGfbe+TrmOjzz06cMmfkJ+PrQmZ9E5ArgQ2NMp4jcA1xujFl5untruz2K3s7BoSOfrrNWhvRmwtSroXAhFCyC7JlnHbaP6w7244pzEOcQ/vH13Tz1zj687jiumJ7Nn8zO4YrpWSR7xs8vfkrFyqna7KiG67EUyUa6O9ivv8UrNYZ2797NjBkzJnSwPs4YQ3V1NaWlpcOOn8fhejHwsDHmT8L73wMwxvzjSc6fB/zUGLPkdPfWcH0avZ3w2e+soSMH/gAdjdZxVyLkz4eChVC4yPqaePYLzPT2hfhgXzPrqhpYV3WEpvYe3HEOlk7LZPmsHK6eOYn0RF1ETanRnM8PNJ53+vpDlD+ynm8tu4DvXDUt1sVRyjbsEKxhXNYzH6gZsl8LXHyK8/8cOOknjSNmeYpE+SYutxdm3mBtxkDrQajZDLUfQe1meO8xCIWXRveVDAbtgoXWipJxp+6BdjsdLLswi2UXZvGDG2bz8aEW3qxs4M2qBjZUH8XxMtxQls+/riwDoKu3nwS3djwpdToarkeobgjQ3tNHUYY31kVRSqlxRURuA8qBL5zsnGjP8jRhiYCv2Nrm3mwd6+2E+m1W0K75CPa9DTt+Zb3n8kLevMGwXbgIkk7+gH6cQygvTqe8OJ2Hri1lV30b6yobBoaIhEKGi//Pev7s0mK++8Xp9IcMOw/7Kc1NJt6pgVupoTRcj7D1UAuArsyolI20trayZs0avv3tb5/Vdddccw1r1qwhLS0tSiU7L5zR7E0icjXwEPCF8AJgKtrcXmue7MmXWvvGgL/GCtq1m63t/cchFF4qPa3IGrNduAgKyiFn7qi92yLCrLxUZuWlDhzr7Q/xrS9MoazQ+lnf29jOjY//EXecg9K8FMoKUikrSuOigjRKMhPH4yc0SkWMhusRthxsYVJKPPlpupSsUnbR2trKz372sxPCdV9fH07nyZvJ119/PdpFOx+cduan8Djrp7AWDjs69kVUgNW7nVZkbXO+ah0LdkH9DmsoSc1HcPCPUBlej83pCfdul1uhu2AhpOSOemuPK45VV0wd2M9N9fDErfPZVtvKtkOtvLSllufetyaISU1wMbcglbLCNG5eUKifBCvb0XA9wpaDLSyY7NPfupWKoZVPvX/ac64qzebuZVMGzv/qggJuLi/kWEcv9/xyy7Bzf/Wtxae814MPPsjevXspKyvD5XLh8Xjw+XxUV1fz6aefcuONN1JTU0N3dzf33Xcfd999NwDFxcVUVFTQ3t7OihUrWLp0Ke+99x75+fm8+uqrJCSM/1/SjTF9InIvsI7BmZ+qhs78BPwz1qxPL4XbzkPGmOtjVmg1yJUARRdb23H+2nDvdoUVuj98Ct77ifVeauHwoSQ5c8F54kONyR4XK+bksmKOFcb7Q4Y9RwNsr2llW00r22r8PL7xMy6fnk1RhpdNnzbyUkUND18/i8ykeIwx+v+smrA0XA9xpK2b2pYuvnlpcayLopQaQ48++iiVlZVs27aNt99+m2uvvZbKysqB6fKeffZZ0tPT6erqYuHChdx0001kZGQMu8eePXt44YUX+PnPf87XvvY1fv3rX3PbbbfFojoRZ4x5HXh9xLH/NeT11WNeKHXuUgusbfZXrP2+nuG92zUfQdXL1ntx8ZAxJTzeu8T6mh7+mlYEznjrNIcwIyeFGTkprFxoPaja2duHO84BwLGOHqrq2gbWj3j0zWre3dPERYVplIW3KVlJxOnKyGoC0HA9xNaD1njr+TreWqmYOl1P86nOT090n/X1Iy1atGjYPNSPPfYYr7zyCgA1NTXs2bPnhHBdUlJCWZk1q8KCBQs4cODA5yqDUmPGGW/NoV24EBavso611Q2O227eC8f2w96N0Nc15EKBlPxw2J48InyX4E3wWUNVgC/PK+DL8woGrrwgM5FddW389/Y61nx4CICkeCdz8lOHBe6cVM8Y/SEoFTkarofYcrAFt9PBrLyUWBdFKRVDiYmJA6/ffvtt1q9fz/vvv4/X6+Xyyy8fdSXJ+Pj4gddxcXF0dXWdcI5S40ZK3uA0gMcZA+1HoWU/tBywAnfLAWt/z++g/cjwe8SnWqH7eE/3kN7vlfMLWbmwiFDIsL+5Y2A4yfaaVn7x7j6C/YYLJyXx1gPWxDNPbtpLbqqHG8ryw0XRYSXq/KXheogth1qYm5+q0wopZTPJyckEAoFR3/P7/fh8PrxeL9XV1XzwwQdjXDqlzhMikDzJ2oouOfH93g5oOXhi+D5SBZ+8Af29Q+4VB2mFOHwlTPEVMyW9hK9MK4ZFJXQnz2L3MfB3BQdOf3VbHbPzUrihLJ9QyLDwH9aTneJhWnYSU4dsxRmJuJ2OaP9JKHVKGq6HuPuyC3S8l1I2lJGRwZIlS5g9ezYJCQlMmjRp4L3ly5fz5JNPUlpayvTp07nkklFChVIK3Ikwaaa1jRTqh0D98N7u4wF816vQdWzgVA8wz5th9XRvy4PkXN6Yn0tfUg7sayWYkM3KuWnsOmbYeqiF17bXDVwb5xAmp3uZkp3ETfPzWT47F2MMnb39JMZr5FFjQ5c/V0rF3O7du09YDnwiG62+5+vy59Gk7bYa0O0/cahJy0ErkLfVQ4//xGvcyZCSS39iDm2uTBolnZpgGnu6kqkMeFk6fy63XFFObVsvS/9pIz+6+SJuWlBAg7+bDdVHB3q7dYl3dS50+fMzUHnYjzEwpyD19CcrpZRSKnI8qZB7kbWNprfDCtmB8NZWN/A6rq0eX+tmfIF6Lgz1cdXxa94F/ugg15vFR5npeHcWQn0h/s5kPt7eyRsmnQaTTk/CJHKys5k6KZmpWVbgnpKdRE6KRz/NVudEw3XYTzbsobohwKb/eUWsi6KUUkqpodyJkDnV2k4mFILOpsHe7kAdtNUTF6gjO9AAbYehajPTu1r456ELU4aguyGeIw0+6kM+GoyPT0wa1yyaSX5ePp/4nfzhcIiVl80l2ZfFMZOEw+0l1evWhyrVqDRchz18/Szq/SfOAKCUUkqpccDhgKRsaztZDzhYq1YGGob1gHsCDRS11ZHXeph+fw3Ozq04t74OW2E61sY+6/J0oMc4aZRkOh3JBN2phDw+HInpxCdn4k3LIjU9G1dSBiT4ICHd+upNtxb1UROehuuw3NQEclP1h14ppZSa0FwJ1vSA6SXDDgvgCm+AFcK7Wga3zmPQ1cLh+sM0NR4h2N6MdB4jrtdPQtd+klt24qOdeAlyUk4PIU8a4k1HEtLB6wsH8CEhPDETErMGv7qTBuYLV+ODhmtg84FjVB32c8uiIjwunYZPKaWUsj1XgrWl5A07nB/ehgqFDI3tPew81kldYzMd/ma+PjsRulr49/Uf0+k/yqqLM6DrGBu2foJpaGGS8xjpcbWkmgDefj9xpm/0cjgThoftpKzw/iibNwPiNNrFmv4NAK9tq+OVjw9z++LiWBdFKaWUUuOMwyFMSvEwKcUDxenAtIH37rhrGaGQgfDDkW2ZtXxyJEDNsU5qjnVx6Fgn/q5eEujBRzs+CXBZbogHl2VCRyMfVH5CblyAyfEdEKgneHg7zu5mJDRaD7lYw08GAncmJGaPCOfZ2iseZRqusVZmnFeUpk8FK6XOSFJSEu3t7dTV1bF69WrWrl17wjmXX345P/zhDykvt9XsekqpUTiG5IuvzC844f227mA4bHdS19qNL9EFZdZ5j2z9AxeXZPD962YS7A9x4d+9gTGGNOlgWlI3U73dFHvaKXB3kO0IkCF+sqSN5P5WaNgJHY3WVIejGdkrHp9sPTzqTgSXF9xeK4Cf8nX4XGe8BvUw24fr9p4+qhvauPfKaac/WSmlhsjLyxs1WCul1NlI8biYlZfKrLwTpwP+7XcuG7b/zDfKaWjrpsFvbbVt3VT4u2lo6CbQYw0tuXvZBfztNaV09PSx5J828NDyC7i5NIHWpjo2bKkiz9lOpsOPz/hJ7mvB1XMMCdRD8x7o7bSmPgx2gAmdeSUkbkgoT7QCuCv81Z14Zq+HBnZ30uD9HONr1U3bh+vtNa2EDCyY7It1UZRSAG88aPW2RFLOHFjx6EnffvDBByksLGTVqlUAPPzwwzidTjZu3EhLSwvBYJBHHnmEG264Ydh1Bw4c4LrrrqOyspKuri7uuOMOtm/fzowZM+jq6opsHZRStueKc3BV6aSTvt/R00dDWzcJ4efH+voN183NpSDLB6kZ7Pd7+e7HRzEma9h1HpeDnBQPWcnxZGXE881LS1hU7KOlLcD2fYeZN8lNqjOI6e1Agh1WAA92WCF84PWQUD70dXebNTtLb3v43PB2VhX3DulRTxx8PXI76XsjQ7vX2o9ST7vtw/WWgy2IQFlhWqyLopSKkZUrV3L//fcPhOsXX3yRdevWsXr1alJSUmhqauKSSy7h+uuvP+m8tk888QRer5fdu3ezY8cO5s+fP5ZVUEopEuOdTMlKGthP9bp45MY5A/vzinx88vcrOBro5khbN/X+wR7w+rZuGgM9VDcECHQHQYTKoz1881f7ePFbi1mUl86rHx/m+7+pJDPZQ2ZSCplJ8WQlx5OZdHxzk5Ucz/ScZLzuU0TMUGgwZPeGQ3qwMxzAj4fz9lGOj9g6Goe/d1ahXayw7UmFv9p1Dn/aJ2f7cL31UAsXZieTmuA6/clKqeg7RQ9ztMybN4+jR49SV1dHY2MjPp+PnJwcHnjgAd555x0cDgeHDx/myJEj5OTkjHqPd955h9WrVwMwd+5c5s6dO5ZVUEqpM+J2OijweSnweU977vwiH6/du4QLwoG9ODORmxYU0NTeQ1N7D58eCfDe3mb8XcMfrvzve5cypyCVl7fW8pMNn/HSXy4mMymeTZ82sr2mdSCIZybHk5U0icz0eBLcEZit7XhoH+hBP761D/am97YPD/Wh/s//fUewdbgOhQxbD7Zw7dy805+slJrQbr75ZtauXUtDQwMrV67k+eefp7GxkS1btuByuSguLqa7WxeaUkrZR2K8k7kFg5/slxWmjfpJf09fP83tvQOhuyQrEYDMpHhm5aWQ7LHi5nt7m3hq075Rv1e800Ga14XP6+blb1+K1+3kjZ317Kpv47tfnA7Azlo/7T19+BJdpCW4SfO6hk+h7HBAfJK1xZCtw/Xexnbauvt0vLVSipUrV3LXXXfR1NTEpk2bePHFF8nOzsblcrFx40YOHjx4yuuXLVvGmjVruPLKK6msrGTHjh1jVHKllIqteGcceWkJ5KUNX4xv2YVZLLtwcHz391aU8t3/MZ3mjh6aAlYYbwz00NTRg78zSEtnL62dQTxOKzBvPdTC6zsbBsL1j3+/h/W7jwz7Hh6XYyBop3ld5Kd5+dHXrBU636pqoD9kWDEnF4CaY524nQ5SE1xRXdfE1uG6pqUTrztOw7VSilmzZhEIBMjPzyc3N5dbb72VL33pS8yZM4fy8nJmzJhxyuvvuece7rjjDkpLSyktLWXBggVjVHKllBo/3E7HGa+K/dC1M3no2plD9ku5c2kxrZ1BWsNh3N8VpLWzl5bOIP7OIE3tPQPnP/f+AbqDoYFw/RfPVfDJkQAwGMqf/saCYb3zkSDGmIjeMFbKy8tNRUXFWV/X1x8iziEnfUhJKRV9u3fvprS0NNbFGDOj1VdEthhjbDUp9rm220opdSY6e/voDoZIT3QDsKH6CPX+blo7g/i7grR09HLf1dPOaPz5SKdqs23dcw3gjBtfcycqpZRSSqnT87qdeN2D+1fOOPk0hpGkyVIppZRSSqkI0XCtlDovTJQhaqdjl3oqpZRdabhWSsWcx+Ohubl5wgdPYwzNzc14PJ5YF0UppVSU2H7MtVIq9goKCqitraWxsTHWRYk6j8dDQUFBrIuhlFIqSjRcK6VizuVyUVJSEutiKKWUUp+bDgtRSimllFIqQjRcK6WUUkopFSEarpVSSimllIqQCbNCo4g0AgfP4dJMoCnCxRkP7FhvO9YZ7Fnv8VjnycaYrFgXYixpu31WtM72Ycd6j8c6n7TNnjDh+lyJSIXdlhwGe9bbjnUGe9bbjnW2Ezv+/Wqd7cOO9Z5oddZhIUoppZRSSkWIhmullFJKKaUiRMM1PB3rAsSIHettxzqDPettxzrbiR3/frXO9mHHek+oOtt+zLVSSimllFKRoj3XSimllFJKRYiGa6WUUkoppSLE1uFaRJaLyCci8pmIPBjr8kSbiBSKyEYR2SUiVSJyX6zLNJZEJE5EPhaR38a6LGNBRNJEZK2IVIvIbhFZHOsyjQUReSD8810pIi+IiCfWZVKRYbc2G+zdbtutzQZ7ttsTsc22bbgWkTjgcWAFMBP4uojMjG2poq4P+K4xZiZwCbDKBnUe6j5gd6wLMYZ+DLxpjJkBXIQN6i4i+cBqoNwYMxuIA26JbalUJNi0zQZ7t9t2a7PBZu32RG2zbRuugUXAZ8aYfcaYXuC/gBtiXKaoMsbUG2O2hl8HsP7R5se2VGNDRAqAa4FnYl2WsSAiqcAy4BcAxpheY0xrbEs1ZpxAgog4AS9QF+PyqMiwXZsN9m237dZmg63b7QnXZts5XOcDNUP2a7FBg3WciBQD84APY1uSMfNvwF8DoVgXZIyUAI3Av4c/Vn1GRBJjXahoM8YcBn4IHALqAb8x5q3YlkpFiK3bbLBdu223Nhts2G5P1DbbzuHatkQkCfg1cL8xpi3W5Yk2EbkOOGqM2RLrsowhJzAfeMIYMw/oACb8GFUR8WH1ZpYAeUCiiNwW21Ip9fnZqd22aZsNNmy3J2qbbedwfRgoHLJfED42oYmIC6uBft4Y83KsyzNGlgDXi8gBrI+SrxSRX8a2SFFXC9QaY473cK3FarQnuquB/caYRmNMEHgZuDTGZVKRYcs2G2zZbtuxzQZ7ttsTss22c7jeDEwTkRIRcWMNoH8txmWKKhERrLFcu40x/xLr8owVY8z3jDEFxphirL/nDcaYcf+b8akYYxqAGhGZHj50FbArhkUaK4eAS0TEG/55v4oJ/kCQjdiuzQZ7ttt2bLPBtu32hGyznbEuQKwYY/pE5F5gHdbTqc8aY6piXKxoWwLcDuwUkW3hY39rjHk9hmVS0fMd4PlwENkH3BHj8kSdMeZDEVkLbMWaZeFjJtiyunZl0zYbtN22G1u12xO1zdblz5VSSimllIoQOw8LUUoppZRSKqI0XCullFJKKRUhGq6VUkoppZSKEA3XSimllFJKRYiGa6WUUkoppSJEw7VSESIil4vIb2NdDqWUUqenbbaKFg3XSimllFJKRYiGa2U7InKbiHwkIttE5CkRiRORdhH5VxGpEpHfi0hW+NwyEflARHaIyCsi4gsfnyoi60Vku4hsFZEp4dsnichaEakWkefDK04ppZQ6R9pmq/FGw7WyFREpBVYCS4wxZUA/cCuQCFQYY2YBm4D/Hb7k/wF/Y4yZC+wccvx54HFjzEXApUB9+Pg84H5gJnAB1upqSimlzoG22Wo8su3y58q2rgIWAJvDHRQJwFEgBPwqfM4vgZdFJBVIM8ZsCh9/DnhJRJKBfGPMKwDGmG6A8P0+MsbUhve3AcXAu9GvllJKTUjaZqtxR8O1shsBnjPGfG/YQZHvjzjPnOP9e4a87kf/jSml1OehbbYad3RYiLKb3wNfFZFsABFJF5HJWP8Wvho+50+Bd40xfqBFRC4LH78d2GSMCQC1InJj+B7xIuId01oopZQ9aJutxh39DU3ZijFml4j8HfCWiDiAILAK6AAWhd87ijXGD+DPgCfDDfE+4I7w8duBp0TkB+F73DyG1VBKKVvQNluNR2LMuX6SotTEISLtxpikWJdDKaXU6Wmbrc5nOixEKaWUUkqpCNGea6WUUkoppSJEe66VUkoppZSKEA3XSimllFJKRYiGa6WUUkoppSJEw7VSSimllFIRouFaKaWUUkqpCPn/0ieK8GWZSJYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot the development of the accuracy and loss during training\n",
    "plt.figure(figsize=(12,4))\n",
    "plt.subplot(1,2,(1))\n",
    "plt.plot(history.history['accuracy'],linestyle='-.')\n",
    "plt.plot(history.history['val_accuracy'])\n",
    "plt.title('model accuracy')\n",
    "plt.ylabel('accuracy')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'valid'], loc='lower right')\n",
    "plt.subplot(1,2,(2))\n",
    "plt.plot(history.history['loss'],linestyle='-.')\n",
    "plt.plot(history.history['val_loss'])\n",
    "plt.title('model loss')\n",
    "plt.ylabel('loss')\n",
    "plt.xlabel('epoch')\n",
    "plt.legend(['train', 'valid'], loc='upper right')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "iCaoy8l4oaCW"
   },
   "source": [
    "#### Prediction on the original test set after training on original data\n",
    "\n",
    "Now, let's use the fcNN that was trained on the flattened MNIST data to predict new unseen data (our testdata).\n",
    "We determine the confusion matrix and the accuracy on the testdata to evaluate the classification performance.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "id": "z1U4S7o5oaCY",
    "outputId": "a844c91b-bc54-475a-94f2-4e963d453967"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 966    0    1    2    0    5    3    1    1    1]\n",
      " [   0 1121    4    0    0    1    4    1    4    0]\n",
      " [   5    2 1000    3    3    2    2    6    9    0]\n",
      " [   0    1    7  972    0   10    0   10   10    0]\n",
      " [   1    0    3    1  950    1    5    3    2   16]\n",
      " [   2    1    0    7    3  867    5    1    5    1]\n",
      " [   7    2    1    0    3   12  929    0    4    0]\n",
      " [   2    7   11    6    1    0    0  993    0    8]\n",
      " [   3    1    5    8    3    5    5    6  936    2]\n",
      " [   5    5    1    7   15    9    0   10    6  951]]\n",
      "Acc_fc_orig_flat =  0.9685\n"
     ]
    }
   ],
   "source": [
    "pred=model.predict(X_test_flat)\n",
    "print(confusion_matrix(np.argmax(Y_test,axis=1),np.argmax(pred,axis=1)))\n",
    "acc_fc_orig = np.sum(np.argmax(Y_test,axis=1)==np.argmax(pred,axis=1))/len(pred)\n",
    "print(\"Acc_fc_orig_flat = \" , acc_fc_orig)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9n6WX-ClgypT"
   },
   "source": [
    "\n",
    "####  *We get an accuray of around 97% on the test data!*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Wv898fZgjb-Z"
   },
   "source": [
    "#### Play the deep learning game and stack more layers and change the activation function from sigmoid to relu\n",
    "<img src=\"https://raw.githubusercontent.com/tensorchiefs/dl_book/master/imgs/paper-pen.png\" width=\"60\" align=\"left\" />  \n",
    "*Exercise: Try to improve the fcNN by adding more hidden layers and/or changing the activation function from \"sigmoid\" to \"relu\". What do you observe? can you improve the performace on the testset?*\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "cJWBu7Troo5i"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "nb_ch02_02a.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}