DeepDive.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Classroom\n",
    "Contact and Copyright: Mark.schutera@mailbox.com\n",
    "\n",
    "\n",
    "\n",
    "I am happy to have you. In the following, you will interactively experience and explore the world of artificial intelligence, machine learning, or, even more specifically, deep learning. You will get to know...\n",
    "- the general design of a deep learning pipeline\n",
    "- the basic principles behind deep learning \n",
    "- the fundamental neural network architecture\n",
    "- the process of training a neural network\n",
    "- the deployment of your trained neural network\n",
    "- the validation of your model\n",
    "\n",
    "After this course, you will know the basics of...\n",
    "- the look and feel of the programming language python\n",
    "- the deep learning packages tensorflow and Keras\n",
    "- the intuitions and approaches within deep learning\n",
    "- much more\n",
    "\n",
    "This course will enable you to set up your own deep learning projects, train your own models on your own data. You will be well on your way to become a deep learning engineer, machine learning researcher, deploying the power of deep learning to generate progress, make the world a better place and solve the pressing problems of today and tomorrow. This sounds exciting - it is! And you are precisely in the right place - Buckle Up!\n",
    "\n",
    "<img src=\"graphics/welcome.svg\" width=\"500\"><br>\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There will be emojis along the way. These are for virtual lectures with this notebook. They help your lecturer stay aware of how you and others progress. Once you reach a smiley, please post it within the chat function of your virtual meeting tool.\n",
    "\n",
    "😍 Smiling Face with Heart-Eyes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Getting comfortable in the driver's seat!\n",
    "\n",
    "The interface you are seeing in front of you is called jupyter notebook, an excellent interactive way to present and communicate code. Think of it as your cockpit. The nice thing is that in the back, we already got your engine preheated and ready to rumble, waiting for you to kick the pedal to the metal. Thanks to [binder](https://mybinder.org/) you can execute the code, on your machine using binders compute power.  \n",
    "\n",
    "In principle, this notebook is structured in cells, which get highlighted once you select them. There are two types of cells. The markdown cells (such as this one) are for everything that is not code. Here you will find explanations, descriptions, figures, and many more. The other type of cell is for code. Here you can interactively change code and run it within your browser. \n",
    "\n",
    "Navigate to the next cell by using the down-arrow or click on the next cell...\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# Great job. This is an interactive python cell with an easy print function. Feel free to change and play around with it.\n",
    "\n",
    "print('Ready Player One!')\n",
    "\n",
    "# A hashtag is used to comment on stuff that is not executed as code. This is useful if you want to add an explanation or further relevant information to your code.\n",
    "# Run a cell by clicking the Run button above, or by Ctrl + Enter (on your german keyboard, this is Strg + Enter) \n",
    "\n",
    "# Notice the output below the cell.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# We will use this cell to import all the packages you will need in the following - think of it as turning on all your systems\n",
    "# in your cockpit\n",
    "\n",
    "# This makes sure that if you change code in your external scripts, they will be updated\n",
    "\n",
    "\n",
    "import checker\n",
    "import generator\n",
    "\n",
    "from IPython.display import display, clear_output\n",
    "import numpy as np\n",
    "import time\n",
    "import math\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sn\n",
    "import pandas as pd\n",
    "\n",
    "import importlib\n",
    "importlib.reload(checker)\n",
    "importlib.reload(generator)\n",
    "\n",
    "# now go ahead and Run the cell. This might take a while...\n",
    "# while the cell is running, you will see ln[*] next to it. Once it finished, you will see the number of execution\n",
    "# In case you want to interrupt the run of a cell, press Ctrl + C (on your german keyboard, this is Strg + C) \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Hint: \n",
    "Below code cells you sometimes find, what we call hints. So in case you get stuck, or are in search of additional information this is the place for additional elaboration on the topic or links. <br>\n",
    "- [print()](https://docs.python.org/3/library/functions.html#print) <br>\n",
    "- [import](https://docs.python.org/3/reference/simple_stmts.html#import)\n",
    "\n",
    "##### Side Quest:\n",
    "Another feature is the side quests. These are things you could try, think about, and tease your brain with. They are not necessary for you to learn the fundamentals. They are nevertheless, valuable for your holistic understanding of the subject. Also, they are fun."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Making sure you are still on track\n",
    "\n",
    "Next, welcome to the world of what we call sanity checks.\n",
    "Every now and then, you will find sanity checks below your interactive code cells. \n",
    "These are to make sure that your code does what it should (this is usually called a [unit test](https://en.wikipedia.org/wiki/Unit_testing)).\n",
    "Once your code passed the unit test, it is safe for you to go ahead with the notebook.\n",
    "So do not touch the code within a sanity check cell. Instead, just run it and fix your code in the respective cell.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# Write a function that adds a to b and returns c\n",
    "\n",
    "def add(a,b):\n",
    "    c = # '''your code goes here''' \n",
    "    # the code goes here flag has to be removed and indicates that you need to pick your brain and write some code.\n",
    "    \n",
    "    return c\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "##### Hint: \n",
    "An addition in python can be done by simply writing a '+' sign connecting the two input arguments. An alternative is to use a dedicated function that takes an array of values. <br>\n",
    "- [sum([a, b])](https://docs.python.org/3/library/functions.html#sum) <br>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# After writing the add function in the cell above, go ahead and run this cell.\n",
    "\n",
    "'''\n",
    " ____   __   __ _  __  ____  _  _     ___  _  _  ____  ___  __ _ \n",
    "/ ___) / _\\ (  ( \\(  )(_  _)( \\/ )   / __)/ )( \\(  __)/ __)(  / )\n",
    "\\___ \\/    \\/    / )(   )(   )  /   ( (__ ) __ ( ) _)( (__  )  ( \n",
    "(____/\\_/\\_/\\_)__)(__) (__) (__/     \\___)\\_)(_/(____)\\___)(__\\_)\n",
    "\n",
    "Do not touch, just run!\n",
    "'''\n",
    "checker.test_add(add)\n",
    "\n",
    "\n",
    "# If your code is working correctly, you will be prompted with the message \"Everything passed, \n",
    "# you are ready to go.\"\n",
    "# This means you can continue with the next cell\n",
    "\n",
    "# If there is a bug in your code, you will be prompted with an AssertionError, \n",
    "# outlining how your code failed.\n",
    "\n",
    "# Go ahead, break your code and see what's happening to get a feeling for the sanity checks. \n",
    "# Then feel free to move on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you know how to drive this thing, you are ready for some real action."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Talking about brains\n",
    "\n",
    "Your brain is the most astonishing multi-purpose computer around. And neuroscientists and engineers alike have been trying to understand the mechanics of brains, to apply their findings to other fields, such as computer science. This is how the idea of artificial neural networks came into being. <br>\n",
    "\n",
    "Your brain comprises approximately 80 billion neurons, or brain cells, which are heavily interconnected with what are called synapses. Neurons communicate with each other by sending electrical impulses through these synapses. If a neuron receives enough of these messages, the neuron itself sends an impulse to the neurons it is connected with. In this way, the brain can process complex data, extract features, and recognize a pattern. <br>\n",
    "\n",
    "And this is not yet, the most exciting ability of the brain. Your brain is further able to learn, adjusting to newly perceived data, patterns, and behavior. This happens by either strengthening or weakening the connection or synapses between neurons, influencing their interplay. <br>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "🙃 Upside-Down Face"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# pip install --upgrade tensorflow==2.7.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we get to concentrate on the training hyperparameters and process and the neural network architecture design. Not to forget adding layers - juhuuu!\n",
    "\n",
    "<br>\n",
    "\n",
    "In the following, you will be introduced to...\n",
    "- .. [tensorflow](https://www.tensorflow.org/) \n",
    "- .. [keras](https://keras.io/)\n",
    "\n",
    "While you will clamber on the \"Hello World!\" of deep learning, the [MNIST](http://yann.lecun.com/exdb/mnist/) dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tensorflow version: 2.7.0 (Expected 2.7.0)\n"
     ]
    }
   ],
   "source": [
    "# There is this joke that once you have added this import line to your code, \n",
    "# you are allowed to sell your product telling everyone that you are using AI. Life is easy, sometimes.\n",
    "\n",
    "import tensorflow as tf        \n",
    "print('Tensorflow version:', tf.__version__, '(Expected 2.7.0)')\n",
    "\n",
    "\n",
    "\n",
    "# Keras is a model-level library, meaning that it is built upon tensorflow (using it as a backend) - allowing for\n",
    "# high-level building blocks. Making it even easier to design neural networks.\n",
    "# We will access it as tf.keras\n",
    "\n",
    "# The tf and k abbreviations are best practice (same for numpy np and pandas pd), \n",
    "# since you do not want to type T E N S O R F L O W all over your code.\n",
    "# They are prevalent all over the industry and academia in a way that you'll risk a fight if you import them differently.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Hint: \n",
    "The [space versus tabs](https://www.youtube.com/watch?v=SsoOG6ZeyUI&feature=emb_logo) war is a birthday party in comparison to breaking import guidelines."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring the MNIST dataset\n",
    "\n",
    "Before any data science or machine learning project, it is essential to get to know your data. This will enable you to detect issues, noise, pitfalls and understand what your model will learn at the end of the day. With the MNIST dataset, you will be working with a beautiful, cleaned, easy to understand, low-memory, large-scale dataset. \n",
    "Full disclaimer, MNIST is excellent for learning and research purposes, yet this is not what you can expect in real-life. \n",
    "\n",
    "This initially becomes obvious as MNIST is so commonly used that the Keras packages got our back with loading the data in one line. We are loading pairs of samples and ground truth annotations for both the training set (for training our model) and the test set (for testing our model). \n",
    "\n",
    "<img src=\"graphics/jars.svg\" width=\"650\"><br>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Loading the MNIST dataset in one line\n",
    "(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()\n",
    "\n",
    "# Printing the shape\n",
    "print('x_train:', x_train.shape)\n",
    "print('y_train:', y_train.shape)\n",
    "print('x_test:', x_test.shape)\n",
    "print('y_test:', y_test.shape)\n",
    "\n",
    "# Plotting data samples\n",
    "print('\\n Plot of the first 25 samples in the MNIST training set')\n",
    "numbers_to_display = 25\n",
    "num_cells = math.ceil(math.sqrt(numbers_to_display))\n",
    "plt.figure(figsize=(10,10))\n",
    "for i in range(numbers_to_display):\n",
    "    plt.subplot(num_cells, num_cells, i+1)\n",
    "    plt.xticks([])\n",
    "    plt.yticks([])\n",
    "    plt.grid(False)\n",
    "    plt.imshow(x_train[i], cmap=plt.cm.binary)\n",
    "    plt.xlabel(y_train[i])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The shape of the data arrays already gave away that we are using way more samples (60k) for training than for testing (10k). A typical ratio is 80% training and 20% testing as a rule of thumb. Actually, the 80% training is further split into the real training and a validation set, but that will not bother us here. \n",
    "\n",
    "[Deep dive on dataset generation, split and pitfalls (slide)](slides/slide_datasetgeneration.pdf)\n",
    "\n",
    "The other thing the shape gave away is the samples' dimensions $28x28x1$ and the dimension of the ground truth $1$.\n",
    "\n",
    "Along with the shape, we also plot the first 25 samples of the MNIST training set together with their ground truth labels (annotations). We can start to guess what the dataset is all about - it is a dataset purposed for digit recognition.\n",
    "\n",
    "\n",
    "### Data preprocessing\n",
    "We already talked quite a bit about [activation functions](https://cs231n.github.io/neural-networks-1/) (or non-linearity) and how they take a single, usually accumulated, number and perform a specific fixed mathematical operation on it. There are quite different ones, but you will soon find out that there emerges a shared pattern within deep neural networks that makes them beneficial in practice.\n",
    "\n",
    "- They usually map any incoming value so that they are asymptotically bounded from above and below. For example, the [sigmoid function](https://en.wikipedia.org/wiki/Sigmoid_function) does map to a range of values between $0$ and $1$.\n",
    "They are usually most sensitive around zero because they tend to saturate to the bounds of the mapping for large numbers.\n",
    "\n",
    "Thus we can speed up the training process by preprocessing our data so that the first layer's activation function has an easier time picking up the pattern.\n",
    "\n",
    "For the MNIST, this is quickly done by normalizing the grayscale values $[0, 255]$ to a range between $[0, 1]$. Remember to do the preprocessing for both the training and the test samples.\n",
    "\n",
    "\n",
    "<img src=\"graphics/standardizedBeans.svg\" width=\"700\"><br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "x_train_normalized = # '''your code goes here''' \n",
    "x_test_normalized = # '''your code goes here''' \n",
    "\n",
    "# in the next step, we also need to reshape our input to fit our input layer later on. \n",
    "# This is due to keras expecting a definition for how many channels your input sample has, as we \n",
    "# deal with gray scale this is 1.\n",
    "x_train= x_train_normalized.reshape(-1, 28, 28, 1)\n",
    "x_test = x_test_normalized.reshape(-1, 28, 28, 1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    " ____   __   __ _  __  ____  _  _     ___  _  _  ____  ___  __ _ \n",
    "/ ___) / _\\ (  ( \\(  )(_  _)( \\/ )   / __)/ )( \\(  __)/ __)(  / )\n",
    "\\___ \\/    \\/    / )(   )(   )  /   ( (__ ) __ ( ) _)( (__  )  ( \n",
    "(____/\\_/\\_/\\_)__)(__) (__) (__/     \\___)\\_)(_/(____)\\___)(__\\_)\n",
    "\n",
    "Do not touch, just run!\n",
    "'''\n",
    "checker.test_normalize(x_train, x_train_normalized)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "##### Hint: \n",
    "In python a division can be done like this in python:  $3, 5~/~ 2 == 2.5$ <br>\n",
    "                "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Designing the neural network architecture\n",
    "\n",
    "Remember how much work it was to set up a single neuron - right. Now enjoy how Keras is helping you with this. We will call our neural network [marvin](https://en.wikipedia.org/wiki/Marvin_the_Paranoid_Android). Feel free to change it to whatever you like, in case you are not okay with it.\n",
    "\n",
    "Our model will follow some design requirements, it will be [sequential](https://www.tensorflow.org/api_docs/python/tf/keras/Sequential?version=stable) or it other words, feed forward, meaning that once a value passed through a unit, the unit will not see a value during that forward pass again. Or in other words, we do not have any recurrence in the architecture (feedback loops and so on). \n",
    "\n",
    "That's a lot to tackle, there will be quite some hyperparameters. This is for the visual learner - need a sandbox before we get started? I got your back [tensorflow sandbox](http://playground.tensorflow.org)\n",
    "\n",
    "I really encourage you to take the time to play in the sandbox. It's worth to build up some intuition. At least play with the number of layers and the number of units per layer. Observe how the features increase in complexity the deeper you go with your layers.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# This is the moment where you define your model's architecture\n",
    "\n",
    "model = tf.keras.models.Sequential([\n",
    "  tf.keras.layers.Flatten(input_shape=(# '''your code goes here''' )),\n",
    "  tf.keras.layers.Dense(128, activation='relu'),\n",
    "  tf.keras.layers.Dropout(0.2),\n",
    "  # '''your code goes here''' \n",
    "])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Adding layers to Marvin is as easy as adding a layer while handing over that layer's specification. In the end, the layers will be stacked following the order in which your code gets compiled.\n",
    "\n",
    "Initially, we need an input layer, which can take in our input samples. Go ahead and set up a 2D convolutional layer, with an input shape fitting our image samples, with a kernel size of 5, 8 channels or filters, a stride of 1, and an activation function that does identity mapping, to begin with."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# “Don't blame you,\" said Marvin and counted five hundred and ninety-seven thousand million sheep \n",
    "# before falling asleep again a second later.”\n",
    "\n",
    "\n",
    "marvin.summary()"
   ]
  },
  {
   "attachments": {
    "image-2.png": {
     "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgoAAAERCAYAAAAaOI+QAAAgAElEQVR4nO3dP2vjzLs38K8ezouwwSkM227jKjIs/DBsc7qV8ELkalk4TU4V0tkJB0fuQrotw11Jhg3WvoCFNDdYqdykDaiwwX4X8xSSrJGssWVbSZzk+4HAfa/s0b+RdM01Y40mhBAgIiIiyvH/XnsDiIiI6HAxUCAiIiIlBgpERESkxECBiIiIlBgoEBERkRIDBSIiIlJioEBERERKDBSIiIhITaxhAwKqv/P/E8aa5f/9v/+t/i7+I9rPVrYh/u98Tdn4H/E/69bd/s/zlc1jVvox+0/7tc7Hpv1qi/+81vngMdt+v3jd85iVcswO97o33Pm6x/1aGt/MSERERCrseiAiIiIlBgpERESkxECBiIiIlBgoAMDDANp3F4vX3o7IYmjCHB7K1hC9JB8DzYQ7e+3tIKLY3oHCYmhCu/LL2JbXMXNh6hM4NxYq2X/XBniNPaucnKNhVTF4eIWV00FZDE1omhb9bfsAXcD9rj1TPdqvbP9Kk/ZLe9v3EHoV6Wtjl+vjFc1cmFHj9C08Qz94RmEB96wDuL9g1V57W2Q6ulMHE/11AhU6DIuhieofE3MhIITA3AU6R2+/TiyGJpqPznK/hBAQF/prbxa9RW2pHvkNdI7eULDwub5snBqf6q+6KRvt/MPKyNw1BPpj9Qd8O/VbTtuPF4yFnfr/vH8bp97lkPod6NQRBmwxlj/TdsRWvxT17ZzvjBXvjzCEMy2y3XPhtA3hTOVybDFesw7V71vH/f1++0pv2VjYyzoXmwunLdezdD2Ur8Vxf/1vqeeuIQx3nvpcUtZ+Za8Xlq3+bLTfvpP8Jjx7jSrvKZv2K1n/8vvr7l100Oaukakb2bqVOdc5n7V9uT5nrre1zy5bOK4R1ftxtJ7s9Zq71eltSv1lnxOH45kzCj4Gf1tJq8G30Vu2knW0+kDvr9Q+erhHDzZax0CY2mxi4s6j749z0vE9NLVr1Kfhcvuug9stUqH+3x6Mb610lwN0dIWAmDowYGO8bPWMoqzDpu0GAA+do3i7BMb9HprL1FKR/Yq25KsN78/9wYydoBc0CzBBA/VUpquC+udM3VPQLwSEmMNpA7aftNxHJ0lt96wqrj/Nc67N/ctWq6D1zYBnVdeMw/HQ0QOci7zret09ZfN++VdVdD6Po+/P4Tw2OR7onVoMb4EbkZxrdHCaOdc9XVvWlXHfQ+efZU3ZUM966DydQ/g2PKuJ4Cz8/ujfTXWpAuu3wNw1YLjzzHXUxaHm1Z45UNDRlVOKxy3YmCCIUkP6DwfG5X1yEf/twXB/hgdrdo8RHPxa3nx0/HSNlZuk7acf4JOg6EW/QPAINOpFbm6ZvVq33SvbFT7w8RiED/yC+wUAqNVh3AUItt5CevNmAbx2HdmEZP2TUd46+uPk4Z65Np9T5WQEMXUAqxr2La8MJDbgTOObZva6Xn9PAbBmv3zcX9oYL79fgXXGYPy9WAxP0bmzcR6d+8pJV+pSjgLUp8zdVKorqfv0xnpmwPkRLW87+HmMHQUI7oxMg+DwPPsYhfSgpSZ68sJaC2a7h/sHIL6I45OMWQDvroOqNFilanmZ0uVWfNjSKdaqAcITtONOrdtulfiBX2i/4vXU0XihmzcdGEWQGDx5z9if6b1cXatZGEktveoWA4fX3lNyRfs1CzBBD015AJy++dt0wKR7adVqYCy3ymcuzA33WfurHAx0IX4ng9q3r2cFRNtUtTx4VjUq10PnSDvozNazBgqLoYnmpZy+H8NOfSKM8np//TB932+lUy/tzICnUgc91VFv7/rdDdudR24dFt2v3PQzfQi5QeLuWbBiXqNlU4F148AoGBBvvqfkkfdL/m709zvziyd6O1L3Ujl172Nw1AGWXbxhur+o3epZATULI7m7QeriLt7IfXnP/6sH6QHpX61GZZWTc9iX1zBvJkkqBwhTPXerfUrlCft7lV0VtToaiLMGOd9WbfcKHwNdGguxzX4p0s/0Eeho9T10zpK0/GJ4ig7Sac5l/X0Y5LSYio9p8K+a6LVNtKRAoayyN1n8O4K3TUC84Z4iS+1XlAlsHvhP0ag8y6B65uJUlblV2aKebU8KXt/APb6cQOGymfub6MrJOWwpNXT9ycmJysIboof0TSocVBgO9JPLLvM34fpXG551q0h56uj6Nnq66je6qu0OJd8LBy6OpDEJRfcrf7AlfRT6hcD4s5Ra/WNivmz9hi3xZT//TR3jnBaTfjGGLV2fqfSm9O/NR6fcspV8DFK/fc9JGa9R6J6ybr9+hwMY5fUfcsqXdhWN/Yrvw0cBzC0yCsWeXbuSur1nAbYMX17FQcwe6V+FI09fPvWygPu9itG33dadv91hmcGZQHfnAS4I+7KOApwf8EhYersWQxPVp/N39/6C97pfRK/pv157A/AwiPqCXqPdHLacRkencL+Mtnvp0rNud9i/1vAFgwQiInpVr/dmxodBNOp4Iv0U6hXULIz8RqoveK0X2O7F8BoTd75fRoKIiKgEB9H1QERERIfpg8/1QEREROswUCAiIiIlBgpERESkxECBiIiIlBgoEBERkRIDBSIiIlJioEBERERKDBSIiIhIaUOgsDqBi6Zp0L5HbzHMzPednRRqMTRXly0nQHrOsrNziWcndlq/7ucsm8es/GP2eudjw7pf8XzwmL2tOszr/h0dswO+7nfFNzMSERGREjMKjJLf/DFj65jH7K3XYV737+iYHfB1vytmFIiIiEiJgxmJiIhIiYECERERKTFQICIiIiUGCkRERKTEQIGIiIiUGCgQERGREgMFIiIiUmKgQEREREqlBQrpt00NEL3/TXqLVPxmKZIthuZWb81KHec937b10vyr5O1jtC0fA15DhbCeEZWrnEBh5uLUApypgBACQnShAwB0dIWAEGPYOxW8gPt9zUU/c2Eug5IX9DAo5/WYDwNUrQbGF7r0j3mv6Ez2sXIyghACc9fYbZ2bjtnDIHkdaMn0Hw4mOh9221vA/d7ExP0Fqxb9U/yqVrnuPeO5K094TatfUYvV19Aq9il+1W32/sB69oGk7sU858+lpEAhgNc20apt/ui7cNyNAiIBIeZwHpswh9venn0M9B5sPw6qZIYUdMmB1xtXszDyG+icHfrD7LAshqfowMGvk0pmiQHj8f7lA+W9VGD9luu2gPBtAA3UawDgY3A0gjmVri90cJq9vh4GaD7asNs5q2A9+xhmLkx9srxXzl2gc/QKDccPoJRAYRFMdv9uqssiiQjD1kIVnTugpyeti/CBHLW6jzrw0ENTEVHmd4cAYavGhDuTW++7VrAK6p932e9r9PpjdI93WukamYzEssW54ZjFrTi9B9x1UFUeN0Um5WEA7cpPHfPc4On4Jxx0cMvUcEE+bi3AubGQDROABsxvE1yrgtRsy1w6X4uhCXO4SE1Ak26ZrznXJfP/9mC4P8NgeBZgsgwagPzrKwqyz36iriqU9ezd8//pwOufL7NslZNz2Ojhnue8fGIP4z4EsPpnuPPsJ4UNQzjTzD9PHWFLn527hkDbEdK/CKcNYfuKDZg6woAtxnnLfFtAWpYuOywXy20K/391u4tQ7Nta6/arWHlz1xDor+75uC8fj7GwkVnPumMmRHjcUudALhvSOjPHzLfD8x8vzxz/IttOOVTnY3kex8KOl6c+mz334f/H52vuGqlrNXvtrT3XZVqpj9G1Ke9T5npItm39/YH17D3Lnvv4nv5M9fSD2yujoF+IpL+87WAepRJHKylShZqFrvTZyhcTxl2AYJ+Nivh/02n9ysk57LsR7qWMg+2Pomi0gtY3A95T8TUnLecmelJUW0yA4M6QWk1ZHjpHm/to8+gXcjeFjlYfmARlJGB93F/a0niKCqwzG96f+2Tb2g7m8fLjFmxMEOT0GVbqDeAxYFq4gEUwAT7Xc7IJMR2tz6st58XwGr22g5/Hyee6fuZ89cfLazV97RU41yXx/+kAcTYhXtdvgfm3UZjV0oGxGKXGZlyntk2N9exjiLPPo29zzN3t7uNUzCv/PDKT3jzqwCup3OAx3WWhaU30Sik7FA8qFEJg/ul6u0FkswDrO2syYxR+56WdFTIDLZuXRb+4wSzAJNVlEXVT7KJWLy0gfO+Cp81XhP7DweRvTtfA2gBjjTLP9dr1uLi+NGB+kbcyvCdUn86jRsgEzWW3yALuWQeN3HE9OVjP3r2eruH603zZQA2ePBiflB1StKNXDRT8qyo6SDIRYupgx7H8uWw/M2hKbpmUaOtMSK2ORvmbEQ3u6aX2e9wvcwU2xiJzTLcJYpbbGcBr19X9y7RU/1Tgiqi1YD5ew81WwExreruxRCWd6zWyfcwAgIdbdO6SjEHlZIS5a6B342Ixu8coNWZJGsOUF6iznr1j0dgVKSsWNxAb9TJrKQGvnlGA1OoJWwvp9lNYGXp5rSUgeuDmDV4JuxJ6+suMgA1veK0tfplQR73t5abl9yd1aTwMVjMKymOWLDcyXTThv7dgtntoljCobXM6nWLF0ucVWGcNjP4kgUAYvMpdEj5uLQ/2WYGHfdFzHWevdqkTUTbB+ZF31chdVgvc//HC+lKzMEoFL3M47ahBkBPEsJ69b/pXG7hsJoNwoyCzVfoAcXreQGGZBm+it+x3T36ZoP9wYFw2l62D4NtqRkG/GMNefiY7kj7sd01aGEnZYUtkkk6flvQb8/SvKTQ0MYYo0GeaiAKZm223J+mqqVoeEB+X+EZds3Del8Y33NThrGQU1McsLuOXC2mMRBxsVWD9Dn8KKu/79j8LDW/89td38YPP53fcgn1XYPT+cQuNOynMrlkYTR1Mlue5iYk7L/grm4Ln+rgVvh9l63EAUaMgb2zPcTf6mZuUNfi87fUVroP17J077kLI9zIdGL+Xn5IfGE0IIV57Iz4mHwOtCfjiGX4iecAeBtBu6piXnMZ+zxZDE9U/5gEeswXc77s+yJ8Z6xlRaRgovKaHwceKgmcuzKMRzOnzjBV5v8IH8ujbvPgvip7ZYmiGWa3+AQYJrGdEpWKgkLG8AeYy4JR881kMzXCE96HdbJ+Bf6Xh/usHy6CUxsdAu0adD7+NWM+IysVAgYiIiJRe/1cPREREdLAYKBAREZESAwUiIiJSYqBARERESgwUiIiISImBAhERESkxUCAiIiIlBgpERESkxECBiIiIlBgoEBERkRIDBSIiIlJioEBERERKDBSIiIhIiYECERERKTFQICIiIiUGCkRERKTEQIGIiIiUGCgQERGREgMFIiIiUmKgQEREREolBwoLuN81DB7KLfWw+RhoJtzZa29HMYuhCXO4eO3NoK29rXr2mvyrj3YPInpeHzejMHNhahq06G+3G8sC7vcmJu4vWLXonx4GyzLjv0N6MFdOztGwqryRvik59Syuv1d+8rGHAbTvLg6ntq3nX2nQ8oKf1DW0unwxNNdeX/oPBxOdQdVHEdeH1Xta2HBd1hX5WgEQBt+b79NhPV0tP/73/GdIsbLfig8aKCzgno1gTgWEEBC+jd4ON5bF8BQdOPh1UskssTEWUdlTB7CqB1RRdHSnDib6ANnLhg6Tup4ZMB7v3+B5DG+i958cGHnLdCyvn7kLdI6kuvowQPWPiXnq+jpNX7s1CyO/gc7Z2wmaaBdhIHAKE3bOUv+qis7ncVhPxBj2ZVO6D8fB9zx1n14JNh4GaD7asNuZNQ9NNBGXHT9D4npasOw3pIRAQY6cqujcZRavaR3EaXA5MlsXla20lja0PNQqsH6PktbZcQs2PARbBQo+bi3AubGQvX2n1CyMfBuedSvd0NdEmw8DaFd+qtWUDjIyUbKWfeAXiGRrFs77PVwfTPBCauvqWQPmt4n6PGayZnKLavO1t6k1tsceXV2jPhXofslbqqMrutCj/6t8MWFgsrw2F8EE+FxPjkWtjkZeMcc/4aCD2zd8c6b1FsNTBGcCo5P66sKZi+tLA86PuCbp+Oka8P7ch8+Qh1t07mycx8F3zcJ5H+j9leu5j4Heg332E9k1BE8ejE/Sv9bqSdBbqOy3Zc9AIRM5iTkcOfKauTD1CZxly72Rbh0A8Kwqrj/No9aDgd5NEgz4V3LZAuK3dLMsUPazerhHr22iVdv80bASxTe77DEbr3YFXDZRfTpfRqpykLEYnkpRsoCQbqqFyo7oX+3koqHDtaGe1U/O0cg9jz4GRx00fKFoUW269uTW2BzOY7O0rJh+IQXpW6p8MWFI++FfNRXHp4LWN+NN35xpvcrJCN1jxcJZAE+uFw8DVC0PuAsQIAo4+63lvXMxNNG8BPAYpJ4/vf44dx36Vxve8t66gHvWgReVV6Tst2a/QOHhVpESDS3+HQFyv+rxTzjtHu7lB1d/jFH0/coXE0Z0ImOqh1mhsgsKbzYOfqoqXd76sy2bdeRWz+weo9QxCyPd1A2t7WB+EVWz4xZsqUUFALhUpJuLlC1tU/ZY0+HZXM90tD6vtpwXw+tMndbR9TPBofLa83F/aWN8sWzXwzp7jcAyvgGfJ9d5zcJIzGH+qULTtDD9+zs/q1epN970zZlKEGfVdGAsxqv30igrXf1jYj51kmtg5uI6dQ1kHHchxBjQw0z66NscIvtZVdlv0LOOUQiePHhWVUqT53RNrKFfzOGgg2pOanTfsmNhtGdjrLjZqARPXvEPzwJMpP/27pJ90jQtjHQLqpyMMO730MzrWtim7FodjexFQwenSD3TfziY5AWDRQPZrFmACZI6pmkaNL23S0l78a+q6EAKmoHoxl9FcJZkOla73yIMhj+2uw6qRwHO48zrLMAEDdTjoPOyCe2mHo53+W2hMgvgteuoRwFqw5eztRkPA2ha2IUWZ25TXePKst+mZx/MaMhdB9GfMl20ogLrdzJoaaKng4X9yg6DhKoFONM1FUKh/ml1GJZyPf+O4MkVtO0kg7HiP1XkmkO/EKmuhVSwULTs7EVDB6lQPau1YD5ew80+ETOt6UUwQXHSgNy8rr9n5l9puQG8/08H3jIdXIH1ew6nrRhv88ZvzrSHWh0GjPS9XaoPlXoDQLp+LbN3s3uM7oCenm6E9vR4nNwC7k0PxjKjraMrxrDvwsze2rJf9CCUZ79AoVaHcTfCfdQq9a/SrfqwH+e0nJ8pZQYt7Vt2EiTs1l9aOK0Z9Y3ZcXR63IJ918FpKf29ddTlMSHblM2b6JtQrJ5VYJ01MPqTBAJhV4LcJeHj1vJgnxV42NdaMNs9NDcNYIwHE5c40BGQggShCODl4xHd1Bv11b166zdn2kOtBbPtSb98iR7u31phfThuwYZcx6Pr46sedW/JQXI49s7208Gy9yRF5g/36MEIG17ryn6rxJ7mriEACADCcOdi3IewfekDvr1cHv7ZYix/tz9OPjt1hLFcPhZ26nth+Slryl5vtWwAAm1HzDd/OVVGal9zt8kQznTz+pfl+HZmO8bCXpYxF047s8397B6vKVv+VD/neNIBUtSz1LWSfC5Vd6aOMBTXz/prT4i8urZaX3LWWYB8z1i5TjLbvFrPV7crr37Hn8tfRu/Cyr02Wx/SdUVZf9fWo6Sc9PLsfTZ7ny9a9tugCSFEeWHHx7IYmuFAlRdMyZZi5sI8CnCuarHRQTnceraA+z36dcQWXWcv4mEQ9hEf3DEjens+6AuXylE5+QUHZXUjvJT4Z3MMEt6KQ6xn4Xs+DjRIiH86vekdJ0RUyLvMKIR9nKqla/o+d1sbBto16juOdXhpi6GJU/xa/iyO3oq3Vc9ek3+l4f7rdgObiUjtXQYKREREVA52PRAREZESAwUiIiJSYqBARERESgwUiIiISImBAhERESkxUCAiIiIlBgpERESkxECBiIiIlBgoEBERkRIDBSIiIlJioEBERERKDBSIiIhIiYECERERKTFQICIiIiUGCkRERKTEQIGIiIiUGCgQERGREgMFIiIiUmKgQEREREoMFIiIiEip5EBhAfe7hsFDuaW+Z4uhCe3Kf+3NeBH+FevG7nwMNBPu7LW34/CxnhGV60NnFPwrDeZwsdf3tX1u3g8DVK0Gxhd69A9hoKVp8t8AbyaMmLkw12yv/sPBROfDbnsLuN+bmLi/YNWif5q5MDUtHWQ+DKB9d7F7jX5ZyuvnYSDV/9Xli6GZukay1zDr2Qeyoa5QOT50oLA7HwNNw/0nB8Y+Zeg92H4XemaJ4c4hhIAQAuN+D823FCysU7Mw8hvonL2dh9khWAxP0YGDXyeVzBIDxuP9G6wb664fHwMdGEf1f+4CnSOp/j8MUP1jYh4tF1MHsE7TDwjWs49h5sLUJ3CmirpCpSkhUAgv+jCiq6JzJy1atjClz2RaPGGrIi8i9DHQBvDjllPOd7MtCznduJLSl1q78feal4BnVZMyCnYB+FfXqE8Ful+2OlApi+E1ev0xusfrP6dfzOG0e7iOWk1xFkTe91SaVT5eeS3OKz/13WxrTH0+VruVkmMcnd+jDjz00FwX4R//hIMObpkaLsjHrQU4NxayYQLQgPltsqwbK9bUhcXQhDlcpM53Ol2fyW6V2D22/vrR0RVJ8Fz5YsLABEFUjxbBBPhcT45FrY5GXjGsZ++e/08HXv98mWWrnJzDRg/3POel2zNQiFOicQt4Dqed/UwPTS28MQgxhn2XXLyLoYnmo7NsHaxGhD00jwKcx2Wjg9P4phin7eOWhW+jVzDdWDkZRa31dOtdXGTb9vn0i1GSAt7JAvd/PNhfi6yvgvpnwHsKlv/iWVVUn86jY2agdxMHUD4GRx00/Gh/xBj2ZTMdDFw2l9/NtsY2nw8VHd2oPAN2ck5E3nGqoPXNQO8v4/5CHu7Ra5toKepb/eQcjT/3OS3nzXXBs6q4/jTPqUeAf1VF5/M4ua4fm3t108n2uX4qX0wY0n74V03F8WE9e98WCB4h3UPDZ1EPwCRgHqls+wUKD7eKlGia7cc3Bh2tfnwifdxaHuyzpKVUOfkFpy1HhAacady6CC/88IG5gHvTg+H+TNL2x12M+x5G/76FShIguDNQL3izrH/KJGjbDuZRUFP5YsK4CxAgylK0HfxcZil0dH0bnvwgkb6LWgtm24taa0XORzkq9QbwGDAtXMBKC3qFjtbn1ZZzobrQH2MUXbtyPQJ83F/a0tiZCqyzzHdfxALuWbrViJqFkZjD/BNmApsYQ/zOy7awnn0UYVasitG3OeaukWpUUTleYIyCjZaUXtcvxPLmBBR/WC5JF36jvj5AOVizAJMtPh48ean/N761pNSrhZGUql3/UMmXROA7nI9d1OrSQ4nWyZ77PPoPB5O8lvMOdQFAVD/lLiQNmt7bpaS9+FdVdCAFtkDUnVJFcJZkOpQDflnP3r2eri2zYqOTCoInD8an+mtv1rvzyoMZvWXfYyhAcKf6bES6+aVTTGEq6k1Q9avm8nF/ieKVP9OCWgSbQ5Ik4NrhfOxiFsBr18HLebOVbFKeWgvm4zXc7BNxh7qQkLuQoj9Fy/05+Fcampc2xpl1hv3S8dieCqzf6TE8Kaxn71jYJStnxeJnwJttQB6w/QKFWh3G3Qj30cPFv8oMZlwr7IaQ+0VX06WSmYtTK+7Xj7ohrFtpNPQtOnc2zuVukOWNMuyvzbbN6p+MV0inAkAd9Xb2oZwn6ndrb+7eAeL0sZyGXu1OSJU+PEXnLs74FDsfy+DsYYCqlTmitToaBQYTbU6nU6xY+rwC66yB0Z8kENi2LqTUWjDbPTQ3DWCMf5pW8ntAlkGCWP1FEID08ZjdY3SX/3BgPXvf9K82cNlMBuFGz4DWhgHitAOxp7lrCAACgDDcuRj3IWw/Wjh1hAFbjNd8f9zH8vtIfXYsbMjLpHJz1g0YwpmmlgqnLS3z87ZF/gwE+uu2VLVe1foLlNF2xHzd9uRs07gfHmelqSMM6fupz/p2ZptXz436fGTKbjti7Bqrxyy1jrxjEu5j9lySSngdrByvlWsrul7kOrWmLsyz526lvNW6uFrvctZZwNrrJ7PNq9fB6nbl1yXWsw8hdb9Z/6yh3WlCCPEM8UcJfAy0a9Sn+/7C4FD5GGhNwBcbfyJZmocBtJs65i+YQj7IbXhjFkMzfHfAwR2zBdzv0a8jCv5i6MWwnhGVhi9cejXhKPSe/oFeEBK/ICX3nQCkUjn5lf5p8AEI38VxoEEC6xlRqRgoZGRf4pT+K/kVocddzN3J5r7gd8L/p4OG/14zRM+pAuv3GI3sGwhfUfwukoMLEsB6RlS2A+56ICIiotfGjAIREREpMVAgIiIiJQYKREREpMRAgYiIiJQYKBAREZESAwUiIiJSYqBARERESgwUiIiISImBAhERESkxUCAiIiIlBgpERESkxECBiIiIlBgoEBERkRIDBSIiIlJioEBERERKDBSIiIhIiYECERERKTFQICIiIiUGCkRERKTEQIGIiIiUSg4UFnC/axg8lFvqYfMx0Ey4s9fejmIWQxPmcPHam0Fbe1v17DmxDhO9rA+bUfCvNGha8rdbcLOA+72JifsLVi36p4dBqlxN0w7qplY5OUfDqn6wYO6ty6lnMxempkG78pOPPQygfXdxOLVtvfAazAt+fAzWXD+swyRbDM38e3h8jaju8ZuWp+7lHztI/7CBgn4hIET0N3Uw0bevCIvhKTpw8OukklliYyyVDat6QMGCju7UwUQfwN/8YToA6npmwHi8f4PnMQwE7j85MFaWxUHRPHX9pG/irMMExBnsU5iw85adjWBOo/uwb6OXusf7GBx10PAVz4CZC1OfwIm+P3eBztHHrW8lBApy9F9F5y6zeE1UFqcQ5dZ9+oaQblmstJbKivhqdTS2/pKPWwtwbixkb9/psi2MfBuedStVsjUtpocBtCt/GSWvtqjCiyPZ72zlXd8ai7fpvN/D9cEEL6S2rp41YH6bqM9jpsUkZx82X3uZenZV3i3Sv7pGfSrQ/ZKz8OEWnTsb53FQVNymK3EAABVZSURBVLNw3gd6fzPrZx3+8BbDUwRnAqOTes7SCqzfoyQDd9yCDQ/BLP7uNXptBz+Po+U1C+d9D6N/w/rk/9OB1z9ffr9ycg4bPdx/0CzWnoFCJvoXczhtaXEmKhN+YyUq86wqrj/No6jNQO8mCQb8K7lsAfFbulkWKLuwh3v02iZatc0f3ek7tToMTKJKmj1m49U06mUT1afzZSQsBxmL4Sk6n8fJMRFd6MsvFig7on+14f25fzNp6g9rQz2rn5yjkXseMy0mMYZ92UwFjuuvvapUz+ZwHpulZcX0C+kGnrEIJkC/tazTi6GJ5iWAx2BlH1mHP7bKyQjd482fU/pcTwXf9U8GvKcAwALBI2B/XdZCuN+b6AGYBB+ztu0XKDzcKlKiocW/I0DuVz3+Caedicr6Y4yi71e+mDDuAgTSYtWNoFDZa0ktJr0H+2xDZiD77WCyUtGU5IzF7B6j1DHT8dM10i2mtoP5RVRJj1uwl0FG5FKRbi5StrRN2WNNh2dzPdPR+tzBbaber7SYoKPrZx6symvPx/2ljXFcB1GBdfbCD+UoW1j9Y2I+dfLrKuswFeRfNVPXQ+WLCePyOtXVcGp5Od8LM+Wjb3PM3TiQ+HiedYxC8OTBs6pSmjyna2IN/WIOBx1Uc1Kj+5YdpqbEssVUv9lu0GHwtFqplGYBJtJ/e3fJPmmahmpOBVVu9ckI434PzbyuhW3KrtXRyAYgdHCK1DP9h4NJXjBYNJDNmgWYIKljcTD9Yi6b0G7qmMdZxFkAr13HSoKZdZgKCLNSNsZyRrpm4ZcLdI6i+n0GnLsGjE9JLevp2jLjNjqpIHjyUss/kmcfzGjIXQfRX/F0kfQwnzqY6OlgYb+y0+tpfdsuWqx/Wh2GpbL4dwQPDdTj7EfbCW+C8t+FvrYMWTIQM+xaSAULRcueBZjI20QHqVA9q7VgPl7DzVbfTLp+EUxQnDQgN6/r75lU6o1w3dK6lFkV1mHaYDE0UbUAZyp30YYqJ6NU3caTh0a9AqCC+mekMm5xd0S4/OPZL1Co1WHcjXAfRfT+VbpVr3+14Vmn5fysJDPgsNSy4ePW8qQ+qc0q9UZuv+mKhwGqlgfbjyrqcQv2XQenpfT31lGXx4RsU7aqlUYHpVg9q8A6a2D0JwkEwq4EuUsiquNFuthqLZjtHpqbBjDGg4lLHOgYdrXJ615zbbIO0xpJkKAeExPzrzQ0MV42NPWvNnDZTBqm0SDb1j5jIt4ysae5awgAAoAw3LkY9yFsX/qAby+Xh3+2GMvf7Y+Tz04dYSyXj4Wd+l5YfsqasjdstXDa6bJT21xIuH0r31vZJkM40/zv5q7ftwXajpinPhuXsbrdqeO3qWz5U/2c40kHSFHPUtdK8rlU3Zk6wlBcP+uvPSHy6tpqfclZZwHyPSP/OknXYdW1yTr8wa3ca+X6snofRKquppfn1qNU+UWfLe+TJoQQZQcfH8ViaIaDrV4gJVuqmQvzKMC5WE3H0eE53Hq2gPs9+nXEFl1npWAdJnoxDBT2Et4oR9/mUl/WofMx0JqAv+t4Dnp5h1fPwrSuB/RfIUhgHSZ6Ue8yUPCvtPC317lsjEtthfgYaNeoF+gHOwSLoYlT/DqYBw4V9bbq2XNiHSZ6We8yUCAiIqJyfNi5HoiIiGgzBgpERESkxECBiIiIlBgoEBERkRIDBSIiIlJioEBERERKDBSIiIhIiYECERERKTFQICIiIiUGCkRERKTEQIGIiIiUGCgQERGREgMFIiIiUmKgQEREREoMFIiIiEiJgQIREREpMVAgIiIiJQYKREREpMRAgYiIiJQYKBAREZESAwUiIiJSYqBARERESgwUiIiISImBAhERESkxUCAiIiKlDYGCj4GmQcv+fXexAICZCzNv+ZUPAFgMzdVlmobBw3OXDfhXOd/VTLizzet+zrJ5zMo/Zq93Pjas+xXPB4/Z26rDvO7f0TE74Ot+V5oQQuxVAhEREb1bzCgwSn7zx4ytYx6zt16Hed2/o2N2wNf9rphRICIiIiUOZiQiIiIlBgpERESkxECBiIiIlBgoEBERkRIDBSIiIlJioEBERERKDBSIiIhIiYECERERKR10oCC/ococLl57c4iIiD6c9YHCwyD3VZOaNkDhF0LOXJjbfF6iXwgIITDu7/Dljyx+jWfeaztT5zR+LWhaGKDlL1PJf+1o/FrSBdzv615LmvOZPV85SkRE5VgfKBx3IYSAEHM4bcBw59H/d6G/0AbSdhZDE9oZYOYFVzMXpj6BMw0DsLkLdI7kIC58j/j9JwfGluuNg7rl39SBAQP1GgBUYP3OLPdtAI1oOeBfVdH5PI6Wj2FfNplFIiI6APt3PWQnoVi2BKPJK4468NBDs0grMp7YogSLoQlzuEi1dONJNzaue5kFkSbgyGxbemIPaZ8eBtCu3KhsE+5DdHxS389M7JGz32H527XqMXNx+nQO8dtCPWex/08HXv8cVvRwrpycw0YP98vJSK5Rnwp0v2yxToXFv6PUula25W8PhvszDDhnLq4vDTg/4vBTx0/XgPfnvrT6QEREu9kzUPAxOOqg4YuclqCO7rJlaWO8bE2Olg+PxfAWuBFJ1gIdnJbYivSsKq4/zZct2J6etJ43r7uHphY+OIUYw77r4DYONGYubvFr2Tqeu0DnTHrYX3YQnAmM+x46eoBzMYZ9N8J99ND3r5qYLLMzAuK3hUoZO1yzMLpQ5XoWCB4B+6u+/H/3exM9AJMg3HL9YqR8sG/Hx60F6cGfMXNxfWnj/CTa61kAr22iFa/7YYCq5QF3AYIyNoeIiHa2V6CwGF6j13bw8zj+Fx1d3y7cEqycdKUHUwWtbwa8pxIfDf0xRvHD6LgFGxMEs+Lrtv34wamj1U8eqKhZ6J4kj/bKFxOG/FCTjsmy1Zyx6RhVTkapoKpMYZalitG3OeZuycccUb1Yl034pwPkHZc4O6UDYzFOnS8iInod+3c9fK7v3hrOdFtULW/vzVnPSx48G9dto3Wc/J9+IZKgI9ttcdTBNluuX4QZjGpul8jz6unaMssyOqkgePJgfMrrqNiVj1vLkzIXGVE3g/klU2vuOqgeBTiPx8DMAkykMQxERPQ69g8UHoN0330wKfjFsNsCUgp+7m47hG5b8eC6/dbtX1XRgYN5auDeNqTBfVMHE/0lgoUK6p+RzrJE3RGNeikdH2GJK1mmtOw4CQBArQ4DBpypNEh2FsBr13PHWhAR0cvZK1AIU+5S333cmjyT+txrdTSkAXNZy4fUzMXpM2YU/KsmenI/+L7rXmZSFnDPtssopNTqaOT8806DGTfQv9rAZTMJSh5u0blLZ042muUNzozlnP/Md9ODFiO1Fsy2J43zWMC96cH41ipn7AYREe1OFDIXThvCcOeri6aOMACB6C/3M769XA4YwplGpbqG9O+2cFxDoD+OvjQWtlTu2vLztjhVNgTajpgrl2fWPXWEAVuMc0te3WfblT7v28t1jfvx9o6Fvdzv1f3K26dw+5JjVUz+MUuOqcici/Q+rhyzzPmKPiWc9up3RbS/2eO88r2+6qjG5W53nomI6HlpQgjx/OHIy1sMTVSfziGUvwKgXflXGpqPDuZl/VqDiIgO1kG/wpkOTPRWRwYJREQfx3+99gbQG3LchRDd194KIiJ6Qe+264GIiIj2x64HIiIiUmKgQEREREoMFIiIiEjpAAKFaCbFEmeOBKIXFl3565evzHhJREREsvWBQvRzuNW/ZBbGtyqcdOklXhv9scQBWN4rqdNTc+fVoV2Cxsy8G9mpv7PToOdO7Z2e9tsscQZTIqK3bn2gcNxNpmFuA8ZyboRu7oyIu4mmo+bv8t+48IF9ChN23uKHAapWYznd+LjfQ1N+WD8MoGn3qG8duEnzZsR/vg3EE0rVLIxEevm4j/QruL9L035PHcCqvuhEXUREh2zProfw4TB4iKcu1lbnJ8hkJeQb8Pr0v4+BNoAvtwizLc01ZS9LuVq/XClV9jZzLizgfjfhzuRWarb1nG7BpvZ95sLUBvDlz6T2O9OClpYthibMoRt9bwA/3ge5/Mwxy2s9h8dsu6zRYniK4ExgdJI3jVM0d4M0tbT+w4FxN8L9LDoeN3XMRRetLdap4v/tKaf3Bnzcy/NNRPNdnMcTZdUsnPeB3t+3njMjIipHKWMU5KmLx30PnX/im6yPwd9WqqXX05MH0Ob0fw/N5dTDY9iZCajWlQ0AuGwutyt3ucrMhalP4EzjshvoHG3z4PTQObpGfSq1nqWHtX91j9ayhTuGLU/UFO+3Fn8/vd+L4S1wI5JMDzo4lR72njVCfTqH0+6heVPHfOrAuLyPtt3HQN4vIU+dvZ/KyQhd5eRSAYI7eWrpcPZObzntt45uWRmlmYvrS+nBn7EYXqMnzV65CCZAv7UMKhZDE81LrMyKSkT0UZUzmFGaulj/aks3WR1dea6F4xZsTKKHQxHy1MM6Wn1gEsS37wJly1MqH7dgr5nFUrb4dwS4v5KpkI9/wmkX+27M9kfL76ePCaBfyF032f3Kfj+9vHLSlaZorqD1zYD3FEj7nDwE82dx9DD6d/0jUL8ou3spFmdDmoAfdgFk93tf/j8dYE024dbC6uyVwDLTUv1jhsHVXYBg9VNERB9OKYGC/VV+YHdT4w3k1L+mNdErY4U7lV1HvV2s3ODJg2dVpbKr6NztubHygyeT/m9eZj+cnvpZv5Ba/pnBedWtpsfW0Y364F9+UKqHzlEVwVmYyegeLxA8SlN9lyGaxjrJXKQthtcrU40DAC6b0G7qmMdjZWYBvHYdeZ0oREQfzbP+PDJM49rLAWxCjPMHur1I2WH6u559SCgkAzeTP3VqvYD4wTNzYeo92H5mcF0hYcoe0rZt/asNaXDf3J2g+SLBQhikGe5cOobbnY8i/H868KSMSmYpbi1vJctSqTcA2BhLwe0imEiDHYmIPrbnf4+C1DLzr8rNKGxTtrI1mUP/asOzTrcYwLiOj4Heg/GtJT14pAfkwyAno7DeshU+c3G6VUYhLXxIrtplMOOGNYVdJNIx3eZ8LOUNzoxF2YTcboXl+hz8zAZ7UZdUMoYkCii+cnpyIiLgmQOFysk57LsOqlGa/PqTI7X6k9H7VcsL079bvPxofdmRuMy473nZatyw7uNuNIBx93dH9PSkS2TizpOug5qF876XlH1Th1M4o6Djp2skZR8FMLfJKGTfi6FPpDEge1qWHQZs8TbGgzQrJyPMXSz3e+35iM9r9lcux63wHK8MNFzAPVuTTYgCqvwxGzq60YDSePvh75k9IiJ6Rzh7ZOkWcL+HffF82JQtPLadz2OIC7b4iYhewgG8wplos/CdGwwSiIhe2n+99gYQFVE5GUGcvPZWEBF9POx6ICIiIiV2PRAREZESAwUiIiJSYqBARERESh88UIhmaMz+Xn8vyYyayrVKr57Om72RiIjoUKwPFOJ5BbLTFJf6YF1n80P3LQonXdrm1c20SWrK8hedw4KI6H0rkFEwYDzev9Mbr45uPBHQa28K7e5hgKrVWM77Me730HyxYJaI6H0rECg0YH6b4FqVIs/MZriSfbjyU629oqn2MD0fztqYvA5Z+v7MhakN4MfdBzldCOlWppmauyG1LPva6ALbva7saA+S7dq2hZt61XJe2So+BtoAvnxOsg/MzGucU9maOFskfz91bJJXLafLXsD9bsIdJnMxxN0rcvnpY5aXKYqOWcHXeC+36aYHQ5paWv/hwLgb4b6UuTqIiD62QmMU6ifnaPy5z2mhhbMZNnxpBsfLZvqhetlE9ek8XO7b8KzbQg/NMD0/h9NGaqbF5ZwJAIAemto16tNo3Xcd3MYPn5mLW/ySZlkEOmfJQ7NyMlo/++K67d5QNgD09Hi7ohZu0YffzIWpT+BE3w3nnNgm0OiheRTgXETHDx2cLs+Hj8HfVjIjpm+jp2fKvuugGn9/6sC4vJYmcroFbuJzkS3bQ+dPHfOpA+OyietPc8xdA72//nK/TqVW/96zcS6Fs1AmU0uHddKDh4CBAhHR3goOZtTR+iw9hCOrM/Lp6Po2PDmoaDuYx6/cPW7BxqTUG7jtj6KJgHS0+sAkiNZcs9CVgorKFxPGXYCgaMHrtrtA2cl2hbNR4rJY983i3xHg/komNzr+Cafdw33hcRqGNNFTNGvjU7xlOrry649zz4f0/VoLZjt54FZOutKkS9myIU26ZOP8JK8zZ9N+RF1BO72iOc52hJM6jeW6QEREOyv8qwf9h4PJ35xH3ef67v372W6LrQeh2WhJrVL9Qs44ZNLkRx3sPiFz1pZl1+ooOsdj8OTBs6rSMQm7X/YizbYo/+IinukxJTX1cwXWb6nlnzlf1W2muK5ZGPl20o1U6hgCD52jcCKuMFOxQPAoTcdNREQ7K/7zyFoL5uM13GyTPDPl7yKYFF97zcJISkULUdKUxwD8qyo6cDCPy546hR/WpZc9C+C166gXLN9w59IxKSFNHwVzi6GJ5qUtpf/Hq1NzK4UpfUjbpuy2UTnuLr87/txBtZRgoY56OzxmyTEKuyPqeVNOExHRVrZ4j0IF1lkDoz9JIBCm3OUuCR+3lieloPdVQf0zkn7ubS2zHQu4Z2VmFLYpOxps961V6JjoX2141ukWAxjXmLk4tTzYX6XwSwpY/KucjMIGy1Z6VPau6p/ygoxdBjNGXSDSMQu7xOTMCBER7Wq7Fy4dt9C4kx4ONQujqYOJnqSyJ6mW3f70i3CA5La/mtB/hIPq4vR98E1u9SddB1XLA+LPFXxArS87lPxSo4rRt7nUJZL8GqJ5iWU3w3K/jrvRAMZdu2S85LvRQNP4fFROzmHfdVCNyr3+5GyRUdDx0zWS/ToKYG6RUcj+4qFqNTAu6WeplZNROKA02u/qHxNz/uSViKgUnD3yXfEx0K5Rn46kQYdERES7++CvcCYiIqJ1GCgQERGRErseiIiISIkZBSIiIlJioEBERERKDBSIiIhIiYECERERKTFQICIiIiUGCkRERKTEQIGIiIiUGCgQERGREgMFIiIiUmKgQEREREoMFIiIiEiJgQIREREpMVAgIiIiJQYKREREpMRAgYiIiNTEGjYgoPo7/z9hrFn+3//73+rv4j+i/WxlG+L/zteUjf8R/7Nu3e3/PF/ZPGalH7P/tF/rfGzar7b4z2udDx6z7feL1z2PWSnH7HCve8Odr3vcr6UJIQSIiIiIcrDrgYiIiJQYKBAREZESAwUiIiJSYqBARERESgwUiIiISImBAhERESkxUCAiIiKl/w+or7F9si9G5wAAAABJRU5ErkJggg=="
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "##### Hint: \n",
    "Find out about Keras activation functions [here](https://keras.io/api/layers/activations/) and about layers [here](https://www.tensorflow.org/api_docs/python/tf/keras/layers). The input shape only needs to be defined for the input layer. Using softmax in your output layer will give you the pleasant effect of a probability distribution for your prediction. The output of softmax transformation has all values non-negative and sum to 1.\n",
    "\n",
    "<br>    \n",
    "\n",
    "Keep on designing your model until your model summary is the same as this one:\n",
    "\n",
    "![image-2.png](attachment:image-2.png)\n",
    "\n",
    "\n",
    "##### Side Quest: \n",
    "Come back once you finished training your model, and try adding layers and playing around with different layer types."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Designing the neural network training\n",
    "\n",
    "Great, that's it. Take a moment to realize how quickly and with how much ease (maybe not during the first time) you designed this architecture with $67k$ parameters. Also, within the blink of an eye, you created a neural network that has more neurons than a box jellyfish ($17.5k$ neurons), a sea slug ($18k$), coming in behind the fruitfly ($250k$). To put things into perspective, humans come in second with ($8.6\\times10^{10}$) neurons, only surpassed by the African elephant. Find a full list [here](https://en.wikipedia.org/wiki/List_of_animals_by_number_of_neurons). \n",
    "\n",
    "Next, we need to train Marvin to assign a class prediction when we hand him over a sample of a digit. \n",
    "\n",
    "Remember, when we set up our loss function and learning rate? That's precisely what we are going to do here as well, yet again with the help of Keras.\n",
    "\n",
    "\n",
    "<img src=\"graphics/blackCoffee.svg\" width=\"500\"><br>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# Define your loss\n",
    "loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)\n",
    "\n",
    "\n",
    "print(-tf.math.log(1/10))\n",
    "\n",
    "sampleID = 100\n",
    "loss_fn(y_train[:1], model(x_train[sampleID-1:sampleID]).numpy()).numpy()\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The loss is crucial for your training. The better your loss reflects the actual objective of your model the more efficient it will converge to the optimal parameter set. \n",
    "\n",
    "The loss is also great to check whether your model behaves as expected. The SparseCategoricalCrossentropy is equal to the negative log probability of the true class: The loss is zero if the model is sure of the correct class.\n",
    "\n",
    "Your untrained model gives probabilities close to random (1/10 for each class,  as there are 10 digits), so the initial loss should be close to:\n",
    "\n",
    "-tf.math.log(1/10) ~= 2.3\n",
    "\n",
    "Obviously different samples will give different losses, play with the sampleID to observe the changes.\n",
    "\n",
    "\n",
    "<img src=\"graphics/grinder.svg\" width=\"700\"><br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compiling basically means to prepare the training routine for your model which consists of the optimizer, \n",
    "# the loss, and the metrics which are to be reported during training\n",
    "\n",
    "marvin.compile(optimizer='adam',\n",
    "              loss=# '''your code goes here''' ,\n",
    "              metrics=['accuracy'])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "##### Hint: \n",
    "Make sure you understand the difference between a [metric](https://keras.io/api/metrics/) (only tells you how good your neural network performs) and a [loss](https://keras.io/api/losses/) (tells you how your model performance, but also directly tells your model how to update its weights to improve performance). <br>\n",
    "\n",
    "\n",
    "##### Side Quest:\n",
    "Feel free to play around with your neural network and the learning configuration.\n",
    "Great places to start are:\n",
    "- The [optimizer](https://keras.io/api/optimizers/), the losses, the metrics.\n",
    "- Adding layers, changing Kernel sizes, strides, and activation functions.\n",
    "- You might have guessed the learning rate. Truth be told, playing around with each hyperparameter will get you some experience and intuition for its behavior. \n",
    "\n",
    "\n",
    "\n",
    "                "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Here is some space for you to write down your thoughts on the side quest questions and experiments\n",
    "\n",
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "😅 Grinning Face with Sweat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train, the model\n",
    "\n",
    "Now it is time to plug our data into Marvin and let him learn to recognize the digit within an image sample. Besides plugging in the data, and the labels, we need to let Keras know how long we want to train Marvin. In deep learning, training on each sample of the training set precisely ones is called an epoch.\n",
    "Obviously, the longer you train, the better your model will fit the underlying pattern. Then you can also set how many samples Marvin will see during each update step (this is called batch size). The more samples, the better the gradient approximation, but at some point, you will also run into hardware limitations.\n",
    "\n",
    "Yet, remember that it is the unseen data of your validation set that machine learning approaches are after. This is also what lets deep learning stand out from pure optimization.\n",
    "\n",
    "\n",
    "<img src=\"graphics/machine.svg\" width=\"800\"><br>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import datetime\n",
    "# import os\n",
    "# %load_ext tensorboard\n",
    "\n",
    "\n",
    "# logdir = os.path.join(\"logs\", datetime.datetime.now().strftime(\"%Y%m%d-%H%M%S\"))\n",
    "# tensorboard_callback = tf.keras.callbacks.TensorBoard(logdir, histogram_freq=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tensorboard is great for versioning, sharing and analyzing your trainings through logs and tensorboards visualization as you will experience soon enough. Sad enough tensorboard is not eagerly supported by binder. But you get the idea [here](https://www.tensorflow.org/tensorboard)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "\n",
    "# get ready for your first training run.\n",
    "\n",
    "marvin.fit(\n",
    "    x_train,\n",
    "    y_train,\n",
    "    epochs= # '''your entry goes here''',\n",
    "    batch_size= # '''your entry goes here''' ,\n",
    "    validation_data=(x_test, y_test)\n",
    "    # callbacks=[tensorboard_callback]\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This line would start up tensorboard for you\n",
    "# %tensorboard --logdir logs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "##### Hint: \n",
    "Even if you stop the training, the model's current state will be saved as long as your juypter notebook session is active. Also, there is more to specify for training within the fit function of Keras. Check it out [here](https://keras.io/api/models/model_training_apis/) <br>\n",
    "\n",
    "##### Side Quest:\n",
    "Reach at least a validation performance of $98 \\%$ or higher before moving on. To achieve this, you are free to change whatever you want within the architecture and training process. Obviously, better performance is an excellent achievement. Further add at least one more argument to the fit routine. In case you are looking for further guidance how to progress with setting up your training pipeline [look no further](https://cs231n.github.io/neural-networks-3/).\n",
    "\n",
    "\n",
    "                "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# after the training finishes, we will also save Marvin in Keras style (HDF5), so we do not have to \n",
    "# train him again\n",
    "# every time we start our computer. Obviously, by changing the model_name, you can also save different\n",
    "# configurations of Marvin. The name has to be a string, like this: 'name.h5'\n",
    "model_name = # ''' Your model name goes here'''\n",
    "marvin.save(model_name, save_format='h5')\n",
    "\n",
    "# It is best practice to indicate what configuration changes you did within the name, so you know\n",
    "# which model you need to load already from its name\n",
    "# Let's say instead of a learning rate of 0.001 you used 0.1, your naming could then look like:\n",
    "# 'marvin_lr01.h5'\n",
    "\n",
    "print('Success! You saved Marvin as: ', model_name)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Evaluating Marvin\n",
    "\n",
    "Yes, you are right. The job is basically done. Marvin has been trained, and you also made sure that the model recognizes more than $98 \\%$ of the unseen samples correctly. Would you say that you know what Marvin learned or what exactly he is capable of in the end? I do not want to spoil it for you, but in real life the hard task begins right after training a model - propper validation is key. Especially in safety critical systems. And there are enough [incidents](https://incidentdatabase.ai/) to demonstrate how models can fail and why it is so hard to prevent these failures.\n",
    "\n",
    "#### Side Quest:\n",
    "Try to answer the following questions:\n",
    "\n",
    "- Which digit is hardest for Marvin to predict correctly?\n",
    "- Which two digits does Marvin confuse most often?\n",
    "- Which digit does he perform best in?\n",
    "- How do digits look that Marvin was not able to predict correctly? Can you see why?\n",
    "\n",
    "These questions are hard to answer and clarify that an in depth analysis of the model's performance is paramount. I like a mix of visualization (which helps my intuition) and number crushing (which helps to write reports and prove my intuition with hard and cold numbers).\n",
    "\n",
    "So in the following, we will go ahead and answer some of the questions mentioned above. Along the way, you will learn some strategies to evaluate your neural network.\n",
    "\n",
    "The worse your model is the easier it will be to pick up effects in your validation, so feel free to either save a random model or a model that has only been trained for a few iterations and use this model in the following.\n",
    "\n",
    "<img src=\"graphics/stars.svg\" width=\"500\"><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "😆 Grinning Squinting Face"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Here is some space for you to write down your thoughts on the side quest questions and experiments\n",
    "\n",
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Intuitive Approach\n",
    "\n",
    "We will start with the intuition building approach. The following code will plot the image samples. The caption below is Marvin's prediction. If he is right, the sample will get a greenish touch. If he is wrong, a reddish one. Try to answer the above questions with the results from the intuitive plot you will generate below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# load a saved marvin configuration you want to evaluate\n",
    "model_name = # '''your model name goes here''' \n",
    "marvin_reloaded = tf.keras.models.load_model(model_name)\n",
    "\n",
    "# Let Marvin predict on the test set, so we have some data to evaluate his performance.\n",
    "predictions = marvin_reloaded.predict([x_test])\n",
    "\n",
    "# Remember that the prediction of Marvin is a probability distribution over all ten-digit classes\n",
    "# We want him to assign the digit class with the highest probability to the sample.\n",
    "predictions = np.argmax(predictions, axis=1)\n",
    "#pd.DataFrame(predictions)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# Plot for the intuitive approach\n",
    "\n",
    "numbers_to_display = 196\n",
    "num_cells = math.ceil(math.sqrt(numbers_to_display))\n",
    "plt.figure(figsize=(15, 15))\n",
    "\n",
    "for plot_index in range(numbers_to_display):    \n",
    "    predicted_label = predictions[plot_index]\n",
    "    plt.xticks([])\n",
    "    plt.yticks([])\n",
    "    plt.grid(False)\n",
    "    color_map = 'Greens' if predicted_label == y_test[plot_index] else 'Reds'\n",
    "    plt.subplot(num_cells, num_cells, plot_index + 1)\n",
    "    plt.imshow(x_test_normalized[plot_index].reshape((28, 28)), cmap=color_map)\n",
    "    plt.xlabel(predicted_label)\n",
    "\n",
    "plt.subplots_adjust(hspace=1, wspace=0.5)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Here is some space for you to write down your thoughts on the side quest questions and experiments\n",
    "\n",
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "### The Empirical Number Crushing Approach\n",
    "\n",
    "Now that you intuitively answered the questions, let's do some data analysis on the results and see how much you were off and where you were right and picked up patterns in Marvin's capabilities. \n",
    "\n",
    "#### Hint: \n",
    "Usually, visualizations to building your intuition are most beneficial during your first training runs on problems you do not know what to expect from your neural network. In visualizations, you might also be able to pick up wrong label assignments (think somebody labeled all \"8\"s as \"9\"s), ambiguous or overlapping classes (think somebody labeled all \"6\"s and \"9\"s as the same class). These things do not usually happen in beautifully cleaned datasets such as MNIST. But I promise you, you will be surprised how messy these public datasets out there are. And further, I promise you that if you are building your own datasets, these bugs will be there too and want to be found.\n",
    "\n",
    "Now for the analytical approach, we will look at one of the most basic strategies, the [confusion matrix](https://en.wikipedia.org/wiki/Confusion_matrix) or error matrix. This table shows the actual class of the sample (horizontal), plotted over which class it was predicted (vertical). To make things clearer, numbers in the diagonal are true positives, meaning Marvin predicted them correctly. \n",
    "\n",
    "\n",
    "<img src=\"graphics/arsenicBeans.svg\" width=\"800\"><br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "confusion_matrix = tf.math.confusion_matrix(y_test, predictions)\n",
    "\n",
    "f, ax = plt.subplots(figsize=(9, 7))\n",
    "sn.heatmap(\n",
    "    confusion_matrix,\n",
    "    annot=True,\n",
    "    linewidths=.7,\n",
    "    fmt=\"d\",\n",
    "    square=True,\n",
    "    ax=ax,\n",
    "    cmap=\"viridis\",\n",
    ")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Here is some space for you to write down your thoughts on the side quest questions and experiments\n",
    "\n",
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Time to say Goodbye\n",
    "Well, actually not quite yet. Now, you are capable of training neural networks, you learned about the fundamentals, and you also can evaluate your model's performance professionally. Next, I want to point out some things you can look into along your path to become a machine learning engineer or deep learning researcher.\n",
    "\n",
    "1. Get a Stackoverflow account (https://stackoverflow.com/)\n",
    "2. Get a GitHub account (https://github.com/)\n",
    "3. Star this repo (https://github.com/schutera/DeepDive)\n",
    "\n",
    "\n",
    "#### In this notebook\n",
    "You can definitely come back to this notebook again and again. Play with the hyperparameters, try to follow through the notebook with other datasets. There are quite a few [directly supported](https://www.tensorflow.org/datasets/catalog/overview) by tensorflow, so they can easily plug into this notebook. Or try to load a dataset from your directory and write your own data pipeline.\n",
    "\n",
    "Come back and try working with other layers, such as Dropout or Batchnorm. Get to know regularization strategies, such as augmentation, and try to implement them here as well. Think of this notebook as your bridgehead to the world of deep learning. This is the place to break things and try stuff.\n",
    "\n",
    "#### Literature\n",
    "Reading helps to get to know new approaches, different ideas, and dive deeper and build up your knowledge. I recommend the following books to start with. From there, start to go with the actual papers.\n",
    "\n",
    "[To start with](http://www.deeplearningbook.org/) Deep Learning Book - Ian Goodfellow and Yoshua Bengio and Aaron Courville\n",
    "Abstract: The Deep Learning textbook is a resource intended to help students and practitioners enter the field of machine learning in general and deep learning in particular. The online version of the book is now complete and will remain available online for free.\n",
    "\n",
    "[Coffee Table Solutions](https://www.amazon.de/dp/B09QRGWWZP) The beginner handbook to Applied Deep Learning\n",
    "Abstract: The neural network was well into training for 42 hours. It all looked good - the gradients were flowing, the weights were updating, and the loss was decreasing. But then came the predictions for validation - all zeroes, no pattern recognized. What did I do wrong? — I asked my computer, who didn't answer. I noticed the dull feeling of despair and hopelessness rise inside my chest. After some more debugging and wasting more of the precious working hours, I would usually rededicate myself to the books, tutorials, and courses I knew so well by then. Somewhere had to be a hint to what I was missing - there was not. To give me a boost and to stay well caffeinated, I would usually go and grab a coffee in the coffee kitchen. Standing there at the coffee table with other students, researchers, developers, and practitioners, I would soon find myself indulging in the soft, warm feeling of getting this issue off my chest. And this is where the magic happened. Either someone already had a similar issue and knew how to fix it. Someone had an idea of narrowing down what the issue's root actually is about. Someone realized there was a conceptional problem in the data set or the model. Most of the time, what was shared at the coffee table were vague hints, interpretations and ideas, heuristics, best practices, experiences from applying deep learning in research and development. Over time we realized that some pitfalls, issues, and knowledge gaps were reoccurring in deep learning novices and were regularly brought to the coffee table while these students developed into deep learning engineers. Shortly after, we were taking notes. Throughout many sessions at the coffee table and hours of debugging, training, and evaluating deep learning approaches, researching and applying state-of-the-art concepts, we polished and enriched these notes with the great ideas that have been around and extensive references for further reading, based on our own experience. The result is this book. We hope it will be of use to you, too.\n",
    "\n",
    "[To understand the fundamentals](http://users.isr.ist.utl.pt/~wurmd/Livros/school/Bishop%20-%20Pattern%20Recognition%20And%20Machine%20Learning%20-%20Springer%20%202006.pdf) Pattern Recognition and Machine Learning - Bishop Abstract: This new textbook reﬂects these recent developments while providing a comprehensive introduction to the ﬁelds of pattern recognition and machine learning. It is aimed at advanced undergraduates or ﬁrst year Ph.D. students and researchers and practitioners and assumes no previous knowledge of pattern recognition or machine learning concepts.\n",
    "\n",
    "[From theory to practice and back](https://www.ksp.kit.edu/site/books/m/10.5445/KSP/1000008476/)\n",
    "Data Mining in der Medizin und Medizintechnik - Ralf Mikut Zusammenfassung: Dieses Buch systematisiert Ziele, Einsatzszenarien, Vorgehensweisen, Methoden und Anwendungsfelder für eine automatisierte Datenanalyse in der Medizin und Medizintechnik. Es wendet sich hauptsächlich an Doktoranden, Diplom- und Masterstudenten der Ingenieurwissenschaften und Informatik. Im Mittelpunkt steht dabei das Spannungsfeld zwischen medizinischen Anwendern und ihren Zielstellungen sowie den Potenzialen vorhandener Data-Mining-Verfahren.\n",
    "\n",
    "[To get practice](https://www.coursera.org/courses?query=andrew%20ng) Deep Learning and Machine Learning Courses on Coursera - Andrew Ng Abstract: In these courses, you will learn the foundations of Deep Learning, understand how to build neural networks, and learn how to lead successful machine learning projects. You will learn about Convolutional networks, RNNs, LSTM, Adam, Dropout, BatchNorm, Xavier/He initialization, and more. You will work on case studies from healthcare, autonomous driving, sign language reading, music generation, and natural language processing. You will master not only the theory but also see how it is applied in industry. You will practice all these ideas in Python and in TensorFlow, which we will teach.\n",
    "Link: \n",
    "\n",
    "[From beginning to mastery](http://cs231n.stanford.edu/) CS231n: Convolutional Neural Networks for Visual Recognition - Andrej Karpathy Abstract: Computer Vision has become ubiquitous in our society, with applications in search, image understanding, apps, mapping, medicine, drones, and self-driving cars. Core to many of these applications is a visual recognition task such as image classification, localization, and detection. Recent developments in neural networks (aka “deep learning”) have significantly advanced the performance of these state-of-the-art visual recognition systems. This course is a deep dive into details of the deep learning architectures focusing on learning end-to-end models for these tasks, particularly image classification. During the 10-week course, students will learn to implement, train, and debug their own neural networks and gain a detailed understanding of cutting-edge research in computer vision. The final assignment will involve training a multi-million parameter convolutional neural network and applying it on the largest image classification dataset (ImageNet). We will focus on teaching how to set up image recognition, the learning algorithms (e.g., backpropagation), practical engineering tricks for training and fine-tuning the networks, and guide the students through hands-on assignments and a final course project. Much of the background and materials of this course will be drawn from the ImageNet Challenge. There is also a YouTube [lecture](https://www.youtube.com/watch?v=NfnWJUyUJYU&list=PLkt2uSq6rBVctENoVBg1TpCC7OQi31AlC). \n",
    "\n",
    "[All things safety and validation](https://users.ece.cmu.edu/~koopman/lectures/index.html)\n",
    "Koopman lectures on why safety and validation is so hard in perception systems. You want to save a job within ADAS and AD in the automotive industry in the next 2 years. In my opinion this is your ticket.\n",
    "\n",
    "\n",
    "#### Mailing list\n",
    "There is so much more out there - different neural network architecture, different learning paradigms, not to mention endless real-world application possibilities.\n",
    "Every now and then, I'll share another part of this world. If you want to keep posted, write an e-mail to mark.schutera@kit.edu with the subject: \"AppliedDeepLearningSchool Newsletter\" - I promise there will be only a little spam.\n",
    "\n",
    "#### Someone deleted the internet, or a link broke in the notebook? \n",
    "You wanted to know more, click a link - and it was dead? That happens, as the internet and especially software is a living thing. In that case, I would be delighted if you'd let me know so that stuff gets fixed for the ones after you. You noticed a bug, a typo, an error, something? Send me an e-mail to mark.schutera@mailbox.org subject: \"AppliedDeepLearningSchool Bug Hunt\".\n",
    "Any other feedback is also highly appreciated.\n",
    "\n",
    "#### Can you use this code for a school or university project?\n",
    "Sure thing, go ahead. Just make sure to cite it appropriately. If you are unsure on how to do this, again, feel free to reach out. I can not wait to see you burning through school and university competitions with deep learning applications. \n",
    "\n",
    "#### You are looking for an intern position, or thesis, some freelancing on the side or some fun deep learning research project to join?\n",
    "Feel free to [connect and reach out](https://www.linkedin.com/in/schuteramark/).\n",
    "\n",
    "\n",
    "<img src=\"graphics/droplet.svg\" width=\"650\"><br>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "😇 Smiling Face with Halo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Here is some space for you to write down your thoughts\n",
    "\n",
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}