notebooks/MNIST-torchvision.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4rU_kfN6q-0v"
      },
      "source": [
        "# MNIST with torchvision and skorch\n",
        "\n",
        "This notebooks shows how to define and train a simple Neural-Network with PyTorch and use it via skorch with the help of torchvision.\n",
        "\n",
        "<table align=\"left\"><td>\n",
        "<a target=\"_blank\" href=\"https://colab.research.google.com/github/skorch-dev/skorch/blob/master/notebooks/MNIST-torchvision.ipynb\">\n",
        "    <img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>  \n",
        "</td><td>\n",
        "<a target=\"_blank\" href=\"https://github.com/skorch-dev/skorch/blob/master/notebooks/MNIST.ipynb\"><img width=32px src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a></td></table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7OYtKLj-q-03"
      },
      "source": [
        "**Note**: If you are running this in [a colab notebook](https://colab.research.google.com/github/skorch-dev/skorch/blob/master/notebooks/MNIST-torchvision.ipynb), we recommend you enable a free GPU by going:\n",
        "\n",
        "> **Runtime**   →   **Change runtime type**   →   **Hardware Accelerator: GPU**\n",
        "\n",
        "If you are running in colab, you should install the dependencies and download the dataset by running the following cell:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "fJW3IR6Mq-06"
      },
      "outputs": [],
      "source": [
        "import subprocess\n",
        "\n",
        "# Installation on Google Colab\n",
        "try:\n",
        "    import google.colab\n",
        "    subprocess.run(['python', '-m', 'pip', 'install', 'skorch' , 'torch', 'torchvision'])\n",
        "except ImportError:\n",
        "    pass"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "nCr4NPnIq-09"
      },
      "outputs": [],
      "source": [
        "from itertools import islice\n",
        "\n",
        "from sklearn.model_selection import train_test_split\n",
        "import torch\n",
        "import torchvision\n",
        "from torchvision.datasets import MNIST\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "35jlO3R5q-1A"
      },
      "outputs": [],
      "source": [
        "USE_TENSORBOARD = True  # whether to use TensorBoard\n",
        "DEVICE = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
        "MNIST_FLAT_DIM = 28 * 28"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cCEm6xAWq-1B"
      },
      "source": [
        "## Loading Data\n",
        "\n",
        "Use torchvision's data repository to provide MNIST data in form of a torch `Dataset`. Originally, the `MNIST` dataset provides 28x28 `PIL` images. To use them with PyTorch, we convert those to tensors by adding the `ToTensor` transform."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "y8MuBRW5q-1C"
      },
      "outputs": [],
      "source": [
        "mnist_train = MNIST('datasets', train=True, download=True, transform=torchvision.transforms.Compose([\n",
        "    torchvision.transforms.ToTensor(),\n",
        "]))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "XzcCY_4bq-1D"
      },
      "outputs": [],
      "source": [
        "mnist_test = MNIST('datasets', train=False, download=True, transform=torchvision.transforms.Compose([\n",
        "    torchvision.transforms.ToTensor(),\n",
        "]))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1Ne8gcaSq-1G"
      },
      "source": [
        "## Taking a look at the data\n",
        "\n",
        "Each entry in the `mnist_train` and `mnist_test` Dataset instances consists of a 28 x 28 images and the corresponding label (numbers between 0 and 9). The image data is already normalized to the range [0; 1]. Let's take a look at the first 5 images of the training set:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "OYMmCHJ9q-1I"
      },
      "outputs": [],
      "source": [
        "X_example, y_example = zip(*islice(iter(mnist_train), 5))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "f6KVmZm1q-1K",
        "outputId": "be31b0b7-c7d5-4be0-9cc6-4552e75541fc"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(tensor(0.), tensor(1.))"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ],
      "source": [
        "\n",
        "X_example[0].min(), X_example[0].max()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wgjNeDTLq-1L"
      },
      "source": [
        "### Print a selection of training images and their labels"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "RF6Ki7_aq-1L"
      },
      "outputs": [],
      "source": [
        "def plot_example(X, y, n=5):\n",
        "    \"\"\"Plot the images in X and their labels in rows of `n` elements.\"\"\"\n",
        "    fig = plt.figure()\n",
        "    rows = len(X) // n + 1\n",
        "    for i, (img, y) in enumerate(zip(X, y)):\n",
        "        ax = fig.add_subplot(rows, n, i + 1)\n",
        "        ax.imshow(img.reshape(28, 28))\n",
        "        ax.set_xticks([])\n",
        "        ax.set_yticks([])\n",
        "        ax.set_title(y)\n",
        "    plt.tight_layout()\n",
        "    return fig"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 120
        },
        "id": "LRhu6GbVq-1M",
        "outputId": "f2d91ae4-d1d4-411c-c477-4605f85e1f47"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 5 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAABnCAYAAABVe9YVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOy9aZBk13Xf+bv3vi1f7pmVVdW19b4BaGwESAIgRYqkREmkTNGkZcmSZqSYsWSP7BjHyAp5JjQzEZIjZmR/8MTEKMbjCNnSWB5LokSam0xzX8AFJNFo7L2vtS9Zued7+d67dz5kdaO70aAAsrszq5C/iIwGqrtenXfrvfu/95xzzxHGGEaMGDFixIhhQw7agBEjRowYMeJWjARqxIgRI0YMJSOBGjFixIgRQ8lIoEaMGDFixFAyEqgRI0aMGDGUjARqxIgRI0YMJSOBGjFixIgRQ8m2EyghxFeFEIEQorX1OTVom7YzQoiSEOITQoi2EOKSEOLvDdqmnYAQ4uDWc/qng7ZlOyOE+EdCiO8LIUIhxB8P2p6dgBDiqBDiy0KIuhDirBDiw4O26bXYdgK1xT8yxmS2PocHbcw25w+BHjAB/BLwfwsh7h2sSTuCPwS+N2gjdgCLwD8H/u2gDdkJCCEs4JPAZ4AS8OvAnwohDg3UsNdguwrUiNuAECINfAT4n40xLWPMk8CngF8ZrGXbGyHELwA14EuDtmW7Y4z5uDHmPwEbg7Zlh3AEmAL+lTEmMcZ8GfgmQ/rOb1eB+t+EEOtCiG8KId49aGO2MYeA2Bhz+rqvPQuMdlA/JEKIHPB7wP8waFtGjHidCOC+QRtxK7ajQP0OsA+YBv4N8GkhxP7BmrRtyQCNm75WB7IDsGWn8PvAHxlj5gdtyIgRt+AUsAr8thDCFkL8JPAuwB+sWbdm2wmUMeYpY0zTGBMaY/6E/vb0ZwZt1zalBeRu+loOaA7Alm2PEOJB4H3Avxq0LSNG3ApjTAT8HPABYBn4LeAvgKFcUFmDNuA2YOhvUUe8cU4DlhDioDHmzNbXHgBeHKBN25l3A3uAy0II6O9QlRDiHmPMwwO0a8SIaxhjnqO/awJACPEt4E8GZ9Frs612UEKIghDi/UIITwhhCSF+Cfgx4HODtm07YoxpAx8Hfk8IkRZCPAF8CPj3g7Vs2/JvgP3Ag1uffw18Fnj/II3azmy95x6g6Iu9t5WJNuKHRAhx/9Y4+kKIfwrsAv54wGbdkm0lUIBNP+V0DVgH/jHwczcF+Ue8Mf47IEXfL/0fgX9ojBntoH4IjDEdY8zy1Q99F2pgjFkbtG3bmN8FusA/A355679/d6AWbX9+BVii/86/F/gJY0w4WJNujRg1LBwxYsSIEcPIdttBjRgxYsSINwkjgRoxYsSIEUPJSKBGjBgxYsRQMhKoESNGjBgxlLyhdE1HuMYjfads2bY02Vw3xlTe6PeNxvPWjMbz9jIaz9vLDzueMBrT1+K1xvQNCZRHmreJ994+q3YIXzR/eemH+b7ReN6a0XjeXkbjeXv5YccTRmP6WrzWmI5cfCNGjBgxYigZCdSIESNGjBhKRgI1YsSIESOGkpFAjRgxYsSIoWQkUCNGjBgxYigZCdSIESNGjBhKRgI1YsSIESOGklFflRE3IFwX6brgulDKY1yLJO0Spy1kbLDqISJKkM02pt7ARDG6G4BOBm36iBEjdhgjgRrxCkKgKmMkk0WC8RSrb7EJy5rc3hrvmXmeK90i3zu5F1WzKJwsUn6hjWoEyEsL6HZ70NaPGDFih7FzBEoIEBIhb+r+LvpeTJMk/VW+EAil+l+Xgq3W3NcwxkCSYLR5c+0KtsbPpFOEZY9OxaI7G5Ebb/F39j7DPyk9zwuR4H/p/RyXq0U6tRyZBRdHCCx75zxGQ8HWMyksG6EkxhhMFIPRMOrf9oORqj8HKPXqdzvRmDja+p/ROG4HtsfMcvWFdRyEZYF8dehM5nMEhyfp5SyMBCNB24IoLdAKSqdCnOcvQTFP/aFxgoKgWxGEY7p/bQ0iEfgLgtyVBKcW4z57gWSjeldvdRAI10WNlTG+x8q7K1TfGuHlWrxv7jx7U+s8nj6DEoJJFfK3dz3DQrnI13IHuXygjH3FZ3+1ArX6oG9jRyB9HzE9iUl7VI/lqR0Cd1Mw9fUGaqmKbjTRzeagzRxKVLlE7/49hAWbxh5Fa1aDoP/RUHpeMPb0JrIVoBeX0UEwaJNH/A1sE4Hq74yk6yL81C3/STJZZOURl7Bs0LbB2AbjJmQrLWyVsOqPMbNQIJgrsPhuTW6qyd/d+yy/WfoeCkHTaNpa8tsXP8LLx3eTXnCZvZiDN4NAOQ7JeJGo6FF9JObfvfvfUlIdplSCJxQSCQgmlMsvZy+iOc+vFp5i5UCKfzH/UzQ/P4M8Nei72BmIlEc4VyQo26y9L+RfPvaXfGztES6sHaaYGJTWI4F6DUQ+x9qDHu0pw/1vO8Mf7vkEnpBIIejohHft+k3cRp7Uegp7sw4jgRp6BidQQvR3REJc2xEJy+oLkGVhPAdcByMExlagBEHBpVewMOLVlwuKkvaeGCvfw7YSLEvjWDGT2SaW1JwcL9M+VKY9qUiNNzlYXmOvu4ovbCISAi2oa5dG6GF1BCoAtL67Y3KXEa6LTHkwVqJ+OEu3LCmMV6moNnmZ4AkLWygSY4hMQkLfLaKNwRZQUV2KTpdq1sIv5DFBiA7D/sWH1YUiFSqTBtvqu4OUxEQRut7AxPGgrQPLIsoowqxAWoam9ghiu/93klt6D97sqFwOkc3Qmy7S2WVgusvB7Bp56WALhS0UigBl9XdURtxiAhkxlAxMoIRlo8bK4NgYJUFKdNajvTtDLy3pTAiCisFYBu0ZjDJM7t7gvbtOY4tXx4Z8FbLPWSMruyihURgCY7Mc52kmKbqP2pwdnyBTbPFbR77MW7xLZGVEXcOatvhG5wgLYZErl8YYOwf+eoxpdwcwMncJIVDjFeLpEo19Ps2PNnnb1CV+ovgiMxbYOKitF7ljIpraoIHAKECQlTCmFHv9dU7MWjj37MFeqmEWl/sxvGGY7G+ByqTRh+aIsw6xp0g8iVMfHneu8FxauxTdCYEx8PmNezm9VqEYDqngDxoh0AdnqR3N0pyTPPbOF/iZ0vMccZYBgUYTGUgw/TXTaBi3FYMRKCEQjo1JpzApB2NJjJJEOYf2uCTKCjpTGrWrg2MnpL0enhXzd2aO8+uFs1sup7+Zum5yHM2aSNiXXae1y2E6U+ded4G9tqauoaoVy3GW890KC0EB1VR4dY3djGFIJ9kfma1EEZNJERZdOhXJY9MX+YWx77DHquOLV9yo/RfcEBhJhKSmPbSReKKBJyzyqksvLwjLLqrtIzdcTBz3k1KGcRdlW0Q5l17eIk4JIl9gpIVrDYm3WymSlCD2DUJANfTp9SyEHsKxHAaEJM66dMuSYEzzeP4c70pdwROSm6c3cyvXy5uFq7vGq4lkW0li/b969biYRIPR/WSxq5e4+n1Xv34Xknbu+lupikVEIUdvpsSV96UIx2OwDMLWSCehlF8nZ0c8kKmxx9/AFglZFWCLhEdT51+3OGk0ZyOPP7j40yxu5gnqLrJhseyV+bXlXyXl9ugEDr3AxoQKq2qhQqicNGRO15GdAN3dQTsoqRBKIVMeYqyE9j1WnihSfSghPV7n3YWTzFoN8jdnQQIric3J3iTnw3E+NX+MZtfl5w88w2+Vj/No6jwf/4kFFh4qoF4uUn4ph1sdnh3JzQjPoz1l0x2TRFmIMobeiiJ//NaxzbuNsS2iDMT5hGK2w2y6xqKfQ1veoE0bWhJPEeVAZ/pzhSckthi5Qq8iXBc1OY7xXMLpPO1dNokrCMqCxIHEN8T+dUKkIXNRUjgfISOD6sQIbQiLDkFBYXcN6QstZKsL1dodfc/vrkAJ0Ren6SLVezze8zPH+dWxb1CWISXZD2YCKPp/Xi9GSog3JE4Ap3sTXHxuiswVidcD1TOAwogMAMWWwWlqZGSwWl1kL8Fa2iReWELvsJReoRTCc/u++tkiYcGm+paY333np5m2N3nYrZKXzqvGODGG1STDC90Zvrexm/pT43gb8Dn/KP+k/DQPuZpPH/1zOibhv536CC+n9+IvusxcyA5ngonr0J6UdCc1STHGL3RpO1mMPwQCIMSWQBnsfMhktsne1DqnUuP0VP6VVfCIawgpiFOSKGtQmYiCauOKIdkNDwky5RFNlYjyDuvHbFqHe7i5kCfmLjCXqvK29DkecavX5t220fzamV/kypOzqACchoOMDa0Z6E1FqJpFxc3ir3q4xkB1847NlXf/NxknyF6C2PKeKQyugIx0X/clIpNQ1T0iAxGCyEg8kTCmFJ6w+kF9EupJGndT4q9oZAwyNjf4oO12gtWKELFGdiJEkmCCcOedfxICNTlOvKtIWHDZPODQy0NxcoNpe5OKamLz2guAtAzZ5dTIu+P9lF0DndDhQiTJy5AJ5eALKLttkowm9lU/rjiEGNemVzDosQjbi1BqOALnwrIQjoPOOMSlmN3lOjknYCNK0wxcMl2DbIeYXjRQO4cGqVC5DMLz6JYlyUTIRLFJVvYz8zomomMMa4nDd7p7uByWiRd8nGaM1Y4g2WHv+C2Q2Swyk0aXC9QP+gRFQXsuYWyyQdlvcySzxJRdwxMRa4nAERpbQGJgLr3J2blxCBVhWyISQTIRUhlrUvUytHel0LaDXb+z7evvrkAZg2m1sVYlqU2f+U6BK3EJX6wyoV7/Zaq6xxc7+1iJ8tTjFI04xaRb58O5Z5ixYkKj6RnDqc4k5RcSct+51P/ZN6t8HPcfVG0wSdI/ENnr3d57HgKEUtTfNs3SOwSUQ376yNMcTS9yr7vAPU5fnF5r1amE4LDdZdZ6mcgojqcP4dQFzUaKP64+wX5vjQ9lX2RMOsylqqQmW4TdLLjOXb7L10eS9bCONvjgnpc51ZzgSq0waJMAkJk0IpulOZ3infe9zG9MfJXP1h/k6ysHqC3lGJ/vYC4t9A/sjkDlMsRH99ArOmw8nPCP3/JVZpwN9lktwOFSbHOyt4v/vHGMZz59D9nLmn2XQ5wLq5iwh+50Bn0LdxYhYN8MtUM5WjMK/ydXeOfYPEfTixzzruCJiIoMsQU8FUzx+cZ9uDJml1PDEz3eX3qen33HCRIjCYxNYgRlq0VBdjgR7Ob/Sb+D1fU0qpchd0LdsaSou76DMr0eIghRgaYReqzGOSZVndBESOQ195x97fzNjWg0HSO4EFaY7xbZ7KVoRh4N32U5ncEXdSIEiZFUez5etUe8tHy3b3N4EP1T9UFRkpprsK+8wa+NfYP7nasrAg+NvpZKfnX8of87AMhKhyxQUi2Qpp/6HyoutsooNEGmv/vIqIC016PrmoHvSF4LYyvGcw0eSF9mrZfhCsMhUNgOJpMiSkvuzSzxiJvwFdWjEbjIrkS1eyQ7fVJ9vQgBrkuv6NAtWzjlNo/7ZyipgKxUaDRN7TPfK3GuXqZ4OiH3wgai0UI3mv0s0528g9pKgopzHp2Kojth+Nldp/lg7gQTqktF9af9BEVkNBtJhtOtcSypaaVcMirksfQZHveaSOSrwitpcZYvFO/hPNBL39n3ZwACFaGbLbylDueP7+IPVt/PnskN3la+SEc7XGyV6WnFRyaP88u5KzcMzOW4y5U4x+cbb+HPv/Y4qWWJCkH2YDk9y5NzRzHpmMnJGg+NLfD0/Cx729GbNrNUuC6qMobJ+rRm4X2zZzngr1BRPeCVpIC67nE8LLGRZHiuM8upxgQzfo3frHyFA7ZLXQfUteFcOIG7Lkkva7Rj81ywh1OT4/xs4QS7rRCFQQoDw6lN15DCoLY+UgzB0yEVwQNzrD7i0t4Tc9hbAmC1l6VV83GaAuIdPKG+AVSlAqU83T0F5t+jYFfAT+07xYTqkpYChSDB8Nn6A3zy1P2YKz57FztQraODoO8i3WHxZeBafFKmUsjxMUw6xeJbU5h31DhU3OQdmdNMqC6Lic9TQZnz4TifXriPaiNNMu/jL0iMhOdcMDZ8/MEH+PVDTzJtb/I2d5kx9cp8UdMpLmyWaC5n2dXaYVl8Juphoh7W0jrj38vQPe9xed80C/vy9AIba9FF9uCT79L83exF3K1sHI3mSpzjW+2D/Jf5I+z+6xjvmUuYIMAEITKXQe+dIs46LL91gi8+mEHP+8jOBm/WV1u6LslkkV7JI5yJ+MXyt6moLiV546+9qQ1Ptg5xsVPmu5d2w0WfZyd7/PQ7n+WA3aauDVfiHBc6ZVLrhvRiDyuwSa1KGvszLNxfRLJ60w+/izf6BriaaawYnkPYQik2j7iMvWeRJworHLZXSYzFephB1GzspkCMXHv9SbiUp7OvSO2gzeNPvMBHx77PfnuDCfWKSzkyMd9e3Yt7PI2/YrCvrBOvrQ3Q8LuDUArhp4imS4RFh9YDAR978I8pyR4lpbBxeCoo84XNe3lmdZrkS2XGFhOyF1rI05f7F3FdhOtw+e/t5tO5+zmUW+Xw2Cpj14VgGtqjsenjriqcZnxDKvrtZmDpLiaK8KoRQtvEvqKtMtg9SK0IZGQ4uzbGUzNpKqrNbsvgCYtzvXG+s7mX6nqWcjPCdDqYXtQXvW6ArHewY42/7NK76JNeFYhuOKhbHBjCdRGOA+NlaoczdMckpco6BRniC3NtV7qedKlqxYlwN19cPMz6Zhaz5uK2BHHD5quNo0TmDM907uf5+hQvLEwxvqmx6v0xlZFFt2LR1v0ElzGrwUy2xmou14/15HLoMMSEg/8dCNtBeC6Rb5Gxe2RVF0cOz6SvFWSckJwV4Ii+eDYiD7shsJvmTRHU/xsREp1L0Rm3CEsw49WYtOpkZQIoIhLWk4QN7bJay1BYNaTWY0y48+LKN6PyOUQhTzKWo3o0RVAW7BpfpyB7KAGLsSE0hq/Uj/CNi/uI11JMrGpS6xGy3kGHIcJxkBkfnU4RZQzTfp1pt4Yj9NZ5yITIaJbjWVTVxlsX/fOid5CBCZSuN3BOXMBxbLIv5UiKfr/PUL3vZ7+S3sXvpn+Og4U1/umuz7PPSvjEykOceXIPhSWBvbxO3A3623VAd7vIK4tg21TW64x910cEPfTSyqBucWCosTLJeJHakSzJL1b5qenTPJE5zYwFCgslBJFJ+FYwxZfq9/DNhb14Hy+w53JIWDJ0iwa7I/lL+1H+yn+Y1CmX0qmE2VqMd3q+/7uzLBzLQtv7WOoVgCs8kbrI9NQmfyR/jLMHjlAK51BLVeKFxUEPCWqshB4v0pqyuT+zzj6rSs4anlps2oaS26ZodbBF32NwsVqicGprkm2P4k9CCpp706w/qvEm2zyRPc1hO8amX9JoJe7x6dZ9nO1MYL2YofK1eUy7i97cHLTpdxYh0AdmqR3J0JyVHP7pM7x37CQPepeYUBZXYs0nGg9xrlPh6187xuwXetitLtb8BqbdwYQhptdDFvK07p2gU1GIoy1+Y+IrlGXIxFbMai2JWUx8nqwdoPS8oPRCv4BxfAeznge3g4pjkq0HR9YbqHUfkgTdaoOQpNYmWd3IYUtNZ8ImocdmkMLbEHibGm5OBzemX504CN7cxTSlwqRc4oJLUJK8b+oMv1r8FgWp8UR/p3M1DX8xKnKmUaGxnmbiXBfr5GXUwRkSxwcgtWSROIrC2YTc04uYICSp1jDRKytSpz5HR/fdK2NK4Yo6+9LrvJwVRDkXWXP7rplB+vyFAM8lznlEvqBodchKjTssOygpQEJKRXiyn0aeYOiFFm5dYzeinVvV5I0gJL20wCp3mS7WqagmvnjFtRcYycVgjPOtMk4NkoXlG57VHYlUSMeml3fpTEi6Uwm/MPldPpReJzH98mQdY3GuU+FMvYK/JEi9MI/pdIlb7RvmUGFZBEVFUBZM5JsctuNrcwZA01gsxEVWu1lSGwlqqYpptu7o7Q3FiTYTx9Du9NO8kwRhSWRs0KEiiC0SBDaKB8qL/Of7SvTyDsWnM7A0aMuHB2FZyGIR4bk0jo1TPaLozMXc6y9QVoaegZUkJDCCauLR0Gk+u3If516ewl9QWJsNTLuDtVAlH8RoR5GddzFKkJpvXuue+4Oyn2wUaWE45C3xH+6N6WVdJmQJ+/xlMAN0UQlJPJGndtCjPWPY5dRIC3nLmo53FamQab8fN8gYDvir7HbW8bcC3nFP4dQjVDN8c6eXy60KKL5Pe1rws4ee52BqhUkVonHp6IjAaI6Hu/nMyWOw6DF1+Qc/qzsBVS4RPLSXsGSz+hbBxAPLvKWwxhFnhcRIvtgt8LXGEZ6rTXPpqRlSq4Kx50J0u9M/TmNujMOaQpbqMVB7mrxj/Ny1g7vQP3v676uP9RNPLvvsX2xgmq1XikPfIYZGoG7Iozf9g7X0JEFkERkLJRKeyJ2hdiTFU84edNYb/Mp8iBCOA6U8SS5F7YDCvKXB0XKVY+48RemxlHRZTlya2uNkOMV6lOX0lQnyLyn8tQRRrZMEAXp+AeYXAHCuJqi8zqwnWyiUEBxxlzh6ZJ4LY2WayxnKUtz8Ltx1gjGPxl6Ip3rM2Rv40h64QAnVFyiTTRNlDfd4C0yrOp5Q/eKmocJqtJCtDnqHT7Y/iH7w30dk03RnYn5n/BtkhA04JMbQNpr1xOZEew7vuRSFcwnpC43+c7uTKeZZfcSlM53w+FtO8n/MfhZvq3p7ZBK+1jjCJ08+gLjisfczHezzy+h25zU9THHeo3jvOr+y57s8mjp/7ZgJQGQ0X5o/RParPv6aRi2sEzcad/wWh0KgbsZog1tPSC3aNEWWFw9NM6FOYYuYg5lVXs5N0Ctl8MYrmFZ71G5cCEQ6TbC7QFC06E5qDpWrHMiuYQtNxyScCMf5WuMItcjnTL1CK3SwFt2+C6mlX1kgXC9EP8SuRyJxiMk5Ab4XYqzMta7Gg0JIgXYEScpguTH2VhmTjnboBA4yFIgBtFaRKY94zwRh0UUXIgqygy00NZ0QGiAWiCiBOHn1IfM3AcKy+p6BQp7e4SmCkoNb6uAKiS3UtcD9pdjneHcvz21O41UN3kbUb0o46Bu4Q6hyCYp5gn1lOlMJqakWB9Or1xY2i3GPauJxvDqLvOKRWhZY9QDT6d5YiGCri/bVXXyn6FBKrTNlb+LLCFA0dY9TUYqFeIzNtSzT6xq3Gt21xJOhFCh0gv/sFXYvl6gfyfEns29nabrA/f5lfqnwXYI5m/9y5HGKYo7U+SqcvfDm3UlJhbAtzNQYV95jY+a6vGvfWf7++NfwZURaxKwkgn9x7qeofmMSuwXZ+YR8W1PuBqh2hOz0MJ3bVxjXFQm7/SpBbHMxNXbbrvuj0EtL9HjI5FY5nMQYLneLRKsp/KqAQZQQmqxw5b0ZurMx7zl6isN2l8AYTkV5FqIidkMhOgEmCN6UWXwym0UU83QOjnHxbwsmZjf49bnj2Lyysk8wfGzzrXzm5DHkFY99J1rI8wvodmdnzglSET6wl7WHXNrTml9457d4b+5FplSTyMCVRPJnm49xvjPG0rem2feZJrLRxSyubI3Jlmxf7cenFOydpr07R/WwxYdKlznmLlGQILE5G3n8j2c+wsJagfJ3bPLfuYTpdO9anH84BQpIqpvIdod0dj/nNzOcLVQ4mlpkt+Ww31slLEBQsnA3fKTr3lgeficexHsNhFIIyyJJO8QTPe6dWuHx/FnudxIiDIuxZE37LG/kqVzQuLUE/2wV0Wz3G0LaVj/b8TZOgI7QZFVAxg4xb6CE1R1DSLQFlhOTtnuoraPbndhBdSQqAJHchfX21or1atsCnfUIdiWMzdS4N7NIVjpo3WMtzjHfKyPDrQO6d8O2IUS4Dtr36OUtpnav8oGpF3k4dfFan7LIJIRGM98pwLKLtypQ6w3iYSxSfJsQUtArWP12RFMd3p19mR/zenSMIDSaauJzujXOhVqJ1ArIs/OYbrcfK7puThRKIV0XXJeomKJTUfQKhgm7QUGCTT/TdzUpsLheQC56+KsJ8dLKXa1VOrQCZaIYDdjLdXLfnuSZC4epPZHiXQf+nDl7g+zb1ljZn2XzaJbMAw9hhQa3plFBgndpk+RNsquSpQKUC7RmUxyam+fDE89wzLuCEoLvhyl+59RHWFnLk/ueR/7lOrITQrWGDnuIbheU6tchfJMF4TWa5XaO1Iogtb6VFXoHuOqmEpk0ZmaCxLdpT3m0xxXdScN9xy7w9uIFHvH7rWTORx7/14UfZ2ktT/mCwbRamCDc8QH/VyEEerxI81Cexh7JO8uLvC19llmrgSTFpg74TlDhSlTm6dN7mHwavI3eHc8qGxhCIF0XkUrRmFPM3r/Aofwq01YDjcULPZfvdffxZPUAL37pEP6iofJiB9Pt9t33W3OhzGaRaR89WWb5sTxBSdCdiSlMbXK4UOPh1AVsBM/2MpwI5vj86j34x1PkLg0mrje0AoVOMGGCWVxh15cVccHn1OQU1X0Ws1aN//Xwp6kdSPPZA/fzzOIMYdvBnnewWxYTpoh94fLQdnW9bQgB+SzBTI72lOSjEy/xi9nL11aYzwezNL4+wfSphMzZKualsyTD2khwAGy2U+SWNKmN+I4VCRaWhUj7UCpQP5wjLAhqRw2TR5Y5mt3kN3d9icN2F1/YSCwuRmWWXxwnd0GSP9shabR2XnX914OQ9MbS1Pcq2nP9BKm3uW3srbTnjUTw5cY9nG1WyJx0KH3jcn+nUKsP2PA7g1AKkUohMmna04bf3/MFpq0aU8qQGMPzwSyfW7mXU+d3cfgzDXj2NCZJ0Dc9OzLto8eLNA5mSd5f430zZ3lX7hTvSi1hC4knLCIjOd7dw8fnH2Th4hgHv9PBOnm5fxbvLs8dwytQW5g4RnYCLCmx13z+bPOtjDsN9jjrlFWLfel1amMpNjMpVkSeXseivuYwfmU3ohuia3VMr9ffIey0F11I4rEM9T02nQnDmNVACUFd96hpON+t4NYM3kYP2ewS30Fxkgj0TVUPpRhu15QQYBQYKX5gr6WrlTmu7zwq/BSkvGs9nLjuGsbuF+rUtiRxJdoR9DKS5u5+3yJT6neITqnomrvxKoGxUZ1+9QgVxJidnol2E8KykM7QwVAAAB3YSURBVPkcwvOoT9m0ZzTOeIeKaiCR1HWPpjY835vmm0v72FjPUl43/V1m2LujZXcGjhQgBcYylFWLrIyQol9g+6XOFKfnJ3CWbUTQwkiBKpQQGR9jW+ish7EVrYpHt6RozQiOlNY5mFpl2tokK2/sPrAQFliu5rBrCtmJtnbxd/9ZHH6B6vVIllcRjsOub+f5RPgOgqmI//4dX+A96ZN8NP99Ppx/msBY1A76NJMUv1f5APOFCbwNQ/mZLGq9jm40d9wBXqEU68d8zAeqPFRa5353AZCcCAt8vXWEL1w5TOVUD/u5i/3uwHdInBQatdUK+qpIya0/h7nLtmtHRL7ATklQrx0sU2NldKWAFv0DtUhJa86nvUuROBBlQNsGZF/wolzCgcNL7ElvYskEW2gsmeDLHlIYFoICS50cGkFgbDqmiyLGFopqnMFfEeQuBaiNJjvcB/AqZD5H+MBewpLFyjsT/sHjX2W3s849ziYaixNhge929vP5paPoT5fZcyHCu7TWX4gmyc5bhF5FSLCsvth4mlmrQ172+991dMTnzh5l+hM2Tr3fLUKOlekcm2bzkE0vB939IV6mx1xpkYcKy+xy6vx45iUmVUhWSMC+9qMio3lqbQ+p4z7+ikGt12+o2nM3GXqBwphrpThSy11yFzKAzWovh/YFJRUxJh2USJC0CU2Nj08ucXwqg7Yl+ZyH3QkRYQitHXRuSgiEbREWBE9MXuawv0JJRYDLWpLjYqdMu+kxVQ+uVey4ExjBDRXB5dbhPo1AG8kwFAt/LRwroesJYk/0uw27r26aKYTAZHx6pRRG0l/BSkGnImlPGbRjSPIJwkkQ0iAkVAot/puZJ3nAXbh2nQhJTXt0tMuX9D1cbJboJjY9o0gMJMKg0QTaRnUNVjPsZxbulOf1dSIch6Bs0R2T5CYa/HT2eQoyJiv7C4ia9rnULbNaz7DrUkzq5WVMs7nzK0aw9SwKAQJ8IfDE1bYZhqjt4C90kEHUjyvn0nQqFp0pQ1RIOLZvgYOZVR7OXOJR7zKeMJSkhSv6VcqvttnpV5kx1LseqXVDqhoPtInr8AvUdailKkVj8DZ9PpZ6gv9YeStH9y3y87u+T8Vq8DZ3g6x0+FDlBNbbNaeqFRasMVJrHsXTBZwXJIQhyU0lPrYb0vcRu6eJCz6dmYTHcueYtTfwt2rsnWjP8dTFPah5D9lp3dFq7sKANoLkak1EDE1t80JzinObY1jtIZhgjUZo0IkiTLYqkwjFB6Zf5GPvcVhteVTvncFqz776ewUEFQ2VsC9AgBCGQq7GkUwTR8b4Vg8lDJuhTy1MYauE/7T+EH8tj7HSybHe8emGDt11HxlInKrE3YT5acP9P7PAdO4EnklIjKGVuHg1jVre7PcuepMg02mE79M7sIvF9xom5tb4WzPPU5ExnpDXUsvX4hwXWyXChovd6GGaTXR3eGoq3jGMxiQaESeISLCYKEompCAtPKH4wP3P89l/eB8kLiR9EcuO17ivtE7Fa/HW7AUqVoOcDKhpB4WhrWMc0SUrBXnpEJmEmo5ZT2xaKxmmTrZR9S5mgOdMt49AGUM8vwALi6TzOfYvz9HLO5x9724+9UTE/sw6B8tPkpeSj2aW+VB6gacm0/wz8RHWV3OAy8RiHtHuIsIQE25fgRLpNJ19BTpjFv50g3emzpOXAl/YRCS83JhEXkjhL96dau4JV1ui9MWoYVwu1kusr2WZbJmBuAZuRmjQsSDSEm369dx/tfB9fvKB56kmGZ59YI5q/Or21bZIeCxzhsfdKnIrxqQQREb3V67G0DSCwCi+193Ds+1ZFrt5npmfodd2sFdtUisCv2GYOdnFqnUR9Ra63iB4/DDPvXOa92VfwDcJWTQd7eBuxkNRYPduInwfxgo09nj8V499nd8uP41EYotX+hBFJmE9yrLcyCIbFla9SbJDkyJuSRxDnCBDwWKcR6sGaZHgS5v/fdfX+P3JrwD9HRX0n1Mp+j2yrnYwuBTHXIqLAEg0SmhmrQZ52XftrSc2C0keZ00hnztLcgdDA6+H7SNQVzEG0+uXiHe0xl9yOXFhlnP5MXzV45C3zDF3gcO2oiw77C+uI4ShPjtG+mgJdzPCbndIhqAFxA+NFGhLoC1QUuOK/sMYmrhf9qWTxt0QuDUNt/vE99ZZnv5p9hz1XTZ51UVj2NQBG4ng+eAwK0sFnCUbt35n+8W8How2eJsJ9mWXxbDMpyYeYjVzFsgC0NQpAm2jbxEwi1BsxBkuqTo9JMtxno52qSU+9cSnox3mu0W6ic3lZpH1Rppe10YtuXiBwN2A1LrGaem+OLU6YClkpUwvb5G3u6RF/0hFTcc0Iw+xkwP9t2Krz1N3d4HuuGDMauEK+1X/TKO53C3RWs7gr8vBHK4eEEZvtVyJItwNyZ+uPsb+9Bo/n/8+MyLGRpGR9lY34R6RMSxqxVqSpqlTXOyN0Uo8rgRFVro5ck6XR3OXqFgNyrJfKb9tNCd705wNJ7A6op9YNmAX8/YTKLZaa8wvIS2LqU6PseeyBOUcf3Hvu+iVNI+99ST/eu5z7LUN/9P0X1Pb5fEn5XfwzUN7SS5mOLhWgW1cgl+ILYGy+3EUb2tlv5gI1pIcK5dLHPheF2uze3vdREIgLBvh2HQe2cPagzad3TH3pBbQaL4Xlvlq4yhfmj/E5Bct8qcbqOXNO1qO/3WhE9JPXWD/mTzB7gIfqz3Bn008QrncYn9xHYBeoohvcapYYrjUKfEVdZSlTo5T53chWxZOTeLUwWobsvMxdjvGjzR7Ig1xiAxbECeIXgRhDxMn/f5lxpA8eJDNwz6NA/CW7CV2W4JLseJMVGC+XUBGg99x3k2EUjTvKbP0uERMdzjoLl+LiVxPYBKevLSP6S8JvI0A1rfvO/yG0QlJq40IQ6ae7PJS7ShP7T6K/4Eefyv7LGUVURT90kQv9LJsJBk+U32Ap5dnaG76ZF5ycWsGq2Owu4aL04pzHxjj8fELZFWXQ6bBYuLy7+af4NJGkdzScHg+tqVAYQy601d9Gcc462mcSpE4VaJbl1w4UiLB4AuHozZoIhaLLxJqi2/39qJ9Z9sXmjWynx6tpMYWEm0MgVE0tIdqS+zVJqLVIbmdq0whEY6N8FyCsqIzG5OZaFGQHRJjWI4KnGuNUdtMs/9KgDw3TzIk8YFkbQ3W1kj1dpM+PE03cllPJJZKXnfb9/V6Bu+Kg1Pr74pS6zF2I8J6+eI1V9PVK72mJEtFkrIIKoJeMWbcapASDpGJWI2ztCMHfxs/l28YIUApwpzETAZMlhpkZZfkpjHQaAJj6DUd/IUAq9YZikaYdxWdYHoae7lO0ZEY6XK6PcGin0XTAvoejMtRiaWoyOlaheZSFmddUX4pwlsNkJ0eohsie+Ost1M0Yo/I9GWgo12Wm1nCTQ+rOxzP4PYUqOswUQztNgIovGyTzntcfDRPaDQ2CbZQSCRHnGWSsmQ9SNPYP0uhcxA2aiQrq3/jz9gOJBgCY9HRbr/4ab3ZL6t/OyoQCIFQCjVWpvPgHEFJsfp2w7sffplprwbA6cjwydUHeeHEHvxFib2xhu4GQ1ehwtQaVE6UiHIWYd4izI2/7u/NB5BejrC6MarVQzYDRNh7QyIspCAo27TnEvyJNiXVQmP4brCXjy89xMKVMkc67R1b6PR6hOuixsqYdIr2tODeuSUOZlYpyBCuq7e3rnt8tbOHC+E47qKNvbSKaXfv2OHqocYYqNZwjaES5Pmedz9PFo+hLTCWQcQCqyuQEbhVw1TVYLdi/As1RKuLyaWJx3N0Kopjk0t8oPgsB+11lLAJjE277WHVLawgGbhrHnaEQPX6KaadDqyv4/g+fPAYPWPQQnP1QT9kC3ZbV1gcL/D/zu3G7hRIGwOra9t6J3UVTf+QZ2BsVCBINmu3bYV5td6fKRf6br2phMcfPsW/nP4ckTGciTOcjsZ58fIuKk8LUusRrN++n387STY3kd9s4ALeVk28183VWo+AMfqVXdIbeX6EJCgIsjN1DpbXKMgAjeJEa45T53fhXbERneEbtzuBcByS8SJR0aM7nfDB8eeYtquUtlq4X2Utcfh89T7O1sZIL7xJGhH+AJKNKlQ3EeclU99V/QO8t2KrLqnRpp9lKyQqu5+w7BJUBO8unuIDfgvw0BgCY5O0LFJ1gdUdjiXS9hQoqZCee62MjMn4YCmMa5P4NqYUoeBa5gpAx0TUtWG1l8Nqg92MEeHOCbIGxnCuN8G5YBwV0n84fxSuNolTCjFWIillaM2m6cwkeJNtCnaX87HDYlzkL1YfZb5VwLrkkdqIcWo9GOYJZCsmNkgXuxTmmmsxMYbVIINVtXEaIIZs13mnkJk0m4eydCsSd7zJuNWgLNvYWzHVlomoaXgx3M3xxRm6qz5TVT0UsZGBYwyYBPOG4ru6X+PQEhjZfwaVkNeOiLS1i1W38KoGqzUcc+O2FCjpuYjZKXTGpXEgS2OPJElBMBkj0xEfvucZstK6oeHW+djh+WCWp9b2UDjXw3nhEroz2BTK28ly4vJXiw/3A5wrP3qAU3ouYm4KnfFYezjD5r0aNRbyG8e+yQOpyxzv7OH/XPoJji/OkPtEhtyFLvs31qBah6jXryE34nWh0ZzeqFB6AVLr0c4teHoTyUyF6kfa/OS+kzyePcsj7jKeEHhbvZ6e7+V4qn2Azy3dQ/ZTWWbOdLDnN/olu0b8UBhLom2BVv0KMMl11V8uhWPkT8HY8QZqdQiSm9hOAnVdqwKR8kjyKaKcQ2dC0plOMJmE3TPrjPtNHs5cukGcNJqOdlmJ8tS7HhONqL9N3iHiBH333maQImy5qPBHuK+teJNIecQFnyjv0JkU5HfX2Vvc4F3pkxy2Q57tznG2NkawnGb38zX0cyfv6IHgnU7QdShuJjj1qH/e5U1A4lvcP3WZ/7r8TSqqR+m6enAazUaS4VJQYqWWZe5S0E9G6QY76r0dJAnyhvqZHe3g1TVqrTY0i6ShFyhhO8hCHuG59PZUaM26hHlB44AhySYUJzd4R2WJgt3lYGqVktXiXmcRuXVrGk1iDBtJhvmwSLfrIGO9MzqUXud6rsgOj09e4LQ/zlJ5D/kfpoutEKhD++nuLdIdU6w9DKYUsXdmnveOn0IJzRdb9/IZbfMfnnkrxe867FrXyNXNN0VQf8TtxUhB0ekwoXqkb3peEwxfrR/l8y/ciztvY21sDmXSzbbCGORKlawxhNkim7c4mD5sbAOBsmCsQJLxWHswRf2BHulSl79/8Lvcl7rCHmuT3Vb/pPTVNhPyutvq15ZK2EgyLHbyRIEF8XCkPt8OhDAoBBWl+bnCcS6mx/iD0p7XDpz+wItJunuLrDxq052O+QdPfOVaQckx6fBcT/HPL3+QCxslyt90qPx/z2J6PeI3yYp/xO3FSEHB7jKhUjfUgoN+VYPjazPknnPwVzRyo0Y8hEk32414eQVWVsmOP3zLyinDxnAJ1NZBUKRAZjOITBqdT9M4mCPMSlpzhrHJBtPZOnvdVSZVg7xMrhU8vEpLhzSNpqYtng+nWYuzfHb5GGcvj+NecZCd+o5xRxkjSDDYQpKVASXVolfSiCP7kO0Aag3oRf108yQBpRC+j7AUJptGpz1QAu1aaEdSPWrTnYvIVlqMWU0UhrXEYS2BE8Ec59fLdNd8Cg3Tb2MyEqfbgrI0cUpiuQrrB1RW3wlYkxPosSLNWYeS1d4qMNxvG3F1kYmBTs/GbRiclsbEO+WNHSzS9xF+ik7WwpPDkQjxgxgqgRJKXXPnde6ZpLbfIRiD1CMb7Ctu8NHCJR5LnyEnQiZUhC8VNjf2MdFoXoo8jnf38r3Gbp58+ijumiJ/VnP4ZBPZqmOu7JA6Z9d5KW0Us1ZEQa5y8P4rnPuFOdyqoPJsGWeje+2Ankmn6OzJEaUVtf2Szt4I6cdMjtXJuwEfKMzz1sx5bNEXnoW4wNOdvbzY2MWp9XHUd3JMLGuy55pvvi6vdwCFQaLIp7t0KlkwFp796jI/OwVhWTQe27NVNaLLI/75a61auC7rViFo1XwmzgRY9S5m62D+iB8BIRAzu+juLVLfqxi3G4O26G9kOARqa9UkHAeRTmF8j+5Yv1R8rxLzwZnTvD1zjiPOCodsh77Zr5h+Nc4UkRAZzUI8xanOJGdqFdKXFOlFTeGlJubES686ob5TUELgo5Ay4Uh+hbNz43RSLt15C5F4KMdC2Yok69EZs+jlBJ3ZmAP7l5lINfnx4kmm7U12W5vstizqusezvTIbic+FTpkzGxVaa2nGVwz+coRsdHfMLnQYcFRCx4PEET+wN9W2ZqtqRLcksXa32FfZ2KoDd2Obk8gkBEZjQonV6CCbXfRoMXRbML5LWLCI0+CJ0Q7qtZEKmfb7Z5lyGXQ2TW88zcqjLmHRwFyXo9PLzPg13pt7iVmrRkW9OhR/Ngo5H5c4H07wqeX7WW1maM7n8K8onLph/HSIvRn00yZ3qDhdRSLxBLwn9xL20YRLu0scH58ladqI0EOGPjqlSe1qkPMD3lJc4dHcBQqqwz5nlayIqGqPM90Mi1GRL24cZbWTZeHlCYovCnY1DbkzDWSjC9XaKJvqNmMENyS+7CRurhrxyMwVjmSWycuIxNjE9NuNXIpj/qrxMOc6FfxLNrLe7leNGLmSf3SEJEk7hAVBlDUjF98PQqi+QOG5RLuKBBMu9d0WM++/xHvHT/Jo6gIPuW1sFEoI5E2uPOjvnM7HJb7aOMr3N+ZY+do0/rLhwMtd1IkzmDjGRP222TtdnKC/i1Io3peq8Z7Ut2nqmFO7cjS0RzXJUI0zjFkNnkhdZEypG8rwKyGIjORUN8dXG0e42Crz7PkZZN1m15OG7GdO9MczSfo7pzfBeN4NkusVSdBvSLcDub5qRDAV8+Gx40xaNQqyn+qcGENoYs7HJT52/iEaKxkmLmtMrb7Vbny0g7odxClFWBTE2Xi0g7oeYVn9UhsTFXQ5h07ZtMsuiSvpjEmCsqA7qXlXbpV9zhoV1cYXN4pSXQfUdb8h3ulonGqc4cvVI7ywvIvuuk952eCvJVi1Ljrceqh34ERqEo3d1jgpSb2d4lIsKMl+8zJbqGtC5UvTH0cZkpYhWRlQUG3yst+NMzIJEQkdnXApTtHQPp/ceIhvXdpLr+3gXnGwW+BtBOhetK2bPA47vt1jzYfYB+QPcURgyBFCoG2JdiTYmoJqk5W9az22QhNT05qLvQqN5SypeQtvI9paYO68d3hQqFBjt0B1JJFRSMS1s1BKaG7RcWag3B2BkgqZSSM8j/Ufn2Pt7Rrjx0xOVsm7AQ/lVrnHX6RktTjmLFFWBvemcxEazQu9LN9qH+S55jTfPXEQZ12RvWiYPh+gOl3Uaq2/2mq1d7RLwHQ6+BfrOJspNi6n+cShh9nrrvIe/yKV6+IXnrCYsWIgJjINIlPHFv3GhgBNHVPVipO9Kf5o4R0s1P//9u5tt43jjuP4d/ZELimedKBkyYlcy3aUOEnTtEHau6ANEKAIgl71DXrfi170GYoCfYA+QA9AetMCCdq6aVokDRq3SOLElSMfJFuyRB0oksvDLrk70wvSjuOoSdPKEk39P5cCRCwHJH+z/52Zf4HkbyVOvdlERR2sdg3VizH1BomE0wNjK8WZ3A5X5k+A8jD+Z1vPP/RsG+07xBkLN9vhtFOnaFmkB5/FzQSux5Nc2Flk9oKi8N4m7NUHG3P1SE40D53RuFsBxasOScqllmSw1R7cc5rEsHkwAXVn0YNtf9KiIZvFZNK0pxXlr+xSzjZ5YeJjZt09nkht8PjdlUuf/XLememvxzMst8ssVyfJrthkNg2Fq22cj2709+NE0bH4IJs4xgraOIAbZLnWngSgnr5FzorvKYtan9yFDmZGGt0fT2OoaYvNZIzr3SlWdsfp7GSYvaFRFy9j4lgWQRyivNPBzXSJ0y44I7hIwlJo20Lb4DianKXIWO7dEnMPiyDxqYZZsushyfL1I77g0aTCCK8W4bSc/uGw+x2JNkRl5gMPKJVKYc+UMekU7YUSwSMOcVoRThkS3zC+uM33H/0n406Ts94mOavLlBUDn15aeyeUVmPFr2vPsdKe4O3l0/hLabw6lK9EuLUQe6ve7447ouW8/ZhejAkC6HaZeq/IxeRJ3ippLpx/jFP5Xb47folXspV99+quxRG/qH+DW2GJv2/MU1vP4zRsCteg2DDkr9RlxdQhsod05ipGkDGYegNHGzJzPpeCk3yUXWLKipm0fTJWlyhvkUwWsLWGxtEvQz/wgLL8NL3ZcXoFj8pzLulnq8yMNXlp+jInvV0WvQpnnHtPfXC5P5yAu89GlrpzvLr8DFElQ/kdxcSF65gwQjdbmLhHDMcmmO7SSb9BnlKMvWUYu5ynO1dgvXaCteky9tcNL2c3uLdlwR234jy/Wfkqe5U8hQ9czv6jhd2IUGsb6GarH07HbTyPmIX1XzdNFOL/kdTqUG/gz09yrTHJpdIc573bjNualNWjl1N0p3zSzc5QNHU9+BKf69EteoQTNtFEwlMT28ykG8x7O8w5exStGFf1jzYJdJfQGHYSl12dITH9232NxVK0wNX2NP+qTxPfGCOz2+8zZFrtwQkGvSMfvCNnDEQRqtXB3UuR3Uhhd23+UjjDj+0ujtW/E7LRJINSyseNMvXrJfxdi2xF49Q6qHZI0glH+rnd0DAaO4Ja4LPl5wj3aTM/kpIEO4xxOjZh22M1dhm3IqZt71MHO4tDYAx2GLOyU+TPhUXSpR7n3B2KdptO2dAIXOx2HreSw8QxunN0XR8OvsQ3lmHvMZf2rOHs+TV+NPc6OdVjwja4KFKDZyKB7nIxGmcrzvH67lO8vzmL1hbGgE4snMtZJj5McNoJZzb2+k3cag2SZksemt4jabZQnRDVCJjebWBSLvoPGZby5/ddsmx1E87VaqhuD9VoohsBOkmOZ3fSI2C0IVuJaS75rHYcNk/neZIAPWzLpw6YCSPczTp2O4OzXuCXe8+zkN7i5bErTNv+F7+AOFB2tUnm3VneWH+a+rfSvHjqdzyRWuf0N29y81yJbiHPXPMkqtnBWttAh0dzfumBB5RxbGIf4rGEKb/JrB2RHvxQJhjapgemRyWxWOlOstErsrw3RWdjrN9PSytUAsVlTf6d1X45LwjQMrvfn+43LTO9LjoI7v758+akcvL40XJaMamaS7dos94bp+rt0ExS6MTG0ozk5MskGjohyrZwgyLLQRltFBX/Bp4KqSY5akmGMHbwRu/tD5+oi7+tMZbF7WaBntFkVcyzpVvkvZD3J3L0iun+wxfXhVEJKKo1pt8tEV11uLT8BC/MPg771NdVrHBbChVDetcMOmWCMgalDdmVJrrZgl5PNumJ0WE03lqVSVMit+bxs8b3+EnO4G8pJiqGdDWBndpRX+WBM0mCbrZQvR7TF8dZaS5wLb3Ab0vPo12DHSnsUOFvGdzba8h09MEyrRb5ax3SVY/Vs5P86tFFclaHpzO3OOdvsnS+zO2whL+VYrrZQXW7/VYnh7zd5MADKtmt4v6pjmcpcsr64rYPd1qT37fcUR7Wi5FkDPHKTazVNTKWYv61wb2u7ndBNtqM5p4znfTv8ANw/1hj5o3B+77/90Eb4nj4Tzh42OlmC2fpJm42Q/apeV5bfJIzuW1+MPFXTjpQWbjIq+4zbK+WmHw/h72VQgMmesgDChiUnR7IKwvx8DMGzOA7chxL14OytDg6Rvfb5WBb+FuGj27Msj5e4KxfYTHV7/ZwprhDtZ4lzro4GR+VJJhD7sk1HKeZCyGEODw6QXdCVBRRfnOD4tUSzUeK/PTFl5ieqfHtEx/zw9nf83PnBT6YfRpvu4QF6FbrUC9TAkoIIY6jQaVL397ErTcohHNUz4+xZRXYncwyY0dMeQGJB8ZzwDn8uJCAEkKIY8zEMbTa2BtVTrydJiy6vP3h1/hO+RlSVcXcpQB7cw/TCL74xQ6YBJQQQhxjJo77G3LX1knd3iRtKYp3Frhp0z+xZ1Q26gohhHhIDdkCN/Vleq0opbaB1Qd3OQ+teWPM1Jf9JxnP/0jG82DJeB6s/2k8Qcb0c+w7pl8qoIQQQojDMnqtO4UQQowECSghhBBDSQJKCCHEUJKAEkIIMZQkoIQQQgwlCSghhBBDSQJKCCHEUJKAEkIIMZQkoIQQQgylfwOSKGoPACnckAAAAABJRU5ErkJggg==\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "plot_example(torch.stack(X_example), y_example, n=5);"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r5OH322Rq-1N"
      },
      "source": [
        "### Preparing a validation split\n",
        "\n",
        "skorch can split the data for us automatically but since we are using `Dataset`s for their lazy-loading property there is no way skorch can do a stratified split automatically without exploring the data completely first (which it doesn't). \n",
        "\n",
        "If we want skorch to do a validation split for us we need to retrieve the `y` values from the dataset and pass these values to `net.fit` later on:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "PZ2clJzBq-1N"
      },
      "outputs": [],
      "source": [
        "y_train = np.array([y for x, y in iter(mnist_train)])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dQhh6yaAq-1O"
      },
      "source": [
        "## Build Neural Network with PyTorch\n",
        "\n",
        "Simple, fully connected neural network with one hidden layer. Input layer has 784 dimensions (28x28), hidden layer has 98 (= 784 / 8) and output layer 10 neurons, representing digits 0 - 9."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "tojAhRAoq-1P"
      },
      "outputs": [],
      "source": [
        "from torch import nn\n",
        "import torch.nn.functional as F"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YHfKnJz_q-1P"
      },
      "source": [
        "A simple neural network classifier with linear layers and a final softmax in PyTorch:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "I6veVp04q-1Q"
      },
      "outputs": [],
      "source": [
        "class ClassifierModule(nn.Module):\n",
        "    def __init__(\n",
        "            self,\n",
        "            input_dim=MNIST_FLAT_DIM,\n",
        "            hidden_dim=98,\n",
        "            output_dim=10,\n",
        "            dropout=0.5,\n",
        "    ):\n",
        "        super(ClassifierModule, self).__init__()\n",
        "        self.dropout = nn.Dropout(dropout)\n",
        "\n",
        "        self.hidden = nn.Linear(input_dim, hidden_dim)\n",
        "        self.output = nn.Linear(hidden_dim, output_dim)\n",
        "\n",
        "    def forward(self, X, **kwargs):\n",
        "        X = X.reshape(-1, self.hidden.in_features)\n",
        "        X = F.relu(self.hidden(X))\n",
        "        X = self.dropout(X)\n",
        "        X = F.softmax(self.output(X), dim=-1)\n",
        "        return X"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rty5y1hDq-1Q"
      },
      "source": [
        "skorch allows to use PyTorch with an sklearn API. We will train the classifier using the classic sklearn `.fit()`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "id": "KkjJEufpq-1R"
      },
      "outputs": [],
      "source": [
        "from skorch import NeuralNetClassifier\n",
        "from skorch.dataset import CVSplit"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4QtvtZMKq-1S"
      },
      "source": [
        "We might also add tensorboard logging. For that, skorch offers the `TensorBoard` callback, which automatically logs useful information to tensorboard\n",
        "\n",
        "**Note**: Using tensorboard requires installing the following Python packages: `tensorboard, future, pillow`\n",
        "\n",
        "After this, to start tensorboard, run:\n",
        "\n",
        "`$ tensorboard --logdir runs`\n",
        "\n",
        "in the directory you are running this notebook in."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "id": "KeRIksQHq-1S"
      },
      "outputs": [],
      "source": [
        "callbacks = []\n",
        "if USE_TENSORBOARD:\n",
        "    from torch.utils.tensorboard import SummaryWriter\n",
        "    from skorch.callbacks import TensorBoard\n",
        "    writer = SummaryWriter()\n",
        "    callbacks.append(TensorBoard(writer))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "id": "zSZKfehwq-1T"
      },
      "outputs": [],
      "source": [
        "torch.manual_seed(0)\n",
        "\n",
        "net = NeuralNetClassifier(\n",
        "    ClassifierModule,\n",
        "    max_epochs=10,\n",
        "    iterator_train__num_workers=2,\n",
        "    iterator_valid__num_workers=2,\n",
        "    lr=0.1,\n",
        "    device=DEVICE,\n",
        "    callbacks=callbacks,\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vNr_KEP3q-1T",
        "outputId": "a56a5865-66ae-4d0b-fbff-267d305b1466"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "  epoch    train_loss    valid_acc    valid_loss      dur\n",
            "-------  ------------  -----------  ------------  -------\n",
            "      1        \u001b[36m0.7973\u001b[0m       \u001b[32m0.8988\u001b[0m        \u001b[35m0.3627\u001b[0m  13.3156\n",
            "      2        \u001b[36m0.4256\u001b[0m       \u001b[32m0.9191\u001b[0m        \u001b[35m0.2840\u001b[0m  7.8078\n",
            "      3        \u001b[36m0.3589\u001b[0m       \u001b[32m0.9297\u001b[0m        \u001b[35m0.2434\u001b[0m  6.6143\n",
            "      4        \u001b[36m0.3169\u001b[0m       \u001b[32m0.9374\u001b[0m        \u001b[35m0.2165\u001b[0m  6.6950\n",
            "      5        \u001b[36m0.2919\u001b[0m       \u001b[32m0.9405\u001b[0m        \u001b[35m0.2016\u001b[0m  6.6883\n",
            "      6        \u001b[36m0.2680\u001b[0m       \u001b[32m0.9474\u001b[0m        \u001b[35m0.1831\u001b[0m  6.5840\n",
            "      7        \u001b[36m0.2542\u001b[0m       \u001b[32m0.9496\u001b[0m        \u001b[35m0.1718\u001b[0m  6.6463\n",
            "      8        \u001b[36m0.2432\u001b[0m       \u001b[32m0.9522\u001b[0m        \u001b[35m0.1629\u001b[0m  6.6640\n",
            "      9        \u001b[36m0.2291\u001b[0m       \u001b[32m0.9548\u001b[0m        \u001b[35m0.1536\u001b[0m  6.6585\n",
            "     10        \u001b[36m0.2232\u001b[0m       \u001b[32m0.9563\u001b[0m        \u001b[35m0.1479\u001b[0m  6.6471\n"
          ]
        }
      ],
      "source": [
        "net.fit(mnist_train, y=y_train);"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "efHqyfCLq-1U"
      },
      "source": [
        "## Prediction"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "id": "63f18IAZq-1U"
      },
      "outputs": [],
      "source": [
        "from sklearn.metrics import accuracy_score"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "id": "LONeNZs8q-1U"
      },
      "outputs": [],
      "source": [
        "y_pred = net.predict(mnist_test)\n",
        "y_test = np.array([y for x, y in iter(mnist_test)])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "l1iY1uS8q-1V",
        "outputId": "7629d989-0013-4987-ef75-6f3c66d7a6e1"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.9578"
            ]
          },
          "metadata": {},
          "execution_count": 19
        }
      ],
      "source": [
        "accuracy_score(y_test, y_pred)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K9V7S_Urq-1W"
      },
      "source": [
        "An accuracy of about 96% for a network with only one hidden layer is not too bad.\n",
        "\n",
        "Let's take a look at some predictions that went wrong.\n",
        "\n",
        "We compute the index of elements that are misclassified and plot a few of those to get an idea\n",
        "of what went wrong."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "id": "7YuhO4G_q-1X"
      },
      "outputs": [],
      "source": [
        "error_mask = y_pred != y_test"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2zJqiCP9q-1Y"
      },
      "source": [
        "Now that we have the mask we need a way to access the images from the `mnist_test` dataset. Luckily, skorch provides a helper class that lets us slice arbitrary `Dataset` objects, `SlicedDataset`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "id": "EhXWuo_3q-1Y"
      },
      "outputs": [],
      "source": [
        "from skorch.helper import SliceDataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "id": "F8rGoDl-q-1Z"
      },
      "outputs": [],
      "source": [
        "mnist_test_sliceable = SliceDataset(mnist_test)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "id": "U_dWUitgq-1a"
      },
      "outputs": [],
      "source": [
        "X_pred = torch.stack(list(mnist_test_sliceable[error_mask]))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 120
        },
        "id": "vGW6BB6Gq-1a",
        "outputId": "30ac782a-1ad2-4550-e1ca-aa0e638429a4"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 5 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAABnCAYAAABVe9YVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOy9eZAc2X2g972XZ2XdVX2fQOOcwZzgHOJohjM8REmkREkkd5da7UreDa6sXdtShGPtXcdK1q60sryxDtkOSeGw1pKtDVkHRVE0KVISObxHHJJzYgaYwY0G+r7qrsrKysz3/Ec1Go0ZzMludDemvggEgOqurPdeZb7f+91Ca02PHj169Oix25A7PYAePXr06NHjRvQEVI8ePXr02JX0BFSPHj169NiV9ARUjx49evTYlfQEVI8ePXr02JX0BFSPHj169NiV9ARUjx49evTYlexJASWE+IQQ4mUhRFMIcUEI8chOj2mvIoQoCCH+cn0tLwsh/uFOj+lWQAhxSAjRFkL80U6PZa8jhPj6+lo21v+c2ekx7VU2reHVP7EQ4rd3elyvhbnTA3irCCF+CPgPwD8AvgcM7+yI9jy/C3SAQeAe4AtCiBNa61M7O6w9z+8CT+30IG4h/mut9f+104PY62itU1f/LYRIAYvAn+/ciF6fvahB/Tvg17TW39FaK631nNZ6bqcHtRcRQiSBjwG/orVuaK2fAD4H/OOdHdneRgjxCaACfGWnx9Kjx+vwMWAZ+NZOD+S12FMCSghhAPcB/UKI80KIWSHE7wghEjs9tj3KYSDSWp/d9NoJ4NgOjWfPI4TIAL8G/Lc7PZZbjN8UQqwKIf5OCPHYTg/mFuHngP+sd3G9uz0loOiaoSzg48AjdE1S9wK/vJOD2sOkgNorXqsC6R0Yy63CrwO/r7We3emB3EL8K2AKGAV+D/i8EOLAzg5pbyOEmAQeBf5wp8fyeuw1AeWv//3bWusFrfUq8FvAh3ZwTHuZBpB5xWsZoL4DY9nzCCHuAT4A/K87PZZbCa31d7XWda11oLX+Q+Dv6D3z3y//GHhCa31ppwfyeuypIAmtdVkIMQtsVkl3rXq6BzgLmEKIQ1rrc+uv3Q30AiTeHo8B+4ArQgjoaqiGEOJ2rfXxHRzXrYYGxE4PYo/zs8D/vNODeCPELjY/3hAhxK8BPwp8GAjpOvW/rrX+lR0d2B5FCPGndB/4T9I1mX4ReKgXxffWEUJ4XK+R/ku6Auufa61XdmRQexwhRA54EPgGENGN3v094N5X+E57vEmEEA8BXwaGtNa72lqypzSodX4d6KN7+m8DnwJ+Y0dHtLf5F8Af0I3mWaO7mfaE09tAa90CWlf/L4RoAO2ecPq+sIB/DxwFYuA08JM94fR98XPAZ3a7cII9qEH16NGjR493BnstSKJHjx49erxD6AmoHj169OixK+kJqB49evTosSvpCagePXr06LEreUtRfLZwtEtyu8ayZ6lTXtVa97/V9/XW88b01nNr6a3n1vJ21xN6a/pavNaaviUB5ZLkQfH+rRvVLcLj+tOX3877eut5Y3rrubX01nNrebvrCb01fS1ea017Jr4ePXr06LEr6QmoHj169OixK+kJqB49evTosSvpCagePXr06LEr6QmoHj169OixK+kJqB49evTosSvpCagePXr06LEr2YvtNt4YIRCGAUIibKv771egtYYwREcRwnGQfQW0bSE6IXRCtFIQBOhYodsBOuzswES2AGkgpEDHMfQq1/d4hyFME5FIdP92bHBskBLt2mjDQFsG2pIIpRGdCBFrRK2JrtfRYYTy26DinZ7GO5ZbUkBJx0EkPYRtowtZlH3jaRrVJrpag8E+Fh8uEuQFTlnjlhVmW+OsBhh+iLFUIlpYvMmz2AKkgZFKgmGggwDl+93Xe4Kqx63O+iFVppLofaOEKZt2v4NflEQJgT+oiV1NnImxswFRaKJLSQxfkjtToHCygdEIkNOzqFbrjT+vx7aw9wWUECC6lkohu/8WSQ+RSqIdmzCfQDmv1qAAhNbIOCbMJWgNCYK+mNg1UJaB0dagHcyWidNIdD9nj23swjAg4SIsC6RAQleTimO00qDVnptTjx6vi+h2ghe23dWakkmCQoIwY9AakPj9giiliUfbOImQ0XyVI5llKmGClzKDNH2HViVFat7FlgLD3Ptb5F5mz6++OTJMPJAnTlq0RlzChKDdJwgKmtjRxMUQw1avep/WoKtprEqWMKMYO7zASKpKue1R6zgEoclCyyHqmAz87SC56Rl0FO3ADN8+xtAASx8cp90nMDogO2A1NZnpALMaYCyXiebmt3UMwnGQjoPWGu37XQEJt5ZgvHpI0pvus1tpfrud9fWXCRc50IdOODSnctTHDYKcoHWwg5vxSXtt9nlNPLPD/uQaWdNnwKoxapVoKoe7M31UI4+/9m7j8lABbz7FxGoRarWdnuE7lr0toIQgHshTO5LGL0iqx2JkLuDwyBIf6D/NoFXlB9zL9N3AB6W05tlOmhfb4+SMFsfdKxRkhCsElrgWO1KKYz64+N+R+7QBe0xAxQNZSo90uGP/HH5k0QotlkoZgu96eMs2Oa1hfmFbN1PpOIhsBhGGqDiGTueakLpFuObjNLqaKYC+tea4qxESYRgIz6MzmifMWiw9YJC8d41j+TX+5ejfcIfV/V6MdQ1LrseHScT6RVqoZAmF4uHUWZ4YP8xfXriL+GupnZhRj3V2v4ASYkNdl5k0OpNCOyZRNkHsSGqTNs0xQZhWJEfq9KWa3JZZZMpZpmA0KEiJJ+wbXBeGjAZ1e5W09MnJiKSQOMLEEtcEWijbaLk3T8Mi1uhQ0gxtsrbPWLJCrCT1pEfoCZR9Y9Pn1g1AIAo5wpE8aDBaBWQYw1qFeGl5ez97m5HJJMLzEKnuphi7BkKBUBoZxJgrNUS7g67ViXsn8K1HCKTjgGUh+4tEAxnaGZvKAYtORtCZCDheWGEquUq/DPDk9YIm0CGx1jR0SFNde75DBN9tHuDri4doLSeR7Rqvtr/sDWQyiUynwDDAtjbMnwDaNFBJF0xJ7JrEjkQoMNoRIlLIVohoB4hNa6PrTVStBkqjo/CmWAl2vYASto0xNID2XMp3FykfkQQFxYE75jiUXmXEqTJsV0jKgFGzjCcDcrJDToKFwJPWa1570hT0G0tIwBM2hhAbJ6tbARGE2Msm026Rxw6d45eGHufx1O38zvAPIZRBlLYxhdy+076QtI4OsnS/hZaASCJiGPxeBvvLa3s3OkoIxMQIrakctQkT/SNl7hyYp9LxqHccrqxlcb83TGJFk3+5jnj+dC+KcouRjoMcGUKlXOYfKVC5LyCVa/GhfS9xwF1mwlpj3KzgiZgR07nuvYEOWYoDmkpyIhjjpD+G0t3NO1Amn3nqPvqfNJhcjRALazsxve8fIWD/OLVDWSJX0C5I1KZzepgEfzLETIbsG1zlBwqXWemkeXZ5lEbLJZ7N4c13hRYACvJnQ5InF9DtAFWp3pTI5t0voAwD7dgoz8bvk7QnOuT76/w3E1/hfmcZTxq4wtwkWCTgvuF1FQpLGKQ2CaRYa2Ku3zRDrUGLV759bxArjLag0zJxjIgjlsG0s4hOxMS2gbK2URgLgbBMgqxBeyhGGxoMDQqCsyaOFNe5bPYayrO7cysKfnLiJT6We5qVOM1anOKb2cN8ZfZutJQkl1ycRKJr2gyjXmDK98t6dJ5IJFAZjyjr0BrW3Dk1x6HUMj9feIID1lVtKUGsFQq9oTEpFC0dsxg7VGKP88Egp+uDGwKqHVu4iya5c03Mahu9WyP41v1uwjBACoR4xR5lGET5BK1+SeQJ2v1dn/xV4nTMwalF9qVKfLhwgh/3aizELf4keTeX/T6+bh2kRRrW3yJiQWLNwPPcrlFU3pw9cfcLKNNEZRJ0cg71fYoP3nmKqcQqh6xVPGlgYbwlraelO1yOBC1lcSUqMB/mUfq131+PXZySALX3NhURxZhNMBoGlU6C+CZtjNLzEKNDqKxH6ZjggXvOkjQ6OEaEH1s8c+ZOMmJva6qy5uMtuYQJm89dupPpwSJHUksccRe4IznP2oNJlltpLh3sJ3XPnThrmv6nK8jlMqrRRNXrOz2FvcVVwXTsEPWDGdp5Sfl2jcpF3Lb/Mh8dfJYhs0pBdu+rQIe0VMhMLPmL6ruY9fOcrgywuJpFhRLRMBGhwK5K7Mqmj1EweD7Emit18x874Q5N+LUxMhlEOkU8XGDx3RmCAihLozbv5hKivpB8XwnPiphcDw65StbyuTd1hX6zziFrBXBICsmD3gUOOUv023VeHhza+P1ObHDCOoAM+3DXQtxWizgItn2uu15AYRhEKZtOzsSabPLrw4+TEhaWcN74vTegpWJOd0ZZCnM8Xx/nTGWA19u2O5GJu7b3hBMAcYzZ0lh1SS18Y61yqxBegvZkHr/fQtzW4DfHPkdaCjxhUVUdHhm+46adwLYFraHWwFm0SJtpamczPFlJ0Dpgc8fwLHe6M3x88hSukHx+coLH776dp2YncWoZMqIb7t8TUG8NYRgI26Z2OMPiQwI56POv7v0SP5i4QL+hyEsXicAQHgAtFVJSiufbE3zq7HHaKwkyZ0wmTwUY7Riz0uwm5VfrqEr1ugOojmOiXazpinSKeChP5WiKwZ+8wk8NP8eIVWbIrGJs2s3SMsQTGlsILARyk5ZlcM2dYQgThSYlHR50QmJd4gOJVeLitWsFWvHTnZ9mvjKGt2iTOOPBWmnb57qjAkqYZjc8tJCDTAoRK2j56ChCt/zrE+T0tftFoWiobkSdFAIDQVvHLMWStjZoapu6cgm1yVqUoq0tGrFLPXaphB4vVwepBw6lapK4YiNez4QXw8iaYi/ao7RtEeS7+V2Dbn0jgmm7EaZJJ2sSZAWeG5CU3cjIqupQUgZyj7qerqMTIlptrJqLN28TtG1OME6145J3WhzPzZA3mzRil4lEmYVClvlDacJkltwZC6NaQ4fRTXM272Wk6yLGhtFJl/q4gTVaZ7xYYZ+1QkHGeOuBTYEOmYtaNJXkqfYEJ5oTnKyMEF5O4q1JkksKu9xGtiNEowWdENXy0TdBE9gyhOgKp9vS1PZJjmdW2Wev0m/U6Teu9wm5QuAIAwNxXeDXZl5pfZJIEAoDg+u99xGjyQqXBoaRoYFOJRCO0zVbb6MveccElDBNZDqNcB0a902wdszEaEN6NsZqKBIXS3DuYvd3I4WMNVFoshgbuCKmomyUlngyJCkiZqIMny7dz7yfYaaWp1xNErdM7EUTsyVwSxq3pJARmK2YdKjJ+RFG8w0irBTI1TLRHgsxB1BZj86dLd5/4BwfLpy4eQEgCZf6hEFrWHNvfg1PGNRVxLfao1wKBrBqe1h7WkfV6wjfxyxXGF3Iom0LlXRRXp5SdoA/P3yITgbSD67wa0c+x33JS5z8xBiLQZavfP5dTC0NQMsnXivv3TJaNwk5NMD8B4fxhzTFBxb51YOfZ8BoMGnGpGRiI1R8JQ74s9rdXPL7+evn7qTwjIld0xw8W0dWW4imj2o00UqhoqgbjbbHUh6EabH0A1nSH1ngwewqP1P8NocsHwuBJa7fziVyyw6lFgYfKrwI74Lv9O2j/UyWRK2IqtW31RqwcxqUYSCSHjrh0Oo3aI3FGE2J2ZIoQ+Am1kNOtOrWyYog7khW4iQGmoryaCuLnNEiLX2mwz7O1AZYqqeoryYx1ywSDUHqisZuxHjzbayZVYgidNCBOEYFAepNnJ72nu4ECIFyTPrydd6VnmbcLCFv0tetLZMwCVEmJm+3sIRBSMR8mOdyu4jssCd9epvRUdRN3G63oVLdeF0AXl8RGe3HL5qsHE3iipAho8bd2UVa2uCLg/egPBcZx3vb1LndrPuddDKBP6QJxjvc33+F9yba15n4Y60IdEhJmZxtDnG+1oc7a1E41cKsteHCDHGzuYMT2UKkIMjDJ0ZOst9ZZspskZVvz93xVhm31jieucJ0vkgnM4Cb8hBRhPD9batMc/MFlDQQlomcHGPuhwfxBzX6QItH91/k1NoQ/ko/dkOj152dym9jzZUwyy593+rjn9U/iVACsyEQsSBOaOKEwvAl3pzAbmqGmhqrFWO0Fc5aG9GOkNUGula/VupH6656egti5LKIdJragMNoapYpe5mc7HCzvm7tWrSHYwYnStyWXABgOkrxB+feTX0xzciVvWkyfbPolo97qYS9lqD5TJZfMH+G8WKFf7PvC0xZNUiHBCMp7JKJLFdvirN5T3E1IOKOw9QPpqmPGeTvX+LBgcv8SPbFDY0p1DEKxd+2sny1djsvlkeY/c4oiWXB4PkQe7YE7YB4FwY6fD9oA7JGi4xs3zSzvSEEI0aL+xMXCUcN/o8ff5TFB/tJzQyQuRJh1WPci6voeuPV7pnvg5suoIRlIh2HzmgW9f4yH5s8xXvSZ3jYrfJHmQP8b0//BMoQG41AdBAQXZkFIRmoNel7NocIQlhYRrcDZF8RVUgj2iEsr65XH76mumut0KxrQe8EW78QiHSaaLSAX5RMeiX2WRWyN/GkrlyLxFCDR4fPc7szh0Qy3ekjOJmj/zykLzX2nGnlraBaLTh/CWEYDDnHWKunmJ5KcWW0wN12Ay8V0BpIo2WCxPRr5+m9U9kcELHwg+CM1PmPhz7Ho4kWJgaGkOvh44qWCvlS5U6++NIxrFmH/Z+tIy/MdfcN378ln3lt6A3L0c2MhR0xHUbMmCnzBEceWmAxyvGfLjzMwski7prJUKeAtWR2jw97TkCtx+0bI0OEI3mq+x0mcgsccJepKZcn2vBMfR92FZxajPTDaxlJWoOO0S0fWbUgjIj9NjoIEM0W0jQgjFDtAN3p3JI35VtBJxzCjE2YEqTMAE/o652kSmxUPdhyhEAbEiljXNk9uYY6pq4SWA2BW4mQrQ63rnhaR3cPSUYtwFtxCfIGs50iK4nLCKEJkwKrJbtZ/j02kK7bTcBNJ6iPS9zRGvuKJQaMBo5wCHRIQwWU4pgTnSEWwyzfmpvCvuyQWBLImt+t+RhFt+w+IGLBWpwibfgo/Ot+FmtNVXVoazgTFnmpPQqAFApLxBxz5jhi1YiBlhaEWlJRzrrLxGY+zNPW18SCgcaRIQaK2905jlpNQmDAqOOKkEP5Fb434dFyXNr9NjJMYvrBlhXXvmkCShgGwnFY+8Fhlt4Tkxmo8MnRb3G3vcjvrL6Hv750O8Fsiv3PtXEuraIr1VdN8KpjWmvdFUTrr9Fq3dTyG7saIYn601SmLFojmgPOEv2GsxEgEWoTom7xWBFt8Vqtm2+VKbGMGEvEKCQNHXIlKJK+rEidWoXyq7/bWxKt4dIM6ZUSRmeSb64eZMxewzJimqOgpSTj3hz/wV5BDvaz8MFhWsOC/ANL/PvDn6UoWxxcr6W3FAdMRymebB7i955/GLHsUHhBMPrUGqLVRi2vdv3Kt/D9Zbbg+foEraTN7dYq2U1qVFV1eCoYYD7M81svvJ/Ed1OgIXZAmeA+uMb/eNsXaCuLk/4Ytcjl+bUx5ldzxHUL77KJtcldp0yIEqBsyD+wxK8e+ityssWI6TMl2vz3I3/D4mCGz5Xv5Wvl46TTSQqRgtm5rZnrllzljZBdlV24Dn6fZHLfIkdySxyyVug3TJaCDO2FJIklibXSQq2V0e1X2+U3HNOveG2vFXHdbqKEQScriFIRGaONiYFCExETagMRSkS09RqUMAyEaaJNgSE1loiJETSVph652A0FlTq65b/xxXYjV7P3r5pLX5lsrNWrnMWq2YRWC7syQjVwKcXdKgexDbEtQO7thOUt42pARMrDHxIEY92AiPe4nY2AiFgrKspkPsxzvjWAMefizQmyF1rEpy/s3dJZbwWlkR1Y8DNkTJ926vr7JwZKcYqFMEe05FF4qYPQECUksSOZ35dh+kAfrdjhbGOAapBgbiWHnHXxqoLCmRi7EnG1hm5sS4KcQZQQrBxKU1MungzwhCAtbVKW5qBVZS41zZdT9xImu8FZYotKqG2rgBKmCYaB0Vekedco7YJB7a4O/2T0eUasMgAzkeLp2XH6n5IkVkPkWoW4HdzSPoptQQiEaSFch9o+m/B4g8N9JcbNEgqD5zqKp/wpPr94F4UXBbnzbZzLa92ExC36fGN0iM5YgfIhh+P9Z3jQu8DpYJgvV+/g65cPMlruoJvNDe13ryBdd6PHWLC/n8gzaOcNgmy3E6sMuxUIkssR7pKP9ENYWkX77W5BWdeh3m+TddqkZZtq1SN/HrzVaO8K661iU0BE7VCaxphB7sEl7u+/wg/nrg+ICHTI7y6/ly+9cAxr1WLgKUViuYM9W9q6+3iXo+OY4ssdLnn7eWlsgtFHK/xw6hQFGVMwrtfGDV90g8Q6EY4CDEEnmeN3Kj+KjMBdFRht6Ktp3EqM4cc4i01EOwDbQtsm7aEklSOSoD/m+Pgsh61lPBkRrpsSX+qkOR0M81fLd5GaFuQuhpgrdeIt+j62V4MyDGTCRfXnWLnXwh+Jeez2M/yT7MsorZmJJXNRhuhyir5vzKKbLeJKdc/1XdoVCIlwHYSXoDEB//T2J9nvLDNuhigET/lTfGb+Xi5eGOTIMzX0idNbmy0vJNFQjvKRBPX98Ej2DA84bR6vF/jbi7cRXUphldaId2tts9dBJD1ENkM4lGXlbpegAMFYh6GRMrGSNHyHKDSon/PInU3hVBXJTtitk5bLEGUT+HmDMadFUgboqk3+tI9ZaaMbt0j489vkakBE9UiGxYc17nCNf3eDgIiuHzPiaxcOM/YFA7fUxjp1hbhU2dVVH7YcFeM+e4nJ6RzVe/p54s4DHHQWwVqjsO7OlOuJMUYgMFaq6GYLVe1GMA+sDVF8IYdsRzC3iGr61zR/QK0LFiObQaZSiAGP8JDP8YkZPtL//Lqp1aCiIuoKnvL385Xlo5ydGWTyYoh3dgW9hSb8bRVQMp2CQo72UBJ/OMYbaTDsVGmpmPnY5k/LD3Kh0UdiSXTrXgWda/10erwlZMKFyVHCfIJOUTFmlxgyq1jrJ9CLfj/TC0XsFRPZCoi3wRwSZmz8AUGYj8gZLQwhUFoQx2LvJJNdbX5nW8hcFhybzkQfzVGHdk7Q2KdQmYhMsclwsobSkqrt0okN5kctysLCapp0UgNYrT5iRxI5gtaQINKSmbDQzfdrdJCtdrdH1jsUYdkYA31oz6U5JEmMVNm/KSDiKmvK52+ak1wIBlHzCdxS0BXufvudYdZ7JZ0Q0fSx6jHnS318I3mUMH2egpwn0JAzWgxbFTp9Mf6RQaxKgHF5qXsYiiJkzUe0O91As1cmiUuje7AaHsAfz1LdbzFYLHEgtcqAWUciCXTEamxRUQmeLE1x5uIwzryFVW2B395SBWP7BJQQqMkhKke7JTl+7N1P8RP5ZwF4KczyF6X7+eanj5O9GDN2towqVbpmvXfiDbcFiOEBZj5UoDWiuO/uczyauExSSFKyG/n0pStH6fuyS2IlgrXy1n++YVCbsOC+Knf2rTFhluGmBsFuDcK2u00Wi3lq9wzRzktWH4z44L0nyFk+Y3YZR4YoLboBJ5twJkNc0aEUp3ixPkY1dKl2EtQDh5xUVIMEX1q6HW9eIC4voHz/TSWK36rIQo76/WO0+gzaDzX47bs/xZBZ56B5/X3zLX+Yf/ulj5GaNpg42cE6eRndCbspJe9A4kYT4bfxLqdYfqqPz8zfx8u3DeGOf5WcbHHMXuaYvcxT79rP15JHMBdTjH/ZxpmpQK2BnltExfGrTe3SwEglEV6CxUf6qD7mM1go8UtTX+EeZ56cBEPY1JXie+2DTLf7OPntgxz5dB1Zr8DSCnHT31L3zDYKKEns2QR5SVBUPJQ+z3vcDmfDDqc7g1xp5smdj8mcWIZyFdUr9/L2WD/xa8/BH1S443XuyszRbziYdHX+UCuadZfBmQCr3O5W0thqpCBMCiYLZaZSq3gyAm7QKHKXI9abu2nPxS9K/H7B4HiZn+//Bu560IfSgrk4y2KYA8ASEbaIOWovctDStHXMZe8sdeVyoTPAhfYAlcjjXK2fUiuB6etuyZ138j0vBMJ1aBUN/AHB/v41Hk20cMS1osahjgl1zExYIHXZoHA6xL1cIS6V3zkmvRuhYrSKEU2fxIpGGyazI1lmBovElmTKbJGWJvdnLjE3keWcOUC76GDVPAy/m4pzo/5k3a7EiW6gSr/g+MQMh1PL3OPMM2l2n+VYa5pKstDJccUvkFgScOIs8Tbdy9smoIQUdHIWzREN/QE5o4VCMR3l+Xb9IJfWCgyVIyh3baQ93h7m6AjxYI7K4TSZQ2XeO3qO+7xLG6r4UtxhKU5AxcJeriBqTeKtDFJY72wqEgliF/rcBgNWHVfswQ1ECORAH+FoAX/Aob4POoUI3XL5d1c+QqWdYGa+gGiZGA2J2RQguomTWkJnICIz0GA0W+UXx7/C7fYaOemzz15hJixS6SQIY4NyUnRNWy0fVa+/43yuRn8/FLI0D+ZZe3fIyFiJDw+e3HSg6gqm/7t2gM8u3MOFmQHGz0R4lypQeoekKLwJdLVG8cUWmSs2a808/2HxwyQHmvzysS9yvzvDlL3MTw49z+nsMF80b2e+kiD3Ypr+Z4sY9QAuzXSjTNcxhgZY+pFxWoMC674yP9Z3giGzSm5doT0fRpwL+3muNckfn7qfeNVh5Eq8rVVhttUHFWQl4XCHwWKdtOxGK013+nm+PEZr1cMuNYlvQsn2WxYhusLpaJrqAcknJl/k72efpt/QSBIEOuJilGUmLGJXJCyuoPz2lkbRCcPYCM6IXRh06vRZdfZqfYS4L0Ntv4vfJ4kmffLZFtWqx8kXJnFWDfY/2cFZrCJLdeKV1W6jOMtCmCbhsUkqh3JcOJhj7eMpJsyACRNiHXLBvMSL7hiNyGHZA1XMIG2rm9f3ThJQQkAhi78/T/mQxcePP8l/Ufg2/YbCEMlrFSJ0yKfnjrP8tVEKS5rUi7NEM/O3dImst0pcqSK+exLHMBidnaJ5PkPlYJZvTxxk3FrjgFXmIbdOmD7Hv+j7BhVl83PFf0qpk8JbcUitJGGTgIoHspTf0+bBqZqqbTcAACAASURBVGl+pPgiH03NrhectYm15lzYz1ert/HM6jjucx7JeUXqYn1bI6631cQXeoJMvsVYukJShICkaDQYTVaYzeXwR5N4jf2Ipo9uttBxvF6qqOeHekPWk2I7xQSNMUkwEDNoVUlLtVE1YkVpPl+5l7O1AZw1AWG09TeTkAjXRScc4oRm1CkzYNawhCDWmqUgQ1jq5liwW2uirdvecRwaYx7VKUnsaVRgUKkkkcs2iRWJU9bYpTay1kI3GuggQAMiisA0kZHqtsjedMC/2jgv1JIhp0qgLF4ciyjdncetZEgBolzpHhzeCf4oIYmzCRrDJu0+zbBdJbveMuMqbR3R1powNkCz0XZcSLEVqTW3FipGa4XR9HHWXBIZg8enj7AcpHkkf44fSp7GQOMKyMkOE4Uylw6kCLImTnkUq5BF+AHab+P3JchkGkwlVxkyq9dVQw+JOemP8cT8FOXlNP0rmsRajGy2t7UqzPYJKNmNXPovDz3JlL3MiBkhcXkoMcP44Br7vTv5ow88gnf7MOkriszFJrIZIKdnt6zQ4C2L7IbvC89j7XabwvsWeDC7wkPeheuqRnzHn+SLf3s/2fMw9FID1Wrd0Pb8/SBsC9WXJ8q5hAMhH0qdoiAhK10CHfLU4jj93zXwVkKoNrbsc7cSmfRQB8YJ8y5zj0n+0Xu/wYvVEU596yCJxW5uh3d2GdHuoKu1bnHXTYL+qgakpehm7FtgrIctzkcBF6MsBpr3JV/GSGpue3SeZ++d5LvzkwR/MUzmYg5rdo1odu6WN18Jw6ByKMnqD4b0DdW4z7vIoHGtZUZETF3FlJSJplsYVVmgHRvhON1uBFctALf4Wr1ptEYtLGGVK/TNZkkuDTCTPcT/8v6DLL87w6hd5pHEeYqG5n/Y90VODY/ybG2Srx64HXu1j+Q8pOZjKgdMfnj8DH8/9xQFGWGsR1KGOqalYj518V7sv8oxWlZkXljqumda21vvcFtNfHFCc6c7w4hRx1s/1RekiWf5LCZmMIZ8WiKB0ZY4VRfLkpheArF+A14NOb+auX9dCPqNVP2rmf23eF6EkAJhmQjbopOF+/sucyixREFGmNgbVSOWoiypGcif8TEXK0RbLJwAkBLtGMSugeFGjBgGCbHuUEXjtxwKKxFOKYBdqiEI2yLMOQQFEznQ5qcyzyLRnG4dIrGqSMw1ic9feu21uxqoYkmULdCmQopu7k5J2Ux3+kkbPoNGg7RUPOad5R73Mo7xEH/Xfxy3YmNWEt3791ZWEYQAKeikBfmBOgfyqxSljyUS1/1aqEFpgSE0yqK7pp6DTKfQ7TZaCHSseqXNNqHabWi3u4ET7QA35VE+MsyZxiChZ3CXM0NS+hyxatxh19lnr/DixDCrXpomDjI0CAqafe4q44ZCCoNYaxSKuoqoKEm97LF/uoNdbqMXV25KV+jtC5IQAmXCiFGn37hWrPTq30ftJT5+9DmuTBa4cKTITDlNXHHInTqCW1bYNYVd6aBNSeQZaENgtmKMVoTsRBjlJmxul2GZxPkk2jIwSk0oVSCKiGuNW89kaBiIVAqd9ujkFI9kzjJilknLbkmjjaoRc3eRWoix5svoan1HHuY4kpitbvFfvVs3k0KOhR9w8UdjDgyucbozxIu1EZLzmsylFnKt9pppXNLzEOMjqEyCxfsThA/UGcvVqcRJvua3+J8ufpi554dRtsYY9EkkOvzAyDQfKTzHYW+Rpz6wysy7kuS+1cfQuqlFlSq3XoTfeqUTmXDxBwQ/s+8EB50lBo3rV9bEoGAYuCLkH4w/zd+6x1jzPU4fLyBaaRJzBqlZjVOLSb+4jC5X0b7f3aB7dN0kzRYiDBl8qsjLjaOcyMGfHz5OMdfgp8ZP8PO5E0yZJf7e5LPMDeW5uK+PuVqG2zJV3uVOI4VgNoK5KMm5zhCfmb+XpVqa7HMO7pUlRNPf2kCr12FbNShtaUZMQVZ6685PjUTiCMlhC365/2kUihhNqBXfbvfzKyM/weJqEnvRwlvwUBZ0cqAsjV0xcSo2ZkuTmnMwWtd8GrFn0Ri1iR1Bas4mAQg/QPhtdHBrCShhGGjPJU676HzIo+4yGekCBhHxRtWImct9HJlpEs/M7VgCtA4lZjNA+J1dW75K5ZLE99b50P6zJM2A88EglyoFsjMdjPNzqNcxYwgvQWsqT7to0LrH5zfv+RwAs50iX2/fxsL3hjn0JyVUwqJ2MEWQ9fjqo0f4wePnOOos8J/v+H9QCH7M/0WKJ/OYFR9Rb9yCAkoibAsch2Ag5r/KP0tGuhgied2vGUKSFQmyEn4he5lPZrtdtcM7Yto65t8sfIAvnboda8HGqeSx1bqA6wmoLire0GzsL1cZ/qqBMTrE2sMjtAY9vvDBO/iF/AtMWSb/PHeq+57BrrXjamv4WBvMRUn+rnmY76zuZ/6bYyQXNIWTTdTFyzc1qGfbBJTWGrMp+IZfZMIss99SeOL6vJiuNtXVqJRQjJoVDhVXuGJFrFpp6gkbZYJORwhT0cmbtGsGRkfQLrrIzqaumo4gKHZNAp2sjVfox2xrEotFjGaAXC4TLS5t13RvCtLzEF4Cchmah4u08wbpXAVLSBSahgpoasXz9QkuXhnAWbC6rS2uFjDduNAbtHjQG17pV//sTWijgY5YiDssxh6iaSBbHUTQ2dVVE5QSBMqk4idoxyaVSpKiH0MnvKFgFabZrTWZz1KfMPEHBIVcE1eEzId5vrR8Gwu1DIklgag1kR0bd9XBCExqswn+eOhBJlMlfrr4XfqNJomCT/lomsSaQ7rWBKVuWBx5ryITLmJihCjvQSbEEhLjRvfXJgwhMdb9qY6wcHTMbckFXhgaYUlmKR11SeaHSF70kOvRkLfKem0J6/lSuunjlmOUIVitJ5mNICsDCtLesGhd9QEqNCEhT7em+KsrxyitZCgsaryVGLPWJr7Jz/D2aVBKk74I//rER9lXLPHr+z7LXa+TtymRHLQ6/PLYF2hqi4ryqMcJpFC4IsQWMU3l0NYWoTaoxB5KX7vBpVDkjBYSxYVgkCt+gflmlnMzg1BzGflGltT/t7Z3b2AhEGPD+PvzNEYs1h4LGB5c5ZPjz+AIi5bu8GLoMR/mefz525n8HNilJiwsXy9UpIF0ndfuQxTH6Djuhk8bxvXVtuMY1Q7eUEgtxB1+v/QQp+uDpKYNmF9G+f6uLRIrwpiw4XG+1sfsSh617JJYkJgrS8SN5g39nTKdRmTTNI700fmRKh8YP8e4W6KtLb5ZPsziZyfJXwjxpteIl1cQhoFbqpCwLBLLw1SfG+eJfZMYH9V8OP88Hz14gud/doyXrgwz2R7GO2OgK1XiTe3k9zKyv8j8+/tpjmiO7z93fX+yN4klDD6ePslDR89xcWqAzxw4zkw9x/yXBhmrNrs153q1PF+FrtdJvrRMIp2gfCzPXx25i/3OMo+4c/StB6gYm/z3oVb84csP0v+pBFOVCPfSQjfKutG86W6C7RNQWuHUFKWFJNPA3FiOKXMVS0gsYWxEmm3GEzbHbOjG6TbX/2zmzUX3LXjTzKcTXAgH+DPzfmarWfxTBdK23TV17VGflEq6BHmTdp/g0Ngyj/Sf5053BuhWi6grl7U4hVkx8aZLiKaPCiOEtX4ykOuVo5NJhHnjDUJHcTdsWgqEZXUd25t+JpVCR2IjGlAIgZYSbYirFfppa4PpVpGZWh6rqbvRg7s0QALoziMwqLUdooaFU5VYdRDhDTa6qy03XAeV8QhykjsHFvhQ7gVqyqUSJ1loZchcjkieXERXaxvh6Fe7jFpArp4HkeNivchiKsd+Z4WJwTXC2KBWGMPNeMh2AKJ2SwQCaMfGH9CEwx0mvdKrnv+rybmvhSEEEsmw4TFmSqbMWayBmIu5fv7TwAfRnouIFRiNXvudV6CjCF2uIDshVr3ATLuAI0NCd25Dc3olnbJL5uVyN7F/ZXXHnt/tM/HFMZmzNSCDX0zzS8v/CCsf8OjUef7ZwNexiZBCY6A3SsXfSGi9HdJCMmL6uGKenxh4noV8jt+/5z0kVu/EXQuxT1wiLm99PbptRUiCQY/KQYk/GvFQ30UeTp5l0qwh8br2Y2LS0kdMtFh4rIgMNWZ7FKE0WnSrHnRSgsY+TZS6gdlKC4y6xGoIlAVhVqHsa9qDXTIYeEaRWA4wl6ro+SVEPkt9n4ffJynmKgCEWtKIHPyOhRsBu7wAsKw2yZ7K01ztg+EQ7941yosZUotDJAspjMUy0dx818Q6OYpKuSzfmaJyBKK+kPvsFhc6A/zR5QdYPtNPYlEyPl1GV2s3rBenmi3kMmTPWcw8PslvDI4zdnSJn5t4kjty83zmA0Os3p1n6LtZUl9qoDvhno9Y0wmbYDji4MQyR7xFJIJYK3zdoa1jfr9yD5+dvYtYvXoPSNodbsstUbCavDf9Eo+5IZ6wuN1epGg0+D8nfUrHiyTWciSebnfTAHpscDW/VAJ2BZ5eGaeWd3hf8jTDht74LqBr4ovZPffZNmpQGn3qHJkzJtlshuKpUTo5h8d/7Hbe9cg0aaO9UcNsylolK7sZeVshpFLSIQUMGorD1hVa+gLPHhvnhcZBkvMuY5czsMcElJACv8/APxAwOFjlA+mTPOBoJN7G77gyJCk73DE6z/MPjKPVtdOREIDU9BXr/O6Rz/KgW7vu+gaCGM03/CJ/1zjMoFXjR1MnGTGvXeNPagf5LfMjpC8lyNsSu1JDZZM0Rg38Ic1d6QoSSQdBM7TpBCZexK7P/teVKv3P5wkKDlc+Ar94+Gt8tf8oz1y6g8hNk1XA/AIimaS1P4ffZ7L6gyG/8MA3kELRih2m232snhjgwOdamNU2enqWuHnjVhqqXkfV68hmi8nWCFEuwflPDGBMKo6nLjP17hXKUZL/t/U+0k94wNX8tb2p+QPECYvCUJUfHTzFXc4MIFDojRDmP734LvRXCogbKD8raZg+MICdDTCOKt7jvoAnbW6zbSZVmztGF3jp2BTJeQvvdBJW1276/HY1WqODAKU0blmxtJhDaUFl0AXi9eC17vex29jeKL44RiuN4bcxyz5CaRJXPP7g0kPYRtzVoKTiYGaV25ILWCJGCoUlYg7YS0yZVVwB2U3OvLeCRCIFuCjytk+cjgk9E/0a5q1dzSsqc6RlB7mpGKslJBkRoIwGB1KrLA2mUfqacJFCI4B9mRKjZo3UpnYGV+3Pse768Qpmk5TR7uahaNajLDVpo0082qYmXbR0yNr7CfImrRFNVAwZSVQxhKClHBYqGaLV9S66uxwdRhi1AFsKjGqCF5tjtCIbfzQmdg20TJOTtxPkXEpHLYK8JtfXIGu0WI3SfHPlICuNJN6S6EbhNX3UmzAz6U4H2WhhAokFjz+Ze4BBr8aD2UsMWxWCgZjgzgmschvj0ixxdW+b+6To+oqlUIBBS3d4KeyW4qqsphheVsjo1fMLfEmYMQk7ktnJ/HU/M4TAMzvEriZ2eh2Kb8imZqadlCBdaDKSqpEUHTaC1F4hnGQqxJ/I4qw5yFp9x7TS7W1YqDXomLjRRF6awbRM9i0Xib6a3mgprKXgxX3DPDl2N1p0M8e1AfKuKj97+HuM2SXel7hMn5F4/c96HSSSw8klnh8dodzJo529V2UbKfAHBZ88+F2mnGVGjOtP046wOGiFhLrBePFbfCz39A0uoUiKiEnzxnZngKTo0GfWsUTEdJhjUYbU130rAL/xwGeJkXy1fBvPLY/i2SHvKyzQbzd4b/olJJILnQH0CxkGzynSF2q7OnoP6Jo/Ls1gJVzyhw7zmb57Sedb/MOHv81+Z4U/nnuAMxcGMdMhP3r4KY54i7giJCkDvl4+wurnx8hfiPCulNCX51Cbqx28DrrTIV5cRtg2Y4/b1C6NMTMlmfh7ZX4q+wwPv+tlnsgcxJrJMPWpEeS5sFuJ+hbxsVyOBP/77A9xca1I7lmb/BOXb+g/0pkU3ko3avXpsXHCsXhT9JlkwK2jCyGduoO2tndL24sIs9vbTKST1PfDvz7yNcatNcbNkKsCajOWkNw7OcPz7z9Ecj7N2GoearVXX/gmcHO+TRVfq5pbqSLOXfuREIJ85Qh2I4MyBNroJvguDia5ON6HFIq2e/lVl7zqUA2JUVojhcBAbNSP2mwqNIQgZbTJJdqUHAXGa2/Qu5nY0Rx0Fhk1K7iv0CglgoSwSQhISc3EDb9ZicIk1ppAR6j19NN406m8qRO0tUVbWxCBITSlKEUpSjJoVfmh5BkKUpIzmgy7VVwZctBZIm34jBh1wKEaezhlSC52kLXW7u9VuH5/iiDArSjMFZuWE/Nw6izvcevUh10+q+5hKFnjo/lnOGjVOB9muNAZYMVPkbkSkzq1hK7UXtOsd0PWTS86CDCvLJNr51FWnkroYaA5llqgMulxUo8Q5Vwsr3tIu1UEVEtZLNQzNEsJhkqKeHHphnMz2m2cYgqhbSrBq8sQOzJC2jHK1l01rcf1yG5rE+3aROmY+xPTFGSEIwxCHW/sA9AV+EprxrwKzw23aXdctLtzB/qdP25ojVgpkdLrjnxDoE1JczjLN/oOMl0s8r7ky4zBxkLORgGfb9zBXJDn6bUJZlbyeF7Ae0YvMu6WeHfyHA864ZYFXewKlCZ1BX71pY9wqLjCL499gWObntWrrTXaWlJVDjXlEmLQVjahvibMKrHHc40Jyh2P5VaateY1H5ZSoltlftXsarMmaAFGAEZbECU1/3G8TcLr0J9uMJqsMuxWGUpWGTQatLTJk4HBs7UJEmsKa6XVbaWyR8xSOo5JXawxYGaornn82dQD1Aov0Ihd9mfWaEY2//bCR2hHJkuLOaxFi8SyYORCCV2poX3/bX/21cCJzEWXv37iXh4fPczRwWUeK54lZ7d44mO34T58hMFnQtyvv4jqhHs2GvUqMYIgMiCUyNeZishlWTmepDWsOTY8vVHAtMebQ9g28UCWMOciUhHpdUffmdCkrly+2TjKM+UJcrbPe/On6TdrDFh1Hjt4jm+ZUwQjGZzSALrRvK49x81g5wUUEC8tw/LKxv+FaZEbu4fFvhTnA5OVyTRXQ8xDHTMTZfjM7L3Mr2VxXvQYOxHS6k/whUfvpG+ghtynuM9++ZY6TOk4JnM5ZOl7BZ6aTDM9WOCYda34aqAjZuIUldhjJiwyF+S7iaehR7gpX2yumePihUGMukFiSeItasRVAaJhZDbAvtxNaNaODYZE1Ls5ECKXoXHHIO2cy5V703AnJIyQQaPBQcvhycDgyeYhzpQGSK9GiKU19E2+ob8vtIZzl8nOJXDX9vHku/ZhCkXOanE0ucgz1QnmnhsmsSyYPB2SPHEFHXRQle+/4ebVwAlLa8a+sg+/L8W5H1f85uRf8mjyNMfff4Xpdh9fVg8w+R0HoTW6s7drTsZIoshAhAIRv/Y8VDZJ5Y6Ikf2rvLd4dqNvVI83h3Bs2n0J2kWTZLpGTkqaWnE6GGa2U+QvLtxD+FKGMKOo3e1yJL3EMW+OD6RPAvDS4B04CzmE0te157gZ7AoBBVz3oOk4xmzG2FWTZs6iqRw250B5MmAsVSFUkpU+l8awSVAUpPItxjNlBs3qdaesWGsCZdEKLUQk9+xDbdVCvP+/vTuJjePK7zj+fbX2wmaTzeYiriJlWZI3xbE1MTKIPTAQHzIzzgbkNMghyCG55pQAAXJIgNwCBDkEOWaAIAEyCDBOxnaMGXg8dsbjNbZlS6IoUdwXsdls9lLdtbz3ciiKoixrZhRbYlN4n2M3SXQXwPpX1fu/33/TRTsu3934deZLV/bf201yzDaHqccZtts5Gu0MUlrEkXNLE51quWSXHdwAMhVNtpKwP1tQa5yddrohT1jpPiAh0qyzIMByHPztEJF4uE2HVuTRSjzU3oJiVfaw3CnRCHyKkYK4e+ON7kTHCYQRbj0kWSzw4+Qknh/juwm1Wp7CmiBbUXjVKN28GH3F3zEM02BdfLaXC/zj1HP0uwFDbp0TmS1eGlFEv3ICt9ZBXF2+L4Gd98rBjfa3EQK7XIZSkeZ0gUy5zcN9W4y6afet1IpQJwQ65mqzjN7I4G8LRNilI10OwcG0k/qUS3tIMNGbriXtKpt3GyeYbw4QrPdQXBNEgc38+ACJsjjm7VLIxOSdMF16saz90O77qXsK1EFakVlvUrpoAw5bSQHY2c/xO+VG/PnYK9SOZfi38jO8OTNDuafFH0++xWlvnWE7wuLWpopK3EO1nsduWoik61dFbqckzqUlhjd6kQMFFpdOcqXw8P7bVqzxdzV2pPEDRT6QiERhhwmom99XJB1EECKSvRifz3Xn6HYHdeO1G2kTe+kSqraLcyHCzWTIT8xQPdHDmhvT0g4KxYX2GG+uzhCv5XF3dtOg3i5vMf88ncTolsS+sszJfxlD5j20cEF4DIYxTqUKYYRutdKx7V/xxm9Zb+JcWMDxfWYaY7z98dM0puE7v/M6L/Z+xCfPjPPjwZOIpSIPfTeBi80je8F1JzdOrMHXjrP5lEtnLObPHvkJz+cvMbw32DBQEYtJwqYs8MHccaZ+JPGqbXT1aG0fuZesQgHRX6R5uoz65g7fnrzE13suA3A+HOUH//sEmRWX0QuS4gerxCN9rOois8MFes91+MPieQbcFsoBHAtxCB2S3VmgABF08Ks53Gb6nDTWcr/5ISc8zrigiFnrnaU54jGeqXEus8S0k4EDxUmhiJHUkyxxx8ELAXm0Tpo3yFoN0Whg1XspOeMkuZuLUFYkcXfaEEY3B0AmCap9a9DpXZ3KPrdgreMIWYsQTguvOY0MHOodn5rMsatqrIV9NBsZnKZAdI7oGsmNztPdOtalENuy0lw8rUFpknu9YVbJNN5ICDzXZaDVj7YK7CZZ8iLhqcIC7eMuP9PTyN4MwnHTO7gjcKyVhljb+3dOllBYlgYblCMQvp9e8bsuwnMJyg6dyYjScJ1fzV7jUe/m/3WMpKoybCRFrJpDdrmG1WinUVwGkD7aUz0ZOn02T48s8/t979FnRXS0ZiMp4lYc8mua/EqbZHEFVyn8nRwyY9GM/b3UH4m2QNuH08LfnQVKa3SjibvpkF/3+P7KWWw0v5a7wtO+3G9+sLB4xF/HGlD0WQGlveN3o5liS4b8tDPGYlTmB58+xsBbHrmKhOrRzTfTUqLbHZzVKrZ/s0CJJH2dJEFFaTsy9+jxmlaa3vmAOJsn7h3gTxb+CJ2TZJY9SkuabFVC7XDaUr8yWt/WUaaVvn93K1qj6w1srekreHzv3XO8MfEQZ8trvFD6lI50uPrUw/QVnyB7bQc5N9/Vd1J2O6a6VuTV7KMUx9qc81cYsVs8O3GVueIgV+xjNMfPAqQB0RaEjwd88+RFZrJbTDkB0LP/967FFn+98G0Wt0sULwusrRq6Ez4wHY5fmmUTT49QfTRHfQbO5NcZtELeaM/wdv0EP12dpvSZpng1wFmtkmiF7slRf0jjHW/wVP8S9h1ikO6n7ixQgGo0EWFEbqCHuYUy/66fxJpUPO3P3fJzZ1yXU24FAIvM/utSazalxyvVJ5irDdL7kc/wy/Podjt99HQU7Z2AVBCkd0Z3eP+eUxJ7dpnhjQLadVA9WbAFVm0bdnYhTtKQ1SPusE92sl6Heh1fCIbfmiAYHuTib0r+dvQ1PCH5i8dniHo9hmUf7pXuHnYo2hH+Rh/z7hAX+kdRvcsM2xZ/UHqX68UCn5XHufjYCJbQ5O0I3054oe9TXshWcYWNK3pu+XtX40Eun5+gsGBRutRJ8+JMcdonLEFzPEP1rCI72uSxzDIl2+b9xjSvffYI/pJP6cMt5Ow8yd6AV1nwyZ6o89vT5/l6/nJXdEF3bYFCSohj7FZIZi3Phijxs94ZvpGbJWdFlKx0Q5mLjStsFIqmConRLCcuq0kfn7Qn+Z+FaeJqhpHrCt1uozvhkVsX+UKHfLWsowjaHURoYUuFti1EM0gfsUj5YBzjLqE7HTLbafFZu97Hy8enuBYO4ZfaBGRprnkMjo6kzSy79a48UYswIrehQbhcnBphqdzGIm14GhGKwE+TTSw0OTvEFZJRZwdfpKeoQEUoFFdiweV4iP/aPkt2wyK3qXB2wyPXjHM/KFegswk5PyZjxdikI2WILRCaeCCPNzmGzvpo36U2k2Okd5Vxr4rC4loiWQn7cUKN1TmcUSZdW6B0kqClxF7eZPI1j6jo8X79FH//GzHH/F2+UbjIkN1k2I4o21kCFTMb+2yrPP96/RneXZpErueY+KEku1zH2qohm639FG7jy1HtDiJKO6aEnV5pKan2ju/Rbn/uNqq2S+69efLZLNKd4K+C38MvtfnO6feY9rf4S/d3cYNxMtsJ/sfXkNvVw/7It1Eb1zn23xaqN8dcdoL/GD7LlFfhnL9KyU447V6jk0uHE964bu+zHGzhEuqYFRlTlRn+ZulbXPhkkuyGzfjrTZzlCqre6N5pzYdFWMR5Qc9AwFhhl7yIkGhCZSMiC+nD5tdy2GdztIc14ZAkN9jgT8fe5lxmiQ/DCV5vnOGt1WkGthLEdi0d3HmfdW2BAkBrVLOFs7CJ3ZMj+8QxLtcGaeR9TmSuA+CLXQpWTEMrVpN+NpIiV2pl5FqO3JpF/uIGyfxC96cZHDV7w9AAtOnsvad0kiAr22DZ9KyNECx5BGR5OLPO89k1/m6oTjBYRmjIuLcnLXQDFYbopRVENotf7WOxXcYVEjuzSo/wKR5YgD+YrB1rSUNFbMg8G0mR+coAPQt2eue0XCFZXTusr9T1lA1ZL6bHDbEPtkcJUJ6mUwahIB6PmBrdZrJQ5bS/zqCt6SiXxaBE0Mow3JHoMIL4/v+jd3eBYi8qvhUg4pjh9/upNoa52D/Ch6cnyfd2mOircaZ3g6vNQT6+MIVbs8mvCkZXJX6tg945ug0RhnELrche22GIfmp1j5eeeBLKcKq0Xr2wZwAABCtJREFUxQfPZWgu5Om9UIRunBytdToIM47Jr2leufQIpf4p7GnN45llppw6024PTdVhNraoqSwL0SBL0QBzzSHenj2BVXMozglKFzvpY73GEV1LPiQuNr9VOo/9lCaUDsHeHsaZngons5sMOnVG7BAbi824yNWdMmrHw2420+UR84jvCyi5vxnRfmOXobds7LERKs+O0S5nuHysyOyxYcSmz8zLMf7iFtTq6WRNpZFHoP3WMH4pWiOvLOAvrjDYPMMHz01Q8lo8WVzixfJH/FPpWZLv93VB79UdaI2KYgorEe3zWXaGfH5UOE21L89zPZeYdBIaKuG99ilWohIf7kxwrTJAuJFj6lWVtpJv1dKGCClvyZA0fjFbCF7Mb/Kt/PotM59sxIFpET6xlmxEvWxXe3B3LKxmiOzcPtfsfuj+AnXQ3mMl3WqnCQjSRiiLTpjBrwrcagcaLXS705ULxYbxpWmVjgdpdIgX+nlVnqGQ71DKtVlYGOJ00Or6x9luPSK76WHFFh+XJrjSV+ad/uP8Z+8GtTjH+e1jtDoeQTWHs+OQrwi8agerHhzalfyDwhYCG5sbD4IPduqFOmZTJtSUx/mdUZxVn+x1gQi/XIzXl3G0CtQetbND7j3Iey54LtpzEVGMqlRRHdPRYzzA9jYSc22Vk/8MKuei3RzayXOqGcDV5cP+hD+fkliziwyuF8B1UK/nUZ5N7A3xkXcMoTR9oaRfKqyohYgSiGKoVJHtzj3b22fAlkz4Xv1JLreGWXtnlOmXmtiNDup65dA+05EsUDpJkFtbv/gHDeMBpRoN+Gw2zUwErL2ctF9mUOJhu7G/6yALbtt10+13gt1OaJBKkNwh81BqjeTmuI2q8rjcGmZud5DsdYE9v5ZGnx1iOseRLFCGYbDXyp+eXLp4j65xGLQis6NZX+rn09hleyQH3FxHqsqQd8IRrie9zLWHWQxKXN0ZIPigTKYC5U876KCdhicf4p5GU6AM4ygzjQLGF9BK41djcks+LZGn9lieWwqUsvlh7VHmmwPMrQ+hV7NkrwuOv1KFxVV0FN0MjT5EpkAZhmE8gNx6RG7TQ1sO/7DwPG8OrOy/t9Yu8tHyOHHTw91yyG0JMhWN1Wghu2gd3xQowzCMB42S2JcWGVwrUM76JD/p47w/sP+2JRUzrRiRtBBhjOhEEMXIyjb6Xif23wVToAzDMB5AB5tRxNztJ3vNXY7fOQSHH1drGIZhGF/AFCjDMAyjK5kCZRiGYXQlcTcx9UKILWDx3n2cI2tKaz14t79kjucdmeP51TLH86v1/zqeYI7pz/GFx/SuCpRhGIZh3C/mEZ9hGIbRlUyBMgzDMLqSKVCGYRhGVzIFyjAMw+hKpkAZhmEYXckUKMMwDKMrmQJlGIZhdCVToAzDMIyuZAqUYRiG0ZX+D3sMGbXfqA2hAAAAAElFTkSuQmCC\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "plot_example(X_pred[:5], y_pred[error_mask][:5]);"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gzUcw21Fq-1b"
      },
      "source": [
        "If tensorboard was enabled, here is how the metrics could look like:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uOh1oAq7q-1c"
      },
      "source": [
        "![tensorboard scalars](https://github.com/sawradip/skorch/blob/master/assets/tensorboard_scalars.png?raw=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YXQnnf78q-1c"
      },
      "source": [
        "# Convolutional Network\n",
        "\n",
        "Next we want to turn it up a notch and use a convolutional neural network which is far better\n",
        "suited for images than simple densely connected layers.\n",
        "\n",
        "PyTorch expects a 4 dimensional tensor as input for its 2D convolution layer. The dimensions represent:\n",
        "\n",
        "* Batch size\n",
        "* Number of channels\n",
        "* Height\n",
        "* Width\n",
        "\n",
        "MNIST data only has one channel since there is no color information. As stated above, each MNIST vector represents a 28x28 pixel image. Hence, the resulting shape for the input tensor needs to be `(x, 1, 28, 28)` where `x` is the batch size and automatically provided by the data loader.\n",
        "\n",
        "Luckily, our data is already formated that way:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZKiPtuCsq-1d",
        "outputId": "1dc788f0-9414-450a-b2c5-dae079e6bb34"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.Size([1, 28, 28])"
            ]
          },
          "metadata": {},
          "execution_count": 25
        }
      ],
      "source": [
        "X_example[0].shape"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KyGRbkU7q-1d"
      },
      "source": [
        "Now let us define the convolutional neural network module using PyTorch:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {
        "id": "KyDI-ZoPq-1e"
      },
      "outputs": [],
      "source": [
        "class Cnn(nn.Module):\n",
        "    def __init__(self, dropout=0.5):\n",
        "        super(Cnn, self).__init__()\n",
        "        self.conv1 = nn.Conv2d(1, 32, kernel_size=3)\n",
        "        self.conv2 = nn.Conv2d(32, 64, kernel_size=3)\n",
        "        self.conv2_drop = nn.Dropout2d(p=dropout)\n",
        "        self.fc1 = nn.Linear(1600, 100) # 1600 = number channels * width * height\n",
        "        self.fc2 = nn.Linear(100, 10)\n",
        "        self.fc1_drop = nn.Dropout(p=dropout)\n",
        "\n",
        "    def forward(self, x):\n",
        "        x = torch.relu(F.max_pool2d(self.conv1(x), 2))\n",
        "        x = torch.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))\n",
        "        \n",
        "        # flatten over channel, height and width = 1600\n",
        "        x = x.view(-1, x.size(1) * x.size(2) * x.size(3))\n",
        "        \n",
        "        x = torch.relu(self.fc1_drop(self.fc1(x)))\n",
        "        x = torch.softmax(self.fc2(x), dim=-1)\n",
        "        return x"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s6aWg-11q-1e"
      },
      "source": [
        "We also want to extend tensorboard logging by two more features:\n",
        "\n",
        "1. Add the predictions for the misclassified images to tensorboard.\n",
        "    \n",
        "    To do this, we subclass the `TensorBoard` callback and call `self.writer.add_figure` with our produced images. When subclassing, don't forget to call `super()` or the other logged metrics won't show.\n",
        "\n",
        "\n",
        "2. Add a graph of the module\n",
        "    \n",
        "    To do this, we use the summary writer's ability to add a traced graph of our module to tensorboard by calling `add_graph`. We also make sure to only call this on the very first batch by inspecting the `self.first_batch_` attribute on `TensorBoard`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {
        "id": "nEjtVJXgq-1f"
      },
      "outputs": [],
      "source": [
        "callbacks = []\n",
        "if USE_TENSORBOARD:\n",
        "    from torch.utils.tensorboard import SummaryWriter\n",
        "    from skorch.callbacks import TensorBoard\n",
        "    writer = SummaryWriter()\n",
        "\n",
        "    class MyTensorBoard(TensorBoard):\n",
        "        def __init__(self, *args, X, **kwargs):\n",
        "            self.X = X\n",
        "            super().__init__(*args, **kwargs)\n",
        "\n",
        "        def add_graph(self, module, X):\n",
        "            \"\"\"\"Add a graph to tensorboard\n",
        "\n",
        "            This requires to run the module with a sample from the\n",
        "            dataset.\n",
        "\n",
        "            \"\"\"\n",
        "            self.writer.add_graph(module, X.to(DEVICE))\n",
        "\n",
        "        def on_batch_begin(self, net, batch, **kwargs):\n",
        "            X, y = batch\n",
        "            if self.first_batch_:\n",
        "                # only add graph on very first batch\n",
        "                self.add_graph(net.module_, X)\n",
        "                \n",
        "        def add_figure(self, net):\n",
        "            # show how difficult images were classified\n",
        "            epoch = net.history[-1, 'epoch']\n",
        "            y_pred = net.predict(self.X)\n",
        "            fig = plot_example(self.X, y_pred)\n",
        "            self.writer.add_figure('difficult images', fig, global_step=epoch)\n",
        "\n",
        "        def on_epoch_end(self, net, **kwargs):\n",
        "            self.add_figure(net)\n",
        "            super().on_epoch_end(net, **kwargs)  # call super last\n",
        "\n",
        "    X_difficult = torch.stack(list(mnist_test_sliceable[error_mask][:15]))\n",
        "    callbacks.append(MyTensorBoard(writer, X=X_difficult))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v9ISDtlLq-1f"
      },
      "source": [
        "As before we can wrap skorch's `NeuralNetClassifier` around our module and start training it like every other sklearn model using `.fit`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {
        "id": "xpzP_ct6q-1g"
      },
      "outputs": [],
      "source": [
        "torch.manual_seed(0)\n",
        "\n",
        "cnn = NeuralNetClassifier(\n",
        "    Cnn,\n",
        "    max_epochs=10,\n",
        "    lr=0.0002,\n",
        "    optimizer=torch.optim.Adam,\n",
        "    device=DEVICE,\n",
        "    iterator_train__num_workers=2,\n",
        "    iterator_valid__num_workers=2,\n",
        "    callbacks=callbacks,\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VeT1tzEHq-1g",
        "outputId": "ad4806ae-00d6-4fbd-fa33-f33251ea5bd0"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "  epoch    train_loss    valid_acc    valid_loss     dur\n",
            "-------  ------------  -----------  ------------  ------\n",
            "      1        \u001b[36m0.9490\u001b[0m       \u001b[32m0.9257\u001b[0m        \u001b[35m0.2535\u001b[0m  9.3405\n",
            "      2        \u001b[36m0.3174\u001b[0m       \u001b[32m0.9561\u001b[0m        \u001b[35m0.1501\u001b[0m  7.4618\n",
            "      3        \u001b[36m0.2211\u001b[0m       \u001b[32m0.9656\u001b[0m        \u001b[35m0.1177\u001b[0m  7.4806\n",
            "      4        \u001b[36m0.1819\u001b[0m       \u001b[32m0.9712\u001b[0m        \u001b[35m0.1002\u001b[0m  7.4573\n",
            "      5        \u001b[36m0.1578\u001b[0m       \u001b[32m0.9735\u001b[0m        \u001b[35m0.0880\u001b[0m  8.0541\n",
            "      6        \u001b[36m0.1435\u001b[0m       \u001b[32m0.9768\u001b[0m        \u001b[35m0.0775\u001b[0m  8.4212\n",
            "      7        \u001b[36m0.1316\u001b[0m       \u001b[32m0.9781\u001b[0m        \u001b[35m0.0728\u001b[0m  7.4816\n",
            "      8        \u001b[36m0.1197\u001b[0m       \u001b[32m0.9795\u001b[0m        \u001b[35m0.0690\u001b[0m  9.6584\n",
            "      9        \u001b[36m0.1100\u001b[0m       \u001b[32m0.9808\u001b[0m        \u001b[35m0.0651\u001b[0m  7.4606\n",
            "     10        \u001b[36m0.1051\u001b[0m       \u001b[32m0.9822\u001b[0m        \u001b[35m0.0620\u001b[0m  7.4750\n"
          ]
        }
      ],
      "source": [
        "cnn.fit(mnist_train, y=y_train);"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {
        "id": "ENbErtU7q-1h"
      },
      "outputs": [],
      "source": [
        "y_pred_cnn = cnn.predict(mnist_test)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wAWCV2ycq-1i",
        "outputId": "f75bafc9-2a77-4adc-a848-ebe298fec6b0"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.9848"
            ]
          },
          "metadata": {},
          "execution_count": 31
        }
      ],
      "source": [
        "accuracy_score(y_test, y_pred_cnn)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sWsPxuspq-1j"
      },
      "source": [
        "An accuracy of >98% should suffice for this example!\n",
        "\n",
        "Let's see how we fare on the examples that went wrong before:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Hgj840CSq-1j",
        "outputId": "273eda81-5330-4f11-b342-2d6d3486defd"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.7132701421800948"
            ]
          },
          "metadata": {},
          "execution_count": 32
        }
      ],
      "source": [
        "accuracy_score(y_test[error_mask], y_pred_cnn[error_mask])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_AGqI1qtq-1k"
      },
      "source": [
        "Great success! The majority of the previously misclassified images are now correctly identified."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wFxdFxLpq-1l"
      },
      "source": [
        "On tensorboard, in the \"IMAGES\" section, we can see how well the CNN classified the difficult images, and how that changed over the epochs:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fLGQyk8Yq-1l"
      },
      "source": [
        "<img src=\"https://github.com/sawradip/skorch/blob/master/assets/tensorboard_digits.png?raw=1\" alt=\"tensorboard digits\" style=\"width: 500px;\"/>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bapzRW6mq-1m"
      },
      "source": [
        "In the \"GRAPHS\" section, we can see the graph of our module."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-Kte4hXwq-1n"
      },
      "source": [
        "<img src=\"https://github.com/sawradip/skorch/blob/master/assets/tensorboard_graph.png?raw=1\" alt=\"tensorboard module graph\" style=\"width: 500px;\"/>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Av8L8UiKq-1o"
      },
      "source": [
        "# Grid searching parameter configurations\n",
        "\n",
        "Finally we want to show an example of how to use sklearn grid search when using torch `Dataset` instances.\n",
        "\n",
        "When doing k-fold validation grid search we have the same problem as before that sklearn is only able to do (stratified) splits when the data is sliceable. While skorch knows how to deal with PyTorch `Dataset` objects and only needs `y` to be known beforehand, sklearn doesn't know how to deal with `Dataset`s and needs a wrapper that makes them sliceable.\n",
        "\n",
        "Fortunately, we already know that skorch provides such a helper: `SliceDataset`.\n",
        "\n",
        "What is left to do is to define our parameter search space and run the grid search with a sliceable instance of `mnist_train`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 33,
      "metadata": {
        "id": "eBgNhs3Dq-1o"
      },
      "outputs": [],
      "source": [
        "from sklearn.model_selection import GridSearchCV"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 34,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "k0a5yr7cq-1p",
        "outputId": "53c240b7-79b2-4cf1-981d-411985f50029"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<class 'skorch.classifier.NeuralNetClassifier'>[initialized](\n",
              "  module_=Cnn(\n",
              "    (conv1): Conv2d(1, 32, kernel_size=(3, 3), stride=(1, 1))\n",
              "    (conv2): Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1))\n",
              "    (conv2_drop): Dropout2d(p=0.5, inplace=False)\n",
              "    (fc1): Linear(in_features=1600, out_features=100, bias=True)\n",
              "    (fc2): Linear(in_features=100, out_features=10, bias=True)\n",
              "    (fc1_drop): Dropout(p=0.5, inplace=False)\n",
              "  ),\n",
              ")"
            ]
          },
          "metadata": {},
          "execution_count": 34
        }
      ],
      "source": [
        "cnn.set_params(max_epochs=2, verbose=False, train_split=False, callbacks=[])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 35,
      "metadata": {
        "id": "8mopSVXPq-1q"
      },
      "outputs": [],
      "source": [
        "params = {\n",
        "    'module__dropout': [0, 0.5, 0.8],\n",
        "}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hvrV6wjdq-1q"
      },
      "source": [
        "The parameter we are interested in here is the dropout rate. We want to see which of the values (no dropout, 50%, 80%) is the best choice for our network.\n",
        "\n",
        "Additionally:\n",
        "\n",
        "- We use only two epochs (`max_epochs=2`) for each `.fit` (only to reduce execution time, normally we wouldn't change this and possibly add an `EarlyStopping` callback).\n",
        "- Disable the network print output (`verbose=False`)\n",
        "- Disable the internal train/validation split (`train_split=False`) since the grid search will do k-fold validation anyway\n",
        "- Turn off tensorboard logging (`callbacks=[]`)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 36,
      "metadata": {
        "id": "LuZ0Ug68q-1r"
      },
      "outputs": [],
      "source": [
        "cnn.initialize();"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 37,
      "metadata": {
        "id": "XgLmSp3sq-1r"
      },
      "outputs": [],
      "source": [
        "gs = GridSearchCV(cnn, param_grid=params, scoring='accuracy', verbose=1, cv=3)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 38,
      "metadata": {
        "id": "yBAcSDdpq-1r"
      },
      "outputs": [],
      "source": [
        "mnist_train_sliceable = SliceDataset(mnist_train)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 39,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "i9hcSRgkq-1s",
        "outputId": "63b1c573-49d4-423a-e465-3c0d2715f42c",
        "scrolled": false
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Fitting 3 folds for each of 3 candidates, totalling 9 fits\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "GridSearchCV(cv=3,\n",
              "             estimator=<class 'skorch.classifier.NeuralNetClassifier'>[initialized](\n",
              "  module_=Cnn(\n",
              "    (conv1): Conv2d(1, 32, kernel_size=(3, 3), stride=(1, 1))\n",
              "    (conv2): Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1))\n",
              "    (conv2_drop): Dropout2d(p=0.5, inplace=False)\n",
              "    (fc1): Linear(in_features=1600, out_features=100, bias=True)\n",
              "    (fc2): Linear(in_features=100, out_features=10, bias=True)\n",
              "    (fc1_drop): Dropout(p=0.5, inplace=False)\n",
              "  ),\n",
              "),\n",
              "             param_grid={'module__dropout': [0, 0.5, 0.8]}, scoring='accuracy',\n",
              "             verbose=1)"
            ]
          },
          "metadata": {},
          "execution_count": 39
        }
      ],
      "source": [
        "gs.fit(mnist_train_sliceable, y_train)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZUI3B1t-q-1s"
      },
      "source": [
        "After running the grid search we now know the best configuration in our search space:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 40,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YlKmRJoYq-1t",
        "outputId": "765c7546-788f-4626-8a91-63f399e2d51b"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'module__dropout': 0}"
            ]
          },
          "metadata": {},
          "execution_count": 40
        }
      ],
      "source": [
        "gs.best_params_"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": []
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "base",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.13 (default, Mar 28 2022, 08:03:21) [MSC v.1916 64 bit (AMD64)]"
    },
    "vscode": {
      "interpreter": {
        "hash": "bd97b8bffa4d3737e84826bc3d37be3046061822757ce35137ab82ad4c5a2016"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}