labs/lab4/lab4.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "lYUHk6sxO2pO"
   },
   "source": [
    "# DSCI 572 Lab 4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "AoVG1Z3FO2pQ"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import os\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from scipy.signal import convolve2d\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To install scikit-image, use\n",
    "\n",
    "```\n",
    "conda install -c conda-forge scikit-image\n",
    "```\n",
    "\n",
    "or \n",
    "\n",
    "```\n",
    "pip install scikit-image\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from skimage.color import rgb2gray\n",
    "from skimage.transform import resize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "G-ePax-DO2pS"
   },
   "outputs": [],
   "source": [
    "plt.rcParams['font.size'] = 16"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "ODE7I44yO2pV",
    "outputId": "28ddbdce-c461-46a2-af6e-329a9e2479a9"
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "from tensorflow.keras.datasets import mnist\n",
    "from tensorflow.keras.layers import Dense, Dropout, Flatten, Activation, Conv2D, MaxPooling2D, GlobalAveragePooling2D\n",
    "from tensorflow.keras.models import Sequential, Model, load_model\n",
    "from tensorflow.keras.optimizers import Adam, SGD\n",
    "from tensorflow.keras.preprocessing.image import img_to_array, load_img\n",
    "from tensorflow.keras import utils\n",
    "from tensorflow.keras.applications.inception_v3 import InceptionV3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "WyZOj0pOO2pa"
   },
   "source": [
    "## Instructions\n",
    "rubric={mechanics:20}\n",
    "\n",
    "Follow the [general lab instructions](https://ubc-mds.github.io/resources_pages/general_lab_instructions/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "J6QQPeGXO2pc"
   },
   "source": [
    "## Exercise 1: convolutions\n",
    "\n",
    "For each of the filters given below, convolve the given image (or a different image of your choice) with the given filter and discuss why the results look the way they do. \n",
    "\n",
    "You can perform 2D convolutions using [`scipy.signal.convolve2d`](https://docs.scipy.org/doc/scipy-0.18.1/reference/generated/scipy.signal.convolve2d.html).\n",
    "\n",
    "The suggested image size is around 100x100 pixels; if the image is too big, it will be hard to see the changes by eye using the very small filters given below. If you want to make an image smaller, try [scipy.misc.imresize](https://docs.scipy.org/doc/scipy-0.18.1/reference/generated/scipy.misc.imresize.html). This will be a lot faster than seam carving :)\n",
    "\n",
    "Note: depending on your versions of various packages, you might get warnings when you run the code. It's OK."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Y-wnQa5UO2pd"
   },
   "outputs": [],
   "source": [
    "def preprocess_image(filename):\n",
    "    img = plt.imread(filename) # read in the image\n",
    "    img = resize(img, (100,100), mode='reflect') # resize it if you want\n",
    "    return rgb2gray(img) # make it grayscale\n",
    "\n",
    "def show_conv(img, filt):\n",
    "\n",
    "    plt.figure(figsize=(8,16))\n",
    "    plt.subplot(1,2,1)\n",
    "    \n",
    "    plt.imshow(img, cmap='gray')\n",
    "    plt.xticks(())\n",
    "    plt.yticks(())\n",
    "    plt.title(\"original\")\n",
    "    I_filt = convolve2d(img,filt, boundary='symm', mode='same')\n",
    "\n",
    "    I_filt = np.maximum(0, I_filt) # set negative values to 0, for visualization purposes\n",
    "    I_filt = np.minimum(1, I_filt) # set values greater than 1 to 1, for visualization purposes\n",
    "    plt.subplot(1,2,2)\n",
    "    if np.sum(filt) == 0: # a trick to make the images easier to see, not part of the \"math\"\n",
    "        plt.imshow(I_filt/np.max(I_filt), cmap='gray')\n",
    "    else:\n",
    "        plt.imshow(I_filt, cmap='gray')\n",
    "    plt.xticks(())\n",
    "    plt.yticks(())\n",
    "    plt.title(\"filtered\")\n",
    "\n",
    "    return I_filt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "jJzVyzf8O2pg"
   },
   "outputs": [],
   "source": [
    "img = preprocess_image(\"milad_cropped.png\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "r_uUh2lbO2pj"
   },
   "source": [
    "**Example** (you don't need to do this one)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ewjVBg4rO2pl",
    "outputId": "84f39f31-1faf-4dac-c835-8d0c5018ec2f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1, 10)\n",
      "[[0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1]]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdAAAADxCAYAAAB72uYnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO29e6w213ndt4aSJYsURX6836/iRTeboqzKsitYUhy4ady6SaukadLGCNIIagHXbZMadgPERRKnRVKgQdoUbdFUdVHIQFQ0rmA4+cOS0lRy1EpWRFIiJVIkRfLjnSKpuyWKb/94z+Ls8zuz3j0z53zkd8hnAR/mm3nnsmdm7znP2s96nmfYbDYqFAqFQqGwDGe81A0oFAqFQuE4ov6AFgqFQqGwAvUHtFAoFAqFFag/oIVCoVAorED9AS0UCoVCYQXqD2ihUCgUCitQf0BPAwzD8OFhGDbDMFxziHO8d+8cv35kDetf85q9a374xbpmoXAqMQzDa4dh+I1hGL46DMMP9vr3L+8tf7HZ72XV9/fu5ZMvdTuOG179UjegUCgUTiP8FUm/KukTkj4i6TlJPzr34GEY7pekzWZzzSloW+E0Q/0BPT3wq5L+S0knD3GO/1fSmyQ9eSQtKhRemfhjkr4l6ec2m80PJGkYhnMk/SNJj7yUDSucfqg/oKcBNpvNIzrk4NxsNt+RdNfRtKhQeMXiUklP+Y+nJG02m2clPfvSNalwuqJ8oCsxDMOPDMPwl4dhuH0Yhu8Ow/D0MAz/eBiG90zsax/n9cMw/MowDF8ZhuH79lcmH+gwDBcNw/C/DMPw5DAM3x6G4VPDMLxvGIZf39v/vc2+kz5Q+zaGYbh4GIbf3DvXd/a23TrR1vfvXfMre9f85jAMnx6G4U8fyYMrFE5DeExJulbS1XvjxmPnF+kDnTj+mr3jr8bx9J2eMQzDvz8Mw2eGYfjW3r9PD8PwJyfOufO7sbfPG4Zh+BvDMNw1DMP3hmF4ahiGfzQMw4+Hdn5gGIbP7+17chiG/3oYhtetfnCvcBQDXYFhGAZJH5X0r0v6kqS/J+mEpD8t6RPDMPyZzWbzDycO/W8lvUPS70j6bUn37rjG2ZL+b0k3Sfq4pM9IeqOkf6ytf2YJzpX0KUlPSfpNbQf5n5T0e8MwvGmz2Tza7PufSbpu73onJZ23d5+/NQzDpZvN5r9ZeO1C4Tjgk3vLX95bup/fP/P4ZyT9FxPHS9K/kF74bnxE0p/S9rvxv+79/scl/R/DMPzyZrP5uxPnnvxuDMNwgbbfiDdp+034HUnnS/o3Jf3RYRh+drPZ/L5PMgzDX5D0P0t6WtI/kPRdbb8DN828xwKx2Wzq38J/kv68pI2kfyLp1c32N0n6traD6exm+4f39r9P0mUT5/Pv1zTb/ubetr+Nff/c3vaNpPc229+7t+3Xsb/3/buShmb7X9vb/qvY/9qJ9p0l6QvaTmOd2Wy/Zu8cH36p30n9q39H8U/bP5j3Y9sv7vXzX2y2Tfb9qeOb3z64d8x/J+lVzfaztDVY/7D9Psz4bnxk7/c/g+1v3BurtzfbzpH0jb3t1zbbz5b0xb3zfPKlfv7H7V9N4a7Dn99b/pXNZvOcN242mzsl/U/adtZ/Y+K4v7PZbB6eeY0/K+k7kn4D2/93ba3XJfi2pF/b7I2YPXx4b/kT7Y6bzeY+HrzZbL6trbX8Bkn/0sJrFwqFLf5Dbdnff7zZbH7ojXvj669Leo22jJA48N3YY59/StLvbDabj7S/bTabe7T9Dr11GIa37m3+BW3/WP4P7RjfbDbf1NZYL6xATeGuwy2Snt5sNrdN/PZJSf/R3j7/G3777JyT76n+rpb0uc1m83T722az2QzD8M8lvXlBe+/eG6QtrPg9F9d+g7bTuL+g7VTumTju0gXXLRQKkoZhOFPSWyU9IOnXtrO5+3Dh3vLmicOnvhvv1FbD8voQ+/2m5nx3SLJP9J9N7Pv/xIYXdqL+gK7DGyTdHX57tNmHeHzm+c/eWz4Rfp97HuOAgnCz2Ty3N4hf5W3DMLxG0j/V9o//57RlqV+X9MO9bb8g6bULr10oFLYaiUFbw/iv7djvrIltU+P9vL3lz+z9653vnB3nemzH8YUdqD+g6/ANSReH3y5u9iHmVi//5t7ywvD7RTPPsxS/oO0fyv9xs9l8sP1hGIZf2fu9UCgsh78Hn9psNv/ywmOnvhs+39/cbDZ/dcY5bERPfTvSt6zQQflA1+FfSDrR+Bda/EyzzypstnFnX5P0pmEYOMU6SPrJtefu4Pq95ccmfvvpU3TNQuHlhB+qmdUx9nyNd2nrl3z9EVzn/9P2D+vcb8EX9pYHwuwkLf2DXthD/QFdh9/cW/6tYRjaKdAbJf0lba293z7kNT6irf/xP8f2f0fL/J9L8MDect8fy70YtX/tFF2zUHg54euSLhiGYcrV8fe0nUr9+8MwHEgPOAzDW4ZhmDW7tNmGnn1U0h8ZhuFDE+c6YxiGdmr3/9J2ZusvDcNwbbPf63XwG1OYiZrCXYfflPRvSfp5SZ8fhuF3NcaBvk7Sn91sNlNTuEvwt7SN5/rLwzC8XdtUfW/UNibzn0j6OUnPH/IaxMe0/SP6K8MwvEVbi/ktkv4VSf+npD9xxNcrFF5u+IS2yvbfHobhU5J+oK1S9nZJ/72kn5L070p67zAMH9dWM3GppB/T1n3ybs3XOHxIW5HQ3x+G4S9q+434lqSr9s5zkfby+G42m2eGYfhlbeNAPzcMw29pjAP9ok6dUf6yRv0BXYHNZvP8MAx/QtJ/Iunf01Z1+z1J/1zSb2w2m396BNf4xrDNavRfacv+3q3ttPAf07bT/5xGX+mRYLPZfHMYhvdL+jvaTuu8b++a/6q2g7z+gBYKu/E3tE1m8Mcl/VFtZ/ke1TYmcyPpz+0Z3H9RW03BmdqKeO6U9B9Iun3uhTabzVPDMLxb2+/PB7T9Fm20TQv6KUn/EPv/g2EYviXp1yT9BW0Tq/yWpL+qbchcYSGG/aGBheOAYRj+mbaW7DmbzeZbL3V7CoVC4ZWI8oGexhiG4UDM5TAM/7a27PDj9cezUCgUXjoUAz2NMQzD7dqmBfyCpO9r6yf5I9r6Od6z2WxWK30LhUKhcDjUH9DTGMMw/Kfaqm6vk/R6bX0Wn5T01zebzRdfwqYVCoXCKx71B7RQKBQKhRVYpMI966yzNidOnHhhfe4f34m8j6ctfE/PP7+NEPne974nSfrBD36wb7/vf//7+/Y744wz9i19Hq/7GXjJ/dP1eX6uT4HH+ppef9WrXrXvWl7+8Ic/3Pc72+7jvZ+38zxs44/+6Dbk7ayztlnFXvOa1+w7njiO/YV44IEHntxsNimT1EuOM888c3Puued2n3VvjJ9O74r9nv3aS+7H7ezPBscw733ts5h6xr3nntp4mGu24PfJS38b5i75DTmO2DWWF/0BPXHihH7pl37pQAf0kpj78NZ0RP6BWgp+7A0Ppm9/e5t7/a677pIkPfroo/v2O3lym4v9m9/cRpL4j8PrXve6fef1H48f+ZEf2bc888wz9y05mL/73e/ua4fP4z/ovl7bft/TH/7hH0qSvvOd7+y7ptt67rnn7jvW53zmmWckSeedd96+e3nta1+773zPPrvNCuZB8txzz+27rtv++tdvE67cdNO23OC73vUuSdJll10mafxDmoyM3h/YXR+BOfvsQjq+t2588IMf/NqqC79IOPfcc/XBD37wyP+ALh3rRwH3Yxu17qfuh9/61lZr5/7/jW98Y99+HmNe93jwWDTcTz2e3H89DnxvXjfSPXvccOxPXZt/MG3Q848+xw6NaBoLvI7b7u+Nv08ey294wzbF99lnn71v3d8UL88555x9x7/61a+ebJ9xKvrF0rGf9v/Qhz4Ux/KqOND0x6e3P8GPVFqfc+7eC0kfPncgDx7/ofTyvvu2lX/cAQh3FP+RcYdxB0zHJXbIZ+rzG+7gHuTtfXpg+9687n08CNw2fxD8ITD8h9TPxPv7eN/j17/+9X336NkJt/npp7eFZO644w5J4wfMf1BvuOGGfeflh2huf5jTX47KVZH60XF1hcxtd2Jjhxm7a8E/IjQY+Qex9wfTx3MGhuPH/ZxL9tteu73kH9oWPQPOY5fsmvv7Gvyd78nfHf5h9h95Gyde+pnRaPYY9zeEf0hpNPN+d+FUjeHDnL/CWAqFQqFQWIHFDHSz2Ryg/QlkU2m6NVm3u7DWf0br9ZFHHpEkffGLW1Hrww9v69Ym36YtMltWnrLwdrM17mcLzNf377QIbeFxOsiWHX2xLcP1sT6Xf/O6LW7fi9vu7WaC9Pu6rbYyvT+tW1v6vudrr71233EPPfTQvv2++tWvSpLe9ra3SZIuuuiifefl1G5iO2TzLXr+LG7vzaqk/Y8jAz1Mm3vMMzHRwzBU9zP3TzNJT9F6nUv3V07Rul8bnJr1useR13cxxxY+v5dki8kn2x6z9Hn1/LtkluwD1FrQPeNvApkoXU5283iWyt8/u578nZzzLJcyxt72NMaLgRYKhUKh8CJhMQOd+us9l0GS1fUEGUvmxXsMg/6Se+65R5J05513SpKeeuopSQed3RbS2GludmVr1JaYfaBPPLGtgW02ZYvLoP/EbMxw+3xet9v72yK0Ref922N61qUtcLNa+kAtIqLIwRa8r+3fbW36mfn89oHS5/P444/vu779ze94xzskSddcc42kg1Zqj4FOYa6PMvXF3rVeqQy0h8Twd4m7kqqb4iAzTi4pHqKfzmPJ48Bj1mPZ69xOX+jcPpFEcfwOcpxNnSv1R4N+Ws5CkZHy2vR1JpGRx7q/GZ618vfR6/7uecnZLf7O7257f0tnheaup+1ztT1SMdBCoVAoFFbhUNVYen+pl87bG5yH3xXWkCwyW062Os2GbrvtNkmj79MWkRWktmLNPG+++WZJI6O0tWum6bbayr3xxhsljSzOlhV9l743y8B9vFmXLT23w1Y11bfts6P/1c/AbXDbqRCmNepruk3ez1Yn35e301fKuFNbt1Q+e/vnP/95SeO7vP766/edP2FXHN1SlWxPrc3tL1cG2gsDWuvr3MU86aejr7O3pIKUOgKqZsk4qaCnf87jhOOG98bZrqRkpR+xHVfubxzLROrf6dtMtkudA6+b/Lg+zs+e46Hn5+W48TfHbH/qHo96nc9obohai2KghUKhUCiswCIGutlsFlnayQrtqXOT72CXypK/2fIxU/zUpz4laWSe559/vqRxDt7nsTLVylAzSjNCW2JXXXWVpIPWLZWjboetWzNTszQed+GF24QXVqzS1+l22H/o36XRojVj87625G1pMxOQ78nbvT8VxvZz+Jpug+HjbEU6FoxWqMHr2vK3T9Ts3Oyf1ukun8XcRAi97Ylh9pKIHAdsNptubO2uY6Xlatpd+/tZMqaQalqvez/O7hgeW0xy4n7m8cD1HvPkmCULJAtz+zw+yDy93ct2H+oaqBzmeooD5XggQ3QbqbJNS8PHe7vvgd9FKpx5PrZzV4KctT7OxDjXzlZJxUALhUKhUFiFQ/lA5/pJiJTiLzHNqf2T1WCLxizm05/+tCTpa1/bZmO65JJLJI2WkJf2gTp20UuzLfsPmZrKVi39hLYuGRvJOChbvd6fVrAtTFvdtuTMoM0ypYPqQ1/TbaQPk5axr234eZs1mwH4GfpZuE20Xs1Yn3zyyX3XMegz8u9+d34nbrevl5jC1Ll7mOs7mqvQO06+0CXssRd3vWTWiKDv3GOmF8dJpSljtqkU9Tjg0v0p+XXJppIPNM1WMJaSsZNzGCgZIn2XBJ9NUjgb9H0yGsBLt5nMOGU08j2l7E0p+1g72zQ3f0CaCUjH9Zhn+UALhUKhUDhFOBQDNXp5aJdWKumdpz2HLSBbrWaaju80Q7Mv08zP1ql9jsyGw6wnF1xwwb7r0U9oduR2uO30s9gvSIZplufjvL/XHadqNjeVx9ZWH31I9iXy2dnK9D34nFTH+ry2Cr2fGaKfDXOJus1OvP/YY4/te0YG41N93bvvvlvSQbZv9u37OkzFh57VmXydR5nN5KXCruc1l3H2tieW1votGd+ZWI5Bxkj/GmMPPZbcf5iX1Uj5YonkA+U6faFud/Ib+r6lg2yVCvakluU9pG9sj0X7fJxV4iwAWTTb5/b3WOSuHADMCDUXPaZZcaCFQqFQKLxEWPQnfRgGnXHGGTH3aGICcxloe51d55EOljAyu3EuW7McW5/ez/67W265RZJ0+eWXSxpZjWGmyNJftsxY2sjWM/2Nvq7bYb8k89aScZIB+3f7FW3ZtX5AW97ex+uMj3NbfQ+M+2QGFmYzob/WbTOL9nFXXnmlpLHqyr333itpnB2wItrw+fxM/A59nO/L7/i6666TNGYuahXJKWvMXPT8KcdZfWvsUjoeRp8wdW4+x5aBcqaDfjfOyrBaivuN+7tnJsxAWUWIma04ttlmrvdmzth3yOY8tjm+Wobl7wc1Eilek+fuVWfhtX0d+kw548YlS8Z5ydkkb0/xrFTftt81fo9YIrKHtardOSgGWigUCoXCCiz2gQ7DEOvfzVXk9azVFAvUWq22cOzjtHLT6/Zl2qIyezHz9JKZgmzpXXzxxZJGy8yWFuvwURXH7D1U+jEDEeOymB2FFVSY67K1wlgXkQzT1qB9lr6GmZ7fK++B16QfhdamLX0/c1/P/maz+i996UuSRr+1nyGVzbZefX1b584q5XXXGW2fy1w1aC8O9LC1b083eDapXd+FXtHz3nNOPrv2Nz5jqq0ZU+ilGab7lfsd87N6/5Qth0yE25PKtnfP/J65n/s8vL/2WMP7pHqdzH/di1/mDFrKA5xmxOj7TApqVsqh4jq90/ZZJLbMYxLmqmxLhVsoFAqFwouEVQx0beagxCx7243WarUa1Tlurfi89NJL9+1rn+ett94qSXrzm98saWQ59FNMxSFJB32UPs6M19aR2ZstLlu/rKZCRkoFq61n7+fzeT+zrtZKYu09tznFpflcZoi+Bq1bWnz0o5ANM9OKn7UZgt+Br/uJT3xCkvTggw9KGq1WqoetKrb61/3FPtWrr776hX1bf2iLpZl3lvo6jwsDlfY/i97YWzu7ROY5pawlK2KsNVmS+7V9nmSe9IFyrHDMMT9sYnNkyikbE1kbl/TxcoZIOugPTm0kI+0xUIIzW4yPp/I1VcahQj/FdzJvMZmor9vONvIeuWT94ISlDLMYaKFQKBQKpwirGGjK2DF32WOwBquE2HqRDvogzTAZN/nud79b0sh6UtYRbk9WDtVptnLJ1mwdU0XH2DMyW/qamNnI13N7bAlKo5KYbWSuULJhnps+UKoPWS0n5Qo1aD17f88O/NRP/ZQk6TOf+Ywk6b777pO0Py5OGp+p/SxmKr6/22+//YV9f/qnf3rftZJF3AOzHfXi1o4TAz3jjDMOHe+ZxjQZEDPjuN+3YGYrxncyjpO1J7mdmbV6OWu5vbdM4KwWmS6rF5Ftttuork1sOPlxe7GPfH8cL2R3ZKb8vedHZgx6YqJTPtC0pGLZ6DHSo1DSFwMtFAqFQmEFVsWBtuv8fWp9qWKPloOtE6s1pYO5ZW2xfPWrX5Uk/ezP/qwk6a1vfaukgz5PtjHFrRlkIqwpaF8AM3zYyqGPIFnDzL/p/X2fzPLTsjTv43MlFS19mbSs6QOiNZmsXqpzGavL373uup/EPffcI+lgNhZbnGaiPt+Xv/zlF461Itdq7Ll9calfJPnBTndYTd9joFyf+/wSU2KfkA4qQv1+Gffn/s1MQ2ae/iZ4TM7NftPzH/JejMRgeK9kmFThcry2vxnpe9Lz6ffutRcDmTIY+ZlzxmxpZi/OgtHv3f4/LakgTlEixNwcuzvPMXvPQqFQKBQKL2CxD3SqAsZca7V3LlrztkqcsabNXMOMGFbl/uRP/qQk6R3veIek0VJKfjxuT23kPZDV2RpOSLFutkZpYZIFkrWxNqc0PoukoEsZpPh7qkZP32rPSqZ1yzg8n8+Wo1mjn4kV1n63VuH6nVCl21amcfYiKzR71R+MHhNNWWeOKwNt1/n71Hpvv6Q0dd+0z75V4bJf8Nwco4wH9FjwLFMvO9LSmN6eLzRlNErZgHh/VLVLWXk6N3aRx/V8qGnM8rxkopzZS9dJ1/M3nrlz25k1+jg9I5Eq0/BbuTQGfMlYLgZaKBQKhcIKLPaBtn+d51qlBi3DxAJtUTi70P333y9pv3LPVsnjjz8uacyH+p73vEfSaJXymlSCJhUa2VvP8rMVmRR9Sd1r0Nqmko/n8bNombPZVrJ8qb5N8XCEr+n3QvZNMMMKt9Mn6v1szfpd3nzzzZKkO+64Q9L4zp2n2OehX0Ya/adXXXWVpDGz1NyYseSr7u13nHAUegaDqnjOXpBRTfm46Mtyf2BmIcZ7ep36g7ks2uj5C3vsit8xjlnGmCfNQPsMeswzseJe1Zb0THpxpHxG1IJ4DKb+wHYZbFe7zrhyzm5wOeVHnbpHbl+DYqCFQqFQKKzAYh/oGWecEeM/231a9KzbxBLNQFvFnmGfivOrvvOd75Q0Mo2Uy5EKVDJQ+sqMngXIvJe2oHpxhMnPw3Xmg2RVmPa3dC2DtUh775HPn/VB52Zo4faUpcbXNRN1hiWzSjNtVqRpY2xtETu7EX2h3pczD0aKde6tHydMsfC1vs/EgDjbMRXnxxq9jKW2vsBM02pbM1DOxvTy9hI9xWqqINKLteRsFr9vnMlpx+lcxkl2nKq0pGeRfJTpHlP7+J2igppKZPpOPZanNCm89+T75JLf8t5YXTOLVAy0UCgUCoUVOFQmIqPnc+B6bz9moLHFadYpjXlUnX3nbW97m6SDuWcTw0yMM8WF9kCriFYTfaQEr8tnQitqF3tIrLdXx5WKYPoleC+0+KhCpL85+V98XbbbjMNxop6RcD8wA7H6lv1GGrMaXXbZZfuOWdpnk/WaWPbpjmE4mNOav7dLIila2Y85Dukza/9P3yerrDDnLWPAe2Mn+ftS1pw0XnqMdaqmZYs5/vUeu03fF85Mcez1tvf8u4mhGimnrt9p+lbQR9qCbJpsllneGE88d3aop8CfQjHQQqFQKBRWoP6AFgqFQqGwAodK5ddun1qm35MIyfT9iSee2LfuKbv2vJ7OsdDkvPPO295QmDLiFChT61GAw/MYbhMLaycBDadEKBxIwdicCuP00NQ0Q0oSkc6VQns4lZu2JzEE32fansQVPK+n66+44gpJ0le+8pV97fCUnlP7SWNqRfedu+++W9KYwJ5J/Jkc20jTOkkKf5zERFMhaQlpejQJblJox1QR5CQ8cUgEl54O5BRdmspNbTfY35IwcC5SOJgxJ43e2oQJvSnb9Gx2TaG25++1M71nJr9g6j5+A9rzsoRaSnSfCq/3hIKHQTHQQqFQKBRWYJWIiP9Pf8kTM+XSVofLlZ08eVLSwXCJyy+//IVzM9SBKft6jIKWEtuUmGkKj2HZMaY0SwIDJkwgkiBhKplBT+hCazFZm7xWsmIZUsP9U9qw9HtqhxkJw1rcX2zVehZCGgsP+FoOZ2HiDT6bxDp6woLjIh5qscsC75Uc7KV9JJNg+Erbxyg8McNk8nj3Ay9TSa0e6+oxUfbLdP50vl7qviQE2oWlx3AMp9KDqY1c5xjtzcCQcTLkxN9rCoRSmtD2t5RAwSJCJuZIiRWOAsVAC4VCoVBYgVOSSKHn80z72V9lZuHj7ANrC2q/5S1v2fdbSkCQLGkyQFrIyULjvSSGS+aZfEO9JBQ9P+Yu5pOedy+NoUGrk4kTyABSWjFasbxOr/iyLU8zDzNNtsPsXxoTbNifbmvUzNSzGQxxYP/pBV0f14QK1DP0ZpG4zvtO7zBpEdrxxWTqTKjgdb9/JkNJPrGlTJSsuhfK1tMtpGdB7LpOCqVJ4XbpWvzezUWPZScmzJkz+kKTT9Tvuk3ewr7VC2fx3xEW6U7P+TBjthhooVAoFAorsJiBSvOZJ7endcP+KVuADn63ytLKW+mg77N3bVtAKc1WUnDRR0kLj0ovWnpMP2ULiym3kqqX7C2VQ2uRLO5kNfZKGyUfKhkpLW/6E/ksjJTEgsezjJlVeVbftlarn7OZqPdhMgb6SXapAdvtxNLEG6cDpsbh3LGa+mNKPUkWOZVIwYkT/H7t+/QYZ5m+uQw0zbQkBpr8fkRSwvYUq2m/o0ioMDf1Hr+XaSatp1vpRSWQcabvJfvLVPrWxESpIGaRbhbanjuzMAfHb9QXCoVCoXAaYFUcaG/OeC4D9dJMwgz0oosukiTdcsstkqTHHntM0hjDJ81nnmSK6XdatSzdlXyiBJOVU8mX2FfyD9IKnuP75L5Gz4pN95J8mD3lKo+fa3mnuD6q6+gLa4uam3Ga8Tz11FOSxiLd9o/QciaSPzfhuPhApd2WeE/JzXdIdbpBX9cUA/U2Mk/6QOf6PrmefP9z2XZKxzmX2fawK5l8mgFZOpYTk1zLxhIj5XFJfU1fJ99t2y6em4zT31TPKqVZxZ4Ku5LJFwqFQqHwImGVCjfFMyVFKa0KznczO5AtUVr9LlXWHtsrXcQ597kqwnQ+7s/SOb1n01MFE8m/skvZl+LYkpIuJQZnG5O6kO+RSuP0DNIzTZY+rVT7OMwm2zhQ9yErt91GF+M2Q02xYXP9WkbPuj0dMQwHC0O0v0n9ccN1jkvG8pJNTm1Lyx7T630LjMQ0kt9wLbub2xemWH3Pl3lUDLTX5sRY+e0wUrx7+u6lWYIpMN6cPlCv+3swd6ah5y/ehWKghUKhUCiswKHKmSV1JhWuKSOE93NmGTMOly9zDJ9jPW3FtqDaq2edJvaT/H0p/ytB9pUUp8mHmdrbsyzb7XPVf2wr0VOipv2NXjHy1G+Sr4J+FDIWM9EHHnjghXNZwe1zsMyc/er2m7JvJUs4le86TsxTGvUM7Xq7JIPoKVapJp+rkN11LY7ptEzlyhLTSLNEnG1gVqU0CzSXBRK79p/LNNMyaScSer7RXjw6Z9LIUFNO8vQO2va38d3SQT0KfZ1kpFTnJn/9GhQDLRQKhUJhBVapcHs+T/ouzBiowDJcOcPq229+85uSpELSGTkAACAASURBVJtvvlmSdOmll0rar7JMSrtkdSY/m60W+u14fDqfQeY5N89ssuBsLSWruhejJh1ksbxmT6XbU8um8/f8ZT3fJ61bvivOaDh3qvuRJD355JOSRuW2ld6e7fDSfaxXJYf3ltaPUzzoFAtM7I7ZpvhcyAJ5HHUPrQ/U748Ftf2dmKtuTf03see5MZVzGagx1xfam8mbgx777bU1oecLnbtM/YnvlGO8/U4uHVO9GbDkjy0VbqFQKBQKLxIO5QNN89xkoGSeZFlWSnJu+vzzz9+3vUWqotK2s21jYjmJyTk+KVmJZEnJ2iXDTT5Ysrme2nHKauqpy2iNzvXrzm1L8p1SRWvQb5asaf/u/uOZCPvGHeNp5tLCqlurcq2+tX/dij37QJOvJ7FyI6moT1d4Nqnn+0x5RJN6nP07ZQ1qz0d2mrLiJFBt3ovZXco857K3XjuPwufWa8uSOPEp9Hyhc9tHHYzHLvMcs6IK95MO+qJTrDyXRi/CYq5GYwrFQAuFQqFQWIEjrQearE1asVRL2dowG7jqqqskHazO0qqvnBeX50wqSa6nmENaO4mhMleuYUvK26nCZXuYY9fn7cV7TrHHlIey58fjdlqR9A/3fJ/pOLa9VyGCLIfWq/uAfeRt/kz3Jfs+WS/WfcoM9cSJE5NtYFsIKoePE6YYaC+7j9GLW+b6LubEfaiWbNvbtpXb5zLFno+zpwxNuaN7Ste1rLBFb5YpzSr1FMnpfEt9oSlONOVEZhWWqThh/r3gd4rf7qSeTt+xdK/FQAuFQqFQOEVYxUD5F7wX+8WlYcZwzz33vHDudrtj9RzT1/q4kvVA1pVivth2/m7LiP6UtGTcJ5cpfipZfMz3mNS3Uz7QFBdHpHMlCz0dN9WWqe0ptqynEuazY11Ib3c/kaT7779/X1vdp8xE/X79fhILMXrW6nFjoIwD7cVasv/R6icr4zchsQDpYAwu+3FS2qcsOAmJcaZavWQ4jA+d+yx6vlW2r0WvX8310fMaiWXP9fvOZaJpNpJ5rZnHts2VPFWZZepafJ9L72Xu9hbFQAuFQqFQWIHFcaBTf5WT3yRZJ/R1WFVpv5VhS8Jq3CmVZVK1pmsmZkrr0WD+RVtOtIp8HqrKyEyZRYOWPbNnULGc7q/dtxeHlvwjPbbNZ2UkRTLPn6zf9O7IXnoZaNp3YqbpttrXSYZPsMbp3BzJxyn+01jDQFNGoZQRjMxjqrpRT6mbmEBiWyl+ucc80zvv7Z90E+l8Ce37mMNOp87Nfkm/YTq+58/t+YfnKpTTclcmufQdS3l002zR0jHbY/NSMdBCoVAoFFZhVTUWMoLeX3haDrYwzRgefvhhSSNr8PmvvfZaSaOS0pmKps7Zi99LvpzEemidMJcqFYK97D60Ssm6en6R5MNtQSsuWYVktVTNcruVcbw3+i2SHyX5cdmP5vodud3+ktZvcuGFF0qSHnnkkX37us/xGST2nnzkPZZ/HNDOKM31Myb2lfxMnN2Y6ufJX8ZzLMXc+M6eqrbHXHsMtTfGp+5vro+9x659HKMBeHyP4Sb/cG+Z7ovfFvpG2xkKjlWjl686PeceA/Vxc2K7i4EWCoVCobACixmolCs09NRMjAmyVeO8pPZxmomaUUxlDaKFlRhD8r8lf19SDpsFe90WXYplpHXje7b6c6k1ZCR/ZXuO5Gcli+azoVXYi3sj+Ox7cVnMiUzw2VAx62dvJuzsQpJ05ZVXShoVnmeddZakMROR2+QsRr2sTGxTYp7HJSMRVbg9H5WR1J7sS0vbIs0fA8ZaFW7vvD1W1XtWXFJXcRjVZzomLTnDYvS+h3PVuWuZKL8pc5DU+kaPkfZi640l76EYaKFQKBQKK7CKgRKJTZERMUbTVVdc/9MM1CzBv5933nmS9qsszSjYBrLV5GfrWSFkoPQDMiaMilGDVhEZK3Pk9nypaX2qrbR8U2aixHJ7fmKzuxQfRwbKfpJy85LB9ti6ZyxamFmanRruY2azjz76qKRxZoC+7d4sy3H0fe5Cjw2m+M65ytcpxrErRnQXUm7U3jtJ7Ggum0r+wLnnn9POpXHHc/1+vWv3nn1PhZtYYs/H2sv8tus3j2VmsOJ7SrlyDzOGi4EWCoVCobACh6rGkpBiyVJcH7NSeD/mKZ2KDbI1Qb9fyviTLKZUVcJsztaNWTD9etwvqTzJ1hLz7GUKmfKvnH322ZPXSvGbvFf6SOf6ntKz9bNq63RK4+xB8tek+C6D/cXvvlVpexbDv5mJ+lypGgSr/MxV3R7XeNCeD425otNYnst8dqHHZtPs0Vx/7Fp2xTHba1ePjbGdU88qzWSlfpl8iSk6oRdDm7It9bL9pHvj9lQflqrcqX3TjNga//tUG5fgeI32QqFQKBROExwJA01/wdM8vq1/MxH7q2688UZJByuakCG1//c5erFWvfytyf9GxWe6J143KUYNnp/tTxYn1b/tM/E90l9r/x7h383O2OZezk+f10tnkvL7sw/bDNS/26ftKipt/GZ7fVqcXnq7fZ+u9enrSQdnP7yv2+LtzG5FdXcPx9kn2o7l3nKuirzHAqZ8XT1/aaqly2XP/7ZWMZqy8qTlXP/vnNmKHsNkmzw2mYubOaG93escy/4m+Pee3zfNNpI99jJUsZZ0+39/J/gtT1mz+Hx7ma2MUuEWCoVCoXCKcSgV7lJrOzEJswL7rR5//HFJI7OwddWqcFNMYy8TEC1qr/vcjkm1BZZy3zILD2NbEzMl4yWrI3oKwNYac5vmqgiTlepamWaMfj+0Uh0b+/TTT+9b0gfq8/mZOF7T+7v+q1W9tDQZd0xWacbdWpy0yN0W7+N7cx9jfdfEbuYytuOAXQx0al/pILuiJiD5wlKWIelgTcg0O5N8lClOuZejlirxnpp2rj5h6e+7nv1cRbHvyf06MUqPWY/ptO4lqxYZZJjMBuax7DHqdX4n12RnSj5PHtNjpmlGYwmKgRYKhUKhsAKrqrH04vJ6FpuXtk6ct/Suu+7adx5bRVPsMqklU224KcbWrpt5ug1mSfax2hIjezv33HP3LclsbXl5mWKSyEB7qsgpX5KfFy1vsgT6QXxvPt4+aT8Dx0qSafL8vI7Py/gsv3fPOFhtbZ/o5ZdfLumgb9SgP8Uq7SeffPKFfWhl+l7dhsTWk0+z1+ePE/M01rQ5qck5I8RrpBkYaewnU/G8U8ey3/pdemkW5iV/Z/+njzUpRLmemAzb3fOVTh3fm0VKMywew88+++y+pcdaWrd+wEvqGRit4NkCf9dcUctLRwR43TM91hyQRabIjXbfFGvPZ2ak/XpjPO03hWKghUKhUCiswKFUuGluOflLmFnGVoyZhy27r33ta5JGa8iM55JLLnmhHbRkjORjZBtseZl5mr3Y8iIztcXG/WmJuT2OSbQl5pqmtsjoQ0jKMtZK3OUHcNvtczTo1/B+Zn5enjx5UtJYwcTrtErJsvk+aaHb8vez97P0dq+7fW7/ddddJ+mgr5Pv2L+3ilpa5Kzow7bNjWNLbOM4xoFOzSbt2lca+51nBzhzQnZFtSX7vZRjoZOP0++WjJP9x/2KS6rCqT7n2KOfz7NSzN3tdcYnpyxou9S/SQnPe/c9e4z6++RZJH87yTw5LsjKU95Y3wv3pz+Z+grWfubsUqrHPPVMjBRpMTe/7lHoGI7PaC8UCoVC4TTCqnqgxNK/5PSBXn311ZJG5mlVpmGryUxVOpixghZbyrtrVuWlLTVbboyZdMyi/Wxe95z+HXfcIWlkcYxhtFXk4+3fu/jii/edz0yVTNbnoYrNFmjLwG1VWsVsa8/3btZsa9TPwD5O+zy9n61TsjTfk61IW96+Xsr2Q+vS79VWsq/DWM3rr79+3/mIqdixFHvntpJJ2lJOFWLmZiA6Lr7QYZiuxpJ8u8wg4+fqfunfyUhTvF+baSZlN0oxi+4f9NuZeXKmhWpy9uvkw099yM/AfcnfAi6d49uMlcx0lzKZOgLfA++ZzNLfMS/9XfL+ZJy8N/o4ezVaU/y835mfMWclqYrnt6Xth5wR4EwmsZRRlg+0UCgUCoUXGaviQFM8TVrSf0K1GzPieLstDW9vK7CkyiKGjzWb+uIXvyhpZGdUrNrSs+XmNtmys0VmtmTLzsfZ+mUcKO/B5zPrs1/3sssukzQyUR9nC58xsVQktv/3PXtfW6lmmPR1+l5oldLSswVty9rs2SpqvxMzUbfd7SLb971S8edn6HflmQfPWLBiCqvDtNvMEtw2X8vXdp/qVWEhjnMGIqNtcxqraawT9FcmhjKlHmd/o2/Sx3IMUcXN7WSmSa3LWMdUx5YzK+6P7u9U5Ps6Xue44MxJ+z7I4Hwv1C0kVa2XvnfOzHnp75qXvje2jbqGFJeeqlj5eParxPLbZ5H86AY1EbxHHj9XSV8MtFAoFAqFU4RVKlz+v5ejMql1bTmYhZlxMJeq0TKMdC1bOrYubYmRXdkyMpuyOvONb3yjpNGiM2tyW81Y7Mu0hejrmNl6u69ja9Xn8Xbfo9mhr+v2XnDBBZJG65WWfKu4pX/UVqrb5HX7QfwezMbo1/Kz8bP1uhmhlz7eoGKQ1q/P79/JBAy388EHH5Q0PkO/AyqtW/8l+wH9XLRik3Xbs0qPm++zxS4VbmKeZC6pUgkzfiU20J6brCa9C8YTJ/Wsf0+K0BRHypmQVKuS6nP6cg3GjpOFTTGixNiT0pRtoaaCz85MMzHQ5APlvTDjkbennLhkfbw/vqOpe07qZd9jL8tRLwNRqXALhUKhUDjFWMxAd1Vb6GXmoBLLMMsy47DlYB9bsnKnzkVVpX2N9ruZ7do6pS/Mx5vZMVsJrVX6Ss3y6BOlxW7QV8CMHsz0wZyXrSVP655ZjsiiDbcpZXfyeX2cnxljAanE473xmdN/wrycnpHwszQzYPuNtr/Rf0Um5Gu6LSmj1SvBFzqFdF/UMXB78oHuUiunqius9et+5GOp/p5iL23bDM6QUK1LVpV8pGyvxwPHsNtJ3yfZXfstY+YsLjmmqAOgZqJXBSUtyZYNfls422SkKi0GZwF2xcamjHcG+2JvWXGghUKhUCi8RDiUCpfo1dVLWS1sqZl5WLVmK4p+walr0H9Cldq1114rafR1MkbRsAVFluP9aKHZh0krlOoyZiUhq7aVakWrmbLZuS1OMtrWKuM+zEHrtrHSQstipdESN2v3+Wzduu1+xoYtO+bB9L15Oyug+HzMLpWUgsnybH2oZMH0gadKLzynsba24HFDL5NSUi72nseu58T+yv6YVNfen6p+zpxQEe++kGbEUjTBVA3e9rpkxOz/jF3nN2GqshL9pmxLurcU6+oxklS09F1TlZt8pT11L1l3yls7lRu8V83GSPWDe1nGioEWCoVCofAiY3E1ll0Wau8vf6rNaSvlmmuukTTGWhpT9UC9LVmRtkbMPJmv0vubVaV8m8wxed9990kasybZx2qmSzbFTENm1/QfmvW5nc4DyzhG+hCmKreTWXrJOFEzfp+TdT75rBw3aqv2gQce2Lfu69vytlLZ79W+TCuOzbaTletn4d97frkW7B9k334WyTfay4Xby9hz3NHL20p/EvuY0YsRlw6+R7Ifvn/6t6myZeYi9zuPUc+cMKY7ZSpiPKrb7vHBuGjPGpEtpqw+U9tTjmF/pxh/7rZ7nTGwKfctZ2bIdDmzRk0GZ5u85MwE2XaqmsUY3CkszSzUiwo5zNgtBlooFAqFwgocygdK9jd3aTAm0+zsoYcekjRaN7Ze7Attj0014pK6klYGFXWO+7z33nslSbfffrsk6c4775Q0KkN9PbeVsZJmXT4v2Zd9rKzYTgvSSH7l1mqldcdYMlvU/t3X8Hb7Jm1F+lmYxTt/r9/Txz/+8X3PwDlr6TP90pe+JEm68sorJY1+aGdEchYmW7k+nn6TVNViCrRo3RbGyblv2aJn3Vaj5yN9uTBQqjaN5CdMv6f6lXyHUlZ9pww0nDHxuc2yWJHEMy6PPfbYvqW3M18sFfasQMM4aTPOFJuZ+mtSpk6BmcfSksrgNnZeOpg7nG3j7GACmSOrxPDeOMOXfOZTM5xp1ofPs6e6XVopac7+xUALhUKhUFiBVdVYkrWd4rnojzKYUcTMxEzIFh9rdE5dg0syjWSp2S/yB3/wB5JGH6etUa+/6U1vkjT69cyGbc36uldccYWkkW250oyZLP07bofbabaXfBEpzqo9N58Rrc1klfr9OPOPLetbbrlF0mix/+7v/q6kkUHa6rRPyMfddNNNkkZ/8Wc/+1lJ47P1s7If2czXlj1nKJK/mwxbOlj5giybsx5pViXlzTSOM/OcykS0NNaOytekPt+Vz5TKz57i1GPYjNNj2Iwz5YPlksp6V0xKbJmaDfs8vXS/dT9OsZtcsspIi17mKzJIqmDN0skUeT5+i5nD2+tsc8pcxLh5RgKk+NSp+04qa8PrfN6s50o/7NS12t8rF26hUCgUCqcIq3ygRs/6ZqwjfQr0S9lv+LnPfU7SyFSYVYP/33XtNE9uBd7v//7vSxr9dLaYHn74YUnSz//8z0uS3vnOd0qSPvrRj0qSPv/5z++7d1tWrMDge/M9sDK7a5+avaW4UR9nFd2uyu3MCMTMLfb52Kfp333PngmwStbM0+/rZ37mZySNzNLviXFwfEd+Blbzmkk4/7D9w8x5y/shIzU8U9Hu63OYjXCmwMzflnpS/PK9GMc1E1HLPlskdXvP35TYYsp8M8XoyV6Z29TvlFm/3D+Z/cugspNsjQyFscPM+pUUqFyn8t/jx31syh9s8LmThTEekwyT9+btzO5mkEF6bPre6Rf2u2FuaSqljcTGd6ltjRSrzeeWsjalaizEmtjuYqCFQqFQKKzAqmosvb/gRvJVcd0WgtmY1ZyOM7S10lYeIWMgkjrXfhOrau+//35JBzN43HrrrZLGiiO+l3e9612SRuvS1rCtHfv/yDhvuOGGfe3w8Waqvnefh5mQkgpvioknvxZjUP0MfQ/nn3++pPF5M68m6x7aH2wGSkvfS/uW3va2t+3b7mft90p/9VTljva+dlXL8D35Hrz0c/V7uvvuu/dtv/nmm/ddO+HloMKdamtvDBusrEPQx5XqSu6Cz+F+SkbIWpdksMzh7BkV90fqJVK8KWMlfc/M7ezxQYU9lfacQZmaTaJfmP2XVZ2Yr3duTH7y/feqrnB2K+XS5bPlWE7taftHqgRD8L0xwsLHcdawp2/YhWKghUKhUCiswJHmwl0KWwhWYdr39uY3v1nSGHNpdVurIEtz51TK0Yr1tbz+Yz/2Y5JG69EszJael7ZirK61/9B+OzNb+ixpLdkCI6uzlWXr2PdMPwkrQuyqasG4OmZdYk5R5s1kjk4q9Qy3mXk2DVqCZAiMfeX+tLqpwCZjbX/ze/FzdoYnt/kLX/iCpHGGYK5/5CjyaB4npGdPxuA+wv12ZeMh26d6n3439z9fy349Mg8q7pNugmOLeWO9nnxu9nF6Sd8nMyox65XRftN66ueeYrmX33epjzrV7Uw1UlMUAb+P6Z20TDXNOBHsW/SB8l6OIjNRMdBCoVAoFFbgUCrcuaCSzLAlwHyzZgNW19mC8O9SjgM1aAm5DbYK3/72t0safWHenqwQW1z22znmjHXwyN5o4dl/Y2vUv9uvyDqgbAetrNYao1qVWUHMylmNwtvJ5LzOupxuK5mAfUG0Ppm/s43XbJ+FmSYtRKonU9yW36E0MnwzUMfX2r/uttoHyuo2c9W1L3cGmtTIfpe08pP/nftN5XBmdR729Z56NWU/oiKffj2PSY9t+ub5DUhxq2mZ1L5s3y7MzetqJLaW3kv6nYyUvtJehZQ0ZpPCmnmxW3B2I/3OusX+liZWfJjZpGKghUKhUCiswCoV7tT/W8ytB2qrw0zmK1/5iqQx64+VsFbK2n8pHbRQeupBMxIq4ngeqtjMlsySaaXaGqKCj3k6bf3Yd2pfq5WsZnWsiWnQ6qIF2Lbd+3pJ/4YtNDLOFHdJfwVzjbLmX6oVaEufPi5aw8xyMhetItQs29sc90mlsX2hvvZcBvlyUOEuAcduyjhkpIxF7Ivt/5mHmEzPYzeNdcadk924T3A7le0pXjQpTN1u+kIZ/8k4xKSY5f9b8HmSXSXlaapy4mfi/VLd21QbNa3z+B6znZqZMNiXkv6FGo2kek5jdc0YLgZaKBQKhcIKLK4HuisO1KBFmPwoKa7KS2b+aH1c9AHSWuR+VKAaZG20Clk3j1ZsUsdSVeu2e90xacz7m3y5tOSm4qloaRtkwanqhcH9eG0+AzNLX5++JcZd+Tg/E1uOU8ri9vpJTTf1u8/l39yHvN3XNPM3Ujxaz2/yckXybSVGwe1kFmSi7T4pwxQZX/LDGfS9J8ZJ/QL7O2MYeS+szkIVLv15nHmjT3UKqR/Sj8q6nUbyC9NHmhT8yU/L/dI3JTFVRgKQZU6B36VU2YXfyKSTMXr6hl0oBlooFAqFwgrUH9BCoVAoFFbgUCIiIpUBSiEnnFrkFC7DJFp6zynMXgFkTs1xyoGFZFn6KMm/mfg8CWI8hegp2xQ2k8r69IRZLZLknKIiJkhIUxwUeaTkzgandzhVZjBBNafZvZ0CB05fcYpYOugG4FS578HbOd3cQ5rePq5Tuundz53C5VQdp9HmTOGy76cQiARON6aCEpyiZTgX320SQqXyZCm0h2K8JQWeU/hKGotJvJO29/rv0qnfXcXup9qT+k/7fz93uqgMig8ZZtQTDVUYS6FQKBQKLxKONJFCL5SETn1bEmYBFnRQ9j1ltaZr9STPTKPFUA63jYwxiYp4fp+PoRypiG5KvZUEBlxvZd+Ux7uNtP4Jhq30krd7nZY2LXfvbzbItIZ+z95uRtpLOE0L3teZSmzOsCA/f8NhLW2SjvbccwPYjyMD3cWA0oxNYk89BpqYaHsMEymkmSu2KYlHGGaSwlsSUzUS6+uFcrCdKZHMVDEEMr25YqDERPl76s+9RCKczZorQjJ4PL/XU/2L+6R+y5mFlMqP+xcDLRQKhULhRcZiBnrGGWdE2bix1Gfh4y+55BJJB60Xs4j2OskCZqB3sjIYzkCrlaEatMAYcuHtTEuXAotpTZHhslxZ8hu3VjN9gSwwnNJ9JYs8laricSkNXmLLqRSS253Yd2IIU2nJfC6z37vuukvSmBDDhQtYVi6hZ7lzv+OGpQkkmKYu+eTIQKdK1CXmNpcdGSn0gsfTp95LR5d0Cmnd4HmTb3YOA01ML6XsS7NC1H4kJBbN3xPmzmCQzXN2cte5iTQzxnCiubNLc1AMtFAoFAqFFVjlA+1Z3T0LkSzQVodLeLkYspWRzzzzjKRpfx+tyJSImpYQlZz0t9Afw6V/T0pBphWjRZgCiclwWcyXfr/2umaCLKOUim/TCuX78LXN0tKzNfhe0+981nwn9F8nRrtLpWkftktd+Rk4HWRKVE6fdk+519t+OmNOAH8CvwE939cuK783W9RTjhqpP6Rgf2KuL5RI/kkilfXrKValzNiScnnuPcxVzaZ7T++MzJfvmHqL9L2V+mXM2BZ+v5LP+jC+zxeuOXvPQqFQKBQKL+BI4kB7f7F7DILM5umnn5Y0ltF65JFHJO0vg9XzhyU/RbISyShpzfZK6JAp0iJnQmzO05Ntsb1m47zv9j58bj8npgfsJYomg/S67y2la+P56Ovkcfzd4Ha+EyOpfdv2XHbZZZJG36b7kp+NnycVvIn1vtxS+Q3D7pScS3UMTIHJGGP62dvxlPoVFdO9d5AUnz31dvJF9t4x2VW6blKupv2mQJ9yKoLRU9vyfD2dQdqPs0e8btrOb30vMmDqnKmtvZmJ3j3NnUndd83uHoVCoVAoFA5gVTL5dr23v9TPEkRL0EWQWbTavtD2mKSEo6VDBR7ZkJEsp3QeFudlPBOVX6nYNK1jZkXxMyBzmvILpHJgvDcW1rZqN5UjI9swaPmROTCxu5Fiy/g7FXrJZ9YqO2+66SZJ4/PyrIbXk2WcksezTT1F6HHAUcwm9YpW97ICtb/xPXIWJj37ue+gx0TT/lxPs1S9OM/EcPhtaLf1fJZJ0UvwWvQHp7j69O3ubZ/rU+0VspcOzm6wjUYvaTyvvdTHPoVioIVCoVAorMAqFe7SueLe/qk0GLP9tJYFt9GSmrLqpIOZiMgoWdqIVrCZJsuf9fx1ZtXGBRdcsO8eDV/v7LPPljSyQTMnL432WZJ5kgm67V6acdpqtfqZ+XutjraP1aDPyOflvRhuO9VxZB5JwZl8tUbbP6xETjMRqX/0YsN6PqHjhKPwgRo9RpTiGtt9UxuSApXoZULbpR+YanPyJ85tT+8+kn+4/T9nezhb47HksZeKTbMtzAjle2V+4B6jJXr+3Z5+YuqZLo3Tncssl47xKRy/UV8oFAqFwmmAQ6lw01/oFBOUslO06lrpoHrUzOSqq656YR9aSpzbN2hVkoEy8w+Pp3VIFWGynMy2zOq8NKtjpiOyRmYymsrg0ra7PSetzBSDle7tiSeekDSyON+L2+78sXyWBmcHGNtKv7HbayWsnwVnAVI1iyllp58jWbfBDDq7Ykrb7cmC5n7HAXPaOvd++Hzox2R+0lZ7wLG8Kwe2lBnrXNabWFVSyxrpG9PLN5sKwfOb1D4Tj+UUVcBzsZg9Z9KMxEAZQ+71XuWlhKW+UD7b9rh0rt7MxVzm2VvuQjHQQqFQKBRW4Ejrga75Cy4drHBi9mUr6Dvf+Y6kMVeuNLKVnqVDK49K0qTopG+ypxSkX85xh08++eTkfqwC06vrNydeitvIQBmTamuTdULddlco8bO48MILJY1MlBZeqgnImQQf55kHKpr5bKm4psU/lZWJ7NT3YHbre0+1RXsq3LnrxwVLdQycHUjPjyySy/b/ZGwpX3SayZqbhYfvNs2MJPZGVsjxw+VcRtT2tZRxiIyelYyMkm28LQAAIABJREFUHusmc02qadbWXIql6txd5+j9PjeXcmUiKhQKhULhJcaRqHC5vfc7123tPPvss5JG1aetLMZU7rpmUuLRUjZ6sVuE92fcp/2EDz74oCTpgQcekDTGH95www2SRuZjBk3/CFlZqqFKNfDUNvoG/YzMIA36jsz0fQ9uk9vqGqlJRcvzJp84lcupXmgv68kucMbhxIkTksZnkHygvX61xlo93TDV9rn30xsna5BylfYYg9HzY3M/jw+PXSvlyeZ4vlSBxt8rszYex+gC+ufb61IjwZkyKu6ZSYt6AYI+Us5S9aIb5jLJuf7LXXHtvWvyOVMvkmb5jkJZXwy0UCgUCoUVOFIf6NpjbVmYrVnlaZbAihlSX0nHa6a4Tu7HTD+0yBhnZf+slavO22v25v0vvfRSSWN8J60qZhqiRWjr1Ps5hrONC/W90cJl3KUtZTNJWu5WO995552SpIcffljSaOV6hsBsmr7VlN2Eql0+W6oge7GFiYG05/Bzc19yHVCvpxmJtExW7HHDMOzOhdtDj4H0rPspNT+f7dzZAF6T/SplSWI8tGvFsl4ol1Tg09fJ+Gv6Kbn0GG4ZKNkpZ6T4PWNu7eTDZ39PdUVTvVeqc3uYm/eX+7X9I1WcMVL/ocJ4ro+0VLiFQqFQKJxirJNW7aHHKHs+CloStsTM2swW7BvddS22iRYarRRaHVShpcoJthJttVple/LkSUkj83zooYckSTfffLOksSqIz8usPGayPSWq22N/TRvf6Hv0NrLpNPefavJ5JoBVcbzdDNbH+3r0BRnJamVWJzJVvpOUsWgqp6nbcOONN0oaM0CRncz1sxnH2fe5Bmksc5wZST3Kfj31W8//lnQOfJccu/6ueAw7NtszXx5TvYombAcjAlKllMRAudz1W4pt5Zj398Tfl/SdNNIzZ7xoioGd+63nO+GYnZPdKekVUqan3szGYVAMtFAoFAqFFTiUCneXD2pq/x58vBmnazqa0VAdt+TcSWWW4kINKvW8dIykfZ+PPfaYpJGJ2io1i7Z/xee///77JUl33HGHpJHJOgb2yiuvlDQyWOehpa+0Va7S7+c2+lpk8j7Wftmrr756X9vNNK3KNav2vVrRmmJyaQHy2RuM1aU6O1mcVBq2784WuM/pGQDGzxlz+9GpsGJfKhxF2/nu0pL+qNZPyG3Jh270so1Rhe4x43HhvuExST0BFar8BjDbj9vrscssYswVneqNtmzM13IbPXbdVvpE6dellsJt9Fhl1i9GPHh/ny9lJuotezlxiV19cq5SPqlue2N3zdguBlooFAqFwgqcUhVuzz+Z9re1dfnll0sa2YPrg0oH/R3pmqlSOjNwMBeu18k8zRRdm9RtNdvz9a655hpJo5/QsAVpNmf/oi1Bszq397bbbpM0KmOZ89Lnk0ZL2yzY1zCTvOKKKySNvh6fizGprAhjBbHv1VaxfUd+P2QbtNCJVJGBvlwy0qSaa9+975GqZ/p2kn84+XbmxiK+UpD808Quqz7543oKUrIaxkAyvtLM032DcZ9kdaybS0Us4e2+Drf7eN4Hq06123gsY7vpgyaz5Mxa8gsaVDIzdpU1gnsxmr040JQxacnfmZ6Pem68ZzHQQqFQKBReJBxKhUss9akkpRZ9EFPKPaq3ejlLjZTzMVmvtibNOL00E3300UcljQzU+WLtNzzvvPMkjT5Mx06+733vkyS95S1vkTRaw75nW5BuDy1KMuR2Xz8bt4UMkFYqsyN5O9W4nhEww2WlGfq9WLvU1ivfK98VWSGtWPppeP/SyDzN6P38qcZOtUe5viZG7JWMNc+px+p7Ss7k8/QsjceY+6vHNhkov0seF/Ztku25b6XsYvyWcDkVB8pZF86Y9Bgkv2tppoVM0khVXtwO7j+3aksvv/DUzM+uGOL2mKTmnuv7TOu7UAy0UCgUCoUVONJcuPx9bR23FDtmC1AarcpenlQfS78GM3AkP4n9fvT/mYlahWvr89prr5U0qm+p/qSvyP5FMkoqZmk92ep1bGbbJi99Tu/L+DQzzRSvyWv5nqgEti+UlTf8TJg3OFl8fIcG/S9z2I3jPc1EDc5czLVKE4qJ7kdiFGlMt9t6vk++u5QnujeWPYapZE8VZdifPWPjbFas7EMWSEaaVLltTDf9t17ymBT7mOLICbK0FEtLX2xirmnGZu43f0p53RvvPTbN8/C4w3wDioEWCoVCobACR+IDXWqFJ6ukx0DNhKTRj5Fiqgyq1bg9WbFkprZafV37QG2ZmXna52nmY3ZH3wFZGbPv0I947733ShrZpdtjBtz+5jabsbP2qC1pvocU6+U2Oweuz2dVNPP02tL38fR9GklVyXfp9tgfw4xJUzG8PJZqwlSrMSn6ev654+gbndPWnqo4PaclSsjUjuT7NJJ+gapZKulTzHeKJ2Q2HsZSMtcqMx6ZCfvb4fFBJb+ZsnQwNjWx5JR/muv0GyekWQCDcadk1/yOGXyXiRHvqu6TlPFr0VPYz0Ex0EKhUCgUVmARAx2GYZblOHWc1M/HmFgilaftvsnCpco25bb1OWnFUn1rK9Gsi7Gq9mXS30hlKn0XjHVkbCarr5hxen/7X9p7s/LUYBYYr1Nty+dP5Z+X9v14f6oZU95MgtYofUbsHymL0C6rlTMURq8OqHEcmeVRoscOE3rK2fb4HltJx/Zq5bpfeuykWMqU+5azQxw3Se3J9jHOM9X2bLMzMRKBFV7YRs4ecQyluMtedp+eSpeq4Lnvkr/PGV8pb3XC2lklHr8LxUALhUKhUFiBQ2Ui6jHItL03L5+qsU/lfU0WEC04+hCYp5IKPmYrsdLUvk/7Op2z1j5P5sGkRZisqFaBJ43M0kzXzNbWrzMcOdazvVefm5a3QXZLXxKfFZV/bhtZNq1MPzvvz/qh9E/7WXA7/ZV8t97eqoyZEYc+l14VlqNQ6J3u2HUPczMq9TLKcHwyZrP9jbV2E1viGE8+TzNQMs/k+ySrI+PkGEzKUc6KmR22s0Xt+b29jQNlNZVUI5eY6+s0ejqU3gwfv8McP7xOisxYg158J/dL64dBMdBCoVAoFFbgUCrcuQq9uXPVBmtk0uKTDuapZIwjWVWvErtZF6vTM++rLSszP1ulXpqB2k9I3yfbxwoOrMLg41g1wcs2Ntas1O+F8W9+VvTt+Jopu4ivxcxCfm9m3352SS3JKi30DZFxGrRu07Ns+2PKpJJyd861Ynsq3JcLerNLc5WznG2Y8pVRF8BzJyaafJ/M7JNy3PLdcVywri0zdlF9y37PnNCc9dhVoYaZx1Kf53eMWg7ux5m3nhK5lxu6N/PQ24/X3YVeHOhSHynPu2YMFwMtFAqFQmEFjjQX7lz0fKW2XlyphL4DKau5kq+gV6Weyj3ntjXztDXriibOdMMct/QLsl4irVr6dxmraYZpNbCzAZnh2hcrjVmPfC98zrROfU9kmFTfUvXK4xgfZyRfKBWDfAa+boof9bMz85iC20hlby8bUlqfi1daNRaOIzKWpUy/PYZslWM6xX8mDUXSYlBx6n7HMeh1jvF0r4y3Tqp2Muh2W8rdneqAknX72VA/wDan+MxetRWjl696rk90TWxyL8tR8ssexaxRMdBCoVAoFFbgRfGBcv+589+OebQVY9YnHbQmmNkn1X2kD4CqW6tszTzN/BznefXVV0saWZCZoP2Atl59XluhvnevMwYsVRKwX5Hszdc165RGS9nWa1IHkgHSN9qz6FNMGNW4VOzZKqZFzvqJKa9xqt6zK7tNyvS01j+f1o+jL3QNW07HUJPAPrErDjT5x9w/Uiwj+2nK2pN8nQYzZjHnbYqxTGw7jWWfh+1njc6pc5B5UrHM2rt8lqmtSVfQq/pCzM17nvabMx57WcPWqnIPg2KghUKhUCiswGIGOmWBJqs+WR89hR8tzYcffljSyAalg1VOkhIusTFmN6Ji1czP53e8JxkkMwX53lgbkFYs25v8Oj6f2SX9Na3lSNZMvwitUVrMjNXz72a7zDhEhTDvwfCzSFax4eN8PbaTvqLENKSDVv1cpnmY6vTHDUfpr01ZpdifuZza5vfKd8hcs6k/pPzXaQaM52f2MH7Pej43nocqc/bXKWVqT13L3M68BsG49IRehrdepq40TuZmINrVJ+deO6lwe8en/XahGGihUCgUCitwpD7QuT7Rngo3/d5WK7B/gpl/6OM0O6JqjerbZMU63tNqV+awtQVo1seYsdQuZj6iRck8nrTKvZ/ZuTT6gHwMq6QkZZzZMf2xvgcySoJ+F8aXnn/++fv2I/M1Q+Wz4LtLzGJKKdirR7jUB/pyzER0FJjr49oVB9jLOLSEuUnzK8EktsX+xG9HUh4nP2VSyqa41fY3Mkx+vzgmes/MSKwtMcPErnssMGHJ+OG+S2eJlua+LQZaKBQKhcIpxqEY6FIFYs93wP12MVJaXr1qAFR0koXRt2j2Y/ZEVRoZpFW4Bn2kzizE2oS0JJkRiZYorVvHq7bHMgsJFZLejxa3maDvhQrA5Mdglh+fx9eZyrTSgj4n+nuYy5e+2qn29arSJ2tzqXW7xIfzSsScsTyXxRJ8N9QFUJk6Nw6QcdKMN041M8k005IMlOvt/6nV4HdgKQubu0z+RO7HWau57C3Ff+66j7nMcanv8zCzTMVAC4VCoVBYgUNVY+kxz7nZK9acj3U0GaNIX+Fc/wkZKONIWSfUVebNGH2csyh56f1Y3Z5+Fmc2oq/U7aRV3PqF6dOhEvjkyZOSRmvWbfK1/Ez9DMxErYr1tX2dc845Z1+bqfBL2aDoDyZjZj9gHdDEVFtG0OtDvd97DDbhOPlG57S1p1eYe3xv3LXb0jFzZxXSeaneZexjO5ba6yc/I/tzqjNKnyn3m4rtJOPkTJTH4pSqedf2lO+X2cSY75dZlTiGk++zt/8a5rlWGT93//KBFgqFQqFwirGKgc5V36bMMWm/Xjxpezwtq7nqPyMpUK08pU/SzPPBBx+UdDDridvzsY99TJJ02223SZIeeughSdJ1110nSfrABz4gacyq5HhWt8e5dqn4s8X56KOPShpjPp0ZqW27nxszufha9sfec889+56J1x977LF992xWbcZ5+eWX73t23m5FNN9Bius0kgqzl7czxdROoec/Ocoq9S8HzM0o0zt+Dcud6wNNNXfZHzyWPIbcr804PRPjGHCP7TQTwhy5Hnfu5ydOnNi3ZP832N/bnNNU6LqtXHfb/T3wPfh3z4yZufpazOfLOsMe0ylTGrfTB8pvd5pdmDPO5sZ3zlUCH2VsdzHQQqFQKBRWYFUmorl5D42k3jSS1UK0FgNVtWaMzHlKJOWmLSj786gwpXrWVqgzFJnd2Sq1//ATn/iEpJG5Or/vDTfcIGmMM3U7bAFSKWs25vtjlRZptHhtXfpYK3VtnXpp69X37GfpmNeLL75Y0miNum0+v/fz776eLX23h/4W5iH2khYj/SX0SdF3ZCt66lw932Zv/56lfBzVt7vafFQMdO7+u86RvjdJGZoYifstY7RZIcnjgrHXHifMlcvYb9YGZta0NLPWqn3NHHvKYi/dRrbF8Dozo9H36e+Wl7xnztglNpd83vz+zokb9T6p3ufaWO65v+9CMdBCoVAoFFZgEQPdbDZ6/vnnFyvzEgNdmx1GOqhSs0VkUK1Ka5MML2XDIQM1qzJzdJUWs7NrrrlGknTrrbdKkt7//vdLGn2htgBt4Zkd+nhXg2GlBvstn3rqKUmjurf1m9hyZl5Ms14zT78Pt9XPzn5ZVsNw2/xefE1b2N7P2/0sbXkznpTPllap9+d5Wc3F79brLQNNzHCtNTr3uOPCRD2Wd/2+ZL23Pakx220pJpcq2Llxfh5r7hc+n9nY3HtgHHOKfUwKVPsj6dtPWcmkkYF6ab8t40N9TtYj9thL9XDpPyaD5b1y/57fMTFPKp/njBtqHpbGca4dy3NQDLRQKBQKhRVY7AOlBdGiZ9Hx2MQOjF4+Relg3U36CGyZ2YLjPHqqQk8VrhmgLTv7/5ypiP4+X8fXNdujFUoVHeNOacUa9k/6+BZkim6zmSTrJ9IHSebJGFvGfbbMr72+nz3j8MhAqdDzM/TvzEhEa5cW6hSSj+ao4jqPozp3DgNdq5BNNX93MVBmE+vFfyYWxVkpzmywTUlhSp8p/X693Lcpy9iuyjQG/aUeY70KNCkGtZfxhz7RudVYkq8zxcQm32dil+3/yab5ew9rZ512oRhooVAoFAorcKT1QGmN9H7v+ad6Fqg0+ghSVRZaPMwLy99tiTH/qmO5LrnkEkkj8zRLY+YiWre2FMnazGTNVFl9gda02+XjWyvJ1h7j2Azm3PS53Sb7QFONVeagpS+S9856nb4nt4/+aca8+nwG+wfPT4YxdSzXUxWKudboknjH0w1TDDQxzrk6hoTWVy/tf15UmvfYTi/u18eRMfL3HvOk2pbsjDNrqWYnf3e/T8v2/5x1SdmQUv7ruZVqUm5wo/ftJhLzTEraxC534VSPuTl9uxhooVAoFAorsFiF28aBGrRykrXaq/uXFF27fFz2AdoHaWZGHyT9eEZPCeqlmafVt1bRknnSqk0xrqzwQHVwstzoR5mqIZiykDBWln5bPl8+K/qN22u3bTT4XnuKPL4Dqi+57v3MFKYY6FoF3lyfadrvdEdS4faYZm9J8N0bbd+ao3Voz8FnnvoTxzCvR8bpJeNCObbTPaX2cQaGTJTx0O3/WUEp1QFNrDr5MomlPs5eP+j5ZBPT5H3MaWv6u3EY3+ZcFAMtFAqFQmEFjkSF25t3Tz7Qni+U8/Mt/JstM7OuNMdPFma2RlUs81K6jfaDMCNHigGjotTKVVqUrAloxmz2SAUf41RbJssMP7T2aHHzHtIy5aj1ef27ZwP4zH0dt8s+UIO+Ufub+Y7oS/I6a7GuwVzLPFnox4V5tpgay4lRpGpGyWearP4lzympcfkuEivjtXo1L3vMNN0DZ0w4Nt2vGVOe6oW250hq1t49JVbGNqcKSOmbnjIKEWmsGqnW8FT/WKpXSDgVTLQYaKFQKBQKK1B/QAuFQqFQWIEjSSbfo/tzp4WMlBaq3Y+ya6e2s+OfTvQ0RWt4f047OkzF64anW7w93RPFRDwPpycpFmL5IoaitMIDihg4BUUxBIVVdOCnxBeU5/NZMlSIgeKcvmHYjZNKeCrX7eE79JLJ8NvfjLlpvdam9zpuYSybzWbf++2JQnpjOk3lpXCFXeFwSWzGMZ+Qpi3TlG3PDcMpwzSVyOM4PtzP7abxkoIhKReZ791Lmspd+j7T/hyDKbFOL3l8ClHclUgh3Ssxd0xz/7S+C8VAC4VCoVBYgUMlk09WRi+dlEGnvzHH0qBAxtdyeS8nVmACAgoCGMTMMBgLVFjKi21mUgJa5BRCkYUxjZ7BYr/+3dasUwxKo9CJSSV8r7SIyThT2EoSMvDZuU28FxbzpciCia5p3TLxg9vjxA98Zu29Jcs9rXP7WkZ6uqPHQJNoqMdciF5o2tQ53C6mtUzJ23vvupc8gP2/x0S5neuJQbPfs0hCO5NDBkr0Eh6ktqUZLzLMdA+pX/Ad8HgjCbLmhEXNDVNJ6Iklud8cFAMtFAqFQmEFVvlAUxLlnrXSnqP9PSWeTgH+7T60SszU6HNkm5iM2fA1Ha5isK1kuPQnkmHyPKlcmhkl05ox6T2TP0sHk7rbt+JrkqmxZJHRC9pO1nFix7xX78c0jPQrM/SI1/M72mWtJszxvUyhF55xnJjpFANNSSyWzi71winad9ljS5ylmZOAfNd6j4H2QpZS8vhe0oO5afDmIH1De/04+TTTPfWYaWKkBhPTeJ1aELZ7zjef6IXy9M6zhokWAy0UCoVCYQVWMVAyzZRMmerLNE+egoN3+a9SYVb648jYDLMdWzpMrWdlp9tmNme2xHu3hZX8vH4WSTnm65hFuv2+3jnnnLPvfGZf7fmYlpDlx6gIpkXvttPXyPORHZP5+llbZWgFsdvhe/N5/XtiKb6e2291ruF34/NLy9nJ3O09q/S4JFTwOO75PHv6hqTC7CUraJ9Tz1eZ/Gxe9tS06ffEVJLCNCWxZzvc96jQTwkAuH977TQjMNe3P9dXPbcf8HvWi8BI45DJWHZhrY9z7nmW+lJbFAMtFAqFQmEFVqlwyQw4508m1PMP0TpKSc2nYoNSXJH9ayyKSzZF/6DvgenuzATpG02+op6/I8WL+vwugm02aDbH8kat/5I+Y1q4ZMlUB1JB7P3pJ2YaRZ/H/mcy0nQ9g+1kXCjb7XdiH+tnPvMZSdJP/MRPvHDOiy66aF9bkz9rrtXJ/Y8L00zYbDZ67rnnYr9dyjiM5POkNmAKc/13ibVwe1LXzk2wzuumsc7+628Hn6X7LWfmOD7bfZKuJKlzjZ46Nvk+5y6X+sQN6iCoj/D3r2X/qUycz+GZp158bg89Vj+FYqCFQqFQKKzAqmTyaX6e8UzMWjE3k4dBy661unjtlPHEWW28n5loKhpNlZjP5+PYNt+rz8PjetlKvJ3xUVQLOw6VyeZ3+U0MW7Z+HyzTlMpBJQs6Xc/ntU+S7zuVL6MPydsZm2tW7v7gZ/DZz35WknTxxRe/cG6Xn5sbn9jDXJZyXJhpYqB8p0uZUFK6GmRf7TFJM9FTUSafZy/Os+cbNXp9KKmDOe64PX27pq7B9Z7mYmlGorkxkkYvPjg9wxQDy/j7KZU244NT2bmlY/swCvpioIVCoVAorMBiH+hzzz13YE66FxdFJGUWLQDGTM7xgSaLzf45t7VVbErZikxWqpkhfa9U1NECp1+P7WURYJZBM8ua8iUxTpMZgJJqkUuqaMlUWVItsRjOIKQMM/Z7MLORz2vVLX28jC1rnwmzGs2d7eD6y0V1O4WWgSZlffKBTjHJFpw12uW/TP2wLe/VnoPr6Xhea+277ym4U39MhbrZ35lpq/0/mVovPjfNIBA933eaeeipeHusPOXd5jNsx3LvHGvVt73f54ztYqCFQqFQKKzAKgbas1LoE6V1kjKLJFa5yweafKEGz+ncsrbwnOuWftmkupybgcXbbW3Sz0N2biuV7TfbI3uzP5BWentuq1lTvkuqBZMFz/eUVHGMiWVh75QfmAWFfX3HvtrHkfzFF154oaQxN257DP2pxlw/V9o/zZYcF9AH2mOec7OMzY21a/t5GlOcUeG5egyU3x/O/vTuITGbnoq351ck65q6Ti++sqea7jFG3mtaJ3rZyOaqsL30OPV30stWb0Emz4LnnM1b48ucuoc5KAZaKBQKhcIKLGagP/zhD2dbpXOVXlPXmVqfyl6SlHhpO+t+Pvnkk5JGlSuZC1mXQfaW/LX02/EZkXnymTBGk2xsKp8pn4Hbbp8m1bQpxsvnM5P1+ax+5T2TibIdrIzDOol+BmaU9n3yGbHKy1VXXSVJOu+88164ZvI9c32pr3Mpcz1dwUxEPR9oUt/y/tNMzprnlJhd6l9LFdWc6eplRpvLro0U8zrnvshOEytnxANz2CafaIqY6C2TL7bnL+b3zt8gRzcwW9oUA+W3NOlMluIwMd7FQAuFQqFQWIFDxYH2/CEppy1BFWViqlN5ZNM+PWWd9zOLcT5Wxm6R6TELD+NKadXSr0dLkhVSeH4yXzLo1vrytXhuM0Zv936p2oqvbX+Er+HtrJZCXxPbSGbL7Ep+Nueff/7ks6Lv0/j6178uSbr++usljf7s9hi3/YknnpA0+mUvu+wySeP7SzmKE/NMFvdxYqS7cuEmv3UvJjJt3+Vrm6t67b2j5BdMSN8Exm1Svc7je8/uMH0iMc8ek0y+0KWqWl43xZrzuLSds0isCDWld6Gqn+f09yS9H/pMe37f8oEWCoVCoXCKcSgfaLL8jFTZnSwxxfel9fYcc5fJuvXcu32LZmusvmKm4jYw76+PT1mWyGgNMkn6FXkcra4WSf1KfwrjLMlcqVL078x+Qms4qWpp/ZrtG/Y/p5qnhhmxr2ff6Vvf+lZJ+7NF8Tn72N/7vd+TNGYqevvb3y5pzJ2b2K/RU50eFwbqsWzMzeE8d2aHStO0lA6O7zTue/U723ubWnIWicyG5+v5cXuMk37HFD+/K/sO275UjTtXrdvzeXJ7ivvsjQ9mD2JdY85eSQc1GIwr9/cuXdP7W1Ph78SuvMxzUQy0UCgUCoUVWOUDTdZPL54qxQSlLBTJAp36LTFNsmCyMW6/4oorJEkPPPCAJOmpp56SNMYk2lLidW0lkUXREmQ1Al7f5zNLI9Nltp7WarUl5rb4HLTg/J7MtsnsaOGxKguZq9vC+Du2kXmDHctK1mJLkQza7Xj00UclSZdeeqkk6cSJE/v2a//ve/K17Ce944471OLHf/zHJUmXX365dmEpIz2d8fzzz0eW1WOgc2Mc05ieyhqVGGZaT9m82M8ZBUBdAWdUkuoztSexxR678/jwuLUmQBrHlsduGv/J98lvci9TERknlfIpX3aq/ETlPKtNcZaJvtD2naZsWD3fuJFmBclE14zdYqCFQqFQKKzAqnqgS63V5POkhUd2t4uBzq20wP2T5Uy2ZZXmI488Ikl67LHHJI1WC+uCEqnaS/LH0LridluEroHpZXufVqXSmqTa1ev0Wfr5k2G6rb6mYYabsv0kf4lZIGv/0c9MP7StcS+vu+46SaMfu7Ugk4rUvs4vf/nLksYZhvvvv1/SyH7bmNJdWBM7djrAY7ldn1oaSUOQcp7OnVWaOiYxvV7VFapijZ5KlozU/Z7xhryHxHx4XqrR3a/JMr2c2kYGynjP5ONMKtqemtfXSRWa0rtJ32z6OhPL9zts9Q8po9RUjvQpsI32v/I9roknLQZaKBQKhcIKLPaBtgy0F//ZHiNlX0fPT7KLgfZUt2m/xERt1ZglUdHqXLq0CO1jM1tK1mmqqp5i1nw9+ytd33RK1csYUjNNVo7x7waZ36J8AAAC9ElEQVRVtykmllYpfTisG0qVW7LcbRF66eubobpdngUwi7Tvc+pZ09Kmwtht9O++5oMPPijpYGaqlHHluDHPFm3bE+NM/fiwuoa2b/RU94nlpjFuJI1GimGkj5THc6yyf5Pt0V/JpcfNlA/UzNO/pZjqufVamZ+c6z2FMJ9R8n2nWYJeRRWD+ot2X36POLZTG/ld9LNkJqNexqkpFAMtFAqFQmEFVvlAl1Yx4F/4lE2/x0Bb62apci9ZzGy72ZoZiJkoGYytRTNCW4pmolbtJgvdx7dWZ/tM/IztdzQTNVNiVp/23N7X+5i9mk3Z18h6hP7dlhrrfnrJ8/s8jO2iqtfPgMo7+zBpuft8ZoXe3xmLmP2p7Y/0AfkeXRfW7NbX8L3Yj/z4449LGlW5zEozN1bwdMacti5V3S6dVZralqqVcAajtzRSXlh+x5IP1aBvlO2mb46KVrLJXQw0MU+yrjTTkpSrnEVK+yXmmd5vyj29NIvUFPNNsab0A3P/1GbOAvb63S4UAy0UCoVCYQVWpWKYa2UnJexcazXFXLbbaJ0kK7WXX5fWp60UVo+nv5AszmpOq3hdWYSWIdW/tIponZqJmoHSvymN9TD93Ki25TH2s7gN9t96nTlzn3nmmX3PLNUmpc+IKmvW/jMLdGUcr588eXLf8c55S9XtVNYgvkczS8eQptgv3/tDDz0kaVTjWp1rLM0Je9zQy0e7VHW7Jg60p3Po6R3YP5iBKNW07OUB7l2P4ycparlsVbhkoGSOc7MdJWaZFMmpZi/Vs8nnmZhnmh1IMzpTswDJX8vMarw2285vLZnpEjVuMdBCoVAoFFZgWOKzGYbhCUlfO3XNKRReNrh6s9lc+FI3IqHGcqEwG3EsL/oDWigUCoVCYYuawi0UCoVCYQXqD2ihUCgUCitQf0ALhUKhUFiB+gNaKBQKhcIK1B/QQqFQKBRWoP6AFgqFQqGwAvUHtFAoFAqFFag/oIVCoVAorED9AS0UCoVCYQX+f/SQzae5rOJbAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 576x1152 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ft = 0.1*np.ones(10)[None]\n",
    "print(ft.shape)\n",
    "print(ft)\n",
    "\n",
    "res = show_conv(img, ft)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "AJfVXMiCO2pp"
   },
   "source": [
    "**Example answer:** The filter is a horizontal bar all containing all $0.1$s. Therefore I would expect a blurring in the horizontal direction, meaning the _vertical_ edges get blurred (because these are the ones that change rapidly in the horizontal direction). This seems to be happening in the result. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1-z9WROaO2pr"
   },
   "source": [
    "#### 1(a)\n",
    "rubric={reasoning:5}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "5nUPLjfCO2pr",
    "outputId": "e9fbcee2-ace6-42aa-fd71-6f6fa162069d"
   },
   "outputs": [],
   "source": [
    "ft = 0.1*np.ones(10)[:,None]\n",
    "print(ft.shape)\n",
    "print(ft)\n",
    "\n",
    "res = show_conv(img, ft)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xfGCG29BO2pv"
   },
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LtPIw0w-O2pw"
   },
   "source": [
    "#### 1(b)\n",
    "rubric={reasoning:5}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ZAhTDb23O2py",
    "outputId": "601583c5-ecba-4bf3-fec9-e28d9b70d645"
   },
   "outputs": [],
   "source": [
    "ft = np.zeros((5,5))\n",
    "ft[2,2] = 1\n",
    "print(ft.shape)\n",
    "print(ft)\n",
    "res = show_conv(img, ft)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "OJ1q0EWxO2p1"
   },
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "32tx26E6O2p1"
   },
   "source": [
    "#### 1(c)\n",
    "rubric={reasoning:5}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Lj1wDS7yO2p3",
    "outputId": "cde75fe3-7732-4499-8eeb-228416439f93"
   },
   "outputs": [],
   "source": [
    "ft = 0.01*np.ones((10,10))\n",
    "print(ft.shape)\n",
    "res = show_conv(img, ft)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jJjpLeB-O2p8"
   },
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "32K6eYDKO2p8"
   },
   "source": [
    "#### 1(d)\n",
    "rubric={reasoning:5}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "zkjVbsI7O2p9",
    "outputId": "9fffe31d-7f36-449f-9964-9f1aae4ec50e"
   },
   "outputs": [],
   "source": [
    "ft = -np.ones((3,3))/8\n",
    "ft[1,1] = 1\n",
    "print(ft.shape)\n",
    "print(ft)\n",
    "res6 = show_conv(img, ft)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zMe1BA1BO2qF"
   },
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "sObjn7S9O2qG"
   },
   "source": [
    "#### (optional) 1(e)\n",
    "rubric={reasoning:1}\n",
    "\n",
    "Earlier in this course we talked about gradients and numerical differentiation. Think about part (d) above: does this have anything to do with the topics from earlier on? Can you relate these edge detection operations to \"derivatives\" or \"gradients\"?\n",
    "\n",
    "Also, by the way, back in the seam carving lab of DSCI 512 we gave you a function that calculated the \"energy\" of an image, and we then looked for low energy seams. Here's the code we gave you:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "cEUGRajqO2qI"
   },
   "outputs": [],
   "source": [
    "from scipy.ndimage.filters import convolve\n",
    "\n",
    "def energy(image): \n",
    "    dy = np.array([-1, 0, 1])[:,None,None]\n",
    "    dx = np.array([-1, 0, 1])[None,:,None]\n",
    "    energy_img = convolve(image, dx)**2 + convolve(image, dy)**2\n",
    "    return np.sum(energy_img, axis=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "h4anrPVxO2qL"
   },
   "source": [
    "(There's no particular reason I switched from [`scipy.ndimage.filters.convolve`](https://docs.scipy.org/doc/scipy-0.16.1/reference/generated/scipy.ndimage.filters.convolve.html) to [`scipy.signal.convolve2d`](https://docs.scipy.org/doc/scipy-0.18.1/reference/generated/scipy.signal.convolve2d.html); they perform the same function for our purposes.) I thought you might enjoy looking back at this formerly mysterious code with your newfound knowledge. And it's also a bit of a hint: the seam carving energy function looked for \"edges\" or \"changes\" or ... derivatives! The above actually calculates the magnitude squared of the \"gradient\" at every point. The whole thing should make sense now as well -- when seam carving we wanted to remove pixels for which there wasn't much going on in the immediate vicinity."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "nvsR_W_tO2qN"
   },
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "BYYZaUzyO2qN"
   },
   "source": [
    "## Exercise 2. Convolutional networks for MNIST\n",
    "\n",
    "Sorry to continue with MNIST so long. It's just _THE_ classic data set for this stuff.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gg208bROO2qN"
   },
   "source": [
    "Below is some code that trains a convnet on MNIST. The code is adapted from the book [Deep Learning with Python](https://machinelearningmastery.com/deep-learning-with-python/) with permission from the author. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "vc8-vt_WO2qQ"
   },
   "outputs": [],
   "source": [
    "# load data\n",
    "(X_train, y_train), (X_test, y_test) = mnist.load_data()\n",
    "\n",
    "# reshape to be [samples][channels][width][height]\n",
    "X_train = X_train.reshape(X_train.shape[0], 28, 28, 1).astype('float32')\n",
    "X_test = X_test.reshape(X_test.shape[0], 28, 28, 1).astype('float32')\n",
    "\n",
    "# normalize inputs from 0-255 to 0-1\n",
    "X_train = X_train / 255\n",
    "X_test = X_test / 255\n",
    "\n",
    "# one hot encode outputs\n",
    "y_train = utils.to_categorical(y_train)\n",
    "y_test = utils.to_categorical(y_test)\n",
    "num_classes = y_test.shape[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "NpcpMxhxO2qT"
   },
   "outputs": [],
   "source": [
    "# take a subset of the data for speed\n",
    "subset_size = 10000\n",
    "X_train = X_train[:subset_size]\n",
    "y_train = y_train[:subset_size]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "LodO4nxkO2qV"
   },
   "outputs": [],
   "source": [
    "# define a simple CNN model\n",
    "def build_mnist_CNN():\n",
    "    mnist_model = Sequential()\n",
    "    mnist_model.add(Conv2D(32, (5, 5), input_shape=(28, 28, 1), activation='relu'))\n",
    "    mnist_model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "    mnist_model.add(Dropout(0.2))\n",
    "    mnist_model.add(Flatten())\n",
    "    mnist_model.add(Dense(128, activation='relu'))\n",
    "    mnist_model.add(Dense(num_classes, activation='softmax'))\n",
    "\n",
    "    # Compile model\n",
    "    mnist_model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "    return mnist_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mnist_model = build_mnist_cnn()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "MXvKbAJ7O2qW",
    "outputId": "e197f9bf-d87d-44e8-c54a-e55c4dfc0b7f",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Fit the model\n",
    "mnist_model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=10, batch_size=256)\n",
    "# Final evaluation of the model\n",
    "scores = mnist_model.evaluate(X_test, y_test, verbose=0)\n",
    "print(\"CNN Error: %.2f%%\" % (100-scores[1]*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "D4aYezc9O2qZ"
   },
   "source": [
    "#### 2(a)\n",
    "rubric={reasoning:15}\n",
    "\n",
    "Run the code above. How does it compare to your fully-connected (\"Dense\") neural net from lab 3? Discuss in 2-3 sentences. (Keep in mind that here we're only using a subset of the training data for speed.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### (optional) 2(b)\n",
    "rubric={reasoning:1}\n",
    "\n",
    "Let's assess what happens if we permute the rows of the images (both both training and testing). Below we permute the images, retrain the network, and re-evaluate the network. The accuracy is now lower. But we used the same data, just shuffled - can you explain why this operation hurt the accuracy?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "perm = np.random.permutation(X_train.shape[1])\n",
    "perm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_plots = 3\n",
    "\n",
    "for i in range(n_plots):\n",
    "    ind = np.random.randint(X_train.shape[0])\n",
    "    plt.subplot(2,2,1)\n",
    "    plt.imshow(X_train[ind,...,0], cmap='gray');\n",
    "    plt.title(\"Original\");\n",
    "    plt.subplot(2,2,2)\n",
    "    plt.imshow(X_train[ind,perm,:,0], cmap='gray');\n",
    "    plt.title(\"Permuted\");\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above: this is what a permuted training example looks like, with its rows shuffled."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mnist_model_perm = build_mnist_CNN()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fit the model\n",
    "mnist_model_perm.fit(X_train[:,perm], y_train, validation_data=(X_test[:,perm], y_test), epochs=10, batch_size=256)\n",
    "# Final evaluation of the model\n",
    "scores = mnist_model_perm.evaluate(X_test[:,perm], y_test, verbose=0)\n",
    "print(\"CNN Error: %.2f%%\" % (100-scores[1]*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "91yxVPKkO2qa"
   },
   "source": [
    "#### 2(c)\n",
    "rubric={reasoning:30}\n",
    "\n",
    "You will now deploy Keras/TensorFlow on the cloud using [Kaggle Kernels](https://www.kaggle.com/kernels). This will allow you to train on a GPU and assess the benefits of training neural networks on GPUs. Kaggle Kernels offers 30 hours of free GPU usage per account. This should be much more than adequate for this lab.\n",
    "\n",
    "Note: last year we used [Google Colab](https://colab.research.google.com/) instead of Kaggle Kernels. That would have been fine for this exercise - they are roughly equivalent. But later in the lab, when we want to access a Kaggle dataset, Kaggle Kernels are way more convenient! (Furthermore... two years ago we used Amazon AWS and that was truly a huge hassle because they wouldn't recognize your @alumni.ubc.ca email addresses as \"student email addresses\".)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "-evt3v1rO2qa"
   },
   "source": [
    "Follow these steps:\n",
    "\n",
    "1. Save this Jupyter notebook so that it contains your latest work. Also push it to GitHub to be safe.\n",
    "2. Go to https://www.kaggle.com/kernels\n",
    "3. Make an account if you don't have one\n",
    "4. Select New Notebook\n",
    "7. Create\n",
    "8. File->Upload notebook\n",
    "9. Upload this notebook itself, lab4.ipynb, which you just saved.\n",
    "5. On the right-hand side, go to Settings.\n",
    "1. Make sure Internet is enabled.\n",
    "1. Make sure GPU is enabled.\n",
    "\n",
    "**SUGGESTION:** once you're done all your work on Kaggle (which means this exercise and the next one), you can download the notebook from Kaggle and overwrite your local version. That way any work you did on Kaggle won't be lost. (They allow working directly off a notebook on GitHub, but that feature won't work for us since we're using github.ubc.ca.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pmF2S-UoO2qb"
   },
   "source": [
    "Now, run the same MNIST experiment as above but on a Kaggle Kernel with the GPU active. \n",
    "\n",
    "1. How much faster is it (as a ratio) to run the exact same code on the GPU vs. your laptop? \n",
    "2. Notice the code above takes a subset of 10,000 training examples for speed. With the speed of the GPU, you should now use the full 60,000 training examples on AWS. Report your performance after 10 epochs when training on the full data set. How does it compare to the validation error you were able to get on your local machine (which presumably required using the smaller training set to run in reasonable time)? \n",
    "3. Again, compare to the fully connected network from lab 3."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kraMgjwZO2qc"
   },
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kXIVN3thVXPh"
   },
   "source": [
    "## Exercise 3: Transfer learning\n",
    "\n",
    "In this exercise we will work on the concept of _transfer learning_, in which we'll use a model trained on one task as a starting point for learning to perform another task. \n",
    "\n",
    "A natural question is, why is transfer learning helpful to us? Why can't we just train a model with the second task's objectives from the beginning? \n",
    "A key motivation is the difficulty in obtaining labeled data: ususally we need a whole lot of data in order to solve complex problems, and it can be hard to collect the data. (Another motivation is the time and effort -- both computational time and human time -- needed to train the original model. Someone did the work already, so we don't have to.)\n",
    "\n",
    "In this exercise we'll apply transfer learning to fine-grained image classification. Here, the goal is to recognize different subclasses within a higher-level class. In our case, the higher level question could be, \"Is this a dog?\" and the fine-grained question is, \"What breed of dog is this?\"\n",
    "\n",
    "We will use the [Kaggle dog breed identification](https://www.kaggle.com/c/dog-breed-identification/data) dataset. \n",
    "In the dataset, each dog image is labeled according to one of 120 breeds. We'll start with a pre-trained model that was already trained on a more high-level image classification task, namely the famous [ImageNet dataset](http://www.image-net.org/). You can see some sample ImageNet images [here](http://image-net.org/explore?wnid=n04516672).\n",
    "\n",
    "We'll consider three approaches to the dog breed classification problem:\n",
    "\n",
    "1. No transfer learning, just end-to-end training of a CNN directly for the dog breed classification task.\n",
    "2. Use the pre-trained model as a feature extractor; then, add new layers in order to train it with the dog-breed dataset.\n",
    "3. Some fine tuning of the weights of the pre-trained model (instead of freezing them all and only adding new layers, is in approach 2).\n",
    "\n",
    "Attribution: In designing this exercise, we took some inspiration from [here](https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html). But I think our version is more interesting because the classes in our new task are not part of the original task."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "X080337PVXQg"
   },
   "source": [
    "### Preliminaries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "z1zEyEZ3O2qf"
   },
   "source": [
    "I am assuming you already have your Kaggle Kernel set up as in the previous exercise, with the GPU and Internet enabled. Next, you will need to add the dataset to your Kaggle Kernel. (FYI: this is the part that is so much easier with Kaggle Kernels than Google Colab, where we had to install the Kaggle API on the Colab instance, set up key-based authentication, and then upload many GB worth of data from one cloud to the other, which turned out to work fine on ubcsecure wifi but not on eduroam wifi... lessons learned!)\n",
    "\n",
    "- Go to https://www.kaggle.com/c/dog-breed-identification/rules, make sure you're signed in to Kaggle, and accept the competition rules.\n",
    "- In your Kaggle Kernel, press \"+ Add Data\" at the upper-right.\n",
    "- From the tabs at the top, select \"Competition Data\" (do not skip this step!)\n",
    "- Search for \"dog breed identification\" in the search box. It should be the first result.\n",
    "- Press \"Add\". Note: this will cause your kernel to restart.\n",
    "- When asked if you want code to load the data, you can select \"No\" - I already have the code for you in this notebook, below.\n",
    "\n",
    "### What you should do\n",
    "\n",
    "As with the previous exercise, you should do this on the GPU on Kaggle. Your task for now is to read along and, **whenever there are code cells below, you should run them as you go along.** There will be some questions interspersed in the document, **which you should answer**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XQEDFbi5O2qp"
   },
   "source": [
    "Next, we take only the first 2000 samples from the original dataset. We want to simulate having only a small labeled dataset available to us, and see the effect of transfer learning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "id": "F3uwOqqNVXQl",
    "outputId": "d94e6d24-0181-4d29-9fe1-33650217167a"
   },
   "outputs": [],
   "source": [
    "data = pd.read_csv('../input/dog-breed-identification/labels.csv')\n",
    "data = data[:2000]\n",
    "data['image_path'] = data.apply( lambda row: (os.path.join(\"../input/dog-breed-identification/train\", row[\"id\"] + \".jpg\") ), axis=1)\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SWFmLeZtO2qv"
   },
   "source": [
    "Above: you can see some of the breeds that we're predicting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "Hw0kE6BgVXQr",
    "outputId": "df467e37-9ac0-4727-d561-73edb636ea4e",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "target_labels = data['breed']\n",
    "total_classes = len(set(target_labels))\n",
    "print(\"number of dog breeds:\", total_classes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "xTCKP_rhVXQ5"
   },
   "outputs": [],
   "source": [
    "# read images from the image directory. \n",
    "images = np.array([img_to_array(\n",
    "                    load_img(img, target_size=(256,256))\n",
    "                    ) for img in data['image_path'].values.tolist()])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "EtOYBdEiVXQ9",
    "outputId": "d068b3a0-733c-4873-8fe6-674c7dfa2668"
   },
   "outputs": [],
   "source": [
    "images.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "bkPvTMP1O2q5"
   },
   "source": [
    "Above: we have 2000 images, each of size $256 \\times 256$ and with 3 colour channels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "XXvcDjzKO2q6"
   },
   "outputs": [],
   "source": [
    "images = images.astype('float32')/255.0 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "T-OqM5tQO2q8"
   },
   "source": [
    "Above: it's very important to scale the images!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 263
    },
    "colab_type": "code",
    "id": "rkbauZ2CO2q9",
    "outputId": "46a895d1-47a2-4a9c-8efe-f07afefc752a"
   },
   "outputs": [],
   "source": [
    "plt.imshow(images[0]);\n",
    "plt.grid(True);\n",
    "plt.xticks([]);\n",
    "plt.yticks([]);\n",
    "plt.title(\"Breed = \" + target_labels[0]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "c49AJfCuO2q_"
   },
   "source": [
    "Above: this is a sample image from the dog breed data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "r8s5GlKHVXRF"
   },
   "outputs": [],
   "source": [
    "X_train, X_valid, y_train, y_valid = train_test_split(images, target_labels, \n",
    "                                                    stratify=np.array(target_labels), \n",
    "                                                    random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "4AxPhnaUVXRI",
    "outputId": "ce3d1213-8f17-4aa5-953d-b88ecea3f0b4"
   },
   "outputs": [],
   "source": [
    "print(X_train.shape)\n",
    "print(X_valid.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3(a)\n",
    "rubric={reasoning:10}\n",
    "\n",
    "Before we start, do some EDA to assess whether there is serious class imbalance in the training data. What training accuracy would you get with `DummyClassifier`? Briefly discuss your results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gpQLM_QdO2rE"
   },
   "source": [
    "#### 3(b)\n",
    "rubric={reasoning:5}\n",
    "\n",
    "How many training examples do we have per breed of dog, roughly? In the context of other classification tasks we've done in MDS, do you consider this to be a lot or a little?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "CCL5JmShO2rF"
   },
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "uOcbM1boO2rG"
   },
   "source": [
    "Next, we do the one-hot encoding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "cy0xw3hVVXRM",
    "outputId": "9c4b368b-fcfc-4ef5-c74b-4cc45995cd44"
   },
   "outputs": [],
   "source": [
    "Y_train = pd.get_dummies(y_train.reset_index(drop=True)).values\n",
    "Y_valid = pd.get_dummies(y_valid.reset_index(drop=True)).values\n",
    "\n",
    "print(Y_train.shape)\n",
    "print(Y_valid.shape)\n",
    "\n",
    "# Note: it would be better to use keras.utils.to_categorical, or something else like that,\n",
    "# just in case one of the classes is absent in one of the two sets.\n",
    "# But this works for now."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0oMSfTq8VXRd"
   },
   "source": [
    "### Approach 1\n",
    "\n",
    "Now, we try Approach 1, which is training an end-to-end CNN on the dog breed classification task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "nyo9AZskVXRe"
   },
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(Conv2D(32, (3, 3), input_shape=(256, 256, 3)))\n",
    "model.add(Activation('relu')) # this is just different syntax for specifying the activation function\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "\n",
    "model.add(Conv2D(32, (3, 3)))\n",
    "model.add(Activation('relu'))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "\n",
    "model.add(Conv2D(64, (3, 3)))\n",
    "model.add(Activation('relu'))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 646
    },
    "colab_type": "code",
    "id": "UEprOsdpVXRh",
    "outputId": "afc2f8fc-4577-43ea-a6ee-8b125f029d33"
   },
   "outputs": [],
   "source": [
    "model.add(Flatten())\n",
    "model.add(Dense(64))\n",
    "model.add(Activation('relu'))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(total_classes))\n",
    "model.add(Activation('softmax'))\n",
    "\n",
    "model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "both",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 374
    },
    "colab_type": "code",
    "id": "4MWU2eWTVXRn",
    "outputId": "2cedd7a6-df25-4b03-b6d5-3899e557044c",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "history = model.fit(X_train, Y_train, epochs=10, validation_data=(X_valid, Y_valid))\n",
    "\n",
    "# FYI: it's often a good idea to save your weights after training or during training.\n",
    "# But you don't have to here.\n",
    "# model.save_weights('my_conv_net.h5')  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "nzNnV6IjO2rX"
   },
   "source": [
    "#### 3(c)\n",
    "rubric={reasoning:1}\n",
    "\n",
    "What do you think of the results? Are you impressed? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8BSLsVLXVXRr"
   },
   "source": [
    "### Approach 2\n",
    "\n",
    "Here we load a pre-trained model and add some layers on top. The syntax is not what you're used to - that's OK, don't worry about it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "JAebsZBqVXRt"
   },
   "outputs": [],
   "source": [
    "# Get the InceptionV3 model trained on the ImageNet data set\n",
    "base_inception = InceptionV3(weights='imagenet', include_top=False, input_shape=(256, 256, 3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_ACKgXs7O2rf"
   },
   "source": [
    "Note the `include_top=False`. This throws away the last layer. It wasn't useful to us anyway. ImageNet has 1000 classes, but we're not interested in those classes. Another way to think of it is that the original model is a crazy feature extractor plus logistic regression for the 1000 ImageNet classes. We are using the feature extractor and discarding the logistic regression part."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "IJnDwXjjVXR4"
   },
   "outputs": [],
   "source": [
    "top_block = base_inception.output\n",
    "top_block = GlobalAveragePooling2D()(top_block) # pool over height/width to reduce number of parameters\n",
    "top_block = Dense(256, activation='relu')(top_block) # add a Dense layer\n",
    "predictions = Dense(total_classes, activation='softmax')(top_block) # add another Dense layer\n",
    "\n",
    "model_transfer = Model(inputs=base_inception.input, outputs=predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above: the syntax is not what you're used to - that's OK, don't worry about it.  If you want to know more, see [this documentation](https://keras.io/models/model/). However, at a high level we're grabbing the base model, doing some pooling, and then adding two new dense layers at the top."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "txHFPz5eVXR7"
   },
   "outputs": [],
   "source": [
    "for layer in base_inception.layers:\n",
    "    layer.trainable = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "uqftzeBcO2rl"
   },
   "source": [
    "Above: this is a key step. We \"freeze\" the layers of the base model, so that only our two new Dense layers at the top are trainable. That means we only update the weights in the new top layers - all the other weights (the ones from the base model) are fixed (\"frozen\") during training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hoqUIzlRVXR-",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "model_transfer.compile(Adam(lr=.001), loss='categorical_crossentropy', metrics=['accuracy']) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 11407
    },
    "colab_type": "code",
    "id": "UF5F_PJNO2rn",
    "outputId": "78aae16d-0755-4989-d7e5-de2a08beee12"
   },
   "outputs": [],
   "source": [
    "model_transfer.summary() # run me if you dare"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Z3YFYrIGO2rq"
   },
   "source": [
    "Above: that's a lot of layers!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 374
    },
    "colab_type": "code",
    "id": "u7xbyNuXO2rr",
    "outputId": "215bec81-f6b0-42db-b9b6-3594b236f177"
   },
   "outputs": [],
   "source": [
    "history = model_transfer.fit(X_train, Y_train, validation_data=(X_valid, Y_valid), epochs=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "oQBtEHkh517q"
   },
   "source": [
    "#### 3(d)\n",
    "rubric={reasoning:1}\n",
    "\n",
    "How does this result compare to the \"from scratch\" CNN?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "PLeqHLO3VXSB"
   },
   "source": [
    "### Approach 3\n",
    "\n",
    "Below, we un-freeze the last \"15\" layers, which is really only the last one or two layers, since the list of Keras layer objects doesn't really correspond to our idea of a layer (see `model.summary()`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "BlBgaXPlVXSL"
   },
   "outputs": [],
   "source": [
    "for i, layer in enumerate(reversed(model_transfer.layers)):\n",
    "    layer.trainable = True\n",
    "#     print(layer)\n",
    "    if i > 15:\n",
    "        break\n",
    "\n",
    "# compile the model with a SGD/momentum optimizer and a very slow learning rate.\n",
    "model_transfer.compile(loss='categorical_crossentropy',\n",
    "              optimizer=SGD(lr=1e-4, momentum=0.9),\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 374
    },
    "colab_type": "code",
    "id": "0pkUjUCzVXSO",
    "outputId": "676a2f96-7ff6-4780-b278-f6dc7762c97f"
   },
   "outputs": [],
   "source": [
    "# fine-tune the unfrozen layers\n",
    "history = model_transfer.fit(X_train, Y_train, validation_data=(X_valid, Y_valid), epochs=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "D5sIpWhhO2ry"
   },
   "source": [
    "#### (optional) 3(e)\n",
    "rubric={reasoning:1}\n",
    "\n",
    "Un-freezing some of the layers seems to have a small effect here. Was it actually useful at all, or could we have achieved the same results by just training our top layers for more epochs?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Ta0BOhAoO2rz"
   },
   "source": [
    "#### 3(f)\n",
    "rubric={reasoning:5}\n",
    "\n",
    "In Lab 3 we noticed that unlike scikit-learn's `fit`, Keras's `fit` doesn't re-initialize the weights, but rather continues on from where you were. In the above code, we benefitted from this. Briefly describe how/why this behaviour was useful to us."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SkatY2bAO2rz"
   },
   "source": [
    "#### 3(g)\n",
    "rubric={reasoning:10}\n",
    "\n",
    "Brainstorm 3 other applications of this type of transfer learning, where you use a pre-trained network plus some modifications. In each case, what is the original task and what is the new task? (It's OK if you don't actually have access to a pre-trained network to do the original task; we're just brainstorming here.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### (optional) 3(h)\n",
    "rubric={reasoning:3}\n",
    "\n",
    "There are two perspectives on what we did in Approach 2: one is that we froze most of the layers and just fine-tuned the last layers. The other perspective is that we used a pre-trained feature extractor and then just used a simple model on top. In the above we added 2 layers on top, but if we added just one layer on top then it would just be a softmax logistic regression. Following this second perspective, can you get reasonable results by chaining together the feature extractor and a multi-class scikit-learn `LogisticRegression`? Perhaps this would be a good use case for a scikit-learn `Pipeline`? \n",
    "\n",
    "WARNING: I have not tried this myself, so there is a chance things will go wrong. If you get something to work, please let me know - I'm curious!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kqeZ-NB1O2r0"
   },
   "source": [
    "(You are now done with your Kaggle Kernel. If you were editing the file there, you should download it to your local machine before closing the Kaggle Kernel!)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "M-ArOMhhO2r0"
   },
   "source": [
    "## Exercise 4: Pondering\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "trsCrShnO2r1"
   },
   "source": [
    "#### 4(a) \n",
    "rubric={reasoning:10}\n",
    "\n",
    "When we previously worked on the handwritten digits dataset, we did something quite silly: we \"flattened\" images into vectors; for example $28\\times 28$ MNIST images became vectors of length $28\\times 28 = 784$. This is arguably insane! One reason it's insane is that we were completely discarding the \"spatial information\" contained in the image and just pretended we had 784 different features, whereas convnets preserve the 2D structure and take 2D convolutions. But there is another, related reason it's a bad idea to just flatten the images... what would go wrong if we tried to use fully connected nets on larger images, like $1000 \\times 1000$ pixels?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "uVPiQ2EzO2r2"
   },
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QX7W97a-O2r2"
   },
   "source": [
    "#### 4(b)\n",
    "rubric={reasoning:10}\n",
    "\n",
    "For each of the following, would increasing this quantity typically increase, decrease, or have no effect on the number of parameters of the model? \n",
    "\n",
    "1. Dropout probability, e.g. `0.2`\n",
    "2. Filter size, e.g. `(5,5)`\n",
    "3. Number of filters, e.g. `32`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1Xd4M7VpO2r3"
   },
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "DctVN-ITO2r3"
   },
   "source": [
    "#### 4(c)\n",
    "rubric={reasoning:10}\n",
    "\n",
    "For each of the following, would increasing this quantity typically increase, decrease, or have no effect on the training error? No need to explain. \n",
    "\n",
    "1. Dropout probability, e.g. `0.2`\n",
    "2. Filter size, e.g. `(5,5)`\n",
    "3. Number of filter, e.g. `32`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "3_u1ue4hO2r3"
   },
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vt94u1XIO2r4"
   },
   "source": [
    "#### 4(d)\n",
    "rubric={reasoning:15}\n",
    "\n",
    "What are the pros/cons of neural nets, vs. approaches previously learned (for both regression and classification)? Choose one method from a previous course (561, 571, 573) and compare it with what you've done in deep learning. Write a paragraph summarizing the results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_6TdwOcvO2r4"
   },
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "sQMacCZUO2r5"
   },
   "source": [
    "-----------------\n",
    "\n",
    "All the rest are optional; if you want to be done, you're done!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "lErYdgUOO2r5"
   },
   "source": [
    "#### (optional) 4(e) \n",
    "rubric={reasoning:1}\n",
    "\n",
    "The code below shows that the MNIST model from Exercise 2 has 592,074 parameters. Explain where this number comes from by going layer by layer and accounting for all the parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "OkGOzmdaO2r6"
   },
   "outputs": [],
   "source": [
    "mnist_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "b-HmSzHoO2r8"
   },
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Oe-tlfm_O2r_"
   },
   "source": [
    "#### (optional) 4(f)\n",
    "rubric={reasoning:1}\n",
    "\n",
    "Consider this CNN architecture:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "M878lE8ZO2sA"
   },
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(Conv2D(32, (5, 5), input_shape=(28, 28, 1), activation='relu'))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model.add(Conv2D(32, (5, 5), activation='relu'))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model.add(Flatten())\n",
    "model.add(Dense(num_classes, activation='softmax'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Kh5iX2q8O2sC"
   },
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "C5WSKdm5O2sD"
   },
   "source": [
    "Now, we remove (comment out) pooling from the _first_ convolutional layer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "R7c6aCvzO2sD"
   },
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(Conv2D(32, (5, 5), input_shape=(28, 28, 1), activation='relu'))\n",
    "# model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model.add(Conv2D(32, (5, 5), activation='relu'))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model.add(Flatten())\n",
    "model.add(Dense(num_classes, activation='softmax'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "QYcZDdofO2sH"
   },
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Xbf-pTMeO2sJ"
   },
   "source": [
    "Why does this change increase the number of parameters in the 3rd (Dense) layer, but not in the 2nd (Conv2D) layer?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mWwo0M-MO2sJ"
   },
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "-dL3QrirO2sJ"
   },
   "source": [
    "#### (optional) 4(g) \n",
    "rubric={reasoning:1}\n",
    "\n",
    "In the code above, the data is transformed to `float32` type. In lecture we discussed floating point representations. The main advantage of using 32-bit floating point numbers (versus 64-bit) is computational speed, but there's a disadvantage in terms of accuracy. Think about dropout and what it does/accomplishes. Did thinking about dropout alleviate your concerns about the potential pitfalls of using a smaller floating point represntation? Briefly discuss."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "USR2nNu9O2sL"
   },
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "H9DAdP6PO2sL"
   },
   "source": [
    "#### (optional) 4(h)\n",
    "rubric={reasoning:1}\n",
    "\n",
    "If you had access to 1000 GPUs, do you think you could get 1000x performance? If not, why? What are the limitations/bottlenecks?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pAWXF-Z1O2sM"
   },
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (optional) Exercise 5: setting priorities and time-management skills\n",
    "rubric={reasoning:100}\n",
    "\n",
    "_Rationale: admission into the MDS program is very competitive. This is great for me, because I enjoy working with such motivated and talented individuals. However, I conjecture that this same admissions process may also select for people with unrealistically high expectations of themselves. For example, I have noticed some students feel they must do all the optional questions even if they aren't particularly interested in the topic or don't really have the time to do them. If that sounds like you, this optional exercise was created for you! My hope is that none of you attempts, let alone completes, this silly exercise. In skipping it, you will need to forego the 100 bonus points.$^*$ I hope that by doing so, you will feel that it's perfectly fine to triage and skip certain lab questions: nothing bad happens! I believe this skill --- or perhaps you could call it a mindset --- will be extremely important for your success and wellbeing in the long-term. Doing everything perfectly is simply not possible forever, and when that time comes, it is important that you can set priorities. I have seen many people be ineffective at their jobs because they cannot skip the unimportant things. And now, the task..._\n",
    "\n",
    "Do the following before the lab deadline:\n",
    "\n",
    "- 400 push-ups\n",
    "- 300 sit-ups\n",
    "- recite the alphabet backwards 200 times\n",
    "- memorize the names of 100 countries\n",
    "- ask someone the following question: \"What is $\\int_1^\\text{cabin} \\frac{1}{x}\\text{d}x$?\"\n",
    "\n",
    "You must include, with your submission, incontrovertible evidence that you did each of these things. Your word of honour will not be sufficient. \n",
    "\n",
    "$^*$_Also, please note that all lab grades in MDS are capped at 100%. That is, you can increase your grade up to 100% with optional questions, but not beyond 100%._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### (optional) 5(b)\n",
    "rubric={reasoning:1}\n",
    "\n",
    "If you completed the first part of this exercise, write a brief reflection in 2-3 sentences. What did you learn about setting priorities and time-management skills?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "name": "lab4.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}