Notebooks/ResNet-on-CIFAR10.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Trains a ResNet on the CIFAR10 dataset.\n",
    "### Dr. Tirthajyoti Sarkar, Fremont, CA\n",
    "\n",
    "* **ResNet paper 1**: https://arxiv.org/pdf/1512.03385.pdf (**a**)\n",
    "* **ResNet paper 2**: https://arxiv.org/pdf/1603.05027.pdf (**b**)\n",
    "\n",
    "### About CIFAR-10 dataset \n",
    "\n",
    "Website: https://www.cs.toronto.edu/~kriz/cifar.html\n",
    "\n",
    "The CIFAR-10 dataset consists of 60000 32x32 colour images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images. \n",
    "\n",
    "The dataset is divided into five training batches and one test batch, each with 10000 images. The test batch contains exactly 1000 randomly-selected images from each class. The training batches contain the remaining images in random order, but some training batches may contain more images from one class than another. Between them, the training batches contain exactly 5000 images from each class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import keras\n",
    "from keras.layers import Dense, Conv2D, BatchNormalization, Activation\n",
    "from keras.layers import AveragePooling2D, Input, Flatten\n",
    "from keras.optimizers import Adam\n",
    "from keras.callbacks import ModelCheckpoint, LearningRateScheduler\n",
    "from keras.callbacks import ReduceLROnPlateau\n",
    "from keras.preprocessing.image import ImageDataGenerator\n",
    "from keras.regularizers import l2\n",
    "from keras import backend as K\n",
    "from keras.models import Model\n",
    "from keras.datasets import cifar10\n",
    "import numpy as np\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_augmentation = True\n",
    "num_classes = 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Subtracting pixel mean improves accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "subtract_pixel_mean = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How many layers? Depends on the model version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def model_depth(version,n=3):\n",
    "    \"\"\"\n",
    "    Computes depth from supplied model parameter n\n",
    "    \"\"\"\n",
    "    if version == 1:\n",
    "        depth = n * 6 + 2\n",
    "    elif version == 2:\n",
    "        depth = n * 9 + 2\n",
    "    return depth"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def model_type(version,n):\n",
    "    # Model name, depth and version\n",
    "    model_type = 'ResNet%dv%d' % (model_depth(version,n), version)\n",
    "    return model_type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ResNet29v2\n"
     ]
    }
   ],
   "source": [
    "model_t = model_type(2,3)\n",
    "print(model_t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Function to show any data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def show_data(sample):\n",
    "    \"\"\"\n",
    "    Shows the given sample data as an image\n",
    "    \"\"\"\n",
    "    import matplotlib.pyplot as plt\n",
    "    plt.imshow(sample)\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load the CIFAR10 data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "(x_train, y_train), (x_test, y_test) = cifar10.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD5CAYAAADhukOtAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAZkElEQVR4nO2dbWxk5XXH/2de/O6119434yWYhW0TRGFBDkIlRSlpIwqhQJUgqIT4gLK0BalI4QOiUqFSPyRVkygfqlSbgkKqFEITEKsKtUGbEBpV3eCFZYFswi67Zl/stXfX2Ou3tT0zpx/mrmrIPWfsOzN3HJ7/T7I885x57j3zzJx5ef5zzhFVBSHk40+m0Q4QQtKBwU5IIDDYCQkEBjshgcBgJyQQGOyEBEKumskicjOAbwHIAvgXVf2qd/v2jg7t7umxjuWdZ9VzANvmyY2+rWQZzDkl73glxwZHEnXVUsvorVWS4yU8WiLfKz0uqz+mP6W256pEklliPJ6zM9NYOH8+1pg42EUkC+CfAPwxgBMAXhOR3ar6S2tOd08PHnjkkVhbc1OTea68YctmbPczYn9oKRaLpm1xcdG0LRm24pI9xzueZysUCqatVDJedBxbJmOvh/ei6Z3LwwpO73je47K0tJRoXq398GzeMZO+wVhks9nY8Zd3v2jOqeZj/HUADqvqEVVdBPAsgNurOB4hpI5UE+z9AI4vu34iGiOErEGqCfa4z36/8XlERHaKyJCIDM3OzFRxOkJINVQT7CcAXLzs+lYAIx+9karuUtVBVR1s7+io4nSEkGqoJthfA7BdRC4VkSYAdwPYXRu3CCG1JvFuvKoWROQhAP+FsvT2lKq+480pFAqYnPgg1pbP2a7k8/nY8UzW242P360EgJIloQEoOrvg1g55qWDvFHu76vXY2fXlyHisnV3A38VP4oc3J+P4nnOeH959TrIbn0QGBgD1VJIaZ5daj4v38Fels6vqSwBequYYhJB04C/oCAkEBjshgcBgJyQQGOyEBAKDnZBAqGo3frWUikVMT07FO5J3pLdcvPSWdaQ3cRJhrIwhoIIsVzSkt9LqEzHK85IlmdQ6QzCJXFdpnnW/PSmvwskSTbPWOOl6eP4nlUuTYEtvju819YAQsmZhsBMSCAx2QgKBwU5IIDDYCQmEVHfji8UiziXZjTcSYbyyVN5uvLsj7JZ+i99tVXEmebu+9izXx8Q72gmo9U69WzUwRcWg1slElebVuiyV/RzgbjwhwcNgJyQQGOyEBAKDnZBAYLATEggMdkICId1EmFIJs7Px5aRzTlKLKb15NegyyeqqubKLZXJeMr1zebXfKvQZss9nSU2JE1A8mycrGjZnTjZFOazWnW4q2ZLWvLNIUoOO7+yEBAKDnZBAYLATEggMdkICgcFOSCAw2AkJhKqkNxEZBjANoAigoKqD3u1LpRIWzi/E23Krb4WUzdmtlTJeSyMnI86Tmiy5w5PQXMnLk9c8Wc7BOp/Xmsi1ORKmZLwsNeN+OzX+UHKOp9657ENaqONH0npxSeclkd6sc3ke1EJn/0NVPVOD4xBC6gg/xhMSCNUGuwL4sYjsE5GdtXCIEFIfqv0Yf4OqjojIJgAvi8ivVPXV5TeIXgR2AkBLa2uVpyOEJKWqd3ZVHYn+jwN4AcB1MbfZpaqDqjqYb2qq5nSEkCpIHOwi0i4inRcuA/g8gLdr5RghpLZU8zF+M4AXItkgB+DfVPU/vQmqiqWlJdNmz4sfLzptl1zpLWExR8vmZTS5mW11oJYZVABcOczKsAMAzRjFOT15ylHlkmabWbZSKVmGWtIWT/VoA2acyDQlDnZVPQLg6qTzCSHpQumNkEBgsBMSCAx2QgKBwU5IIDDYCQmEVAtOJpfe4m1ZdQpOJpTDPOkqUdZbHeSYJH3KPHmtWLQlTFfI8+5bgh5rbkJc0TZ6/lu2kiPb+rbay3JJpDe7h509h+/shAQCg52QQGCwExIIDHZCAoHBTkgg/FbvxpecOZmS/TqWtBVPkkSYeti8tbJ89Hbj3eM5fmSKbm+o+HO598s+mpO34q5VoRBfpzDt3fiaJrs4uI9lKh4QQhoOg52QQGCwExIIDHZCAoHBTkggMNgJCYQ1I70lkZqS1qDzEleS1KDzEjGyOftcuZK9/N56eOezpKYmp7Jvc3Ozact6STKOTGkmapgzfOmt4CTCWPcZSJYI47WGWivSmy1HO1JpTT0ghKxZGOyEBAKDnZBAYLATEggMdkICgcFOSCBUlN5E5CkAXwAwrqpXRmM9AH4AYADAMIC7VPWDSsdS9bKQVi+9eTKZOLZs1rN59emMGnQZR+bL2UtczDiyVsaWtTo620xbz/ru2PHWlhZzjpdS5q1HzrtvhuS1uLhozlk0ZFkAWCrY84oLts2SepeKtlzniWSJ21B50luCen1JfFjJO/t3Adz8kbFHAexR1e0A9kTXCSFrmIrBHvVbn/jI8O0Ano4uPw3gjhr7RQipMUm/s29W1VEAiP5vqp1LhJB6UPefy4rITgA7ASDv/GSTEFJfkr6zj4lIHwBE/8etG6rqLlUdVNXBbC6f8HSEkGpJGuy7AdwXXb4PwIu1cYcQUi9WIr09A+CzADaIyAkAjwP4KoDnROR+AMcAfGlFZ1M1pbckLZSSZF0BvmTnSm/Ga6MlyVVzrvaOdtPW1d1l2rq718WOb+rdYJ+r1ZbyZmdnTdvp06dN25mxU7Hjo6fixyudy8ts8+jo6Igdb22PHweARS97LYEcBlSQ3mqId56Kwa6q9ximzyV1iBCSPvwFHSGBwGAnJBAY7IQEAoOdkEBgsBMSCOkWnIQmKryXJPvHk96843lyWHNTfGHGFiejzCvm6NHcbP8AaWRkxLQdevfd2PH1nfGSHABcu+Ma03bZZZeZtiNHjpi2dw8dih2fn58353gZcQsLC6bNW+OFhfOx4z2OXNrU2mra5ufmTJvV3w7wi1haz0avk555LCeDke/shAQCg52QQGCwExIIDHZCAoHBTkggMNgJCYRUpTePJJKcN8eT3rq67KyxgYEB07Zt27bY8fVGkUcAGBsbM23vvPOOaevp7TFthaJTYNGQr95zZLL3jw6btrvuusu03Xbbbaatpyfe//3795tzpqenTdspJ1vOyyy0VNapyUlzzkZDYgWA1mZbZj175qxp8yRdm9VLzuz1RghhsBMSCgx2QgKBwU5IIDDYCQmEVHfjRQT5/OorzFo7mZs3bzbnbN++3bR5yR1WzTIAmJuLr5E2M2vvIh8/ccy0nZ2wa7ht3NRr2rz71n9Rf+z4gcwb5pzRk3ZizSuvvGLaPv3pT5u2W2+9NXb8kksuMee88YbtY5NThvzkyZOmzUq8yeftHXctmcWScfnl9nNHC/ZO+OnT9jG9Gob2JMMHJsIQQhjshAQCg52QQGCwExIIDHZCAoHBTkggrKT901MAvgBgXFWvjMaeAPBlABe0o8dU9aVKx8pIxqzX5iW1WHJYr5FsAQBXXnmlaevvj5enACCTtV//isX4FkTnpuykitZWO3Hi6quvMm1NTl21jRs3mrbC0lLs+ITTqml2Zsa0revsNG2bN9udujuNeVdd9XvmnE2b7OMNXDpg2oadRJ7ZmXi51KsbmM3aMp/3/Fjfvd60FZwaelNGAlCS9mZu2zPT8v98F8DNMePfVNUd0V/FQCeENJaKwa6qrwKYSMEXQkgdqeY7+0MickBEnhIR+/MLIWRNkDTYvw3gMgA7AIwC+Lp1QxHZKSJDIjJUKMR/nySE1J9Ewa6qY6pa1HLl++8AuM657S5VHVTVwVxu9b+LJ4TUhkTBLiJ9y67eCeDt2rhDCKkXK5HengHwWQAbROQEgMcBfFZEdqBcJGsYwAMrOZmIoDkXn8EmYruSz8TLCe8dPmzOUUeCuOlzN5m21vZ207bFyLLb4Ehhn7rClgCzxlpUwsuymzDqoF1//fXmnD+44QbT1tXRZtoyGVsuXSrGS035vP249HbbGYc3fub3Tdsntm41bccOD8eOn5s8Z86ZWoiX6wBAmuzHrMmQZgFgfZMt9c3n4ltUlZwadKb0Zs5YQbCr6j0xw09WmkcIWVvwF3SEBAKDnZBAYLATEggMdkICgcFOSCCkWnAykxG0tsRnc6nVpwdAR1tr7LiXZTTiFCE8evSoaet22i6dPx8vkWzstYtDdnfbraE6O+yMsjZH8iqWbImnWCzGjntFNrdvs23d62w/Tp6yC1WOj8W3a5oat1skvfeW3Q5rfsFueVX0iphqvFSWy9tP/WzBfg/Mt9rZiLPTdvagFuIfF8Bu9XV2wk5JMVteVZn1Rgj5GMBgJyQQGOyEBAKDnZBAYLATEggMdkICIfVeb81GIUVLMgKAdqPgZLFoy3Wd7XYGlSfLeb3o5mfnYscnHYmkq6srkW19jy0r5nL2a/TsbLz8s2/vXnPO8+fsLLoH/+ovTFvXelumHD78fuz4//zkv805P3t+t2nrdTILd9x0o2nrMYqLZoz+gQDQm7d7CBbFfs6db7Ol1HUtdjbleyPxEubUOTszL0nWG9/ZCQkEBjshgcBgJyQQGOyEBAKDnZBASHk3PmPudntta6zWUC3Ndl2vpUU7WWRqwm7XtO5Ke0d1yagxNjtr1yzzbOPj46ato9NWE9ra7Pt9bHg4dnzPnj3mnEUnyaSp2W6FdNuf3mbaDh05Ejs+MTllzrn62mtN2+1f/DPTdskOu6WUec+c59vc+XnTNnJq1LQdWxp2/LDr9c3Nx6s8uVyC8GQiDCGEwU5IIDDYCQkEBjshgcBgJyQQGOyEBMJK2j9dDOB7ALYAKAHYparfEpEeAD8AMIByC6i7VPUD71iZjKC1Nb6e3OKiLf9YEkRW7NeqUsGW3k47iStnTp82bb/zyd+NHfeSKgpOgo9V0w4AFhzb2bN2Hbfjx4/Hjps1ywC0tdl15vbte920fcKpXXfKWMdZJ1Pjnvv+3LQ1d9o+/u+b+0zb7HS8jLawaHcUnjxnS7NTk7ZNsvYaj5yOr8kHAAWjPl1Tky17mj5UKb0VAHxFVT8F4HoAD4rIFQAeBbBHVbcD2BNdJ4SsUSoGu6qOqurr0eVpAAcB9AO4HcDT0c2eBnBHvZwkhFTPqr6zi8gAgGsA7AWwWVVHgfILAoBNtXaOEFI7VhzsItIB4EcAHlZVO6v+N+ftFJEhERlaWIhv40sIqT8rCnYRyaMc6N9X1eej4TER6YvsfQBif+itqrtUdVBVB60qNYSQ+lMx2KW8vfckgIOq+o1lpt0A7osu3wfgxdq7RwipFStJq7kBwL0A3hKR/dHYYwC+CuA5EbkfwDEAX6p0oGw2i04jm2tpyZZCLNu5afvbxOQHtkQyfsqWQX72k1dM2/DR4djxLX1bzDm9Tmuotja7LllJbclubsZuM7Qwb0h2duk0tLTYWXSZjC0rvj5kS14FI8tLs7Y0dHzyjGk7dWjMtB07dsK0dbbEP98u6rvInDNvZKEBvkQ8v2DLpZNOPbmsId02Ndn1EC2JzZNYKwa7qv4cdh27z1WaTwhZG/AXdIQEAoOdkEBgsBMSCAx2QgKBwU5IIKRacDKbzWDduviCjktLdpbawmL8L++WFuw52ZwtGeWcLLVTRiseADgzHi//5PN2dlJHhy2v9fTYslxvr91aqbPTPiaMddzYu8GcclFfn2k7evSoaRs7YUtePYb/lswEAG8MvWHaZpzCnaq2rthlSG8LjkymJUendJibsX3MOY2ZWo0Cot5aWcltmQwLThISPAx2QgKBwU5IIDDYCQkEBjshgcBgJyQQ0pXeMlmsW7cu1lZwCkQWjaKN3evWm3P6+/pN2yeNwpEAMD9v9/myCj16PdsmztrFLUdO2jIfnN5gvpwXL3l5WYVzjqzl1SCYnp42ba1OJp15vCk7M+zXv/q1aevvtzPYBi7+ROz4jJM5eNp5PI84UuSEU8h0YGDAtLW3xxfT9DLYLJs7x7QQQj5WMNgJCQQGOyGBwGAnJBAY7IQEQqq78Zlsst34UsnYmS7ZP/qX9bataB0PQMHZtd66NX6Hf9bZzZ6amjJt3u6t1+JpYsK2jRvJOpdfvt2c4+3Unzlj14XzHjPLf6+ceD5v11w7c9b2w6tFODI6asyxlYTJKbt+4aKzVps3bzZt3n2zbF77J7MlmlMzkO/shAQCg52QQGCwExIIDHZCAoHBTkggMNgJCYSK0puIXAzgewC2oJydsUtVvyUiTwD4MoDT0U0fU9WXvGNlMhm0tcX/6D+J9KZFW0Lz6pKZUh4AbbaXpLUUnxTS1m4nfXR1x9fcA4AtW+wu1+edGmnT5+wkDkvyStJeCwAmJx0ZymmFZM3zpDfvMfPqsXnJS1ayTs5prdRpyMMA0NXVZdo86a2z034edHTE18nz2nJZ0lsmW0X7JwAFAF9R1ddFpBPAPhF5ObJ9U1X/cQXHIIQ0mJX0ehsFMBpdnhaRgwDs/FFCyJpkVd/ZRWQAwDUA9kZDD4nIARF5SkTs5HJCSMNZcbCLSAeAHwF4WFXPAfg2gMsA7ED5nf/rxrydIjIkIkMzM/ZPFAkh9WVFwS4ieZQD/fuq+jwAqOqYqhZVtQTgOwCui5urqrtUdVBVBzs67E0KQkh9qRjsUu76/iSAg6r6jWXjy9uI3Ang7dq7RwipFSvZjb8BwL0A3hKR/dHYYwDuEZEdABTAMIAHKh1IREzJwMOSykpiy3XqyGsZJzPIl+wsi9emx5Z4so5M0tRsz2tpbjVtVlahJ6F5dcs8SXRubs60eRKVhVg9jWBLTYBfJ8+Setva7Tp+HUaLMsBe30p+eDKaZfMy5ay18iTKlezG/xyIbVTlauqEkLUFf0FHSCAw2AkJBAY7IYHAYCckEBjshARCqgUnoWq2ckqS9VYqJihSCV9e8yiV4ucZw+VzOa+n4rzU5rzCgRmneGFzfJHCVqPFEABs2LTRtBXdzELHZiyKt/IZR3rL5hxJyZHlctl4m+Tsxc86x/OyyiRWtIrmOfKmGDb7aLb05smXfGcnJBAY7IQEAoOdkEBgsBMSCAx2QgKBwU5IIKQqvZVKJTNTypLkPFtSec07VxJZTtWWOzwfk5JIOnRlLUe6yjvznAwrTwKq5ZxK8yyb85ABjrzmtBcEnMdFHdFRDQk50f1yfOA7OyGBwGAnJBAY7IQEAoOdkEBgsBMSCAx2QgIhVemtWCphdnY21pZERkuSKecdr5LNljtsiSRphl2tSSoBevKPt/4WXvZXUj9qLb1pIdm5PJL6v9o5JUpvhBAGOyGBwGAnJBAY7IQEAoOdkECouBsvIi0AXgXQHN3+h6r6uIhcCuBZAD0AXgdwr6ouesdSLeH8+fOxNi85xSJpQkvSHXJr2hrZcHdJuhufdPfcwtt59pYxST0215Zx/HBsHrVWDJLg1QVcySO5AOAmVb0a5fbMN4vI9QC+BuCbqrodwAcA7q+Br4SQOlEx2LXMTHQ1H/0pgJsA/DAafxrAHXXxkBBSE1banz0bdXAdB/AygPcATKrqhV9VnADQXx8XCSG1YEXBrqpFVd0BYCuA6wB8Ku5mcXNFZKeIDInI0Nys3eKXEFJfVrX7oqqTAF4BcD2AbhG5sMG3FcCIMWeXqg6q6mCb06iAEFJfKga7iGwUke7ociuAPwJwEMBPAXwxutl9AF6sl5OEkOpZSSJMH4CnRSSL8ovDc6r6HyLySwDPisjfA3gDwJOVDlQqaU2lt+QSWjJZzmr/lLb0VkupphJJ17jWNehqLl058ppbZ87Bbb1UYwnTwnqOAisIdlU9AOCamPEjKH9/J4T8FsBf0BESCAx2QgKBwU5IIDDYCQkEBjshgSBp1kgTkdMA3o+ubgBwJrWT29CPD0M/Psxvmx+XqOrGOEOqwf6hE4sMqepgQ05OP+hHgH7wYzwhgcBgJyQQGhnsuxp47uXQjw9DPz7Mx8aPhn1nJ4SkCz/GExIIDQl2EblZRH4tIodF5NFG+BD5MSwib4nIfhEZSvG8T4nIuIi8vWysR0ReFpFD0f/1DfLjCRE5Ga3JfhG5JQU/LhaRn4rIQRF5R0T+OhpPdU0cP1JdExFpEZFfiMibkR9/F41fKiJ7o/X4gYg0rerAqprqH4AsymWttgFoAvAmgCvS9iPyZRjAhgac90YA1wJ4e9nYPwB4NLr8KICvNciPJwA8kvJ69AG4NrrcCeBdAFekvSaOH6muCcrFdDuiy3kAe1EuGPMcgLuj8X8G8JerOW4j3tmvA3BYVY9oufT0swBub4AfDUNVXwUw8ZHh21Eu3AmkVMDT8CN1VHVUVV+PLk+jXBylHymvieNHqmiZmhd5bUSw9wM4vux6I4tVKoAfi8g+EdnZIB8usFlVR4Hykw7Apgb68pCIHIg+5tf968RyRGQA5foJe9HANfmIH0DKa1KPIq+NCPa4ch6NkgRuUNVrAfwJgAdF5MYG+bGW+DaAy1DuETAK4OtpnVhEOgD8CMDDqnourfOuwI/U10SrKPJq0YhgPwHg4mXXzWKV9UZVR6L/4wBeQGMr74yJSB8ARP/HG+GEqo5FT7QSgO8gpTURkTzKAfZ9VX0+Gk59TeL8aNSaROdedZFXi0YE+2sAtkc7i00A7gawO20nRKRdRDovXAbweQBv+7Pqym6UC3cCDSzgeSG4Iu5ECmsi5eJtTwI4qKrfWGZKdU0sP9Jek7oVeU1rh/Eju423oLzT+R6Av2mQD9tQVgLeBPBOmn4AeAblj4NLKH/SuR9AL4A9AA5F/3sa5Me/AngLwAGUg60vBT8+g/JH0gMA9kd/t6S9Jo4fqa4JgKtQLuJ6AOUXlr9d9pz9BYDDAP4dQPNqjstf0BESCPwFHSGBwGAnJBAY7IQEAoOdkEBgsBMSCAx2QgKBwU5IIDDYCQmE/wNH+sWQ3Ez3egAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_data(x_train[250])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Input image dimensions.\n",
    "input_shape = x_train.shape[1:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(32, 32, 3)\n"
     ]
    }
   ],
   "source": [
    "print(input_shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Normalize data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train = x_train.astype('float32') / 255\n",
    "x_test = x_test.astype('float32') / 255"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD5CAYAAADhukOtAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAZkElEQVR4nO2dbWxk5XXH/2de/O6119434yWYhW0TRGFBDkIlRSlpIwqhQJUgqIT4gLK0BalI4QOiUqFSPyRVkygfqlSbgkKqFEITEKsKtUGbEBpV3eCFZYFswi67Zl/stXfX2Ou3tT0zpx/mrmrIPWfsOzN3HJ7/T7I885x57j3zzJx5ef5zzhFVBSHk40+m0Q4QQtKBwU5IIDDYCQkEBjshgcBgJyQQGOyEBEKumskicjOAbwHIAvgXVf2qd/v2jg7t7umxjuWdZ9VzANvmyY2+rWQZzDkl73glxwZHEnXVUsvorVWS4yU8WiLfKz0uqz+mP6W256pEklliPJ6zM9NYOH8+1pg42EUkC+CfAPwxgBMAXhOR3ar6S2tOd08PHnjkkVhbc1OTea68YctmbPczYn9oKRaLpm1xcdG0LRm24pI9xzueZysUCqatVDJedBxbJmOvh/ei6Z3LwwpO73je47K0tJRoXq398GzeMZO+wVhks9nY8Zd3v2jOqeZj/HUADqvqEVVdBPAsgNurOB4hpI5UE+z9AI4vu34iGiOErEGqCfa4z36/8XlERHaKyJCIDM3OzFRxOkJINVQT7CcAXLzs+lYAIx+9karuUtVBVR1s7+io4nSEkGqoJthfA7BdRC4VkSYAdwPYXRu3CCG1JvFuvKoWROQhAP+FsvT2lKq+480pFAqYnPgg1pbP2a7k8/nY8UzW242P360EgJIloQEoOrvg1g55qWDvFHu76vXY2fXlyHisnV3A38VP4oc3J+P4nnOeH959TrIbn0QGBgD1VJIaZ5daj4v38Fels6vqSwBequYYhJB04C/oCAkEBjshgcBgJyQQGOyEBAKDnZBAqGo3frWUikVMT07FO5J3pLdcvPSWdaQ3cRJhrIwhoIIsVzSkt9LqEzHK85IlmdQ6QzCJXFdpnnW/PSmvwskSTbPWOOl6eP4nlUuTYEtvju819YAQsmZhsBMSCAx2QgKBwU5IIDDYCQmEVHfji8UiziXZjTcSYbyyVN5uvLsj7JZ+i99tVXEmebu+9izXx8Q72gmo9U69WzUwRcWg1slElebVuiyV/RzgbjwhwcNgJyQQGOyEBAKDnZBAYLATEggMdkICId1EmFIJs7Px5aRzTlKLKb15NegyyeqqubKLZXJeMr1zebXfKvQZss9nSU2JE1A8mycrGjZnTjZFOazWnW4q2ZLWvLNIUoOO7+yEBAKDnZBAYLATEggMdkICgcFOSCAw2AkJhKqkNxEZBjANoAigoKqD3u1LpRIWzi/E23Krb4WUzdmtlTJeSyMnI86Tmiy5w5PQXMnLk9c8Wc7BOp/Xmsi1ORKmZLwsNeN+OzX+UHKOp9657ENaqONH0npxSeclkd6sc3ke1EJn/0NVPVOD4xBC6gg/xhMSCNUGuwL4sYjsE5GdtXCIEFIfqv0Yf4OqjojIJgAvi8ivVPXV5TeIXgR2AkBLa2uVpyOEJKWqd3ZVHYn+jwN4AcB1MbfZpaqDqjqYb2qq5nSEkCpIHOwi0i4inRcuA/g8gLdr5RghpLZU8zF+M4AXItkgB+DfVPU/vQmqiqWlJdNmz4sfLzptl1zpLWExR8vmZTS5mW11oJYZVABcOczKsAMAzRjFOT15ylHlkmabWbZSKVmGWtIWT/VoA2acyDQlDnZVPQLg6qTzCSHpQumNkEBgsBMSCAx2QgKBwU5IIDDYCQmEVAtOJpfe4m1ZdQpOJpTDPOkqUdZbHeSYJH3KPHmtWLQlTFfI8+5bgh5rbkJc0TZ6/lu2kiPb+rbay3JJpDe7h509h+/shAQCg52QQGCwExIIDHZCAoHBTkgg/FbvxpecOZmS/TqWtBVPkkSYeti8tbJ89Hbj3eM5fmSKbm+o+HO598s+mpO34q5VoRBfpzDt3fiaJrs4uI9lKh4QQhoOg52QQGCwExIIDHZCAoHBTkggMNgJCYQ1I70lkZqS1qDzEleS1KDzEjGyOftcuZK9/N56eOezpKYmp7Jvc3Ozact6STKOTGkmapgzfOmt4CTCWPcZSJYI47WGWivSmy1HO1JpTT0ghKxZGOyEBAKDnZBAYLATEggMdkICgcFOSCBUlN5E5CkAXwAwrqpXRmM9AH4AYADAMIC7VPWDSsdS9bKQVi+9eTKZOLZs1rN59emMGnQZR+bL2UtczDiyVsaWtTo620xbz/ru2PHWlhZzjpdS5q1HzrtvhuS1uLhozlk0ZFkAWCrY84oLts2SepeKtlzniWSJ21B50luCen1JfFjJO/t3Adz8kbFHAexR1e0A9kTXCSFrmIrBHvVbn/jI8O0Ano4uPw3gjhr7RQipMUm/s29W1VEAiP5vqp1LhJB6UPefy4rITgA7ASDv/GSTEFJfkr6zj4lIHwBE/8etG6rqLlUdVNXBbC6f8HSEkGpJGuy7AdwXXb4PwIu1cYcQUi9WIr09A+CzADaIyAkAjwP4KoDnROR+AMcAfGlFZ1M1pbckLZSSZF0BvmTnSm/Ga6MlyVVzrvaOdtPW1d1l2rq718WOb+rdYJ+r1ZbyZmdnTdvp06dN25mxU7Hjo6fixyudy8ts8+jo6Igdb22PHweARS97LYEcBlSQ3mqId56Kwa6q9ximzyV1iBCSPvwFHSGBwGAnJBAY7IQEAoOdkEBgsBMSCOkWnIQmKryXJPvHk96843lyWHNTfGHGFiejzCvm6NHcbP8AaWRkxLQdevfd2PH1nfGSHABcu+Ma03bZZZeZtiNHjpi2dw8dih2fn58353gZcQsLC6bNW+OFhfOx4z2OXNrU2mra5ufmTJvV3w7wi1haz0avk555LCeDke/shAQCg52QQGCwExIIDHZCAoHBTkggMNgJCYRUpTePJJKcN8eT3rq67KyxgYEB07Zt27bY8fVGkUcAGBsbM23vvPOOaevp7TFthaJTYNGQr95zZLL3jw6btrvuusu03Xbbbaatpyfe//3795tzpqenTdspJ1vOyyy0VNapyUlzzkZDYgWA1mZbZj175qxp8yRdm9VLzuz1RghhsBMSCgx2QgKBwU5IIDDYCQmEVHfjRQT5/OorzFo7mZs3bzbnbN++3bR5yR1WzTIAmJuLr5E2M2vvIh8/ccy0nZ2wa7ht3NRr2rz71n9Rf+z4gcwb5pzRk3ZizSuvvGLaPv3pT5u2W2+9NXb8kksuMee88YbtY5NThvzkyZOmzUq8yeftHXctmcWScfnl9nNHC/ZO+OnT9jG9Gob2JMMHJsIQQhjshAQCg52QQGCwExIIDHZCAoHBTkggrKT901MAvgBgXFWvjMaeAPBlABe0o8dU9aVKx8pIxqzX5iW1WHJYr5FsAQBXXnmlaevvj5enACCTtV//isX4FkTnpuykitZWO3Hi6quvMm1NTl21jRs3mrbC0lLs+ITTqml2Zsa0revsNG2bN9udujuNeVdd9XvmnE2b7OMNXDpg2oadRJ7ZmXi51KsbmM3aMp/3/Fjfvd60FZwaelNGAlCS9mZu2zPT8v98F8DNMePfVNUd0V/FQCeENJaKwa6qrwKYSMEXQkgdqeY7+0MickBEnhIR+/MLIWRNkDTYvw3gMgA7AIwC+Lp1QxHZKSJDIjJUKMR/nySE1J9Ewa6qY6pa1HLl++8AuM657S5VHVTVwVxu9b+LJ4TUhkTBLiJ9y67eCeDt2rhDCKkXK5HengHwWQAbROQEgMcBfFZEdqBcJGsYwAMrOZmIoDkXn8EmYruSz8TLCe8dPmzOUUeCuOlzN5m21vZ207bFyLLb4Ehhn7rClgCzxlpUwsuymzDqoF1//fXmnD+44QbT1tXRZtoyGVsuXSrGS035vP249HbbGYc3fub3Tdsntm41bccOD8eOn5s8Z86ZWoiX6wBAmuzHrMmQZgFgfZMt9c3n4ltUlZwadKb0Zs5YQbCr6j0xw09WmkcIWVvwF3SEBAKDnZBAYLATEggMdkICgcFOSCCkWnAykxG0tsRnc6nVpwdAR1tr7LiXZTTiFCE8evSoaet22i6dPx8vkWzstYtDdnfbraE6O+yMsjZH8iqWbImnWCzGjntFNrdvs23d62w/Tp6yC1WOj8W3a5oat1skvfeW3Q5rfsFueVX0iphqvFSWy9tP/WzBfg/Mt9rZiLPTdvagFuIfF8Bu9XV2wk5JMVteVZn1Rgj5GMBgJyQQGOyEBAKDnZBAYLATEggMdkICIfVeb81GIUVLMgKAdqPgZLFoy3Wd7XYGlSfLeb3o5mfnYscnHYmkq6srkW19jy0r5nL2a/TsbLz8s2/vXnPO8+fsLLoH/+ovTFvXelumHD78fuz4//zkv805P3t+t2nrdTILd9x0o2nrMYqLZoz+gQDQm7d7CBbFfs6db7Ol1HUtdjbleyPxEubUOTszL0nWG9/ZCQkEBjshgcBgJyQQGOyEBAKDnZBASHk3PmPudntta6zWUC3Ndl2vpUU7WWRqwm7XtO5Ke0d1yagxNjtr1yzzbOPj46ato9NWE9ra7Pt9bHg4dnzPnj3mnEUnyaSp2W6FdNuf3mbaDh05Ejs+MTllzrn62mtN2+1f/DPTdskOu6WUec+c59vc+XnTNnJq1LQdWxp2/LDr9c3Nx6s8uVyC8GQiDCGEwU5IIDDYCQkEBjshgcBgJyQQGOyEBMJK2j9dDOB7ALYAKAHYparfEpEeAD8AMIByC6i7VPUD71iZjKC1Nb6e3OKiLf9YEkRW7NeqUsGW3k47iStnTp82bb/zyd+NHfeSKgpOgo9V0w4AFhzb2bN2Hbfjx4/Hjps1ywC0tdl15vbte920fcKpXXfKWMdZJ1Pjnvv+3LQ1d9o+/u+b+0zb7HS8jLawaHcUnjxnS7NTk7ZNsvYaj5yOr8kHAAWjPl1Tky17mj5UKb0VAHxFVT8F4HoAD4rIFQAeBbBHVbcD2BNdJ4SsUSoGu6qOqurr0eVpAAcB9AO4HcDT0c2eBnBHvZwkhFTPqr6zi8gAgGsA7AWwWVVHgfILAoBNtXaOEFI7VhzsItIB4EcAHlZVO6v+N+ftFJEhERlaWIhv40sIqT8rCnYRyaMc6N9X1eej4TER6YvsfQBif+itqrtUdVBVB60qNYSQ+lMx2KW8vfckgIOq+o1lpt0A7osu3wfgxdq7RwipFStJq7kBwL0A3hKR/dHYYwC+CuA5EbkfwDEAX6p0oGw2i04jm2tpyZZCLNu5afvbxOQHtkQyfsqWQX72k1dM2/DR4djxLX1bzDm9Tmuotja7LllJbclubsZuM7Qwb0h2duk0tLTYWXSZjC0rvj5kS14FI8tLs7Y0dHzyjGk7dWjMtB07dsK0dbbEP98u6rvInDNvZKEBvkQ8v2DLpZNOPbmsId02Ndn1EC2JzZNYKwa7qv4cdh27z1WaTwhZG/AXdIQEAoOdkEBgsBMSCAx2QgKBwU5IIKRacDKbzWDduviCjktLdpbawmL8L++WFuw52ZwtGeWcLLVTRiseADgzHi//5PN2dlJHhy2v9fTYslxvr91aqbPTPiaMddzYu8GcclFfn2k7evSoaRs7YUtePYb/lswEAG8MvWHaZpzCnaq2rthlSG8LjkymJUendJibsX3MOY2ZWo0Cot5aWcltmQwLThISPAx2QgKBwU5IIDDYCQkEBjshgcBgJyQQ0pXeMlmsW7cu1lZwCkQWjaKN3evWm3P6+/pN2yeNwpEAMD9v9/myCj16PdsmztrFLUdO2jIfnN5gvpwXL3l5WYVzjqzl1SCYnp42ba1OJp15vCk7M+zXv/q1aevvtzPYBi7+ROz4jJM5eNp5PI84UuSEU8h0YGDAtLW3xxfT9DLYLJs7x7QQQj5WMNgJCQQGOyGBwGAnJBAY7IQEQqq78Zlsst34UsnYmS7ZP/qX9bataB0PQMHZtd66NX6Hf9bZzZ6amjJt3u6t1+JpYsK2jRvJOpdfvt2c4+3Unzlj14XzHjPLf6+ceD5v11w7c9b2w6tFODI6asyxlYTJKbt+4aKzVps3bzZt3n2zbF77J7MlmlMzkO/shAQCg52QQGCwExIIDHZCAoHBTkggMNgJCYSK0puIXAzgewC2oJydsUtVvyUiTwD4MoDT0U0fU9WXvGNlMhm0tcX/6D+J9KZFW0Lz6pKZUh4AbbaXpLUUnxTS1m4nfXR1x9fcA4AtW+wu1+edGmnT5+wkDkvyStJeCwAmJx0ZymmFZM3zpDfvMfPqsXnJS1ayTs5prdRpyMMA0NXVZdo86a2z034edHTE18nz2nJZ0lsmW0X7JwAFAF9R1ddFpBPAPhF5ObJ9U1X/cQXHIIQ0mJX0ehsFMBpdnhaRgwDs/FFCyJpkVd/ZRWQAwDUA9kZDD4nIARF5SkTs5HJCSMNZcbCLSAeAHwF4WFXPAfg2gMsA7ED5nf/rxrydIjIkIkMzM/ZPFAkh9WVFwS4ieZQD/fuq+jwAqOqYqhZVtQTgOwCui5urqrtUdVBVBzs67E0KQkh9qRjsUu76/iSAg6r6jWXjy9uI3Ang7dq7RwipFSvZjb8BwL0A3hKR/dHYYwDuEZEdABTAMIAHKh1IREzJwMOSykpiy3XqyGsZJzPIl+wsi9emx5Z4so5M0tRsz2tpbjVtVlahJ6F5dcs8SXRubs60eRKVhVg9jWBLTYBfJ8+Setva7Tp+HUaLMsBe30p+eDKaZfMy5ay18iTKlezG/xyIbVTlauqEkLUFf0FHSCAw2AkJBAY7IYHAYCckEBjshARCqgUnoWq2ckqS9VYqJihSCV9e8yiV4ucZw+VzOa+n4rzU5rzCgRmneGFzfJHCVqPFEABs2LTRtBXdzELHZiyKt/IZR3rL5hxJyZHlctl4m+Tsxc86x/OyyiRWtIrmOfKmGDb7aLb05smXfGcnJBAY7IQEAoOdkEBgsBMSCAx2QgKBwU5IIKQqvZVKJTNTypLkPFtSec07VxJZTtWWOzwfk5JIOnRlLUe6yjvznAwrTwKq5ZxK8yyb85ABjrzmtBcEnMdFHdFRDQk50f1yfOA7OyGBwGAnJBAY7IQEAoOdkEBgsBMSCAx2QgIhVemtWCphdnY21pZERkuSKecdr5LNljtsiSRphl2tSSoBevKPt/4WXvZXUj9qLb1pIdm5PJL6v9o5JUpvhBAGOyGBwGAnJBAY7IQEAoOdkECouBsvIi0AXgXQHN3+h6r6uIhcCuBZAD0AXgdwr6ouesdSLeH8+fOxNi85xSJpQkvSHXJr2hrZcHdJuhufdPfcwtt59pYxST0215Zx/HBsHrVWDJLg1QVcySO5AOAmVb0a5fbMN4vI9QC+BuCbqrodwAcA7q+Br4SQOlEx2LXMTHQ1H/0pgJsA/DAafxrAHXXxkBBSE1banz0bdXAdB/AygPcATKrqhV9VnADQXx8XCSG1YEXBrqpFVd0BYCuA6wB8Ku5mcXNFZKeIDInI0Nys3eKXEFJfVrX7oqqTAF4BcD2AbhG5sMG3FcCIMWeXqg6q6mCb06iAEFJfKga7iGwUke7ociuAPwJwEMBPAXwxutl9AF6sl5OEkOpZSSJMH4CnRSSL8ovDc6r6HyLySwDPisjfA3gDwJOVDlQqaU2lt+QSWjJZzmr/lLb0VkupphJJ17jWNehqLl058ppbZ87Bbb1UYwnTwnqOAisIdlU9AOCamPEjKH9/J4T8FsBf0BESCAx2QgKBwU5IIDDYCQkEBjshgSBp1kgTkdMA3o+ubgBwJrWT29CPD0M/Psxvmx+XqOrGOEOqwf6hE4sMqepgQ05OP+hHgH7wYzwhgcBgJyQQGhnsuxp47uXQjw9DPz7Mx8aPhn1nJ4SkCz/GExIIDQl2EblZRH4tIodF5NFG+BD5MSwib4nIfhEZSvG8T4nIuIi8vWysR0ReFpFD0f/1DfLjCRE5Ga3JfhG5JQU/LhaRn4rIQRF5R0T+OhpPdU0cP1JdExFpEZFfiMibkR9/F41fKiJ7o/X4gYg0rerAqprqH4AsymWttgFoAvAmgCvS9iPyZRjAhgac90YA1wJ4e9nYPwB4NLr8KICvNciPJwA8kvJ69AG4NrrcCeBdAFekvSaOH6muCcrFdDuiy3kAe1EuGPMcgLuj8X8G8JerOW4j3tmvA3BYVY9oufT0swBub4AfDUNVXwUw8ZHh21Eu3AmkVMDT8CN1VHVUVV+PLk+jXBylHymvieNHqmiZmhd5bUSw9wM4vux6I4tVKoAfi8g+EdnZIB8usFlVR4Hykw7Apgb68pCIHIg+5tf968RyRGQA5foJe9HANfmIH0DKa1KPIq+NCPa4ch6NkgRuUNVrAfwJgAdF5MYG+bGW+DaAy1DuETAK4OtpnVhEOgD8CMDDqnourfOuwI/U10SrKPJq0YhgPwHg4mXXzWKV9UZVR6L/4wBeQGMr74yJSB8ARP/HG+GEqo5FT7QSgO8gpTURkTzKAfZ9VX0+Gk59TeL8aNSaROdedZFXi0YE+2sAtkc7i00A7gawO20nRKRdRDovXAbweQBv+7Pqym6UC3cCDSzgeSG4Iu5ECmsi5eJtTwI4qKrfWGZKdU0sP9Jek7oVeU1rh/Eju423oLzT+R6Av2mQD9tQVgLeBPBOmn4AeAblj4NLKH/SuR9AL4A9AA5F/3sa5Me/AngLwAGUg60vBT8+g/JH0gMA9kd/t6S9Jo4fqa4JgKtQLuJ6AOUXlr9d9pz9BYDDAP4dQPNqjstf0BESCPwFHSGBwGAnJBAY7IQEAoOdkEBgsBMSCAx2QgKBwU5IIDDYCQmE/wNH+sWQ3Ez3egAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_data(x_train[250])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### If `subtract_pixel_mean` is enabled"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# If subtract pixel mean is enabled\n",
    "if subtract_pixel_mean:\n",
    "    x_train_mean = np.mean(x_train, axis=0)\n",
    "    x_train -= x_train_mean\n",
    "    x_test -= x_train_mean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_train shape: (50000, 32, 32, 3)\n",
      "50000 train samples\n",
      "10000 test samples\n",
      "y_train shape: (50000, 1)\n"
     ]
    }
   ],
   "source": [
    "print('x_train shape:', x_train.shape)\n",
    "print(x_train.shape[0], 'train samples')\n",
    "print(x_test.shape[0], 'test samples')\n",
    "print('y_train shape:', y_train.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD5CAYAAADhukOtAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAATNElEQVR4nO3de4xd1XXH8e+68zAeezB+gHEMiYFYbShJTXBppDyUtyiiJUhNBGoj/ojiNApSkdI/EJUa2r+aKoDyVyqnoJAqTUJCUlAUlSCUiiZ/JBgCtqkLAToBw3hsHsYv5nlX/7jHzeCcte/Muec+zP59JGvunHXPOdtn7rrn3rPO3tvcHRF582v0uwEi0htKdpFMKNlFMqFkF8mEkl0kE0p2kUwMd7KymV0BfBUYAv7F3f+xzfMdK39/sYalVixfTGKdlES1senNxHrBisnypUqb0lvuXpoYVrXObmZDwFPAx4D9wMPAde7+3+E6jSFnxVhpbGTFinBfwyMjpcuHGvF7VeqNYGFhIYzNzMzE683Olgdm43UgWEekS6Jk7+Rj/OXA0+7+rLvPAt8Bru5geyLSRZ0k+2bg+UW/7y+WicgA6uQ7e9lHhd/5TmBmO4Ad8Soi0gudJPt+4PxFv58HvHjqk9x9J7ATiu/sItIXnXyMfxjYamYXmNkocC1wXz3NEpG6VT6zu/u8md0A3E+r9Hanuz+RXqkJ08dKQ3PTx8PV5qJmDiWab0NxrBlfjac5H8eIYvrAIoOvcumt0s7MEjtLfZ9XsossVTdKbyJyGlGyi2RCyS6SCSW7SCaU7CKZGKCr8SJSB12NF8mckl0kE0p2kUwo2UUyoWQXyYSSXSQTSnaRTCjZRTKhZBfJhJJdJBNKdpFMKNlFMtHR9E8isWiYMfWF6hed2UUyoWQXyYSSXSQTSnaRTCjZRTKhZBfJREelNzObAI4CC8C8u2+vo1EySFIz9aTOFVGsWbEdqZJd1W3mpY46+4fc/aUatiMiXaSP8SKZ6DTZHfiJmT1iZjvqaJCIdEenH+Pf6+4vmtk5wANm9j/u/tDiJxRvAnojEOmz2iaJMLNbgGPu/pXEc3Rj9GlHF+hON7VPEmFmq8xs/ORj4OPA3qrbE5Hu6uRj/Ebgh2Z2cjv/5u7/UUur5DSROuvXuY7UQXO9SRup5ByqsF7Vl0BqvYWK23xz0lxvIplTsotkQskukgklu0gmlOwimdCAk9JG1ZtZ6r4arxtnOqUzu0gmlOwimVCyi2RCyS6SCSW7SCZ0NV46oCvkpxOd2UUyoWQXyYSSXSQTSnaRTCjZRTKhZBfJhJJdJBNKdpFMKNlFMqFkF8mEkl0kE0p2kUwo2UUy0bbXm5ndCVwFHHT3S4pl64DvAluACeBT7v5q95opnUvN3pIaFy61XurlE83SMptYR7ppKWf2bwBXnLLsJuBBd98KPFj8LiIDrG2yF/Otv3LK4quBu4rHdwGfqLldIlKzqt/ZN7r7JEDx85z6miQi3dD1kWrMbAewo9v7EZG0qmf2KTPbBFD8PBg90d13uvt2d99ecV8iUoOqyX4fcH3x+Hrg3nqaIyLdYu7p6XjM7NvAB4ENwBTwJeDfgbuBtwLPAZ9091Mv4pVtq+rcP7JUw6tKF4+t3xCusmrlWBg7fvx4GDtxKPxAB0wnYtJN7l4691bb7+zufl0Q+khHLRKRntIddCKZULKLZELJLpIJJbtIJpTsIploW3qrdWcqvQ2mlevC0JaLLgpjE3sf7kZrpENR6U1ndpFMKNlFMqFkF8mEkl0kE0p2kUwo2UUyodLbcoyWl6jG1p4VrnJiaiqxwbhH2aCwNeeGscsuuyyM7fr5z8sDM4c7bZK0odKbSOaU7CKZULKLZELJLpIJJbtIJro+lHT/xP+1NW+JO3esHl8dxo6fOFG6/OiJo4l2dOGK+9pNcSxoIzOvVdqVv3YgjDWG4qmhPnrVVaXLn9izN1xn8qnHlt4wWTad2UUyoWQXyYSSXSQTSnaRTCjZRTKhZBfJRNvSm5ndCVwFHHT3S4pltwCfBQ4VT7vZ3X/crUZWsent7wxjm8/bHMYsUU6aX5gvXX7kSNy5Y2psZRibnpkJYyMrVoSxs88+O4zNzc2VLp/cU/94ceeeG3eSGR8fL10+lDi+GxPbe/KpJ8PY6wcmwhhU6XsVtzF9fiw/9oNiKWf2bwBXlCy/3d23Ff8GKtFF5He1TXZ3fwhoO2mjiAy2Tr6z32Bmu83sTjNbW1uLRKQrqib714CLgG3AJHBr9EQz22Fmu8xsV8V9iUgNKiW7u0+5+4K7N4GvA5cnnrvT3be7+/aqjRSRzlVKdjNb3BPjGiDu3SAiA6HtGHRm9m3gg8AGYAr4UvH7Nlp1jQngc+4+2XZnAzIG3Xl/8MdhbOWqVWFs48aNpcvHEuW10UQJbWg4LvGkDtTR43Evu1deerl0+UsHD4brjCbKYWtWj4WxC7fGvQdHzzijdPmJY8fCdU4ciWMjo/FxnJh4Low9//RE6fJjhxO9AD3VU3EkEZtOxJqJWL2iMeja1tnd/bqSxXd03CIR6SndQSeSCSW7SCaU7CKZULKLZELJLpKJN/GAk7H9ExNhbPWG9WFsena2dPnZ6+N11qyN7yQeX13eMwxg5Xhc8momyjgvH3qpdPnWt28N19l6UVxCO+vMuB0vHHgxjB0KSn2vTZWXBgGe2fNEGHt9pvzYAzRHEi/jZnlZcXh0NFxlPtEbkZXlJUUAXg8G+xwQOrOLZELJLpIJJbtIJpTsIplQsotkQskukom2vd5q3dmA9HpLaax7axiLBqNclegpt2bNmjB2ZiK2dv26MDY8HL9H73/++dLlTz8Rl7U4Gvc2+4vP/1UYW7P2rDC2+5FflS7/2f33x+2gWumqYWeGsfWbywcXbTTiY9gYiXsBNhMv4enDcW/E116Kex1W/X9Hol5vOrOLZELJLpIJJbtIJpTsIplQsotkQlfjT7HubfG0UbPB9E/Ts4mOE5ReGAVgONGBY9X46jA2NhZ3xng+6uTzygvhOkkr4o48H/uzPw1jex/fXbp88qnHKjXjA5d+KIxt2XZJGIu6z1ji73JiOr46PnkgHmrxuWf/N4wd+E0cg9cTseXT1XiRzCnZRTKhZBfJhJJdJBNKdpFMKNlFMrGU6Z/OB74JnEtrDpud7v5VM1sHfBfYQmsKqE+5+6tttjXwpTdWbQpDb//93ytd3khM4zS/EI8XNz0TTxc0PR3HZufi2LGJfWGsbhsv/qMwNhWVqF7ZH67z/vdfGcbGxuMptiamDoSxE0fKy1ozs3PhOoePxC/j2VcPhzGG4nIeC3Enmbp1UnqbB77o7u8A3gN8wcwuBm4CHnT3rcCDxe8iMqDaJru7T7r7o8Xjo8A+YDNwNXBX8bS7gE90q5Ei0rllfWc3sy3ApcAvgI0nZ24tfp5Td+NEpD5LHjfezFYD9wA3uvsRs8T3kzeutwPYUa15IlKXJZ3ZzWyEVqJ/y91/UCyeMrNNRXwTUDoUh7vvdPft7r69jgaLSDVtk91ap/A7gH3uftui0H3A9cXj64F762+eiNRlKaW39wH/BeyB/5936GZa39vvBt4KPAd80t1fabOtwS+9VTEcjxc3tmFDHBuLx65r+kIYO3IsLv/MH3oujNUvLjkSTlEVvwQueOflYezAoakw9vqBuJxHo7z34PpNcYn15ZcPxdubjsfrg1Tvx96JSm9tv7O7+8+I+2l+pJNGiUjv6A46kUwo2UUyoWQXyYSSXSQTSnaRTPRhwMmoAJC6Iy/uoZSn0UQsGmKxl9urqgvtaJRPsTW+Li6XHj36Wry9meOJnQ126U1ndpFMKNlFMqFkF8mEkl0kE0p2kUwo2UUy0dvSW2PYh1aUl0JSmkETPR7LEebK52VrSZVx4sEc5fQzMraxdPl8M37x+HSi19tpQKU3kcwp2UUyoWQXyYSSXSQTSnaRTPT0avzwyAo/c91bSmPJZgR9ZBqJMdAaFscWEjubn4+v4s/OlV/Fnzme6BzRTHSqSIzHJpIWnaebuhovkjslu0gmlOwimVCyi2RCyS6SCSW7SCbazghjZucD3wTOpTWnz053/6qZ3QJ8FjjZa+Bmd/9xaluNRoOxlWOlsUpFqArlOoBUudETh2Rlc0Xp8rmxM8J1ZmfHE7G4Q87cTKJDznxqHLRUByAZTMHrxxJj8jWC8/TC0XCVpUzZPA980d0fNbNx4BEze6CI3e7uX1nCNkSkz5Yy19skMFk8Pmpm+4DN3W6YiNRrWd/ZzWwLcCmtGVwBbjCz3WZ2p5mtrbltIlKjJSe7ma0G7gFudPcjwNeAi4BttM78twbr7TCzXWa2q7kQT0MsIt21pHvjzWwE+BFwv7vfVhLfAvzI3S9JbWd0xUo/Z9OFpbHBuUAXawZD5szNxZNYpC7C6QKdtNR7gc59vtq98WZmwB3AvsWJbmaLZ7O/Btjbblsi0j9LuRr/XuDTwB4ze6xYdjNwnZlto3UynAA+125DZsbwcPkuPXFOjSKeGEcsNZmUJ9/j4nY0Ldifx9sziw/x0FC83ujoSBibnV0ZxuZmgimI5lO971KD+ckblZdfWxJn4tE4tiKIDQ/HPTc9+JQ5fSz+1LeUq/E/ozx3kjV1ERksuoNOJBNKdpFMKNlFMqFkF8mEkl0kE0spvdXG3VkI7qJbSJTRwu014zvyUjfHVLwXh6iJzcRNOs1maotxbGg4Lr2d0Yj/bCNBGWdhoby3IaRvCmouJP4uqfm3ooOVPPiJY5UoUw4F5VyA4eBY2XDq2MfbazRSd2vFIUv936JYKiei11yifTqzi2RCyS6SCSW7SCaU7CKZULKLZELJLpKJnpbems0mx0+cCKKp8lV5rHq/9LikUWXOuWBqrbb7SpWa0r32EpuM1kn8xxpR32igMRT3vBpKxMJSU6IdqfJUqv2p9Sw4Im6J107i4M+nXiCpNib+omF5s8I8jMm/87K3JiKnJSW7SCaU7CKZULKLZELJLpIJJbtIJnpfejt+LIimik3l5hPj0KcGo0wWNCr1ykqOW53aYKVQWoUeVMnNxQ1JD1od9cpKnF8qH6pE6S1of7L0lurZVrEomv5zhgXTZe9LpTcRUbKL5ELJLpIJJbtIJpTsIploezXezM4AHqI1780w8H13/5KZXQB8B1gHPAp82t3jmQppXSGffT2YsLDCTf/JMdCqTRXZBZUvq9es6hRPdZ8Pqh6PVKenRAeaKvtLXo1PSHXkqbvyEuVLouqylL/kDPBhd/9DWtMzX2Fm7wG+DNzu7luBV4HPLLO5ItJDbZPdW04Wx0eKfw58GPh+sfwu4BNdaaGI1GJJn9HMbKiYwfUg8ADwDHDY3U/eV7Ef2NydJopIHZaU7O6+4O7bgPOAy4F3lD2tbF0z22Fmu8xs1+B8jxbJz7Kuvrj7YeA/gfcAZ9lvJx8/D3gxWGenu2939+2Dc7FKJD9tk93Mzjazs4rHK4GPAvuAnwJ/XjzteuDebjVSRDpnqRvnAczsXbQuwA3RenO4293/wcwu5Lelt18Bf+nuM2225XG1r8pH/KpfC6pODjUoevkJqerxqNK5o2oHlCrHoxvHMLXNukuY0XGcw718zrG2yV4nJXtdlOxLj9W5Tifb7H+y6w46kUwo2UUyoWQXyYSSXSQTSnaRTPR0DDrgJZj/TfF4Q+v3vjsN29HVikFNx6Pj6sop7ehbleR0e328LQr0tPT2hh2b7WrdVddfaofakUs79DFeJBNKdpFM9DPZd/Zx34upHW+kdrzRm6YdffvOLiK9pY/xIpnoS7Kb2RVm9qSZPW1mN/WjDUU7Jsxsj5k91hpco2f7vdPMDprZ3kXL1pnZA2b26+Ln2j614xYze6E4Jo+Z2ZU9aMf5ZvZTM9tnZk+Y2V8Xy3t6TBLt6OkxMbMzzOyXZvZ40Y6/L5ZfYGa/KI7Hd81sdFkbdvee/qPVVfYZ4EJgFHgcuLjX7SjaMgFs6MN+PwC8G9i7aNk/ATcVj28CvtyndtwC/E2Pj8cm4N3F43HgKeDiXh+TRDt6ekxodZ9bXTweAX5Ba8CYu4Fri+X/DHx+Odvtx5n9cuBpd3/WW0NPfwe4ug/t6Bt3fwh45ZTFV9MaNwB6NIBn0I6ec/dJd3+0eHyU1uAom+nxMUm0o6e8pfZBXvuR7JuB5xf93s/BKh34iZk9YmY7+tSGkza6+yS0XnTAOX1syw1mtrv4mN/1rxOLmdkW4FJaZ7O+HZNT2gE9PibdGOS1H8le1rG+XyWB97r7u4E/Ab5gZh/oUzsGydeAi2jNETAJ3NqrHZvZauAe4EZ3P9Kr/S6hHT0/Jt7BIK+RfiT7fuD8Rb+Hg1V2m7u/WPw8CPyQ1kHtlykz2wRQ/DzYj0a4+1TxQmsCX6dHx8TMRmgl2Lfc/QfF4p4fk7J29OuYFPte9iCvkX4k+8PA1uLK4ihwLXBfrxthZqvMbPzkY+DjwN70Wl11H62BO6GPA3ieTK7CNfTgmJiZAXcA+9z9tkWhnh6TqB29PiZdG+S1V1cYT7naeCWtK53PAH/bpzZcSKsS8DjwRC/bAXyb1sfBOVqfdD4DrAceBH5d/FzXp3b8K7AH2E0r2Tb1oB3vo/WRdDfwWPHvyl4fk0Q7enpMgHfRGsR1N603lr9b9Jr9JfA08D1gxXK2qzvoRDKhO+hEMqFkF8mEkl0kE0p2kUwo2UUyoWQXyYSSXSQTSnaRTPwfpv2KpQ8u3voAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_data(x_train[250])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convert class vectors to binary class matrices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train = keras.utils.to_categorical(y_train, num_classes)\n",
    "y_test = keras.utils.to_categorical(y_test, num_classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Function for auto-scaling learning based on epochs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def lr_schedule(epoch):\n",
    "    \"\"\"Learning Rate Schedule\n",
    "\n",
    "    Learning rate is scheduled to be reduced after 80, 120, 160, 180 epochs.\n",
    "    Called automatically every epoch as part of callbacks during training.\n",
    "\n",
    "    # Arguments\n",
    "        epoch (int): The number of epochs\n",
    "\n",
    "    # Returns\n",
    "        lr (float32): learning rate\n",
    "    \"\"\"\n",
    "    lr = 1e-3\n",
    "    if epoch > 180:\n",
    "        lr *= 0.5e-3\n",
    "    elif epoch > 160:\n",
    "        lr *= 1e-3\n",
    "    elif epoch > 120:\n",
    "        lr *= 1e-2\n",
    "    elif epoch > 80:\n",
    "        lr *= 1e-1\n",
    "    print('Learning rate: ', lr)\n",
    "    return lr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Function of a single ResNet layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def resnet_layer(inputs,\n",
    "                 num_filters=16,\n",
    "                 kernel_size=3,\n",
    "                 strides=1,\n",
    "                 activation='relu',\n",
    "                 batch_normalization=True,\n",
    "                 conv_first=True):\n",
    "    \"\"\"2D Convolution-Batch Normalization-Activation stack builder\n",
    "\n",
    "    # Arguments\n",
    "        inputs (tensor): input tensor from input image or previous layer\n",
    "        num_filters (int): Conv2D number of filters\n",
    "        kernel_size (int): Conv2D square kernel dimensions\n",
    "        strides (int): Conv2D square stride dimensions\n",
    "        activation (string): activation name\n",
    "        batch_normalization (bool): whether to include batch normalization\n",
    "        conv_first (bool): conv-bn-activation (True) or bn-activation-conv (False)\n",
    "\n",
    "    # Returns\n",
    "        x (tensor): tensor as input to the next layer\n",
    "    \"\"\"\n",
    "    conv = Conv2D(num_filters,\n",
    "                  kernel_size=kernel_size,\n",
    "                  strides=strides,\n",
    "                  padding='same',\n",
    "                  kernel_initializer='he_normal',\n",
    "                  kernel_regularizer=l2(1e-4))\n",
    "\n",
    "    x = inputs\n",
    "    if conv_first:\n",
    "        x = conv(x)\n",
    "        if batch_normalization:\n",
    "            x = BatchNormalization()(x)\n",
    "        if activation is not None:\n",
    "            x = Activation(activation)(x)\n",
    "    else:\n",
    "        if batch_normalization:\n",
    "            x = BatchNormalization()(x)\n",
    "        if activation is not None:\n",
    "            x = Activation(activation)(x)\n",
    "        x = conv(x)\n",
    "    return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ResNet Version 1 model builder "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def resnet_v1(input_shape, depth, num_classes=10):\n",
    "    \"\"\"ResNet Version 1 Model builder [a]\n",
    "\n",
    "    Stacks of 2 x (3 x 3) Conv2D-BN-ReLU\n",
    "    Last ReLU is after the shortcut connection.\n",
    "    At the beginning of each stage, the feature map size is halved (downsampled)\n",
    "    by a convolutional layer with strides=2, while the number of filters is\n",
    "    doubled. Within each stage, the layers have the same number filters and the\n",
    "    same number of filters.\n",
    "    Features maps sizes:\n",
    "    stage 0: 32x32, 16\n",
    "    stage 1: 16x16, 32\n",
    "    stage 2:  8x8,  64\n",
    "    The Number of parameters is approx the same as Table 6 of [a]:\n",
    "    ResNet20 0.27M\n",
    "    ResNet32 0.46M\n",
    "    ResNet44 0.66M\n",
    "    ResNet56 0.85M\n",
    "    ResNet110 1.7M\n",
    "\n",
    "    # Arguments\n",
    "        input_shape (tensor): shape of input image tensor\n",
    "        depth (int): number of core convolutional layers\n",
    "        num_classes (int): number of classes (CIFAR10 has 10)\n",
    "\n",
    "    # Returns\n",
    "        model (Model): Keras model instance\n",
    "    \"\"\"\n",
    "    if (depth - 2) % 6 != 0:\n",
    "        raise ValueError('depth should be 6n+2 (eg 20, 32, 44 in [a])')\n",
    "    # Start model definition.\n",
    "    num_filters = 16\n",
    "    num_res_blocks = int((depth - 2) / 6)\n",
    "\n",
    "    inputs = Input(shape=input_shape)\n",
    "    x = resnet_layer(inputs=inputs)\n",
    "    # Instantiate the stack of residual units\n",
    "    for stack in range(3):\n",
    "        for res_block in range(num_res_blocks):\n",
    "            strides = 1\n",
    "            if stack > 0 and res_block == 0:  # first layer but not first stack\n",
    "                strides = 2  # downsample\n",
    "            y = resnet_layer(inputs=x,\n",
    "                             num_filters=num_filters,\n",
    "                             strides=strides)\n",
    "            y = resnet_layer(inputs=y,\n",
    "                             num_filters=num_filters,\n",
    "                             activation=None)\n",
    "            if stack > 0 and res_block == 0:  # first layer but not first stack\n",
    "                # linear projection residual shortcut connection to match\n",
    "                # changed dims\n",
    "                x = resnet_layer(inputs=x,\n",
    "                                 num_filters=num_filters,\n",
    "                                 kernel_size=1,\n",
    "                                 strides=strides,\n",
    "                                 activation=None,\n",
    "                                 batch_normalization=False)\n",
    "            x = keras.layers.add([x, y])\n",
    "            x = Activation('relu')(x)\n",
    "        num_filters *= 2\n",
    "\n",
    "    # Add classifier on top.\n",
    "    # v1 does not use BN after last shortcut connection-ReLU\n",
    "    x = AveragePooling2D(pool_size=8)(x)\n",
    "    y = Flatten()(x)\n",
    "    outputs = Dense(num_classes,\n",
    "                    activation='softmax',\n",
    "                    kernel_initializer='he_normal')(y)\n",
    "\n",
    "    # Instantiate model.\n",
    "    model = Model(inputs=inputs, outputs=outputs)\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ResNet Version 2 model builder "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "def resnet_v2(input_shape, depth, num_classes=10):\n",
    "    \"\"\"ResNet Version 2 Model builder [b]\n",
    "\n",
    "    Stacks of (1 x 1)-(3 x 3)-(1 x 1) BN-ReLU-Conv2D or also known as\n",
    "    bottleneck layer\n",
    "    First shortcut connection per layer is 1 x 1 Conv2D.\n",
    "    Second and onwards shortcut connection is identity.\n",
    "    At the beginning of each stage, the feature map size is halved (downsampled)\n",
    "    by a convolutional layer with strides=2, while the number of filter maps is\n",
    "    doubled. Within each stage, the layers have the same number filters and the\n",
    "    same filter map sizes.\n",
    "    Features maps sizes:\n",
    "    conv1  : 32x32,  16\n",
    "    stage 0: 32x32,  64\n",
    "    stage 1: 16x16, 128\n",
    "    stage 2:  8x8,  256\n",
    "\n",
    "    # Arguments\n",
    "        input_shape (tensor): shape of input image tensor\n",
    "        depth (int): number of core convolutional layers\n",
    "        num_classes (int): number of classes (CIFAR10 has 10)\n",
    "\n",
    "    # Returns\n",
    "        model (Model): Keras model instance\n",
    "    \"\"\"\n",
    "    if (depth - 2) % 9 != 0:\n",
    "        raise ValueError('depth should be 9n+2 (eg 56 or 110 in [b])')\n",
    "    # Start model definition.\n",
    "    num_filters_in = 16\n",
    "    num_res_blocks = int((depth - 2) / 9)\n",
    "\n",
    "    inputs = Input(shape=input_shape)\n",
    "    # v2 performs Conv2D with BN-ReLU on input before splitting into 2 paths\n",
    "    x = resnet_layer(inputs=inputs,\n",
    "                     num_filters=num_filters_in,\n",
    "                     conv_first=True)\n",
    "\n",
    "    # Instantiate the stack of residual units\n",
    "    for stage in range(3):\n",
    "        for res_block in range(num_res_blocks):\n",
    "            activation = 'relu'\n",
    "            batch_normalization = True\n",
    "            strides = 1\n",
    "            if stage == 0:\n",
    "                num_filters_out = num_filters_in * 4\n",
    "                if res_block == 0:  # first layer and first stage\n",
    "                    activation = None\n",
    "                    batch_normalization = False\n",
    "            else:\n",
    "                num_filters_out = num_filters_in * 2\n",
    "                if res_block == 0:  # first layer but not first stage\n",
    "                    strides = 2    # downsample\n",
    "\n",
    "            # bottleneck residual unit\n",
    "            y = resnet_layer(inputs=x,\n",
    "                             num_filters=num_filters_in,\n",
    "                             kernel_size=1,\n",
    "                             strides=strides,\n",
    "                             activation=activation,\n",
    "                             batch_normalization=batch_normalization,\n",
    "                             conv_first=False)\n",
    "            y = resnet_layer(inputs=y,\n",
    "                             num_filters=num_filters_in,\n",
    "                             conv_first=False)\n",
    "            y = resnet_layer(inputs=y,\n",
    "                             num_filters=num_filters_out,\n",
    "                             kernel_size=1,\n",
    "                             conv_first=False)\n",
    "            if res_block == 0:\n",
    "                # linear projection residual shortcut connection to match\n",
    "                # changed dims\n",
    "                x = resnet_layer(inputs=x,\n",
    "                                 num_filters=num_filters_out,\n",
    "                                 kernel_size=1,\n",
    "                                 strides=strides,\n",
    "                                 activation=None,\n",
    "                                 batch_normalization=False)\n",
    "            x = keras.layers.add([x, y])\n",
    "\n",
    "        num_filters_in = num_filters_out\n",
    "\n",
    "    # Add classifier on top.\n",
    "    # v2 has BN-ReLU before Pooling\n",
    "    x = BatchNormalization()(x)\n",
    "    x = Activation('relu')(x)\n",
    "    x = AveragePooling2D(pool_size=8)(x)\n",
    "    y = Flatten()(x)\n",
    "    outputs = Dense(num_classes,\n",
    "                    activation='softmax',\n",
    "                    kernel_initializer='he_normal')(y)\n",
    "\n",
    "    # Instantiate model.\n",
    "    model = Model(inputs=inputs, outputs=outputs)\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Choose model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def choose_model(version=1):\n",
    "    \"\"\"\n",
    "    Chooses the model based on version\n",
    "    \n",
    "    # Argument:\n",
    "        An (int) version number (1 or 2)\n",
    "    # Returns:\n",
    "        keras model\n",
    "    \"\"\"\n",
    "    if version == 2:\n",
    "        model = resnet_v2(input_shape=input_shape, depth=depth)\n",
    "    else:\n",
    "        model = resnet_v1(input_shape=input_shape, depth=depth)\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Choose version, parameter `n`, compile, and show summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "version = 1\n",
    "n = 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "depth = model_depth(version=version,n=n)\n",
    "model = choose_model(version)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ResNet20v1\n"
     ]
    }
   ],
   "source": [
    "print(model_type(version,n))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Learning rate:  0.001\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_3 (InputLayer)            (None, 32, 32, 3)    0                                            \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_43 (Conv2D)              (None, 32, 32, 16)   448         input_3[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_39 (BatchNo (None, 32, 32, 16)   64          conv2d_43[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "activation_39 (Activation)      (None, 32, 32, 16)   0           batch_normalization_39[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_44 (Conv2D)              (None, 32, 32, 16)   2320        activation_39[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_40 (BatchNo (None, 32, 32, 16)   64          conv2d_44[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "activation_40 (Activation)      (None, 32, 32, 16)   0           batch_normalization_40[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_45 (Conv2D)              (None, 32, 32, 16)   2320        activation_40[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_41 (BatchNo (None, 32, 32, 16)   64          conv2d_45[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "add_19 (Add)                    (None, 32, 32, 16)   0           activation_39[0][0]              \n",
      "                                                                 batch_normalization_41[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "activation_41 (Activation)      (None, 32, 32, 16)   0           add_19[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_46 (Conv2D)              (None, 32, 32, 16)   2320        activation_41[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_42 (BatchNo (None, 32, 32, 16)   64          conv2d_46[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "activation_42 (Activation)      (None, 32, 32, 16)   0           batch_normalization_42[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_47 (Conv2D)              (None, 32, 32, 16)   2320        activation_42[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_43 (BatchNo (None, 32, 32, 16)   64          conv2d_47[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "add_20 (Add)                    (None, 32, 32, 16)   0           activation_41[0][0]              \n",
      "                                                                 batch_normalization_43[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "activation_43 (Activation)      (None, 32, 32, 16)   0           add_20[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_48 (Conv2D)              (None, 32, 32, 16)   2320        activation_43[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_44 (BatchNo (None, 32, 32, 16)   64          conv2d_48[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "activation_44 (Activation)      (None, 32, 32, 16)   0           batch_normalization_44[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_49 (Conv2D)              (None, 32, 32, 16)   2320        activation_44[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_45 (BatchNo (None, 32, 32, 16)   64          conv2d_49[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "add_21 (Add)                    (None, 32, 32, 16)   0           activation_43[0][0]              \n",
      "                                                                 batch_normalization_45[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "activation_45 (Activation)      (None, 32, 32, 16)   0           add_21[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_50 (Conv2D)              (None, 16, 16, 32)   4640        activation_45[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_46 (BatchNo (None, 16, 16, 32)   128         conv2d_50[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "activation_46 (Activation)      (None, 16, 16, 32)   0           batch_normalization_46[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_51 (Conv2D)              (None, 16, 16, 32)   9248        activation_46[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_52 (Conv2D)              (None, 16, 16, 32)   544         activation_45[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_47 (BatchNo (None, 16, 16, 32)   128         conv2d_51[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "add_22 (Add)                    (None, 16, 16, 32)   0           conv2d_52[0][0]                  \n",
      "                                                                 batch_normalization_47[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "activation_47 (Activation)      (None, 16, 16, 32)   0           add_22[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_53 (Conv2D)              (None, 16, 16, 32)   9248        activation_47[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_48 (BatchNo (None, 16, 16, 32)   128         conv2d_53[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "activation_48 (Activation)      (None, 16, 16, 32)   0           batch_normalization_48[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_54 (Conv2D)              (None, 16, 16, 32)   9248        activation_48[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_49 (BatchNo (None, 16, 16, 32)   128         conv2d_54[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "add_23 (Add)                    (None, 16, 16, 32)   0           activation_47[0][0]              \n",
      "                                                                 batch_normalization_49[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "activation_49 (Activation)      (None, 16, 16, 32)   0           add_23[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_55 (Conv2D)              (None, 16, 16, 32)   9248        activation_49[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_50 (BatchNo (None, 16, 16, 32)   128         conv2d_55[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "activation_50 (Activation)      (None, 16, 16, 32)   0           batch_normalization_50[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_56 (Conv2D)              (None, 16, 16, 32)   9248        activation_50[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_51 (BatchNo (None, 16, 16, 32)   128         conv2d_56[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "add_24 (Add)                    (None, 16, 16, 32)   0           activation_49[0][0]              \n",
      "                                                                 batch_normalization_51[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "activation_51 (Activation)      (None, 16, 16, 32)   0           add_24[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_57 (Conv2D)              (None, 8, 8, 64)     18496       activation_51[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_52 (BatchNo (None, 8, 8, 64)     256         conv2d_57[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "activation_52 (Activation)      (None, 8, 8, 64)     0           batch_normalization_52[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_58 (Conv2D)              (None, 8, 8, 64)     36928       activation_52[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_59 (Conv2D)              (None, 8, 8, 64)     2112        activation_51[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_53 (BatchNo (None, 8, 8, 64)     256         conv2d_58[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "add_25 (Add)                    (None, 8, 8, 64)     0           conv2d_59[0][0]                  \n",
      "                                                                 batch_normalization_53[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "activation_53 (Activation)      (None, 8, 8, 64)     0           add_25[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_60 (Conv2D)              (None, 8, 8, 64)     36928       activation_53[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_54 (BatchNo (None, 8, 8, 64)     256         conv2d_60[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "activation_54 (Activation)      (None, 8, 8, 64)     0           batch_normalization_54[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_61 (Conv2D)              (None, 8, 8, 64)     36928       activation_54[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_55 (BatchNo (None, 8, 8, 64)     256         conv2d_61[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "add_26 (Add)                    (None, 8, 8, 64)     0           activation_53[0][0]              \n",
      "                                                                 batch_normalization_55[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "activation_55 (Activation)      (None, 8, 8, 64)     0           add_26[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_62 (Conv2D)              (None, 8, 8, 64)     36928       activation_55[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_56 (BatchNo (None, 8, 8, 64)     256         conv2d_62[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "activation_56 (Activation)      (None, 8, 8, 64)     0           batch_normalization_56[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "conv2d_63 (Conv2D)              (None, 8, 8, 64)     36928       activation_56[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "batch_normalization_57 (BatchNo (None, 8, 8, 64)     256         conv2d_63[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "add_27 (Add)                    (None, 8, 8, 64)     0           activation_55[0][0]              \n",
      "                                                                 batch_normalization_57[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "activation_57 (Activation)      (None, 8, 8, 64)     0           add_27[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "average_pooling2d_3 (AveragePoo (None, 1, 1, 64)     0           activation_57[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "flatten_3 (Flatten)             (None, 64)           0           average_pooling2d_3[0][0]        \n",
      "__________________________________________________________________________________________________\n",
      "dense_3 (Dense)                 (None, 10)           650         flatten_3[0][0]                  \n",
      "==================================================================================================\n",
      "Total params: 274,442\n",
      "Trainable params: 273,066\n",
      "Non-trainable params: 1,376\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer=Adam(lr=lr_schedule(0)),\n",
    "              metrics=['accuracy'])\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are, in total, 274442 parameters in this model!\n"
     ]
    }
   ],
   "source": [
    "print(\"There are, in total, {} parameters in this model!\".format(model.count_params()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare model saving directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "save_dir = os.path.join(os.getcwd(), 'saved_models')\n",
    "model_name = 'cifar10_%s_model.{epoch:03d}.h5' % model_type\n",
    "if not os.path.isdir(save_dir):\n",
    "    os.makedirs(save_dir)\n",
    "filepath = os.path.join(save_dir, model_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare callbacks for model saving and for learning rate adjustment\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "checkpoint = ModelCheckpoint(filepath=filepath,\n",
    "                             monitor='val_acc',\n",
    "                             verbose=1,\n",
    "                             save_best_only=True,\n",
    "                             period=3\n",
    "                            )\n",
    "\n",
    "lr_scheduler = LearningRateScheduler(lr_schedule)\n",
    "\n",
    "lr_reducer = ReduceLROnPlateau(factor=np.sqrt(0.1),\n",
    "                               cooldown=0,\n",
    "                               patience=5,\n",
    "                               min_lr=0.5e-6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For now, we will not save the models, so won't include the `checkpoint` in the `callbacks`\n",
    "callbacks = [lr_reducer, lr_scheduler]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training (sit back and enjoy a cuppa, because this is gonna take some time!)\n",
    "First start with a very small number of epochs to check the training speed on your particular hardware. Thereafter, adjust accordingly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "from time import time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 32  # orig paper trained all networks with batch_size=128\n",
    "epochs = 20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 50000 samples, validate on 10000 samples\n",
      "Epoch 1/20\n",
      "Learning rate:  0.001\n",
      "50000/50000 [==============================] - 56s 1ms/step - loss: 1.5058 - acc: 0.5118 - val_loss: 1.5499 - val_acc: 0.5298\n",
      "Epoch 2/20\n",
      "Learning rate:  0.001\n",
      "50000/50000 [==============================] - 44s 883us/step - loss: 1.0880 - acc: 0.6704 - val_loss: 1.4282 - val_acc: 0.5853\n",
      "Epoch 3/20\n",
      "Learning rate:  0.001\n",
      "50000/50000 [==============================] - 44s 886us/step - loss: 0.9160 - acc: 0.7345 - val_loss: 1.1339 - val_acc: 0.6702\n",
      "Epoch 4/20\n",
      "Learning rate:  0.001\n",
      "50000/50000 [==============================] - 44s 882us/step - loss: 0.8145 - acc: 0.7745 - val_loss: 0.8868 - val_acc: 0.7585\n",
      "Epoch 5/20\n",
      "Learning rate:  0.001\n",
      "50000/50000 [==============================] - 44s 883us/step - loss: 0.7448 - acc: 0.8023 - val_loss: 1.0334 - val_acc: 0.7252\n",
      "Epoch 6/20\n",
      "Learning rate:  0.001\n",
      "50000/50000 [==============================] - 46s 915us/step - loss: 0.6937 - acc: 0.8214 - val_loss: 0.8521 - val_acc: 0.7684\n",
      "Epoch 7/20\n",
      "Learning rate:  0.001\n",
      "50000/50000 [==============================] - 44s 888us/step - loss: 0.6477 - acc: 0.8404 - val_loss: 1.0899 - val_acc: 0.7255\n",
      "Epoch 8/20\n",
      "Learning rate:  0.001\n",
      "50000/50000 [==============================] - 45s 894us/step - loss: 0.6116 - acc: 0.8540 - val_loss: 0.9564 - val_acc: 0.7511\n",
      "Epoch 9/20\n",
      "Learning rate:  0.001\n",
      "50000/50000 [==============================] - 45s 893us/step - loss: 0.5839 - acc: 0.8664 - val_loss: 1.0141 - val_acc: 0.7446\n",
      "Epoch 10/20\n",
      "Learning rate:  0.001\n",
      "50000/50000 [==============================] - 45s 897us/step - loss: 0.5591 - acc: 0.8764 - val_loss: 0.9252 - val_acc: 0.7786\n",
      "Epoch 11/20\n",
      "Learning rate:  0.001\n",
      "50000/50000 [==============================] - 45s 898us/step - loss: 0.5380 - acc: 0.8847 - val_loss: 0.9336 - val_acc: 0.7706\n",
      "Epoch 12/20\n",
      "Learning rate:  0.001\n",
      "50000/50000 [==============================] - 46s 925us/step - loss: 0.5214 - acc: 0.8937 - val_loss: 0.9568 - val_acc: 0.7729\n",
      "Epoch 13/20\n",
      "Learning rate:  0.001\n",
      "50000/50000 [==============================] - 47s 934us/step - loss: 0.5056 - acc: 0.9002 - val_loss: 1.2603 - val_acc: 0.7169\n",
      "Epoch 14/20\n",
      "Learning rate:  0.001\n",
      "50000/50000 [==============================] - 45s 895us/step - loss: 0.4919 - acc: 0.9070 - val_loss: 1.3367 - val_acc: 0.7052\n",
      "Epoch 15/20\n",
      "Learning rate:  0.001\n",
      "50000/50000 [==============================] - 47s 934us/step - loss: 0.4843 - acc: 0.9121 - val_loss: 1.1203 - val_acc: 0.7629\n",
      "Epoch 16/20\n",
      "Learning rate:  0.001\n",
      "50000/50000 [==============================] - 45s 893us/step - loss: 0.4765 - acc: 0.9171 - val_loss: 1.0507 - val_acc: 0.7716\n",
      "Epoch 17/20\n",
      "Learning rate:  0.001\n",
      "50000/50000 [==============================] - 48s 951us/step - loss: 0.4738 - acc: 0.9184 - val_loss: 0.9458 - val_acc: 0.7960\n",
      "Epoch 18/20\n",
      "Learning rate:  0.001\n",
      "50000/50000 [==============================] - 45s 910us/step - loss: 0.4654 - acc: 0.9230 - val_loss: 1.1894 - val_acc: 0.7486\n",
      "Epoch 19/20\n",
      "Learning rate:  0.001\n",
      "50000/50000 [==============================] - 46s 912us/step - loss: 0.4630 - acc: 0.9265 - val_loss: 1.2325 - val_acc: 0.7490\n",
      "Epoch 20/20\n",
      "Learning rate:  0.001\n",
      "50000/50000 [==============================] - 45s 906us/step - loss: 0.4568 - acc: 0.9293 - val_loss: 1.0133 - val_acc: 0.7787\n"
     ]
    }
   ],
   "source": [
    "t1 = time()\n",
    "model.fit(x_train, y_train,\n",
    "              batch_size=batch_size,\n",
    "              epochs=epochs,\n",
    "              verbose=1,\n",
    "              validation_data=(x_test, y_test),\n",
    "              shuffle=True,\n",
    "              callbacks=callbacks)\n",
    "t2 = time()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training of 20 epochs took 15.3 minutes total.\n"
     ]
    }
   ],
   "source": [
    "time_delta=round((t2-t1)/60,2)\n",
    "\n",
    "print (\"Training of {} epochs took {} minutes total.\".format(epochs, time_delta))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Score trained model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000/10000 [==============================] - 3s 282us/step\n",
      "Test loss: 1.0133228489875794\n",
      "Test accuracy: 0.7787\n"
     ]
    }
   ],
   "source": [
    "scores = model.evaluate(x_test, y_test, verbose=1)\n",
    "print('Test loss:', scores[0])\n",
    "print('Test accuracy:', scores[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A smaller learning rate and larger batch size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "version = 1\n",
    "n = 3\n",
    "\n",
    "depth = model_depth(version=version,n=n)\n",
    "model = choose_model(version)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Hard coding the learning rate = 1e-4\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer=Adam(lr=1e-4),\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 128  # orig paper trained all networks with batch_size=128\n",
    "epochs = 20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 50000 samples, validate on 10000 samples\n",
      "Epoch 1/20\n",
      "50000/50000 [==============================] - 36s 720us/step - loss: 2.3569 - acc: 0.2939 - val_loss: 1.8272 - val_acc: 0.3618\n",
      "Epoch 2/20\n",
      "50000/50000 [==============================] - 23s 469us/step - loss: 1.6489 - acc: 0.4519 - val_loss: 1.6397 - val_acc: 0.4504\n",
      "Epoch 3/20\n",
      "50000/50000 [==============================] - 24s 474us/step - loss: 1.5019 - acc: 0.5100 - val_loss: 1.4866 - val_acc: 0.5148\n",
      "Epoch 4/20\n",
      "50000/50000 [==============================] - 24s 479us/step - loss: 1.4017 - acc: 0.5501 - val_loss: 1.4173 - val_acc: 0.5457\n",
      "Epoch 5/20\n",
      "50000/50000 [==============================] - 24s 484us/step - loss: 1.3207 - acc: 0.5802 - val_loss: 1.3643 - val_acc: 0.5613\n",
      "Epoch 6/20\n",
      "50000/50000 [==============================] - 24s 480us/step - loss: 1.2535 - acc: 0.6048 - val_loss: 1.3097 - val_acc: 0.5914\n",
      "Epoch 7/20\n",
      "50000/50000 [==============================] - 24s 482us/step - loss: 1.1997 - acc: 0.6240 - val_loss: 1.3519 - val_acc: 0.5710\n",
      "Epoch 8/20\n",
      "50000/50000 [==============================] - 24s 483us/step - loss: 1.1461 - acc: 0.6448 - val_loss: 1.2997 - val_acc: 0.5953\n",
      "Epoch 9/20\n",
      "50000/50000 [==============================] - 24s 488us/step - loss: 1.1043 - acc: 0.6593 - val_loss: 1.3115 - val_acc: 0.5915\n",
      "Epoch 10/20\n",
      "50000/50000 [==============================] - 24s 488us/step - loss: 1.0596 - acc: 0.6756 - val_loss: 1.2338 - val_acc: 0.6173\n",
      "Epoch 11/20\n",
      "50000/50000 [==============================] - 24s 485us/step - loss: 1.0229 - acc: 0.6894 - val_loss: 1.1934 - val_acc: 0.6316\n",
      "Epoch 12/20\n",
      "50000/50000 [==============================] - 24s 485us/step - loss: 0.9888 - acc: 0.7012 - val_loss: 1.2169 - val_acc: 0.6242\n",
      "Epoch 13/20\n",
      "50000/50000 [==============================] - 24s 488us/step - loss: 0.9513 - acc: 0.7161 - val_loss: 1.2631 - val_acc: 0.6203\n",
      "Epoch 14/20\n",
      "50000/50000 [==============================] - 25s 509us/step - loss: 0.9161 - acc: 0.7306 - val_loss: 1.2307 - val_acc: 0.6240\n",
      "Epoch 15/20\n",
      "50000/50000 [==============================] - 24s 486us/step - loss: 0.8845 - acc: 0.7398 - val_loss: 1.1589 - val_acc: 0.6480\n",
      "Epoch 16/20\n",
      "50000/50000 [==============================] - 26s 517us/step - loss: 0.8525 - acc: 0.7534 - val_loss: 1.2173 - val_acc: 0.6284\n",
      "Epoch 17/20\n",
      "50000/50000 [==============================] - 24s 487us/step - loss: 0.8201 - acc: 0.7663 - val_loss: 1.2008 - val_acc: 0.6338\n",
      "Epoch 18/20\n",
      "50000/50000 [==============================] - 25s 498us/step - loss: 0.7928 - acc: 0.7757 - val_loss: 1.1804 - val_acc: 0.6399\n",
      "Epoch 19/20\n",
      "50000/50000 [==============================] - 25s 492us/step - loss: 0.7627 - acc: 0.7872 - val_loss: 1.3991 - val_acc: 0.5958\n",
      "Epoch 20/20\n",
      "50000/50000 [==============================] - 24s 489us/step - loss: 0.7325 - acc: 0.7977 - val_loss: 1.2367 - val_acc: 0.6341\n"
     ]
    }
   ],
   "source": [
    "# Note, we turn off the callbacks as we want to use the new learning rate\n",
    "t1 = time()\n",
    "model.fit(x_train, y_train,\n",
    "              batch_size=batch_size,\n",
    "              epochs=epochs,\n",
    "              verbose=1,\n",
    "              validation_data=(x_test, y_test),\n",
    "              shuffle=True)\n",
    "t2 = time()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training of 20 epochs took 8.36 minutes total.\n"
     ]
    }
   ],
   "source": [
    "time_delta=round((t2-t1)/60,2)\n",
    "\n",
    "print (\"Training of {} epochs took {} minutes total.\".format(epochs, time_delta))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### With ResNet v2 model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model type: ResNet47v2\n",
      "There are, in total, 1398826 parameters in this model!\n"
     ]
    }
   ],
   "source": [
    "version = 2\n",
    "n = 5\n",
    "\n",
    "depth = model_depth(version=version,n=n)\n",
    "model = choose_model(version)\n",
    "\n",
    "print(\"Model type:\",model_type(version,n))\n",
    "print(\"There are, in total, {} parameters in this model!\".format(model.count_params()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Hard coding the learning rate = 3e-4\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer=Adam(lr=3e-4),\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 128  # orig paper trained all networks with batch_size=128\n",
    "epochs = 20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 50000 samples, validate on 10000 samples\n",
      "Epoch 1/20\n",
      "50000/50000 [==============================] - 110s 2ms/step - loss: 2.4201 - acc: 0.4460 - val_loss: 2.2206 - val_acc: 0.4953\n",
      "Epoch 2/20\n",
      "50000/50000 [==============================] - 80s 2ms/step - loss: 1.9250 - acc: 0.5916 - val_loss: 1.8745 - val_acc: 0.5946\n",
      "Epoch 3/20\n",
      "50000/50000 [==============================] - 80s 2ms/step - loss: 1.6684 - acc: 0.6650 - val_loss: 1.7502 - val_acc: 0.6214\n",
      "Epoch 4/20\n",
      "50000/50000 [==============================] - 81s 2ms/step - loss: 1.4783 - acc: 0.7201 - val_loss: 1.7839 - val_acc: 0.5964\n",
      "Epoch 5/20\n",
      "50000/50000 [==============================] - 81s 2ms/step - loss: 1.3153 - acc: 0.7691 - val_loss: 1.9695 - val_acc: 0.5663\n",
      "Epoch 6/20\n",
      "50000/50000 [==============================] - 81s 2ms/step - loss: 1.1650 - acc: 0.8159 - val_loss: 2.0570 - val_acc: 0.5618\n",
      "Epoch 7/20\n",
      "50000/50000 [==============================] - 81s 2ms/step - loss: 1.0283 - acc: 0.8606 - val_loss: 1.7825 - val_acc: 0.6336\n",
      "Epoch 8/20\n",
      "50000/50000 [==============================] - 81s 2ms/step - loss: 0.9047 - acc: 0.9012 - val_loss: 2.1000 - val_acc: 0.6138\n",
      "Epoch 9/20\n",
      "50000/50000 [==============================] - 81s 2ms/step - loss: 0.8266 - acc: 0.9233 - val_loss: 2.4783 - val_acc: 0.5880\n",
      "Epoch 10/20\n",
      "50000/50000 [==============================] - 81s 2ms/step - loss: 0.7639 - acc: 0.9430 - val_loss: 2.8940 - val_acc: 0.5157\n",
      "Epoch 11/20\n",
      "50000/50000 [==============================] - 81s 2ms/step - loss: 0.7229 - acc: 0.9545 - val_loss: 2.5297 - val_acc: 0.5880\n",
      "Epoch 12/20\n",
      "50000/50000 [==============================] - 81s 2ms/step - loss: 0.7020 - acc: 0.9579 - val_loss: 2.2610 - val_acc: 0.6122\n",
      "Epoch 13/20\n",
      "50000/50000 [==============================] - 82s 2ms/step - loss: 0.6785 - acc: 0.9634 - val_loss: 2.0715 - val_acc: 0.6554\n",
      "Epoch 14/20\n",
      "50000/50000 [==============================] - 82s 2ms/step - loss: 0.6669 - acc: 0.9641 - val_loss: 2.2667 - val_acc: 0.6290\n",
      "Epoch 15/20\n",
      "50000/50000 [==============================] - 82s 2ms/step - loss: 0.6467 - acc: 0.9693 - val_loss: 2.6547 - val_acc: 0.6102\n",
      "Epoch 16/20\n",
      "50000/50000 [==============================] - 82s 2ms/step - loss: 0.6351 - acc: 0.9710 - val_loss: 2.8041 - val_acc: 0.5869\n",
      "Epoch 17/20\n",
      "50000/50000 [==============================] - 81s 2ms/step - loss: 0.6197 - acc: 0.9736 - val_loss: 2.4689 - val_acc: 0.6145\n",
      "Epoch 18/20\n",
      "50000/50000 [==============================] - 81s 2ms/step - loss: 0.6084 - acc: 0.9751 - val_loss: 4.2802 - val_acc: 0.4618\n",
      "Epoch 19/20\n",
      "50000/50000 [==============================] - 81s 2ms/step - loss: 0.6149 - acc: 0.9713 - val_loss: 2.5493 - val_acc: 0.6121\n",
      "Epoch 20/20\n",
      "50000/50000 [==============================] - 81s 2ms/step - loss: 0.5959 - acc: 0.9760 - val_loss: 2.2993 - val_acc: 0.6470\n"
     ]
    }
   ],
   "source": [
    "# Note, we turn off the callbacks as we want to use the new learning rate\n",
    "t1 = time()\n",
    "model.fit(x_train, y_train,\n",
    "              batch_size=batch_size,\n",
    "              epochs=epochs,\n",
    "              verbose=1,\n",
    "              validation_data=(x_test, y_test),\n",
    "              shuffle=True)\n",
    "t2 = time()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training of 20 epochs took 27.57 minutes total.\n"
     ]
    }
   ],
   "source": [
    "time_delta=round((t2-t1)/60,2)\n",
    "\n",
    "print (\"Training of {} epochs took {} minutes total.\".format(epochs, time_delta))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000/10000 [==============================] - 8s 773us/step\n",
      "Test loss: 2.2992931884765624\n",
      "Test accuracy: 0.647\n"
     ]
    }
   ],
   "source": [
    "scores = model.evaluate(x_test, y_test, verbose=1)\n",
    "print('Test loss:', scores[0])\n",
    "print('Test accuracy:', scores[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### More variation and longer training..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model type: ResNet26v1\n",
      "There are, in total, 372330 parameters in this model!\n"
     ]
    }
   ],
   "source": [
    "version = 1\n",
    "n = 4\n",
    "\n",
    "depth = model_depth(version=version,n=n)\n",
    "model = choose_model(version)\n",
    "\n",
    "print(\"Model type:\",model_type(version,n))\n",
    "print(\"There are, in total, {} parameters in this model!\".format(model.count_params()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Hard coding the learning rate = 1e-3\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer=Adam(lr=1e-3),\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 128  # orig paper trained all networks with batch_size=128\n",
    "epochs = 50"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [],
   "source": [
    "callbacks = [lr_reducer]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 50000 samples, validate on 10000 samples\n",
      "Epoch 1/50\n",
      "50000/50000 [==============================] - 32s 631us/step - loss: 1.0204 - acc: 0.7070 - val_loss: 1.3248 - val_acc: 0.6230\n",
      "Epoch 2/50\n",
      "50000/50000 [==============================] - 31s 621us/step - loss: 0.8886 - acc: 0.7562 - val_loss: 1.8254 - val_acc: 0.5249\n",
      "Epoch 3/50\n",
      "50000/50000 [==============================] - 31s 622us/step - loss: 0.7859 - acc: 0.7894 - val_loss: 0.9916 - val_acc: 0.7328\n",
      "Epoch 4/50\n",
      "50000/50000 [==============================] - 31s 624us/step - loss: 0.7010 - acc: 0.8206 - val_loss: 1.5349 - val_acc: 0.6127\n",
      "Epoch 5/50\n",
      "50000/50000 [==============================] - 31s 630us/step - loss: 0.6452 - acc: 0.8400 - val_loss: 0.9615 - val_acc: 0.7466\n",
      "Epoch 6/50\n",
      "50000/50000 [==============================] - 31s 630us/step - loss: 0.5894 - acc: 0.8588 - val_loss: 0.9448 - val_acc: 0.7522\n",
      "Epoch 7/50\n",
      "50000/50000 [==============================] - 32s 636us/step - loss: 0.5459 - acc: 0.8761 - val_loss: 1.1748 - val_acc: 0.7155\n",
      "Epoch 8/50\n",
      "50000/50000 [==============================] - 31s 630us/step - loss: 0.5060 - acc: 0.8900 - val_loss: 1.3657 - val_acc: 0.6986\n",
      "Epoch 9/50\n",
      "50000/50000 [==============================] - 31s 628us/step - loss: 0.4642 - acc: 0.9045 - val_loss: 1.4428 - val_acc: 0.6682\n",
      "Epoch 10/50\n",
      "50000/50000 [==============================] - 32s 631us/step - loss: 0.4307 - acc: 0.9177 - val_loss: 1.3192 - val_acc: 0.7142\n",
      "Epoch 11/50\n",
      "50000/50000 [==============================] - 31s 629us/step - loss: 0.4162 - acc: 0.9235 - val_loss: 1.6899 - val_acc: 0.6612\n",
      "Epoch 12/50\n",
      "50000/50000 [==============================] - 31s 628us/step - loss: 0.2813 - acc: 0.9769 - val_loss: 0.9175 - val_acc: 0.8019\n",
      "Epoch 13/50\n",
      "50000/50000 [==============================] - 31s 628us/step - loss: 0.2341 - acc: 0.9920 - val_loss: 0.9335 - val_acc: 0.8052\n",
      "Epoch 14/50\n",
      "50000/50000 [==============================] - 31s 629us/step - loss: 0.2139 - acc: 0.9964 - val_loss: 1.0254 - val_acc: 0.7984\n",
      "Epoch 15/50\n",
      "50000/50000 [==============================] - 32s 631us/step - loss: 0.2044 - acc: 0.9973 - val_loss: 1.0121 - val_acc: 0.8029\n",
      "Epoch 16/50\n",
      "50000/50000 [==============================] - 32s 631us/step - loss: 0.1974 - acc: 0.9974 - val_loss: 1.1791 - val_acc: 0.7870\n",
      "Epoch 17/50\n",
      "50000/50000 [==============================] - 32s 630us/step - loss: 0.1964 - acc: 0.9953 - val_loss: 1.1734 - val_acc: 0.7929\n",
      "Epoch 18/50\n",
      "50000/50000 [==============================] - 32s 631us/step - loss: 0.1835 - acc: 0.9988 - val_loss: 1.0928 - val_acc: 0.8059\n",
      "Epoch 19/50\n",
      "50000/50000 [==============================] - 32s 631us/step - loss: 0.1766 - acc: 0.9997 - val_loss: 1.0741 - val_acc: 0.8076\n",
      "Epoch 20/50\n",
      "50000/50000 [==============================] - 32s 634us/step - loss: 0.1730 - acc: 0.9999 - val_loss: 1.0837 - val_acc: 0.8075\n",
      "Epoch 21/50\n",
      "50000/50000 [==============================] - 32s 632us/step - loss: 0.1701 - acc: 0.9999 - val_loss: 1.1360 - val_acc: 0.8025\n",
      "Epoch 22/50\n",
      "50000/50000 [==============================] - 32s 632us/step - loss: 0.1662 - acc: 1.0000 - val_loss: 1.1012 - val_acc: 0.8078\n",
      "Epoch 23/50\n",
      "50000/50000 [==============================] - 32s 632us/step - loss: 0.1635 - acc: 1.0000 - val_loss: 1.1208 - val_acc: 0.8077\n",
      "Epoch 24/50\n",
      "50000/50000 [==============================] - 32s 631us/step - loss: 0.1621 - acc: 1.0000 - val_loss: 1.1175 - val_acc: 0.8084\n",
      "Epoch 25/50\n",
      "50000/50000 [==============================] - 33s 654us/step - loss: 0.1604 - acc: 1.0000 - val_loss: 1.1125 - val_acc: 0.8088\n",
      "Epoch 26/50\n",
      "50000/50000 [==============================] - 32s 637us/step - loss: 0.1589 - acc: 1.0000 - val_loss: 1.1141 - val_acc: 0.8097\n",
      "Epoch 27/50\n",
      "50000/50000 [==============================] - 32s 631us/step - loss: 0.1568 - acc: 1.0000 - val_loss: 1.1280 - val_acc: 0.8074\n",
      "Epoch 28/50\n",
      "50000/50000 [==============================] - 32s 633us/step - loss: 0.1553 - acc: 1.0000 - val_loss: 1.1258 - val_acc: 0.8089\n",
      "Epoch 29/50\n",
      "50000/50000 [==============================] - 32s 633us/step - loss: 0.1545 - acc: 1.0000 - val_loss: 1.1250 - val_acc: 0.8076\n",
      "Epoch 30/50\n",
      "50000/50000 [==============================] - 32s 633us/step - loss: 0.1536 - acc: 1.0000 - val_loss: 1.1288 - val_acc: 0.8074\n",
      "Epoch 31/50\n",
      "50000/50000 [==============================] - 32s 634us/step - loss: 0.1526 - acc: 1.0000 - val_loss: 1.1328 - val_acc: 0.8079\n",
      "Epoch 32/50\n",
      "50000/50000 [==============================] - 32s 634us/step - loss: 0.1516 - acc: 1.0000 - val_loss: 1.1321 - val_acc: 0.8083\n",
      "Epoch 33/50\n",
      "50000/50000 [==============================] - 32s 634us/step - loss: 0.1509 - acc: 1.0000 - val_loss: 1.1336 - val_acc: 0.8078\n",
      "Epoch 34/50\n",
      "50000/50000 [==============================] - 32s 634us/step - loss: 0.1505 - acc: 1.0000 - val_loss: 1.1374 - val_acc: 0.8083\n",
      "Epoch 35/50\n",
      "50000/50000 [==============================] - 33s 654us/step - loss: 0.1501 - acc: 1.0000 - val_loss: 1.1369 - val_acc: 0.8076\n",
      "Epoch 36/50\n",
      "50000/50000 [==============================] - 32s 634us/step - loss: 0.1498 - acc: 1.0000 - val_loss: 1.1373 - val_acc: 0.8073\n",
      "Epoch 37/50\n",
      "50000/50000 [==============================] - 32s 633us/step - loss: 0.1494 - acc: 1.0000 - val_loss: 1.1381 - val_acc: 0.8072\n",
      "Epoch 38/50\n",
      "50000/50000 [==============================] - 32s 634us/step - loss: 0.1491 - acc: 1.0000 - val_loss: 1.1379 - val_acc: 0.8074\n",
      "Epoch 39/50\n",
      "50000/50000 [==============================] - 32s 643us/step - loss: 0.1490 - acc: 1.0000 - val_loss: 1.1406 - val_acc: 0.8073\n",
      "Epoch 40/50\n",
      "50000/50000 [==============================] - 32s 638us/step - loss: 0.1489 - acc: 1.0000 - val_loss: 1.1385 - val_acc: 0.8072\n",
      "Epoch 41/50\n",
      "50000/50000 [==============================] - 32s 639us/step - loss: 0.1487 - acc: 1.0000 - val_loss: 1.1407 - val_acc: 0.8076\n",
      "Epoch 42/50\n",
      "50000/50000 [==============================] - 33s 666us/step - loss: 0.1485 - acc: 1.0000 - val_loss: 1.1399 - val_acc: 0.8070\n",
      "Epoch 43/50\n",
      "50000/50000 [==============================] - 32s 641us/step - loss: 0.1484 - acc: 1.0000 - val_loss: 1.1395 - val_acc: 0.8079\n",
      "Epoch 44/50\n",
      "50000/50000 [==============================] - 32s 638us/step - loss: 0.1484 - acc: 1.0000 - val_loss: 1.1411 - val_acc: 0.8079\n",
      "Epoch 45/50\n",
      "50000/50000 [==============================] - 32s 637us/step - loss: 0.1483 - acc: 1.0000 - val_loss: 1.1398 - val_acc: 0.8074\n",
      "Epoch 46/50\n",
      "50000/50000 [==============================] - 32s 637us/step - loss: 0.1483 - acc: 1.0000 - val_loss: 1.1410 - val_acc: 0.8073\n",
      "Epoch 47/50\n",
      "50000/50000 [==============================] - 32s 641us/step - loss: 0.1481 - acc: 1.0000 - val_loss: 1.1405 - val_acc: 0.8075\n",
      "Epoch 48/50\n",
      "50000/50000 [==============================] - 32s 640us/step - loss: 0.1481 - acc: 1.0000 - val_loss: 1.1414 - val_acc: 0.8072\n",
      "Epoch 49/50\n",
      "50000/50000 [==============================] - 32s 637us/step - loss: 0.1480 - acc: 1.0000 - val_loss: 1.1417 - val_acc: 0.8075\n",
      "Epoch 50/50\n",
      "50000/50000 [==============================] - 32s 635us/step - loss: 0.1479 - acc: 1.0000 - val_loss: 1.1411 - val_acc: 0.8073\n"
     ]
    }
   ],
   "source": [
    "# Note, we turn off the callbacks as we want to use the new learning rate\n",
    "t1 = time()\n",
    "model.fit(x_train, y_train,\n",
    "              batch_size=batch_size,\n",
    "              epochs=epochs,\n",
    "              verbose=1,\n",
    "              validation_data=(x_test, y_test),\n",
    "              shuffle=True,\n",
    "              callbacks=callbacks)\n",
    "t2 = time()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training of 50 epochs took 26.55 minutes total.\n"
     ]
    }
   ],
   "source": [
    "time_delta=round((t2-t1)/60,2)\n",
    "\n",
    "print (\"Training of {} epochs took {} minutes total.\".format(epochs, time_delta))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000/10000 [==============================] - 4s 432us/step\n",
      "Test loss: 1.1410504387378693\n",
      "Test accuracy: 0.8073\n"
     ]
    }
   ],
   "source": [
    "scores = model.evaluate(x_test, y_test, verbose=1)\n",
    "print('Test loss:', scores[0])\n",
    "print('Test accuracy:', scores[1])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}