week_09/Week_09.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Week 09.ipynb",
      "provenance": [],
      "toc_visible": true,
      "machine_shape": "hm"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JXDbwvnXH7u_"
      },
      "source": [
        "# 1.0 Introduction"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "teKSGqKplfXo"
      },
      "source": [
        "There are discrete architectural elements from milestone models that you can use to design your convolutional neural networks. Specifically, models that have achieved state-of-the-art results for tasks like image classification use discrete architecture elements repeated multiple times, such as the VGG block in the [VGG](https://arxiv.org/abs/1409.1556) models, the inception module in the [GoogLeNet](https://arxiv.org/abs/1409.4842), and the residual module in the [ResNet](https://arxiv.org/abs/1512.03385). Once you can implement parameterized versions of these architectural elements, you can use them to design your models for computer\n",
        "vision and other applications. This lesson will discover how to implement the critical architecture elements from milestone convolutional neural network models from scratch. After completing this lesson, you will know:\n",
        "\n",
        "- How to implement a VGG module used in the VGG-16 and VGG-19 convolutional neural network models.\n",
        "- How to implement the naive and optimized inception module used in the GoogLeNet model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "itx0EimHmH2n"
      },
      "source": [
        "# 2.0 VGGnet"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m7Uv0SNunzV9"
      },
      "source": [
        "In our previous lesson, we discussed LeNet and AlexNet, two seminal Convolutional Neural Networks in the deep learning and computer vision literature. Simonyan and Zisserman first introduced VGGnet, (sometimes referred to as simply VGG) in their 2015 paper, [Very Deep Learning Convolutional Neural Networks for Large-Scale Image Recognition]((https://arxiv.org/abs/1409.1556)). \n",
        "\n",
        "> The primary contribution of their work was demonstrating that architecture with tiny (3x3) filters can be trained to increasingly higher depths (16-19 layers) and obtain state-of-the-art classification on the challenging ImageNet classification challenge.\n",
        "\n",
        "This network is **characterized by its simplicity**, using only 3×3 convolutional layers stacked on top of each other in increasing depth. **Reducing the spatial dimensions of volumes is accomplished through the usage of max pooling**. Two fully connected layers, each with 4,096 nodes (and dropout in between), are followed by a softmax classifier.\n",
        "\n",
        "VGG is often used today for <font color=\"red\">transfer learning</font> (we will describe this technique later in this course) **as the network demonstrates an above-average ability to generalize to datasets it was not trained on** (compared to other network types such as GoogLeNet and ResNet). More times than not, if you are reading a publication or lab journal that applies transfer learning, it likely uses VGG as the base model.\n",
        "\n",
        "Unfortunately, training VGG from scratch is a pain, to say the least. The network is brutally slow to train, and the network architecture weights themselves are quite large (over 500MB). Due to the depth of the network along with the fully-connected layers, the backpropagation phase is excruciatingly slow.\n",
        "\n",
        "> References from practitioners such as [Adrian Rosebrock](https://www.pyimagesearch.com/), training VGG on eight GPUs took $\\approx$ 10 days – with any less than four GPUs, training VGG from scratch will likely take prohibitively long (unless you can be very patient).\n",
        "\n",
        "That said, it’s important as a deep learning practitioner to understand the history of deep learning, especially the concept of <font color=\"red\">pre-training</font> and how **we later learned to avoid this expensive operation by optimizing our initialization weight functions**."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9SMwntsHn54F"
      },
      "source": [
        "## 2.1 Implementing VGGNet"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zRzfkf_Hz3wC"
      },
      "source": [
        "When implementing VGG, Simonyan and Zisserman tried variants of VGG that increased in depth. The figure below was extracted from their publication is and highlights their experiments. In particular, we are most interested in configurations A, B, D, and E. Looking at these architectures, you will notice two patterns:\n",
        "\n",
        "1. The first is that the **network uses only 3×3 filters**. \n",
        "2. The second is as the depth of the network increases, the number of filters learned increases as well – to be exact, **the number of filters doubles each time max pooling is applied** to reduce volume size. \n",
        "\n",
        "> The notion of doubling the number of filters each time you decrease spatial dimensions is of historical importance in the deep learning literature and even a pattern you will see today.\n",
        "\n",
        "<img width=\"600\" src=\"https://drive.google.com/uc?export=view&id=19cSs4edO6rQ4ZodRzgU54Inv8P3m566K\"/>\n",
        "\n",
        "We perform this doubling of filters to ensure no single layer block is more biased than the others. Layers earlier in the network architecture have fewer filters, but their spatial volumes are also much larger, implying there is **more (spatial) data** to learn from.\n",
        "\n",
        "However, we know that applying a max-pooling operation will reduce our spatial input volumes. \n",
        "\n",
        "> If we reduce the spatial volumes without increasing the number of filters, our layers become unbalanced and potentially biased, implying that layers earlier in the network may influence our output classification more than layers deeper. \n",
        "\n",
        "To combat this imbalance, we keep in mind the ratio of volume size to the number of filters. If we reduce the input volume size by 50-75%, we double the number of filters in the next set of CONV layers to maintain the balance.\n",
        "\n",
        "The issue with training such deep architectures is that Simonyan and Zisserman found training VGG16 and VGG19 to be extremely challenging due to their depth. **If these architectures were randomly initialized and trained from scratch, they would often struggle to learn and gain any initial traction – the networks were too deep for basic random initialization**. Therefore, to train deeper variants of VGG, Simonyan and Zisserman came up with a clever concept called <font color=\"red\">pre-training</font>.\n",
        "\n",
        "> Pre-training is the practice of training smaller versions of your network architecture with fewer weight layers first and then using these converged network weights as the initializations for larger,\n",
        "deeper networks.\n",
        "\n",
        "In the case of VGG, the authors first trained configuration A, VGG11. VGG11 was able to converge to the level of reasonably low loss but not state-of-the-art accuracy worthy.\n",
        "\n",
        "The weights from VGG11 were then used as initializations to configuration B, VGG13. The conv3-64 and conv3-128 layers (highlighted in bold in above figure) in VGG13 were randomly initialized while the remainder of the layers were copied over from the pre-trained VGG11 network. Using the initializations, Simonyan and Zisserman were able to train VGG13 successfully–but still not obtain state-of-the-art accuracy.\n",
        "\n",
        "This pre-training pattern continued to configuration D, which we commonly know as VGG16. This time three new layers were randomly initialized while the other layers were copied over from VGG13. The network was then trained using these **warmed pre-trained up** layers, thereby allowing the randomly initialized layers to converge and learn discriminating patterns. Ultimately, VGG16 was able to perform very well on the ImageNet classification challenge.\n",
        "\n",
        "> As a final experiment, Simonyan and Zisserman once again applied pre-training to configuration E, VGG19. This very deep architecture copied the weights from the pre-trained VGG16 architecture and then added another additional three convolutional layers. After training, it was found that VGG19 obtained the highest classification accuracy from their experiments; however, the size of the model (574MB) and the amount of time it took to train and evaluate the network, all for meager gains, made it less appealing to deep learning practitioners.\n",
        "\n",
        "**If pre-training sounds like a painful, tedious process, that is because it is**. Training smaller variations of your network architecture and then using the converged weights as initializations to your deeper versions of the network is a clever trick; however, it requires training and tuning the\n",
        "hyperparameters to N separate networks, where N is your final network architecture along with the number of previous (smaller) networks required to obtain the end model. Performing this process is\n",
        "extremely time-consuming, especially for deeper networks with many fully-connected layers such as VGG.\n",
        "\n",
        "<font color=\"red\">The good news is that we no longer perform pre-training</font> when training very deep Convolutional Neural Networks – instead, we rely on a good initialization function. Instead of pure random weight initializations we now use [Xavier/Glorot](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf) or MSRA (also known as He et al. initialization).\n",
        "Through the work of both Mishkin and Mtas in their 2015 paper, [All you need is a good init](https://arxiv.org/abs/1511.06422) and He et al. in [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://arxiv.org/abs/1502.01852), we found that we can skip the pre-training phase entirely and jump directly to the deeper variations of the network architectures.\n",
        "\n",
        "> After these papers were published, Simonyan and Zisserman re-evaluated their experiments and found that these **smarter initialization** schemes and activation functions could replicate their previous performance without the usage of tedious pre-training.\n",
        "\n",
        "Additionally, **we recommend using batch normalization after the activation functions in the network**. \n",
        "\n",
        "> Apply batch normalization was not discussed in the original Simonyan and Zisserman paper, but as other lessons have discussed, batch normalization can stabilize your training and reduce the total number of epochs required to obtain a reasonably performing model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6xBJgBMTDl-Q"
      },
      "source": [
        "## 2.2 The VGG Family of Networks"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M9mrWjqWUjhz"
      },
      "source": [
        "Two key components can characterize the VGG family of Convolutional Neural Networks:\n",
        "\n",
        "1. All CONV layers in the network using only 3x3 filters.\n",
        "2. Stacking multiple CONV => RELU layer sets (where the number of consecutive CONV => RELU layers typically increases the deeper we go) before applying a POOL operation.\n",
        "\n",
        "In this section, we will discuss a variant of the VGGNet architecture, which we call “MiniVGGNet,” because the network is substantially more shallow than its big brother."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "js_H5Ss6VTdI"
      },
      "source": [
        "### 2.2.1 The (Mini) VGGNet Architecture"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9-WIqSRAXeiC"
      },
      "source": [
        "In LeNet-5, we have applied a series of CONV => RELU => POOL layers. However, in VGGNet, we stack multiple CONV => RELU layers before applying a single POOL layer. This allows the network to learn more rich features from the CONV layers before downsampling the spatial input size via the POOL operation.\n",
        "\n",
        "Overall, MiniVGGNet consists of two sets of CONV => RELU => CONV => RELU => POOL layers, followed by a set of FC => RELU => FC => SOFTMAX layers. The first two CONV layers will learn 32 filters, each of size 3x3. The second two CONV layers will learn 64 filters, again, each of size 3x3. Our POOL layers will perform max pooling over a 2x2 window with a 2x2 stride. We will also be inserting batch normalization layers after the activations and dropout layers (DO) after the POOL and FC layers.\n",
        "\n",
        "The network architecture itself is detailed in Figure below, where the initial input image size is assumed to be 32x32x3 as we will be training MiniVGGNet on [CIFAR-10](https://www.cs.toronto.edu/~kriz/cifar.html) later in this section.\n",
        "\n",
        "<img width=\"450\" src=\"https://drive.google.com/uc?export=view&id=1hcjInhUrh0AuZXV73mu49RoksoIkzayU\"/>\n",
        "\n",
        "\n",
        "Again, notice how the batch normalization and dropout layers are included in the network architecture based on **Best Practices** described in Lesson 5. Applying batch normalization will help reduce the effects of overfitting and increase our classification accuracy on CIFAR-10."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UUW_xNn0dag5"
      },
      "source": [
        "### 2.2.2 CIFAR-10"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TGQQChxCdeFl"
      },
      "source": [
        "Just like MNIST, CIFAR-10 is considered another standard benchmark dataset for image classification\n",
        "in the computer vision and machine learning literature. CIFAR-10 consists of 60,000\n",
        "32x32x3 (RGB) images resulting in a feature vector dimensionality of 3072.\n",
        "\n",
        "As the name suggests, CIFAR-10 consists of 10 classes, including: airplanes, automobiles,\n",
        "birds, cats, deer, dogs, frogs, horses, ships, and trucks.\n",
        "\n",
        "While it’s quite easy to train a model that obtains > 97% classification accuracy on MNIST,\n",
        "it’s substantially harder to obtain such a model for CIFAR-10 (and it’s bigger brother, CIFAR-100).\n",
        "\n",
        "The challenge comes from the dramatic variance in how objects appear. For example, we can\n",
        "no longer assume that an image containing a green pixel at a given (x;y)-coordinate is a frog. This\n",
        "pixel could be part of the background of a forest that contains a deer. Or, the pixel could simply be\n",
        "the color of a green truck.\n",
        "\n",
        "These assumptions are a stark contrast to the MNIST dataset where the network can learn assumptions regarding the spatial distribution of pixel intensities. For example, the spatial distribution of foreground pixels of the number 1 is substantially different than a 0 or 5.\n",
        "\n",
        "> While being a small dataset, CIFAR-10 is still regularly used to benchmark new CNN architectures."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2rV_GBjXYnLi"
      },
      "source": [
        "## 2.3 Implementing MiniVGGNet"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "biTxxtguZPYe"
      },
      "source": [
        "# import the necessary packages\n",
        "from tensorflow.keras.models import Sequential\n",
        "from tensorflow.keras.layers import BatchNormalization\n",
        "from tensorflow.keras.layers import Conv2D\n",
        "from tensorflow.keras.layers import MaxPooling2D\n",
        "from tensorflow.keras.layers import Activation\n",
        "from tensorflow.keras.layers import Flatten\n",
        "from tensorflow.keras.layers import Dropout\n",
        "from tensorflow.keras.layers import Dense\n",
        "from tensorflow.keras import backend as K\n",
        "\n",
        "class MiniVGGNet:\n",
        "\t@staticmethod\n",
        "\tdef build(width, height, depth, classes):\n",
        "\t\t# initialize the model along with the input shape to be\n",
        "\t\t# \"channels last\" and the channels dimension itself\n",
        "\t\tmodel = Sequential()\n",
        "\t\tinputShape = (height, width, depth)\n",
        "\t\tchanDim = -1\n",
        "\n",
        "\t\t# if we are using \"channels first\", update the input shape\n",
        "\t\t# and channels dimension\n",
        "\t\tif K.image_data_format() == \"channels_first\":\n",
        "\t\t\tinputShape = (depth, height, width)\n",
        "\t\t\tchanDim = 1\n",
        "\n",
        "\t\t# first CONV => RELU => CONV => RELU => POOL layer set\n",
        "\t\tmodel.add(Conv2D(32, (3, 3), padding=\"same\",input_shape=inputShape))\n",
        "\t\tmodel.add(Activation(\"relu\"))\n",
        "\t\tmodel.add(BatchNormalization(axis=chanDim))\n",
        "\t\tmodel.add(Conv2D(32, (3, 3), padding=\"same\"))\n",
        "\t\tmodel.add(Activation(\"relu\"))\n",
        "\t\tmodel.add(BatchNormalization(axis=chanDim))\n",
        "\t\tmodel.add(MaxPooling2D(pool_size=(2, 2)))\n",
        "\t\tmodel.add(Dropout(0.25))\n",
        "\n",
        "\t\t# second CONV => RELU => CONV => RELU => POOL layer set\n",
        "\t\tmodel.add(Conv2D(64, (3, 3), padding=\"same\"))\n",
        "\t\tmodel.add(Activation(\"relu\"))\n",
        "\t\tmodel.add(BatchNormalization(axis=chanDim))\n",
        "\t\tmodel.add(Conv2D(64, (3, 3), padding=\"same\"))\n",
        "\t\tmodel.add(Activation(\"relu\"))\n",
        "\t\tmodel.add(BatchNormalization(axis=chanDim))\n",
        "\t\tmodel.add(MaxPooling2D(pool_size=(2, 2)))\n",
        "\t\tmodel.add(Dropout(0.25))\n",
        "\n",
        "\t\t# first (and only) set of FC => RELU layers\n",
        "\t\tmodel.add(Flatten())\n",
        "\t\tmodel.add(Dense(512))\n",
        "\t\tmodel.add(Activation(\"relu\"))\n",
        "\t\tmodel.add(BatchNormalization())\n",
        "\t\tmodel.add(Dropout(0.5))\n",
        "\n",
        "\t\t# softmax classifier\n",
        "\t\tmodel.add(Dense(classes))\n",
        "\t\tmodel.add(Activation(\"softmax\"))\n",
        "\n",
        "\t\t# return the constructed network architecture\n",
        "\t\treturn model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q27hX2aFlL0z"
      },
      "source": [
        "model = MiniVGGNet.build(width=32, height=32, depth=3, classes=10)\n",
        "model.summary()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kpTz5T7zZnKw"
      },
      "source": [
        "## 2.4 MiniVGGNet on CIFAR-10"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hwDY1UK8v5ZN"
      },
      "source": [
        "%%capture\n",
        "!pip install wandb"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1FfZKTg6v9kO"
      },
      "source": [
        "!wandb login"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Viz3lGIQagQj"
      },
      "source": [
        "import matplotlib\n",
        "\n",
        "# import the necessary packages\n",
        "from sklearn.preprocessing import LabelBinarizer\n",
        "from sklearn.metrics import classification_report\n",
        "from tensorflow.keras.optimizers import SGD\n",
        "from tensorflow.keras.datasets import cifar10\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import wandb\n",
        "from wandb.keras import WandbCallback\n",
        "\n",
        "# load the training and testing data, then scale it into the\n",
        "# range [0, 1]\n",
        "print(\"[INFO] loading CIFAR-10 data...\")\n",
        "(train_x, train_y), (test_x, test_y) = cifar10.load_data()\n",
        "train_x = train_x.astype(\"float\") / 255.0\n",
        "test_x = test_x.astype(\"float\") / 255.0\n",
        "\n",
        "# convert the labels from integers to vectors\n",
        "lb = LabelBinarizer()\n",
        "train_y = lb.fit_transform(train_y)\n",
        "test_y = lb.transform(test_y)\n",
        "\n",
        "# initialize the label names for the CIFAR-10 dataset\n",
        "labelNames = [\"airplane\", \"automobile\", \"bird\", \"cat\", \"deer\",\n",
        "              \"dog\", \"frog\", \"horse\", \"ship\", \"truck\"]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LdIGC9mMwbTH"
      },
      "source": [
        "# Set an experiment name to group training and evaluation\n",
        "experiment_name = wandb.util.generate_id()\n",
        "\n",
        "# setup wandb\n",
        "wandb.init(project=\"lesson07_VGG\", \n",
        "           group=experiment_name,\n",
        "           config={\n",
        "               \"epoch\": 40,\n",
        "               \"batch_size\": 64,\n",
        "           })\n",
        "config = wandb.config"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pCXHS5OJwUiB"
      },
      "source": [
        "%%wandb\n",
        "\n",
        "# initialize the optimizer and model\n",
        "print(\"[INFO] compiling model...\")\n",
        "\n",
        "# An decay parameter was used. This argument is used to slowly reduce the learning rate over time.\n",
        "# Rate Schedulers, decaying the learning rate is helpful in reducing overfitting\n",
        "# and obtaining higher classification accuracy – the smaller the learning rate is, \n",
        "# the smaller the weight updates will be. A common setting for decay is to divide\n",
        "# the initial learning rate by the total number of epochs – in this case, \n",
        "# we’ll be training our network for a total of 40 epochs with an initial learning rate of 0.01,\n",
        "# therefore decay = 0.01 / 40.\n",
        "\n",
        "opt = SGD(lr=0.01, decay=0.01/40, momentum=0.9, nesterov=True)\n",
        "model = MiniVGGNet.build(width=32, height=32, depth=3, classes=10)\n",
        "model.compile(loss=\"categorical_crossentropy\", optimizer=opt,metrics=[\"accuracy\"])\n",
        "\n",
        "# train the network\n",
        "print(\"[INFO] training network...\")\n",
        "H = model.fit(train_x, train_y, validation_data=(test_x, test_y),\n",
        "              batch_size=config.batch_size, \n",
        "              epochs=config.epoch,\n",
        "              verbose=1,\n",
        "              callbacks=[WandbCallback()])\n",
        "wandb.finish()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FYvs8Rx2Bf7T"
      },
      "source": [
        "**Log Analysis**\n",
        "\n",
        "Next, log an analysis run, using the same experiment name as the group parameter so that this run and the previous run are grouped together in W&B."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TklsWV-FBcSB"
      },
      "source": [
        "%%capture\n",
        "# Install dependencies\n",
        "!pip install scikit-plot -qqq"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2U3M91iyBsZl"
      },
      "source": [
        "import numpy as np\n",
        "from sklearn.metrics import f1_score\n",
        "import matplotlib.pyplot as plt\n",
        "import tensorflow as tf\n",
        "from scikitplot.metrics import plot_confusion_matrix, plot_roc, plot_precision_recall\n",
        "\n",
        "wandb.init(project=\"lesson07_VGG\", group=experiment_name)\n",
        "\n",
        "# Class proportions\n",
        "train_y_labels = [labelNames[i] for i in np.argmax(train_y, axis=1)]\n",
        "test_y_labels = [labelNames[i] for i in np.argmax(test_y, axis=1)]\n",
        "wandb.log({'Class Proportions': wandb.sklearn.plot_class_proportions(train_y_labels,\n",
        "                                                                     test_y_labels,\n",
        "                                                                     labelNames)},\n",
        "           commit=False) # Hold on, more incoming!\n",
        "\n",
        "# Log F1 Score\n",
        "test_y_pred = np.asarray(model.predict(test_x))\n",
        "test_y_pred_class = np.argmax(test_y_pred, axis=1)\n",
        "f1 = f1_score(np.argmax(test_y, axis=1), test_y_pred_class, average='micro')\n",
        "wandb.log({\"f1\": f1}, commit=False)\n",
        "\n",
        "#test_y_labels = [labelNames[i] for i in np.argmax(test_y, axis=1)]\n",
        "test_y_pred_labels = [labelNames[i] for i in test_y_pred_class]\n",
        "\n",
        "\n",
        "# Log Confusion Matrix\n",
        "fig, ax = plt.subplots(figsize=(16, 12))\n",
        "plot_confusion_matrix(test_y_labels, test_y_pred_labels, ax=ax)\n",
        "wandb.log({\"confusion_matrix\": wandb.Image(fig)}, commit=False)\n",
        "\n",
        "# Log ROC Curve\n",
        "fig, ax = plt.subplots(figsize=(16, 12))\n",
        "plot_roc(test_y_labels, test_y_pred, ax=ax)\n",
        "wandb.log({\"plot_roc\": wandb.Image(fig)},commit=False)  # Now we've logged everything for this step\n",
        "\n",
        "# Precision vs Recall\n",
        "fig, ax = plt.subplots(figsize=(16, 12))\n",
        "plot_precision_recall(test_y_labels, test_y_pred, ax=ax)\n",
        "wandb.log({\"plot_precision_recall\": wandb.Image(fig)},commit=False)  # Now we've logged everything for this step\n",
        "\n",
        "# Class Scores\n",
        "class_score_data = []\n",
        "for test, pred in zip(test_y_labels, test_y_pred):\n",
        "    class_score_data.append([test, pred])\n",
        "\n",
        "wandb.log({\"class_scores\": wandb.Table(data=class_score_data,\n",
        "                                           columns=[\"test\", \"pred\"])}, commit=False)\n",
        "\n",
        "# \n",
        "# Visualize Predictions\n",
        "# \n",
        "# visualize 18 images\n",
        "def show_image(train_image, label, index):\n",
        "    plt.subplot(3, 6, index+1)\n",
        "    plt.imshow(tf.squeeze(train_image), cmap=plt.cm.gray)\n",
        "    plt.title(label)\n",
        "    plt.grid(b=False)\n",
        "\n",
        "# predictions\n",
        "predictions = model.predict(test_x)\n",
        "results = np.argmax(predictions, axis = 1)\n",
        "\n",
        "# visualize the first 18 test results\n",
        "plt.figure(figsize=(12, 8))\n",
        "for index in range(18):\n",
        "    label = results[index]\n",
        "    image_pixels = test_x[index,:,:,:]\n",
        "    show_image(image_pixels, labelNames[label], index)\n",
        "plt.tight_layout()\n",
        "\n",
        "wandb.log({\"Predictions\": plt}, commit=True)\n",
        "\n",
        "wandb.finish()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FaaSsw2fxX4F"
      },
      "source": [
        "# evaluate the network\n",
        "print(\"[INFO] evaluating network...\")\n",
        "predictions = model.predict(test_x, batch_size=64)\n",
        "print(classification_report(test_y.argmax(axis=1),\n",
        "                            predictions.argmax(axis=1), target_names=labelNames))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PMexmGUScjNl"
      },
      "source": [
        "## 2.5 Extensions"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tX32HtwWewNc"
      },
      "source": [
        "When evaluating MinIVGGNet you can performed some experiments:\n",
        "1. Disable GPU and use the only CPU to investigate the ratio of benefit to using more powerful hardware.\n",
        "2. Experiment the model without batch normalization.\n",
        "3. Let’s go ahead and take a look at these results to compare how network performance increases\n",
        "when applying batch normalization.\n",
        "4. Consider using Data Augmentation in a combination with 2 and 3. Tell us what you find.\n",
        "5. Try to combat the overfiting problem using techniques presented in previous lessons."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u7leU0DAXFJV"
      },
      "source": [
        "# 3.0 GoogLeNet"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u3LfKTWUXIjW"
      },
      "source": [
        "This section will study the **GoogLeNet architecture**, introduced by Szegedy et al. in their 2014 paper, [Going Deeper With Convolutions](https://arxiv.org/pdf/1409.4842.pdf). This paper is essential for two reasons. First, the model architecture is tiny compared to AlexNet and VGGNet ( 28MB for the weights themselves). The authors can obtain such a dramatic drop in network architecture size (while still increasing the depth of the overall network) by removing fully connected layers and instead of using global average pooling. Most of the weights in a CNN can be found in the dense FC layers – if these layers can be removed, the memory savings are massive.\n",
        "\n",
        "Secondly, the Szegedy et al. paper makes usage of a network in-network or micro-architecture when constructing the overall macro-architecture. Up to this point, we have seen only sequential neural networks where the output of one network feeds directly into the next. We are now going to see micro-architectures, small building blocks used inside the rest of the architecture, where the output from one layer can split into various paths and be rejoined later.\n",
        "\n",
        "Specifically, Szegedy et al. contributed the Inception module to the deep learning community, a building block that fits into a Convolutional Neural Network, enabling it to learn CONV layers with **multiple filter sizes**, turning the module into a multi-level feature extractor.\n",
        "\n",
        "Micro-architectures such as Inception have inspired other significant variants, including the Residual module in [ResNet](https://arxiv.org/abs/1512.03385) and the Fire module in [SqueezeNet](https://arxiv.org/abs/1602.07360). We will be discussing the Inception module (and its variants) later in this section. Once we have examined the Inception module and ensure we know how it works, we will then implement a smaller version of GoogLeNet called \"MiniGoogLeNet\" – we will train this architecture on the CIFAR-10 dataset and obtain higher accuracy than in any of our previous section using VGG.\n",
        "\n",
        "From there, we will move on to the more difficult [cs231n Tiny ImageNet Challenge](http://cs231n.stanford.edu/project.html). This challenge is offered to students enrolled in [Stanford’s cs231n Convolutional Neural Networks for Visual Recognition class](http://cs231n.stanford.edu/) as part of their final project. It means to give them a taste of the challenges associated with large-scale deep learning on modern architectures without being as time-consuming or taxing to work with as the entire ImageNet dataset.\n",
        "\n",
        "By training GoogLeNet from scratch on Tiny ImageNet, we will demonstrate how to obtain a top ranking position on the Tiny ImageNet leaderboard. Furthermore, in our next section, we will utilize ResNet to claim the top position from models trained from scratch. Let us go ahead and get this section started by discussing the Inception module."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k0KUD4f1bhL7"
      },
      "source": [
        "## 3.1 The Inception Module (and its Variants)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BYKoL0TxcSxi"
      },
      "source": [
        "Modern state-of-the-art Convolutional Neural Networks utilize **micro-architectures**, also called **network-in-network modules**, initially proposed by [Lin et al](https://arxiv.org/abs/1312.4400). I prefer the term micro-architecture better describes these modules as building blocks in the context of the overall macro-architecture (i.e., what you build and train).\n",
        "\n",
        "Micro-architectures are tiny building blocks designed by deep learning practitioners to enable networks to learn (1) faster and (2) more efficiently, all while increasing network depth. \n",
        "> These micro-architecture building blocks are stacked, along with conventional layer types such as CONV, POOL, etc., to form the overall macro-architecture.\n",
        "\n",
        "In 2014, Szegedy et al. introduced the Inception module. The general idea behind the Inception module is two-fold:\n",
        "\n",
        "1. It can be hard to decide the size of the filter you need to learn at given CONV layers. Should they be 5x5 filters? What about 3x3 filters? Should we learn local features using 1x1 filters? Instead, why not learn them all and let the model decide? Inside the Inception module, we learn all three 5x5, 3x3, and 1x1 filters (computing them in parallel), concatenating the resulting feature maps along the channel dimension. The next layer in\n",
        "the GoogLeNet architecture (which could be another Inception module) receives these concatenated, mixed filters and performs the same process. Taken as a whole, this process enables GoogLeNet to learn both local features via smaller convolutions and abstracted\n",
        "features with larger convolutions – we do not have to sacrifice our level of abstraction at the expense of smaller features.\n",
        "2. By learning multiple filter sizes, we can turn the module into a multi-level feature extractor. The 5x5 filters have a larger receptive size and can learn more abstract features. The 1x1 filters are, by definition, local. The 3x3 filters sit as a balance in between."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2ap9n_4df8NB"
      },
      "source": [
        "### 3.1.1 Inception"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jIm6Tf14gDi_"
      },
      "source": [
        "Now that we’ve discussed the motivation behind the Inception module, let’s look at the actual\n",
        "module itself in Figure below (the original Inception module used in GoogLeNet).\n",
        "\n",
        "<img width=\"450\" src=\"https://drive.google.com/uc?export=view&id=1ja1dLUSBtZSxBwsbimHUbho0cEvno0Cb\"/>\n",
        "\n",
        "Specifically, take note of how the Inception module branches into four distinct paths from the input layer. The first branch in the Inception module learns a series of 1x1 local features from the input.\n",
        "\n",
        "The second batch first applies 1x1 convolutions, not only as a form of learning local features but instead as dimensionality reduction. Larger convolutions (i.e., 3x3 and 5x5) by definition take more computation to perform. Therefore, if we can reduce the dimensionality of the inputs\n",
        "to these larger filters by applying 1x1 convolutions, we can reduce the computation required by our network. Therefore, the number of filters learned in the 1x1 CONV in the second branch will always be smaller than the number of 3x3 filters learned directly afterward.\n",
        "\n",
        "The third branch applies the same logic as the second branch, only this time to learn 5x5 filters. We once again reduce dimensionality via 1x1 convolutions, then feed the output into the 5x5 filters.\n",
        "\n",
        "The fourth and final branch of the Inception module performs 3x3 max pooling with a stride of 1x1 – this branch is commonly referred to as the pool projection branch. Historically, models that perform pooling have demonstrated an ability to obtain higher accuracy, although we now\n",
        "know through the work of Springenberg et al. in their 2014 paper, [Striving for Simplicity: The All Convolutional Net](https://arxiv.org/abs/1412.6806) that this is not necessarily true and that POOL layers can be replaced with CONV layers for reducing volume size.\n",
        "\n",
        "In the case of Szegedy et al., this POOL layer was added simply because it was thought that they were needed for CNNs to perform reasonably. The output of the POOL is then fed into another series of 1x1 convolutions to learn local features.\n",
        "\n",
        "Finally, all four-interception modules converge where they are concatenated together along the channel dimension. Special care is taken during the implementation (via zero paddings) to ensure the output of each branch has the same volume size, thereby allowing the outputs to be concatenated. The output of the Inception module is then fed into the next layer in the network. In practice, we often stack multiple Inception modules on top of each other before performing a pooling operation to reduce volume size."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wVCUMHbNidli"
      },
      "source": [
        "### 3.1.2 Miniception"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4mA5jF38lrYD"
      },
      "source": [
        "Of course, the original Inception module was designed for GoogLeNet. It could be trained on the ImageNet dataset (where each input image is assumed to be 224x22x43) and obtain state-of-the-art accuracy. We can simplify the Inception module for smaller datasets (with smaller image spatial dimensions) where fewer network parameters are required.\n",
        "\n",
        "A tiny version of Inception was developed from Zhang et al. in a 2017\n",
        "publication, [Understanding Deep Learning Requires Re-Thinking Generalization](https://arxiv.org/pdf/1611.03530.pdf). The top row of the figure below describes three modules used in their MiniGoogLeNet implementation.\n",
        "\n",
        "<img width=\"750\" src=\"https://drive.google.com/uc?export=view&id=1TlTRgCzY_Af-laTm_x29VT-Ud1V5lgpA\"/>\n",
        "\n",
        "- **Left**: A convolution module responsible for performing convolution, batch normalization, and activation.\n",
        "- **Middle**: The Miniception module, which performs two sets of convolutions, one for 1x1 filters and the other for 3x3 filters, then concatenates the results. No dimensionality reduction is performed before the 3x3 filter as (1) the input volumes will be smaller already (since we’ll be using the CIFAR-10 dataset) and (2) to reduce the number of parameters in\n",
        "the network.\n",
        "- **Right**: A downsample module which applies both convolution and max-pooling to reduce dimensionality, then concatenates across the filter dimension.\n",
        "\n",
        "\n",
        "These building blocks are then used to build the MiniGoogLeNet architecture on the bottom row. You’ll notice here that the authors placed the batch normalization before the activation (presumably because this is what Szegedy et al. did as well), in contrast to what is now recommended when implementing CNNs.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oAoCjjWX_3UY"
      },
      "source": [
        "## 3.2 MiniGoogLeNet on CIFAR-10"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e1ue5E2J-WnK"
      },
      "source": [
        "In this section, we are going to implement the MiniGoogLeNet architecture using the Miniception module. We’ll then train MiniGoogLeNet on the CIFAR-10 dataset. As our results demonstrate, this architecture will obtain > 90% accuracy on CIFAR-10, far better than our previous attempts using miniVGG."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FCt2QjPg-7hJ"
      },
      "source": [
        "### 3.2.1 Implementing MiniGoogLeNet"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7u00tYNr-sQN"
      },
      "source": [
        "# import the necessary packages\n",
        "from tensorflow.keras.layers import BatchNormalization\n",
        "from tensorflow.keras.layers import Conv2D\n",
        "from tensorflow.keras.layers import AveragePooling2D\n",
        "from tensorflow.keras.layers import MaxPooling2D\n",
        "from tensorflow.keras.layers import Activation\n",
        "from tensorflow.keras.layers import Dropout\n",
        "from tensorflow.keras.layers import Dense\n",
        "from tensorflow.keras.layers import Flatten\n",
        "from tensorflow.keras.layers import Input\n",
        "from tensorflow.keras.models import Model\n",
        "from tensorflow.keras.layers import concatenate\n",
        "from tensorflow.keras import backend as K\n",
        "\n",
        "class MiniGoogLeNet:\n",
        "\t@staticmethod\n",
        "\tdef conv_module(x, K, kX, kY, stride, chanDim, padding=\"same\"):\n",
        "\t\t# define a CONV => BN => RELU pattern\n",
        "\t\tx = Conv2D(K, (kX, kY), strides=stride, padding=padding)(x)\n",
        "\t\tx = BatchNormalization(axis=chanDim)(x)\n",
        "\t\tx = Activation(\"relu\")(x)\n",
        "\n",
        "\t\t# return the block\n",
        "\t\treturn x\n",
        "\n",
        "\t@staticmethod\n",
        "\tdef inception_module(x, numK1x1, numK3x3, chanDim):\n",
        "\t\t# define two CONV modules, then concatenate across the\n",
        "\t\t# channel dimension\n",
        "\t\tconv_1x1 = MiniGoogLeNet.conv_module(x, numK1x1, 1, 1,(1, 1), chanDim)\n",
        "\t\tconv_3x3 = MiniGoogLeNet.conv_module(x, numK3x3, 3, 3,(1, 1), chanDim)\n",
        "\t\tx = concatenate([conv_1x1, conv_3x3], axis=chanDim)\n",
        "\n",
        "\t\t# return the block\n",
        "\t\treturn x\n",
        "\n",
        "\t@staticmethod\n",
        "\tdef downsample_module(x, K, chanDim):\n",
        "\t\t# define the CONV module and POOL, then concatenate\n",
        "\t\t# across the channel dimensions\n",
        "\t\tconv_3x3 = MiniGoogLeNet.conv_module(x, K, 3, 3, (2, 2),chanDim, padding=\"valid\")\n",
        "\t\tpool = MaxPooling2D((3, 3), strides=(2, 2))(x)\n",
        "\t\tx = concatenate([conv_3x3, pool], axis=chanDim)\n",
        "\n",
        "\t\t# return the block\n",
        "\t\treturn x\n",
        "\n",
        "\t@staticmethod\n",
        "\tdef build(width, height, depth, classes):\n",
        "\t\t# initialize the input shape to be \"channels last\" and the\n",
        "\t\t# channels dimension itself\n",
        "\t\tinputShape = (height, width, depth)\n",
        "\t\tchanDim = -1\n",
        "\n",
        "\t\t# if we are using \"channels first\", update the input shape\n",
        "\t\t# and channels dimension\n",
        "\t\tif K.image_data_format() == \"channels_first\":\n",
        "\t\t\tinputShape = (depth, height, width)\n",
        "\t\t\tchanDim = 1\n",
        "\n",
        "\t\t# define the model input and first CONV module\n",
        "\t\tinputs = Input(shape=inputShape)\n",
        "\t\tx = MiniGoogLeNet.conv_module(inputs, 96, 3, 3, (1, 1),chanDim)\n",
        "\n",
        "\t\t# two Inception modules followed by a downsample module\n",
        "\t\tx = MiniGoogLeNet.inception_module(x, 32, 32, chanDim)\n",
        "\t\tx = MiniGoogLeNet.inception_module(x, 32, 48, chanDim)\n",
        "\t\tx = MiniGoogLeNet.downsample_module(x, 80, chanDim)\n",
        "\n",
        "\t\t# four Inception modules followed by a downsample module\n",
        "\t\tx = MiniGoogLeNet.inception_module(x, 112, 48, chanDim)\n",
        "\t\tx = MiniGoogLeNet.inception_module(x, 96, 64, chanDim)\n",
        "\t\tx = MiniGoogLeNet.inception_module(x, 80, 80, chanDim)\n",
        "\t\tx = MiniGoogLeNet.inception_module(x, 48, 96, chanDim)\n",
        "\t\tx = MiniGoogLeNet.downsample_module(x, 96, chanDim)\n",
        "\n",
        "\t\t# two Inception modules followed by global POOL and dropout\n",
        "\t\tx = MiniGoogLeNet.inception_module(x, 176, 160, chanDim)\n",
        "\t\tx = MiniGoogLeNet.inception_module(x, 176, 160, chanDim)\n",
        "\t\tx = AveragePooling2D((7, 7))(x)\n",
        "\t\tx = Dropout(0.5)(x)\n",
        "\n",
        "\t\t# softmax classifier\n",
        "\t\tx = Flatten()(x)\n",
        "\t\tx = Dense(classes)(x)\n",
        "\t\tx = Activation(\"softmax\")(x)\n",
        "\n",
        "\t\t# create the model\n",
        "\t\tmodel = Model(inputs, x, name=\"googlenet\")\n",
        "\n",
        "\t\t# return the constructed network architecture\n",
        "\t\treturn model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JLIHQfdJAGvD"
      },
      "source": [
        "### 3.2.2 Training and Evaluating MiniGoogLeNet on CIFAR-10"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k0Bqdf-VQ0JK"
      },
      "source": [
        "In this training we are importing the **LearningRateScheduler class**, which\n",
        "implies that we’ll be defining a specific learning rate for our optimizer to follow when training the\n",
        "network. Specifically, we’ll be defining a polynomial decay learning rate schedule. A polynomial learning rate scheduler will follow the equation:\n",
        "\n",
        "$$\n",
        "\\displaystyle \\alpha = \\alpha_0 \\times (1 - \\frac{e}{e_{max}})^p\n",
        "$$\n",
        "\n",
        "Where $\\alpha_0$ is the **initial learning rate**, $e$ is the **current epoch number**, $e_{max}$ is the **maximum number of epochs** we are going to perform, and $p$ is the **power of the polynomial**. Applying this equation yields the learning rate $\\alpha$ for the current epoch."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "x9KU0iv4NKFm"
      },
      "source": [
        "# definine the total number of epochs to train for along with the\n",
        "# initial learning rate\n",
        "NUM_EPOCHS = 70\n",
        "INIT_LR = 5e-3\n",
        "power_base = 1.0\n",
        "\n",
        "def poly_decay(epoch):\n",
        "\t# initialize the maximum number of epochs, base learning rate,\n",
        "\t# and power of the polynomial\n",
        "\tmaxEpochs = NUM_EPOCHS\n",
        "\tbaseLR = INIT_LR\n",
        "\tpower = power_base\n",
        "\n",
        "\t# compute the new learning rate based on polynomial decay\n",
        "\talpha = baseLR * (1 - (epoch / float(maxEpochs))) ** power\n",
        "\n",
        "\t# return the new learning rate\n",
        "\treturn alpha\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NSa9sseDSKSl"
      },
      "source": [
        "Given the **maximum number of epochs**, the learning rate will decay to zero. This learning rate scheduler can also be made linear by setting the power to 1.0 – which is often done – and, in fact, what we are going to do in this example. We have included a number of example polynomial learning rate schedules using a maximum of **70 epochs**, an initial learning rate of $5e-3$, and varying powers in Figure below. Notice how as the power increases, the faster the learning rate drops. Using a power of 1.0 turns the curve into a linear decay."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E_in5I4ANO1Y"
      },
      "source": [
        "result = dict()\n",
        "for p in [1.0, 1.5, 2.0, 3.0]:\n",
        "  power_base = p\n",
        "  result[power_base] = []\n",
        "  for i in range(70):\n",
        "    result[power_base].append(poly_decay(i))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "X6yroJmKOM99"
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "plt.style.use(\"fivethirtyeight\")\n",
        "fig, ax = plt.subplots(1,1, figsize=(8,6))\n",
        "for i in range(4):\n",
        "  ax.plot(range(70),list(result.values())[i])\n",
        "\n",
        "ax.set_title(\"Examples of Polynomical Learning Rate Decay\")\n",
        "ax.set_ylabel(\"Learning Rate\")\n",
        "ax.set_xlabel(\"Epoch\")\n",
        "ax.legend([\"p = 1.0\", \"p = 1.5\", \"p = 2.0\", \"p = 3.0\"])\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JbOFbmHkXUHc"
      },
      "source": [
        "%%capture\n",
        "!pip install wandb"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YaeJikmzXUHd"
      },
      "source": [
        "!wandb login"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JSZcRDbrXfuH"
      },
      "source": [
        "import wandb\n",
        "\n",
        "# Set an experiment name to group training and evaluation\n",
        "experiment_name = wandb.util.generate_id()\n",
        "\n",
        "# setup wandb\n",
        "wandb.init(project=\"lesson07_MiniGoogLeNet\", \n",
        "           group=experiment_name,\n",
        "           config={\n",
        "               \"epoch\": 70,\n",
        "               \"init_lr\": 5e-3\n",
        "           })\n",
        "config = wandb.config"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NZDhO_gxAYMi"
      },
      "source": [
        "%%wandb\n",
        "\n",
        "# set the matplotlib backend so figures can be saved in the background\n",
        "import matplotlib\n",
        "\n",
        "# import the necessary packages\n",
        "from sklearn.preprocessing import LabelBinarizer\n",
        "from tensorflow.keras.preprocessing.image import ImageDataGenerator\n",
        "from tensorflow.keras.callbacks import LearningRateScheduler\n",
        "from tensorflow.keras.optimizers import SGD\n",
        "from tensorflow.keras.datasets import cifar10\n",
        "import numpy as np\n",
        "from wandb.keras import WandbCallback\n",
        "\n",
        "\n",
        "# definine the total number of epochs to train for along with the\n",
        "# initial learning rate\n",
        "NUM_EPOCHS = config.epoch\n",
        "INIT_LR = config.init_lr\n",
        "power_base = 1.0\n",
        "\n",
        "# load the training and testing data, converting the images from\n",
        "# integers to floats\n",
        "print(\"[INFO] loading CIFAR-10 data...\")\n",
        "((train_x, train_y), (test_x, test_y)) = cifar10.load_data()\n",
        "train_x = train_x.astype(\"float\")\n",
        "test_x = test_x.astype(\"float\")\n",
        "\n",
        "# apply mean subtraction to the data\n",
        "mean = np.mean(train_x, axis=0)\n",
        "train_x -= mean\n",
        "test_x -= mean\n",
        "\n",
        "# convert the labels from integers to vectors\n",
        "lb = LabelBinarizer()\n",
        "train_y = lb.fit_transform(train_y)\n",
        "test_y = lb.transform(test_y)\n",
        "\n",
        "# construct the image generator for data augmentation\n",
        "aug = ImageDataGenerator(width_shift_range=0.1,\n",
        "                         height_shift_range=0.1,\n",
        "                         horizontal_flip=True,\n",
        "                         fill_mode=\"nearest\")\n",
        "\n",
        "# construct the set of callbacks\n",
        "callbacks = [LearningRateScheduler(poly_decay),\n",
        "             WandbCallback()]\n",
        "\n",
        "# initialize the optimizer and model\n",
        "print(\"[INFO] compiling model...\")\n",
        "opt = SGD(lr=INIT_LR, momentum=0.9)\n",
        "model = MiniGoogLeNet.build(width=32, height=32, depth=3, classes=10)\n",
        "model.compile(loss=\"categorical_crossentropy\", optimizer=opt,metrics=[\"accuracy\"])\n",
        "\n",
        "# train the network\n",
        "print(\"[INFO] training network...\")\n",
        "model.fit(aug.flow(train_x, train_y, batch_size=64),\n",
        "                    validation_data=(test_x, test_y),\n",
        "                    steps_per_epoch=len(train_x) // 64,\n",
        "                    epochs=NUM_EPOCHS, \n",
        "                    callbacks=callbacks, verbose=1)\n",
        "\n",
        "wandb.finish()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4jltnz0AnATt"
      },
      "source": [
        "model.summary()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sWPNDk2lgq0r"
      },
      "source": [
        "**Log Analysis**\n",
        "\n",
        "Next, log an analysis run, using the same experiment name as the group parameter so that this run and the previous run are grouped together in W&B."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bZjQgiZwgq0t"
      },
      "source": [
        "%%capture\n",
        "# Install dependencies\n",
        "!pip install scikit-plot -qqq"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OMIg_nYRgq0t"
      },
      "source": [
        "import numpy as np\n",
        "from sklearn.metrics import f1_score\n",
        "import matplotlib.pyplot as plt\n",
        "import tensorflow as tf\n",
        "from scikitplot.metrics import plot_confusion_matrix, plot_roc, plot_precision_recall\n",
        "\n",
        "wandb.init(project=\"lesson07_MiniGoogLeNet\", group=experiment_name)\n",
        "\n",
        "# initialize the label names for the CIFAR-10 dataset\n",
        "labelNames = [\"airplane\", \"automobile\", \"bird\", \"cat\", \"deer\",\n",
        "              \"dog\", \"frog\", \"horse\", \"ship\", \"truck\"]\n",
        "\n",
        "# Class proportions\n",
        "train_y_labels = [labelNames[i] for i in np.argmax(train_y, axis=1)]\n",
        "test_y_labels = [labelNames[i] for i in np.argmax(test_y, axis=1)]\n",
        "wandb.log({'Class Proportions': wandb.sklearn.plot_class_proportions(train_y_labels,\n",
        "                                                                     test_y_labels,\n",
        "                                                                     labelNames)},\n",
        "           commit=False) # Hold on, more incoming!\n",
        "\n",
        "# Log F1 Score\n",
        "test_y_pred = np.asarray(model.predict(test_x))\n",
        "test_y_pred_class = np.argmax(test_y_pred, axis=1)\n",
        "f1 = f1_score(np.argmax(test_y, axis=1), test_y_pred_class, average='micro')\n",
        "wandb.log({\"f1\": f1}, commit=False)\n",
        "\n",
        "#test_y_labels = [labelNames[i] for i in np.argmax(test_y, axis=1)]\n",
        "test_y_pred_labels = [labelNames[i] for i in test_y_pred_class]\n",
        "\n",
        "# Log Confusion Matrix\n",
        "fig, ax = plt.subplots(figsize=(16, 12))\n",
        "plot_confusion_matrix(test_y_labels, test_y_pred_labels, ax=ax)\n",
        "wandb.log({\"confusion_matrix\": wandb.Image(fig)}, commit=False)\n",
        "\n",
        "# Log ROC Curve\n",
        "fig, ax = plt.subplots(figsize=(16, 12))\n",
        "plot_roc(test_y_labels, test_y_pred, ax=ax)\n",
        "wandb.log({\"plot_roc\": wandb.Image(fig)},commit=False)  # Now we've logged everything for this step\n",
        "\n",
        "# Precision vs Recall\n",
        "fig, ax = plt.subplots(figsize=(16, 12))\n",
        "plot_precision_recall(test_y_labels, test_y_pred, ax=ax)\n",
        "wandb.log({\"plot_precision_recall\": wandb.Image(fig)},commit=False)  # Now we've logged everything for this step\n",
        "\n",
        "# Class Scores\n",
        "class_score_data = []\n",
        "for test, pred in zip(test_y_labels, test_y_pred):\n",
        "    class_score_data.append([test, pred])\n",
        "\n",
        "wandb.log({\"class_scores\": wandb.Table(data=class_score_data,\n",
        "                                           columns=[\"test\", \"pred\"])}, commit=False)\n",
        "\n",
        "# \n",
        "# Visualize Predictions\n",
        "# \n",
        "# visualize 18 images\n",
        "def show_image(train_image, label, index):\n",
        "    plt.subplot(3, 6, index+1)\n",
        "    plt.imshow(tf.squeeze(train_image), cmap=plt.cm.gray)\n",
        "    plt.title(label)\n",
        "    plt.grid(b=False)\n",
        "\n",
        "# predictions\n",
        "predictions = model.predict(test_x)\n",
        "results = np.argmax(predictions, axis = 1)\n",
        "\n",
        "# visualize the first 18 test results\n",
        "plt.figure(figsize=(12, 8))\n",
        "for index in range(18):\n",
        "    label = results[index]\n",
        "    image_pixels = test_x[index,:,:,:]/255\n",
        "    show_image(image_pixels, labelNames[label], index)\n",
        "plt.tight_layout()\n",
        "\n",
        "wandb.log({\"Predictions\": plt}, commit=True)\n",
        "\n",
        "wandb.finish()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6zBRodsPCXxn"
      },
      "source": [
        "from sklearn.metrics import classification_report\n",
        "\n",
        "# evaluate the network\n",
        "print(\"[INFO] evaluating network...\")\n",
        "predictions = model.predict(test_x, batch_size=64)\n",
        "print(classification_report(test_y.argmax(axis=1),\n",
        "                            predictions.argmax(axis=1), target_names=labelNames))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kXg9aJG6jHtm"
      },
      "source": [
        "## 3.3 The Tiny ImageNet Challenge"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wy2ssnLaNOuU"
      },
      "source": [
        "The Tiny ImageNet Visual Recognition Challenge (a sample of which can be seen in Figure below) is part of the [cs231n Stanford course on Convolutional Neural Networks](http://cs231n.stanford.edu/) for Visual Recognition. As part of their final project, students can compete in the classification by either training a CNN from scratch or performing transfer learning via fine-tuning (this topic will be presented later in our course).\n",
        "\n",
        "<img width=\"850\" src=\"https://drive.google.com/uc?export=view&id=1IVTouM03vjm30eOb7M6p9pM0yHLHLcpm\"/>\n",
        "\n",
        "The [Tiny ImageNet dataset](https://drive.google.com/file/d/1ZZcGmX3s5bOb9A_El5RAHeJyau_zFgn_/view?usp=sharing) is actually a subset of the entire [ImageNet dataset](https://image-net.org/), consisting of 200 diverse classes, including everything from Egyptian cats to volleyballs to lemons. Given that there are 200 classes, guessing at random, we would expect to be correct 1/200 = 0.5%\n",
        "of the time; therefore, our CNN needs to obtain at least 0.5% to demonstrate it has learned underlying discriminative patterns in the respective classes.\n",
        "\n",
        "> Each class includes 450 training images, 50 validation images, and 50 testing images. \n",
        "\n",
        "Groundtruth labels are only provided for the training and validation images. Since we do not have access to the Tiny ImageNet evaluation server, we will use part of the training set to form our testing set to evaluate the performance of our classification algorithms.\n",
        "\n",
        "The images in the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) have varying widths and heights. Therefore, whenever we work with ILSVRC, we first need to resize all images in the dataset to a fixed width and height before we can train our network. To help students focus strictly on the deep learning and image classification component (and not get caught up in image processing details), all images in the Tiny ImageNet dataset have been resized to 64x64 pixels and center cropped.\n",
        "\n",
        "In some ways, having the images resized makes Tiny ImageNet a bit more challenging than its bigger brother, ILSVRC. In ILSVRC, we are free to apply any type of resizing, cropping, etc., operations that we see fit. However, with Tiny ImageNet, much of the image has already been discarded for us. As we’ll find out, obtaining a reasonable rank-1 and rank-5 accuracy on Tiny ImageNet isn’t as easy as one might think, making it a great, insightful dataset for budding deep learning practitioners to learn and practice on.\n",
        "\n",
        "In the next few sections, you will learn how to obtain the Tiny ImageNet dataset, understand its structure, and create HDF5 files for the training, validation, and testing images.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JSEWn6mOnsyk"
      },
      "source": [
        "# --- tiny-imagenet-200\n",
        "#   |--- test\n",
        "#       | --- images\n",
        "#             | --- test_00.JPEG \n",
        "#   |--- train\n",
        "#       | --- nxxyyzzuu \n",
        "#             | --- images\n",
        "#                   | --- nxxyyzzuu_00.JPEG \n",
        "#             | --- nxxyyzzuu_boxes.txt\n",
        "#   |--- val\n",
        "#       | --- images\n",
        "#             | --- val_00.JPEG \n",
        "#             | --- val_annotations.txt\n",
        "#   |--- wnids.txt\n",
        "#   |--- words.txt\n",
        "\n",
        "# download Tiny ImageNet dataset (tiny-imagenet-200.zip)\n",
        "!gdown https://drive.google.com/uc?id=1ZZcGmX3s5bOb9A_El5RAHeJyau_zFgn_"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0t2ZauzHFDR1"
      },
      "source": [
        "!unzip tiny-imagenet-200.zip"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N2I-oNtGFTcM"
      },
      "source": [
        "Inside the test directory are the testing images – we will be ignoring these images since we do not have access to the cs231n evaluation server (the labels are purposely left out from the download to ensure no one can “cheat” in the challenge).\n",
        "\n",
        "We then have the train directory, which contains subdirectories with strange names starting with the letter <font color=\"red\">n</font> followed by a series of numbers. These subdirectories are the [WordNet](https://wordnet.princeton.edu/) IDs called “synonym set” or “synsets” for short. Each WordNet ID maps to a specific word/object.\n",
        "Every image inside a given WordNet subdirectory contains examples of that object.\n",
        "\n",
        "We can lookup the human-readable label for a WordNet ID by parsing the **words.txt** file, which is simply a tab-separated file with the WordNet ID in the first column and the human-readable word/object in the second column. The **wnids.txt** file lists out the 200 WordNet IDs (one per line) in the ImageNet dataset.\n",
        "\n",
        "Finally, the val directory stores our validation set. Inside the val directory, you will find an images subdirectory and a file named **val_annotations.txt**. The **val_annotations.txt** provides the WordNet IDs for every image in the val directory.\n",
        "\n",
        "Therefore, before we can even get started training GoogLeNet on Tiny ImageNet, we first need to write a script to parse them and put them into the HDF5 format. Keep in mind that being a deep learning practitioner is not about implementing Convolutional Neural Networks and training them from scratch. Being a deep learning practitioner involves using your programming skills to build simple scripts that can parse data.\n",
        "\n",
        "The more general-purpose programming skills you have, the better deep learning practitioner you can become – while other deep learning researchers are struggling to organize files on disk or understand how a dataset is structured, you’ll have already converted your entire dataset to a format suitable for training a CNN.\n",
        "\n",
        "In the next section, we’ll teach you how to define your project configuration file and create a single, simple Python script that will convert the Tiny ImageNet dataset into an HDF5 representation."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iSCpVoelLMcA"
      },
      "source": [
        "### 3.3.1 Building the Tiny ImageNet Dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l0M8Gf5iS6d5"
      },
      "source": [
        "Let’s go ahead and define the project structure for Tiny ImageNet + GoogLeNet:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bvDLGopySkgg"
      },
      "source": [
        "# import the necessary packages\n",
        "from os import path\n",
        "\n",
        "# define the paths to the training and validation directories\n",
        "TRAIN_IMAGES = \"tiny-imagenet-200/train\"\n",
        "VAL_IMAGES = \"tiny-imagenet-200/val/images\"\n",
        "\n",
        "# define the path to the file that maps validation filenames to\n",
        "# their corresponding class labels\n",
        "VAL_MAPPINGS = \"tiny-imagenet-200/val/val_annotations.txt\"\n",
        "\n",
        "# define the paths to the WordNet hierarchy files which are used\n",
        "# to generate our class labels\n",
        "WORDNET_IDS = \"tiny-imagenet-200/wnids.txt\"\n",
        "WORD_LABELS = \"tiny-imagenet-200/words.txt\"\n",
        "\n",
        "# since we do not have access to the testing data we need to\n",
        "# take a number of images from the training data and use it instead\n",
        "NUM_CLASSES = 200\n",
        "NUM_TEST_IMAGES = 50 * NUM_CLASSES\n",
        "\n",
        "# define the path to the output training, validation, and testing\n",
        "# HDF5 files\n",
        "TRAIN_HDF5 = \"tiny-imagenet-200/hdf5/train.hdf5\"\n",
        "VAL_HDF5 = \"tiny-imagenet-200/hdf5/val.hdf5\"\n",
        "TEST_HDF5 = \"tiny-imagenet-200/hdf5/test.hdf5\"\n",
        "\n",
        "# define the path to the dataset mean\n",
        "DATASET_MEAN = \"tiny-imagenet-200/output/tiny-image-net-200-mean.json\"\n",
        "\n",
        "# define the path to the output directory used for storing plots,\n",
        "# classification reports, etc.\n",
        "OUTPUT_PATH = \"tiny-imagenet-200/output\"\n",
        "MODEL_PATH = path.sep.join([OUTPUT_PATH,\"epoch_60.hdf5\"])\n",
        "FIG_PATH = path.sep.join([OUTPUT_PATH,\"deepergooglenet_tinyimagenet.png\"])\n",
        "JSON_PATH = path.sep.join([OUTPUT_PATH,\"deepergooglenet_tinyimagenet.json\"])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NPUB6a_RWkft"
      },
      "source": [
        "!mkdir tiny-imagenet-200/hdf5\n",
        "!mkdir tiny-imagenet-200/output"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J2FG8tGiUd6-"
      },
      "source": [
        "As you can see, this configuration file is fairly straightforward. We are mainly just defining paths to input directories of images/label mappings along with output files. However, taking the time to create this configuration file makes our life much easier when actually building Tiny ImageNet and converting it to HDF5."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v0hRRgWSVKwe"
      },
      "source": [
        "# import the necessary packages\n",
        "import h5py\n",
        "import os\n",
        "\n",
        "class HDF5DatasetWriter:\n",
        "\tdef __init__(self, dims, outputPath, dataKey=\"images\",bufSize=1000):\n",
        "\t\t# check to see if the output path exists, and if so, raise\n",
        "\t\t# an exception\n",
        "\t\tif os.path.exists(outputPath):\n",
        "\t\t\traise ValueError(\"The supplied `outputPath` already \"\n",
        "\t\t\t\t\"exists and cannot be overwritten. Manually delete \"\n",
        "\t\t\t\t\"the file before continuing.\", outputPath)\n",
        "\n",
        "\t\t# open the HDF5 database for writing and create two datasets:\n",
        "\t\t# one to store the images/features and another to store the\n",
        "\t\t# class labels\n",
        "\t\tself.db = h5py.File(outputPath, \"w\")\n",
        "\t\tself.data = self.db.create_dataset(dataKey, dims,dtype=\"float\")\n",
        "\t\tself.labels = self.db.create_dataset(\"labels\", (dims[0],),dtype=\"int\")\n",
        "\n",
        "\t\t# store the buffer size, then initialize the buffer itself\n",
        "\t\t# along with the index into the datasets\n",
        "\t\tself.bufSize = bufSize\n",
        "\t\tself.buffer = {\"data\": [], \"labels\": []}\n",
        "\t\tself.idx = 0\n",
        "\n",
        "\tdef add(self, rows, labels):\n",
        "\t\t# add the rows and labels to the buffer\n",
        "\t\tself.buffer[\"data\"].extend(rows)\n",
        "\t\tself.buffer[\"labels\"].extend(labels)\n",
        "\n",
        "\t\t# check to see if the buffer needs to be flushed to disk\n",
        "\t\tif len(self.buffer[\"data\"]) >= self.bufSize:\n",
        "\t\t\tself.flush()\n",
        "\n",
        "\tdef flush(self):\n",
        "\t\t# write the buffers to disk then reset the buffer\n",
        "\t\ti = self.idx + len(self.buffer[\"data\"])\n",
        "\t\tself.data[self.idx:i] = self.buffer[\"data\"]\n",
        "\t\tself.labels[self.idx:i] = self.buffer[\"labels\"]\n",
        "\t\tself.idx = i\n",
        "\t\tself.buffer = {\"data\": [], \"labels\": []}\n",
        "\n",
        "\tdef storeClassLabels(self, classLabels):\n",
        "\t\t# create a dataset to store the actual class label names,\n",
        "\t\t# then store the class labels\n",
        "\t\tdt = h5py.special_dtype(vlen=str) # `vlen=unicode` for Py2.7\n",
        "\t\tlabelSet = self.db.create_dataset(\"label_names\",(len(classLabels),), dtype=dt)\n",
        "\t\tlabelSet[:] = classLabels\n",
        "\n",
        "\tdef close(self):\n",
        "\t\t# check to see if there are any other entries in the buffer\n",
        "\t\t# that need to be flushed to disk\n",
        "\t\tif len(self.buffer[\"data\"]) > 0:\n",
        "\t\t\tself.flush()\n",
        "\n",
        "\t\t# close the dataset\n",
        "\t\tself.db.close()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EQLCrRt7UCoa"
      },
      "source": [
        "from sklearn.preprocessing import LabelEncoder\n",
        "from sklearn.model_selection import train_test_split\n",
        "from imutils import paths\n",
        "import numpy as np\n",
        "import progressbar\n",
        "import json\n",
        "import cv2\n",
        "import os\n",
        "\n",
        "# grab the paths to the training images, then extract the training\n",
        "# class labels and encode them\n",
        "trainPaths = list(paths.list_images(TRAIN_IMAGES))\n",
        "trainLabels = [p.split(os.path.sep)[-3] for p in trainPaths]\n",
        "le = LabelEncoder()\n",
        "trainLabels = le.fit_transform(trainLabels)\n",
        "\n",
        "# perform stratified sampling from the training set to construct a\n",
        "# a testing set\n",
        "split = train_test_split(trainPaths, trainLabels,test_size=NUM_TEST_IMAGES, \n",
        "                         stratify=trainLabels,random_state=42)\n",
        "(trainPaths, testPaths, trainLabels, testLabels) = split\n",
        "\n",
        "# load the validation filename => class from file and then use these\n",
        "# mappings to build the validation paths and label lists\n",
        "M = open(VAL_MAPPINGS).read().strip().split(\"\\n\")\n",
        "M = [r.split(\"\\t\")[:2] for r in M]\n",
        "valPaths = [os.path.sep.join([VAL_IMAGES, m[0]]) for m in M]\n",
        "valLabels = le.transform([m[1] for m in M])\n",
        "\n",
        "# construct a list pairing the training, validation, and testing\n",
        "# image paths along with their corresponding labels and output HDF5\n",
        "# files\n",
        "datasets = [\n",
        "\t(\"train\", trainPaths, trainLabels, TRAIN_HDF5),\n",
        "\t(\"val\", valPaths, valLabels, VAL_HDF5),\n",
        "\t(\"test\", testPaths, testLabels, TEST_HDF5)]\n",
        "\n",
        "# initialize the lists of RGB channel averages\n",
        "(R, G, B) = ([], [], [])\n",
        "\n",
        "# loop over the dataset tuples\n",
        "for (dType, paths, labels, outputPath) in datasets:\n",
        "\t# create HDF5 writer\n",
        "\tprint(\"[INFO] building {}...\".format(outputPath))\n",
        "\twriter = HDF5DatasetWriter((len(paths), 64, 64, 3), outputPath)\n",
        "\n",
        "\t# initialize the progress bar\n",
        "\twidgets = [\"Building Dataset: \", progressbar.Percentage(), \" \",\n",
        "            progressbar.Bar(), \" \", progressbar.ETA()]\n",
        "\tpbar = progressbar.ProgressBar(maxval=len(paths),widgets=widgets).start()\n",
        "\n",
        "\t# loop over the image paths\n",
        "\tfor (i, (path, label)) in enumerate(zip(paths, labels)):\n",
        "\t\t# load the image from disk\n",
        "\t\timage = cv2.imread(path)\n",
        "\n",
        "\t\t# if we are building the training dataset, then compute the\n",
        "\t\t# mean of each channel in the image, then update the\n",
        "\t\t# respective lists\n",
        "\t\tif dType == \"train\":\n",
        "\t\t\t(b, g, r) = cv2.mean(image)[:3]\n",
        "\t\t\tR.append(r)\n",
        "\t\t\tG.append(g)\n",
        "\t\t\tB.append(b)\n",
        "\n",
        "\t\t# add the image and label to the HDF5 dataset\n",
        "\t\twriter.add([image], [label])\n",
        "\t\tpbar.update(i)\n",
        "\n",
        "\t# close the HDF5 writer\n",
        "\tpbar.finish()\n",
        "\twriter.close()\n",
        "\n",
        "# construct a dictionary of averages, then serialize the means to a\n",
        "# JSON file\n",
        "print(\"[INFO] serializing means...\")\n",
        "D = {\"R\": np.mean(R), \"G\": np.mean(G), \"B\": np.mean(B)}\n",
        "f = open(DATASET_MEAN, \"w\")\n",
        "f.write(json.dumps(D))\n",
        "f.close()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GUdk5fIaWeHT"
      },
      "source": [
        "# evaluate the generated dataset\n",
        "import h5py\n",
        "\n",
        "filenames = [TRAIN_HDF5, VAL_HDF5, TEST_HDF5]\n",
        "for filename in filenames:\n",
        "  db = h5py.File(filename, \"r\")\n",
        "  print(db[\"images\"].shape)\n",
        "  db.close()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1wUg4TL7DQje"
      },
      "source": [
        "# copy hdf5 files to your google drive\n",
        "!cp -r tiny-imagenet-200/hdf5 /content/drive/MyDrive/Atividades/Ensino/Disciplinas/POS-GRADUAÇÃO/Deep\\ Learning/Lessons/Lesson\\ #07/dataset\n",
        "!cp -r tiny-imagenet-200/output /content/drive/MyDrive/Atividades/Ensino/Disciplinas/POS-GRADUAÇÃO/Deep\\ Learning/Lessons/Lesson\\ #07/"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z5-gVeJxFYdo"
      },
      "source": [
        "<font color=\"red\"> Only execute the cell below if you already have hdf5 files stored in your google drive </font>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qNS9YzB7EZqx"
      },
      "source": [
        "# \n",
        "# only in case loading data from drive\n",
        "#\n",
        "!mkdir tiny-imagenet-200\n",
        "!cp -r /content/drive/MyDrive/Atividades/Ensino/Disciplinas/POS-GRADUAÇÃO/Deep\\ Learning/Lessons/Lesson\\ #07/dataset/hdf5 tiny-imagenet-200\n",
        "!cp -r /content/drive/MyDrive/Atividades/Ensino/Disciplinas/POS-GRADUAÇÃO/Deep\\ Learning/Lessons/Lesson\\ #07/output tiny-imagenet-200"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "by2g9Pq8diZN"
      },
      "source": [
        "## 3.4 DeeperGoogLeNet on Tiny ImageNet"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zkAYal3Adjfi"
      },
      "source": [
        "Now that we have our HDF5 representation of the **Tiny ImageNet dataset**, we are ready to train GoogLeNet on it – but instead of using **MiniGoogLeNet** as in the previous section, we are going to use a deeper variant which more closely models the Szegedy et al. implementation. This deeper variation will use the original Inception module as which will help you understand the original architecture and implement it on your own in the future. To get started, we’ll first learn how to implement this deeper network architecture. \n",
        "\n",
        "We’ll then train DeeperGoogLeNet on the Tiny ImageNet dataset and evaluate the results in terms of **rank-1** and **rank-5** accuracy.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s4qJwYp7N57I"
      },
      "source": [
        "## 3.5 Training DeeperGoogLeNet on Tiny ImageNet"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bEsqjvS4PaZw"
      },
      "source": [
        "\n",
        "We have provided a figure (replicated and modified from Szegedy et al.) detailing our Deeper-GoogLeNet architecture in Figure below. There are only two primary differences between our implementation and the **full GoogLeNet architecture** used by Szegedy et al. when training the network on the complete ImageNet dataset:\n",
        "\n",
        "1. Instead of using 7\u0002x7 filters with a stride of 2\u0002x2 in the first CONV layer, we use 5x5 filters with a 1x1 stride. We use these due to the fact that our implementation of GoogLeNet is only able to accept 64x64x3 input images while the original implementation was constructed to accept 224x224x3 images. If we applied 7x7 filters with a 2x2 stride, we would reduce our input dimensions too quickly.\n",
        "2. Our implementation is slightly shallower with two fewer Inception modules – in the original Szegedy et al. paper, two more Inception modules were added prior to the average pooling operation. This implementation of GoogLeNet will be more than enough for us to perform well on Tiny ImageNet and claim a spot on the cs231n Tiny ImageNet leaderboard. \n",
        "\n",
        "| type           | patch size/stride | output size | depth | #1x1 | #3x3 reduce | #3x3 | #5x5 reduce | #5x5 | pool proj |\n",
        "|----------------|-------------------|-------------|-------|------|-------------|------|-------------|------|-----------|\n",
        "| convolution    | 5x5/1             | 64, 64, 64  | 1     |      |             |      |             |      |           |\n",
        "| max pool       | 3x3/2             | 32, 32, 64  | 0     |      |             |      |             |      |           |\n",
        "| convolution    | 3x3/1             | 32, 32, 192 | 2     |      | 64          | 192  |             |      |           |\n",
        "| max pool       | 3x3/2             | 16, 16, 192 | 0     |      |             |      |             |      |           |\n",
        "| inception (3a) |                   | 16, 16, 256 | 2     | 64   | 96          | 128  | 16          | 32   | 32        |\n",
        "| inception (3b) |                   | 16, 16, 480 | 2     | 128  | 128         | 192  | 32          | 96   | 64        |\n",
        "| max pool       | 3x3/2             | 8, 8, 480   | 0     |      |             |      |             |      |           |\n",
        "| inception (4a) |                   | 8, 8, 512   | 2     | 192  | 96          | 208  | 16          | 48   | 64        |\n",
        "| inception (4b) |                   | 8, 8, 512   | 2     | 160  | 112         | 224  | 24          | 64   | 64        |\n",
        "| inception (4c) |                   | 8, 8, 512   | 2     | 128  | 128         | 256  | 24          | 64   | 64        |\n",
        "| inception (4d) |                   | 8, 8, 528   | 2     | 112  | 144         | 288  | 32          | 64   | 64        |\n",
        "| inception (4e) |                   | 8, 8, 832   | 2     | 256  | 160         | 320  | 32          | 128  | 128       |\n",
        "| max pool       | 3x3/2             | 4, 4, 832   | 0     |      |             |      |             |      |           |\n",
        "| avg pool       | 4x4/1             | 1, 1, 832   | 0     |      |             |      |             |      |           |\n",
        "| dropout (40%)  |                   | 1, 1, 832   | 0     |      |             |      |             |      |           |\n",
        "| linear         |                   | 1, 1, 200   | 1     |      |             |      |             |      |           |\n",
        "| softmax        |                   | 1, 1, 200   | 0     |      |             |      |             |      |           |\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zw0-b-WzSzS3"
      },
      "source": [
        "# import the necessary packages\n",
        "from tensorflow.keras.layers import BatchNormalization\n",
        "from tensorflow.keras.layers import Conv2D\n",
        "from tensorflow.keras.layers import AveragePooling2D\n",
        "from tensorflow.keras.layers import MaxPooling2D\n",
        "from tensorflow.keras.layers import Activation\n",
        "from tensorflow.keras.layers import Dropout\n",
        "from tensorflow.keras.layers import Dense\n",
        "from tensorflow.keras.layers import Flatten\n",
        "from tensorflow.keras.layers import Input\n",
        "from tensorflow.keras.models import Model\n",
        "from tensorflow.keras.layers import concatenate\n",
        "from tensorflow.keras.regularizers import l2\n",
        "from tensorflow.keras import backend as K\n",
        "\n",
        "class DeeperGoogLeNet:\n",
        "\t@staticmethod\n",
        "\tdef conv_module(x, K, kX, kY, stride, chanDim,padding=\"same\", reg=0.0005, name=None):\n",
        "\t\t# initialize the CONV, BN, and RELU layer names\n",
        "\t\t(convName, bnName, actName) = (None, None, None)\n",
        "\n",
        "\t\t# if a layer name was supplied, prepend it\n",
        "\t\tif name is not None:\n",
        "\t\t\tconvName = name + \"_conv\"\n",
        "\t\t\tbnName = name + \"_bn\"\n",
        "\t\t\tactName = name + \"_act\"\n",
        "\n",
        "\t\t# define a CONV => BN => RELU pattern\n",
        "\t\tx = Conv2D(K, (kX, kY), strides=stride, padding=padding, kernel_regularizer=l2(reg), name=convName)(x)\n",
        "\t\tx = BatchNormalization(axis=chanDim, name=bnName)(x)\n",
        "\t\tx = Activation(\"relu\", name=actName)(x)\n",
        "\n",
        "\t\t# return the block\n",
        "\t\treturn x\n",
        "\n",
        "\t@staticmethod\n",
        "\tdef inception_module(x, num1x1, num3x3Reduce, num3x3,num5x5Reduce, num5x5, num1x1Proj, chanDim, stage, reg=0.0005):\n",
        "\t\t# define the first branch of the Inception module which\n",
        "\t\t# consists of 1x1 convolutions\n",
        "\t\tfirst = DeeperGoogLeNet.conv_module(x, num1x1, 1, 1,\n",
        "\t\t\t(1, 1), chanDim, reg=reg, name=stage + \"_first\")\n",
        "\n",
        "\t\t# define the second branch of the Inception module which\n",
        "\t\t# consists of 1x1 and 3x3 convolutions\n",
        "\t\tsecond = DeeperGoogLeNet.conv_module(x, num3x3Reduce, 1, 1, (1, 1), chanDim, reg=reg, name=stage + \"_second1\")\n",
        "\t\tsecond = DeeperGoogLeNet.conv_module(second, num3x3, 3, 3,  (1, 1), chanDim, reg=reg, name=stage + \"_second2\")\n",
        "\n",
        "\t\t# define the third branch of the Inception module which\n",
        "\t\t# are our 1x1 and 5x5 convolutions\n",
        "\t\tthird = DeeperGoogLeNet.conv_module(x, num5x5Reduce, 1, 1, (1, 1), chanDim, reg=reg, name=stage + \"_third1\")\n",
        "\t\tthird = DeeperGoogLeNet.conv_module(third, num5x5, 5, 5, (1, 1), chanDim, reg=reg, name=stage + \"_third2\")\n",
        "\n",
        "\t\t# define the fourth branch of the Inception module which\n",
        "\t\t# is the POOL projection\n",
        "\t\tfourth = MaxPooling2D((3, 3), strides=(1, 1), padding=\"same\", name=stage + \"_pool\")(x)\n",
        "\t\tfourth = DeeperGoogLeNet.conv_module(fourth, num1x1Proj, 1, 1, (1, 1), chanDim, reg=reg, name=stage + \"_fourth\")\n",
        "\n",
        "\t\t# concatenate across the channel dimension\n",
        "\t\tx = concatenate([first, second, third, fourth], axis=chanDim,\n",
        "\t\t\tname=stage + \"_mixed\")\n",
        "\n",
        "\t\t# return the block\n",
        "\t\treturn x\n",
        "\n",
        "\t@staticmethod\n",
        "\tdef build(width, height, depth, classes, reg=0.0005):\n",
        "\t\t# initialize the input shape to be \"channels last\" and the\n",
        "\t\t# channels dimension itself\n",
        "\t\tinputShape = (height, width, depth)\n",
        "\t\tchanDim = -1\n",
        "\n",
        "\t\t# if we are using \"channels first\", update the input shape\n",
        "\t\t# and channels dimension\n",
        "\t\tif K.image_data_format() == \"channels_first\":\n",
        "\t\t\tinputShape = (depth, height, width)\n",
        "\t\t\tchanDim = 1\n",
        "\n",
        "\t\t# define the model input, followed by a sequence of CONV =>\n",
        "\t\t# POOL => (CONV * 2) => POOL layers\n",
        "\t\tinputs = Input(shape=inputShape)\n",
        "\t\tx = DeeperGoogLeNet.conv_module(inputs, 64, 5, 5, (1, 1), chanDim, reg=reg, name=\"block1\")\n",
        "\t\tx = MaxPooling2D((3, 3), strides=(2, 2), padding=\"same\", name=\"pool1\")(x)\n",
        "\t\tx = DeeperGoogLeNet.conv_module(x, 64, 1, 1, (1, 1), chanDim, reg=reg, name=\"block2\")\n",
        "\t\tx = DeeperGoogLeNet.conv_module(x, 192, 3, 3, (1, 1), chanDim, reg=reg, name=\"block3\")\n",
        "\t\tx = MaxPooling2D((3, 3), strides=(2, 2), padding=\"same\", name=\"pool2\")(x)\n",
        "\n",
        "\t\t# apply two Inception modules followed by a POOL\n",
        "\t\tx = DeeperGoogLeNet.inception_module(x, 64, 96, 128, 16, 32, 32, chanDim, \"3a\", reg=reg)\n",
        "\t\tx = DeeperGoogLeNet.inception_module(x, 128, 128, 192, 32, 96, 64, chanDim, \"3b\", reg=reg)\n",
        "\t\tx = MaxPooling2D((3, 3), strides=(2, 2), padding=\"same\", name=\"pool3\")(x)\n",
        "\n",
        "\t\t# apply five Inception modules followed by POOL\n",
        "\t\tx = DeeperGoogLeNet.inception_module(x, 192, 96, 208, 16, 48, 64, chanDim, \"4a\", reg=reg)\n",
        "\t\tx = DeeperGoogLeNet.inception_module(x, 160, 112, 224, 24, 64, 64, chanDim, \"4b\", reg=reg)\n",
        "\t\tx = DeeperGoogLeNet.inception_module(x, 128, 128, 256, 24, 64, 64, chanDim, \"4c\", reg=reg)\n",
        "\t\tx = DeeperGoogLeNet.inception_module(x, 112, 144, 288, 32, 64, 64, chanDim, \"4d\", reg=reg)\n",
        "\t\tx = DeeperGoogLeNet.inception_module(x, 256, 160, 320, 32, 128, 128, chanDim, \"4e\", reg=reg)\n",
        "\t\tx = MaxPooling2D((3, 3), strides=(2, 2), padding=\"same\", name=\"pool4\")(x)\n",
        "\n",
        "\t\t# apply a POOL layer (average) followed by dropout\n",
        "\t\tx = AveragePooling2D((4, 4), name=\"pool5\")(x)\n",
        "\t\tx = Dropout(0.4, name=\"do\")(x)\n",
        "\n",
        "\t\t# softmax classifier\n",
        "\t\tx = Flatten(name=\"flatten\")(x)\n",
        "\t\tx = Dense(classes, kernel_regularizer=l2(reg), name=\"labels\")(x)\n",
        "\t\tx = Activation(\"softmax\", name=\"softmax\")(x)\n",
        "\n",
        "\t\t# create the model\n",
        "\t\tmodel = Model(inputs, x, name=\"googlenet\")\n",
        "\n",
        "\t\t# return the constructed network architecture\n",
        "\t\treturn model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hZJjPW5EUkfR"
      },
      "source": [
        "model = DeeperGoogLeNet.build(64,64,3,200)\n",
        "model.summary()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3xOksB6EGKfg"
      },
      "source": [
        "### 3.5.1 Creating the training pre-processing"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1alnTRfBGN26"
      },
      "source": [
        "#### 3.5.1.1 Image Preprocessors"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fgmBX1ST1F8U"
      },
      "source": [
        "# import the necessary packages\n",
        "from tensorflow.keras.preprocessing.image import img_to_array\n",
        "\n",
        "class ImageToArrayPreprocessor:\n",
        "\tdef __init__(self, dataFormat=None):\n",
        "\t\t# store the image data format\n",
        "\t\tself.dataFormat = dataFormat\n",
        "\n",
        "\tdef preprocess(self, image):\n",
        "\t\t# apply the Keras utility function that correctly rearranges\n",
        "\t\t# the dimensions of the image\n",
        "\t\treturn img_to_array(image, data_format=self.dataFormat)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2XgdXcA7aPfT"
      },
      "source": [
        "#### 3.5.1.2 Mean preprocessor"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mfbg-dCsUEuW"
      },
      "source": [
        "# import the necessary packages\n",
        "import cv2\n",
        "\n",
        "class MeanPreprocessor:\n",
        "\tdef __init__(self, rMean, gMean, bMean):\n",
        "\t\t# store the Red, Green, and Blue channel averages across a\n",
        "\t\t# training set\n",
        "\t\tself.rMean = rMean\n",
        "\t\tself.gMean = gMean\n",
        "\t\tself.bMean = bMean\n",
        "\n",
        "\tdef preprocess(self, image):\n",
        "\t\t# split the image into its respective Red, Green, and Blue\n",
        "\t\t# channels\n",
        "\t\t(B, G, R) = cv2.split(image.astype(\"float32\"))\n",
        "\n",
        "\t\t# subtract the means for each channel\n",
        "\t\tR -= self.rMean\n",
        "\t\tG -= self.gMean\n",
        "\t\tB -= self.bMean\n",
        "\n",
        "    # Keep in mind that OpenCV represents images in BGR order\n",
        "\t\t# merge the channels back together and return the image\n",
        "\t\treturn cv2.merge([B, G, R])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2NF8faSog8so"
      },
      "source": [
        "#### 3.5.1.3 HDF5 dataset generators"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uMo22QKUg7me"
      },
      "source": [
        "# import the necessary packages\n",
        "from tensorflow.keras.utils import to_categorical\n",
        "import numpy as np\n",
        "import h5py\n",
        "\n",
        "class HDF5DatasetGenerator:\n",
        "\tdef __init__(self, dbPath, batchSize, preprocessors=None, aug=None, binarize=True, classes=2):\n",
        "\t\t# store the batch size, preprocessors, and data augmentor,\n",
        "\t\t# whether or not the labels should be binarized, along with\n",
        "\t\t# the total number of classes\n",
        "\t\tself.batchSize = batchSize\n",
        "\t\tself.preprocessors = preprocessors\n",
        "\t\tself.aug = aug\n",
        "\t\tself.binarize = binarize\n",
        "\t\tself.classes = classes\n",
        "\n",
        "\t\t# open the HDF5 database for reading and determine the total\n",
        "\t\t# number of entries in the database\n",
        "\t\tself.db = h5py.File(dbPath, \"r\")\n",
        "\t\tself.numImages = self.db[\"labels\"].shape[0]\n",
        "\n",
        "\tdef generator(self, passes=np.inf):\n",
        "\t\t# initialize the epoch count\n",
        "\t\tepochs = 0\n",
        "\n",
        "\t\t# keep looping infinitely -- the model will stop once we have\n",
        "\t\t# reach the desired number of epochs\n",
        "\t\twhile epochs < passes:\n",
        "\t\t\t# loop over the HDF5 dataset\n",
        "\t\t\tfor i in np.arange(0, self.numImages, self.batchSize):\n",
        "\t\t\t\t# extract the images and labels from the HDF dataset\n",
        "\t\t\t\timages = self.db[\"images\"][i: i + self.batchSize]\n",
        "\t\t\t\tlabels = self.db[\"labels\"][i: i + self.batchSize]\n",
        "\n",
        "\t\t\t\t# check to see if the labels should be binarized\n",
        "\t\t\t\tif self.binarize:\n",
        "\t\t\t\t\tlabels = to_categorical(labels,\n",
        "\t\t\t\t\t\tself.classes)\n",
        "\n",
        "\t\t\t\t# check to see if our preprocessors are not None\n",
        "\t\t\t\tif self.preprocessors is not None:\n",
        "\t\t\t\t\t# initialize the list of processed images\n",
        "\t\t\t\t\tprocImages = []\n",
        "\n",
        "\t\t\t\t\t# loop over the images\n",
        "\t\t\t\t\tfor image in images:\n",
        "\t\t\t\t\t\t# loop over the preprocessors and apply each\n",
        "\t\t\t\t\t\t# to the image\n",
        "\t\t\t\t\t\tfor p in self.preprocessors:\n",
        "\t\t\t\t\t\t\timage = p.preprocess(image)\n",
        "\n",
        "\t\t\t\t\t\t# update the list of processed images\n",
        "\t\t\t\t\t\tprocImages.append(image)\n",
        "\n",
        "\t\t\t\t\t# update the images array to be the processed\n",
        "\t\t\t\t\t# images\n",
        "\t\t\t\t\timages = np.array(procImages)\n",
        "\n",
        "\t\t\t\t# if the data augmenator exists, apply it\n",
        "\t\t\t\tif self.aug is not None:\n",
        "\t\t\t\t\t(images, labels) = next(self.aug.flow(images,\n",
        "\t\t\t\t\t\tlabels, batch_size=self.batchSize))\n",
        "\n",
        "\t\t\t\t# yield a tuple of images and labels\n",
        "\t\t\t\tyield (images, labels)\n",
        "\n",
        "\t\t\t# increment the total number of epochs\n",
        "\t\t\tepochs += 1\n",
        "\n",
        "\tdef close(self):\n",
        "\t\t# close the database\n",
        "\t\tself.db.close()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5Ou-_XnZqjrY"
      },
      "source": [
        "#### 3.5.1.4 Simple preprocessor"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Juf1YRHzqmeU"
      },
      "source": [
        "# import the necessary packages\n",
        "import cv2\n",
        "\n",
        "class SimplePreprocessor:\n",
        "\tdef __init__(self, width, height, inter=cv2.INTER_AREA):\n",
        "\t\t# store the target image width, height, and interpolation\n",
        "\t\t# method used when resizing\n",
        "\t\tself.width = width\n",
        "\t\tself.height = height\n",
        "\t\tself.inter = inter\n",
        "\n",
        "\tdef preprocess(self, image):\n",
        "\t\t# resize the image to a fixed size, ignoring the aspect\n",
        "\t\t# ratio\n",
        "\t\treturn cv2.resize(image, (self.width, self.height),\n",
        "\t\t\tinterpolation=self.inter)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1bJUx9Prq2xE"
      },
      "source": [
        "#### 3.5.1.5 Training monitor"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "m1JGywYeq5Wj"
      },
      "source": [
        "# import the necessary packages\n",
        "from tensorflow.keras.callbacks import BaseLogger\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "import json\n",
        "import os\n",
        "\n",
        "class TrainingMonitor(BaseLogger):\n",
        "\tdef __init__(self, figPath, jsonPath=None, startAt=0):\n",
        "\t\t# store the output path for the figure, the path to the JSON\n",
        "\t\t# serialized file, and the starting epoch\n",
        "\t\tsuper(TrainingMonitor, self).__init__()\n",
        "\t\tself.figPath = figPath\n",
        "\t\tself.jsonPath = jsonPath\n",
        "\t\tself.startAt = startAt\n",
        "\n",
        "\tdef on_train_begin(self, logs={}):\n",
        "\t\t# initialize the history dictionary\n",
        "\t\tself.H = {}\n",
        "\n",
        "\t\t# if the JSON history path exists, load the training history\n",
        "\t\tif self.jsonPath is not None:\n",
        "\t\t\tif os.path.exists(self.jsonPath):\n",
        "\t\t\t\tself.H = json.loads(open(self.jsonPath).read())\n",
        "\n",
        "\t\t\t\t# check to see if a starting epoch was supplied\n",
        "\t\t\t\tif self.startAt > 0:\n",
        "\t\t\t\t\t# loop over the entries in the history log and\n",
        "\t\t\t\t\t# trim any entries that are past the starting\n",
        "\t\t\t\t\t# epoch\n",
        "\t\t\t\t\tfor k in self.H.keys():\n",
        "\t\t\t\t\t\tself.H[k] = self.H[k][:self.startAt]\n",
        "\n",
        "\tdef on_epoch_end(self, epoch, logs={}):\n",
        "\t\t# loop over the logs and update the loss, accuracy, etc.\n",
        "\t\t# for the entire training process\n",
        "\t\tfor (k, v) in logs.items():\n",
        "\t\t\tl = self.H.get(k, [])\n",
        "\t\t\tl.append(float(v))\n",
        "\t\t\tself.H[k] = l\n",
        "\n",
        "\t\t# check to see if the training history should be serialized\n",
        "\t\t# to file\n",
        "\t\tif self.jsonPath is not None:\n",
        "\t\t\tf = open(self.jsonPath, \"w\")\n",
        "\t\t\tf.write(json.dumps(self.H))\n",
        "\t\t\tf.close()\n",
        "\n",
        "\t\t# ensure at least two epochs have passed before plotting\n",
        "\t\t# (epoch starts at zero)\n",
        "\t\tif len(self.H[\"loss\"]) > 1:\n",
        "\t\t\t# plot the training loss and accuracy\n",
        "\t\t\tN = np.arange(0, len(self.H[\"loss\"]))\n",
        "\t\t\tplt.style.use(\"ggplot\")\n",
        "\t\t\tplt.figure()\n",
        "\t\t\tplt.plot(N, self.H[\"loss\"], label=\"train_loss\")\n",
        "\t\t\tplt.plot(N, self.H[\"val_loss\"], label=\"val_loss\")\n",
        "\t\t\tplt.plot(N, self.H[\"accuracy\"], label=\"train_acc\")\n",
        "\t\t\tplt.plot(N, self.H[\"val_accuracy\"], label=\"val_acc\")\n",
        "\t\t\tplt.title(\"Training Loss and Accuracy [Epoch {}]\".format(\n",
        "\t\t\t\tlen(self.H[\"loss\"])))\n",
        "\t\t\tplt.xlabel(\"Epoch #\")\n",
        "\t\t\tplt.ylabel(\"Loss/Accuracy\")\n",
        "\t\t\tplt.legend()\n",
        "\n",
        "\t\t\t# save the figure\n",
        "\t\t\tplt.savefig(self.figPath)\n",
        "\t\t\tplt.close()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3wm5dfeNNWJ-"
      },
      "source": [
        "#### 3.5.1.6 Epoch CheckPoint"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eDrs03uINbB4"
      },
      "source": [
        "# import the necessary packages\n",
        "from tensorflow.keras.callbacks import Callback\n",
        "import os\n",
        "\n",
        "class EpochCheckpoint(Callback):\n",
        "\tdef __init__(self, outputPath, every=5, startAt=0):\n",
        "\t\t# call the parent constructor\n",
        "\t\tsuper(Callback, self).__init__()\n",
        "\n",
        "\t\t# store the base output path for the model, the number of\n",
        "\t\t# epochs that must pass before the model is serialized to\n",
        "\t\t# disk and the current epoch value\n",
        "\t\tself.outputPath = outputPath\n",
        "\t\tself.every = every\n",
        "\t\tself.intEpoch = startAt\n",
        "\n",
        "\tdef on_epoch_end(self, epoch, logs={}):\n",
        "\t\t# check to see if the model should be serialized to disk\n",
        "\t\tif (self.intEpoch + 1) % self.every == 0:\n",
        "\t\t\tp = os.path.sep.join([self.outputPath,\n",
        "\t\t\t\t\"epoch_{}.hdf5\".format(self.intEpoch + 1)])\n",
        "\t\t\tself.model.save(p, overwrite=True)\n",
        "\n",
        "\t\t# increment the internal epoch counter\n",
        "\t\tself.intEpoch += 1"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nk2HBr6JAMQ8"
      },
      "source": [
        "#### 3.5.1.7 Rank accuracy\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AJCbMd1sAPP8"
      },
      "source": [
        "# import the necessary packages\n",
        "import numpy as np\n",
        "\n",
        "def rank5_accuracy(preds, labels):\n",
        "\t# initialize the rank-1 and rank-5 accuracies\n",
        "\trank1 = 0\n",
        "\trank5 = 0\n",
        "\n",
        "\t# loop over the predictions and ground-truth labels\n",
        "\tfor (p, gt) in zip(preds, labels):\n",
        "\t\t# sort the probabilities by their index in descending\n",
        "\t\t# order so that the more confident guesses are at the\n",
        "\t\t# front of the list\n",
        "\t\tp = np.argsort(p)[::-1]\n",
        "\n",
        "\t\t# check if the ground-truth label is in the top-5\n",
        "\t\t# predictions\n",
        "\t\tif gt in p[:5]:\n",
        "\t\t\trank5 += 1\n",
        "\n",
        "\t\t# check to see if the ground-truth is the #1 prediction\n",
        "\t\tif gt == p[0]:\n",
        "\t\t\trank1 += 1\n",
        "\n",
        "\t# compute the final rank-1 and rank-5 accuracies\n",
        "\trank1 /= float(len(preds))\n",
        "\trank5 /= float(len(preds))\n",
        "\n",
        "\t# return a tuple of the rank-1 and rank-5 accuracies\n",
        "\treturn (rank1, rank5)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vrmCQKbMHI-4"
      },
      "source": [
        "#### 3.5.1.8 SimpleDatasetLoader"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wj4RbTs8HH62"
      },
      "source": [
        "# import the necessary packages\n",
        "import numpy as np\n",
        "import cv2\n",
        "import os\n",
        "\n",
        "# helper to load images\n",
        "class SimpleDatasetLoader:\n",
        "\tdef __init__(self, preprocessors=None):\n",
        "\t\t# store the image preprocessor\n",
        "\t\tself.preprocessors = preprocessors\n",
        "\n",
        "\t\t# if the preprocessors are None, initialize them as an\n",
        "\t\t# empty list\n",
        "\t\tif self.preprocessors is None:\n",
        "\t\t\tself.preprocessors = []\n",
        "\n",
        "\tdef load(self, imagePaths, verbose=-1):\n",
        "\t\t# initialize the list of features and labels\n",
        "\t\tdata = []\n",
        "\t\tlabels = []\n",
        "\n",
        "\t\t# loop over the input images\n",
        "\t\tfor (i, imagePath) in enumerate(imagePaths):\n",
        "\t\t\t# load the image and extract the class label assuming\n",
        "\t\t\t# that our path has the following format:\n",
        "\t\t\t# /path/to/dataset/{class}/{image}.jpg\n",
        "\t\t\timage = cv2.imread(imagePath)\n",
        "\t\t\tlabel = imagePath.split(os.path.sep)[-2]\n",
        "\n",
        "\t\t\t# check to see if our preprocessors are not None\n",
        "\t\t\tif self.preprocessors is not None:\n",
        "\t\t\t\t# loop over the preprocessors and apply each to\n",
        "\t\t\t\t# the image\n",
        "\t\t\t\tfor p in self.preprocessors:\n",
        "\t\t\t\t\timage = p.preprocess(image)\n",
        "\n",
        "\t\t\t# treat our processed image as a \"feature vector\"\n",
        "\t\t\t# by updating the data list followed by the labels\n",
        "\t\t\tdata.append(image)\n",
        "\t\t\tlabels.append(label)\n",
        "\n",
        "\t\t\t# show an update every `verbose` images\n",
        "\t\t\tif verbose > 0 and i > 0 and (i + 1) % verbose == 0:\n",
        "\t\t\t\tprint(\"[INFO] processed {}/{}\".format(i + 1,\n",
        "\t\t\t\t\tlen(imagePaths)))\n",
        "\n",
        "\t\t# return a tuple of the data and labels\n",
        "\t\treturn (np.array(data), np.array(labels))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MhDNJXiJM_lx"
      },
      "source": [
        "### 3.5.2 Creating the Training "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3_C-54macaSJ"
      },
      "source": [
        "# import the necessary packages\n",
        "from os import path\n",
        "\n",
        "# define the paths to the training and validation directories\n",
        "TRAIN_IMAGES = \"tiny-imagenet-200/train\"\n",
        "VAL_IMAGES = \"tiny-imagenet-200/val/images\"\n",
        "\n",
        "# define the path to the file that maps validation filenames to\n",
        "# their corresponding class labels\n",
        "VAL_MAPPINGS = \"tiny-imagenet-200/val/val_annotations.txt\"\n",
        "\n",
        "# define the paths to the WordNet hierarchy files which are used\n",
        "# to generate our class labels\n",
        "WORDNET_IDS = \"tiny-imagenet-200/wnids.txt\"\n",
        "WORD_LABELS = \"tiny-imagenet-200/words.txt\"\n",
        "\n",
        "# since we do not have access to the testing data we need to\n",
        "# take a number of images from the training data and use it instead\n",
        "NUM_CLASSES = 200\n",
        "NUM_TEST_IMAGES = 50 * NUM_CLASSES\n",
        "\n",
        "# define the path to the output training, validation, and testing\n",
        "# HDF5 files\n",
        "TRAIN_HDF5 = \"tiny-imagenet-200/hdf5/train.hdf5\"\n",
        "VAL_HDF5 = \"tiny-imagenet-200/hdf5/val.hdf5\"\n",
        "TEST_HDF5 = \"tiny-imagenet-200/hdf5/test.hdf5\"\n",
        "\n",
        "# define the path to the dataset mean\n",
        "DATASET_MEAN = \"tiny-imagenet-200/output/tiny-image-net-200-mean.json\"\n",
        "\n",
        "# define the path to the output directory used for storing plots,\n",
        "# classification reports, etc.\n",
        "OUTPUT_PATH = \"tiny-imagenet-200/output\"\n",
        "MODEL_PATH = path.sep.join([OUTPUT_PATH,\"epoch_40.hdf5\"])\n",
        "FIG_PATH = path.sep.join([OUTPUT_PATH,\"deepergooglenet_tinyimagenet.png\"])\n",
        "JSON_PATH = path.sep.join([OUTPUT_PATH,\"deepergooglenet_tinyimagenet.json\"])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aZXME7qzZ8Y1"
      },
      "source": [
        "# Global variables used in the train\n",
        "\n",
        "RESUME_MODEL = \"tiny-imagenet-200/output/epoch_40.hdf5\"\n",
        "RESUME = False\n",
        "START_EPOCH = 0"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oiRLEqb7NsRo"
      },
      "source": [
        "# set the matplotlib backend so figures can be saved in the background\n",
        "import matplotlib\n",
        "matplotlib.use(\"Agg\")\n",
        "\n",
        "from tensorflow.keras.preprocessing.image import ImageDataGenerator\n",
        "from tensorflow.keras.optimizers import Adam,SGD\n",
        "from tensorflow.keras.models import load_model\n",
        "import tensorflow.keras.backend as K\n",
        "import argparse\n",
        "import json\n",
        "\n",
        "# construct the training image generator for data augmentation\n",
        "aug = ImageDataGenerator(rotation_range=18, zoom_range=0.15, \n",
        "                         width_shift_range=0.2, height_shift_range=0.2, shear_range=0.15,\n",
        "                         horizontal_flip=True, fill_mode=\"nearest\")\n",
        "\n",
        "# load the RGB means for the training set\n",
        "means = json.loads(open(DATASET_MEAN).read())\n",
        "\n",
        "# initialize the image preprocessors\n",
        "sp = SimplePreprocessor(64, 64)\n",
        "mp = MeanPreprocessor(means[\"R\"], means[\"G\"], means[\"B\"])\n",
        "iap = ImageToArrayPreprocessor()\n",
        "\n",
        "# initialize the training and validation dataset generators\n",
        "trainGen = HDF5DatasetGenerator(TRAIN_HDF5, 64, aug=aug,\n",
        "                                preprocessors=[sp, mp, iap], classes=NUM_CLASSES)\n",
        "valGen = HDF5DatasetGenerator(VAL_HDF5, 64,\n",
        "                              preprocessors=[sp, mp, iap], classes=NUM_CLASSES)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GSn_YiX9Zt3n"
      },
      "source": [
        "# if there is no specific model checkpoint supplied, then initialize the network and compile the model\n",
        "if not RESUME:\n",
        "  print(\"[INFO] compiling model...\")\n",
        "  model = DeeperGoogLeNet.build(width=64, height=64, depth=3, classes=NUM_CLASSES, reg=0.0002)\n",
        "  opt = SGD(1e-3,momentum=0.9)\n",
        "  model.compile(loss=\"categorical_crossentropy\", optimizer=opt,metrics=[\"accuracy\"])\n",
        "# otherwise, load the checkpoint from disk\n",
        "else:\n",
        "  print(\"[INFO] loading {}...\".format(RESUME_MODEL))\n",
        "  model = load_model(RESUME_MODEL)\n",
        "\n",
        "  # update the learning rate\n",
        "  print(\"[INFO] old learning rate: {}\".format(K.get_value(model.optimizer.lr)))\n",
        "  K.set_value(model.optimizer.lr, 1e-5)\n",
        "  print(\"[INFO] new learning rate: {}\".format(K.get_value(model.optimizer.lr)))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yopLg5pKayq8"
      },
      "source": [
        "# construct the set of callbacks\n",
        "callbacks = [EpochCheckpoint(\"tiny-imagenet-200/output\",every=5,startAt=START_EPOCH),\n",
        "             TrainingMonitor(FIG_PATH, \n",
        "                             jsonPath=JSON_PATH,\n",
        "                             startAt=START_EPOCH)]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UJozqEEXb_AH"
      },
      "source": [
        "# train the network\n",
        "model.fit(trainGen.generator(),\n",
        "          steps_per_epoch=trainGen.numImages // 64,\n",
        "          validation_data=valGen.generator(),\n",
        "          validation_steps=valGen.numImages // 64,\n",
        "          epochs=40,\n",
        "          max_queue_size=10,\n",
        "          callbacks=callbacks, verbose=1,initial_epoch=START_EPOCH)\n",
        "\n",
        "# close the databases\n",
        "trainGen.close()\n",
        "valGen.close()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gndVLgJ7GgA2"
      },
      "source": [
        "# save a backup of results\n",
        "!cp tiny-imagenet-200/output/deepergooglenet_tinyimagenet.png /content/drive/MyDrive/Atividades/Ensino/Disciplinas/POS-GRADUAÇÃO/Deep\\ Learning/Lessons/Lesson\\ #07/output/models_origin/deepergooglenet_tinyimagenet.png \n",
        "!cp tiny-imagenet-200/output/epoch_40.hdf5 /content/drive/MyDrive/Atividades/Ensino/Disciplinas/POS-GRADUAÇÃO/Deep\\ Learning/Lessons/Lesson\\ #07/output/models_origin/epoch_40.hdf5 "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W3BOhu34cM9D"
      },
      "source": [
        "### 3.5.3 Creating the Evaluation Script"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IoQABFecLZmS"
      },
      "source": [
        "Once we are satisfied with our model performance on the training and validation set, we can move on to evaluating the network on the testing set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4_HAw9WFLpMG"
      },
      "source": [
        "from tensorflow.keras.models import load_model\n",
        "import json\n",
        "\n",
        "# load the RGB means for the training set\n",
        "means = json.loads(open(DATASET_MEAN).read())\n",
        "\n",
        "# initialize the image preprocessors\n",
        "sp = SimplePreprocessor(64, 64)\n",
        "mp = MeanPreprocessor(means[\"R\"], means[\"G\"], means[\"B\"])\n",
        "iap = ImageToArrayPreprocessor()\n",
        "\n",
        "# initialize the testing dataset generator\n",
        "testGen = HDF5DatasetGenerator(TEST_HDF5, 64, \n",
        "                               preprocessors=[sp, mp, iap],\n",
        "                               classes=NUM_CLASSES)\n",
        "\n",
        "# load the pre-trained network\n",
        "print(\"[INFO] loading model...\")\n",
        "model = load_model(MODEL_PATH)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OT2ahH-pMpns"
      },
      "source": [
        "# make predictions on the testing data\n",
        "print(\"[INFO] predicting on test data...\")\n",
        "predictions = model.predict(testGen.generator(),\n",
        "                            steps=testGen.numImages // 64,\n",
        "                            max_queue_size=10)\n",
        "\n",
        "# compute the rank-1 and rank-5 accuracies\n",
        "(rank1, rank5) = rank5_accuracy(predictions, testGen.db[\"labels\"])\n",
        "print(\"[INFO] rank-1: {:.2f}%\".format(rank1 * 100))\n",
        "print(\"[INFO] rank-5: {:.2f}%\".format(rank5 * 100))\n",
        "\n",
        "# close the database\n",
        "testGen.close()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7zBKWM-pNWnW"
      },
      "source": [
        "### 3.5.4 DeeperGoogLeNet Experiments"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XFxJwV8IZOpX"
      },
      "source": [
        "In the following sections we have included the results of three separate experiments we ran when training DeeperGoogLeNet on Tiny ImageNet. After each experiment we evaluated the results and then made an educated decision on how the hyperparameters and network architecture should be updated to increase accuracy.\n",
        "\n",
        "Case studies like these are especially helpful to you as a budding deep learning practitioner. Not only do they demonstrate that deep learning is an iterative process requiring many experiments, but they also show which parameters you should be paying attention to and how to update them.\n",
        "\n",
        "Finally, it’s worth noting that some of these experiments required changes to the code. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Zh5gS9OQoncZ"
      },
      "source": [
        "#### 3.5.4.1 Experiment 01"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jZc8kLWSoraC"
      },
      "source": [
        "Given that this was my first time training a network on the Tiny ImageNet challenge, we wasn’t sure what the optimal depth should be for a given architecture on this dataset. While we knew Tiny ImageNet would be a challenging classification task, <font color=\"red\">we didn’t think Inception modules 4a-4e were required</font>, so we **removed** them from our DeeperGoogLeNet implementation above, leading to a substantially more shallow network architecture.\n",
        "\n",
        "1. We decided to train DeeperGoogLenet using SGD with an initial learning rate of $1e-2$ and momentum term of $0.9$ (no Nesterov acceleration was applied).\n",
        "2. [<font color=\"red\">best practice</font>] you should first try SGD to obtain a baseline, and then if need be, use more advanced optimization methods.\n",
        "3. The learning rate schedule detailed in Table below can then be used. This table implies that after epoch 25 we stopped training, lowered the learning rate to $1e-3$, then resumed training for another 10 epochs.\n",
        "4. After epoch 35 we can again stopped training, lowered the learning rate to $1e-4$, and then resumed training for thirty more epochs. Training for an extra thirty epochs can be excessive, to say the least; however, we wanted to get a feel for the level of overfitting to expect for a large number of epochs after the original learning rate had been dropped (as this was the first time we had worked with GoogLeNet + Tiny ImageNet).\n",
        "\n",
        "| Epoch   | Learning Rate |\n",
        "|---------|---------------|\n",
        "| 1 - 25  | $1e-02$      |\n",
        "| 26 - 35 | $1e-03$      |\n",
        "| 36 - 65 | $1e-04$      |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-i0kQYKQo_nT"
      },
      "source": [
        "#### 3.5.4.2 Experiment 02"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LxyDwwfysXhm"
      },
      "source": [
        "In our second experiment with DeeperGoogLeNet + Tiny ImageNet, I decided to switch out the SGD optimizer for Adam. This decision was made strictly because I wasn’t convinced that the network architecture needed to be deeper (yet). The Adam optimizer was used with the default initial learning rate of $1e-3$. I then used the learning rate schedule in Table below.\n",
        "\n",
        "| Epoch   | Learning Rate |\n",
        "|---------|---------------|\n",
        "| 1 - 20  | 1e-3          |\n",
        "| 21 - 30 | 1e-4          |\n",
        "| 31 - 40 | 1e-5          |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SDBxgr3et3Zm"
      },
      "source": [
        "#### 3.5.4.3 Experiment 03"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QHQQtbaVuZZw"
      },
      "source": [
        "Supposing a learning stagnation, we can postulate that the network was not deep enough to model the underlying patterns in the Tiny ImageNet dataset. Therefore, I decided to enable the Inception modules 4a-4e again, creating a much deeper network architecture capable of learning deeper, more discriminative features. The Adam optimizer with an initial learning rate of $1e-3$ was used to train the network. I left the L2 weight decay term at $0.0002$. DeeperGoogLeNet was then trained according to Table below.\n",
        "\n",
        "| Epoch   | Learning Rate |\n",
        "|---------|---------------|\n",
        "| 1 - 40  | 1e-3          |\n",
        "| 41 - 60 | 1e-4          |\n",
        "| 61 - 70 | 1e-5          |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FjWhf_6cutUg"
      },
      "source": [
        "#### 3.5.4.4 Tips"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IVTl_kATvUeU"
      },
      "source": [
        "For readers interested in trying to boost the accuracy of DeeperGoogLeNet further, I would suggest the following experiments:\n",
        "\n",
        "1. Change the **conv_module** to use CONV => RELU => BN instead of the original CONV => BN => RELU ordering.\n",
        "2. Attempt using **ELUs** instead of **ReLUs**."
      ]
    }
  ]
}