finetuning_torchvision_models.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "finetuning_torchvision_models_tutorial.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "metadata": {
        "id": "dh8HAMp4bffx",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "\n",
        "Finetuning Torchvision Models\n",
        "=============================\n",
        "\n",
        "**Author:** `Nathan Inkawhich <https://github.com/inkawhich>`__\n",
        "\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "JfYCZqACbffx",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "In this tutorial we will take a deeper look at how to finetune and\n",
        "feature extract the `torchvision\n",
        "models <https://pytorch.org/docs/stable/torchvision/models.html>`__, all\n",
        "of which have been pretrained on the 1000-class Imagenet dataset. This\n",
        "tutorial will give an indepth look at how to work with several modern\n",
        "CNN architectures, and will build an intuition for finetuning any\n",
        "PyTorch model. Since each model architecture is different, there is no\n",
        "boilerplate finetuning code that will work in all scenarios. Rather, the\n",
        "researcher must look at the existing architecture and make custom\n",
        "adjustments for each model.\n",
        "\n",
        "In this document we will perform two types of transfer learning:\n",
        "finetuning and feature extraction. In **finetuning**, we start with a\n",
        "pretrained model and update *all* of the model’s parameters for our new\n",
        "task, in essence retraining the whole model. In **feature extraction**,\n",
        "we start with a pretrained model and only update the final layer weights\n",
        "from which we derive predictions. It is called feature extraction\n",
        "because we use the pretrained CNN as a fixed feature-extractor, and only\n",
        "change the output layer. For more technical information about transfer\n",
        "learning see `here <https://cs231n.github.io/transfer-learning/>`__ and\n",
        "`here <https://ruder.io/transfer-learning/>`__.\n",
        "\n",
        "In general both transfer learning methods follow the same few steps:\n",
        "\n",
        "-  Initialize the pretrained model\n",
        "-  Reshape the final layer(s) to have the same number of outputs as the\n",
        "   number of classes in the new dataset\n",
        "-  Define for the optimization algorithm which parameters we want to\n",
        "   update during training\n",
        "-  Run the training step\n",
        "\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "ZLj5xQz1bffr",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "%matplotlib inline"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "cp6DM890cRp6",
        "colab_type": "code",
        "outputId": "3ce5ab00-bea7-4603-b3cb-e57c2f4c362f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 55
        }
      },
      "cell_type": "code",
      "source": [
        "!pip3 install https://download.pytorch.org/whl/cpu/torch-1.0.0-cp36-cp36m-linux_x86_64.whl"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: torch==1.0.0 from https://download.pytorch.org/whl/cpu/torch-1.0.0-cp36-cp36m-linux_x86_64.whl in /usr/local/lib/python3.6/dist-packages (1.0.0)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "99UxaCpCdeXR",
        "colab_type": "code",
        "outputId": "18b55e11-1e4c-4369-d308-9504538837df",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 335
        }
      },
      "cell_type": "code",
      "source": [
        "!pip3 install torchvision"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: torchvision in /usr/local/lib/python3.6/dist-packages (0.2.1)\n",
            "Collecting pillow>=4.1.1 (from torchvision)\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/85/5e/e91792f198bbc5a0d7d3055ad552bc4062942d27eaf75c3e2783cf64eae5/Pillow-5.4.1-cp36-cp36m-manylinux1_x86_64.whl (2.0MB)\n",
            "\u001b[K    100% |████████████████████████████████| 2.0MB 12.9MB/s \n",
            "\u001b[?25hRequirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.11.0)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.14.6)\n",
            "Requirement already satisfied: torch in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.0.0)\n",
            "Installing collected packages: pillow\n",
            "  Found existing installation: Pillow 4.0.0\n",
            "    Uninstalling Pillow-4.0.0:\n",
            "      Successfully uninstalled Pillow-4.0.0\n",
            "Successfully installed pillow-5.4.1\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.colab-display-data+json": {
              "pip_warning": {
                "packages": [
                  "PIL"
                ]
              }
            }
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "Wvab0O7geOrR",
        "colab_type": "code",
        "outputId": "b2dbe58e-fab0-4691-e215-f363a23f1b03",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 235
        }
      },
      "cell_type": "code",
      "source": [
        "!pip install kaggle"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: kaggle in /usr/local/lib/python3.6/dist-packages (1.5.1.1)\n",
            "Requirement already satisfied: urllib3<1.23.0,>=1.15 in /usr/local/lib/python3.6/dist-packages (from kaggle) (1.22)\n",
            "Requirement already satisfied: six>=1.10 in /usr/local/lib/python3.6/dist-packages (from kaggle) (1.11.0)\n",
            "Requirement already satisfied: certifi in /usr/local/lib/python3.6/dist-packages (from kaggle) (2018.11.29)\n",
            "Requirement already satisfied: python-dateutil in /usr/local/lib/python3.6/dist-packages (from kaggle) (2.5.3)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from kaggle) (2.18.4)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from kaggle) (4.28.1)\n",
            "Requirement already satisfied: python-slugify in /usr/local/lib/python3.6/dist-packages (from kaggle) (2.0.1)\n",
            "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->kaggle) (3.0.4)\n",
            "Requirement already satisfied: idna<2.7,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->kaggle) (2.6)\n",
            "Requirement already satisfied: Unidecode>=0.04.16 in /usr/local/lib/python3.6/dist-packages (from python-slugify->kaggle) (1.0.23)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "c7bPRrtDedUV",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "024f3ede-341e-4122-cc07-da469b938e98"
      },
      "cell_type": "code",
      "source": [
        "!mkdir .kaggle"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "mkdir: cannot create directory ‘.kaggle’: File exists\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "9DBhwEC7eif0",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!cp kaggle.json ~/.kaggle/"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "qlEH7CnGe29j",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!chmod 600 ~/.kaggle/kaggle.json"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "oOA3eoVJeoN1",
        "colab_type": "code",
        "outputId": "5fcf96da-54dc-413b-92e6-ec4ad3c2b709",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 125
        }
      },
      "cell_type": "code",
      "source": [
        "!kaggle competitions download -c oxford-102-flower-pytorch"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading sample_submission.csv to /content\n",
            "\r  0% 0.00/15.2k [00:00<?, ?B/s]\n",
            "100% 15.2k/15.2k [00:00<00:00, 12.8MB/s]\n",
            "Downloading flower_data.zip to /content\n",
            "100% 330M/330M [00:03<00:00, 97.2MB/s]\n",
            "100% 330M/330M [00:03<00:00, 110MB/s] \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "D0W1mQCfgxzN",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!unzip flower_data.zip"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "xrDolA3xbffz",
        "colab_type": "code",
        "outputId": "eaa0c20a-9ebe-468a-c022-270548abc517",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        }
      },
      "cell_type": "code",
      "source": [
        "from __future__ import print_function \n",
        "from __future__ import division\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.optim as optim\n",
        "from torch.optim import lr_scheduler\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import torchvision\n",
        "from torchvision import datasets, models, transforms\n",
        "from torch.utils import data\n",
        "import matplotlib.pyplot as plt\n",
        "import time\n",
        "import os\n",
        "import copy\n",
        "import json\n",
        "\n",
        "from PIL import Image\n",
        "\n",
        "print(\"PyTorch Version: \",torch.__version__)\n",
        "print(\"Torchvision Version: \",torchvision.__version__)"
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "PyTorch Version:  1.0.0\n",
            "Torchvision Version:  0.2.1\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "NmdxAz2Kbff1",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Inputs\n",
        "------\n",
        "\n",
        "Here are all of the parameters to change for the run. We will use the\n",
        "*hymenoptera_data* dataset which can be downloaded\n",
        "`here <https://download.pytorch.org/tutorial/hymenoptera_data.zip>`__.\n",
        "This dataset contains two classes, **bees** and **ants**, and is\n",
        "structured such that we can use the\n",
        "`ImageFolder <https://pytorch.org/docs/stable/torchvision/datasets.html#torchvision.datasets.ImageFolder>`__\n",
        "dataset, rather than writing our own custom dataset. Download the data\n",
        "and set the ``data_dir`` input to the root directory of the dataset. The\n",
        "``model_name`` input is the name of the model you wish to use and must\n",
        "be selected from this list:\n",
        "\n",
        "::\n",
        "\n",
        "   [resnet, alexnet, vgg, squeezenet, densenet, inception]\n",
        "\n",
        "The other inputs are as follows: ``num_classes`` is the number of\n",
        "classes in the dataset, ``batch_size`` is the batch size used for\n",
        "training and may be adjusted according to the capability of your\n",
        "machine, ``num_epochs`` is the number of training epochs we want to run,\n",
        "and ``feature_extract`` is a boolean that defines if we are finetuning\n",
        "or feature extracting. If ``feature_extract = False``, the model is\n",
        "finetuned and all model parameters are updated. If\n",
        "``feature_extract = True``, only the last layer parameters are updated,\n",
        "the others remain fixed.\n",
        "\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "tpedhpsWbff2",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "cellView": "both",
        "outputId": "35103d1e-fdce-4cea-e203-b8e21269662b"
      },
      "cell_type": "code",
      "source": [
        "# Top level data directory. Here we assume the format of the directory conforms \n",
        "#   to the ImageFolder structure\n",
        "#data_dir = \"./data/hymenoptera_data\"\n",
        "\n",
        "data_dir = 'flower_data'\n",
        "\n",
        "# Models to choose from [resnet, alexnet, vgg, squeezenet, densenet, inception]\n",
        "model_name = \"inception\" #@param [\"resnet\", \"alexnet\", \"vgg\",\"squeezenet\",'densenet','inception']\n",
        "\n",
        "# Number of classes in the dataset\n",
        "with open('cat_to_name.json', 'r') as f:\n",
        "    cat_to_name = json.load(f)\n",
        "print(len(cat_to_name.keys())) # 102\n",
        "num_classes = len(cat_to_name.keys())\n",
        "\n",
        "# Batch size for training (change depending on how much memory you have)\n",
        "batch_size = 8 #@param {type:\"integer\"}\n",
        "\n",
        "# Number of epochs to train for \n",
        "num_epochs = 5 #@param {type:\"integer\"}\n",
        "\n",
        "# Flag for feature extracting. When False, we finetune the whole model, \n",
        "#   when True we only update the reshaped layer params\n",
        "feature_extract = True #@param {type:\"boolean\"}"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "102\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "Hg6HBmE8bff5",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Helper Functions\n",
        "----------------\n",
        "\n",
        "Before we write the code for adjusting the models, lets define a few\n",
        "helper functions.\n",
        "\n",
        "Model Training and Validation Code\n",
        "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n",
        "\n",
        "The ``train_model`` function handles the training and validation of a\n",
        "given model. As input, it takes a PyTorch model, a dictionary of\n",
        "dataloaders, a loss function, an optimizer, a specified number of epochs\n",
        "to train and validate for, and a boolean flag for when the model is an\n",
        "Inception model. The *is_inception* flag is used to accomodate the\n",
        "*Inception v3* model, as that architecture uses an auxiliary output and\n",
        "the overall model loss respects both the auxiliary output and the final\n",
        "output, as described\n",
        "`here <https://discuss.pytorch.org/t/how-to-optimize-inception-model-with-auxiliary-classifiers/7958>`__.\n",
        "The function trains for the specified number of epochs and after each\n",
        "epoch runs a full validation step. It also keeps track of the best\n",
        "performing model (in terms of validation accuracy), and at the end of\n",
        "training returns the best performing model. After each epoch, the\n",
        "training and validation accuracies are printed.\n",
        "\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "Pe9tOTwcbff6",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def train_model(model, dataloaders, criterion, optimizer, scheduler, num_epochs=25, is_inception=False):\n",
        "    since = time.time()\n",
        "\n",
        "    val_acc_history = []\n",
        "    \n",
        "    best_model_wts = copy.deepcopy(model.state_dict())\n",
        "    best_acc = 0.0\n",
        "\n",
        "    for epoch in range(num_epochs):\n",
        "        print('Epoch {}/{}'.format(epoch, num_epochs - 1))\n",
        "        print('-' * 10)\n",
        "\n",
        "        # Each epoch has a training and validation phase\n",
        "        for phase in ['train', 'valid']:\n",
        "            if phase == 'train':\n",
        "                scheduler.step()\n",
        "                model.train()  # Set model to training mode\n",
        "            else:\n",
        "                model.eval()   # Set model to evaluate mode\n",
        "\n",
        "            running_loss = 0.0\n",
        "            running_corrects = 0\n",
        "\n",
        "            # Iterate over data.\n",
        "            for inputs, labels in dataloaders[phase]:\n",
        "                inputs = inputs.to(device)\n",
        "                labels = labels.to(device)\n",
        "\n",
        "                # zero the parameter gradients\n",
        "                optimizer.zero_grad()\n",
        "\n",
        "                # forward\n",
        "                # track history if only in train\n",
        "                with torch.set_grad_enabled(phase == 'train'):\n",
        "                    # Get model outputs and calculate loss\n",
        "                    # Special case for inception because in training it has an auxiliary output. In train\n",
        "                    #   mode we calculate the loss by summing the final output and the auxiliary output\n",
        "                    #   but in testing we only consider the final output.\n",
        "                    if is_inception and phase == 'train':\n",
        "                        # From https://discuss.pytorch.org/t/how-to-optimize-inception-model-with-auxiliary-classifiers/7958\n",
        "                        outputs, aux_outputs = model(inputs)\n",
        "                        loss1 = criterion(outputs, labels)\n",
        "                        loss2 = criterion(aux_outputs, labels)\n",
        "                        loss = loss1 + 0.4*loss2\n",
        "                    else:\n",
        "                        outputs = model(inputs)\n",
        "                        loss = criterion(outputs, labels)\n",
        "\n",
        "                    _, preds = torch.max(outputs, 1)\n",
        "\n",
        "                    # backward + optimize only if in training phase\n",
        "                    if phase == 'train':\n",
        "                        loss.backward()\n",
        "                        optimizer.step()\n",
        "\n",
        "                # statistics\n",
        "                running_loss += loss.item() * inputs.size(0)\n",
        "                running_corrects += torch.sum(preds == labels.data)\n",
        "\n",
        "            epoch_loss = running_loss / len(dataloaders[phase].dataset)\n",
        "            epoch_acc = running_corrects.double() / len(dataloaders[phase].dataset)\n",
        "\n",
        "            print('{} Loss: {:.4f} Acc: {:.4f}'.format(phase, epoch_loss, epoch_acc))\n",
        "\n",
        "            # deep copy the model\n",
        "            if phase == 'valid' and epoch_acc > best_acc:\n",
        "                best_acc = epoch_acc\n",
        "                best_model_wts = copy.deepcopy(model.state_dict())\n",
        "            if phase == 'valid':\n",
        "                val_acc_history.append(epoch_acc)\n",
        "\n",
        "        print()\n",
        "\n",
        "    time_elapsed = time.time() - since\n",
        "    print('Training complete in {:.0f}m {:.0f}s'.format(time_elapsed // 60, time_elapsed % 60))\n",
        "    print('Best val Acc: {:4f}'.format(best_acc))\n",
        "\n",
        "    # load best model weights\n",
        "    model.load_state_dict(best_model_wts)\n",
        "    return model, val_acc_history"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "XQzbUIk2bff-",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Set Model Parameters’ .requires_grad attribute\n",
        "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n",
        "\n",
        "This helper function sets the ``.requires_grad`` attribute of the\n",
        "parameters in the model to False when we are feature extracting. By\n",
        "default, when we load a pretrained model all of the parameters have\n",
        "``.requires_grad=True``, which is fine if we are training from scratch\n",
        "or finetuning. However, if we are feature extracting and only want to\n",
        "compute gradients for the newly initialized layer then we want all of\n",
        "the other parameters to not require gradients. This will make more sense\n",
        "later.\n",
        "\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "4EqZriPzbff-",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def set_parameter_requires_grad(model, feature_extracting):\n",
        "    if feature_extracting:\n",
        "        for param in model.parameters():\n",
        "            param.requires_grad = False"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "H7T6L3StbfgA",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Initialize and Reshape the Networks\n",
        "-----------------------------------\n",
        "\n",
        "Now to the most interesting part. Here is where we handle the reshaping\n",
        "of each network. Note, this is not an automatic procedure and is unique\n",
        "to each model. Recall, the final layer of a CNN model, which is often\n",
        "times an FC layer, has the same number of nodes as the number of output\n",
        "classes in the dataset. Since all of the models have been pretrained on\n",
        "Imagenet, they all have output layers of size 1000, one node for each\n",
        "class. The goal here is to reshape the last layer to have the same\n",
        "number of inputs as before, AND to have the same number of outputs as\n",
        "the number of classes in the dataset. In the following sections we will\n",
        "discuss how to alter the architecture of each model individually. But\n",
        "first, there is one important detail regarding the difference between\n",
        "finetuning and feature-extraction.\n",
        "\n",
        "When feature extracting, we only want to update the parameters of the\n",
        "last layer, or in other words, we only want to update the parameters for\n",
        "the layer(s) we are reshaping. Therefore, we do not need to compute the\n",
        "gradients of the parameters that we are not changing, so for efficiency\n",
        "we set the .requires_grad attribute to False. This is important because\n",
        "by default, this attribute is set to True. Then, when we initialize the\n",
        "new layer and by default the new parameters have ``.requires_grad=True``\n",
        "so only the new layer’s parameters will be updated. When we are\n",
        "finetuning we can leave all of the .required_grad’s set to the default\n",
        "of True.\n",
        "\n",
        "Finally, notice that inception_v3 requires the input size to be\n",
        "(299,299), whereas all of the other models expect (224,224).\n",
        "\n",
        "Resnet\n",
        "~~~~~~\n",
        "\n",
        "Resnet was introduced in the paper `Deep Residual Learning for Image\n",
        "Recognition <https://arxiv.org/abs/1512.03385>`__. There are several\n",
        "variants of different sizes, including Resnet18, Resnet34, Resnet50,\n",
        "Resnet101, and Resnet152, all of which are available from torchvision\n",
        "models. Here we use Resnet18, as our dataset is small and only has two\n",
        "classes. When we print the model, we see that the last layer is a fully\n",
        "connected layer as shown below:\n",
        "\n",
        "::\n",
        "\n",
        "   (fc): Linear(in_features=512, out_features=1000, bias=True) \n",
        "\n",
        "Thus, we must reinitialize ``model.fc`` to be a Linear layer with 512\n",
        "input features and 2 output features with:\n",
        "\n",
        "::\n",
        "\n",
        "   model.fc = nn.Linear(512, num_classes)\n",
        "\n",
        "Alexnet\n",
        "~~~~~~~\n",
        "\n",
        "Alexnet was introduced in the paper `ImageNet Classification with Deep\n",
        "Convolutional Neural\n",
        "Networks <https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf>`__\n",
        "and was the first very successful CNN on the ImageNet dataset. When we\n",
        "print the model architecture, we see the model output comes from the 6th\n",
        "layer of the classifier\n",
        "\n",
        "::\n",
        "\n",
        "   (classifier): Sequential(\n",
        "       ...\n",
        "       (6): Linear(in_features=4096, out_features=1000, bias=True)\n",
        "    ) \n",
        "\n",
        "To use the model with our dataset we reinitialize this layer as\n",
        "\n",
        "::\n",
        "\n",
        "   model.classifier[6] = nn.Linear(4096,num_classes)\n",
        "\n",
        "VGG\n",
        "~~~\n",
        "\n",
        "VGG was introduced in the paper `Very Deep Convolutional Networks for\n",
        "Large-Scale Image Recognition <https://arxiv.org/pdf/1409.1556.pdf>`__.\n",
        "Torchvision offers eight versions of VGG with various lengths and some\n",
        "that have batch normalizations layers. Here we use VGG-11 with batch\n",
        "normalization. The output layer is similar to Alexnet, i.e.\n",
        "\n",
        "::\n",
        "\n",
        "   (classifier): Sequential(\n",
        "       ...\n",
        "       (6): Linear(in_features=4096, out_features=1000, bias=True)\n",
        "    )\n",
        "\n",
        "Therefore, we use the same technique to modify the output layer\n",
        "\n",
        "::\n",
        "\n",
        "   model.classifier[6] = nn.Linear(4096,num_classes)\n",
        "\n",
        "Squeezenet\n",
        "~~~~~~~~~~\n",
        "\n",
        "The Squeeznet architecture is described in the paper `SqueezeNet:\n",
        "AlexNet-level accuracy with 50x fewer parameters and <0.5MB model\n",
        "size <https://arxiv.org/abs/1602.07360>`__ and uses a different output\n",
        "structure than any of the other models shown here. Torchvision has two\n",
        "versions of Squeezenet, we use version 1.0. The output comes from a 1x1\n",
        "convolutional layer which is the 1st layer of the classifier:\n",
        "\n",
        "::\n",
        "\n",
        "   (classifier): Sequential(\n",
        "       (0): Dropout(p=0.5)\n",
        "       (1): Conv2d(512, 1000, kernel_size=(1, 1), stride=(1, 1))\n",
        "       (2): ReLU(inplace)\n",
        "       (3): AvgPool2d(kernel_size=13, stride=1, padding=0)\n",
        "    ) \n",
        "\n",
        "To modify the network, we reinitialize the Conv2d layer to have an\n",
        "output feature map of depth 2 as\n",
        "\n",
        "::\n",
        "\n",
        "   model.classifier[1] = nn.Conv2d(512, num_classes, kernel_size=(1,1), stride=(1,1))\n",
        "\n",
        "Densenet\n",
        "~~~~~~~~\n",
        "\n",
        "Densenet was introduced in the paper `Densely Connected Convolutional\n",
        "Networks <https://arxiv.org/abs/1608.06993>`__. Torchvision has four\n",
        "variants of Densenet but here we only use Densenet-121. The output layer\n",
        "is a linear layer with 1024 input features:\n",
        "\n",
        "::\n",
        "\n",
        "   (classifier): Linear(in_features=1024, out_features=1000, bias=True) \n",
        "\n",
        "To reshape the network, we reinitialize the classifier’s linear layer as\n",
        "\n",
        "::\n",
        "\n",
        "   model.classifier = nn.Linear(1024, num_classes)\n",
        "\n",
        "Inception v3\n",
        "~~~~~~~~~~~~\n",
        "\n",
        "Finally, Inception v3 was first described in `Rethinking the Inception\n",
        "Architecture for Computer\n",
        "Vision <https://arxiv.org/pdf/1512.00567v1.pdf>`__. This network is\n",
        "unique because it has two output layers when training. The second output\n",
        "is known as an auxiliary output and is contained in the AuxLogits part\n",
        "of the network. The primary output is a linear layer at the end of the\n",
        "network. Note, when testing we only consider the primary output. The\n",
        "auxiliary output and primary output of the loaded model are printed as:\n",
        "\n",
        "::\n",
        "\n",
        "   (AuxLogits): InceptionAux(\n",
        "       ...\n",
        "       (fc): Linear(in_features=768, out_features=1000, bias=True)\n",
        "    )\n",
        "    ...\n",
        "   (fc): Linear(in_features=2048, out_features=1000, bias=True)\n",
        "\n",
        "To finetune this model we must reshape both layers. This is accomplished\n",
        "with the following\n",
        "\n",
        "::\n",
        "\n",
        "   model.AuxLogits.fc = nn.Linear(768, num_classes)\n",
        "   model.fc = nn.Linear(2048, num_classes)\n",
        "\n",
        "Notice, many of the models have similar output structures, but each must\n",
        "be handled slightly differently. Also, check out the printed model\n",
        "architecture of the reshaped network and make sure the number of output\n",
        "features is the same as the number of classes in the dataset.\n",
        "\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "1XlQWf1CbfgB",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 7433
        },
        "outputId": "247f05a8-b32c-46a8-c40f-83622739cecb"
      },
      "cell_type": "code",
      "source": [
        "def initialize_model(model_name, num_classes, feature_extract, use_pretrained=True):\n",
        "    # Initialize these variables which will be set in this if statement. Each of these\n",
        "    #   variables is model specific.\n",
        "    model_ft = None\n",
        "    input_size = 0\n",
        "\n",
        "    if model_name == \"resnet\":\n",
        "        \"\"\" Resnet18\n",
        "        \"\"\"\n",
        "        model_ft = models.resnet18(pretrained=use_pretrained)\n",
        "        set_parameter_requires_grad(model_ft, feature_extract)\n",
        "        num_ftrs = model_ft.fc.in_features\n",
        "        model_ft.fc = nn.Linear(num_ftrs, num_classes)\n",
        "        input_size = 224\n",
        "\n",
        "    elif model_name == \"alexnet\":\n",
        "        \"\"\" Alexnet\n",
        "        \"\"\"\n",
        "        model_ft = models.alexnet(pretrained=use_pretrained)\n",
        "        set_parameter_requires_grad(model_ft, feature_extract)\n",
        "        num_ftrs = model_ft.classifier[6].in_features\n",
        "        model_ft.classifier[6] = nn.Linear(num_ftrs,num_classes)\n",
        "        input_size = 224\n",
        "\n",
        "    elif model_name == \"vgg\":\n",
        "        \"\"\" VGG11_bn\n",
        "        \"\"\"\n",
        "        model_ft = models.vgg11_bn(pretrained=use_pretrained)\n",
        "        set_parameter_requires_grad(model_ft, feature_extract)\n",
        "        num_ftrs = model_ft.classifier[6].in_features\n",
        "        model_ft.classifier[6] = nn.Linear(num_ftrs,num_classes)\n",
        "        input_size = 224\n",
        "\n",
        "    elif model_name == \"squeezenet\":\n",
        "        \"\"\" Squeezenet\n",
        "        \"\"\"\n",
        "        model_ft = models.squeezenet1_0(pretrained=use_pretrained)\n",
        "        set_parameter_requires_grad(model_ft, feature_extract)\n",
        "        model_ft.classifier[1] = nn.Conv2d(512, num_classes, kernel_size=(1,1), stride=(1,1))\n",
        "        model_ft.num_classes = num_classes\n",
        "        input_size = 224\n",
        "\n",
        "    elif model_name == \"densenet\":\n",
        "        \"\"\" Densenet\n",
        "        \"\"\"\n",
        "        model_ft = models.densenet121(pretrained=use_pretrained)\n",
        "        set_parameter_requires_grad(model_ft, feature_extract)\n",
        "        num_ftrs = model_ft.classifier.in_features\n",
        "        model_ft.classifier = nn.Linear(num_ftrs, num_classes) \n",
        "        input_size = 224\n",
        "\n",
        "    elif model_name == \"inception\":\n",
        "        \"\"\" Inception v3 \n",
        "        Be careful, expects (299,299) sized images and has auxiliary output\n",
        "        \"\"\"\n",
        "        model_ft = models.inception_v3(pretrained=use_pretrained)\n",
        "        set_parameter_requires_grad(model_ft, feature_extract)\n",
        "        # Handle the auxilary net\n",
        "        num_ftrs = model_ft.AuxLogits.fc.in_features\n",
        "        model_ft.AuxLogits.fc = nn.Linear(num_ftrs, num_classes)\n",
        "        # Handle the primary net\n",
        "        num_ftrs = model_ft.fc.in_features\n",
        "        model_ft.fc = nn.Linear(num_ftrs,num_classes)\n",
        "        input_size = 299\n",
        "\n",
        "    else:\n",
        "        print(\"Invalid model name, exiting...\")\n",
        "        exit()\n",
        "    \n",
        "    return model_ft, input_size\n",
        "\n",
        "# Initialize the model for this run\n",
        "model_ft, input_size = initialize_model(model_name, num_classes, feature_extract, use_pretrained=True)\n",
        "\n",
        "# Print the model we just instantiated\n",
        "print(model_ft)"
      ],
      "execution_count": 74,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Inception3(\n",
            "  (Conv2d_1a_3x3): BasicConv2d(\n",
            "    (conv): Conv2d(3, 32, kernel_size=(3, 3), stride=(2, 2), bias=False)\n",
            "    (bn): BatchNorm2d(32, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "  )\n",
            "  (Conv2d_2a_3x3): BasicConv2d(\n",
            "    (conv): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), bias=False)\n",
            "    (bn): BatchNorm2d(32, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "  )\n",
            "  (Conv2d_2b_3x3): BasicConv2d(\n",
            "    (conv): Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "    (bn): BatchNorm2d(64, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "  )\n",
            "  (Conv2d_3b_1x1): BasicConv2d(\n",
            "    (conv): Conv2d(64, 80, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "    (bn): BatchNorm2d(80, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "  )\n",
            "  (Conv2d_4a_3x3): BasicConv2d(\n",
            "    (conv): Conv2d(80, 192, kernel_size=(3, 3), stride=(1, 1), bias=False)\n",
            "    (bn): BatchNorm2d(192, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "  )\n",
            "  (Mixed_5b): InceptionA(\n",
            "    (branch1x1): BasicConv2d(\n",
            "      (conv): Conv2d(192, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(64, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch5x5_1): BasicConv2d(\n",
            "      (conv): Conv2d(192, 48, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(48, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch5x5_2): BasicConv2d(\n",
            "      (conv): Conv2d(48, 64, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2), bias=False)\n",
            "      (bn): BatchNorm2d(64, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch3x3dbl_1): BasicConv2d(\n",
            "      (conv): Conv2d(192, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(64, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch3x3dbl_2): BasicConv2d(\n",
            "      (conv): Conv2d(64, 96, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(96, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch3x3dbl_3): BasicConv2d(\n",
            "      (conv): Conv2d(96, 96, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(96, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch_pool): BasicConv2d(\n",
            "      (conv): Conv2d(192, 32, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(32, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "  )\n",
            "  (Mixed_5c): InceptionA(\n",
            "    (branch1x1): BasicConv2d(\n",
            "      (conv): Conv2d(256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(64, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch5x5_1): BasicConv2d(\n",
            "      (conv): Conv2d(256, 48, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(48, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch5x5_2): BasicConv2d(\n",
            "      (conv): Conv2d(48, 64, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2), bias=False)\n",
            "      (bn): BatchNorm2d(64, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch3x3dbl_1): BasicConv2d(\n",
            "      (conv): Conv2d(256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(64, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch3x3dbl_2): BasicConv2d(\n",
            "      (conv): Conv2d(64, 96, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(96, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch3x3dbl_3): BasicConv2d(\n",
            "      (conv): Conv2d(96, 96, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(96, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch_pool): BasicConv2d(\n",
            "      (conv): Conv2d(256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(64, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "  )\n",
            "  (Mixed_5d): InceptionA(\n",
            "    (branch1x1): BasicConv2d(\n",
            "      (conv): Conv2d(288, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(64, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch5x5_1): BasicConv2d(\n",
            "      (conv): Conv2d(288, 48, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(48, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch5x5_2): BasicConv2d(\n",
            "      (conv): Conv2d(48, 64, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2), bias=False)\n",
            "      (bn): BatchNorm2d(64, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch3x3dbl_1): BasicConv2d(\n",
            "      (conv): Conv2d(288, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(64, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch3x3dbl_2): BasicConv2d(\n",
            "      (conv): Conv2d(64, 96, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(96, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch3x3dbl_3): BasicConv2d(\n",
            "      (conv): Conv2d(96, 96, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(96, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch_pool): BasicConv2d(\n",
            "      (conv): Conv2d(288, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(64, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "  )\n",
            "  (Mixed_6a): InceptionB(\n",
            "    (branch3x3): BasicConv2d(\n",
            "      (conv): Conv2d(288, 384, kernel_size=(3, 3), stride=(2, 2), bias=False)\n",
            "      (bn): BatchNorm2d(384, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch3x3dbl_1): BasicConv2d(\n",
            "      (conv): Conv2d(288, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(64, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch3x3dbl_2): BasicConv2d(\n",
            "      (conv): Conv2d(64, 96, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(96, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch3x3dbl_3): BasicConv2d(\n",
            "      (conv): Conv2d(96, 96, kernel_size=(3, 3), stride=(2, 2), bias=False)\n",
            "      (bn): BatchNorm2d(96, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "  )\n",
            "  (Mixed_6b): InceptionC(\n",
            "    (branch1x1): BasicConv2d(\n",
            "      (conv): Conv2d(768, 192, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(192, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch7x7_1): BasicConv2d(\n",
            "      (conv): Conv2d(768, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(128, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch7x7_2): BasicConv2d(\n",
            "      (conv): Conv2d(128, 128, kernel_size=(1, 7), stride=(1, 1), padding=(0, 3), bias=False)\n",
            "      (bn): BatchNorm2d(128, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch7x7_3): BasicConv2d(\n",
            "      (conv): Conv2d(128, 192, kernel_size=(7, 1), stride=(1, 1), padding=(3, 0), bias=False)\n",
            "      (bn): BatchNorm2d(192, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch7x7dbl_1): BasicConv2d(\n",
            "      (conv): Conv2d(768, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(128, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch7x7dbl_2): BasicConv2d(\n",
            "      (conv): Conv2d(128, 128, kernel_size=(7, 1), stride=(1, 1), padding=(3, 0), bias=False)\n",
            "      (bn): BatchNorm2d(128, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch7x7dbl_3): BasicConv2d(\n",
            "      (conv): Conv2d(128, 128, kernel_size=(1, 7), stride=(1, 1), padding=(0, 3), bias=False)\n",
            "      (bn): BatchNorm2d(128, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch7x7dbl_4): BasicConv2d(\n",
            "      (conv): Conv2d(128, 128, kernel_size=(7, 1), stride=(1, 1), padding=(3, 0), bias=False)\n",
            "      (bn): BatchNorm2d(128, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch7x7dbl_5): BasicConv2d(\n",
            "      (conv): Conv2d(128, 192, kernel_size=(1, 7), stride=(1, 1), padding=(0, 3), bias=False)\n",
            "      (bn): BatchNorm2d(192, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch_pool): BasicConv2d(\n",
            "      (conv): Conv2d(768, 192, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(192, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "  )\n",
            "  (Mixed_6c): InceptionC(\n",
            "    (branch1x1): BasicConv2d(\n",
            "      (conv): Conv2d(768, 192, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(192, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch7x7_1): BasicConv2d(\n",
            "      (conv): Conv2d(768, 160, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(160, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch7x7_2): BasicConv2d(\n",
            "      (conv): Conv2d(160, 160, kernel_size=(1, 7), stride=(1, 1), padding=(0, 3), bias=False)\n",
            "      (bn): BatchNorm2d(160, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch7x7_3): BasicConv2d(\n",
            "      (conv): Conv2d(160, 192, kernel_size=(7, 1), stride=(1, 1), padding=(3, 0), bias=False)\n",
            "      (bn): BatchNorm2d(192, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch7x7dbl_1): BasicConv2d(\n",
            "      (conv): Conv2d(768, 160, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(160, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch7x7dbl_2): BasicConv2d(\n",
            "      (conv): Conv2d(160, 160, kernel_size=(7, 1), stride=(1, 1), padding=(3, 0), bias=False)\n",
            "      (bn): BatchNorm2d(160, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch7x7dbl_3): BasicConv2d(\n",
            "      (conv): Conv2d(160, 160, kernel_size=(1, 7), stride=(1, 1), padding=(0, 3), bias=False)\n",
            "      (bn): BatchNorm2d(160, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch7x7dbl_4): BasicConv2d(\n",
            "      (conv): Conv2d(160, 160, kernel_size=(7, 1), stride=(1, 1), padding=(3, 0), bias=False)\n",
            "      (bn): BatchNorm2d(160, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch7x7dbl_5): BasicConv2d(\n",
            "      (conv): Conv2d(160, 192, kernel_size=(1, 7), stride=(1, 1), padding=(0, 3), bias=False)\n",
            "      (bn): BatchNorm2d(192, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch_pool): BasicConv2d(\n",
            "      (conv): Conv2d(768, 192, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(192, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "  )\n",
            "  (Mixed_6d): InceptionC(\n",
            "    (branch1x1): BasicConv2d(\n",
            "      (conv): Conv2d(768, 192, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(192, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch7x7_1): BasicConv2d(\n",
            "      (conv): Conv2d(768, 160, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(160, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch7x7_2): BasicConv2d(\n",
            "      (conv): Conv2d(160, 160, kernel_size=(1, 7), stride=(1, 1), padding=(0, 3), bias=False)\n",
            "      (bn): BatchNorm2d(160, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch7x7_3): BasicConv2d(\n",
            "      (conv): Conv2d(160, 192, kernel_size=(7, 1), stride=(1, 1), padding=(3, 0), bias=False)\n",
            "      (bn): BatchNorm2d(192, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch7x7dbl_1): BasicConv2d(\n",
            "      (conv): Conv2d(768, 160, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(160, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch7x7dbl_2): BasicConv2d(\n",
            "      (conv): Conv2d(160, 160, kernel_size=(7, 1), stride=(1, 1), padding=(3, 0), bias=False)\n",
            "      (bn): BatchNorm2d(160, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch7x7dbl_3): BasicConv2d(\n",
            "      (conv): Conv2d(160, 160, kernel_size=(1, 7), stride=(1, 1), padding=(0, 3), bias=False)\n",
            "      (bn): BatchNorm2d(160, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch7x7dbl_4): BasicConv2d(\n",
            "      (conv): Conv2d(160, 160, kernel_size=(7, 1), stride=(1, 1), padding=(3, 0), bias=False)\n",
            "      (bn): BatchNorm2d(160, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch7x7dbl_5): BasicConv2d(\n",
            "      (conv): Conv2d(160, 192, kernel_size=(1, 7), stride=(1, 1), padding=(0, 3), bias=False)\n",
            "      (bn): BatchNorm2d(192, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch_pool): BasicConv2d(\n",
            "      (conv): Conv2d(768, 192, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(192, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "  )\n",
            "  (Mixed_6e): InceptionC(\n",
            "    (branch1x1): BasicConv2d(\n",
            "      (conv): Conv2d(768, 192, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(192, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch7x7_1): BasicConv2d(\n",
            "      (conv): Conv2d(768, 192, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(192, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch7x7_2): BasicConv2d(\n",
            "      (conv): Conv2d(192, 192, kernel_size=(1, 7), stride=(1, 1), padding=(0, 3), bias=False)\n",
            "      (bn): BatchNorm2d(192, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch7x7_3): BasicConv2d(\n",
            "      (conv): Conv2d(192, 192, kernel_size=(7, 1), stride=(1, 1), padding=(3, 0), bias=False)\n",
            "      (bn): BatchNorm2d(192, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch7x7dbl_1): BasicConv2d(\n",
            "      (conv): Conv2d(768, 192, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(192, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch7x7dbl_2): BasicConv2d(\n",
            "      (conv): Conv2d(192, 192, kernel_size=(7, 1), stride=(1, 1), padding=(3, 0), bias=False)\n",
            "      (bn): BatchNorm2d(192, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch7x7dbl_3): BasicConv2d(\n",
            "      (conv): Conv2d(192, 192, kernel_size=(1, 7), stride=(1, 1), padding=(0, 3), bias=False)\n",
            "      (bn): BatchNorm2d(192, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch7x7dbl_4): BasicConv2d(\n",
            "      (conv): Conv2d(192, 192, kernel_size=(7, 1), stride=(1, 1), padding=(3, 0), bias=False)\n",
            "      (bn): BatchNorm2d(192, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch7x7dbl_5): BasicConv2d(\n",
            "      (conv): Conv2d(192, 192, kernel_size=(1, 7), stride=(1, 1), padding=(0, 3), bias=False)\n",
            "      (bn): BatchNorm2d(192, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch_pool): BasicConv2d(\n",
            "      (conv): Conv2d(768, 192, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(192, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "  )\n",
            "  (AuxLogits): InceptionAux(\n",
            "    (conv0): BasicConv2d(\n",
            "      (conv): Conv2d(768, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(128, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (conv1): BasicConv2d(\n",
            "      (conv): Conv2d(128, 768, kernel_size=(5, 5), stride=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(768, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (fc): Linear(in_features=768, out_features=102, bias=True)\n",
            "  )\n",
            "  (Mixed_7a): InceptionD(\n",
            "    (branch3x3_1): BasicConv2d(\n",
            "      (conv): Conv2d(768, 192, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(192, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch3x3_2): BasicConv2d(\n",
            "      (conv): Conv2d(192, 320, kernel_size=(3, 3), stride=(2, 2), bias=False)\n",
            "      (bn): BatchNorm2d(320, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch7x7x3_1): BasicConv2d(\n",
            "      (conv): Conv2d(768, 192, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(192, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch7x7x3_2): BasicConv2d(\n",
            "      (conv): Conv2d(192, 192, kernel_size=(1, 7), stride=(1, 1), padding=(0, 3), bias=False)\n",
            "      (bn): BatchNorm2d(192, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch7x7x3_3): BasicConv2d(\n",
            "      (conv): Conv2d(192, 192, kernel_size=(7, 1), stride=(1, 1), padding=(3, 0), bias=False)\n",
            "      (bn): BatchNorm2d(192, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch7x7x3_4): BasicConv2d(\n",
            "      (conv): Conv2d(192, 192, kernel_size=(3, 3), stride=(2, 2), bias=False)\n",
            "      (bn): BatchNorm2d(192, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "  )\n",
            "  (Mixed_7b): InceptionE(\n",
            "    (branch1x1): BasicConv2d(\n",
            "      (conv): Conv2d(1280, 320, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(320, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch3x3_1): BasicConv2d(\n",
            "      (conv): Conv2d(1280, 384, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(384, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch3x3_2a): BasicConv2d(\n",
            "      (conv): Conv2d(384, 384, kernel_size=(1, 3), stride=(1, 1), padding=(0, 1), bias=False)\n",
            "      (bn): BatchNorm2d(384, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch3x3_2b): BasicConv2d(\n",
            "      (conv): Conv2d(384, 384, kernel_size=(3, 1), stride=(1, 1), padding=(1, 0), bias=False)\n",
            "      (bn): BatchNorm2d(384, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch3x3dbl_1): BasicConv2d(\n",
            "      (conv): Conv2d(1280, 448, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(448, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch3x3dbl_2): BasicConv2d(\n",
            "      (conv): Conv2d(448, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(384, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch3x3dbl_3a): BasicConv2d(\n",
            "      (conv): Conv2d(384, 384, kernel_size=(1, 3), stride=(1, 1), padding=(0, 1), bias=False)\n",
            "      (bn): BatchNorm2d(384, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch3x3dbl_3b): BasicConv2d(\n",
            "      (conv): Conv2d(384, 384, kernel_size=(3, 1), stride=(1, 1), padding=(1, 0), bias=False)\n",
            "      (bn): BatchNorm2d(384, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch_pool): BasicConv2d(\n",
            "      (conv): Conv2d(1280, 192, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(192, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "  )\n",
            "  (Mixed_7c): InceptionE(\n",
            "    (branch1x1): BasicConv2d(\n",
            "      (conv): Conv2d(2048, 320, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(320, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch3x3_1): BasicConv2d(\n",
            "      (conv): Conv2d(2048, 384, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(384, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch3x3_2a): BasicConv2d(\n",
            "      (conv): Conv2d(384, 384, kernel_size=(1, 3), stride=(1, 1), padding=(0, 1), bias=False)\n",
            "      (bn): BatchNorm2d(384, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch3x3_2b): BasicConv2d(\n",
            "      (conv): Conv2d(384, 384, kernel_size=(3, 1), stride=(1, 1), padding=(1, 0), bias=False)\n",
            "      (bn): BatchNorm2d(384, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch3x3dbl_1): BasicConv2d(\n",
            "      (conv): Conv2d(2048, 448, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(448, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch3x3dbl_2): BasicConv2d(\n",
            "      (conv): Conv2d(448, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(384, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch3x3dbl_3a): BasicConv2d(\n",
            "      (conv): Conv2d(384, 384, kernel_size=(1, 3), stride=(1, 1), padding=(0, 1), bias=False)\n",
            "      (bn): BatchNorm2d(384, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch3x3dbl_3b): BasicConv2d(\n",
            "      (conv): Conv2d(384, 384, kernel_size=(3, 1), stride=(1, 1), padding=(1, 0), bias=False)\n",
            "      (bn): BatchNorm2d(384, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "    (branch_pool): BasicConv2d(\n",
            "      (conv): Conv2d(2048, 192, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
            "      (bn): BatchNorm2d(192, eps=0.001, momentum=0.1, affine=True, track_running_stats=True)\n",
            "    )\n",
            "  )\n",
            "  (fc): Linear(in_features=2048, out_features=102, bias=True)\n",
            ")\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "hD3nB56ZbfgE",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Load Data\n",
        "---------\n",
        "\n",
        "Now that we know what the input size must be, we can initialize the data\n",
        "transforms, image datasets, and the dataloaders. Notice, the models were\n",
        "pretrained with the hard-coded normalization values, as described\n",
        "`here <https://pytorch.org/docs/master/torchvision/models.html>`__.\n",
        "\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "i4uixJZPbfgH",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "outputId": "22dcfc5a-36f9-445b-8ba8-139be94f53b4"
      },
      "cell_type": "code",
      "source": [
        "# Data augmentation and normalization for training\n",
        "# Just normalization for validation\n",
        "data_transforms = {\n",
        "    'train': transforms.Compose([\n",
        "        transforms.RandomResizedCrop(input_size),\n",
        "        transforms.RandomHorizontalFlip(),\n",
        "        transforms.ToTensor(),\n",
        "        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])\n",
        "    ]),\n",
        "    'valid': transforms.Compose([\n",
        "        transforms.Resize(input_size),\n",
        "        transforms.CenterCrop(input_size),\n",
        "        transforms.ToTensor(),\n",
        "        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])\n",
        "    ]),\n",
        "}\n",
        "\n",
        "print(\"Initializing Datasets and Dataloaders...\")\n",
        "\n",
        "# Create training and validation datasets\n",
        "image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x), data_transforms[x]) for x in ['train', 'valid']}\n",
        "# Create training and validation dataloaders\n",
        "dataloaders_dict = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=batch_size, shuffle=True, num_workers=4) for x in ['train', 'valid']}\n",
        "\n",
        "# Detect if we have a GPU available\n",
        "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
        "device\n"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Initializing Datasets and Dataloaders...\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "device(type='cuda', index=0)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 20
        }
      ]
    },
    {
      "metadata": {
        "id": "iO76XxZwbfgL",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Create the Optimizer\n",
        "--------------------\n",
        "\n",
        "Now that the model structure is correct, the final step for finetuning\n",
        "and feature extracting is to create an optimizer that only updates the\n",
        "desired parameters. Recall that after loading the pretrained model, but\n",
        "before reshaping, if ``feature_extract=True`` we manually set all of the\n",
        "parameter’s ``.requires_grad`` attributes to False. Then the\n",
        "reinitialized layer’s parameters have ``.requires_grad=True`` by\n",
        "default. So now we know that *all parameters that have\n",
        ".requires_grad=True should be optimized.* Next, we make a list of such\n",
        "parameters and input this list to the SGD algorithm constructor.\n",
        "\n",
        "To verify this, check out the printed parameters to learn. When\n",
        "finetuning, this list should be long and include all of the model\n",
        "parameters. However, when feature extracting this list should be short\n",
        "and only include the weights and biases of the reshaped layers.\n",
        "\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "YDiPisrubfgM",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 107
        },
        "outputId": "f9257df4-0a94-4a1e-9fe7-96263d1629bd"
      },
      "cell_type": "code",
      "source": [
        "# Send the model to GPU\n",
        "model_ft = model_ft.to(device)\n",
        "\n",
        "# Gather the parameters to be optimized/updated in this run. If we are\n",
        "#  finetuning we will be updating all parameters. However, if we are \n",
        "#  doing feature extract method, we will only update the parameters\n",
        "#  that we have just initialized, i.e. the parameters with requires_grad\n",
        "#  is True.\n",
        "params_to_update = model_ft.parameters()\n",
        "print(\"Params to learn:\")\n",
        "if feature_extract:\n",
        "    params_to_update = []\n",
        "    for name,param in model_ft.named_parameters():\n",
        "        if param.requires_grad == True:\n",
        "            params_to_update.append(param)\n",
        "            print(\"\\t\",name)\n",
        "else:\n",
        "    for name,param in model_ft.named_parameters():\n",
        "        if param.requires_grad == True:\n",
        "            print(\"\\t\",name)\n",
        "\n",
        "# Observe that all parameters are being optimized\n",
        "optimizer_ft = optim.SGD(params_to_update, lr=0.001, momentum=0.9) #@param [\"optim.SGD(params_to_update, lr=0.001, momentum=0.9)\", \"optim.Adam(model_ft.parameters(), lr=0.001)\"] {type:\"raw\"}\n",
        "\n",
        "# Decay LR by a factor of 0.1 every 7 epochs\n",
        "# ACTIVATION: add \"scheduler\" arg. to train_model \n",
        "exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=7, gamma=0.1)"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Params to learn:\n",
            "\t AuxLogits.fc.weight\n",
            "\t AuxLogits.fc.bias\n",
            "\t fc.weight\n",
            "\t fc.bias\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "26agXGqGbfgS",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Run Training and Validation Step\n",
        "--------------------------------\n",
        "\n",
        "Finally, the last step is to setup the loss for the model, then run the\n",
        "training and validation function for the set number of epochs. Notice,\n",
        "depending on the number of epochs this step may take a while on a CPU.\n",
        "Also, the default learning rate is not optimal for all of the models, so\n",
        "to achieve maximum accuracy it would be necessary to tune for each model\n",
        "separately.\n",
        "\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "_d93gTuSZKrn",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1853
        },
        "outputId": "686799d9-8bd3-421c-fb1f-da1be9bc16a7"
      },
      "cell_type": "code",
      "source": [
        "# Setup the loss fxn\n",
        "criterion = nn.CrossEntropyLoss() #@param [\"nn.CrossEntropyLoss()\", \"nn.NLLLoss()\"] {type:\"raw\"}\n",
        "num_epochs = 20 #@param {type:\"integer\"}\n",
        "# Train and evaluate\n",
        "model_ft, hist = train_model(model_ft, dataloaders_dict, criterion, optimizer_ft, exp_lr_scheduler, num_epochs=num_epochs, is_inception=(model_name==\"inception\"))"
      ],
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 0/19\n",
            "----------\n",
            "train Loss: 5.5954 Acc: 0.1389\n",
            "valid Loss: 3.3892 Acc: 0.3655\n",
            "\n",
            "Epoch 1/19\n",
            "----------\n",
            "train Loss: 4.3263 Acc: 0.3223\n",
            "valid Loss: 2.6362 Acc: 0.4829\n",
            "\n",
            "Epoch 2/19\n",
            "----------\n",
            "train Loss: 3.5739 Acc: 0.4222\n",
            "valid Loss: 2.1613 Acc: 0.6198\n",
            "\n",
            "Epoch 3/19\n",
            "----------\n",
            "train Loss: 3.0949 Acc: 0.4971\n",
            "valid Loss: 1.7882 Acc: 0.6760\n",
            "\n",
            "Epoch 4/19\n",
            "----------\n",
            "train Loss: 2.7514 Acc: 0.5482\n",
            "valid Loss: 1.5323 Acc: 0.7323\n",
            "\n",
            "Epoch 5/19\n",
            "----------\n",
            "train Loss: 2.5046 Acc: 0.5870\n",
            "valid Loss: 1.3559 Acc: 0.7604\n",
            "\n",
            "Epoch 6/19\n",
            "----------\n",
            "train Loss: 2.3215 Acc: 0.6165\n",
            "valid Loss: 1.2318 Acc: 0.7885\n",
            "\n",
            "Epoch 7/19\n",
            "----------\n",
            "train Loss: 2.1923 Acc: 0.6398\n",
            "valid Loss: 1.2237 Acc: 0.7885\n",
            "\n",
            "Epoch 8/19\n",
            "----------\n",
            "train Loss: 2.1721 Acc: 0.6499\n",
            "valid Loss: 1.1930 Acc: 0.7910\n",
            "\n",
            "Epoch 9/19\n",
            "----------\n",
            "train Loss: 2.1730 Acc: 0.6445\n",
            "valid Loss: 1.2262 Acc: 0.7751\n",
            "\n",
            "Epoch 10/19\n",
            "----------\n",
            "train Loss: 2.0999 Acc: 0.6540\n",
            "valid Loss: 1.1573 Acc: 0.7848\n",
            "\n",
            "Epoch 11/19\n",
            "----------\n",
            "train Loss: 2.1239 Acc: 0.6461\n",
            "valid Loss: 1.1514 Acc: 0.8020\n",
            "\n",
            "Epoch 12/19\n",
            "----------\n",
            "train Loss: 2.1148 Acc: 0.6490\n",
            "valid Loss: 1.1333 Acc: 0.7946\n",
            "\n",
            "Epoch 13/19\n",
            "----------\n",
            "train Loss: 2.0742 Acc: 0.6574\n",
            "valid Loss: 1.1444 Acc: 0.7885\n",
            "\n",
            "Epoch 14/19\n",
            "----------\n",
            "train Loss: 2.0725 Acc: 0.6688\n",
            "valid Loss: 1.1126 Acc: 0.8044\n",
            "\n",
            "Epoch 15/19\n",
            "----------\n",
            "train Loss: 2.0824 Acc: 0.6603\n",
            "valid Loss: 1.1421 Acc: 0.7824\n",
            "\n",
            "Epoch 16/19\n",
            "----------\n",
            "train Loss: 2.0681 Acc: 0.6674\n",
            "valid Loss: 1.1453 Acc: 0.7934\n",
            "\n",
            "Epoch 17/19\n",
            "----------\n",
            "train Loss: 2.0761 Acc: 0.6648\n",
            "valid Loss: 1.1323 Acc: 0.7946\n",
            "\n",
            "Epoch 18/19\n",
            "----------\n",
            "train Loss: 2.0719 Acc: 0.6525\n",
            "valid Loss: 1.1007 Acc: 0.7983\n",
            "\n",
            "Epoch 19/19\n",
            "----------\n",
            "train Loss: 2.0602 Acc: 0.6647\n",
            "valid Loss: 1.1233 Acc: 0.7934\n",
            "\n",
            "Training complete in 45m 58s\n",
            "Best val Acc: 0.804401\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "cf04zEDCvf1Z",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Save the checkpoint\n"
      ]
    },
    {
      "metadata": {
        "id": "5NsYj-dUvdmC",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 127
        },
        "outputId": "ecec3448-b04e-420b-abf8-900c66234c70"
      },
      "cell_type": "code",
      "source": [
        "# Auth drive\n",
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=email%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdocs.test%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive.photos.readonly%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fpeopleapi.readonly&response_type=code\n",
            "\n",
            "Enter your authorization code:\n",
            "··········\n",
            "Mounted at /content/drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "ptxNPafuvtuk",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "model_ft.class_to_idx = image_datasets['train'].class_to_idx"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "dhWYdRs7yf_N",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Save the checkpoint\n",
        "\n",
        "checkpoint = {\n",
        "            'epochs': num_epochs,\n",
        "            'model_state_dict': model_ft.state_dict(),\n",
        "            'optimizer_state_dict': optimizer_ft.state_dict(),\n",
        "            'fc': model_ft.fc,\n",
        "            'aux_fc':model_ft.AuxLogits.fc,\n",
        "            'class_to_idx': model_ft.class_to_idx,\n",
        "            'criterion':criterion,\n",
        "            'optimizer':optimizer_ft,\n",
        "            'accuracy_history':hist,\n",
        "            'model_name':model_name\n",
        "            }"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "NLsqu0600zt5",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "10cb4803-eb60-4f77-db23-b0159bdc6b48"
      },
      "cell_type": "code",
      "source": [
        "torch.save(checkpoint, 'drive/My Drive/Udacity Pytorch/incV3_NOlogsoft_sgd_Cross_20e_80a.pt')\n",
        "checkpoint.keys()\n",
        "print('Saved')"
      ],
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Saved\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "QeEg-0181XUg",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Loading the checkpoint"
      ]
    },
    {
      "metadata": {
        "id": "Ek1lLWbLyuI4",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "\n",
        "def load_checkpoint(checkpoint_path):    \n",
        "    \"\"\"\n",
        "        Loads a checkpoint and rebuilds the model\n",
        "        \n",
        "    \"\"\"\n",
        "    checkpoint = torch.load(checkpoint_path)\n",
        "\n",
        "    if model_name == \"resnet\":\n",
        "            \"\"\" Resnet18\n",
        "            \"\"\"\n",
        "            model = models.resnet18()\n",
        "            set_parameter_requires_grad(model_ft, feature_extract)\n",
        "            num_ftrs = model_ft.fc.in_features\n",
        "            model_ft.fc = nn.Linear(num_ftrs, num_classes)\n",
        "            input_size = 224\n",
        "\n",
        "    elif model_name == \"alexnet\":\n",
        "            \"\"\" Alexnet\n",
        "            \"\"\"\n",
        "            model_ft = models.alexnet(pretrained=use_pretrained)\n",
        "            set_parameter_requires_grad(model_ft, feature_extract)\n",
        "            num_ftrs = model_ft.classifier[6].in_features\n",
        "            model_ft.classifier[6] = nn.Linear(num_ftrs,num_classes)\n",
        "            input_size = 224\n",
        "\n",
        "    elif model_name == \"vgg\":\n",
        "            \"\"\" VGG11_bn\n",
        "            \"\"\"\n",
        "            model_ft = models.vgg11_bn(pretrained=use_pretrained)\n",
        "            set_parameter_requires_grad(model_ft, feature_extract)\n",
        "            num_ftrs = model_ft.classifier[6].in_features\n",
        "            model_ft.classifier[6] = nn.Linear(num_ftrs,num_classes)\n",
        "            input_size = 224\n",
        "\n",
        "    elif model_name == \"squeezenet\":\n",
        "            \"\"\" Squeezenet\n",
        "            \"\"\"\n",
        "            model_ft = models.squeezenet1_0(pretrained=use_pretrained)\n",
        "            set_parameter_requires_grad(model_ft, feature_extract)\n",
        "            model_ft.classifier[1] = nn.Conv2d(512, num_classes, kernel_size=(1,1), stride=(1,1))\n",
        "            model_ft.num_classes = num_classes\n",
        "            input_size = 224\n",
        "\n",
        "    elif model_name == \"densenet\":\n",
        "            \"\"\" Densenet\n",
        "            \"\"\"\n",
        "            model_ft = models.densenet121(pretrained=use_pretrained)\n",
        "            set_parameter_requires_grad(model_ft, feature_extract)\n",
        "            num_ftrs = model_ft.classifier.in_features\n",
        "            model_ft.classifier = nn.Linear(num_ftrs, num_classes) \n",
        "            input_size = 224\n",
        "\n",
        "    elif model_name == \"inception\":\n",
        "            \"\"\" Inception v3 \n",
        "            Be careful, expects (299,299) sized images and has auxiliary output\n",
        "            \"\"\"\n",
        "            model = models.inception_v3()\n",
        "            \n",
        "            # Handle the auxilary net      \n",
        "            model.AuxLogits.fc = checkpoint['aux_fc']\n",
        "            # Handle the primary net\n",
        "            \n",
        "            model.fc = checkpoint['fc']\n",
        "    else:\n",
        "        print('Something went wrong!')\n",
        "        print(model_name)\n",
        "\n",
        "    model.load_state_dict(checkpoint['model_state_dict'])\n",
        "\n",
        "    return model, checkpoint['class_to_idx']       \n",
        "          "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "8wIuZtcN9V-8",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "154e72a0-e7bc-4480-e69a-eb858525701a"
      },
      "cell_type": "code",
      "source": [
        "# Get the name of the flowers\n",
        "model,class_to_idx = load_checkpoint('drive/My Drive/Udacity Pytorch/incV3_NOlogsoft_sgd_Cross_20e_80a.pt')\n",
        "idx_to_class = { v : k for k,v in class_to_idx.items()}\n",
        "print('Loaded')"
      ],
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Loaded\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "xaWokBHrBA3S",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Check performance of trained model on the validation dataset"
      ]
    },
    {
      "metadata": {
        "id": "l9ePBQEmCy-9",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Get Predictions for Kaggle"
      ]
    },
    {
      "metadata": {
        "id": "Yo9xXYXp8Z1c",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        " input_size should be 229 for inception model, 224 for others.\n"
      ]
    },
    {
      "metadata": {
        "id": "7g-d5SAuMnJg",
        "colab_type": "code",
        "colab": {},
        "cellView": "both"
      },
      "cell_type": "code",
      "source": [
        "def process_image(image):\n",
        "    ''' Scales, crops, and normalizes a PIL image for a PyTorch model,\n",
        "        returns an Numpy array\n",
        "    '''\n",
        "    \n",
        "    # TODO: Process a PIL image for use in a PyTorch model\n",
        "    size = 256, 256\n",
        "    image.thumbnail(size, Image.ANTIALIAS)\n",
        "    \n",
        "    # Crop \n",
        "    image_resize = 299 #@param {type:\"integer\"}\n",
        "    left_margin = (image.width-image_resize)/2\n",
        "    bottom_margin = (image.height-image_resize)/2\n",
        "    right_margin = left_margin + image_resize\n",
        "    top_margin = bottom_margin + image_resize\n",
        "    image = image.crop((left_margin, bottom_margin, right_margin,   \n",
        "                      top_margin))\n",
        "    \n",
        "    #image = image.crop((128 - 112, 128 - 112, 128 + 112, 128 + 112))\n",
        "    npImage = np.array(image)\n",
        "    npImage = npImage/255.\n",
        "        \n",
        "    imgA = npImage[:,:,0]\n",
        "    imgB = npImage[:,:,1]\n",
        "    imgC = npImage[:,:,2]\n",
        "    \n",
        "    imgA = (imgA - 0.485)/(0.229) \n",
        "    imgB = (imgB - 0.456)/(0.224)\n",
        "    imgC = (imgC - 0.406)/(0.225)\n",
        "        \n",
        "    npImage[:,:,0] = imgA\n",
        "    npImage[:,:,1] = imgB\n",
        "    npImage[:,:,2] = imgC\n",
        "    \n",
        "    npImage = np.transpose(npImage, (2,0,1))\n",
        "    \n",
        "    return npImage"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "tfi4DGxNSvX3",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def predict(image_path, model, top_num=5):\n",
        "    # To cpu\n",
        "    model.to('cpu')\n",
        "    # Process image\n",
        "    img = process_image(Image.open(image_path))\n",
        "    \n",
        "    # Numpy -> Tensor\n",
        "    image_tensor = torch.from_numpy(img).type(torch.FloatTensor)\n",
        "    # Add batch of size 1 to image\n",
        "    model_input = image_tensor.unsqueeze(0)\n",
        "    \n",
        "    # Probs\n",
        "    probs = torch.softmax(model.forward(model_input), dim=1)\n",
        "    \n",
        "    # Top probs\n",
        "    top_probs, top_labs = probs.topk(top_num)\n",
        "    top_probs = top_probs.detach().numpy().tolist()[0] \n",
        "    top_labs = top_labs.detach().numpy().tolist()[0]\n",
        "    \n",
        "    # Convert indices to classes\n",
        "    idx_to_class = {val: key for key, val in    \n",
        "                                      class_to_idx.items()}\n",
        "    top_labels = [idx_to_class[lab] for lab in top_labs]\n",
        "    top_flowers = [cat_to_name[idx_to_class[lab]] for lab in top_labs]\n",
        "    return top_probs, top_labels, top_flowers"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "VRusp54nTaba",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "outputId": "775497fc-cfd9-4a42-eb4a-ac4a57daec5e"
      },
      "cell_type": "code",
      "source": [
        "TEST_DIR = 'flower_data/test'\n",
        "test_image_files = os.listdir(TEST_DIR)\n",
        "test_image_files[:3]"
      ],
      "execution_count": 61,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['image_06321.jpg', 'image_01609.jpg', 'image_05039.jpg']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 61
        }
      ]
    },
    {
      "metadata": {
        "id": "TnInK5rxMLoY",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "preds = list()\n",
        "names = list()\n",
        "probs = list()\n",
        "for image_file in test_image_files:\n",
        "    image_path = os.path.join(TEST_DIR, image_file)\n",
        "    top_probs, top_labels, top_flowers = predict(image_path, model, top_num=1)\n",
        "    prob = top_probs[0]\n",
        "    label = top_labels[0]\n",
        "    name = top_flowers[0]\n",
        "    #print(f'prediction for {image_file}: label = {label}, prob = {prob}, name = {name}')\n",
        "    probs.append(prob)\n",
        "    names.append(name)\n",
        "    preds.append(label)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "qtz8QLP1C9c0",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "outputId": "1d277e50-46cf-45ae-b682-d399800d0f87"
      },
      "cell_type": "code",
      "source": [
        "results = {\n",
        "    'file_name': test_image_files,\n",
        "    'label': preds,\n",
        "    'probability': probs,\n",
        "    'name': names\n",
        "}\n",
        "prediction_results = pd.DataFrame(results)\n",
        "prediction_results.head()"
      ],
      "execution_count": 63,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>file_name</th>\n",
              "      <th>label</th>\n",
              "      <th>name</th>\n",
              "      <th>probability</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>image_06321.jpg</td>\n",
              "      <td>51</td>\n",
              "      <td>petunia</td>\n",
              "      <td>0.080226</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>image_01609.jpg</td>\n",
              "      <td>51</td>\n",
              "      <td>petunia</td>\n",
              "      <td>0.116209</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>image_05039.jpg</td>\n",
              "      <td>51</td>\n",
              "      <td>petunia</td>\n",
              "      <td>0.086448</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>image_06457.jpg</td>\n",
              "      <td>51</td>\n",
              "      <td>petunia</td>\n",
              "      <td>0.106978</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>image_06031.jpg</td>\n",
              "      <td>51</td>\n",
              "      <td>petunia</td>\n",
              "      <td>0.261860</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "         file_name label     name  probability\n",
              "0  image_06321.jpg    51  petunia     0.080226\n",
              "1  image_01609.jpg    51  petunia     0.116209\n",
              "2  image_05039.jpg    51  petunia     0.086448\n",
              "3  image_06457.jpg    51  petunia     0.106978\n",
              "4  image_06031.jpg    51  petunia     0.261860"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 63
        }
      ]
    },
    {
      "metadata": {
        "id": "HQLEYEHdVL2w",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "outputId": "7b8982c0-29b5-4a7c-cd0b-f8847a6e3dad"
      },
      "cell_type": "code",
      "source": [
        "submission_df = prediction_results.drop(['probability', 'name'], axis=1)\n",
        "submission_df = submission_df.rename(columns={'label' : 'id'})\n",
        "submission_df.head()"
      ],
      "execution_count": 64,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>file_name</th>\n",
              "      <th>id</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>image_06321.jpg</td>\n",
              "      <td>51</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>image_01609.jpg</td>\n",
              "      <td>51</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>image_05039.jpg</td>\n",
              "      <td>51</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>image_06457.jpg</td>\n",
              "      <td>51</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>image_06031.jpg</td>\n",
              "      <td>51</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "         file_name  id\n",
              "0  image_06321.jpg  51\n",
              "1  image_01609.jpg  51\n",
              "2  image_05039.jpg  51\n",
              "3  image_06457.jpg  51\n",
              "4  image_06031.jpg  51"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 64
        }
      ]
    },
    {
      "metadata": {
        "id": "hIGyPPLONN-Y",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "prediction_results.to_csv('test_data_predictions.csv', sep=',', index=False)\n",
        "submission_df.to_csv('classifier_submission299.csv', sep=',', index=False)\n"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}