notebooks_EN/Lecture 17 - Convolutional neural networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "g_OXy39qk3zW"
   },
   "source": [
    "SIN-392 - Introduction to Digital Image Processing (2023-1)\n",
    "\n",
    "# Lecture 17 - Convolutional neural networks\n",
    "\n",
    "Prof. João Fernando Mari ([*joaofmari.github.io*](https://joaofmari.github.io/))\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Mounting Google Drive\n",
    "---\n",
    "\n",
    "* If you are running on Google Colab, don't forget to enable access to the GPU.\n",
    "     * Edit >> Notebook Settings >> Hardware Accelerator\n",
    "     * Select GPU\n",
    "     * OK\n",
    "* After use, disable access."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "False\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    import google.colab\n",
    "    IN_COLAB = True\n",
    "except:\n",
    "    IN_COLAB = False\n",
    "\n",
    "# DEBUG\n",
    "print(IN_COLAB)\n",
    "\n",
    "if IN_COLAB:\n",
    "    from google.colab import drive\n",
    "    drive.mount('/content/drive')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xtwkvxQSk5Ov"
   },
   "source": [
    "## Importing the required libraries\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "WD5GFozxkhwf"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "from sklearn import metrics\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "\n",
    "import torchvision\n",
    "import torchvision.transforms as transforms\n",
    "\n",
    "### %matplotlib notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rojqemqDndc4"
   },
   "source": [
    "## Checking GPU Access\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "u08Rir3zNy3m"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Device: cuda\n"
     ]
    }
   ],
   "source": [
    "DEVICE = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "print('\\nDevice: {0}'.format(DEVICE))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "wOXBEudInd-w",
    "outputId": "2946e33c-c5aa-4947-f344-b80e0bef6888"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tue Oct  3 17:49:48 2023       \n",
      "+---------------------------------------------------------------------------------------+\n",
      "| NVIDIA-SMI 535.113.01             Driver Version: 535.113.01   CUDA Version: 12.2     |\n",
      "|-----------------------------------------+----------------------+----------------------+\n",
      "| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
      "| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |\n",
      "|                                         |                      |               MIG M. |\n",
      "|=========================================+======================+======================|\n",
      "|   0  NVIDIA GeForce GTX 1050 Ti     Off | 00000000:01:00.0 Off |                  N/A |\n",
      "| 46%   31C    P8              N/A /  75W |      8MiB /  4096MiB |      0%      Default |\n",
      "|                                         |                      |                  N/A |\n",
      "+-----------------------------------------+----------------------+----------------------+\n",
      "                                                                                         \n",
      "+---------------------------------------------------------------------------------------+\n",
      "| Processes:                                                                            |\n",
      "|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |\n",
      "|        ID   ID                                                             Usage      |\n",
      "|=======================================================================================|\n",
      "|    0   N/A  N/A       996      G   /usr/lib/xorg/Xorg                            4MiB |\n",
      "+---------------------------------------------------------------------------------------+\n"
     ]
    }
   ],
   "source": [
    "!nvidia-smi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FMBopsEdk-5f"
   },
   "source": [
    "## Settings for reproducibility\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "ZlZtCMr4kz-d"
   },
   "outputs": [],
   "source": [
    "np.random.seed(1234)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gacAYGO4lG0C"
   },
   "source": [
    "## Setting some hyperparameters\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "5bgQsLBKlEbU"
   },
   "outputs": [],
   "source": [
    "# Mini-batch size\n",
    "batch_size = 8 \n",
    "\n",
    "# Number of training epochs\n",
    "max_epochs = 50 # 20"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-RAjZmYrlV4z"
   },
   "source": [
    "## The dataset\n",
    "---\n",
    "* In this first example, we will use the CIFAR-10 dataset.\n",
    "* CIFAR-10 is made up of 60,000 images measuring 32 x 32 organized into 10 classes, with 6,000 images per class.\n",
    "     * Classes: 'plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck'\n",
    "* CIFAR-10 is already divided into a training set with 50,000 images and a test set with 10,000.\n",
    "     * In other words, if necessary, a part of the training set must be separated to be used as validation.\n",
    "* CIFAR-10 can be accessed directly from the torchvision library.\n",
    "     * That is, there is no need to load the dataset from local storage."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Class names\n",
    "classes = ('plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 149,
     "referenced_widgets": [
      "5229cef2e24d4638b23cf913f297a045",
      "8f8ddb43215c4d2681850d17375e1a57",
      "a3c5a43743464f59a0cd587e1747b7ad",
      "64d5ed6f790e43f7977ad2d6153cdf7a",
      "b4588739ee6e4e9eb99c28163fcbbeb8",
      "813a473ba3654eee95de0739163efd86",
      "5fac6ea4ebfe447a9ac22ac65a9ef6c2",
      "2d248bdaf3f14eb7bf3769d6c21efd4c",
      "0e9f622b52e240ddb244ffaf0e500d39",
      "4014f51fb3e1451d8495d8fe663994ce",
      "8275df47ac1044fa820e724be92fc15b"
     ]
    },
    "id": "yrir1oYOlWb3",
    "outputId": "62f17e57-335e-4c12-f11f-09b19f831cda"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Files already downloaded and verified\n",
      "Files already downloaded and verified\n"
     ]
    }
   ],
   "source": [
    "# Set a sequence of transformations that will be applied to the images in the datasets\n",
    "transform = transforms.Compose([# Converts the mini-batch to tensor. Automatically converts values to range [0, 1]\n",
    "                                transforms.ToTensor(), \n",
    "                                # Convert values to the range [-1, 1]\n",
    "                                transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))]) \n",
    "\n",
    "# Datasets\n",
    "# --------\n",
    "# Training set\n",
    "dataset_train = torchvision.datasets.CIFAR10(root='./data', \n",
    "                                             train=True,\n",
    "                                             download=True, \n",
    "                                             transform=transform)\n",
    "# Test set\n",
    "dataset_test = torchvision.datasets.CIFAR10(root='./data', \n",
    "                                            train=False,\n",
    "                                            download=True, \n",
    "                                            transform=transform)\n",
    "\n",
    "# Dataloaders\n",
    "# -----------\n",
    "# Training set\n",
    "dataloader_train = torch.utils.data.DataLoader(dataset_train, batch_size=batch_size,\n",
    "                                               shuffle=True, num_workers=2)\n",
    "# Test set\n",
    "dataloader_test = torch.utils.data.DataLoader(dataset_test, batch_size=batch_size,\n",
    "                                              shuffle=False, num_workers=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "riys3dokleJm"
   },
   "source": [
    "## Building a simple Convolutional Neural Network\n",
    "---\n",
    "\n",
    "* Let's build a simple convolutional neural network from scratch using the PyTorch library.\n",
    "* Our CNN has the following structure:\n",
    "    1. Convolutional layer 1: Inputs with 3 channels and 6 filters with size 5 x 5\n",
    "    2. ReLU activation function\n",
    "    3. Max Pooling Layer\n",
    "    4. Convolutional layer 2: Input with 6 channels and 16 filters with size 5 x 5\n",
    "    2. ReLU activation function\n",
    "    3. Max Pooling Layer\n",
    "    4. Flattening of exits\n",
    "    5. Fully connected layer with 120 neurons\n",
    "    6. ReLU activation function\n",
    "    7. Fully connected layer with 84 neurons\n",
    "    8. ReLU activation function\n",
    "    9. Output layer with 10 neurons (dataset with 10 classes)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "qbhO_YVPlgU1"
   },
   "outputs": [],
   "source": [
    "class Net(nn.Module):\n",
    "    \"\"\"\n",
    "    Considering each image having 32 rows by 32 columns:\n",
    "    \n",
    "    Input [3, 32, 32] \n",
    "    Conv1(3, 6, 5) [6, 28, 28] \n",
    "    Pool(2, 2) [6, 14, 14] \n",
    "    Conv2(6, 16, 5) [16, 10, 10]\n",
    "    Pool(2, 2) [16, 5, 5]\n",
    "    Flatten [400] (16 x 5 x 5 = 400)\n",
    "    Fc1 [120]\n",
    "    Fc2 [84]\n",
    "    Fc3 [10]\n",
    "    \"\"\"\n",
    "    def __init__(self):\n",
    "        \"\"\"\n",
    "        torch.nn.Conv2d(in_channels, out_channels, kernel_size, stride=1, padding=0, \n",
    "                        dilation=1, groups=1, bias=True, padding_mode='zeros', device=None, dtype=None)\n",
    "        torch.nn.MaxPool2d(kernel_size, stride=None, padding=0, dilation=1, return_indices=False, ceil_mode=False)\n",
    "        torch.nn.Linear(in_features, out_features, bias=True, device=None, dtype=None)\n",
    "        \"\"\"\n",
    "        super().__init__()\n",
    "        self.conv1 = nn.Conv2d(3, 6, 5)\n",
    "        self.pool = nn.MaxPool2d(2, 2)\n",
    "        self.conv2 = nn.Conv2d(6, 16, 5)\n",
    "        self.fc1 = nn.Linear(16 * 5 * 5, 120)\n",
    "        self.fc2 = nn.Linear(120, 84)\n",
    "        self.fc3 = nn.Linear(84, 10)\n",
    "\n",
    "    def forward(self, x):\n",
    "        # conv1(x): Input: [32, 32, 3]. Output: [28, 28, 6].\n",
    "        # - With padding=0 and kernel_size=5, the image is \"reduced\" 2 lines (5-1/2 = 2) top and bottom and 2 columns left and right.\n",
    "        x = self.conv1(x)\n",
    "        x = F.relu(x)\n",
    "        # pool: Input: [28, 28, 6], Output: [14, 14, 6]\n",
    "        x = self.pool(x)\n",
    "        # conv2: Input: [14, 14, 6]. Output: [10, 10, 16].\n",
    "        # - With padding=0 and kernel_size=5, the image is \"reduced\" 2 lines (5-1/2 = 2) top and bottom and 2 columns left and right.\n",
    "        x = self.conv2(x)\n",
    "        x = F.relu(x)\n",
    "        # pool: Input: [10, 10, 16], Output: [5, 5, 16]\n",
    "        x = self.pool(x)\n",
    "        # flatten: Input: [5, 5, 16]. Output: [400]\n",
    "        x = torch.flatten(x, 1) \n",
    "        # fc1: Input: [400]. Output: [120]\n",
    "        x = self.fc1(x)\n",
    "        x = F.relu(x)\n",
    "        # fc2: Input: [120]. Output: [84]\n",
    "        x = self.fc2(x)\n",
    "        x = F.relu(x)\n",
    "        # fc3: Input: [80]. Output: [num_classes]\n",
    "        x = self.fc3(x)\n",
    "        \n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "pb2tLCROn8Jy"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Net(\n",
      "  (conv1): Conv2d(3, 6, kernel_size=(5, 5), stride=(1, 1))\n",
      "  (pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
      "  (conv2): Conv2d(6, 16, kernel_size=(5, 5), stride=(1, 1))\n",
      "  (fc1): Linear(in_features=400, out_features=120, bias=True)\n",
      "  (fc2): Linear(in_features=120, out_features=84, bias=True)\n",
      "  (fc3): Linear(in_features=84, out_features=10, bias=True)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "# Instantiate an object of the Net class\n",
    "model = Net()\n",
    "\n",
    "# Send model to GPU\n",
    "model = model.cuda() # Cuda\n",
    "\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GQ5ztheDlhxx"
   },
   "source": [
    "## Loss function and optimizer\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "nTAAdvNxllhe"
   },
   "outputs": [],
   "source": [
    "# Loss function - Cross entropy\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "\n",
    "# Optimizer - Stochastic Gradient Descent\n",
    "optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9PSrdawplqM_"
   },
   "source": [
    "## Training the model\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "vGFNN1T1lsgZ",
    "outputId": "336cb57d-d250-429b-cdbb-02ed7bb79091"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0/49 - TRAIN Loss: 91059.5220\n",
      "Epoch 1/49 - TRAIN Loss: 71219.0159\n",
      "Epoch 2/49 - TRAIN Loss: 63322.4090\n",
      "Epoch 3/49 - TRAIN Loss: 58354.1272\n",
      "Epoch 4/49 - TRAIN Loss: 54607.4862\n",
      "Epoch 5/49 - TRAIN Loss: 51305.5750\n",
      "Epoch 6/49 - TRAIN Loss: 48686.3224\n",
      "Epoch 7/49 - TRAIN Loss: 46180.8776\n",
      "Epoch 8/49 - TRAIN Loss: 44208.9232\n",
      "Epoch 9/49 - TRAIN Loss: 42288.2284\n",
      "Epoch 10/49 - TRAIN Loss: 40336.2707\n",
      "Epoch 11/49 - TRAIN Loss: 38699.2404\n",
      "Epoch 12/49 - TRAIN Loss: 37199.3423\n",
      "Epoch 13/49 - TRAIN Loss: 35760.1628\n",
      "Epoch 14/49 - TRAIN Loss: 34489.2443\n",
      "Epoch 15/49 - TRAIN Loss: 33462.9016\n",
      "Epoch 16/49 - TRAIN Loss: 32366.0826\n",
      "Epoch 17/49 - TRAIN Loss: 31418.7436\n",
      "Epoch 18/49 - TRAIN Loss: 30438.6922\n",
      "Epoch 19/49 - TRAIN Loss: 29356.8523\n",
      "Epoch 20/49 - TRAIN Loss: 28557.5266\n",
      "Epoch 21/49 - TRAIN Loss: 28080.4109\n",
      "Epoch 22/49 - TRAIN Loss: 27369.6549\n",
      "Epoch 23/49 - TRAIN Loss: 26697.6281\n",
      "Epoch 24/49 - TRAIN Loss: 25784.3382\n",
      "Epoch 25/49 - TRAIN Loss: 25403.9090\n",
      "Epoch 26/49 - TRAIN Loss: 24871.5800\n",
      "Epoch 27/49 - TRAIN Loss: 24375.8124\n",
      "Epoch 28/49 - TRAIN Loss: 23636.1738\n",
      "Epoch 29/49 - TRAIN Loss: 23348.9754\n",
      "Epoch 30/49 - TRAIN Loss: 23014.9346\n",
      "Epoch 31/49 - TRAIN Loss: 22713.6155\n",
      "Epoch 32/49 - TRAIN Loss: 22286.7526\n",
      "Epoch 33/49 - TRAIN Loss: 22363.1931\n",
      "Epoch 34/49 - TRAIN Loss: 21826.8920\n",
      "Epoch 35/49 - TRAIN Loss: 21283.5947\n",
      "Epoch 36/49 - TRAIN Loss: 21221.4489\n",
      "Epoch 37/49 - TRAIN Loss: 21016.0782\n",
      "Epoch 38/49 - TRAIN Loss: 20544.8891\n",
      "Epoch 39/49 - TRAIN Loss: 20366.7916\n",
      "Epoch 40/49 - TRAIN Loss: 19998.2274\n",
      "Epoch 41/49 - TRAIN Loss: 20157.2189\n",
      "Epoch 42/49 - TRAIN Loss: 19795.5249\n",
      "Epoch 43/49 - TRAIN Loss: 19432.6024\n",
      "Epoch 44/49 - TRAIN Loss: 19533.5936\n",
      "Epoch 45/49 - TRAIN Loss: 19400.6033\n",
      "Epoch 46/49 - TRAIN Loss: 19045.7163\n",
      "Epoch 47/49 - TRAIN Loss: 18916.4821\n",
      "Epoch 48/49 - TRAIN Loss: 19327.4723\n",
      "Epoch 49/49 - TRAIN Loss: 18586.5067\n",
      "\n",
      "Treinamento finalizado!\n"
     ]
    }
   ],
   "source": [
    "# Iterates over the dataset for a number of epochs.\n",
    "for epoch in range(max_epochs):  \n",
    "\n",
    "    # Enables the model for training mode\n",
    "    model.train() \n",
    "\n",
    "    # Epoch loss\n",
    "    train_loss = 0.0\n",
    "\n",
    "    # Training\n",
    "    for i, (inputs, labels) in enumerate(dataloader_train):\n",
    "\n",
    "        # Send data to GPU\n",
    "        inputs = inputs.to(DEVICE) \n",
    "        labels = labels.to(DEVICE) \n",
    "\n",
    "        # Zero the parameter gradients\n",
    "        optimizer.zero_grad()\n",
    "\n",
    "        # Forward \n",
    "        outputs = model(inputs)\n",
    "        \n",
    "        # Compute the loss function\n",
    "        loss = criterion(outputs, labels)\n",
    "        \n",
    "        # Backward\n",
    "        loss.backward()\n",
    "        \n",
    "        # Optimizes the parameters (weights)\n",
    "        optimizer.step()\n",
    "\n",
    "        # Update the epoch training loss\n",
    "        train_loss += loss.item() * inputs.size(0)\n",
    "\n",
    "    # Print epoch information\n",
    "    print(f'Epoch {epoch}/{max_epochs - 1} - TRAIN Loss: {train_loss:.4f}')\n",
    "        \n",
    "print('\\nTraining finished!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GeoyzzlQl0Yg"
   },
   "source": [
    "## Testing the model\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "_WeQQ4rBl2pq",
    "outputId": "4b633c24-7ecb-4b9b-cbbf-6ea77cac875f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Acurácia de testes: 0.5973\n"
     ]
    }
   ],
   "source": [
    "# Number of correctly classified images\n",
    "correct = 0\n",
    "# Total number of images\n",
    "total = 0\n",
    "\n",
    "# List of real classes and predicted classes\n",
    "true_test_list = []\n",
    "pred_test_list = []\n",
    "\n",
    "# It is not necessary to calculate the gradients.\n",
    "with torch.no_grad():\n",
    "    for inputs, labels in dataloader_test:\n",
    "        \n",
    "        inputs = inputs.to(DEVICE) \n",
    "        labels = labels.to(DEVICE) \n",
    "        \n",
    "        # Computation of the outputs\n",
    "        outputs = model(inputs)\n",
    "\n",
    "        # The image is classified according to the largest output\n",
    "        _, predicted = torch.max(outputs.data, 1)\n",
    "\n",
    "        # Update the number of images\n",
    "        total += labels.size(0)\n",
    "        # Updates the number of correct classifications\n",
    "        correct += (predicted == labels).sum().item()\n",
    "\n",
    "        true_test_list += list(labels.cpu())\n",
    "        pred_test_list += list(predicted.cpu())\n",
    "\n",
    "# Compute accuracy over the test set\n",
    "accuracy = correct / total\n",
    "\n",
    "print(f'Acurácia de testes: {accuracy}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Confusion matrix\n",
      "[[704  28  63  18  32   9  11  14  76  45]\n",
      " [ 43 747   8  12   7  10  19  11  42 101]\n",
      " [122  10 440  67 122  60  91  50  15  23]\n",
      " [ 46  12  74 387 103 169 101  53  21  34]\n",
      " [ 41   8  95  71 532  43  97  88  17   8]\n",
      " [ 24   6  67 205  75 447  61  90   9  16]\n",
      " [ 16   8  53  72  65  36 710  12  12  16]\n",
      " [ 33   6  44  46 102  54  15 671   4  25]\n",
      " [124  36  16  22  18   8  14  13 710  39]\n",
      " [ 81 120  19  23  17  17  20  22  56 625]]\n",
      "\n",
      "Class. report\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "       plane     0.5705    0.7040    0.6303      1000\n",
      "         car     0.7615    0.7470    0.7542      1000\n",
      "        bird     0.5006    0.4400    0.4683      1000\n",
      "         cat     0.4193    0.3870    0.4025      1000\n",
      "        deer     0.4958    0.5320    0.5133      1000\n",
      "         dog     0.5240    0.4470    0.4825      1000\n",
      "        frog     0.6234    0.7100    0.6639      1000\n",
      "       horse     0.6553    0.6710    0.6630      1000\n",
      "        ship     0.7380    0.7100    0.7238      1000\n",
      "       truck     0.6706    0.6250    0.6470      1000\n",
      "\n",
      "    accuracy                         0.5973     10000\n",
      "   macro avg     0.5959    0.5973    0.5949     10000\n",
      "weighted avg     0.5959    0.5973    0.5949     10000\n",
      "\n",
      "\n",
      "\n",
      "Acc.: 0.5973\n"
     ]
    }
   ],
   "source": [
    "# Confusion matrix\n",
    "conf_mat = metrics.confusion_matrix(true_test_list, pred_test_list)\n",
    "print('\\nConfusion matrix')\n",
    "print(conf_mat)\n",
    "\n",
    "# Classification report - Scikit-learn\n",
    "class_rep = metrics.classification_report(true_test_list, \n",
    "                                          pred_test_list, \n",
    "                                          target_names=classes, \n",
    "                                          digits=4,\n",
    "                                          zero_division=0)\n",
    "print('\\nClass. report')\n",
    "print(class_rep)\n",
    "\n",
    "# Accuracy\n",
    "acc = metrics.accuracy_score(true_test_list, pred_test_list)\n",
    "print('\\n\\nAcc.: {:.4f}'.format(acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dyvLmgHxl9cc"
   },
   "source": [
    "## Bibliography\n",
    "---\n",
    "* PyTorch. Training a Classifier\n",
    "    * https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html\n",
    "* Microsoft. Train your image classifier model with PyTorch.\n",
    "    * https://learn.microsoft.com/en-us/windows/ai/windows-ml/tutorials/pytorch-train-model\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "gpuClass": "standard",
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "0e9f622b52e240ddb244ffaf0e500d39": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "2d248bdaf3f14eb7bf3769d6c21efd4c": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4014f51fb3e1451d8495d8fe663994ce": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "5229cef2e24d4638b23cf913f297a045": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_8f8ddb43215c4d2681850d17375e1a57",
       "IPY_MODEL_a3c5a43743464f59a0cd587e1747b7ad",
       "IPY_MODEL_64d5ed6f790e43f7977ad2d6153cdf7a"
      ],
      "layout": "IPY_MODEL_b4588739ee6e4e9eb99c28163fcbbeb8"
     }
    },
    "5fac6ea4ebfe447a9ac22ac65a9ef6c2": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "64d5ed6f790e43f7977ad2d6153cdf7a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_4014f51fb3e1451d8495d8fe663994ce",
      "placeholder": "​",
      "style": "IPY_MODEL_8275df47ac1044fa820e724be92fc15b",
      "value": " 170498071/170498071 [00:13&lt;00:00, 13971152.44it/s]"
     }
    },
    "813a473ba3654eee95de0739163efd86": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "8275df47ac1044fa820e724be92fc15b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "8f8ddb43215c4d2681850d17375e1a57": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_813a473ba3654eee95de0739163efd86",
      "placeholder": "​",
      "style": "IPY_MODEL_5fac6ea4ebfe447a9ac22ac65a9ef6c2",
      "value": "100%"
     }
    },
    "a3c5a43743464f59a0cd587e1747b7ad": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_2d248bdaf3f14eb7bf3769d6c21efd4c",
      "max": 170498071,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_0e9f622b52e240ddb244ffaf0e500d39",
      "value": 170498071
     }
    },
    "b4588739ee6e4e9eb99c28163fcbbeb8": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}