CNN-exercise2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CNN Exercise 2: Image Classification with CNNs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use the CIFAR10 dataset, which consists of 60000 colour images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images. The classes are: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck. In this exercise we will only use the dog and cat classes and we will train a CNN to distinguish between the two.\n",
    "\n",
    "In the following cell we load the dataset again from ```torchvision.datasets```. There is a bunch of code that is not relevant for this exercise, but you can have a look at it if you are interested. The important part is that we load training and test data, only select the dog and cat classes, and do some augmentation and normalizing of the data.\n",
    "\n",
    "Thanks to [this Gist](https://gist.github.com/Miladiouss/6ba0876f0e2b65d0178be7274f61ad2f) for the code to load the CIFAR10 dataset.\n",
    "\n",
    "Created by Cedric Ewen in 2023\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import torch\n",
    "from torch import nn\n",
    "from torch import optim\n",
    "from torch.utils.data import Dataset, DataLoader\n",
    "import torchvision\n",
    "from torchvision import transforms\n",
    "from torchvision.datasets import CIFAR10\n",
    "\n",
    "from torchsummary import summary\n",
    "from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# device configuration\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "device"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# transformations\n",
    "RC = transforms.RandomCrop(32, padding=4)\n",
    "RHF = transforms.RandomHorizontalFlip()\n",
    "NRM = transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))\n",
    "TT = transforms.ToTensor()\n",
    "\n",
    "# transforms object for trainset with augmentation\n",
    "transform_with_aug = transforms.Compose([TT, RC, RHF, NRM])\n",
    "# transforms object for testset w/o augmentation\n",
    "transform_no_aug = transforms.Compose([TT, NRM])\n",
    "\n",
    "# downloading/louding CIFAR10 data\n",
    "trainset = CIFAR10(\n",
    "    root='data',\n",
    "    train=True,\n",
    "    download=True,\n",
    "    transform=transform_with_aug)\n",
    "testset = CIFAR10(\n",
    "    root='data',\n",
    "    train=False,\n",
    "    download=True,\n",
    "    transform=transform_no_aug)\n",
    "classDict = {'plane': 0, 'car': 1, 'bird': 2, 'cat': 3, 'deer': 4,\n",
    "             'dog': 5, 'frog': 6, 'horse': 7, 'ship': 8, 'truck': 9}\n",
    "classes = ['plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog',\n",
    "           'horse', 'ship', 'truck']\n",
    "\n",
    "# separating trainset/testset data/label\n",
    "x_train = trainset.data\n",
    "x_test = testset.data\n",
    "y_train = np.array(trainset.targets)\n",
    "y_test = np.array(testset.targets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now choose the classes we want to use for the classification. We start with cats and dogs and will later add more classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This code block will remove all data samples that do not belong to\n",
    "# class 'cat' or to class 'dog'.\n",
    "classes = ['cat', 'dog']\n",
    "\n",
    "# all classes of CIFAR-10\n",
    "#classes = ['plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck']\n",
    "\n",
    "# initialize masks and map according to chosen classes\n",
    "mask_train = np.zeros(len(y_train), dtype=bool)\n",
    "mask_test = np.zeros(len(y_test), dtype=bool)\n",
    "lable_map = {}\n",
    "for i, class_name in enumerate(classes):\n",
    "    mask_train = mask_train | (y_train == classDict[class_name])\n",
    "    mask_test = mask_test | (y_test == classDict[class_name])\n",
    "    lable_map[classDict[class_name]] = i\n",
    "\n",
    "# select data\n",
    "trainset.data = x_train[mask_train]\n",
    "testset.data = x_test[mask_test]\n",
    "trainset.targets = y_train[mask_train].tolist()\n",
    "testset.targets = y_test[mask_test].tolist()\n",
    "\n",
    "# make class labels continuous again\n",
    "trainset.targets = [lable_map[e] for e in trainset.targets]\n",
    "testset.targets = [lable_map[e] for e in testset.targets]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 64\n",
    "\n",
    "# create dataset loaders from trainset and testset\n",
    "kwargs = {'num_workers': 4, 'pin_memory': False,\n",
    "          'batch_size': batch_size}\n",
    "trainloader = DataLoader(\n",
    "    trainset, shuffle=True, **kwargs)\n",
    "testloader = DataLoader(\n",
    "    testset, shuffle=False, **kwargs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have a training and test set of the selected classes that are one-hot-encoded. Let's look at some of the images.\n",
    "- What's the size of the images?\n",
    "- How many images do we have in training and test set?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# functions to show an image\n",
    "def imshow(img):\n",
    "    img = img / 2 + 0.5     # unnormalize\n",
    "    npimg = img.numpy()\n",
    "    plt.imshow(np.transpose(npimg, (1, 2, 0)))\n",
    "    plt.show()\n",
    "\n",
    "# get some random training images\n",
    "dataiter = iter(trainloader)\n",
    "\n",
    "images, labels = next(dataiter)\n",
    "\n",
    "images_to_plot = 8\n",
    "\n",
    "# show images\n",
    "imshow(torchvision.utils.make_grid(images[:images_to_plot]))\n",
    "# print labels\n",
    "print(' '.join(f'{classes[labels[j]]:5s}' for j in range(images_to_plot)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: print image shape and dataset length"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Define the Network\n",
    "This is the only area you will need to change"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CNN(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(CNN, self).__init__()\n",
    "        # TODO define the layers            \n",
    "\n",
    "    def forward(self, x):\n",
    "        # TODO define the forward pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = CNN()\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "summary(model, (3, 32, 32), batch_size=batch_size, device='cpu')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cross Entropy Loss already includes Softmax\n",
    "loss_func = nn.CrossEntropyLoss()   \n",
    "loss_func"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# move model to GPU\n",
    "model = model.to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learning_rate = 0.003\n",
    "optimizer = optim.Adam(model.parameters(), lr = learning_rate)   \n",
    "optimizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The training loop is the same as in the previous exercise. The only difference is that we have added a validation loop, where we evaluate the model on the validation set (in our case the test data). We additionally save the loss and accuracy for both the training and validation set in a list, so that we can plot them later. The accuracy is the fraction of correctly classified images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train(num_epochs, model, trainloader, testloader):\n",
    "    \n",
    "    train_losses = []\n",
    "    train_accuracy = []\n",
    "    \n",
    "    val_losses = []\n",
    "    val_accuracy = []\n",
    "        \n",
    "    # Train the model    \n",
    "    for epoch in range(num_epochs):\n",
    "        \n",
    "        # training\n",
    "        model.train()\n",
    "        train_loss = 0.0\n",
    "        train_acc = 0.0\n",
    "        for i, (images, labels) in enumerate(trainloader, 0):\n",
    "            \n",
    "            # move data to GPU\n",
    "            images = images.to(device)\n",
    "            labels = labels.to(device)\n",
    "            \n",
    "            # run model\n",
    "            output = model(images)     \n",
    "            # compute loss          \n",
    "            loss = loss_func(output, labels)\n",
    "            \n",
    "            # clear gradients for this training step   \n",
    "            optimizer.zero_grad()           \n",
    "            \n",
    "            # backpropagation, compute gradients \n",
    "            loss.backward()    \n",
    "            # apply gradients             \n",
    "            optimizer.step()   \n",
    "            \n",
    "            # save metrics for ploting\n",
    "            train_loss += loss.item()\n",
    "            train_acc += (output.argmax(dim=1) == labels).float().mean().item()\n",
    "            \n",
    "        train_loss = train_loss / len(trainloader)\n",
    "        train_acc = train_acc / len(trainloader)\n",
    "        train_losses.append(train_loss)\n",
    "        train_accuracy.append(train_acc)\n",
    "        \n",
    "        # validation \n",
    "        model.eval()\n",
    "        with torch.no_grad():\n",
    "            val_loss = 0.0  \n",
    "            val_acc = 0.0\n",
    "            for i, (images, labels) in enumerate(testloader, 0):  \n",
    "                images = images.to(device)\n",
    "                labels = labels.to(device)\n",
    "                pred = model(images)\n",
    "                v_loss = loss_func(pred, labels)\n",
    "                val_loss += v_loss.item()\n",
    "                val_acc += (pred.argmax(dim=1) == labels).float().mean().item()\n",
    "                \n",
    "            val_loss = val_loss / len(testloader)\n",
    "            val_acc = val_acc / len(testloader)\n",
    "            val_losses.append(val_loss)     \n",
    "            val_accuracy.append(val_acc)   \n",
    "\n",
    "        print(f\"Epoch: {epoch+1}/{num_epochs} | loss: {train_loss} - acc: {train_acc} | val_loss: {val_loss} - val_acc: {val_acc}\")\n",
    "    \n",
    "    print(f\"Finished training after {num_epochs} epochs\")\n",
    "    print(f\"Best validation accuracy: {max(val_accuracy)}\")\n",
    "    return train_losses, val_losses, train_accuracy, val_accuracy\n",
    "\n",
    "num_epochs = 50\n",
    "train_losses, val_losses, train_accuracy, val_accuracy = train(num_epochs, model, trainloader, testloader)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(np.array(train_losses), label='Training loss')\n",
    "plt.plot(np.array(val_losses), label='Validation loss')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(np.array(train_accuracy), label='Training Accuracy')\n",
    "plt.plot(np.array(val_accuracy), label='Validation Accuracy')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following, we will evaluate the last epoch on the test dataset and calculate the accuracy, look at some outputs and the confusion matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# predicting labels for test data and calculating accuracy\n",
    "correct = 0\n",
    "total = 0\n",
    "test_preds = []\n",
    "test_labels = []\n",
    "test_images = []\n",
    "# since we're not training, we don't need to calculate the gradients for our outputs\n",
    "with torch.no_grad():\n",
    "    for data in testloader:\n",
    "        images, labels = data\n",
    "        # calculate outputs by running images through the network\n",
    "        outputs = model(images.to(device))\n",
    "        # the class with the highest energy is what we choose as prediction\n",
    "        _, predicted = torch.max(outputs.data, 1)\n",
    "        test_preds.append(predicted.cpu())\n",
    "        test_labels.append(labels.cpu())\n",
    "        test_images.append(images.cpu())\n",
    "        total += labels.size(0)\n",
    "        correct += (predicted.cpu() == labels).sum().item()\n",
    "    test_preds = torch.cat(test_preds).numpy()\n",
    "    test_labels = torch.cat(test_labels).numpy()\n",
    "    test_images = torch.cat(test_images)\n",
    "\n",
    "print(f'Accuracy of the network on the {total} test images: {100 * correct // total} %')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "images_to_plot = 8\n",
    "\n",
    "# show images\n",
    "imshow(torchvision.utils.make_grid(test_images[:images_to_plot]))\n",
    "# print labels\n",
    "print('Truth:     ', ' '.join(f'{classes[test_labels[j]]:5s}' for j in range(images_to_plot)))\n",
    "print('Predicted: ', ' '.join(f'{classes[test_preds[j]]:5s}' for j in range(images_to_plot)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create confusion matrix and normalizes it over true labels\n",
    "confusion = confusion_matrix(test_labels, test_preds, normalize='true')\n",
    "\n",
    "# Create a plot of the confusion matrix\n",
    "fig, ax = plt.subplots(figsize=(15, 7))\n",
    "ConfusionMatrixDisplay(confusion, display_labels=classes).plot(ax=ax)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Task\n",
    "\n",
    "**Create a CNN that achieves a validation accuracy of at least 80%**\n",
    "\n",
    "- Feel free to experiment on the network structure yourself. Use a combination of convolutional and pooling layers, similar to the first exercise.\n",
    "- If your train and validation metrics diverge, you should consider including [Dropout layers](https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html).\n",
    "\n",
    "# Possible next steps\n",
    "\n",
    "- A network with many parameters may perform better but a small network is trained faster and requires less computing resources. Can you reach the 70% validation accuracy in less than 20 epochs?\n",
    "- The dataset does not only contain Cat and Dog pictures but 10 classes in total. You could rebuild the model as a multi-classifier. To achieve this, you would need to:\n",
    "    - Include additional classes in your input. Do this by commenting out the one line of code in the data loading section.\n",
    "    - What else do you need to change to make the model a multi-classifier?\n",
    "    - Try to achieve the best validation accuracy you can. Which other parameters besides the architecture can you change to affect the performance?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}