labs/arap_2020_lab03_transfer_todo.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"dlai_2020_lab07_transfer_todo.ipynb","provenance":[{"file_id":"1U1-XVGuI4tjyK2b3LszgHgZEIAP9bNCM","timestamp":1573677711864},{"file_id":"1gERmVY5BC32CE2b2q98xtRR9i66barAQ","timestamp":1573671613475},{"file_id":"1vBHRUqLNlkqwB8GijBavV8wXGjkHbMCV","timestamp":1572717190064},{"file_id":"1DBcGTSYcu2GzwfSk0oEDFTuht0787scu","timestamp":1560100314210},{"file_id":"1cJ_X7DzotOjmQ2Bb82d6NY0jh4uciF8o","timestamp":1550331701836},{"file_id":"1ZanlQigkk9GwvEcz3nV4cu5eeomtrPFN","timestamp":1550221078788},{"file_id":"1rYmI2Tv9OU6ejof1B_no155Pehck7iNM","timestamp":1538258997652}],"collapsed_sections":[]},"kernelspec":{"name":"python3","display_name":"Python 3"},"accelerator":"GPU"},"cells":[{"cell_type":"markdown","metadata":{"id":"cnlisHjUUtlM"},"source":["# Transfer Learning\n","\n","In this session we will work with convolutional neural networks, when dealing with small datasets, which is a very usual situation, as data can be difficult to obtain in certain scenearios, specially labellel data.\n","\n","Notebook created by [Daniel Fojo](https://www.linkedin.com/in/daniel-fojo/) and [Xavier Giro-i-Nieto](https://imatge.upc.edu/web/people/xavier-giro) for the [Postgraduate course in artificial intelligence with deep learning](https://www.talent.upc.edu/ing/estudis/formacio/curs/310400/postgrau-artificial-intelligence-deep-learning/) in [UPC School](https://www.talent.upc.edu/ing/) (2019).\n","Based on previous versions by [Miriam Bellver](https://imatge.upc.edu/web/people/miriam-bellver) for the ([Barcelona Technology School](https://barcelonatechnologyschool.com/) 2018).\n","\n","Related [slides](https://github.com/telecombcn-dl/dlai-2019/raw/master/slides/dlai_2019_d09l1_transfer.pdf) by [Ramon Morros](https://imatge.upc.edu/web/people/josep-ramon-morros), and [slides](https://github.com/telecombcn-dl/dlai-2019/raw/master/slides/dlai_2019_d05l2_methodology.pdf) by [Javier Ruiz](https://imatge.upc.edu/web/people/javier-ruiz-hidalgo) from [Deep Learning for Artificial Intelligence](https://telecombcn-dl.github.io/dlai-2019/) (UPC TelecomBCN 2019).\n","\n","Related video lectures by [Kevin McGuinness](https://www.youtube.com/watch?v=UKleTP1Zy1U) in ([UPC TelecomBCN 2016](http://imatge-upc.github.io/telecombcn-2016-dlcv/)), and [Ramon Morros](https://www.youtube.com/watch?v=YQvTxkPV8LQ&feature=emb_logo) in Deep Learning for Artificial Intelligence ([UPC TelecomBCN 2018](https://telecombcn-dl.github.io/2018-dlai/))\n","\n","![Daniel Fojo](https://telecombcn-dl.github.io/2018-dlcv/img/assistants/DanielFojo-160x160.jpg)\n","![Miriam Bellver](https://telecombcn-dl.github.io/2018-dlcv/img/instructors/MiriamBellver-160x160.jpg)\n","![Xavier Giro-i-Nieto](https://telecombcn-dl.github.io/2019-dlcv/img/instructors/XavierGiro-160x160.jpg)\n","![Kevin McGuinness](https://telecombcn-dl.github.io/2019-dlcv/img/instructors/Kevin160x160.jpg)\n","![Ramon Morros](https://telecombcn-dl.github.io/dlai-2019/img/instructors/RamonMorros.jpg)\n","![Javier Ruiz](https://telecombcn-dl.github.io/dlai-2019/img/instructors/JavierRuiz.jpg)\n","\n"]},{"cell_type":"code","metadata":{"id":"f1LmKoW84dra"},"source":["import torch\n","import torch.nn as nn\n","import torch.nn.functional as F\n","import torch.optim as optim\n","import numpy as np\n","import torchvision.transforms as transforms\n","from torchvision.datasets import ImageFolder\n","from torch.utils.data import DataLoader\n","from tqdm.notebook import tqdm\n","import matplotlib.pyplot as plt\n","\n","if not torch.cuda.is_available():\n","    raise Exception(\"You should enagle GPU in the Runtime menu\")\n","device = torch.device(\"cuda\")"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"QI_xjZurR-0Y"},"source":["## Downloading the database\n","\n","During this  lab session, we will work with a small database of images with dogs and cats. The cats vs. dogs dataset that we will use isn't packaged with PyTorch. It was made available by Kaggle.com as part of a computer vision competition in late 2013, back when convnets weren't quite mainstream."]},{"cell_type":"markdown","metadata":{"id":"xOn9v1DmWpYB"},"source":["The following command line will download to your remote machine the Kaggle database that we will need for this lab session. It will take a few seconds."]},{"cell_type":"code","metadata":{"id":"MvMy9CfbNy8I"},"source":["!wget --load-cookies /tmp/cookies.txt \"https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=1-tLBJFvpnz7Fo2umGbUQsJounvoyfMKt' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\\1\\n/p')&id=1-tLBJFvpnz7Fo2umGbUQsJounvoyfMKt\" -O dogs-vs-cats.zip && rm -rf /tmp/cookies.txt"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"j-qhVicIXKWa"},"source":["Now you will need to unzip the dataset that you have just downloaded, with the following line:"]},{"cell_type":"code","metadata":{"id":"iWLc6s6xOtY7"},"source":["!unzip -qq dogs-vs-cats.zip -d dogs-vs-cats/\n","!unzip -qq dogs-vs-cats/train.zip -d /datalab/"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"_CVObhON4drr"},"source":["Now you already have the raw images at your remote machine. \n","\n","## Training a convnet from scratch on a small dataset\n","\n","Having to train an image classification model using only very little data is a common situation, which you likely encounter yourself in \n","practice if you ever do computer vision in a professional context.\n","\n","Having \"few\" samples can mean anywhere from a few hundreds to a few tens of thousands of images. As a practical example, we will focus on \n","classifying images as \"dogs\" or \"cats\", in a dataset containing only 800 pictures of cats and dogs (400 cats, 400 dogs). We will use 400 \n","pictures for training and 400 for validation.\n","\n","In this section, we will review one basic strategy to tackle this problem: training a new model from scratch on what little data we have. We \n","will start by naively training a small convnet on our 400 training samples, without any regularization, to set a baseline for what can be \n","achieved. At that point, our main issue will be overfitting. Then we will introduce \n","*data augmentation*, a powerful technique for mitigating overfitting in computer vision. By leveraging data augmentation, we will improve \n","our network.\n","\n","In the last section of today's session, we will review two more essential techniques for applying deep learning to small datasets: *doing feature extraction \n","with a pre-trained network* and *fine-tuning a pre-trained network*. Together, these three strategies -- training a small model from scratch, doing feature extracting using a \n","pre-trained model, and fine-tuning a pre-trained model -- will constitute your future toolbox for tackling the problem of doing computer \n","vision with small datasets."]},{"cell_type":"markdown","metadata":{"id":"kkHfoBNx4dr2"},"source":["### The relevance of deep learning for small-data problems\n","\n","You will sometimes hear that deep learning only works when lots of data is available. This is in part a valid point: one fundamental \n","characteristic of deep learning is that it is able to find interesting features in the training data on its own, without any need for manual \n","feature engineering, and this can only be achieved when lots of training examples are available. This is especially true for problems where \n","the input samples are very high-dimensional, like images.\n","\n","However, what constitutes \"lots\" of samples is relative -- relative to the size and depth of the network you are trying to train, for \n","starters. It isn't possible to train a convnet to solve a complex problem with just a few tens of samples, but a few hundreds can \n","potentially suffice if the model is small and well-regularized and if the task is simple. \n","Because convnets learn local, translation-invariant features, they are very \n","data-efficient on perceptual problems. Training a convnet from scratch on a very small image dataset will still yield reasonable results \n","despite a relative lack of data, without the need for any custom feature engineering. You will see this in action in this section.\n","\n","But what's more, deep learning models are by nature highly repurposable: you can take, say, an image classification or speech-to-text model \n","trained on a large-scale dataset then reuse it on a significantly different problem with only minor changes. Specifically, in the case of \n","computer vision, many pre-trained models (usually trained on the ImageNet dataset) are now publicly available for download and can be used \n","to bootstrap powerful vision models out of very little data. That's what we will do in the next section.\n","\n","For now, let's get started by getting our hands on the data."]},{"cell_type":"markdown","metadata":{"id":"DDIvPB9P4dr7"},"source":["## Preparing the data\n","\n","Some sample images of the cats vs. dogs dataset look like this:\n","\n","![cats_vs_dogs_samples](https://s3.amazonaws.com/book.keras.io/img/ch5/cats_vs_dogs_samples.jpg)"]},{"cell_type":"markdown","metadata":{"id":"AikOvezq4dr9"},"source":["Unsurprisingly, the cats vs. dogs Kaggle competition in 2013 was won by entrants who used convnets. The best entries could achieve up to \n","95% accuracy. In our own example, we will get fairly close to this accuracy (in the next section), even though we will be training our \n","models on less than 5% of the data that was available to the competitors.\n","This original dataset contains 25,000 images of dogs and cats (12,500 from each class) and is 543MB large (compressed). After downloading \n","and uncompressing it, we will create a new dataset containing three subsets: a training set with 200 samples of each class and a validation \n","set with 200 samples of each class.\n","\n","Here are a few lines of code to do this:"]},{"cell_type":"code","metadata":{"id":"PPBTmOxOiYUe"},"source":["import os, shutil"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"0X2LSPKe4dsI"},"source":["# The path to the directory where the original\n","# dataset was uncompressed\n","original_dataset_dir = '/datalab/train/'\n","\n","# The directory where we will\n","# store our smaller dataset\n","base_dir = '/content/processed_datalab'\n","if not os.path.exists(base_dir):\n","    os.makedirs(base_dir)\n","\n","# Directories for our training,\n","# validation and test splits\n","train_dir = os.path.join(base_dir, 'train')\n","if not os.path.exists(train_dir):\n","    os.mkdir(train_dir)\n","validation_dir = os.path.join(base_dir, 'validation')\n","if not os.path.exists(validation_dir):\n","    os.mkdir(validation_dir)\n","test_dir = os.path.join(base_dir, 'test')\n","if not os.path.exists(test_dir):\n","    os.mkdir(test_dir)\n","\n","# Directory with our training cat pictures\n","train_cats_dir = os.path.join(train_dir, 'cats')\n","if not os.path.exists(train_cats_dir):\n","    os.mkdir(train_cats_dir)\n","\n","# Directory with our training dog pictures\n","train_dogs_dir = os.path.join(train_dir, 'dogs')\n","if not os.path.exists(train_dogs_dir):\n","    os.mkdir(train_dogs_dir)\n","\n","# Directory with our validation cat pictures\n","validation_cats_dir = os.path.join(validation_dir, 'cats')\n","if not os.path.exists(validation_cats_dir):\n","    os.mkdir(validation_cats_dir)\n","\n","# Directory with our validation dog pictures\n","validation_dogs_dir = os.path.join(validation_dir, 'dogs')\n","if not os.path.exists(validation_dogs_dir):\n","    os.mkdir(validation_dogs_dir)\n","\n","# Directory with our validation cat pictures\n","test_cats_dir = os.path.join(test_dir, 'cats')\n","if not os.path.exists(test_cats_dir):\n","    os.mkdir(test_cats_dir)\n","\n","# Directory with our validation dog pictures\n","test_dogs_dir = os.path.join(test_dir, 'dogs')\n","if not os.path.exists(test_dogs_dir):\n","    os.mkdir(test_dogs_dir)\n","\n","# Copy first 500 cat images to train_cats_dir\n","fnames = ['cat.{}.jpg'.format(i) for i in range(200)]\n","for fname in fnames:\n","    src = os.path.join(original_dataset_dir, fname)\n","    dst = os.path.join(train_cats_dir, fname)\n","    shutil.copyfile(src, dst)\n","\n","# Copy next 500 cat images to validation_cats_dir\n","fnames = ['cat.{}.jpg'.format(i) for i in range(200, 400)]\n","for fname in fnames:\n","    src = os.path.join(original_dataset_dir, fname)\n","    dst = os.path.join(validation_cats_dir, fname)\n","    shutil.copyfile(src, dst)\n","    \n","# Copy first 500 dog images to train_dogs_dir\n","fnames = ['dog.{}.jpg'.format(i) for i in range(200)]\n","for fname in fnames:\n","    src = os.path.join(original_dataset_dir, fname)\n","    dst = os.path.join(train_dogs_dir, fname)\n","    shutil.copyfile(src, dst)\n","    \n","# Copy next 500 dog images to validation_dogs_dir\n","fnames = ['dog.{}.jpg'.format(i) for i in range(200, 400)]\n","for fname in fnames:\n","    src = os.path.join(original_dataset_dir, fname)\n","    dst = os.path.join(validation_dogs_dir, fname)\n","    shutil.copyfile(src, dst)"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"6wcc0TSZ4dsQ"},"source":["As a sanity check, let's count how many pictures we have in each training split (train/validation/test):"]},{"cell_type":"code","metadata":{"id":"qfSlaL5i4dsS"},"source":["print('total training cat images:', len(os.listdir(train_cats_dir)))\n","print('total training dog images:', len(os.listdir(train_dogs_dir)))\n","print('total validation cat images:', len(os.listdir(validation_cats_dir)))\n","print('total validation dog images:', len(os.listdir(validation_dogs_dir)))"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"1QsbKDtP4dtY"},"source":["\n","So we have indeed 400 training images and 400 validation images. In each split, there is the same number of samples from each class: this is a balanced binary classification problem, which means that classification accuracy will be an appropriate measure of performance."]},{"cell_type":"markdown","metadata":{"id":"LTQ_f5Qv4dta"},"source":["\n","## Building our network\n","### Exercise 0\n","\n","We've already built a small convnet for MNIST in the previous lab session, so you should be familiar with them. We will reuse the same  general structure: our convnet will be a stack of alternated `Conv2D` (with `relu` activation) and `MaxPool2D` layers.\n","\n","However, since we are dealing with bigger images and a more complex problem, we will make our network accordingly larger: it will have one  more `Conv2D` + `MaxPool2D` stage. This serves both to augment the capacity of the network, and to further reduce the size of the  feature maps, so that they aren't overly large when we reach the `Flatten` layer. Here, since we start from inputs of size 150x150 (a  somewhat arbitrary choice), and we want to end up with feature maps of size 7x7 right before the `Flatten` layer.\n","\n","**Exercise 0:** Write the definition of the model using 2D convolutional layers with spatial kernel size 3x3 and (64,128,256,256) neurons per each of the four layers, respectively. Include a a ReLu and a 2x2 max pooling operation after each convolutional operation.  At the end, add 2 fully-connected layers with 512 hidden units before the final binary classification with a `Sigmoid` activation between cats and dogs.\n"]},{"cell_type":"code","metadata":{"id":"Iuw9n1BY4dtb"},"source":["# TODO\n","model = TODO\n","\n","model.to(device)"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"rX4J6Din4dtv"},"source":["For the optimizer, we will use Adam. Since we are dealing with binary classification, we will use binary cross entropy as our loss function."]},{"cell_type":"code","metadata":{"id":"s_49hpel4dtx"},"source":["optimizer = optim.Adam(model.parameters(), lr=0.001)\n","loss_fn = nn.BCELoss()"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"5Ga_Boci4duF"},"source":["## Data preprocessing\n","\n","Data should be formatted into appropriately pre-processed floating point tensors before being fed into our \n","network. Currently, our data sits on a drive as JPEG files, so the steps for getting it into our network are roughly:\n","\n","* Read the picture files.\n","* Decode the JPEG content to RBG grids of pixels.\n","* Resize and crop the pictures to the desired size (150x150)\n","* Convert these into floating point tensors.\n","* Rescale the pixel values (between 0 and 255) to the [-1, 1] interval (neural networks prefer to deal with small input values).\n","\n","It may seem a bit daunting, but thankfully PyTorch has utilities included in the TorchVision library to take care of these steps automatically. We will use the `ImageFolder` Dataset class, which reads images from different folders, where each folder represents a different category. We will use `torchvision.transforms` to preprocess our data."]},{"cell_type":"code","metadata":{"id":"tIkVMWvf4duG"},"source":["transform = transforms.Compose([\n","                                transforms.Resize(150), # Resize the short side of the image to 150 keeping aspect ratio\n","                                transforms.CenterCrop(150), # Crop a square in the center of the image\n","                                transforms.ToTensor(), # Convert the image to a tensor with pixels in the range [0, 1]\n","                                ])\n","batch_size = 64\n","\n","train_dataset = ImageFolder(train_dir, transform=transform)\n","train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)\n","val_dataset = ImageFolder(validation_dir, transform=transform)\n","val_loader = DataLoader(val_dataset, batch_size=batch_size, shuffle=False)\n"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"7C1ERbtA4duZ"},"source":["Let's take a look at the output of one of these generators: it yields batches of 150x150 RGB images (shape `(64, 3, 150, 150)`) and binary \n","labels (shape `(64,)`). 64 is the number of samples in each batch (the batch size)."]},{"cell_type":"code","metadata":{"id":"Sn27VHRS4dub"},"source":["for data_batch, labels_batch in train_loader:\n","    print('data batch shape:', data_batch.shape)\n","    print('labels batch shape:', labels_batch.shape)\n","    break"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"BmcmTJDq4dup"},"source":["Let's fit our model to the data using the DataLoader. "]},{"cell_type":"code","metadata":{"id":"20LMmxuk4duq"},"source":["class AverageMeter(object):\n","    \"\"\"Computes and stores the average and current value\"\"\"\n","    def __init__(self):\n","        self.reset()\n","\n","    def reset(self):\n","        self.val = 0\n","        self.avg = 0\n","        self.sum = 0\n","        self.count = 0\n","\n","    def update(self, val, n=1):\n","        self.val = val\n","        self.sum += val * n\n","        self.count += n\n","        self.avg = self.sum / self.count\n","\n","\n","def train_model(model, optimizer, loss_fn, train_loader, val_loader, epochs):\n","\n","    train_accuracies, train_losses, val_accuracies, val_losses = [], [], [], []\n","    val_loss = AverageMeter()\n","    val_accuracy = AverageMeter()\n","    train_loss = AverageMeter()\n","    train_accuracy = AverageMeter()\n","\n","    for epoch in range(epochs):\n","        # train\n","        model.train()\n","        train_loss.reset()\n","        train_accuracy.reset()\n","        train_loop = tqdm(train_loader, unit=\" batches\")  # For printing the progress bar\n","        for data, target in train_loop:\n","            train_loop.set_description('[TRAIN] Epoch {}/{}'.format(epoch + 1, epochs))\n","            data, target = data.float().to(device), target.float().to(device)\n","            target = target.unsqueeze(-1)\n","            optimizer.zero_grad()\n","            output = model(data)\n","            loss = loss_fn(output, target)\n","            loss.backward()\n","            optimizer.step()\n","\n","            train_loss.update(loss.item(), n=len(target))\n","            pred = output.round()  # get the prediction\n","            acc = pred.eq(target.view_as(pred)).sum().item()/len(target)\n","            train_accuracy.update(acc, n=len(target))\n","            train_loop.set_postfix(loss=train_loss.avg, accuracy=train_accuracy.avg)\n","\n","        train_losses.append(train_loss.avg)\n","        train_accuracies.append(train_accuracy.avg)\n","\n","        # validation\n","        model.eval()\n","        val_loss.reset()\n","        val_accuracy.reset()\n","        val_loop = tqdm(val_loader, unit=\" batches\")  # For printing the progress bar\n","        with torch.no_grad():\n","            for data, target in val_loop:\n","                val_loop.set_description('[VAL] Epoch {}/{}'.format(epoch + 1, epochs))\n","                data, target = data.float().to(device), target.float().to(device)\n","                target = target.unsqueeze(-1)\n","                output = model(data)\n","                loss = loss_fn(output, target)\n","                val_loss.update(loss.item(), n=len(target))\n","                pred = output.round()  # get the prediction\n","                acc = pred.eq(target.view_as(pred)).sum().item()/len(target)\n","                val_accuracy.update(acc, n=len(target))\n","                val_loop.set_postfix(loss=val_loss.avg, accuracy=val_accuracy.avg)\n","\n","        val_losses.append(val_loss.avg)\n","        val_accuracies.append(val_accuracy.avg)\n","        \n","    return train_accuracies, train_losses, val_accuracies, val_losses\n"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"uuX6WGmTS8iU"},"source":["epochs = 60\n","train_accuracies, train_losses, val_accuracies, val_losses = train_model(model, optimizer, loss_fn, train_loader, val_loader, epochs)"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"_nX57HF0gmM8"},"source":["Let's plot the loss and accuracy of the model over the training and validation data during training:"]},{"cell_type":"code","metadata":{"id":"1HrGpk_b4dvL"},"source":["epochs = range(len(train_accuracies))\n","\n","plt.plot(epochs, train_accuracies, 'b', label='Training acc')\n","plt.plot(epochs, val_accuracies, 'r', label='Validation acc')\n","plt.title('Training and validation accuracy')\n","plt.legend()\n","\n","plt.figure()\n","\n","plt.plot(epochs, train_losses, 'b', label='Training loss')\n","plt.plot(epochs, val_losses, 'r', label='Validation loss')\n","plt.title('Training and validation loss')\n","plt.legend()\n","\n","plt.show()"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"EjQltbgE4dvQ"},"source":["These plots are characteristic of overfitting. Our training accuracy increases linearly over time, until it reaches nearly 100%, while our \n","validation accuracy stalls at about 60%. Our validation loss reaches its minimum after only five epochs then stalls, while the training loss \n","keeps decreasing linearly until it reaches nearly 0.\n","\n","Because we only have very few training samples (400), overfitting is going to be our number one concern. You already know about a \n","number of techniques that can help mitigate overfitting, such as dropout and weight decay (L2 regularization). We are now going to \n","introduce a new one, specific to computer vision, and used almost universally when processing images with deep learning models: *data \n","augmentation*."]},{"cell_type":"markdown","metadata":{"id":"1NSgzqgj4dvR"},"source":["## 1.  Using data augmentation\n","\n","Overfitting is caused by having too few samples to learn from, rendering us unable to train a model able to generalize to new data. \n","Given infinite data, our model would be exposed to every possible aspect of the data distribution at hand: we would never overfit. Data \n","augmentation takes the approach of generating more training data from existing training samples, by \"augmenting\" the samples via a number \n","of random transformations that yield believable-looking images. The goal is that at training time, our model would never see the exact same \n","picture twice. This helps the model get exposed to more aspects of the data and generalize better.\n","\n","In PyTorch, this can be done by passaing TorchVision transformations to the Dataset. Let's get started with an example:"]},{"cell_type":"code","metadata":{"id":"hFFPyTvt4dvR"},"source":["train_transform = transforms.Compose([transforms.RandomHorizontalFlip(), transforms.RandomRotation(10), transforms.RandomResizedCrop(150), transforms.ToTensor()])\n","val_transform = transforms.Compose([transforms.Resize(150), transforms.CenterCrop(150), transforms.ToTensor()])\n","\n","batch_size = 64\n","augmented_dataset = ImageFolder(train_dir, transform=train_transform)\n","augmented_loader = DataLoader(augmented_dataset, batch_size=batch_size, shuffle=True)\n","val_dataset = ImageFolder(validation_dir, transform=val_transform)\n","val_loader = DataLoader(val_dataset, batch_size=batch_size, shuffle=False)"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"PbaKS38t4dvV"},"source":["These are just a few of the options available (for more, see in the [documentation](https://pytorch.org/docs/stable/torchvision/transforms.html)). Let's quickly go over what we just wrote:\n","\n","* `RandomRotation` is a value in degrees (0-180), a range within which to randomly rotate pictures.\n","* `RandomHorizontalFlip` is for randomly flipping half of the images horizontally -- relevant when there are no assumptions of horizontal \n","asymmetry (e.g. real-world pictures).\n","* `RandomResizedCrop` randomly resize the image and returns a random crop with a small aspect ratio distorsion. Very popular transformation for training imagenet.\n","\n","Let's take a look at our augmented images:"]},{"cell_type":"code","metadata":{"id":"rZuSjj7o4dvW"},"source":["random_indices = [0, 42, 200, 242]\n","augmented_images = [augmented_dataset[i][0] for i in random_indices]\n","augmented_images = [np.asarray(transforms.functional.to_pil_image(im)) for im in augmented_images]\n","\n","images = [train_dataset[i][0] for i in random_indices]\n","images = [np.asarray(transforms.functional.to_pil_image(im)) for im in images]\n","\n","fig, axes = plt.subplots(len(random_indices), 2, figsize=(15, 15))\n","for i in range(len(random_indices)):\n","    axes[i, 0].imshow(images[i])\n","    axes[i, 1].imshow(augmented_images[i])"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"qRNS0KLFrGxV"},"source":["**bold text**### Drop out regularization"]},{"cell_type":"markdown","metadata":{"id":"US13lC9m4dvl"},"source":["If we train a new network using this data augmentation configuration, our network will never see twice the same input. However, the inputs \n","that it sees are still heavily intercorrelated, since they come from a small number of original images -- we cannot produce new information, \n","we can only remix existing information. As such, this might not be quite enough to completely get rid of overfitting. \n","\n","\n","**Exercise 1:** To further fight  overfitting, add a [Dropout](https://pytorch.org/docs/stable/nn.html?highlight=dropout#torch.nn.Dropout) layer with probability 0.5 between the convolutional and fully connected layers."]},{"cell_type":"code","metadata":{"id":"OJwzdXoK4dvm"},"source":["# TODO\n","model = TODO\n","\n","model.to(device)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"y1QOBE_tr_53"},"source":["optimizer = optim.Adam(model.parameters(), lr=0.001)\n","loss_fn = nn.BCELoss() "],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"3VgtrZzr4dvq"},"source":["Let's train our network using data augmentation and dropout:"]},{"cell_type":"code","metadata":{"id":"9y7S2PYI4dvs"},"source":["epochs = 60\n","train_accuracies, train_losses, val_accuracies, val_losses = train_model(model, optimizer, loss_fn, augmented_loader, val_loader, epochs)\n"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"vztCCMN54dv9"},"source":["Let's save a checkpoint of our model for future references."]},{"cell_type":"code","metadata":{"id":"TnUgdwyU4dv_"},"source":["torch.save(model.state_dict(), 'cats_and_dogs_small_2_class.pt')\n","np.savez('history_cats_and_dogs_small_2_class.npz', train_accuracies=train_accuracies, train_losses=train_losses, val_accuracies=val_accuracies, val_losses=val_losses)"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"q1yZBQUILFM1"},"source":["Plot the curves with the following lines:"]},{"cell_type":"code","metadata":{"id":"CLCUDr1m4dwD"},"source":["epochs = range(len(train_accuracies))\n","\n","plt.plot(epochs, train_accuracies, 'b', label='Training acc')\n","plt.plot(epochs, val_accuracies, 'r', label='Validation acc')\n","plt.title('Training and validation accuracy')\n","plt.legend()\n","\n","plt.figure()\n","\n","plt.plot(epochs, train_losses, 'b', label='Training loss')\n","plt.plot(epochs, val_losses, 'r', label='Validation loss')\n","plt.title('Training and validation loss')\n","plt.legend()\n","\n","plt.show()"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"U01tymX_4dwK"},"source":["Thanks to data augmentation and dropout, we are no longer overfitting: the training curves are rather closely tracking the validation \n","curves. We are now able to reach an accuracy of about 66%.\n","\n","By leveraging regularization techniques even further and by tuning the network's parameters (such as the number of filters per convolution \n","layer, or the number of layers in the network), we may be able to get an even better accuracy, likely up to 70%. However, it would prove \n","very difficult to go any higher just by training our own convnet from scratch, simply because we have so little data to work with. As a \n","next step to improve our accuracy on this problem, we will have to leverage a pre-trained model, which will be the focus of the next two \n","sections."]},{"cell_type":"markdown","metadata":{"id":"Qh0ffGJra2oP"},"source":["# 2 Using a pre-trained convnet\n","\n","A common and highly effective approach to deep learning on small image datasets is to leverage a pre-trained network. A pre-trained network is simply a saved network previously trained on a large dataset, typically on a large-scale image classification task. If this original dataset is large enough and general enough, then the spatial feature hierarchy learned by the pre-trained network can effectively act as a generic model of our visual world, and hence its features can prove useful for many different computer vision problems, even though these new problems might involve completely different classes from those of the original task. For instance, one might train a network on ImageNet (where classes are mostly animals and everyday objects) and then re-purpose this trained network for something as remote as identifying furniture items in images. Such portability of learned features across different problems is a key advantage of deep learning compared to many older shallow learning approaches, and it makes deep learning very effective for small-data problems.\n","\n","In our case, we will consider a large convnet trained on the ImageNet dataset (1.4 million labeled images and 1000 different classes). ImageNet contains many animal classes, including different species of cats and dogs, and we can thus expect to perform very well on our cat vs. dog classification problem.\n","\n","We will use the VGG16 architecture, developed by Karen Simonyan and Andrew Zisserman in 2014, a simple and widely used convnet architecture for ImageNet. Although it is a bit of an older model, far from the current state of the art and somewhat heavier than many other recent models, we chose it because its architecture is similar to what you are already familiar with, and easy to understand without introducing any new concepts. \n","\n","There are two ways to leverage a pre-trained network: feature extraction and fine-tuning. We will cover both of them. Let's start with feature extraction. Feature extraction consists of using the representations learned by a previous network to extract interesting features from new samples. These features are then run through a new classifier, which is trained from scratch.\n","\n","As we saw previously, convnets used for image classification comprise two parts: they start with a series of pooling and convolution layers, and they end with a densely-connected classifier. The first part is called the \"convolutional base\" of the model. In the case of convnets, \"feature extraction\" will simply consist of taking the convolutional base of a previously-trained network, running the new data through it, and training a new classifier on top of the output.\n","\n","\n","![settings](https://camo.githubusercontent.com/2de162f6f3587a0422389355909594413ae17742/68747470733a2f2f73332e616d617a6f6e6177732e636f6d2f626f6f6b2e6b657261732e696f2f696d672f6368352f7377617070696e675f66635f636c61737369666965722e706e67)\n","\n","Why only reuse the convolutional base? Could we reuse the densely-connected classifier as well? In general, it should be avoided. The reason is simply that the representations learned by the convolutional base are likely to be more generic and therefore more reusable: the feature maps of a convnet are presence maps of generic concepts over a picture, which is likely to be useful regardless of the computer vision problem at hand. On the other end, the representations learned by the classifier will necessarily be very specific to the set of classes that the model was trained on -- they will only contain information about the presence probability of this or that class in the entire picture. Additionally, representations found in densely-connected layers no longer contain any information about where objects are located in the input image: these layers get rid of the notion of space, whereas the object location is still described by convolutional feature maps. For problems where object location matters, densely-connected features would be largely useless.\n","\n","Note that the level of generality (and therefore reusability) of the representations extracted by specific convolution layers depends on the depth of the layer in the model. Layers that come earlier in the model extract local, highly generic feature maps (such as visual edges, colors, and textures), while layers higher-up extract more abstract concepts (such as \"cat ear\" or \"dog eye\"). So if your new dataset differs a lot from the dataset that the original model was trained on, you may be better off using only the first few layers of the model to do feature extraction, rather than using the entire convolutional base.\n","\n","In our case, since the ImageNet class set did contain multiple dog and cat classes, it is likely that it would be beneficial to reuse the information contained in the densely-connected layers of the original model. However, we will chose not to, in order to cover the more general case where the class set of the new problem does not overlap with the class set of the original model.\n","\n","Let's put this in practice by using the convolutional base of the VGG16 network, trained on ImageNet, to extract interesting features from our cat and dog images, and then training a cat vs. dog classifier on top of these features.\n","\n","The VGG16 model, among others, comes pre-packaged with PyTorch. You can import it from the torchvision.models module.\n"]},{"cell_type":"code","metadata":{"id":"DZXUflM5bawo"},"source":["from torchvision.models import vgg16\n","\n","pretrained_model = vgg16(pretrained=True)\n","pretrained_model.eval()\n","pretrained_model.to(device)"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"5AOoq6n6aUeo"},"source":["The following line takes the first part of the model, to use it as a feature_extractor."]},{"cell_type":"code","metadata":{"id":"LnDLUwx8acug"},"source":["feature_extractor = pretrained_model.features\n","feature_extractor"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"3am0EQmKb-Es"},"source":["\n","\n","The final feature map has shape (512, 4, 4). That's the feature on top of which we will stick a densely-connected classifier.\n","\n","The method we will use consists in running the convolutional base over our dataset, recording its output to a Numpy array on disk, then using this data as input to a standalone densely-connected classifier. This solution is very fast and cheap to run, because it only requires running the convolutional base once for every input image, and the convolutional base is by far the most expensive part of the pipeline. However, for the exact same reason, this technique would not allow us to leverage data augmentation at all.\n","\n","\n","We will start by simply running instances of the previously-introduced ImageFolder Dataset to extract features from these images.\n"]},{"cell_type":"code","metadata":{"id":"V8p5xtJXb8Mq"},"source":["base_dir = '/content/processed_datalab'\n","\n","train_dir = os.path.join(base_dir, 'train')\n","validation_dir = os.path.join(base_dir, 'validation')\n","\n","\n","def extract_features(directory):\n","    transform = transforms.Compose([transforms.Resize(150), transforms.CenterCrop(150), transforms.ToTensor(), transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])])  # Normalize with Imagenet parameters\n","    dataset = ImageFolder(directory, transform=transform)\n","    batch_size = 100\n","    loader = DataLoader(dataset, batch_size=batch_size, shuffle=False)\n","    features = np.zeros(shape=(len(dataset), 512, 4, 4))\n","    labels = np.zeros(shape=(len(dataset),))\n","    with torch.no_grad():\n","        for i, (inputs_batch, labels_batch) in enumerate(loader):\n","            inputs_batch, labels_batch = inputs_batch.to(device), labels_batch.to(device)\n","            features_batch = feature_extractor(inputs_batch)\n","            features[i * batch_size : (i + 1) * batch_size] = features_batch.cpu().numpy()\n","            labels[i * batch_size : (i + 1) * batch_size] = labels_batch.cpu().numpy()\n","\n","    return features, labels\n","\n","train_features, train_labels = extract_features(train_dir)\n","validation_features, validation_labels = extract_features(validation_dir)\n"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"yFCd6TO9cDPV"},"source":["The extracted features are currently of shape (samples, 512, 4, 4). We will feed them to a densely-connected classifier, so first we must flatten them to (samples, 8192):"]},{"cell_type":"code","metadata":{"id":"Q8ZjKJrgcEJo"},"source":["train_features = np.reshape(train_features, (-1, 4 * 4 * 512))\n","validation_features = np.reshape(validation_features, (-1, 4 * 4 * 512))"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"_eVlyjpFczgx"},"source":["### Exercise 2\n","At this point, we can define a neural network with a single hiden layer trained independetly with the visual features extracted from the VGG model. \n","\n","**Exercise 2:** Implement this simple neural network with a 256 hidden neurons with a ReLU activation, and a dropout factor of 0.5 for both sets of parameters (input > hidden, hidden > output)."]},{"cell_type":"code","metadata":{"id":"O_z54_9Gc2Px"},"source":["# TODO\n","feature_classifier = TODO\n","\n","feature_classifier.to(device)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"MYwxc7b0TcJw"},"source":["optimizer = optim.Adam(feature_classifier.parameters(), lr=0.001)\n","loss_fn = nn.BCELoss()"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"w33pDKojX5Nl"},"source":["from torch.utils.data import TensorDataset\n","\n","batch_size = 64\n","train_features_dataset = TensorDataset(torch.tensor(train_features), torch.tensor(train_labels))\n","train_features_loader = DataLoader(train_features_dataset, batch_size=batch_size, shuffle=True)\n","\n","val_features_dataset = TensorDataset(torch.tensor(validation_features), torch.tensor(validation_labels))\n","val_features_loader = DataLoader(val_features_dataset, batch_size=batch_size, shuffle=False)\n"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"dLE7AxMTYYft"},"source":["epochs = 10\n","train_accuracies, train_losses, val_accuracies, val_losses = train_model(feature_classifier, optimizer, loss_fn, train_features_loader, val_features_loader, epochs)\n"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"wOhod1-BeO2k"},"source":["Training is very fast, since we only have to deal with two Dense layers -- an epoch takes less than one second even on CPU.\n","\n","Let's take a look at the loss and accuracy curves during training:\n"]},{"cell_type":"code","metadata":{"id":"QDzaOmvxfxUW"},"source":["epochs = range(len(train_accuracies))\n","\n","plt.plot(epochs, train_accuracies, 'b', label='Training acc')\n","plt.plot(epochs, val_accuracies, 'r', label='Validation acc')\n","plt.title('Training and validation accuracy')\n","plt.legend()\n","\n","plt.figure()\n","\n","plt.plot(epochs, train_losses, 'b', label='Training loss')\n","plt.plot(epochs, val_losses, 'r', label='Validation loss')\n","plt.title('Training and validation loss')\n","plt.legend()\n","\n","plt.show()"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"VmLl33Y3gB4Y"},"source":["\n","\n","We reach a validation accuracy of about 96%, much better than what we could achieve in the previous section with our small model trained from scratch. However, our plots also indicate that we are overfitting almost from the start -- despite using dropout with a fairly large rate. This is because this technique does not leverage data augmentation, which is essential to preventing overfitting with small image datasets.\n","\n"]},{"cell_type":"markdown","metadata":{"id":"Z7m850N1iGYN"},"source":["# 3. Fine-tuning\n","\n","Another widely used technique for model reuse, complementary to feature extraction, is fine-tuning. Fine-tuning consists in unfreezing a few of the top layers of a frozen model base used for feature extraction, and jointly training both the newly added part of the model (in our case, the fully-connected classifier) and these top layers. This is called \"fine-tuning\" because it slightly adjusts the more abstract representations of the model being reused, in order to make them more relevant for the problem at hand.\n","\n","![finetuning](https://camo.githubusercontent.com/28a6cb7048317aba34f39cb02f15520708189008/68747470733a2f2f73332e616d617a6f6e6177732e636f6d2f626f6f6b2e6b657261732e696f2f696d672f6368352f76676731365f66696e655f74756e696e672e706e67)"]},{"cell_type":"markdown","metadata":{"id":"tQYNSVvHihVS"},"source":["We have stated before that it was necessary to freeze the convolution base of VGG16 in order to be able to train a randomly initialized classifier on top. For the same reason, it is only possible to fine-tune the top layers of the convolutional base once the classifier on top has already been trained. If the classified wasn't already trained, then the error signal propagating through the network during training would be too large, and the representations previously learned by the layers being fine-tuned would be destroyed. Thus the steps for fine-tuning a network are as follow:\n","\n","\n","1.   Add your custom network on top of an already trained base network.\n","2.   Freeze the base network.\n","3.   Train the part you added.\n","4.   Unfreeze some layers in the base network.\n","5.   Jointly train both these layers and the part you added.\n","\n","We have already completed the first 3 steps when doing feature extraction. Let's proceed with the 4th step.\n","\n","As a reminder, this is what our convolutional base looks like:\n"]},{"cell_type":"code","metadata":{"id":"eg0LQQbDjVID"},"source":["pretrained_model = vgg16(pretrained=True)\n","pretrained_model.to(device)\n","feature_extractor = pretrained_model.features\n","feature_extractor"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"FSIHCK_Vjj21"},"source":["for layer in feature_extractor[:24]:  # Freeze layers 0 to 23\n","    for param in layer.parameters():\n","        param.requires_grad = False\n","\n","for layer in feature_extractor[24:]:  # Train layers 24 to 30\n","    for param in layer.parameters():\n","        param.requires_grad = True"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"OeRNge_VjgnY"},"source":["We will fine-tune the last 3 convolutional layers.\n","\n","Why not fine-tune more layers? Why not fine-tune the entire convolutional base? We could. However, we need to consider that:\n","\n","1.   Earlier layers in the convolutional base encode more generic, reusable features, while layers higher up encode more specialized features. It is more useful to fine-tune the more specialized features, as these are the ones that need to be repurposed on our new problem. There would be fast-decreasing returns in fine-tuning lower layers.\n","2.   The more parameters we are training, the more we are at risk of overfitting. The convolutional base has 15M parameters, so it would be risky to attempt to train it on our small dataset.\n","\n","Thus, in our situation, it is a good strategy to only fine-tune the top 2 to 3 layers in the convolutional base.\n","\n","Let's set this up, starting from where we left off in the previous example:\n"]},{"cell_type":"markdown","metadata":{"id":"onAdiWY4k387"},"source":["**Exercise 3:** Create a model with the `feature_classifier` of top of the `feature_extrator`. Note that you will need to flatten the features. "]},{"cell_type":"code","metadata":{"id":"ONSyBBUdmQc0"},"source":["# TODO\n","model = TODO\n","\n","model.to(device)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"bgg6TahImU_D"},"source":[""],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"s4hU2suDkNsR"},"source":["Now we can start fine-tuning our network. We will do this with the Adam optimizer, using a lower learning rate. The reason for using a low learning rate is that we want to limit the magnitude of the modifications we make to the representations of the 3 layers that we are fine-tuning. Updates that are too large may harm these representations.\n","\n","Now let's proceed with fine-tuning:\n"]},{"cell_type":"markdown","metadata":{"id":"cR5MccYQmeKS"},"source":["In this case, we can do data augmentation, with the same configuration we used in our previous example:"]},{"cell_type":"code","metadata":{"id":"eTzJvmxYmjDW"},"source":["train_transform = transforms.Compose([transforms.RandomHorizontalFlip(), transforms.RandomRotation(10), transforms.RandomResizedCrop(150), transforms.ToTensor(), transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])])  # We have to add ImageNet normalization\n","val_transform = transforms.Compose([transforms.Resize(150), transforms.CenterCrop(150), transforms.ToTensor(), transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])])\n","\n","batch_size = 64\n","augmented_dataset = ImageFolder(train_dir, transform=train_transform)\n","augmented_loader = DataLoader(augmented_dataset, batch_size=batch_size, shuffle=True)\n","val_dataset = ImageFolder(validation_dir, transform=val_transform)\n","val_loader = DataLoader(val_dataset, batch_size=batch_size, shuffle=False)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"DhMFf_L6Y4Ve"},"source":["optimizer = optim.Adam(model.parameters(), lr=1e-4)\n","loss_fn = nn.BCELoss()"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"xydsaVGikSZP"},"source":["epochs = 30\n","train_accuracies, train_losses, val_accuracies, val_losses = train_model(model, optimizer, loss_fn, augmented_loader, val_loader, epochs)\n"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"r0gEUh5HoG2T"},"source":["And finally plot the curves: "]},{"cell_type":"code","metadata":{"id":"bEq7fonOoIo7"},"source":["epochs = range(len(train_accuracies))\n","\n","plt.plot(epochs, train_accuracies, 'b', label='Training acc')\n","plt.plot(epochs, val_accuracies, 'r', label='Validation acc')\n","plt.title('Training and validation accuracy')\n","plt.legend()\n","\n","plt.figure()\n","\n","plt.plot(epochs, train_losses, 'b', label='Training loss')\n","plt.plot(epochs, val_losses, 'r', label='Validation loss')\n","plt.title('Training and validation loss')\n","plt.legend()\n","\n","plt.show()"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"VTk9U7-9dYe8"},"source":["Here we get a validation accuracy of 98%. In the original Kaggle competition around this dataset, this would have been one of the top results. However, using modern deep learning techniques, we managed to reach this result using only a very small fraction of the training data available (only about 2% of the data). There is a huge difference between being able to train on 20,000 samples compared to 400 samples!\n","\n","# Take-aways: using convnets with small datasets\n","\n","Here's what you should take away from these exercises:\n","\n","\n","*   Convnets are the best type of machine learning models for computer vision tasks. It is possible to train one from scratch even on a very small dataset, with decent results.\n","*   On a small dataset, overfitting will be the main issue. Data augmentation is a powerful way to fight overfitting when working with image data.\n","*    It is easy to reuse an existing convnet on a new dataset, via feature extraction. This is a very valuable technique for working with small image datasets.\n","*   As a complement to feature extraction, one may use fine-tuning, which adapts to a new problem some of the representations previously learned by an existing model. This pushes performance a bit further.\n","\n","Now you have a solid set of tools for dealing with image classification problems, in particular with small datasets."]}]}