bark-texture-classification-with-resnet50.ipynb

{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Changes\n### Most changes between versions are small. All experiments should have only one small change each, so it would be easy to understand how changes affect the result. \n\n**v1**: Made a functional CNN with 94% accuracy.\n\n**v2**: Resizing to 256 px before cropping. This way we can retain more of the image after the crop, and this helps us too as most of the images are actually focused and centered on the trees. Accuracy boosted to 96%.\n\n**v3**: 30 epochs has been overfitting the model over the validation data. Reduced to 15 epochs. Retains the original 96-97% accuracy as compared to 30 epochs\n\n**v4**: Added L2 regularization while training and used updated model weights for ResNet50 using new API\n\n**v5**: Increased number of epochs to 45 due to new lr decay\n\n**v6**: Moved from ResNet50 to ResNet34, with 1% performance change in testing, but considerable reduction in model execution (training, testing) time. \n\n**v7**: Added AutoAugment and moved back to Resnet50. Creates marginal performance improvement.\n\n**Accuracies**:\n\n - *ResNet50* - 97%\n\n - *ResNet34* - 96%\n \n We can choose ResNet50 for accuracy ResNet34 for speed","metadata":{}},{"cell_type":"markdown","source":"### Transfer Learning\n\nHere I decided to use a transfer learning model, as using a base model did not quite meet my personal expectations on the accuracy of the model.\n\nTransfer Learning allows us to apply features already learned from other datasets to our dataset, leading to lower training times and higher accuracies. Here we use models trained on the IMAGENET dataset, which work very well for most CNN image classification tasks.","metadata":{}},{"cell_type":"code","source":"import numpy as np\nfrom glob import glob\nimport os\nimport torch\nimport torchvision.models as models\nimport torchvision.transforms as transforms\nfrom torchvision import datasets\nfrom PIL import ImageFile\nfrom torch.utils.data.sampler import SubsetRandomSampler\nimport matplotlib.pyplot as plt\nimport torch.optim as optim\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Creating our transforms**:\n- Cropping to 224 x 224 px for transfer learning model input param. This reduction of the size of the images also helps decrease processing time\n- We augumented the dataset to add more training variations by horizontal flipping and slight rotation. \n- **v2**: Resizing to 256 px before cropping. This way we can retain more of the image after the crop, and this helps us too as most of the images are actually focused and centered on the trees. Accuracy boosted to 96%.\n- **v7**: Added AutoAugment, a data augmentation strategy based on the paper [AutoAugment: Learning Augmentation Strategies from Data](https://arxiv.org/pdf/1805.09501.pdf)","metadata":{}},{"cell_type":"code","source":"import torchvision.transforms as transforms\n\n# Declare the transforms for train, valid and test sets.\n# Convert to Tensor\n# Normalize images because the values of images should be loaded between [0 - 1]\ntransforms = {\n    \n    # RandomHorizontalFlip() & RandomRotation() to augement data in train transformation\n    'train' : transforms.Compose([transforms.Resize(256),\n                                  transforms.AutoAugment(transforms.AutoAugmentPolicy.IMAGENET),\n                                  transforms.RandomHorizontalFlip(),\n                                  transforms.RandomRotation(30),\n                                  transforms.CenterCrop(224),\n                                  transforms.ToTensor(),\n                                  transforms.Normalize(mean=[0.485, 0.456, 0.406],\n                                                       std=[0.229, 0.224, 0.225])]),\n    \n    'valid' : transforms.Compose([transforms.Resize(256),\n                                  transforms.CenterCrop(224),\n                                  transforms.ToTensor(),\n                                  transforms.Normalize(mean=[0.485, 0.456, 0.406],\n                                                       std=[0.229, 0.224, 0.225])]),\n    \n    'test' : transforms.Compose([transforms.Resize(256),\n                                 transforms.CenterCrop(224),\n                                 transforms.ToTensor(),\n                                 transforms.Normalize(mean=[0.485, 0.456, 0.406],\n                                                      std=[0.229, 0.224, 0.225])])\n}\n\n# number of subprocesses to use for data loading\nnum_workers = 2\n# how many samples per batch to load\nbatch_size = 20\n# where our data is stored\ndata_dir = \"./BarkVN-50_mendeley\"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Data Exploration\n\nWe plot our data distribution and observe that our data has 50 classes/directories with roughly 100 images in each directory.","metadata":{}},{"cell_type":"code","source":"x_plt = []\ny_plt = []\nfor directory in os.listdir(data_dir):\n    x_plt.append(directory)\n    y_plt.append(len(os.listdir(os.path.join(data_dir, directory))))\n\n# creating the bar plot\nfig, ax = plt.subplots(figsize =(16, 16))\nplt.barh(x_plt, y_plt, color ='maroon')\n# Remove x, y Ticks\nax.xaxis.set_ticks_position('none')\nax.yaxis.set_ticks_position('none')\n \n# Add padding between axes and labels\nax.xaxis.set_tick_params(pad = 5)\nax.yaxis.set_tick_params(pad = 10)\n \n# Show top values\nax.invert_yaxis()\n\n\nplt.ylabel(\"Bark Type\")\nplt.xlabel(\"No. of images\")\nplt.title(\"Bark Texture Dataset\")\nplt.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Based on this, we allocated 10% testing, 9% to validation and 81% to training.\n\nWe need to be careful of overfitting since this is a small dataset and hence during training, we will save the model only when the validation accuracy goes down. ","metadata":{}},{"cell_type":"code","source":"# how much % we want to allocate for testing\ntest_size=0.1\n# how much % of the remaining train data we want to allocate for validation\nvalid_size=0.1\n\n# load the dataset\ntrain_dataset = datasets.ImageFolder(\n    root=data_dir, transform=transforms[\"train\"],\n)\n\nvalid_dataset = datasets.ImageFolder(\n    root=data_dir, transform=transforms[\"valid\"],\n)\n\ntest_dataset = datasets.ImageFolder(\n    root=data_dir, transform=transforms[\"test\"],\n)\n\nnum_train = len(train_dataset)\nindices = list(range(num_train))\n\n# Splitting test into test and train\nsplit = int(np.floor(test_size * num_train))\n\nnp.random.shuffle(indices)\n\ntrain_idx, test_idx = indices[split:], indices[:split]\ntrain_sampler = SubsetRandomSampler(train_idx)\ntest_sampler = SubsetRandomSampler(test_idx)\n\n# Splitting remaining test into test and validation\nsplit = int(np.floor(valid_size * num_train))\n\nnp.random.shuffle(indices)\n\ntrain_idx, valid_idx = indices[split:], indices[:split]\ntrain_sampler = SubsetRandomSampler(train_idx)\nvalid_sampler = SubsetRandomSampler(valid_idx)\n\n# Creating dataloaders\ntrain_loader = torch.utils.data.DataLoader(\n    train_dataset, batch_size=batch_size, sampler=train_sampler,num_workers=num_workers\n)\nvalid_loader = torch.utils.data.DataLoader(\n    valid_dataset, batch_size=batch_size, sampler=valid_sampler, num_workers=num_workers\n)\ntest_loader = torch.utils.data.DataLoader(\n    test_dataset, batch_size=batch_size, sampler=test_sampler, num_workers=num_workers\n)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Here we can group the dataloaders into a single object to pass it into our train/test functions as a single object","metadata":{}},{"cell_type":"code","source":"dataloaders = {\n    \"train\": train_loader,\n    \"valid\": valid_loader,\n    \"test\": test_loader\n}","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get the all labels\nclass_names = train_dataset.classes\nprint(class_names)\nlen(class_names)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Testing to see if the dataloaders work correctly","metadata":{}},{"cell_type":"code","source":"from torchvision import utils\n\ndef visualize_sample_images(inp):\n    inp = inp.numpy().transpose((1, 2, 0))\n    inp = inp * np.array((0.229, 0.224, 0.225)) + np.array((0.485, 0.456, 0.406))\n    inp = np.clip(inp, 0, 1)\n    \n    fig = plt.figure(figsize=(60, 25))\n    plt.axis('off')\n    plt.imshow(inp)\n    plt.pause(0.001)\n    \n# Get a batch of training data.    \ninputs, classes = next(iter(train_loader))\n\n# Convert the batch to a grid.\ngrid = utils.make_grid(inputs, nrow=5)\n\n# Display!\nvisualize_sample_images(grid)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Model Architecture\n\nThe ResNet model had been chosen as in all first tests, resnet performed better compared to candidates like efficientnet, alexnet and a custom neural network\n\nSmaller datasets tend to do well with less complex networks and it shows here. Resnet50 works better than ResNet101 or ResNet152\n\n**v4**: Used updated model weights for ResNet50 using new API\n\n**v6**: Moved from ResNet50 to ResNet34, with 1% performance change in testing, but considerable reduction in model execution (training, testing) time. ","metadata":{}},{"cell_type":"code","source":"import torchvision.models as models\nfrom torchvision.models import resnet34, ResNet34_Weights\nfrom torchvision.models import resnet50, ResNet50_Weights\nimport torch.nn as nn\n\n## TODO: Specify model architecture \n#model_transfer = models.resnet34(weights=ResNet34_Weights.DEFAULT)\nmodel_transfer = models.resnet50(weights=ResNet50_Weights.DEFAULT)\n\n# Freeze parameters so we don't backprop through them\nfor param in model_transfer.parameters():\n    param.requires_grad = False\n\n# Replace the last fully connected layer with a Linnear layer with 50 out features\nmodel_transfer.fc = nn.Linear(2048, len(class_names)) # Resnet 50+\n#model_transfer.fc = nn.Linear(512, len(class_names)) # Resnet18,34\n\nuse_cuda = torch.cuda.is_available()\n\nif use_cuda:\n    model_transfer = model_transfer.cuda()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Model Training Optimization Algorithms\n\nThe optimizers are responsible for the feedforwand and backpropagation mechanisms. In more appropriate words, they will hold the current state and will update the parameters based on the computed gradients.\n\n**v4**: Added L2 regularization while training.\n\nThe Adam optimizer in pytorch does not implement true weight decay but rather uses L2 Regularization. It is mysteriously added in the Optimization functions because loss functions are used during Optimization. More has been discussed in this paper: [Fixing Weight Decay Regularization in Adam](https://openreview.net/pdf?id=rk6qdGgCZ)","metadata":{}},{"cell_type":"code","source":"criterion = nn.CrossEntropyLoss()\noptimizer = optim.Adam(model_transfer.fc.parameters(), lr=0.001, weight_decay=5e-5)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Training our model\n\nAfter obtaining the data, creating the model architecture, definiting optimization algorithm we can finally begin our training. Here we only save the model every time the validation loss goes down in order to make sure we don't have an overfitted model. If the validation loss stops decreasing for a few epochs, we can know that we have started overfitting and can accordingly lower the training epochs. Conversely, if it keeps decreasing, we can increase the training epochs. \n\nAn important thing to keep in mind is to make sure during training, our model is in training mode by using `model.train()` and during evaluation, it is in evaluation mode by using `model.eval()`. This ensures that during a feedforward operation, the gradients/weights are not updated, if we are currently just evaluating the model on our validation data. And conversely, the gradients/weights *are* updated if we wish to train our parameters.\n\nHere I perform the training on my RTX 2060 Mobile GPU.\n\n**v3**: 30 epochs has been overfitting the model over the validation data. Reduced to 15 epochs. Retains the original 96-97% accuracy as compared to 30 epochs\n\n**v5**: Increased number of epochs to 45 due to new lr decay (L2 regularization)","metadata":{}},{"cell_type":"code","source":"def train(n_epochs, loaders, model, optimizer, criterion, use_cuda, save_path):\n    \"\"\"returns trained model\"\"\"\n    # initialize tracker for minimum validation loss\n    valid_loss_min = np.Inf \n    \n    for epoch in range(1, n_epochs+1):\n        # initialize variables to monitor training and validation loss\n        train_loss = 0.0\n        valid_loss = 0.0\n        \n        ###################\n        # train the model #\n        ###################\n        model.train()\n        for batch_idx, (data, target) in enumerate(loaders['train']):\n            # move to GPU\n            if use_cuda:\n                data, target = data.cuda(), target.cuda()\n            ## find the loss and update the model parameters accordingly\n            ## record the average training loss, using something like\n            ## train_loss = train_loss + ((1 / (batch_idx + 1)) * (loss.data - train_loss))\n            \n            # clear the gradients of all optimized variables, initialize weights to zero\n            optimizer.zero_grad()\n            # forward pass\n            output = model(data)\n            # calculate batch loss\n            loss = criterion(output, target)\n            # backward pass\n            loss.backward()\n            # parameter update\n            optimizer.step()\n            # update training loss\n            train_loss += loss.item() * data.size(0)\n            \n        ######################    \n        # validate the model #\n        ######################\n        model.eval()\n        for batch_idx, (data, target) in enumerate(loaders['valid']):\n            # move to GPU\n            if use_cuda:\n                data, target = data.cuda(), target.cuda()\n            ## update the average validation loss\n            # forward pass\n            output = model(data)\n            # batch loss\n            loss = criterion(output, target)\n            # update validation loss\n            valid_loss += loss.item() * data.size(0)\n            \n        # calculate average losses\n        train_loss = train_loss/len(loaders['train'].dataset)\n        valid_loss = valid_loss/len(loaders['valid'].dataset)\n            \n        # print training/validation statistics \n        print('Epoch: {} \\tTraining Loss: {:.6f} \\tValidation Loss: {:.6f}'.format(\n            epoch, \n            train_loss,\n            valid_loss\n            ))\n        \n        ## TODO: save the model if validation loss has decreased\n        if valid_loss <= valid_loss_min:\n            print('Validation loss decreased ({:.6f} --> {:.6f}).    Saving model...'.\n                 format(valid_loss_min, valid_loss))\n            torch.save(model.state_dict(), save_path)\n            valid_loss_min = valid_loss\n            \n    # return trained model\n    return model\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# train the model\nmodel_transfer =  train(45, dataloaders, model_transfer, optimizer, criterion, use_cuda, 'model_transfer.pt')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Testing the model\n\nOnce we finish training, we can finally test our model to see how well it will perform with data it hasn't seen before. It is a fairly simple process, and we get a final accuracy of 97%","metadata":{}},{"cell_type":"code","source":"def test(loaders, model, criterion, use_cuda):\n\n    # monitor test loss and accuracy\n    test_loss = 0.\n    correct = 0.\n    total = 0.\n\n    model.eval()\n    for batch_idx, (data, target) in enumerate(loaders['test']):\n        # move to GPU\n        if use_cuda:\n            data, target = data.cuda(), target.cuda()\n        # forward pass: compute predicted outputs by passing inputs to the model\n        output = model(data)\n        # calculate the loss\n        loss = criterion(output, target)\n        # update average test loss \n        test_loss = test_loss + ((1 / (batch_idx + 1)) * (loss.data - test_loss))\n        # convert output probabilities to predicted class\n        pred = output.data.max(1, keepdim=True)[1]\n        # compare predictions to true label\n        correct += np.sum(np.squeeze(pred.eq(target.data.view_as(pred))).cpu().numpy())\n        total += data.size(0)\n            \n    print('Test Loss: {:.6f}\\n'.format(test_loss))\n\n    print('\\nTest Accuracy: %2d%% (%2d/%2d)' % (\n        100. * correct / total, correct, total))\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test(dataloaders, model_transfer, criterion, use_cuda)","metadata":{},"execution_count":null,"outputs":[]}]}