torch_brain_tumor_classifier.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "brain_tumor_classifier_resnet_torch.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "43754Epan7bx",
        "colab_type": "text"
      },
      "source": [
        "# Brain Tumor Classification\n",
        "\n",
        "A Brain Tumor Classifier using fine-tuned ResNet50 Neural Network architecture with almost 99 % Accuracy achieved by applying the method of Transfer Learning.\n",
        "\n",
        "Given an MRI image of brain, classify the tumor into **Meningioma, Glioma, and Pitutary**, if present. \n",
        "\n",
        "Project done using Google Colab with follwing specifications:\n",
        "\n",
        "* Ubuntu 18.04 64-bit OS\n",
        "* 12 GB DDR4 RAM\n",
        "* 16 GB NVidia P100 GPU\n",
        "* 40 GB of Non-Persistent Storage\n",
        "\n",
        "Refer [brain_tumor_dataset_preparation.ipynb](brain_tumor_dataset_preparation.ipynb) for dataset preparation & pre-processing that we did."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s4k9AZlhoVgg",
        "colab_type": "text"
      },
      "source": [
        "## Import Necessary Libraries\n",
        "\n",
        "We'll be using the following libraries to complete our classification problem:\n",
        "\n",
        "* Numpy - For linear algebra operations \n",
        "* Torch - Pytorch Deep Learning Framework\n",
        "* Torch NN - Neural network class from Pytorch library\n",
        "* Torch NN Functional - Functional Neural Network class from Pytorch library\n",
        "* Torch Utils Data: DataLoader, Dataset - Dataset class used to create custom dataset class by subclassing it and DataLoader is used to laod data in batches using dataset class in real-time.\n",
        "* Torchvision: Transforms, Models - Trochvision provides augmentation techniques using transforms class and transfer learning models are available in models class\n",
        "* OS - To use Operating System methods\n",
        "* Random - To set random seed at specific places where random operations take place just so it happens the same way everytime it is executed\n",
        "* Pandas - To create DataFrame, CSV files, etc\n",
        "* Time - To perform date time operations\n",
        "* Seaborn - For sophisticated visualization\n",
        "* Pickle - To save and load binary files of our training data\n",
        "* Scikit-Learn - For evaluating our Classifier and for cross-validation split\n",
        "* Matplotlib - To visualize images, losses and accuracy\n",
        "* Google Colab Drive - To mount Google Drive so we can perform storage and loading operations using it\n",
        "\n",
        "To install the requirements, refer [requirements.txt](requirements.txt) file\n",
        "\n",
        " `pip3 install -r requirements.txt`\n",
        "                      \n",
        "Let's go ahead and import the required libraries"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ppY0E2oBcXNu",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        },
        "outputId": "523f44ab-6260-44fd-8e22-1ec57ddd7605"
      },
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "from torch.utils.data import DataLoader, Dataset\n",
        "from torchvision import transforms, models\n",
        "from torchvision.utils import make_grid\n",
        "import os\n",
        "import random\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import pickle\n",
        "import time\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.metrics import confusion_matrix, classification_report, jaccard_similarity_score\n",
        "from google.colab import drive "
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/statsmodels/tools/_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.\n",
            "  import pandas.util.testing as tm\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YyPnviwkoaRh",
        "colab_type": "text"
      },
      "source": [
        "Print Pytorch's version"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uY8S0R76sj90",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "012126c7-b199-473a-d7c3-88a22c6eeb3b"
      },
      "source": [
        "torch.__version__"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'1.5.0+cu101'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8uWrMvwCbmsw",
        "colab_type": "text"
      },
      "source": [
        "Check GPU"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4NKtY3ESbmSl",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 372
        },
        "outputId": "e54e9295-b586-4ee6-ff51-70e63e9dd854"
      },
      "source": [
        "!nvidia-smi"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Sat Jun 20 12:12:51 2020       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 450.36.06    Driver Version: 418.67       CUDA Version: 10.1     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                               |                      |               MIG M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla P100-PCIE...  Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   45C    P0    39W / 250W |      0MiB / 16280MiB |      0%      Default |\n",
            "|                               |                      |                 ERR! |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                                  |\n",
            "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
            "|        ID   ID                                                   Usage      |\n",
            "|=============================================================================|\n",
            "|  No running processes found                                                 |\n",
            "+-----------------------------------------------------------------------------+\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mQ_bNRuloctt",
        "colab_type": "text"
      },
      "source": [
        "Import Google Drive for persistent storage of our training data, neural network model weights and other required files."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SMpvVU6Tq2jy",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "daadc193-1cba-405d-d50d-06a05f3e8348"
      },
      "source": [
        "drive.mount('/content/drive')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "go-hqw0wokrN",
        "colab_type": "text"
      },
      "source": [
        "Empty GPU's memory/cache for training so we'd clear garbage values in it and more memory will be available"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XOjZidblq5JL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "torch.cuda.empty_cache()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zjPKpYJTooMG",
        "colab_type": "text"
      },
      "source": [
        "## Custom Dataset Class\n",
        "\n",
        "Create a custom dataset class that augments each image into 4 different angles: 0, 45, 90, 120, 180, 270, 300, 330 degrees. Fuse it with Pytorch's DataLoader class so data can be loaded, augmented and trained in realtime instead of caching all training samples in memory for augmenting."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "V4PuME0qmFvZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class BrainTumorDataset(Dataset):\n",
        "  def __init__(self, images, labels):\n",
        "    # images\n",
        "    self.X = images\n",
        "    # labels\n",
        "    self.y = labels\n",
        "    \n",
        "    # Transformation for converting original image array to an image and then convert it to a tensor\n",
        "    self.transform = transforms.Compose([transforms.ToPILImage(),\n",
        "        transforms.ToTensor()\n",
        "    ])\n",
        "\n",
        "    # Transformation for converting original image array to an image, rotate it randomly between -45 degrees and 45 degrees, and then convert it to a tensor\n",
        "    self.transform1 = transforms.Compose([\n",
        "        transforms.ToPILImage(),                                          \n",
        "        transforms.RandomRotation(45),\n",
        "        transforms.ToTensor()                                 \n",
        "    ])\n",
        "\n",
        "    # Transformation for converting original image array to an image, rotate it randomly between -90 degrees and 90 degrees, and then convert it to a tensor\n",
        "    self.transform2 = transforms.Compose([\n",
        "        transforms.ToPILImage(),\n",
        "        transforms.RandomRotation(90),\n",
        "        transforms.ToTensor()                                  \n",
        "    ])\n",
        "\n",
        "    # Transformation for converting original image array to an image, rotate it randomly between -120 degrees and 120 degrees, and then convert it to a tensor\n",
        "    self.transform3 = transforms.Compose([\n",
        "        transforms.ToPILImage(),\n",
        "        transforms.RandomRotation(120),\n",
        "        transforms.ToTensor()                                  \n",
        "    ])\n",
        "\n",
        "    # Transformation for converting original image array to an image, rotate it randomly between -180 degrees and 180 degrees, and then convert it to a tensor\n",
        "    self.transform4 = transforms.Compose([\n",
        "        transforms.ToPILImage(),\n",
        "        transforms.RandomRotation(180),\n",
        "        transforms.ToTensor()                                \n",
        "    ])\n",
        "\n",
        "    # Transformation for converting original image array to an image, rotate it randomly between -270 degrees and 270 degrees, and then convert it to a tensor\n",
        "    self.transform5 = transforms.Compose([\n",
        "        transforms.ToPILImage(),\n",
        "        transforms.RandomRotation(270),\n",
        "        transforms.ToTensor()                                \n",
        "    ])\n",
        "\n",
        "    # Transformation for converting original image array to an image, rotate it randomly between -300 degrees and 300 degrees, and then convert it to a tensor\n",
        "    self.transform6 = transforms.Compose([\n",
        "        transforms.ToPILImage(),\n",
        "        transforms.RandomRotation(300),\n",
        "        transforms.ToTensor()                               \n",
        "    ])\n",
        "\n",
        "    # Transformation for converting original image array to an image, rotate it randomly between -330 degrees and 330 degrees, and then convert it to a tensor\n",
        "    self.transform7 = transforms.Compose([\n",
        "        transforms.ToPILImage(),\n",
        "        transforms.RandomRotation(330),\n",
        "        transforms.ToTensor()                                 \n",
        "    ])\n",
        "\n",
        "  def __len__(self):\n",
        "    # return length of image samples\n",
        "    return len(self.X)\n",
        "\n",
        "  def __getitem__(self, idx):\n",
        "    # perform transformations on one instance of X\n",
        "    # Original image as a tensor\n",
        "    data = self.transform(self.X[idx])\n",
        "\n",
        "    # Augmented image at 45 degrees as a tensor\n",
        "    aug45 = self.transform1(self.X[idx])\n",
        "\n",
        "    # Augmented image at 90 degrees as a tensor\n",
        "    aug90 = self.transform2(self.X[idx])\n",
        "\n",
        "    # Augmented image at 120 degrees as a tensor\n",
        "    aug120 = self.transform3(self.X[idx])\n",
        "\n",
        "    # Augmented image at 180 degrees as a tensor\n",
        "    aug180 = self.transform4(self.X[idx])\n",
        "\n",
        "    # Augmented image at 270 degrees as a tensor\n",
        "    aug270 = self.transform5(self.X[idx])\n",
        "\n",
        "    # Augmented image at 300 degrees as a tensor\n",
        "    aug300 = self.transform6(self.X[idx])\n",
        "\n",
        "    # Augmented image at 330 degrees as a tensor\n",
        "    aug330 = self.transform7(self.X[idx])      \n",
        "    \n",
        "    # store the transformed images in a list\n",
        "    new_batch = [data, aug45, aug90, aug120, aug180, aug270, aug300, aug330]\n",
        "\n",
        "    # one-hot encode the labels\n",
        "    labels = torch.zeros(4, dtype=torch.float32)\n",
        "    labels[int(self.y[idx])] = 1.0\n",
        "\n",
        "    new_labels = [labels, labels, labels, labels, labels, labels, labels, labels]\n",
        "\n",
        "    # 8 augmented images and corresponding labels per sample will be returned\n",
        "    return (torch.stack(new_labels), torch.stack(new_batch))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zE6mTnC6pQRD",
        "colab_type": "text"
      },
      "source": [
        "## Load the Dataset\n",
        "\n",
        "* Load the **training_data.pickle** file. \n",
        "\n",
        "* Store the images and labels in separate lists called Xt & yt."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "trcS9hq8q7Dq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "training_data = pickle.load(open('/content/drive/My Drive/new_dataset/training_data.pickle', 'rb'))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-FUX5SScHhEg",
        "colab_type": "text"
      },
      "source": [
        "Create empty lists for storing our data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hY6k--StlVAC",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "Xt = []\n",
        "yt = []\n",
        "features = None\n",
        "labels = None\n",
        "label = []"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "chU6n1FDHllX",
        "colab_type": "text"
      },
      "source": [
        "Store images in Xt and labels in yt iteratively"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9gQx6_RtlwfE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "for features,labels in training_data:\n",
        "  Xt.append(features)\n",
        "  yt.append(labels)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QOlU2ZoCpitc",
        "colab_type": "text"
      },
      "source": [
        "## Train Validation Test split\n",
        "\n",
        "Split the dataset for training using cross-validation method.\n",
        "\n",
        "* 70 % of images for training \n",
        "* 15% of images for validating\n",
        "* 15% of images for testing\n",
        "\n",
        "Set random seed and random_state to any arbitrary number, so the train_test_split happens the same way everytime the function is called."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wxT6Znuul5Kq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# 70 % training, 15% validating, 15% testing\n",
        "X_train, X_test, y_train, y_test = train_test_split(Xt, yt, test_size=0.3, shuffle=True)  # 70% training, 30% testing\n",
        "X_valid, X_test, y_valid, y_test = train_test_split(X_test, y_test, test_size=0.5, shuffle=True)  # split testing set into 50% validation , 50% testing "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BWlXaeaVqINY",
        "colab_type": "text"
      },
      "source": [
        "Empty the previously used lists and arrays to free up RAM / Cache"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "34qYiYRlJoXH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "Xt = None\n",
        "yt = None\n",
        "features = None\n",
        "labels = None\n",
        "label = None\n",
        "training_data = None "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oKlDZh7VqaJj",
        "colab_type": "text"
      },
      "source": [
        "Create training set, validation set and test set using our custom dataset class"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rnTKHLR5mCeI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "train_set = BrainTumorDataset(X_train, y_train)\n",
        "valid_set = BrainTumorDataset(X_valid, y_valid)\n",
        "test_set = BrainTumorDataset(X_test, y_test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tfWO3u4YqiUP",
        "colab_type": "text"
      },
      "source": [
        "Print original number of samples in each set"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MEL7oQ4Y8NIe",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 70
        },
        "outputId": "b4d33819-b7f1-4b5a-ce44-d7aa9c9d9b0c"
      },
      "source": [
        "print(f\"Number of training samples: {len(X_train)}\")\n",
        "print(f\"Number of validation samples: {len(X_valid)}\")\n",
        "print(f\"Number of testing samples: {len(X_test)}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Number of training samples: 2144\n",
            "Number of validation samples: 460\n",
            "Number of testing samples: 460\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0udsS0Qqql3g",
        "colab_type": "text"
      },
      "source": [
        "Print augmented number of samples in each set"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8PJu7hjEi0Ij",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 70
        },
        "outputId": "b9cb23ac-8182-4b4b-cd7f-d7b93670b615"
      },
      "source": [
        "print(f\"Number of augmented training samples: {len(X_train) * 8}\")\n",
        "print(f\"Number of augmented validation samples: {len(X_valid)* 8}\")\n",
        "print(f\"Number of augmented testing samples: {len(X_test)* 8}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Number of augmented training samples: 17152\n",
            "Number of augmented validation samples: 3680\n",
            "Number of augmented testing samples: 3680\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S9XaKzTZqpW0",
        "colab_type": "text"
      },
      "source": [
        "Create a DataLoader for each set with batch size of 4 and shuffling enabled"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-5Eil06jrRUT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "train_gen = DataLoader(train_set, batch_size=4, shuffle=True, pin_memory=True, num_workers=8)\n",
        "valid_gen = DataLoader(valid_set, batch_size=4, shuffle=True, pin_memory=True, num_workers=8)\n",
        "test_gen = DataLoader(test_set, batch_size=10, shuffle=True, pin_memory=True, num_workers=8)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VE1aH9bJq1HD",
        "colab_type": "text"
      },
      "source": [
        "Get device to set the training to run on GPU or CPU later based on its availibility"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ka_a8OAPtNHY",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "device_name = \"cuda:0:\" if torch.cuda.is_available() else \"cpu\"\n",
        "device = torch.device(device_name)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eHncA4uwrAnB",
        "colab_type": "text"
      },
      "source": [
        "## Build the Model\n",
        "\n",
        "* Instantiate the transfer learning model using torchvision's models class.\n",
        "\n",
        "* RESNET50 is the CNN model that we're going to use by transfer learning.\n",
        "\n",
        "* Set all the pretrained weights to trainable by enabling every layer's parameters as true\n",
        "\n",
        "* Build the top layer by creating a custom output sequential layer and assign it to model's fc."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Lq3_lCEttAJk",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "feb12130-9d1d-4c44-d546-6327f266745a"
      },
      "source": [
        "# instantiate transfer learning model\n",
        "resnet_model = models.resnet50(pretrained=True)\n",
        "\n",
        "# set all paramters as trainable\n",
        "for param in resnet_model.parameters():\n",
        "    param.requires_grad = True\n",
        "\n",
        "# get input of fc layer\n",
        "n_inputs = resnet_model.fc.in_features\n",
        "\n",
        "# redefine fc layer / top layer/ head for our classification problem\n",
        "resnet_model.fc = nn.Sequential(nn.Linear(n_inputs, 2048),\n",
        "                                nn.SELU(),\n",
        "                                nn.Dropout(p=0.4),\n",
        "                                nn.Linear(2048, 2048),\n",
        "                                nn.SELU(),\n",
        "                                nn.Dropout(p=0.4),\n",
        "                                nn.Linear(2048, 4),\n",
        "                                nn.LogSigmoid())\n",
        "\n",
        "# set all paramters of the model as trainable\n",
        "for name, child in resnet_model.named_children():\n",
        "  for name2, params in child.named_parameters():\n",
        "    params.requires_grad = True\n",
        "\n",
        "# set model to run on GPU or CPU absed on availibility\n",
        "resnet_model.to(device)\n",
        "\n",
        "# print the trasnfer learning NN model's architecture\n",
        "resnet_model"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "ResNet(\n",
              "  (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)\n",
              "  (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  (relu): ReLU(inplace=True)\n",
              "  (maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)\n",
              "  (layer1): Sequential(\n",
              "    (0): Bottleneck(\n",
              "      (conv1): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (conv3): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "      (bn3): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu): ReLU(inplace=True)\n",
              "      (downsample): Sequential(\n",
              "        (0): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      )\n",
              "    )\n",
              "    (1): Bottleneck(\n",
              "      (conv1): Conv2d(256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (conv3): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "      (bn3): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu): ReLU(inplace=True)\n",
              "    )\n",
              "    (2): Bottleneck(\n",
              "      (conv1): Conv2d(256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (conv3): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "      (bn3): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu): ReLU(inplace=True)\n",
              "    )\n",
              "  )\n",
              "  (layer2): Sequential(\n",
              "    (0): Bottleneck(\n",
              "      (conv1): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
              "      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "      (bn3): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu): ReLU(inplace=True)\n",
              "      (downsample): Sequential(\n",
              "        (0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
              "        (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      )\n",
              "    )\n",
              "    (1): Bottleneck(\n",
              "      (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "      (bn3): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu): ReLU(inplace=True)\n",
              "    )\n",
              "    (2): Bottleneck(\n",
              "      (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "      (bn3): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu): ReLU(inplace=True)\n",
              "    )\n",
              "    (3): Bottleneck(\n",
              "      (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "      (bn3): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu): ReLU(inplace=True)\n",
              "    )\n",
              "  )\n",
              "  (layer3): Sequential(\n",
              "    (0): Bottleneck(\n",
              "      (conv1): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
              "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "      (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu): ReLU(inplace=True)\n",
              "      (downsample): Sequential(\n",
              "        (0): Conv2d(512, 1024, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
              "        (1): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      )\n",
              "    )\n",
              "    (1): Bottleneck(\n",
              "      (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "      (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu): ReLU(inplace=True)\n",
              "    )\n",
              "    (2): Bottleneck(\n",
              "      (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "      (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu): ReLU(inplace=True)\n",
              "    )\n",
              "    (3): Bottleneck(\n",
              "      (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "      (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu): ReLU(inplace=True)\n",
              "    )\n",
              "    (4): Bottleneck(\n",
              "      (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "      (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu): ReLU(inplace=True)\n",
              "    )\n",
              "    (5): Bottleneck(\n",
              "      (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "      (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu): ReLU(inplace=True)\n",
              "    )\n",
              "  )\n",
              "  (layer4): Sequential(\n",
              "    (0): Bottleneck(\n",
              "      (conv1): Conv2d(1024, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
              "      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (conv3): Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "      (bn3): BatchNorm2d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu): ReLU(inplace=True)\n",
              "      (downsample): Sequential(\n",
              "        (0): Conv2d(1024, 2048, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
              "        (1): BatchNorm2d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      )\n",
              "    )\n",
              "    (1): Bottleneck(\n",
              "      (conv1): Conv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (conv3): Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "      (bn3): BatchNorm2d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu): ReLU(inplace=True)\n",
              "    )\n",
              "    (2): Bottleneck(\n",
              "      (conv1): Conv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (conv3): Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "      (bn3): BatchNorm2d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu): ReLU(inplace=True)\n",
              "    )\n",
              "  )\n",
              "  (avgpool): AdaptiveAvgPool2d(output_size=(1, 1))\n",
              "  (fc): Sequential(\n",
              "    (0): Linear(in_features=2048, out_features=2048, bias=True)\n",
              "    (1): SELU()\n",
              "    (2): Dropout(p=0.4, inplace=False)\n",
              "    (3): Linear(in_features=2048, out_features=2048, bias=True)\n",
              "    (4): SELU()\n",
              "    (5): Dropout(p=0.4, inplace=False)\n",
              "    (6): Linear(in_features=2048, out_features=4, bias=True)\n",
              "    (7): LogSigmoid()\n",
              "  )\n",
              ")"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_QdrEPESsKQi",
        "colab_type": "text"
      },
      "source": [
        "## Set Training Configuration\n",
        "\n",
        "* Set model's loss function as CrossEntropyLoss\n",
        "\n",
        "* Set SGD optimizer with 0.9 momentum and learning rate 3e-4 as the model's optimizer. According to many Deep learning experts and researchers such as [Andrej karpathy](https://github.com/karpathy) 3e-4 is a good learning rate choice.\n",
        "\n",
        "* Run the model for 10 total iterations\n",
        "\n",
        "* Create empty lists to store training losses, validation losses, training accuracies, and validation accuracies."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cokrpXp3ud8_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# loss function\n",
        "# if GPU is available set loss function to use GPU\n",
        "criterion = nn.CrossEntropyLoss().to(device)\n",
        "\n",
        "# optimizer\n",
        "optimizer = torch.optim.SGD(resnet_model.parameters(), momentum=0.9, lr=3e-4)\n",
        "\n",
        "# number of training iterations\n",
        "epochs = 30\n",
        "\n",
        "# empty lists to store losses and accuracies\n",
        "train_losses = []\n",
        "test_losses = []\n",
        "train_correct = []\n",
        "test_correct = []"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7E_rg4eJs7Xq",
        "colab_type": "text"
      },
      "source": [
        "## Util function\n",
        "\n",
        "### Checkpoint Saver\n",
        "A function to save the model using checkpoints based on best loss achieved during every iteration compared with previous iteration's loss. We'll load the checkpoint and resume training in case Colab's runtime get's disconnected due to inactivity or any other issues."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MnUksQpFNxNK",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def save_checkpoint(state, is_best, filename='/content/drive/My Drive/bt_resnet50_ckpt_v2.pth.tar'):\n",
        "    torch.save(state, filename)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ci1Lw3UgtU13",
        "colab_type": "text"
      },
      "source": [
        "## Train the model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0TWAlOEau6k7",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "d1c0a256-466f-4420-c9ad-a1bf15630b97"
      },
      "source": [
        "# set training start time\n",
        "start_time = time.time()\n",
        "\n",
        "# set best_prec loss value as 2 for checkpoint threshold\n",
        "best_prec1 = 2\n",
        "\n",
        "# empty batch variables\n",
        "b = None\n",
        "train_b = None\n",
        "test_b = None\n",
        "\n",
        "# start training\n",
        "for i in range(epochs):\n",
        "    # empty training correct and test correct counter as 0 during every iteration\n",
        "    trn_corr = 0\n",
        "    tst_corr = 0\n",
        "    \n",
        "    # set epoch's starting time\n",
        "    e_start = time.time()\n",
        "    \n",
        "    # train in batches\n",
        "    for b, (y, X) in enumerate(train_gen):\n",
        "        # set label as cuda if device is cuda\n",
        "        X, y = X.to(device), y.to(device)\n",
        "\n",
        "        # forward pass image sample\n",
        "        y_pred = resnet_model(X.view(-1, 3, 512, 512))\n",
        "        # calculate loss\n",
        "        loss = criterion(y_pred.float(), torch.argmax(y.view(32, 4), dim=1).long())\n",
        "\n",
        "        # get argmax of predicted tensor, which is our label\n",
        "        predicted = torch.argmax(y_pred, dim=1).data\n",
        "        # if predicted label is correct as true label, calculate the sum for samples\n",
        "        batch_corr = (predicted == torch.argmax(y.view(32, 4), dim=1)).sum()\n",
        "        # increment train correct with correcly predicted labels per batch\n",
        "        trn_corr += batch_corr\n",
        "        \n",
        "        # set optimizer gradients to zero\n",
        "        optimizer.zero_grad()\n",
        "        # back propagate with loss\n",
        "        loss.backward()\n",
        "        # perform optimizer step\n",
        "        optimizer.step()\n",
        "\n",
        "    # set epoch's end time\n",
        "    e_end = time.time()\n",
        "    # print training metrics\n",
        "    print(f'Epoch {(i+1)} Batch {(b+1)*4}\\nAccuracy: {trn_corr.item()*100/(4*8*b):2.2f} %  Loss: {loss.item():2.4f}  Duration: {((e_end-e_start)/60):.2f} minutes') # 4 images per batch * 8 augmentations per image * batch length\n",
        "\n",
        "    # some metrics storage for visualization\n",
        "    train_b = b\n",
        "    train_losses.append(loss)\n",
        "    train_correct.append(trn_corr)\n",
        "\n",
        "    X, y = None, None\n",
        "\n",
        "    # validate using validation generator\n",
        "    # do not perform any gradient updates while validation\n",
        "    with torch.no_grad():\n",
        "        for b, (y, X) in enumerate(valid_gen):\n",
        "            # set label as cuda if device is cuda\n",
        "            X, y = X.to(device), y.to(device)\n",
        "\n",
        "            # forward pass image\n",
        "            y_val = resnet_model(X.view(-1, 3, 512, 512))\n",
        "\n",
        "            # get argmax of predicted tensor, which is our label\n",
        "            predicted = torch.argmax(y_val, dim=1).data\n",
        "\n",
        "            # increment test correct with correcly predicted labels per batch\n",
        "            tst_corr += (predicted == torch.argmax(y.view(32, 4), dim=1)).sum()\n",
        "\n",
        "    # get loss of validation set\n",
        "    loss = criterion(y_val.float(), torch.argmax(y.view(32, 4), dim=1).long())\n",
        "    # print validation metrics\n",
        "    print(f'Validation Accuracy {tst_corr.item()*100/(4*8*b):2.2f} Validation Loss: {loss.item():2.4f}\\n')\n",
        "\n",
        "    # if current validation loss is less than previous iteration's validatin loss create and save a checkpoint\n",
        "    is_best = loss < best_prec1\n",
        "    best_prec1 = min(loss, best_prec1)\n",
        "    save_checkpoint({\n",
        "            'epoch': i + 1,\n",
        "            'state_dict': resnet_model.state_dict(),\n",
        "            'best_prec1': best_prec1,\n",
        "        }, is_best)\n",
        "\n",
        "    # some metrics storage for visualization\n",
        "    test_b  = b\n",
        "    test_losses.append(loss)\n",
        "    test_correct.append(tst_corr)\n",
        "\n",
        "# set total training's end time\n",
        "end_time = time.time() - start_time    \n",
        "\n",
        "# print training summary\n",
        "print(\"\\nTraining Duration {:.2f} minutes\".format(end_time/60))\n",
        "print(\"GPU memory used : {} kb\".format(torch.cuda.memory_allocated()))\n",
        "print(\"GPU memory cached : {} kb\".format(torch.cuda.memory_cached()))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1 Batch 2144\n",
            "Accuracy: 56.80 %  Loss: 0.4113  Duration: 7.17 minutes\n",
            "Validation Accuracy 69.44 Validation Loss: 1.3394\n",
            "\n",
            "Epoch 2 Batch 2144\n",
            "Accuracy: 79.44 %  Loss: 1.0226  Duration: 7.14 minutes\n",
            "Validation Accuracy 87.03 Validation Loss: 1.3007\n",
            "\n",
            "Epoch 3 Batch 2144\n",
            "Accuracy: 85.72 %  Loss: 0.0820  Duration: 7.13 minutes\n",
            "Validation Accuracy 85.75 Validation Loss: 0.2911\n",
            "\n",
            "Epoch 4 Batch 2144\n",
            "Accuracy: 88.46 %  Loss: 0.5032  Duration: 7.13 minutes\n",
            "Validation Accuracy 89.91 Validation Loss: 0.0314\n",
            "\n",
            "Epoch 5 Batch 2144\n",
            "Accuracy: 90.29 %  Loss: 1.4266  Duration: 7.14 minutes\n",
            "Validation Accuracy 92.57 Validation Loss: 0.3442\n",
            "\n",
            "Epoch 6 Batch 2144\n",
            "Accuracy: 91.64 %  Loss: 0.0373  Duration: 7.15 minutes\n",
            "Validation Accuracy 90.90 Validation Loss: 0.0449\n",
            "\n",
            "Epoch 7 Batch 2144\n",
            "Accuracy: 93.24 %  Loss: 0.1904  Duration: 7.14 minutes\n",
            "Validation Accuracy 93.64 Validation Loss: 0.0942\n",
            "\n",
            "Epoch 8 Batch 2144\n",
            "Accuracy: 95.27 %  Loss: 0.0215  Duration: 7.15 minutes\n",
            "Validation Accuracy 95.94 Validation Loss: 0.0230\n",
            "\n",
            "Epoch 9 Batch 2144\n",
            "Accuracy: 96.08 %  Loss: 0.0065  Duration: 7.14 minutes\n",
            "Validation Accuracy 92.82 Validation Loss: 0.0080\n",
            "\n",
            "Epoch 10 Batch 2144\n",
            "Accuracy: 97.21 %  Loss: 0.0347  Duration: 7.14 minutes\n",
            "Validation Accuracy 94.19 Validation Loss: 0.1907\n",
            "\n",
            "Epoch 11 Batch 2144\n",
            "Accuracy: 96.34 %  Loss: 0.0050  Duration: 7.13 minutes\n",
            "Validation Accuracy 96.85 Validation Loss: 0.1216\n",
            "\n",
            "Epoch 12 Batch 2144\n",
            "Accuracy: 97.03 %  Loss: 0.1557  Duration: 7.14 minutes\n",
            "Validation Accuracy 95.34 Validation Loss: 0.0142\n",
            "\n",
            "Epoch 13 Batch 2144\n",
            "Accuracy: 97.79 %  Loss: 0.0323  Duration: 7.13 minutes\n",
            "Validation Accuracy 97.18 Validation Loss: 0.0280\n",
            "\n",
            "Epoch 14 Batch 2144\n",
            "Accuracy: 98.07 %  Loss: 0.0075  Duration: 7.14 minutes\n",
            "Validation Accuracy 96.93 Validation Loss: 0.0036\n",
            "\n",
            "Epoch 15 Batch 2144\n",
            "Accuracy: 98.29 %  Loss: 0.0080  Duration: 7.13 minutes\n",
            "Validation Accuracy 96.79 Validation Loss: 0.0067\n",
            "\n",
            "Epoch 16 Batch 2144\n",
            "Accuracy: 98.33 %  Loss: 0.3111  Duration: 7.13 minutes\n",
            "Validation Accuracy 97.89 Validation Loss: 0.2079\n",
            "\n",
            "Epoch 17 Batch 2144\n",
            "Accuracy: 98.46 %  Loss: 0.0145  Duration: 7.14 minutes\n",
            "Validation Accuracy 97.64 Validation Loss: 0.0079\n",
            "\n",
            "Epoch 18 Batch 2144\n",
            "Accuracy: 99.09 %  Loss: 0.0176  Duration: 7.13 minutes\n",
            "Validation Accuracy 97.51 Validation Loss: 0.0313\n",
            "\n",
            "Epoch 19 Batch 2144\n",
            "Accuracy: 98.92 %  Loss: 0.0012  Duration: 7.14 minutes\n",
            "Validation Accuracy 96.24 Validation Loss: 0.0181\n",
            "\n",
            "Epoch 20 Batch 2144\n",
            "Accuracy: 99.42 %  Loss: 0.0017  Duration: 7.14 minutes\n",
            "Validation Accuracy 97.75 Validation Loss: 0.0065\n",
            "\n",
            "Epoch 21 Batch 2144\n",
            "Accuracy: 99.52 %  Loss: 0.0013  Duration: 7.13 minutes\n",
            "Validation Accuracy 97.59 Validation Loss: 0.0034\n",
            "\n",
            "Epoch 22 Batch 2144\n",
            "Accuracy: 99.85 %  Loss: 0.0014  Duration: 7.14 minutes\n",
            "Validation Accuracy 98.27 Validation Loss: 0.0242\n",
            "\n",
            "Epoch 23 Batch 2144\n",
            "Accuracy: 99.39 %  Loss: 0.0474  Duration: 7.13 minutes\n",
            "Validation Accuracy 97.81 Validation Loss: 0.1353\n",
            "\n",
            "Epoch 24 Batch 2144\n",
            "Accuracy: 99.81 %  Loss: 0.2521  Duration: 7.13 minutes\n",
            "Validation Accuracy 98.19 Validation Loss: 0.2287\n",
            "\n",
            "Epoch 25 Batch 2144\n",
            "Accuracy: 99.65 %  Loss: 0.0075  Duration: 7.13 minutes\n",
            "Validation Accuracy 98.71 Validation Loss: 0.0178\n",
            "\n",
            "Epoch 26 Batch 2144\n",
            "Accuracy: 99.85 %  Loss: 0.0745  Duration: 7.13 minutes\n",
            "Validation Accuracy 98.33 Validation Loss: 0.0034\n",
            "\n",
            "Epoch 27 Batch 2144\n",
            "Accuracy: 100.05 %  Loss: 0.0002  Duration: 7.13 minutes\n",
            "Validation Accuracy 98.79 Validation Loss: 0.0046\n",
            "\n",
            "Epoch 28 Batch 2144\n",
            "Accuracy: 100.01 %  Loss: 0.0014  Duration: 7.13 minutes\n",
            "Validation Accuracy 98.66 Validation Loss: 0.1200\n",
            "\n",
            "Epoch 29 Batch 2144\n",
            "Accuracy: 99.86 %  Loss: 0.0452  Duration: 7.13 minutes\n",
            "Validation Accuracy 97.94 Validation Loss: 0.0074\n",
            "\n",
            "Epoch 30 Batch 2144\n",
            "Accuracy: 100.08 %  Loss: 0.0005  Duration: 7.13 minutes\n",
            "Validation Accuracy 98.88 Validation Loss: 0.0016\n",
            "\n",
            "\n",
            "Training Duration 231.13 minutes\n",
            "GPU memory used : 490444800 kb\n",
            "GPU memory cached : 16034824192 kb\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "U5hiXkZkwW-W",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "torch.cuda.empty_cache()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9Rzl2f2kteb8",
        "colab_type": "text"
      },
      "source": [
        "## Save the model\n",
        "\n",
        "Save the model after the training is completed"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "K4yzcJMmwPnY",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "torch.save(resnet_model.state_dict(), '/content/drive/My Drive/bt_resnet50_model.pt')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q-2R8Yputlvh",
        "colab_type": "text"
      },
      "source": [
        "## Evaluation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hPEBr7E1tobU",
        "colab_type": "text"
      },
      "source": [
        "Print the validation accuracy of the model calculated using validation set during training "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4w0EuNLQvmNV",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "685ce068-21f1-4419-c278-555cde282b40"
      },
      "source": [
        "print(f'Validation accuracy: {test_correct[-1].item()*100/(test_b*8*4):.2f}%')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Validation accuracy: 98.88%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EnauSi8Dt04I",
        "colab_type": "text"
      },
      "source": [
        "Plot the loss graph"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TttSopdFvtr0",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 294
        },
        "outputId": "6bc80323-bb7c-4a70-eae5-544ed8c363fb"
      },
      "source": [
        "plt.plot(train_losses, label='Training loss')\n",
        "plt.plot(test_losses, label='Validation loss')\n",
        "plt.title('Loss Metrics')\n",
        "plt.ylabel('Loss')\n",
        "plt.xlabel('Epochs')\n",
        "plt.legend()\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAEWCAYAAACEz/viAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nOzdeXxU1fn48c+ZJZNlshNkCRD2HQIEARHEfRdxQ+pGbbVaq63+au23i1qr/bbf2tZaxdZq3apF64JYcamKsonsO0HCJmEn+z7JzPn9cecmk20yk8ydBHjerxcvmHvv3HsCOs+c85zzHKW1RgghxKnN1tkNEEII0fkkGAghhJBgIIQQQoKBEEIIJBgIIYRAgoEQQggkGAjR5SmlpimldnR2O8TJTYKBOKEppfYqpc7rhOe+qJTSSqmZTY7/yX98boj30UqpQcGu0Vov1VoP7UBzhWiTBAMh2u9r4GbzhVLKAVwH7IrUA/z3FMJyEgzESUkp5VJKPaGUOuj/9YRSyuU/100p9R+lVLFSqlAptVQpZfOfe0ApdUApVaaU2qGUOjfIY94DzlRKpfpfXwRsAg43acutSqntSqkipdRHSql+/uNL/JdsVEqVK6VmK6VmKKXy/e04DLxgHgu4Xx+l1NtKqWNKqQKl1FP+44OUUl8opUqUUseVUq9H4K9SnCIkGIiT1c+ByUA2MBY4HfiF/9z/A/KBDOA04GeAVkoNBX4ATNRaJwIXAnuDPKMaeBe43v/6ZuDlwAv8w0g/A67yP28p8C8ArfV0/2VjtdZurbX54d0DSAP6Abc3uZ8d+A+wD8gCegPz/ad/DXwMpAKZwF+CtF2IRiQYiJPVDcAjWuujWutjwK+Am/znaoGeQD+tda1/TF4DXsAFjFBKObXWe7XWbQ35vAzcrJRKAc4CFjQ5fwfwv1rr7VrrOuA3QLbZO2iFD3hIa12jta5qcu50oBdwv9a6QmtdrbVeFvBz9QN6NTkuRJskGIiTVS+Mb8+mff5jAL8H8oCPlVK7lVI/BdBa5wE/Ah4Gjiql5iulehGE/wM3A6Mn8p8WPrz7AX/2D0kVA4WAwvhG35pjWuvqVs71Afb5A0tTP/Hfe5VSaqtS6tZgbRcikAQDcbI6iPFBbOrrP4bWukxr/f+01gOAK4D7zNyA1vo1rfWZ/vdq4HchPOufGENPL7dwbj/wPa11SsCvOK31iiD3C1ZKeD/Qt6XEstb6sNb6Nq11L+B7wLy2ZioJYZJgIE4GTqVUbMAvB8a4/C+UUhlKqW7Agxgf2iilLvMnWxVQgjE85FNKDVVKneNPNFcDVRhDNm15EjgfWNLCub8C/6OUGul/drJS6tqA80eAAWH8rKuAQ8BvlVIJ/p93qv/e1yqlMv3XFWEElVDaL4QEA3FSWITxwW3+ehh4FFiDMbtnM7DOfwxgMPAJUA58CczTWi/GyBf8FjiOMSOoO/A/bT1ca12otf5Ut7A5iNb6HYzexXylVCmwBbg44JKHgZf8w0jXhfAsL3A5MAj4BiMRPtt/eiLwlVKqHFgI/FBrvbutewoBoGRzGyGEENIzEEIIIcFACCGEBAMhhBBIMBBCCAGccEWwunXrprOysjq7GUIIcUJZu3btca11RmvnT7hgkJWVxZo1azq7GUIIcUJRSu0Ldl6GiYQQQkgwEEIIIcFACCEEJ2DOQAgRfbW1teTn51Nd3VoxVdFVxMbGkpmZidPpDOt9EgyEEG3Kz88nMTGRrKwsjPp+oivSWlNQUEB+fj79+/cP670yTCSEaFN1dTXp6ekSCLo4pRTp6ent6sFJMBBChEQCwYmhvf9OEgy6mEpPHW+uzUeqyQohokmCQRfzwebD/PjfG9l1rKKzmyJEl1FQUEB2djbZ2dn06NGD3r1717/2eDxB37tmzRruueeeNp9xxhlnRKStn3/+OZdddllE7hVNliWQlVL/AC4DjmqtRwW5biLGBiPXa63ftKo9J4pj5TUAlFbXdnJLhOg60tPT2bBhAwAPP/wwbrebH//4x/Xn6+rqcDha/jjLyckhJyenzWesWBFsJ9KTn5U9gxeBi4JdoJSyY+wC9bGF7TihFFYY33LKq1va71wIYZo7dy533HEHkyZN4ic/+QmrVq1iypQpjBs3jjPOOIMdO3YAjb+pP/zww9x6663MmDGDAQMG8OSTT9bfz+12118/Y8YMrrnmGoYNG8YNN9xQP2y7aNEihg0bxoQJE7jnnnva7AEUFhZy5ZVXMmbMGCZPnsymTZsA+OKLL+p7NuPGjaOsrIxDhw4xffp0srOzGTVqFEuXLo3431kwlvUMtNZLlFJZbVx2N/AWxnZ9Aigo9weDGgkGomv61Xtb2XawNKL3HNEriYcuHxn2+/Lz81mxYgV2u53S0lKWLl2Kw+Hgk08+4Wc/+xlvvfVWs/fk5uayePFiysrKGDp0KHfeeWezOfnr169n69at9OrVi6lTp7J8+XJycnL43ve+x5IlS+jfvz9z5sxps30PPfQQ48aNY8GCBXz22WfcfPPNbNiwgccff5ynn36aqVOnUl5eTmxsLM8++ywXXnghP//5z/F6vVRWVob999ERnbbOQCnVG5gFnE0bwUApdTtwO0Dfvn2tb1wnKqwwhokkGAjRtmuvvRa73Q5ASUkJt9xyCzt37kQpRW1ty0Otl156KS6XC5fLRffu3Tly5AiZmZmNrjn99NPrj2VnZ7N3717cbjcDBgyon78/Z84cnn322aDtW7ZsWX1AOueccygoKKC0tJSpU6dy3333ccMNN3DVVVeRmZnJxIkTufXWW6mtreXKK68kOzu7Q3834erMRWdPAA9orX1tTYXSWj8LPAuQk5NzUk+zkWEi0dW15xu8VRISEur//Mtf/pKzzz6bd955h7179zJjxowW3+Nyuer/bLfbqatr/v9aKNd0xE9/+lMuvfRSFi1axNSpU/noo4+YPn06S5Ys4f3332fu3Lncd9993HzzzRF9bjCdOZsoB5ivlNoLXAPMU0pd2Ynt6RIKKmSYSIj2KCkpoXfv3gC8+OKLEb//0KFD2b17N3v37gXg9ddfb/M906ZN49VXXwWMXES3bt1ISkpi165djB49mgceeICJEyeSm5vLvn37OO2007jtttv47ne/y7p16yL+MwTTaT0DrXX9Wmml1IvAf7TWCzqrPV1FoQQDIdrlJz/5CbfccguPPvool156acTvHxcXx7x587joootISEhg4sS2U51mwnrMmDHEx8fz0ksvAfDEE0+wePFibDYbI0eO5OKLL2b+/Pn8/ve/x+l04na7efnllyP+MwSjrFrcpJT6FzAD6AYcAR4CnABa6782ufZFjGDQ5tTSnJwcfbJublNd62XYLz8EYM7pffnfq0Z3couEMGzfvp3hw4d3djM6XXl5OW63G601d911F4MHD+bee+/t7GY109K/l1Jqrda61Tm2Vs4majvV3nDtXKvacSIxh4hAegZCdEV///vfeemll/B4PIwbN47vfe97nd2kiJGqpV1IYXlAMJBFZ0J0Offee2+X7AlEwqlTjqKyEJb+AWrKOrslrSrwTyt1OWzSMxBCRNWpEwx2fgyfPgJPjIFlT4Cn69X+MZPHfdPiKZOppUKIKDp1gsHY6+G2z6D3ePjkIfjzWPhyHtR2nZ2bzGDQLz1eegZCiKg6dYIBQO8JcONbcOtH0H04fPQ/8GQ2rPo71NV0dusoqPDgsCl6pcRJMBBCRNWpFQxMfSfDLe8Zv1L6waIfw18mwNqXwNt5idvCcg+pCTEkxjoor66TPQ2E8Dv77LP56KOPGh174oknuPPOO1t9z4wZMzCnoV9yySUUFxc3u+bhhx/m8ccfD/rsBQsWsG3btvrXDz74IJ988kk4zW9RVyt1fWoGA1P/6XDrh0ZvISED3rsHnpoIG/4FPm/Um1NQ4SE9IQa3y0mdT1NT54t6G4ToiubMmcP8+fMbHZs/f35IxeLAqDaakpLSrmc3DQaPPPII5513Xrvu1ZWd2sEAQCkYdJ6RT5gzH1xuWHAHfPqrqDelsKKGtIQY3LHGjF9JIgthuOaaa3j//ffrN7LZu3cvBw8eZNq0adx5553k5OQwcuRIHnrooRbfn5WVxfHjxwF47LHHGDJkCGeeeWZ9mWsw1hBMnDiRsWPHcvXVV1NZWcmKFStYuHAh999/P9nZ2ezatYu5c+fy5pvG+thPP/2UcePGMXr0aG699VZqamrqn/fQQw8xfvx4Ro8eTW5ubtCfryuUupZ1BialYOjFMPhCePYsOLQp6k0orPAwqncyiS7jn6W8po6MRFcb7xIiyj74KRzeHNl79hgNF/+21dNpaWmcfvrpfPDBB8ycOZP58+dz3XXXoZTiscceIy0tDa/Xy7nnnsumTZsYM2ZMi/dZu3Yt8+fPZ8OGDdTV1TF+/HgmTJgAwFVXXcVtt90GwC9+8Quef/557r77bq644gouu+wyrrnmmkb3qq6uZu7cuXz66acMGTKEm2++mWeeeYYf/ehHAHTr1o1169Yxb948Hn/8cZ577rlWf76uUOpaegZN2WyQ2AOqCqP+aHOYKMEMBtIzEKJe4FBR4BDRG2+8wfjx4xk3bhxbt25tNKTT1NKlS5k1axbx8fEkJSVxxRVX1J/bsmUL06ZNY/To0bz66qts3bo1aHt27NhB//79GTJkCAC33HILS5YsqT9/1VVXATBhwoT64natWbZsGTfddBPQcqnrJ598kuLiYhwOBxMnTuSFF17g4YcfZvPmzSQmJga9d6ikZ9CSuDQ4GrxbF2meOh9l1XWkJbhw+4NBWY2sQhZdUJBv8FaaOXMm9957L+vWraOyspIJEyawZ88eHn/8cVavXk1qaipz586lurp908Xnzp3LggULGDt2LC+++CKff/55h9prlsHuSAnsaJa6lp5BS+LTot4zKKo0xkLT3MZsIpCegRCB3G43Z599Nrfeemt9r6C0tJSEhASSk5M5cuQIH3zwQdB7TJ8+nQULFlBVVUVZWRnvvfde/bmysjJ69uxJbW1tfdlpgMTERMrKmlcuGDp0KHv37iUvLw+AV155hbPOOqtdP1tXKHUtPYOWxKWBp9xYe+CIzpi9ud2lMZuoIWcghGgwZ84cZs2aVT9cNHbsWMaNG8ewYcPo06cPU6dODfr+8ePHM3v2bMaOHUv37t0blaH+9a9/zaRJk8jIyGDSpEn1AeD666/ntttu48knn6xPHAPExsbywgsvcO2111JXV8fEiRO544472vVzdYVS15aVsLZKVEpYr34e3r8P7suFpJ7WPstv2c7j3Pj8V8y/fTKDurvJefQTHpk5kpunZEXl+UIEIyWsTyztKWEtw0QtiU8zfo/iUJFZpC6wZyBTS4UQ0SLBoCVx/mBQGb1gYNYlSkuIweWw4bQrGSYSQkSNBIOWdELPoLDCg1KQEh+DUgq3y0GFBAPRhZxoQ8qnqvb+O0kwaEkn9AwKKjykxsdgtykA3P76REJ0BbGxsRQUFEhA6OK01hQUFBAbGxv2ey2bTaSU+gdwGXBUaz2qhfM3AA8ACigD7tRab7SqPWExewaVBVF7ZGG5h7SEmPrXbpeTMukZiC4iMzOT/Px8jh071tlNEW2IjY0lMzMz7PdZObX0ReApoLV5T3uAs7TWRUqpi4FngUkWtid0zjhwxkNVUdQeWVjROBgkuqRnILoOp9NJ//79O7sZwkKWDRNprZcArY6zaK1XaK3NT9uVQPihzEpxadFNIFcapShM7liHJJCFEFHTVXIG3wGCLx2MtvjUqCeQGw8TSTAQQkRPp69AVkqdjREMzgxyze3A7QB9+/aNTsOi2DPw+jRFLfQMZJ2BECJaOrVnoJQaAzwHzNRat5qt1Vo/q7XO0VrnZGRkRKdx8elRSyAXV3rQmuY5AylUJ4SIkk4LBkqpvsDbwE1a6687qx2timKxuvoFZ+6GOkhul4PqWh+1XtntTAhhPSunlv4LmAF0U0rlAw8BTgCt9V+BB4F0YJ5SCqAuWN2MqItLg6piY/tLm93SRxVUNBSpM5m7nVXU1JESH9Pi+4QQIlIsCwZa66Cbk2qtvwt816rnd1h8GqChuqRh3YFFAktRmALrE0kwEEJYravMJup6orgKuaWeQf2eBjKjSAgRBRIMWhOfbvwehSRyoX8vg9QmK5BBgoEQIjokGLQmPtX4PQpJ5MKKGpJiHTjtDf8cCS4jTyGrkIUQ0SDBoDVRHiZKdzfeUc0cJpL6REKIaJBg0JoolrFuuvoYAoaJpGcghIgCCQatcSWBzRGdnEFLwaA+gSwLz4QQ1pNg0BqlolaSoqCicSkKgHinHaWkZyCEiA4JBsFEYRWy1pqiFnoGNpvCHeOQnIEQIiokGAQTlwaV1u5pUFpVR51PNwsGILudCSGiR4JBMFHoGRRU1ACQ7m4hGEgZayFElEgwCCYu1fIEckMpClezc7LBjRAiWiQYBBOfbiSQLdwEvKVSFCbpGQghokWCQTDxaeCrBU+5ZY9oqUidKVFyBkKIKJFgEEwUViEHCwbSMxBCRIsEg2CisAq5oNxDQoydWGfzPRPcLqf0DIQQUSHBIJj6noF1SeTCihrSWphJBP4EsqcOn8+6nIUQQoAEg+Dqy1hbt9agoMLT4kwiMPZB1hoqa72WPV8IIUCCQXBRGCYqbKEUham+PpEMFQkhLCbBoAUr8o6z/VApxKYYByxOILeUPIaGrS+lWJ0QwmqWBQOl1D+UUkeVUltaOa+UUk8qpfKUUpuUUuOtaku47n9zE0988jXYHRCbbFnPQGvdYpE6k9kzKJOegRDCYlb2DF4ELgpy/mJgsP/X7cAzFrYlZLVeH4dKqijwb0Vp1CeyJoFc4fHiqfO12jNIdMk+yEKI6LAsGGitlwDBvlLPBF7WhpVAilKqp1XtCdXhkmp8umH+f/0qZAuYex+3OkwkOQMhRJR0Zs6gN7A/4HW+/1gzSqnblVJrlFJrjh07Zmmj8ouqACisNIOBdcXqghWpA0iIka0vhRDRcUIkkLXWz2qtc7TWORkZGZY+K7+oEoDiylrqvD5Ly1gHK1IHDfsgS89ACGG1zgwGB4A+Aa8z/cc6ldkzACiqrLW4Z9B6kTqABMkZCCGipDODwULgZv+soslAidb6UCe2B2gaDDxGz8BTDnU1EX9WsLpEAE67jVinTYKBEMJyDqturJT6FzAD6KaUygceApwAWuu/AouAS4A8oBL4tlVtCceB4krsNoXXp40ZRfEBxeqSIpvfLqzw4HLYiI9pXpfI5HY5ZWqpEMJylgUDrfWcNs5r4C6rnt9e+UVVDO7uJvdwmfHNPXAVcoSDQUG5scZAKdXqNYmywY0QIgpOiARytNR5fRwqqWZMZjLgn1FkYRnrYEXqTG6Xg/JqWYEshLCWBIMAR8pq8Po0o3v7g0G5x9L6RIVBitSZZE8DIUQ0SDAIkF9oTCvN6pZAYqzDSCDXVy6N/CrkYKUoTO5Yh+QMhBCWk2AQwJxJlJkaT3pCjDH109JhotaL1JkSpWcghIgCCQYBDhQbwaBncixpCTEUVtSAMxac8VAV2YVn1bVeKj3eNoOBWxLIQogokGAQIL+oku6JLmKddn8w8Cdu49Ii3jNoa8GZye1yUFFThzH5SgghrCHBIEB+URWZqXEADT0DgPjUiOcM2ipSZ3LHOqj1amrqfBF9vhBCBJJgECC/qIreqfGAUS+oqKLW+EYenx7x2URtFakzSRlrIUQ0SDDw8/o0h0oCewZOPF6f8SFswTBRW0XqTFLGWggRDRIM/I6WVVPr1QHBwPiQrl+FHOGeQVt1iUxulxOQnoEQwloSDPzMaaW9Uxp6BuD/0I5Lg6pi8Hkj9ryCCg9OuyIpNnhFEHMfZFlrIISwkgQDvwMBawyghZ4B2ggIEVJY7iE1PnhdIgjY00B6BkIIC0kw8DM3tTGHicwpn0Yw8K9CjuBQUUEIC86goWdQXiP1iYQQ1pFg4JdfVEU3t7HGACA1MBhYsAq5sKKmzZlEIAlkIUR0SDDwM6aVxtW/ToixE+Ow+XsGqcbBCPYMQilSBwE5AxkmEkJYSIKB34HihmmlAEop0uJjLOsZhFKkDsDlsOGwKekZCCEsJcEA8Pk0B4oaBwMwVyEH7nYWmVXInjofZdV1IeUMlFJSn0gIYTkJBsCx8ho8Xh+ZKY2DQbo7xtjgxpUENkfEhomKKkNbY2AyNriRYCCEsI4EAxqXrg6Uag4TKRXRVcgF5aEVqTO5XQ7JGQghLGVpMFBKXaSU2qGUylNK/bSF832VUouVUuuVUpuUUpdY2Z7WNJ1WakpLiKkvKBfJVcihrj42JcZKz0AIYS3LgoFSyg48DVwMjADmKKVGNLnsF8AbWutxwPXAPKvaE0z96uMWgkFZTR2eOp+/ZxCZPQ1CLVJnkq0vhRBWs7JncDqQp7XerbX2APOBmU2u0UCS/8/JwEEL29OqA8VVpCXEEB/TuDSE+c3d2P4yLWIJ5FCL1JncsU4JBkIIS1kZDHoD+wNe5/uPBXoYuFEplQ8sAu5u6UZKqduVUmuUUmuOHTsW8YbmtzCTCBrG9AvKI1usrrDCg01BSpwzpOvdLtkHWQhhrc5OIM8BXtRaZwKXAK8opZq1SWv9rNY6R2udk5GREfFG5BdVthgMUgN7BmYCOQI7jhVUGHWJbLbgdYlMibEOKUchhLCUlcHgANAn4HWm/1ig7wBvAGitvwRigW4WtqkZrY01Br1TgvQMzLUGvlrwlHf4mYXlodUlMrldDqprfdR6ZbczIYQ1rAwGq4HBSqn+SqkYjATxwibXfAOcC6CUGo4RDCI/DhTE8XIPNXW+ZtNKIaBnEOFVyIUhFqkzmSUpKiRvIISwSEjBQCmVYA7fKKWGKKWuUEoFHfDWWtcBPwA+ArZjzBraqpR6RCl1hf+y/wfcppTaCPwLmKujvPN7a9NKAX+J6YCeAUQkiVwQYpE6k1msTvIGQgirBN9ZpcESYJpSKhX4GONb/2zghmBv0lovwkgMBx57MODP24Cp4TQ40lpbcAZgtylS4pwUVtREtIx1uD0Dcx/kCo8EAyGENUIdJlJa60rgKmCe1vpaYKR1zYqeA8UtrzEwpSbEUFRRGzBM1LG1Bl6fpriqNuRppSBlrIUQ1gs5GCilpmD0BN73H7Nb06Toyi+qJCXeWT8u31R6QoyxSMwcJupgz6Co0oPWoZeiACljLYSwXqjB4EfA/wDv+Mf9BwCLrWtW9LS2xsCUZvYMYlOMAx3MGYRbigICtr6UnoEQwiIh5Qy01l8AXwD4E8nHtdb3WNmwaMkvqmJgRkKr59MSYli7rxjsDohN7vBsonCL1AG4XUauXlYhCyGsEupsoteUUklKqQRgC7BNKXW/tU2znrnGoKXksSktIYaiSg8+nzaSyB0cJqrvGbRjNpH0DIQQVgl1mGiE1roUuBL4AOgP3GRZq6KksMJDVa036DBRanwMXp82pnVGoIx1ob9IXTjDRPFOO0pJzkAIYZ1Qg4HTv67gSmCh1roWo8jcCa2+WmkLq49N5nqA+iRyB3sGBf6eQWp86MHAZlO4Y6SMtRDCOqEGg78Be4EEYIlSqh9QalWjosWcVhp8mMiYAtqoPlEHFFZ4SI5z4rSHt/g7wSX1iYQQ1gk1gfwk8GTAoX1KqbOtaVL0mKuPW1tjAJAW36RyaUcTyBWesJLHJtkHWQhhpVATyMlKqT+aZaSVUn/A6CWc0PKLqkiKdZAcpJS0megtNEtS1FZAXU27nxlukTqTlLEWQlgp1LGKfwBlwHX+X6XAC1Y1KloOFFXRO8gQETT0DAorI1OsLtxSFKZE6RkIISwUajAYqLV+yL9r2W6t9a+AAVY2LBraWnAGEBdjJ85pN/ZCjsAq5IIKT1hF6kxulySQhRDWCTUYVCmlzjRfKKWmAlXWNCk6tNatbmrTVFpCTJOeQftWIft8mqLK9g8TSc9ACGGVUKuW3gG8rJRK9r8uAm6xpknRUVxZS4XHG3RaqSktIcafM/BXLm3nMFFpdS1enw6rSJ3JHSs9AyGEdUKdTbQRGKuUSvK/LlVK/QjYZGXjrBTKtFJTQzDINA60c5jIXGPQntlEiS4H5Z46fD4d8naZQggRqrAmu2utS/0rkQHus6A9URNsU5um6oNBBxPI7SlSZ3LHOtAaKmu97Xq2EEIE05FtL0/or6fm6uM+4fQMnLHgjIeq9u1pYBapa1/OwF+sToaKhBAW6EgwOKHLUeQXVeF2OUiKa3ukLC0hhkqPl+par38VcvsSyGbPoF2zicxidbIKWQhhgaCfhEqpMlr+0FdA2+MrXZg5rVSptjs45jf5wgoPvTqwCrk9RepM5taXsvBMCGGFoD0DrXWi1jqphV+JWus2v1IrpS5SSu1QSuUppX7ayjXXKaW2KaW2KqVea+8PEq5Qp5VC42DQkWJ1BRUe3C4HLkf4m8Q19AwkGAghIi/UqaVhU0rZgaeB84F8YLVSaqHWelvANYMxdlCbqrUuUkp1t6o9TR0oqmJS/7SQrk0PDAZxaVC8v13PbO/qY2jY+lJyBkIIK3QkZ9CW04E8/4plDzAfmNnkmtuAp7XWRQBa66MWtqdeSVUtZTV1IU0rBUiNUM8gIsFAegZCCAtYGQx6A4FfofP9xwINAYYopZYrpVYqpS5q6UZKqdvNInnHjh3rcMPCmVYKDT2DArNnUFUMvvCneBaUt69iKQTsgyzBQAhhASuDQSgcwGBgBjAH+LtSKqXpRVrrZ7XWOVrrnIyMjA4/1JxWGmrPICnWid2mKKpfhayNgBCmjvQMEmSYSAhhISuDwQGgT8DrTP+xQPn4d07TWu8BvsYIDpY6YO5wFmLPwGZTpMY7jZ5BO4vVaa2NYNCOaaUATruNWKdNegZCCEtYGQxWA4OVUv2VUjHA9cDCJtcswOgVoJTqhjFstNvCNgFGzyA+xk5qfOv7GDSVlhBj9AzauQq5vKYOj9fX7mEiMBaeyT7IQggrWBYMtNZ1wA+Aj4DtwBta661KqUeUUlf4L/sIKFBKbQMWA/drrdu3oisM5rTSUNYYmFLjzfpEqcaBMHsGDaUowi9SZ0qUYnVCCItYNrUUQGu9CFjU5NiDAdx06JsAACAASURBVH/WGDWOolrn6EBxVUjVSgOlu2PYcbgM4noYB8JchdyRInWmBJddhomEEJbo7ARypzBWH4eWPDalxsdQVFnb7jLWhR2oS2SSDW6EEFY55YJBaXUtJVW1IU8rNaUnxFBU6cHrdIPN0YFhIskZCCG6nlMuGIQ7k8iUlhCD1lBSXecvVhdeMCjoQJE6k7EPshSqE0JE3ikbDMIeJqpfhVxjTC8NM2dQWFFDrNNGfEz70zQyTCSEsMopFwzCXX1sSvfPAiooN1chh7enQUGFp/4e7eWONfZBNvLuQggROadgMKgi1mkLe1ZPaoKxJqGo0r/wLNwEcgdWH5vcLge1Xk1Nna9D9xFCiKZOuWBgTisNZ40BBPQM2lmsLhLBQOoTCSGscsoFg/ZMK4WAnoG5CrmyEMIYrjlYXEWPpNiwnxtIylgLIaxyCgaD0De1CeRy2HG7HA09A18t1JSF9N5jZTUcL/cwrGdi2M8NJGWshRBWOaWCQUVNHUWVtWFPKzWlJcQ0bHADIQ8V5R4uBWBYj6R2Pddk7nYmW18KISLtlAoGB4rbN63UVB8MwlyFnHvI6EEM69GxnkGiyxiqkp6BECLSTqlg0N5ppaaGYBBez2D74VJ6JMXWr1Vor4Z9kGXhmRAisk6xYGD2DCI0TFQZ2lqD3ENlHc4XgCSQhRDWOaWCwYGiKmIcNrq1c/GXGQx0nL+MdQirkGu9PvKOlnc4XwANU0ulPpEQItJOqWCQX1RFZkocNlt4awxMaQkx1NT5qLT7v+WHMEy053gFHq+P4RHoGbgcNhw2JT0DIUTEnWLBoLLdM4mgoeJoYZUPYlNCSiBvPxSZmUQASincsQ4qpGcghIiwUyoYHCiuane+ACAt3ixWF/oq5NzDZTjtigEZCe1+biC3yyHDREKIiDtlgkGVx8vxck+7p5UC9ZvZFwauQm5D7qFSBnVPxGmPzF+1VC4VQljB0mCglLpIKbVDKZWnlPppkOuuVkpppVSOVW05UNyxaaXQQs8ghATy9kNlDO/g+oJAif7KpUIIEUmWBQOllB14GrgYGAHMUUqNaOG6ROCHwFdWtQVgv7mpTZh7Hwdq1jNoo4x1UYWHw6XVEZlWanK7JBgIISLPyp7B6UCe1nq31toDzAdmtnDdr4HfAdUWtoV4p53pQzLom97+YaJElwOnXVFY6V+F3MYwUe5hc+Vxx5PHpgQZJhJCWKD92261rTewP+B1PjAp8AKl1Higj9b6faXU/Ra2hUkD0pk0IL1D91BKkRofY2xu3z0VaiugrgYcLa9bqK9JFMGeQWKsJJCFEJHXaQlkpZQN+CPw/0K49nal1Bql1Jpjx45Z37gg0hJijMql9auQW+8d5B4qIz0hhgx3x3Y4CyQJZCGEFawMBgeAPgGvM/3HTInAKOBzpdReYDKwsKUkstb6Wa11jtY6JyMjw8Imty0tIaZhtzMImkTOPVzKsJ6JYW+kE4zb5aSq1kudV3Y7E0JEjpXBYDUwWCnVXykVA1wPLDRPaq1LtNbdtNZZWussYCVwhdZ6jYVt6rBQy1h7fZodR8oimi+AhmJ1FTXeiN5XCHFqsywYaK3rgB8AHwHbgTe01luVUo8opa6w6rlWSw+xjPW+ggqqa30dLlvdVKLLrE8klUuFEJFjZQIZrfUiYFGTYw+2cu0MK9sSKakJMZRU1VLrSsEJrfYMzJlEw3ta0zOQ6aVCiEg6ZVYgR0q6vz5REW7jQCs9g9xDpdgUDOrujujzpYy1EMIKEgzCZG5QU1RjB2d8q8Fg++EyBmS4iXXaw3tATTms/ydo3eJpt5SxFkJYQIJBmMzKpQUVNUbeoNVhotL25QvWvwLv3gXfrGzxdKL0DIQQFpBgEKZ0/8Y4xoyi1BZ7BmXVtewvrGpfvmDfcuP3vUtbPC05AyGEFSQYhCk1wdiUvihIGeuvj5hlKMLsGWgN+1YYf96zpMVLJGcghLCCBIMwpcabw0RmGevmi862H/IHg3B7Bse/Nu6X0B3yV0Nt83JNCTGSMxBCRJ4EgzA57TaS45wNPYMWholyD5eSGOugV3JseDc3h4im/hDqquFA8/V3NpuSkhQnkLv/tZ7Xvvqms5shRJskGLRDfX2i+HSoLgFf49XAuYfKGN4jKfwyFPtWgLsHjLsRlA32tJI3cDkol0VnXd7Rsmre23iQ11dLMBBdnwSDdmhckkJDVXH9Oa01uYfLwq9UqjXsXQ79zoC4FOgxJmgSWRLIXd9Xu41e4+YDJZRWS/AWXZsEg3ZIjfcHg4RuxoGyg/Xn8ouqKK+pC78mUfE+4z79zjBe95/mzxtUNbvU7XJQJsNEXd7K3UY+yadh1e62t0gVojNJMGiH+vpEffzbM+z+vP5c/YY24fYMzFlE/aYav2dNA68H9q9qdmlirIMK6Rl0eSt3FzB1UDouh40vd7e9RaoQnUmCQTukuY0y1jo5E7qPgK8/qj+Xe8jY0GboaeEGg+XGuoWMYcbrvlNA2VscKpKtL7u+o2XV7DpWwVlDMpjQL5UVuyQYiK5NgkE7pMXHUOvVxvTOwefDN19CtREEcg+X0S89ngRXmDUA962AvmeAzf9PEpsEvbJbTCLLbKKuz8wXTB6QzpQB6Ww/VGrMQBOii5Jg0A5mSYrCcg8MvhB8dbB7MQDb21OGovQQFO5uyBeYsqbBgbXgqWh0OMElW192dSt3F5DocjCiZxJnDDLKnX+1R3oHouuSYNAO9cGg0p83cCXD1x9T5fGy93hF+Mnjb8x8QZNg0H8a+Gph/1eNDif6ZxPpVorZic63cncBE/un4bDbGJOZQnyMXYaKIuB3H+aydGfnbn17spJg0A6NegZ2Bww6B/L+y84jJfg0DG9P8jjGbUwnDdRnMtgczYaK3C4HWkOlR3Y764rMfMHkAcZueE67jZysNL6UYNAhuYdLeebzXTy9OK+zm3JSkmDQDo16BmAMFZUf4cgOY+ZP2D2DfSuMHoa9SZ7B5YZe45slkaVYXdcWmC8wnTEwnZ1HyzlWVtNZzTrhvb3O2EJ99d4iiisl/xJpEgzaoT4YmAnBQecBCseu/xLntNM3LT70m1UWwtFtzYeITP2nwYF1UFNWf8gsVidrDbqmwHyBaYo/MMgU0/ap8/p4Z/0BZidvI8N3nM9yj3Z2k046EgzaIT7GjsthawgG7gzoPZ7M48sY2iMRmy2MMhTffGn8bq4vaCprGmgvfNOQN0iUnkGXFpgvMI3slURirEOGitppad5xzqj4lN/VPMr/xb3Ex1uPdHaTTjqWBgOl1EVKqR1KqTyl1E9bOH+fUmqbUmqTUupTpVQ/K9sTKUqphpIUfnrwBQys3UFORpgf0PtWgN0Fvce3fL7PJLA5YW9DSWu3yyijLdNLu56m+QKTw25jUv80vtx1vJNadmJbveJTfhfzd7TdxVS9nk07d1FdKzmzSLIsGCil7MDTwMXACGCOUmpEk8vWAzla6zHAm8D/WdWeSGsaDAp6zsCG5izbpvButG85ZE4Eh6vl8zHxkJnTKIlcv6eBFKvrcprlC9a+VL+KfPKAdPYWVHKopHmJEdG6soID3LT3Z1Q701Dfeh07Xs7xrmCFBNaIsrJncDqQp7XerbX2APOBmYEXaK0Xa60r/S9XApkWtieimgaDzTqLYzqZEeUtb1fZopoyOLSx9XyBKWsaHNpgVEilYZhIcgZdT6N8QelB+M+P4IMHADhjoFHLSoaKwlDnofqfN5BCOUcufQEGzMCXMZyrHStkqCjCrAwGvYH9Aa/z/cda8x3gg5ZOKKVuV0qtUUqtOXasa8wxbhoMcg9XsNibTdqhpeAN8UN6/1egfW0Hg/7TjOv2GfmFhp5BaM/RWnO0rPlGOVFTcbzFgnsno0b5go3zjX+3g+vg6HaG9UgkNd4p6w1CpTUs+jEZRet5PO4ehow9A5TCNnY249QOtm/bhM8na20ipUskkJVSNwI5wO9bOq+1flZrnaO1zsnIyIhu41rRPBiUsjF2IqqmBPKbF5dr0b4VxjqCPqcHvy7zdCOv4J9imhDm1pevrfqGKf/7GdsOlobWrkiq88DfpsNfz4SifdF/fhQ1yhdoDRtehe4jjX/jDa9isykm9U+XnkGo1jwP617i6borSJ88p2F/kNHXAjC9ejHr9xcHuYEIh5XB4ADQJ+B1pv9YI0qp84CfA1dorU+YSdjpCTGU19RRU2cksXIPlVHc60zjf/yAwnVB7VsBPbMhJiH4dc5YI6/gDwYxDhsuhy2knkFNnZenPsvD69M8t2x3aO2KpO0LofQAFO+Hf1wIR7ZFvw1R0ihfkL8aCvJg8h0w5CLY+Dp4azljUDoHiqvYX1jZxt1OcXuXwwcPsDt1Kn/0XsescQGDCsmZ1PWdypWOFfx36+HOa+NJxspgsBoYrJTqr5SKAa4HFgZeoJQaB/wNIxCcUBOHU/1rDYoqaqmp87LrWDn9evU0qo3u/LjtG9RWGXWH2hoiMvWfBoc2QVURYOQNQqlP9O81+RwqqWZsnxTe23iQI6VRHi5a/Tyk9ofbPjNev3ARfBNGXuUE0ihfsOFVcMbDyFmQfQNUHIW8T+rXG0jyM4jib+CNm9Gp/bmz6k6mDOxOz+S4Rpc4xs5moDrI3i3LOqmRJx/LgoHWug74AfARsB14Q2u9VSn1iFLqCv9lvwfcwL+VUhuUUgtbuV2Xk+4PBgUVNew6WkGdTzOsZxIMvsBYRFa8P/gNDqw19itobX1BU1nTAF2/70EolUs9dT6e+XwX4/um8Jfrx+H1aV5csTe050XCka1G3aWJ34Eeo+DWjyAhA16eCTs+jF47oqQ+X+Cthi1vw/ArwJVoVLZNyID1/2RQdzfd3C4ZKmqNpxLmfwu8Hjad+Qw7im1cPaGFVOOImXiVk9NLP2HXsfLot/MkZGnOQGu9SGs9RGs9UGv9mP/Yg1rrhf4/n6e1Pk1rne3/dUXwO3YdqfENPYPcw8ZY/PAeiTDkQuOCtnoH+1YACvpOCu2BmTngiIW9xjehULa+fGtdPgeKq/jheUPomx7PhSN78OrKfdHbGGf1c0abs28wXqf2MwJC9+HG//AbXmv3rb8+UkZBedcZVWyUL8h9H2pKIftbxkm7E8bMhq8/RFUWMGVgOit2FUihwaa0hnfvgsNb4OrneHVXDAkxdi4c2aP5tXEpeAZewOX2FXyypdnos2iHLpFAPhGluxt6BrmHy4ix2+jfLQG6DYGUfrDzv8FvsG85nDbK2NAmFA6XsQDNv96grZ5BrdfH04vzGNsnhemDjSmN3502gNLqOt5cmx/aMzuiutQYJx91NcQHLMBK6Aa3vGcMey24E5b/OexbHy+vYeZTy7nx+VXUeX0RbHT7NcoXbHgVkvv6e3N+2TcYpc43vcGUAekcLath9/GKVu52ilr+BGx9G859kKqs81i0+TAXj+5JfEzLe4PETfgWGaqUoxtDzNGJoCQYtFNagrFIrKjCw/ZDpQw+zW1MJ1TKGCra8wXUtjI+7601FiKFmi8wZU2DI5uhshC3yxk0Z/D2unzyi6r40bmD62dhTOiXyri+KTy/bA9eq6fkbXodaiuMIaKmXInwrTeM8fT/Pggf/8L4VhiiZ5fspqrWy/ZDpbz0ZdeYoVSfL4gvMbZBzZ7TsFERwGkjoNc42PAqZ/hXJ8tQUYCvP4ZPfgUjr4Iz7+XjbYcpr6nj6vFBlh4NPp9qRxKjCz6UAoARIMGgnZLjnChlFKvLPVzWuFLpkAuhtrJ+SKeZQxuN8+EGg/7+b5p7l/n3NGh5BXKt18dTi/MYk5nMjKGNp+LeNm0A3xRW8t9tFs7C0NoYIuo1DnpPaPkahwuufh4m3gYr/gILvm8EyTYcK6vh5S/3Mmtcb84emsEfP97B4ZJOXEPhV58v2PIGoGHsnOYXZd8AR7bQr3YnPZNjJRiYjufBW/680synQCneXJtP75Q4JvVPa/19DheVg6/gAtsavti8J3rtPUlJMGgnu02RGh9TX5a40R4GWWeCI671vMG+5cbv4QaDXuONGSp7lwYdJlqw/gD7C6u455yGXoHpwpE96JMWx9+XWvg/z77lcCwXJn43+HU2O1zye5jxM9j4Grx+o5FADOJvX+zCU+fj7nMG8asrRlHn0/z6P507XbU+X9A/1ciD9DsT0vo3v3D0NWB3oTa8xpQB6azcLXkDvHXw9m3GfwvXvwYxCRwuqWZ53nGuHt+7zaKPqZNvIF7VULTunSg1+OQlwaADUuOd9SWJG/UMnHHQfzrs/Kjl4Y99KyB9MLi7h/dARwz0nQx7l+GOdVBR07xQV52/VzCyVxLnDm9+f7tNcevU/qzdV8S6b4rCe36oVj8HsSlGl78tSsGMB+DSPxrrM16Z1epq5aNl1fzzq31cmd2bARlu+qbHc/c5g3h/8yG++LrzVqab+YJzEvYY25eaieOm4lJh2KWw+d9MzUqkoMLD10dO8Zkwy/5krNC+9I+Q0heAd9YfwKdhVrAhIj/VZzJFMT0ZevSD6E2MCMWepVB2YpXLkGDQAekJLoorjaGNYU13NxtyARTtheM7Gx/3+Yyy1f2mtO+hWWfC0W10U2V4vL76RW+mhRsPsq+gknvObd4rMF2b04fEWAfPW9E7KDsM29+DcTcaRfZCNfE7cM3zsH+lkUdowd++2E2tV3P3uYPrj902fQADMhJ48N0tnVbF0swXDMh/F5wJMGJm6xePuwGqipih1gCn+HqDQxvhi98aXxpGGV8ctNa8vS6fCf1SjQkZbbHZKB8yi6lqE19t7iILGvetgJcug9euDWnos6uQYNAB5iY33dwuurmbVB0dfIHxe9OhoqPbjIJzoa4vaCprOgADK9YDjUtSeH2apz7LY3jPJC4YcVqrt3C7HHxrUl8+2HIo8ith171szJrJuTX89466GiZ/H1Y922w21tHSav650ugVBH5IuBx2Hp05in0Flcz7fFdHW98uK3cXMLVfHLZtC2DklcYOda0ZcDYk9SZ955v0SYs7dfMGdTXwzh0Qnw6X/qH+8OYDJew8Wh48cdxEzzNvxq40patft6Kl4amtgnd/YPSMD200ej4nCAkGHWCuQm5xz+OUvpAx3BgqCuRfNBZ2vsDUKxti3PQpMb5ZBq41eG/jQXYfr+CH5w5qtVdgmntGFjal+MfyCPYOvHWw5gUYeA6kD2zfPc59CLqPMBLKFQ3fmv/6xW7qfJq7zxnU7C1nDOrGldm9+Ovnu9gd5QVIZr5gdsJ68JS1PkRkstlh7PWQ9wkX9oGv9hRaP7OrK/r8f40vRlf8pdHU47fW5hPjsHHpmJ4h38rRYzj7Y4cw+Miizp9qvPg3ULgLrnvJqKH0xe+MygEnAAkGHWCuQh7Wo4VgAMZQ0b4vjTn3pn3LIblP/fho2OxO6DuF0wqNYGCWsfb6NH/5bCdDT0vkghEBi3R8Xlj5V3i+cV2gnslxXD62F2+s3k9JVYS6sjsWQdnBthPHwThj4aq/Q3UxLLwHtOZoaTWvfrWPWeN6k9XK0MHPLx2By2njl+9uiWpS1swX5JR8CKlZ0DeEIJ99A2gfs+xLKamqZfuhTigg2Jm++cpYXzLupoZFmhgr5hduPMj5I04jOc4Z1i3Lh1zFSHazZePqSLc2dPlr4cunYMJcGDADLv4/o+ez4E6jYGMXJ8GgA1Lrg0FSyxcMvgB8tca8czCSyftWtL9XYOo/DXfZLjIoru8ZvL/5ELuOVXDPuYMbZmAc2wEvXAwfPgAH18OLlxhlMPy+c2Z/Kjxe5q/6pmPtMa1+DpIyYfCFbV8bTI9RcN7DsON9WPcS8z7f1WqvwJSR6OInFw5leV4BCzce7Njzw7BydwFDXEUkHlwBY7/VeG1Ba9IHQt8pDD20ENCn1lCRpwIW3GH8d3LhbxqdWrzjKEWVtVwTxhCRqd9ZN+HVirJV7V/V3iF1Ncbq6cSecP4jxrH4NLj8z3BkCyxpsSBzlyLBoAN6p8QCMKp3cssX9JkEruSGoaKCXUbBso4Gg6wzAZhs20Z5dR0+n+Yvn+5kcHc3F4/qYSStljxulI0+/jXM+hvc9RW4kuClK+pXMY/qncyUAem8uGIvtR3tXh/72lhol/NtsLe8YjQsk+6EATPQH/yUFatWcvX43vRLD55Q/NakfozJTObR97dTWh2dxN3K3QXcmeovWT72+tDfmH0DjqI8LkvLP7WSyJ88bMy4uvJpiG38Jeqttfl0c7uY5l8xH4749Ey2x41n4JFFaF94EwkqPXUd700ueRyObYfLnoDYgM+DoRcbXxKW/sH4QtaFSTDogPNH9OCd75/B0NaGiexOGHSOkQzVOmB9QTuTx6YeY/HFJDLFto3ymjo+2HKYnUfLufvcwdiObIK/nwOf/dr4D/GuVcaHVFp/uPVDSM6EV6+pL7N92/T+HCqp5v1NhzrWpjX/MPZqHn9zx+5jstngymeo1g5+b3uaH0zPavMtdpvisStHU1Bewx8+2hGZdgRxtKya3cfKOLf6E2MqcWoYW3iPvBKc8dwSu4zVe4s6f6w7GnZ/bkwOmHSn8fcVoLDCw+IdR7kyu5exkr8dSodcTS99lG82Lg75PSvyjjPpN58y94XV7Z+NdmgTLPujsdBwyAXNz1/0v8Y08nfuNHoQXZQEgw6w2xTj+rZRW2jwBVB+xJhZsG+FUb0yvfXhjtAe7KA2czKTbdsora7lyU93Mqybk8uOPQfPnm0877pX4LqXG69lSOoFcxc1FIrb/CYzhnRnYEYCzy3b3f5vR54KY7HViJnhr50I4rBO4yee7zDWtou+W/4S0ntGZyZz85QsXlm5j835JRFrS0u+2l3IRLWDpOoDDcX4QuVKhBEzyS79jLqaCjYfsLatna66BBbcZayvOe+hZqff23iQWq/m6gnt3/l28FnXU6ldlH71akjXv7vhALe8sIqkWCdLdh7juy+tocoTZkDw1hrDQ3FpzYa96sWlGInyY9uNxHkXJcHAaoPOB5QxxdTMF7Qx0ycUtv7TGWA7zOLVG0k4upZ/8wC2ZX8wegF3fQUjWikAm5AONy80hrDe+i62dS/ynTMHsOVAKSv9ydCwbX4Tako6ljhuwTOf5/GBdxIVw2cb3ewQ90G474IhpLtd/HzBZktn6qzcXcC3YpaiYxJh+OXh3yD7Bpx1FVxkW33yb4X54c+MyQWz/mosymzirXX5jOiZxPCereTfQpCRns7q2ClkHfk46DdwrTXPfL6LH87fwIR+qSz64TR+f81Ylu86zrdfXBXe4rXlf4bDm+CyPzYuyNjU4PONhPnyP0P+mjB+quiRYGA1dwb0Hg/rX4GSbzo+ROTnGGjUKbrz+GO86foVbrsHbnwLrpzXdiXU2CTj2sHnw39+xLWet0lPiOG5pe3YCU1rWP13Y3vHvpPb8ZO07FBJFf9atZ9rJmSScOUfjNlXb9/WeGZWK5JinfzyshFsyi/h1a+sK2S3YVc+F9lWokZe2fZudS3pNxVS+nFL3DJW7j6Jg8GOD2DDP+HM+4xS7E3kHi5lU34JV40PtkV6aMqGXEWiLqdww/stnvf6NA8t3MrvPszl8rG9eOnW00mOc3LNhEyemJ3Nqj2FzH1hVWj7ix/NNaaOjpwV2peBCx+DxF7G7KIuuCe4BINoGHyBsXsTdDx57Kd6jKGEBE637WB31hzU91fCoPNCv4EzDma/CiNn4fz0IZ7uuYhPc4+QdzTMefr5a+DwZmMFcQR6PKZ5i3fh05q7zh5kDKnMehZK8uGDB0J6/+VjenLmoG78/sMdHC2LfCG7o2XVDC9aTKyuDn+IyGSzQfYNjPNu4uDeXDx1J2HeoKLAmCJ82mg4q/G/nc+nefWrfcz+20riY+zMzO54MBg2dSbHdRJlq5sPFVXXernzn2t5+ct9fG/6AP48OxuXw15/fmZ2b56cM4513xRz8/NfBZ+E4PMaw0Mxbrg4xJlCsclGIb7jX8Pix/j6SBk3Pf8VY3/1MfM+z2tWTSDaJBhEg7kaOTbZWFAVCTY7P3P9jDti/4+sm542PjDD5YgxKoeOv5nJ+f/gkZhXeGFZmKt4Vz8HMYkw5roWT+cdLedP//2al1bspbgytLnWB4ureH31fq7N6UOfNH9Ji76TYPr9RkG7rW0XJVNK8cjMkdTU+XjkvW0dny3VxFe7C7nGvoSapKyO9Yiy56BRXOb7gg2hbu5eVQQ1J0BNI63h/fuM9s76q/Hfm9/m/BJmPbOCn7+zhWE9Enn3rqlkJLqC3Cw0g3qksCRmGj2PfAFVDX+fhRUevvX3lfx3+xEevnwE/3PJ8BaL4F02phdPzRnHpvwSbnp+VetrcFY+AwfWGIUW3RktX9OSgWdTkz0XveIpfvHkc2zcX8yo3kn834c7uOiJpSzO7bzdfyMwB1C0qWe2Mf+413hjBWqEXHPVdWS4Xe2efQEY7bn8SXAlcfOXT/HWhioKzvsX6UkhDHtUHDc2Ixl/S6NgVFJZy8JNB3lzbT4b9xejlPG58Nii7VwyqgezJ/Zl8oC0VldJz/s8D43mrrObrGKefj/kfQLv/QgyT4fkVr5Jemvh+E4GHNnCa1lL2bNtL7/8dTb2YRdzbvZApg7q1ugbYXvs2L6Jy23b8Y7/ecd6RCl98fabxtV7l/BO3jFOb61k8/E8Y91F7vvGXhhoY5Fb95HGhIDTRhh/Th9ozGLrCra8BdsWGKvKe4wCoKSqlj98vINXVu4jPcHFE7OzmZndq80V8+EoHXwVMdvep2rTO8RN+jbfFFQy94VV5BdXMe9b47l4dPDVzReP7sk8m+Ku19Zxw3Mr+ed3JpES3xDIKNjln613iVFCJUR1Xh//WvUNz2w8h9f1+8xLeA7b95eTlpLC5zuO8sh72/j2i6s5d1h3Hrx8RJtTqSNNnWgldHNycvSaNV0zARPU8Z3GB2ZiC1v4dQVaU/DBo6SvepwCWzdKBizkiAAADm5JREFUUkdh6z2O9METSew/seVZQsv+ZMwb//5K6tKHsmTnMd5ae4D/bjuCx+tj6GmJXDMhk5njenG0tIbXV+9nwYYDlFXXkZUez+yJfbl6Qm+6J8bW3/JAcRUzfr+Ya3P68JtZo5s/s2AX/HWaMfZ80wKoLDAW9RzZYuy5fHiLUT7bZ3yj0/YYau3xxHiK8eBgmXcUn9sm4xtyMdOzhzF9SAaxzvADw8u/uZ0bPW9gu3eLMV23Iza9AW/fxiPpv+PBu+8wjvl86PzV+HLfh9xF2AuNgodV6SMp6nMuifFxJJZ8bawqL8gD7R9isMcYu+11H2EEiIzh0G2wsfteJNZ/BFNbZUyS2PUZ7FoMR7dC5kT49odom5131h/gN4u2U1jh4eYpWdx7/pCwVxqHYs2eAtJeOIPkbr04fNnL3PJaLrVezfO35JCTFSTJ28RnuUe445V1DOzu5tXvTjJqkfl8RhG6w1uMiRpJoZXNWJF3nF+9t40dR8qYPCCN340vod9/ZhvTbC/+LWCswH5h+R6e/HQntV7NbdP7c9fZg1rd6S1cSqm1WuvmSRvzvJXBQCl1EfBnwA48p7X+bZPzLuBlYAJQAMzWWu8Nds8TNhicID6c/zTOnR+QVZvHQFvD2oMiezeKkkegemWTNuh0kgdMgH9cSGVcL/6U+SfeWX+Q4+U1pCXEcMXYXlwzIZORvZKafeOr8nj5YMsh5q/ez6o9hThsinOHd+f6iX2ZPiSDB9/dwhtr9vP5/WfTO6X5rBPAKIa38G6jGFh1wNCKu4fxDfS0kcYY9WkjjQ9CZYf81Xi3LcSzeQFxFfl4UXzlHc5iNYmqgRdzxvgxzBiaEdL/eEdLK/H8YTSkDSLzhxHYctFTSfXvBvFfzxiWxp/LFM9KzvStIUMVU6dtrPQN57++HD7xjucADUMSpyW5GNcnlQm945iUVMhQ2ze4CnKNAHF0O5QGbG9qc0LaAOPvI32Q//fBxu/BZsEEo7XxnF2fwa5PjUBQV20EpL5TYNC5kH0jO8pi+OW7W1i1p5DsPik8euWo1hdqRoDXp5n36F3c7TPyBgfUaST3n4C733joMRp6jjF66iH0Rr74+hi3vbyGAd0S+Od3J9Ft28uw6Mcw82mjMm8bvimo5LFF2/ho6xEyU+P4xaXDuXBkD+P/i0X3G+su5r5fv5AU4EhpNb/9IJd31h+gZ3IsP7tkOJeN6dnh3lOnBQOllB34GjgfyAdWA3O01tsCrvk+MEZrfYdS6npgltZ6drD7SjCIjpKqWnL35nP069XU5a8nsWgLWZ48BqhD2FTDfzN3ee7hI6ZwzrDuXD0hk7OHdifGEdqw1a5j5byxej9vrs2noMJDz+RYjpfXcF1OHx5rqVdg0ho+fcQol13/4T/K2F+5LVrD4c14t75L9eYFJJTkAbDeN4hPOJ2K+ExcvmpcuhqXrsFFDS5dQ2zA6yRdxhQ28c2MP9N3xtyQfta2lLzxfZK3GR9eNbY48pKnsCf9LA5lTEPFpxIf4yAuxkac047LYWdfQQUb9hezfn8x+wqMyrN2m2LIaYlk90lhXN8UJnRX9Nf52ArzjKTl8Two2Iku3IPyNYyF67g0dPogdEJ3Y2KBMx5iElDOeKMMuTMeFROPikkwznkqYPdiIwiU+b8wdBtqfPgPPMeYJRUTT0VNHX/+dCf/WLYHd6yDBy4axuycPm1uWBMJP/33Bg6uX8S5yYeY3aeY2ONbjJXPpvh06DHGCA49xhj5PF9dwC9v/Z93Hi7mtZW76R6n+J7vX9j6ToYb3w4aTMpr6pi3OI/nlu7BYVfcdfYgvnNm/8a9UE8FPDPV+EJz2igjKMenG7/i0thd6eLF9aVsKLCT1SeTH1w2iSF9erV7WLIzg8EU4OH/397dx9ZV13Ecf39uu7GuM+WpPDg6BmyDRMENEYIiAok6NAYfJg9RggYD8qAzJjL0D0Wi0eBDzIRgRpiZis5F2NgfBFgG4SES2MCWDYY8ZbiVPtKUrdvatfd8/eP3a3dX7u1d197enXu/r6Q55/x6dvb79tfe7zm/c87vZ2afj9s/BjCzX+Xs81jc5zlJtUA70GhjVMqTQfns7h/ktXfepeP1zQzufIlkby+7z1/KlxadynGjh/Aeh/1DCRu3dbB60w5ea9/F2ps/xYcLXRVMtq7Xyb76MPta1jGrZ2veXQY1ncHMUQxqRlhmZrB35smcduMaMkeNY86GsfTuCI/ozv10eDu39tB/nu/1DdCys5fm/4Xk0Lyjd2QAw9qMRu7ZJGYkBjVkaVInp6stfr3LGZk2GtjDTPqp0wB17GcmAwcl/oOqa/U8m5zNM8nZPJOcQxvHAfnncrr6E03ctviskSHfp8L27j2s2byDmy+dx6yj4tXewO7QvdO+BdpbwrJzG2QPfRC5bhq4pe43tNJINjGGEgvLbEJiMJQkZBNjMBt+EF9dNJvbFp/FSQ0z8h+wfQs8dRfs6YK9PaHbc18PWP4HHpqbrmXh9XeP62cxrJzJYAmw2My+E7evBS4ws1tz9tka99kZt9+K+3SPOtYNwA0Ac+bM+fg77xwZk6C7CvN+KwzsOnB2PK0uTF9a6r72SZYkxtvd4crhrTikd0aQkRDhSauMREbhJFPSyMnm8MeBmZEkRiY7QG2yj9psP7VDe6lJ+kmo4b36MzDVcNA5as4Zq+LmxQsaObfYW/rlNLQf3nsDBvvDwxSZ2pyvg7eb3+1j1eYusppGbUbUZERtTVxmMnE5XJ7h0jMbi49QkE+ShJc4h5PD3h729HbwdMtrnHTmhSz6TIEXSosolgxS8VtuZiuAFRCuDMpcHVepGmYDE3/WvdwyGTHvhFnMO2GMSXZcUDs9dDMegoULGlm4IM/c1pMtkwkvjtYdMzIvSD1w+QUl/m9LeOxWoCln+5RYlnef2E3UQLiR7JxzbgqVMhlsAuZLOk3SdOBqYP2ofdYD18X1JcATY90vcM45Vxol6yYysyFJtwKPER4tXWlmr0i6E9hsZuuB+4G/SnoT6CEkDOecc1OspPcMzOwR4JFRZT/NWe8Hvl7KOjjnnCvOxyZyzjnnycA555wnA+ecc3gycM45RwpHLZXUBRzuK8jHA91F90qXSoup0uKByoup0uKByospXzynmlnByRdSlwwmQtLmsV7HTqNKi6nS4oHKi6nS4oHKi+lw4vFuIuecc54MnHPOVV8yWFHuCpRApcVUafFA5cVUafFA5cU07niq6p6Bc865/KrtysA551wengycc85VTzKQtFjSfyW9Ken2ctdnMkjaLmmLpGZJqZsLVNJKSZ1xxrvhsmMlbZD0RlwewdNkfVCBmO6Q1BrbqVnSF8pZx/GQ1CTpSUmvSnpF0tJYnsp2GiOeNLfRDEkvSGqJMf08lp8m6fn4mffPOJVA4eNUwz0DSTXA68BngZ2EuRauMbNXy1qxCZK0HThv9DShaSHpYqAP+IuZfTSW3QX0mNmvY9I+xsyWlbOe41EgpjuAPjP7bTnrdjgknQycbGYvSfoQ8CLwZeBbpLCdxojnStLbRgLqzaxP0jTgWWAp8EPgITNbLelPQIuZ3VvoONVyZXA+8KaZvW1m+4HVwBVlrlPVM7OnCfNY5LoCWBXXVxH+UFOjQEypZWZtZvZSXN8NbCPMDZrKdhojntSyoC9uTotfBlwG/CuWF22jakkGs4EdOds7SfkvQGTA45JelHRDuSszSU40s7a43g6cWM7KTKJbJb0cu5FS0aUymqS5wCLgeSqgnUbFAyluI0k1kpqBTmAD8BbQa2ZDcZein3nVkgwq1UVmdi5wOXBL7KKoGHEK1Erox7wXOANYCLQBvytvdcZP0izgQeAHZrYr93tpbKc88aS6jcwsa2YLCXPNnw+cNd5jVEsyaAWacrZPiWWpZmatcdkJrCX8EqRdR+zXHe7f7SxzfSbMzDriH2sC3EfK2in2Qz8IPGBmD8Xi1LZTvnjS3kbDzKwXeBK4EDha0vBslkU/86olGWwC5se769MJcy2vL3OdJkRSfbwBhqR64HPA1rH/VSqsB66L69cBD5exLpNi+EMz+gopaqd4c/J+YJuZ/T7nW6lsp0LxpLyNGiUdHdfrCA/KbCMkhSVxt6JtVBVPEwHER8X+ANQAK83sl2Wu0oRIOp1wNQBhLuu/py0mSf8ALiEMt9sB/AxYB6wB5hCGKr/SzFJzQ7ZATJcQuh8M2A7cmNPffkSTdBHwDLAFSGLxTwj97KlrpzHiuYb0ttE5hBvENYQT/DVmdmf8jFgNHAv8B/immQ0UPE61JAPnnHOFVUs3kXPOuTF4MnDOOefJwDnnnCcD55xzeDJwzjmHJwPnRkjK5oxa2TyZo9tKmps7kqlzR5ra4rs4VzX2xVf6nas6fmXgXBFx3oi74twRL0iaF8vnSnoiDm62UdKcWH6ipLVxfPkWSZ+Mh6qRdF8cc/7x+LYokr4fx9d/WdLqMoXpqpwnA+cOqBvVTXRVzvfeN7OzgbsJb7ID/BFYZWbnAA8Ay2P5cuApM/sYcC7wSiyfD9xjZh8BeoGvxfLbgUXxON8tVXDOjcXfQHYuktRnZrPylG8HLjOzt+MgZ+1mdpykbsJEKYOxvM3MjpfUBZyS++p/HC55g5nNj9vLgGlm9gtJjxImxFkHrMsZm965KeNXBs4dGiuwPh6548JkOXDP7ovAPYSriE05I006N2U8GTh3aK7KWT4X1/9NGAEX4BuEAdAANgI3wcikIw2FDiopAzSZ2ZPAMqAB+MDViXOl5mcgzh1QF2eLGvaomQ0/XnqMpJcJZ/fXxLLvAX+W9COgC/h2LF8KrJB0PeEK4CbChCn51AB/iwlDwPI4Jr1zU8rvGThXRLxncJ6ZdZe7Ls6VincTOeec8ysD55xzfmXgnHMOTwbOOefwZOCccw5PBs455/Bk4JxzDvg/vla8xjC+1bcAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8L-_dup0uCl5",
        "colab_type": "text"
      },
      "source": [
        "Plot the accuracy graph\n",
        "\n",
        "* Training set - Total length of training samples divided by 100 for every trained sample\n",
        "\n",
        "  * `int((2144 * 8)/100) = int(171.52) = 171`\n",
        "\n",
        "* Testing set - Total length of testing samples divided by 100 for every testing sample\n",
        "\n",
        "  * `int((460 * 8)/100) = int(36.8) = 36`\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YKzECNFZvx9u",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 312
        },
        "outputId": "7a995797-5511-4b7d-8227-ea9af093cc18"
      },
      "source": [
        "plt.plot([t/171 for t in train_correct], label='Training accuracy')\n",
        "plt.plot([t/36 for t in test_correct], label='Validation accuracy')\n",
        "plt.title('Accuracy Metrics')\n",
        "plt.ylabel('Accuracy')\n",
        "plt.xlabel('Epochs')\n",
        "plt.legend()\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/pytorch/aten/src/ATen/native/BinaryOps.cpp:81: UserWarning: Integer division of tensors using div or / is deprecated, and in a future release div will perform true division as in Python 3. Use true_divide or floor_divide (// in Python) instead.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEWCAYAAABi5jCmAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deViVddrA8e8NCIjgAoobKpZrbqiopVNZttmulUtW2qY2NWWzVNM07zQzNdNM9ba8zTRZjZotWpo2NWmLZZtl7mbmLiguiAgCKuu53z+eBwTZDggc4Nyf6+I659l/zzn63Oe3i6pijDHGFArwdQKMMcbULxYYjDHGlGCBwRhjTAkWGIwxxpRggcEYY0wJFhiMMcaUYIHBGD8gIj+KyEhfp8M0DBYYTL0kIstFJE1EQnydltogIiNFREVk0SnrB7jrl3t5ntki8lhl+6lqH1X16pzGWGAw9Y6IxALnAgpcXcfXDqrDy6UA54hIVLF1k4FtNXWBOr4f00hYYDD10S3Ad8BsnAdlERHpJCLvikiKiKSKyAvFtt0pIj+JSKaIbBaRQe56FZFuxfYr+pXt/nJPEpEHReQgMEtEWonIB+410tz3McWOjxSRWSKy392+2F2/SUSuKrZfExE5LCIDy7nPXGAxMMHdPxAYD7xxyj33EpFPROSIiGwVkXHu+qnAJOABEckSkffd9Qnu/WwEjolIkLvuosLriMjDIrLT/azWuJ+riMgzInJIRDJE5AcR6evld2YaEQsMpj66Befh+AZwqYi0haIH5wdAIhALdATmudtuAB51j22Ok9NI9fJ67YBIoAswFef/xSx3uTNwAnih2P5zgTCgDxANPOOufw24qdh+lwMHVHVdBdd+zU0zwKXAJmB/4UYRaQZ8ArzpXmsC8E8ROUtVZ+J8Rn9X1XBVvarYeScCVwAtVTX/lGv+0t1+Oc5ndRtwHLgEOA/oAbQAxuH9Z2gaEQsMpl4RkZ/hPJDfVtU1wE7gRnfzUKAD8BtVPaaq2ar6tbvtDpwH5Cp17FDVRC8v6wH+oKo5qnpCVVNVdaGqHlfVTOBx4Hw3fe2B0cB0VU1T1TxV/cI9z+vA5SLS3F2+GSeIlEtVVwCRItITJ0C8dsouVwIJqjpLVfPdILMQuKGSe3peVfeq6okytt0BPKKqW93PaoOqpgJ5QATQCxBV/UlVD1RyHdMIWWAw9c1k4GNVPewuv8nJ4qROQGIZv4ALt+2s5jVTVDW7cEFEwkTkJRFJFJEM4EugpZtj6QQcUdW0U0+iqvuBb4DrRKQlTgB549T9yjAXuAe4AFh0yrYuwDARSS/8wyk+alfJOfdWsK3Mz0pVP8PJGf0DOCQiM4sFOeNHrGLK1Bsi0hSn+CLQLe8HCMF5KA/Aedh1FpGgMoLDXuDMck59HKfop1A7IKnY8qlDDP8K6AkMU9WDIhIHrAPEvU6kiLRU1fQyrjUH5xd5EPCtqu4r/46LzAV2AK+p6nEROfW+vlDVi8s5trzhkSsaNrnws9pU6iDV54HnRSQaeBv4DfD7ipNvGhvLMZj65FqgADgLiHP/egNf4RSzfA8cAJ4QkWYiEioiI9xjXwF+LSKD3UrUbiLSxd22HrjRrXS9DLdYqAIROPUK6SISCfyhcINbtLIEp5y/lVvBfF6xYxcDg4D7KF0sVCZV3e2m6XdlbP4A6CEiN7vXaiIiQ0Skt7s9GTjDm+sU8wrwZxHp7n5W/UUkyj3vMBFpAhwDsnGK2YyfscBg6pPJwCxV3aOqBwv/cIo3JuH8Yr8K6AbswfnVPx5AVd/BqQt4E8jEeUBHuue9zz2usBhmcSXpeBZoChzGaR219JTtN+OUx28BDgEzCje4ZfoLga7Au97euKp+7RZFnbo+E6dSeAJOpfRB4G84OSmAV4Gz3GKmyu6r0P/i5AY+BjLcczTFqYh+GUjDqeBPBZ709h5M4yE2UY8xNUtE/gfooao3VbqzMfWQ1TEYU4PcoqfbcXIVxjRIVpRkTA0RkTtxKnaXqOqXvk6PMdVlRUnGGGNKsByDMcaYEhp0HUPr1q01NjbW18kwxpgGZc2aNYdVtU152xt0YIiNjWX16tW+ToYxxjQoIlLhcDFWlGSMMaYECwzGGGNKsMBgjDGmBAsMxhhjSrDAYIwxpoRaCwwi8m93isBNxdZFulMUbndfW7nrRUSeF5EdIrJR3CkZjTHG1L3azDHMBi47Zd1DwDJV7Q4sc5fBmdCku/s3FXixFtNljDGmArXWj0FVvxSR2FNWXwOMdN/PAZYDD7rrX1NnfI7vRKSliLS3aQWNMafF44Gdn0HGPuhzLYS2OP1TFhTww5fvkpy0m21Ro8gNCq9w/4AA4fwebRjYudVpXxsgZX8COz98nvbn3kKXnnE1cs5T1XUHt7bFHvYHgbbu+46UnIowyV1XKjCIyFScXAWdO3euvZQaYxqu7AzY8BasfAmOuLOYLv0txE2EoVOhTc8qnzIjPZXNH75IzPbXGeA+xobveJp3Pecyp+ASdmnHMo9ThWc/3c6AmBZMGRHL5f3aExIUWKVrq8fD1tXLOPbVP+mf8QVD8bBqQ7taCwy1Ooiem2P4QFX7usvpqtqy2PY0VW0lIh8ATxRO7C4iy4AHVbXCbs3x8fFqPZ+NMUUOb4fvZ8L6NyE3C2KGwLDp0KorrHoFNi2Aglw480IYOg26XwIBFZeoJ25dz4FPnqdfyn9pJtlsadKbrLjbGdA3jiZr/13ynMOmQ7eLS5wzKyefRWuTmL0igZ0px2gdHsyNw7pw07DORDcPrfDaOdnH2Lh0Fi1/mEX3gh1kEMbmttfQ6dL76HhG7wqPrYiIrFHV+HK313Fg2AqMVNUDItIeWK6qPUXkJff9W6fuV9H5LTAYY/B4YMen8P1LzmtAE+h7HQybCh0Hl9w3KwXWzoZVr0LmASdgDJ0KAyeVKGbyFBTwwxcLkO9n0j97NbkaxIaWo2g58h66Dzyv9DnXzIbVFZ9TVfl6x2Fmf5PAZ1sPESjC5f3aM2VELAM7taT4XN+H9u1m15Ln6JG0kEgySAzoxMFet9B39FSaRbTkdNW3wPAkkKqqT4jIQ0Ckqj4gIlcA9wCXA8OA51V1aGXnt8Bg/J3Ho3hq+v+wKhR7SNXMOT3OX03KyUI2ziNg9cvIkV1oeDs8g29DB02G8OiKjy3IQ7Z8QMCql5C9K9EmzdABEzh21kR+/O4jYra/ToweIIVW7Ogynh6X/4KotjGVnpOf/uMUX+1dCU2anSy6ijyzxK6JR44x97tEFqxOIjMnn/4dW3DLOV3oWbCNE1//k/4ZXxKIhw3NziHo7On0/dlVSCU5m6rwWWAQkbdwKppb40xY/gecuXbfBjrjzCk7TlWPiBMqX8BpxXQcuLWyYiSwwGAaiNSdTjHGxvnQpjcMmwY9L4fA6lfxJRw+xpxvE4oeLDVlmPzEC8HPs93TkdkFl/KpZzCe02i82FUOcEvgx1wf+CURcqLG0lncGk93ZudfylLPUPKqUW3aR3YzJfAjrg78lhDJA2BLk7M4Fnc7/S++mSbBIZWcoQz718HKmSeLmaqgpoqLKuLTHENts8Bg6i2PB3Z97vx63P4xBARCj8vgwEY4ugdadIIht8OgyRAW6eUpla92HGb2N7v5fGsKTQKdoohubSpuFeOtXskfcOH2x8gMaU+g5hKRk8zRkA780OF6Nre9hpwmXrboUQ9d0r5lwP75dElbQYE0YXubi0hr2qVG0nlSAHtanc2hiLNq5GyhuWl0S1tOzFnn0D3u3Bo5J1kp8OO7TmV4BRRlT+pxDgdE0mvULTVSXFQRCwzG1KWcTNgwzwkIqduhWTTE3wbxt0JEO/AUwNYlTnn47i8hKBT63eDkItr1K/OUWTn5LFyTxJwVCew6fIw2ESFMGtaZG4d1Jjqi4spLr3g8sPwv8OWT0PV8GPcaBIfD1g+d+0j8GoKaQv9xTjrb9in7PNkZTqXv9zOdlkDh7ZzgN3hK5UU7pk5ZYDCmLqTuhO9fhvVvQE6GU+k5bDqcdQ0ElVMUkbzZCRAb5kP+CejyM6fCtOcVEBjE7sPHmLMigQVrksjKySeuU0tuHRHL6L7tCQ6qofLmvGxYfJfzq3bgzXDlMxDYpOQ+Bzc56dz4NuRnQ+y5J4vDAgLLaAk01Nne+2oICq6ZdJoaZYHBNExHdkHCNxA3qdLmhF479BNsWlhpJeix3AK2J2dyIq/Aq9NGH9vGGenf4pFAtkZdxNp2N3Agoq/XyQrNO0q/lPeJO7iAljkHyAhuy4qQn7EjLZ8AEbpHh9MvpgXtymzaKHDG+c7DuqoVxscOw1sTIel7uOhRGDGj4nMcPwJrX3PqS47uhRadIbIr7P4CAoOdlkBDp0JHG9GmvrPAYBqeE+nw8gVOcOh1JYx9GYLDTu+c2z6GBbdC7jHnV24ZVMGjiqeK/yVSac78ggt5y3MRh6l+2XAAHi6UtdwSuJQhAVsJECVAhAof954CQCH6LOdXer9x3n1WKdvgzRsg8yCMecnpFeytgnzYtsQpZjq6F+JucouLyp0p0tQzFhhMw+LxwPxJToVt/O1OEUWHgTBxHkS0rfz4snz/Mix5ANr2hRvnQ/MORZty8z0s2XSA2SsSWLcnnWbBgdwQ34mbz+nCmTVUqVur8k7ADwuch3TyDxDaEgbdAkPugFblVPbu+gLevtn5lT9xHsSU+3wwjZQFBtOwfPU0LPsTjP678wt4y4ew8HYIi4Ib34a2VWiB4imAj34HK1+EHqPhulcgxHnYH8rM5s2Ve3hj5R5SMnPo2roZk8/pwnWDY4gIbVLJieshVdjzLaz8F/z0AaBOHcDQqdD1vJNFROteh/fvg6huzudZXvAwjZoFBlN9tdHRqSI7P4fXx0Kfsc5DvPDa+9fDWxMgJwvGzYZuF5U4rMCj5HtOqTfIzSJo0VQCty8lf+g0Ci56DAIC+elAJnNWJPDBxv3kFSgje7ZhyvBYzuvehoCAOrzX2nQ0yenZu2Y2nDji9p2YCul74ev/hTNGwg1zoGntNok09ZcFBlM9R/fB7Muh/3i44OHTPl1hh6xF6/bRtXUzpgw/pXVN+l6Yeb7TvPPOZRDcrHR63hwPhzbD5U/CkNtLtdop1JYjvBr8FL0lkUfzJzO34JISpwoPCeL6wTHcck4XzmgIxUXVlZftVLav/Bcc3OisGzQZrni6dMsj41csMJiqy891gkLSKmd53Fw46+oqn+bUDllBAcJFvduy5WAGCanHaRMRwk3DunDj4La0eecap9nj1OXQulvZJ8zJRBfcjmz/iKUR1/HzlDEEBgZyeb/29GgbAUDrrK1c8cMMgguy+Kj3E+yJGlHiFK3Dg7mifwfCQ+p6YGEfUoU930HWQTjr2rrNBZp6qbLA4Ef/O4zXPnrYCQpjX3HK5xf/HKJ7Q+vuXh1+aoes1uEh3DeqO5Pc0SQ9HuWLbSnMXpHAM59uo80XD3Fj4Fp2j3qJruUEhczsPBauOczc/dO4KR9uzVzIJx0OE3HTLKIjo5ydti6FBXc6RSQ3fsxV5XQY8zsi0OUcX6fCNCCWYzAlbZgPi6bCOffApY875dUvnecU8dzxaVHlbVnK6pA1Zbgz/nx5HbIOffVvopfdz6t6NX/OmcDAzi1LFDPtSsnitW8TS3XyuuL4fwj65GFo199pWbP5Pfjot07v4YnzoXn72vqEjGnwrCjJeO/gJnjlIqeD0i3/OTnI267lMHcM9BkD171aoijC41G+2J7CnBUJLHfH77myfwcmD48lrlMllZsHNsKrF0PMEDLHvcPCdQeZ820iu91hH7pHh7NiZ2r559y6FBbc5vRLyMlwegxf93Lp+gljTAkWGIx3CjuV5R6HaV+W7jNQ2Iz0sr/B2dPdop2kEg/yKo3fcyINZo506jOmfVE0lo7Ho3y53Slm2n34GGMHxjBxWKfyz3lgAyy8E3pcAhf9sdzOa8aYk6yOwVTO43HqEdL3wJT/lt2RbMT9kLQG/fh3/HtXC/53SyuO5RYwsHNLnpsQV7XxezweeHea09Lo1g9LDLAWECCM7BnNyJ5eDrrWfgDc8713+xpjvGKBwcA3z8DW/zq5gc5nl9rsFBcdZv6xO3kwfzVXbn2IPT3/zdjzzmZAZcVFZfnqadj+EYx+EjpVOh+TMaaOWWDwdzs/h88eg77XOz2Ni8nMzmPBmiReK1ZctGLws9z4w238Mfdp6PBe1a+3Yxl8/rgzps/QO2voJowxNckCgz87muQMN9G6B1z1XIlK5VUJR5g2dw1HjuWWLi6Kfc5pufTpo07LJW+kbHOHZn7Dafp61bPWnt6YesoCg7/Kz4G3b3Eqf8e/XqIZ6uJ1+3hgwUZiWjXl1cnxDOzcquSxA8Y7/Ry+fQFihpQ/MqfHAzs+cXre7vzs5NDMFz5iLYeMqccsMPirpb+FfWucXs1uxzVV5bll23n20+0M6xrJSzcPpmVYOROtXPoXOLAe3rvbyQG06XlyW/ZRWPeGk0NI2w0R7Z1gMGiKDc1sTANggcEfrX8LVr8Kw+8tGuoiJ7+Ahxb+wKJ1+xg7qCNPjO1fcSujoGBnILaXzoP5N8Odn0HGPre46C3IOwadzoZRv3dm8rKxeYxpMCww+Jt9a+GDGc6MX6P+AEDasVymzV3D9wlH+NXFPbjnwm6IN+X/LTrC9f+GudfCP4ZBRpJTXNTvBme45w5xtXwzxpjaYIHBn2z7CN65FZq1cR7o7rzCt81exb70Ezw3IY5r4jpW7ZxnnA+X/tWZE9iKi4xpFCww+IuVL8HSh06OJRQezcpdqUx7fQ0BIrx5xzDiYyOrd+6zpzt/xphGwQJDY+cpcCqav3+pxFhCi9Yl8cCCjXSKDGPWlCF0ibJWQsYYhwWG+q4g/+RgdlWVk+X0U9i21Bkt9eI/oRLAs59s47ll2zn7jEheuimeFmFWMWyMOckCQ32Vn+PMzfvjYug/zumV3LaP98cf3QdvjYfkzc6MXUPuICe/gAcXrGfx+v1cPziGv4zp5/34RsYYv2GBoT46fgTmTYI9K5z5jTfOh7VznJZEw6Y5k7xXNIrogQ3ONJg5Wc6E790v4sixXKbNXc2qhDR+fUkP7r7Ay5ZHxhi/Y4GhvkndCW/c4AxXcd2r0O96J1CsfQ1WvQLzb4IWnWHI7TDoFgg7pcJ46xJYcDs0bQW3fwRt+7AzJYvbZq/iwNFs/m/iQK4a0ME392aMaRBsPob6JOEbmD8JJAAmvFl6pNOCfNi2xGlhlPAVBDWF/jfA0GnQri989y9nFrP2A5xZzSLa8e3OVKa/vobAAOHlWwYzuEs1Wx4ZYxoNm4+hodgw3xleolUsTHobIs8ovU9gEPS+yvk7uMnpZbzxbSc30aYXpGyBXlfC2JkQ3IyFa5J46N2NdI4MY9aUoXSOCqvz2zLGNDxW8+hrqvD5X53RSjufDXd8UnZQOFW7vnD18/DLzXDxn5xmqSPug3GvoU3CePrjrfzqnQ0MiY3k3Z+PsKBgjPGa5Rh8KT8H3rsHfngb4ibBlc86YxBVRVikExBG3AdAdl4Bv1mwnvc37GdcfAyPXWstj4wxVWOBwVeOpTr1CXu+hQt/D+f+6rTnJ0jNymHq3DWsSUzjgct6ctf5Z1rLI2NMlVlg8IXDO+CN6yFjvzNmUd/rTvuUOw45LY+SM7L5x42DuKJ/+xpIqDHGH1lgqGs5WTBrNGgBTH4fOg87rdNtOZjBnBUJLFq3j/CQIN6aejaDTp1YxxhjqsACQ11b+xocOwS3fVTtoFDgUT7ZnMycFQl8uyuVkKAAro3ryC9GdSOmlVUyG2NOj08Cg4jcB9wJCPCyqj4rIpHAfCAWSADGqWqaL9JXa/Jznekwu4wo3UfBC+nHc5m3ai9zv01kX/oJOrZsyoOX9WLCkE60albFSmtjjClHnQcGEemLExSGArnAUhH5AJgKLFPVJ0TkIeAh4MG6Tl+t2rTAmeXsymerdFjx4qLsPA9nnxHJ7688i4t6RxMUaC2OjDE1yxc5ht7ASlU9DiAiXwBjgWuAke4+c4DlNKbA4PHA189CdB/ofrFXh/yQdJTHP9zMd7uOENokgDEDO3LLObH0bt+8lhNrjPFnvggMm4DHRSQKOAFcDqwG2qrqAXefg0Dbsg4Wkak4uQs6d+5c+6mtKduWwuGtMPYVr5qlLt10kBnz19E8tAm/Hd2L8UM60TLMiouMMbWvzgODqv4kIn8DPgaOAeuBglP2UREpcxAnVZ0JzARnrKRaTm7NUIWvn4GWnaHPmEp2VV75ajd/WfITA2Ja8vIt8bSJCKmjhBpjjI+GxFDVV1V1sKqeB6QB24BkEWkP4L4e8kXaasWebyHpexh+b4WT7uQXePjd4k08/uFPjO7bjnlTz7agYIypc75qlRStqodEpDNO/cLZQFdgMvCE+/qeL9JWK75+BsKinGEvypGZncfdb67jy20p3DXyTH5zSU8CAqzXsjGm7vmqH8NCt44hD7hbVdNF5AngbRG5HUgExvkobTXr4CbY/jFc8AgEl93HICntOLfPXs3OlCz+dl0/xg9pQHUnxphGxyeBQVXPLWNdKjDKB8mpXd88B8HhMPSOMjdv2JvO7XNWk5NfwJzbhjKiW+s6TqAxxpRkPZ9rU1oibFoIZ9/lzKh2iqWbDjBj/npah4fw1p3D6N42wgeJNMaYkiww1KZvX3BmYzv75yVWqyovf7WLvy7ZYi2PjDH1jgWGymyY50yc02lo1Y47dhjWzoX+46FFx6LVqsr/vPcjc79L5Ip+7Xl63ABCmwTWcKKNMab6LDBUJD8HFk2HgCC4+v8gbqL3x658CfKzYcS9JVa/+vVu5n6XyJ3nduW3o3tbyyNjTL1jA+1UJH0voNC0JSyeDp895nRWq0xOpjMfc68roE3PotUrd6Xy1yVbuLRPWx6+3IKCMaZ+ssBQkfQE5/W6V2DQLfDlk7DwdsjLrvi4NXMgOx1+dn/RqkMZ2dzz1jq6RIbx1A0DbGY1Y0y9ZUVJFUlLdF6jusFVz0PkmfDpH+BoEkx4E5qV0bQ0Pxe+/QfEngsx8QDkFXi4+821ZGXn88Ydw4gIbVKHN2GMMVVjOYaKpCdCYDBEtHcGvvvZDLhhDhzYAK+MgpRtpY/54W3I3A8jZhSt+uuHW1iVkMYT1/WjhzVJNcbUcxYYKpKWAC06QUCxVkN9roUp/4XcY/DqRbD7y5PbPB6nQ1vbftDN6av3/ob9/Pub3UwZHss1cR0xxpj6zgJDRdISoVWX0utj4uGOZU5OYu4YWPeGs37rh3B4m5OzEGF7ciYPLtzI4C6tePjy3nWbdmOMqSarY6hIeiJ0GFj2tlZdnHmb35kM7/0cjux0cg8tu8BZ15KZnce019cQFhzEPycNIjjIYrAxpmGwwFCe7KNwIg1axZa/T9OWMGkB/PdX8NXTzrornkYDAnlgwVoSU4/zxh3DaNs8tE6SbIwxNcECQ3kKWySVVZRUXGATuOo5aN0Ddi6DuEm8/NUulmw6yMOX9+LsM6JqP63GGFODrHyjPOluYGhZSWAAp8XS8Hvg5kV8t/c4f1u6ldF923HnuWfUbhqNMaYWWGAoT1GOIdbrQw4ezeaeN9fSJSqMJ60TmzGmgbLAUJ70RAiOKHO47LLk5jud2I7nFvDSTYMJD7FSOmNMw2RPr/KkJTq5BS9/9c/6ZjdrEtN44caBNq+CMaZBsxxDedISKq94dqkqb36/h2FdI7myf4faTZcxxtQyCwxlUYX0Pd5VPAPf7TpCYupxJgztVMsJM8aY2meBoSxZhyD/hNc5hvmr9hARGsTovu1rOWHGGFP7LDCUJd37FklHj+exZNNBro3raDOxGWMaBQsMZUnzvg/Dexv2kZPvYfwQK0YyxjQOFhjKkpbgvLbsXOmu877fS9+OzenbsUXtpskYY+qIBYaypCdAs2gIDqtwt037jrL5QAbj4y23YIxpPCwwlKWwD0Ml5q3aQ0hQAFfbPAvGmEbEAkNZ0suZh6GYE7kFvLduP1f0a0+LpjZVpzGm8bDAcKqCPGdO50oqnpdsOkBmTj7jrNLZGNPIWGA41dEkUE+lOYZ5q/YSGxXGsK6RdZQwY4ypG5UGBhG5SkT8J4B4Mdz2rpQsvt99hPFDOtsIqsaYRsebB/54YLuI/F1EetV2gnzOi+G2316dRGCAcN1gq3Q2xjQ+lQYGVb0JGAjsBGaLyLciMlVEGucQoumJIIHQvOyHfl6BhwVrkriwVzTRETZlpzGm8fGqiEhVM4AFwDygPTAGWCsiv6jFtPlGWgK0iIHAskck/2zLIQ5n5TDBKp2NMY2UN3UMV4vIImA50AQYqqqjgQHAr2o3eT6QVnFT1fmr9hIdEcL5PdrUYaKMMabueDNRz3XAM6r6ZfGVqnpcRG6vnWT5UHoi9Bxd5qaDR7NZvvUQd408k6BA/6mPN8b4F28Cw6PAgcIFEWkKtFXVBFVdVlsJ84ncY3AspdwWSQvW7MWjMM6GwDDGNGLe/Ox9B/AUWy5w1zU+FbRI8niU+av3cs4ZUXSJala36TLGmDrkTWAIUtXcwgX3ffDpXFRE7heRH0Vkk4i8JSKhItJVRFaKyA4RmS8ip3WNaqmgD8N3u1LZe+SEzdJmjGn0vAkMKSJydeGCiFwDHK7uBUWkI3AvEK+qfYFAYALwN5y6jG5AGlD39RdFOYbSgWHeqr20aNqES/u0q+NEGWNM3fImMEwHHhaRPSKyF3gQmHaa1w0CmopIEBCGU4dxIU6TWIA5wLWneY2qS0+EJmHQrGSLo/TjuSz98SBjBtosbcaYxq/SymdV3QmcLSLh7nLW6VxQVfeJyFPAHuAE8DGwBkhX1Xx3tySgzB5mIjIVmArQuXPlE+lUSVqiU4x0yjAXi9ftIzffY5XOxhi/4E2rJETkCqAPEFo4NpCq/qk6FxSRVsA1QFcgHaci+zJvj1fVmcBMgPj4eK1OGsqVllCqGElVmbdqL/1jWnBWh+Y1ejljjKmPvOng9i+c8ZJ+AQhwA1D5ZMjluwjYraopqpoHvAuMAFq6RUsAMcC+07hG1ak6RUmnVDxvTDrKloOZNqezMcZveFPHMFxVbwHSVPWPwDlAj4D5a68AABmBSURBVNO45h6coqkwcbIfo4DNwOfA9e4+k4H3TuMaVXf8CORmlWqqOn/1XkKbBHDVgA51mhxjjPEVbwJDtvt6XEQ6AHk44yVVi6quxKlkXgv84KZhJk6l9i9FZAcQBbxa3WtUS3qC81qsKElV+WDDfkb3bU/zUJulzRjjH7ypY3hfRFoCT+I8zBV4+XQuqqp/AP5wyupdwNDTOe9pSUtwXosVJWXm5JORnU8fq1swxviRCgODO0HPMlVNBxaKyAdAqKoerZPU1aUy+jAcynAyS9HNbXhtY4z/qLAoSVU9wD+KLec0yqAATsVzWBSEnJxmIjkjB4C2ESG+SpUxxtQ5b+oYlonIddLY57BMK90iKdnNMbS1HIMxxo94Exim4fQ1yBGRDBHJFJGMWk5X3UsvPQ9DYY4hurnlGIwx/sObqT0jVDVAVYNVtbm73LhqYz0FkL63zBxDRGgQYcFe9QM0xphGodInnoicV9b6UyfuadAy9oMnr1SO4VBmthUjGWP8jjc/hX9T7H0oTpPSNTiD3jUO6WXPw5CckUNbK0YyxvgZbwbRu6r4soh0Ap6ttRT5QlrZ8zAkZ2QzNDbSBwkyxhjfqc7ExUlA75pOiE+lJQACLU6Oh6SqHMrIsT4Mxhi/400dw//h9HYGJ5DE4fSAbjzSE6F5Rwg6OWlc+vE8cgs8VpRkjPE73tQxrC72Ph94S1W/qaX0+EZaYun6hUzrw2CM8U/eBIYFQLaqFgCISKCIhKnq8dpNWh1KT4QzS9alF/V6thyDMcbPeNXzGWhabLkp8GntJMcH8rIh80C5vZ6jIyzHYIzxL94EhtDi03m678NqL0l1LH2P83pqH4aiAfQsx2CM8S/eBIZjIjKocEFEBuPM1dw4pJfXVDWHVmFNCAkK9EGijDHGd7ypY5gBvCMi+3Gm9myHM9Vn41A4D0Opzm3W69kY45+86eC2SkR6AT3dVVvduZobh/RECAyB8LYlVidnWh8GY4x/qrQoSUTuBpqp6iZV3QSEi8jPaz9pdSQtEVp2hoCSH8WhjGybh8EY45e8qWO4053BDQBVTQPurL0k1bG0hFIVzx6Pcigzx4qSjDF+yZvAEFh8kh4RCQSCK9i/YUkv3bkt9VguBR61PgzGGL/kTeXzUmC+iLzkLk8DltRekurQiXTIPlp+HwbLMRhj/JA3geFBYCow3V3eiNMyqeErGm679DwMYMNhGGP8kzczuHmAlUACzlwMFwI/1W6y6khhU9Uy+jCADYdhjPFP5eYYRKQHMNH9OwzMB1DVC+omaXUgrbwJerIRgdbhFhiMMf6noqKkLcBXwJWqugNARO6vk1TVlfRECG0BTVuWWJ2ckUNUsxCaBFZnugpjjGnYKnryjQUOAJ+LyMsiMgqn53PjkZZYqhgJ3D4MVoxkjPFT5QYGVV2sqhOAXsDnOENjRIvIiyJySV0lsFalJ5aqeAZnLgareDbG+CtvKp+Pqeqb7tzPMcA6nJZKDZvHU26OITkjx3IMxhi/VaVCdFVNU9WZqjqqthJUZ7KSoSCnVMVzfoGHw1k5Ng+DMcZv+W/tanrZLZIOZ+Wian0YjDH+y38DQ1p58zAUdm6zoiRjjH/y48CQ4Ly27Fxi9cnAYDkGY4x/8t/AkJ4IEe2hSckAkJzp9HqOtiG3jTF+yn8DQwV9GAIEoqzXszHGT/lvYCivD0NGNm0iQggMaFx9+Ywxxlv+GRjycyFjXwV9GKx+wRjjv+o8MIhITxFZX+wvQ0RmiEikiHwiItvd11a1loije0E95eYYrA+DMcaf1XlgUNWtqhqnqnHAYOA4sAh4CFimqt2BZe5y7SinDwPgTulp9QvGGP/l66KkUcBOVU0ErgHmuOvnANfW2lXL6cOQk1/AkWO5VpRkjPFrvg4ME4C33PdtVfWA+/4g0LasA0RkqoisFpHVKSkp1btqkzDoMAiadyixOiXTJugxxhifBQYRCQauBt45dZuqKqBlHeeO1RSvqvFt2rSp3sUHjIepn0NAYInVhTO32VzPxhh/5sscw2hgraomu8vJItIewH09VNcJOlTY69kqn40xfsyXgWEiJ4uRAP4DTHbfTwbeq+sE2ThJxhjjo8AgIs2Ai4F3i61+ArhYRLYDF7nLdSo5M4cmgUKrsOC6vrQxxtQbFc35XGtU9RgQdcq6VJxWSj5T2IchwHo9G2P8mK9bJdUrhzJyiLZiJGOMn7PAUExyRrZVPBtj/J4FhmKSM7Kt4tkY4/csMLhO5BaQkZ1vfRiMMX7PAoPrUKbN3GaMMWCBoUhhr2crSjLG+DsLDC6b69kYYxwWGFzJNhyGMcYAFhiKHMrMISQogOZNfdLnzxhj6g0LDC6nqWooItbr2Rjj3ywwuKwPgzHGOCwwuJzhMKx+wRhjLDC4bDgMY4xxWGAAsnLyOZZbYEVJxhiDBQbA+jAYY0xxFhg4GRhsyG1jjLHAADgVz2A5BmOMAQsMgBUlGWNMcRYYcAbQaxYcSHiI9Xo2xhgLDEByZrblFowxxmWBATiUkW0Vz8YY47LAgFOUZDkGY4xx+H1gUNWiAfSMMcZYYCDjRD45+R6iI6woyRhjwAIDyTbXszHGlGCBwfowGGNMCRYYino9W1GSMcaABYaT4yTZkNvGGANYYOBQRjbNQ4NoGhzo66QYY0y94PdjQFgfBtNY5OXlkZSURHZ2tq+TYuqJ0NBQYmJiaNKkSZWOs8Bgw2GYRiIpKYmIiAhiY2MREV8nx/iYqpKamkpSUhJdu3at0rFWlJSRY8NhmEYhOzubqKgoCwoGABEhKiqqWjlIvw4MHo9yyHIMphGxoGCKq+6/B78ODGnHc8krUNpar2djjCni14Eh2WZuM6bGpKamEhcXR1xcHO3ataNjx45Fy7m5uRUeu3r1au69995KrzF8+PCaSq6pgF9XPhcOhxFtgcGY0xYVFcX69esBePTRRwkPD+fXv/510fb8/HyCgsp+5MTHxxMfH1/pNVasWFEzia1DBQUFBAY2rObwPgkMItISeAXoCyhwG7AVmA/EAgnAOFVNq810HCoaDsOKkkzj8sf3f2Tz/owaPedZHZrzh6v6VOmYKVOmEBoayrp16xgxYgQTJkzgvvvuIzs7m6ZNmzJr1ix69uzJ8uXLeeqpp/jggw949NFH2bNnD7t27WLPnj3MmDGjKDcRHh5OVlYWy5cv59FHH6V169Zs2rSJwYMH8/rrryMifPjhh/zyl7+kWbNmjBgxgl27dvHBBx+USFdCQgI333wzx44dA+CFF14oyo387W9/4/XXXycgIIDRo0fzxBNPsGPHDqZPn05KSgqBgYG888477N27tyjNAPfccw/x8fFMmTKF2NhYxo8fzyeffMIDDzxAZmYmM2fOJDc3l27dujF37lzCwsJITk5m+vTp7Nq1C4AXX3yRpUuXEhkZyYwZMwD43e9+R3R0NPfdd1/1v7wq8lWO4TlgqapeLyLBQBjwMLBMVZ8QkYeAh4AHazMRhUVJbayOwZhak5SUxIoVKwgMDCQjI4OvvvqKoKAgPv30Ux5++GEWLlxY6pgtW7bw+eefk5mZSc+ePbnrrrtKtcVft24dP/74Ix06dGDEiBF88803xMfHM23aNL788ku6du3KxIkTy0xTdHQ0n3zyCaGhoWzfvp2JEyeyevVqlixZwnvvvcfKlSsJCwvjyJEjAEyaNImHHnqIMWPGkJ2djcfjYe/evRXed1RUFGvXrgWcYrY777wTgEceeYRXX32VX/ziF9x7772cf/75LFq0iIKCArKysujQoQNjx45lxowZeDwe5s2bx/fff1/lz/101HlgEJEWwHnAFABVzQVyReQaYKS72xxgObUeGLKJbBZMSFDDyuYZU5mq/rKvTTfccENRUcrRo0eZPHky27dvR0TIy8sr85grrriCkJAQQkJCiI6OJjk5mZiYmBL7DB06tGhdXFwcCQkJhIeHc8YZZxS12584cSIzZ84sdf68vDzuuece1q9fT2BgINu2bQPg008/5dZbbyUsLAyAyMhIMjMz2bdvH2PGjAGcTmPeGD9+fNH7TZs28cgjj5Cenk5WVhaXXnopAJ999hmvvfYaAIGBgbRo0YIWLVoQFRXFunXrSE5OZuDAgURFRXl1zZriixxDVyAFmCUiA4A1wH1AW1U94O5zEGhb1sEiMhWYCtC5c+fTSkhyRo7Nw2BMLWvWrFnR+9///vdccMEFLFq0iISEBEaOHFnmMSEhJ/9fBgYGkp+fX619yvPMM8/Qtm1bNmzYgMfj8fphX1xQUBAej6do+dT+AsXve8qUKSxevJgBAwYwe/Zsli9fXuG577jjDmbPns3Bgwe57bbbqpy20+WLVklBwCDgRVUdCBzDKTYqoqqKU/dQiqrOVNV4VY1v06bNaSXE+jAYU7eOHj1Kx44dAZg9e3aNn79nz57s2rWLhIQEAObPn19uOtq3b09AQABz586loKAAgIsvvphZs2Zx/PhxAI4cOUJERAQxMTEsXrwYgJycHI4fP06XLl3YvHkzOTk5pKens2zZsnLTlZmZSfv27cnLy+ONN94oWj9q1ChefPFFwKmkPnr0KABjxoxh6dKlrFq1qih3UZd8ERiSgCRVXekuL8AJFMki0h7AfT1U2wlxpvS0HIMxdeWBBx7gt7/9LQMHDqzSL3xvNW3alH/+859cdtllDB48mIiICFq0aFFqv5///OfMmTOHAQMGsGXLlqJf95dddhlXX3018fHxxMXF8dRTTwEwd+5cnn/+efr378/w4cM5ePAgnTp1Yty4cfTt25dx48YxcODActP15z//mWHDhjFixAh69epVtP65557j888/p1+/fgwePJjNmzcDEBwczAUXXMC4ceN80qJJnB/ndXxRka+AO1R1q4g8ChTmuVKLVT5HquoDFZ0nPj5eV69eXa00FHiU7r/7kLsv6MavLulZrXMYU5/89NNP9O7d29fJ8LmsrCzCw8NRVe6++266d+/O/fff7+tkVYnH42HQoEG88847dO/e/bTOVda/CxFZo6rltg/2VQe3XwBviMhGIA74C/AEcLGIbAcucpdrTWpWDh61PgzGNDYvv/wycXFx9OnTh6NHjzJt2jRfJ6lKNm/eTLdu3Rg1atRpB4Xq8klzVVVdD5QVrUbVVRqKej1b5bMxjcr999/f4HIIxZ111llF/Rp8xW+HxLC5no0xpmz+GxgyLTAYY0xZ/DcwZOQgAq3Dg32dFGOMqVf8NjAcysimdXgIQYF++xEYY0yZ/PapaH0YjKlZF1xwAR999FGJdc8++yx33XVXuceMHDmSwibnl19+Oenp6aX2efTRR4v6E5Rn8eLFRX0AAP7nf/6HTz/9tCrJN8X4cWDIoW2E1S8YU1MmTpzIvHnzSqybN29euQPZnerDDz+kZcuW1br2qYHhT3/6ExdddFG1zuUrhb2v6wO/nY/hUGY2AzpV7x+hMfXekofg4A81e852/WB0+d2Lrr/+eh555BFyc3MJDg4mISGB/fv3c+6553LXXXexatUqTpw4wfXXX88f//jHUsfHxsayevVqWrduzeOPP86cOXOIjo6mU6dODB48GHD6KJw6fPX69ev5z3/+wxdffMFjjz3GwoUL+fOf/8yVV17J9ddfz7Jly/j1r39Nfn4+Q4YM4cUXXyQkJITY2FgmT57M+++/T15eHu+8806JXsngv8Nz+2WOIa/Aw+GsXCtKMqYGRUZGMnToUJYsWQI4uYVx48YhIjz++OOsXr2ajRs38sUXX7Bx48Zyz7NmzRrmzZvH+vXr+fDDD1m1alXRtrFjx7Jq1So2bNhA7969efXVVxk+fDhXX301Tz75JOvXr+fMM88s2j87O5spU6Ywf/58fvjhB/Lz84vGJgJo3bo1a9eu5a677iqzuKpweO61a9cyf/78onkhig/PvWHDBh54wBmkYdKkSdx9991s2LCBFStW0L59+0o/t8LhuSdMmFDm/QFFw3Nv2LCBtWvX0qdPH2677baikVkLh+e+6aabKr2eN/wyx5CSaVN6mkaugl/2tamwOOmaa65h3rx5RQ+2t99+m5kzZ5Kfn8+BAwfYvHkz/fv3L/McX331FWPGjCka+vrqq68u2lbe8NXl2bp1K127dqVHjx4ATJ48mX/84x9Fv7LHjh0LwODBg3n33XdLHe+vw3P7ZWBItpnbjKkV11xzDffffz9r167l+PHjDB48mN27d/PUU0+xatUqWrVqxZQpU0oNUe2tqg5fXZnCobvLG7bbX4fn9suipMLhMKKt8tmYGhUeHs4FF1zAbbfdVlTpnJGRQbNmzWjRogXJyclFRU3lOe+881i8eDEnTpwgMzOT999/v2hbecNXR0REkJmZWepcPXv2JCEhgR07dgDOKKnnn3++1/fjr8Nz+2VgOGS9no2pNRMnTmTDhg1FgWHAgAEMHDiQXr16ceONNzJixIgKjx80aBDjx49nwIABjB49miFDhhRtK2/46gkTJvDkk08ycOBAdu7cWbQ+NDSUWbNmccMNN9CvXz8CAgKYPn261/fir8Nz+2TY7ZpS3WG3P/7xIAvWJPGvmwYTECC1kDJj6p4Nu+1/vBmeuyENu+1Tl/Rpx8xb4i0oGGMarNocntsvK5+NMaahq83huf0yx2BMY9WQi4ZNzavuvwcLDMY0EqGhoaSmplpwMIATFFJTU6vXxLYW0mOM8YGYmBiSkpJISUnxdVJMPREaGkpMTEyVj7PAYEwj0aRJE7p27errZJhGwIqSjDHGlGCBwRhjTAkWGIwxxpTQoHs+i0gKkFjNw1sDh2swOfVBY7unxnY/0PjuqbHdDzS+eyrrfrqoapvyDmjQgeF0iMjqirqEN0SN7Z4a2/1A47unxnY/0PjuqTr3Y0VJxhhjSrDAYIwxpgR/DgwzfZ2AWtDY7qmx3Q80vntqbPcDje+eqnw/flvHYIwxpmz+nGMwxhhTBgsMxhhjSvDLwCAil4nIVhHZISIP+To9p0tEEkTkBxFZLyJVn9KuHhCRf4vIIRHZVGxdpIh8IiLb3ddWvkxjVZRzP4+KyD73e1ovIpf7Mo1VJSKdRORzEdksIj+KyH3u+gb5PVVwPw32exKRUBH5XkQ2uPf0R3d9VxFZ6T7z5otIcIXn8bc6BhEJBLYBFwNJwCpgoqpu9mnCToOIJADxqtpgO+WIyHlAFvCaqvZ11/0dOKKqT7gBvJWqPujLdHqrnPt5FMhS1ad8mbbqEpH2QHtVXSsiEcAa4FpgCg3we6rgfsbRQL8nERGgmapmiUgT4GvgPuCXwLuqOk9E/gVsUNUXyzuPP+YYhgI7VHWXquYC84BrfJwmv6eqXwJHTll9DTDHfT8H5z9tg1DO/TRoqnpAVde67zOBn4CONNDvqYL7abDUkeUuNnH/FLgQWOCur/Q78sfA0BHYW2w5iQb+jwHni/9YRNaIyFRfJ6YGtVXVA+77g0BbXyamhtwjIhvdoqYGUeRSFhGJBQYCK2kE39Mp9wMN+HsSkUARWQ8cAj4BdgLpqprv7lLpM88fA0Nj9DNVHQSMBu52izEaFXXKPBt6ueeLwJlAHHAAeNq3yakeEQkHFgIzVDWj+LaG+D2VcT8N+ntS1QJVjQNicEpIelX1HP4YGPYBnYotx7jrGixV3ee+HgIW4fxjaAyS3XLgwvLgQz5Oz2lR1WT3P60HeJkG+D255dYLgTdU9V13dYP9nsq6n8bwPQGoajrwOXAO0FJECidmq/SZ54+BYRXQ3a2lDwYmAP/xcZqqTUSauRVniEgz4BJgU8VHNRj/ASa77ycD7/kwLaet8OHpGkMD+57cis1XgZ9U9X+LbWqQ31N599OQvycRaSMiLd33TXEa2fyEEyCud3er9Dvyu1ZJAG7zs2eBQODfqvq4j5NUbSJyBk4uAZypWt9siPcjIm8BI3GGCE4G/gAsBt4GOuMMrz5OVRtEhW459zMSp3hCgQRgWrGy+XpPRH4GfAX8AHjc1Q/jlMs3uO+pgvuZSAP9nkSkP07lciDOD/+3VfVP7nNiHhAJrANuUtWccs/jj4HBGGNM+fyxKMkYY0wFLDAYY4wpwQKDMcaYEiwwGGOMKcECgzHGmBIsMBhTBhEpKDa65vqaHIVXRGKLj7pqTH0TVPkuxvilE+6wAsb4HcsxGFMF7twXf3fnv/heRLq562NF5DN34LVlItLZXd9WRBa54+NvEJHh7qkCReRld8z8j91eqojIve78ABtFZJ6PbtP4OQsMxpSt6SlFSeOLbTuqqv2AF3B60AP8HzBHVfsDbwDPu+ufB75Q1QHAIOBHd3134B+q2gdIB65z1z8EDHTPM722bs6YiljPZ2PKICJZqhpexvoE4EJV3eUOwHZQVaNE5DDOpC957voDqtpaRFKAmOLDD7hDPH+iqt3d5QeBJqr6mIgsxZngZzGwuNjY+sbUGcsxGFN1Ws77qig+Tk0BJ+v7rgD+gZO7WFVsRExj6owFBmOqbnyx12/d9ytwRuoFmIQzOBvAMuAuKJpApUV5JxWRAKCTqn4OPAi0AErlWoypbfZrxJiyNXVnwSq0VFULm6y2EpGNOL/6J7rrfgHMEpHfACnAre76+4CZInI7Ts7gLpzJX8oSCLzuBg8BnnfH1DemTlkdgzFV4NYxxKvqYV+nxZjaYkVJxhhjSrAcgzHGmBIsx2CMMaYECwzGGGNKsMBgjDGmBAsMxhhjSrDAYIwxpoT/BzFS9pF633JnAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nEKbGLWsIt6P",
        "colab_type": "text"
      },
      "source": [
        "Empty out training set and validation set to free up RAM / Cache"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SsVJuIwych21",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# resnet_model.load_state_dict(torch.load('/content/drive/My Drive/bt_resnet_torch.pt'))\n",
        "train_gen = None\n",
        "valid_gen = None\n",
        "train_set = None\n",
        "valid_set = None"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LfVhP2dmvdoi",
        "colab_type": "text"
      },
      "source": [
        "Set model to evaluation mode\n",
        "\n",
        "Calculate loss, correctly classified samples, predicted values, labels and store them in a list using test dataloader"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WZJ7w9ztv1pb",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "e59c6347-59f8-42e8-aff4-c95e2567f495"
      },
      "source": [
        "# set model to evaluation mode\n",
        "resnet_model.eval()\n",
        "\n",
        "# perform no gradient updates\n",
        "with torch.no_grad():\n",
        "    # soem metrics storage for visualization and analysis\n",
        "    correct = 0\n",
        "    test_loss = []\n",
        "    test_corr = []\n",
        "    labels = []\n",
        "    pred = []\n",
        "    # perform test set evaluation batch wise\n",
        "    for (y, X) in test_gen:\n",
        "        # set label to use CUDA if available\n",
        "        X, y = X.to(device), y.to(device)\n",
        "\n",
        "        # append original labels\n",
        "        labels.append(torch.argmax(y.view(10 * 8, 4), dim=1).data)\n",
        "\n",
        "        # perform forward pass\n",
        "        y_val = resnet_model(X.view(-1, 3, 512, 512))\n",
        "\n",
        "        # get argmax of predicted values, which is our label\n",
        "        predicted = torch.argmax(y_val, dim=1).data\n",
        "        # append predicted label\n",
        "        pred.append(predicted)\n",
        "\n",
        "        # calculate loss\n",
        "        loss = criterion(y_val.float(), torch.argmax(y.view(10 * 8, 4), dim=1).long())\n",
        "\n",
        "        # increment correct with correcly predicted labels per batch\n",
        "        correct += (predicted == torch.argmax(y.view(10 * 8, 4), dim=1)).sum()\n",
        "\n",
        "        # append correct samples labels and losses\n",
        "        test_corr.append(correct)\n",
        "        test_loss.append(loss)\n",
        "        \n",
        "print(f\"Test Loss: {test_loss[-1].item():.4f}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Test Loss: 0.0096\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AqDm19ZSvyY2",
        "colab_type": "text"
      },
      "source": [
        "Print the test accuracy\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HJm2yjzHbLlP",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "7b227ca6-3d4d-44cd-a030-a0e0cc8d744e"
      },
      "source": [
        "print(f'Test accuracy: {test_corr[-1].item()*100/(460*8):.2f}%')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Test accuracy: 99.32%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T6fxSb68v0uf",
        "colab_type": "text"
      },
      "source": [
        "Convert list of tensors to tensors"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jDRv0IhThKhP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "labels = torch.stack(labels)\n",
        "pred = torch.stack(pred)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ps2QnU4ov4Sr",
        "colab_type": "text"
      },
      "source": [
        "Define ground-truth labels as a list"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7BsASCMlL2_z",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "LABELS = ['Meningioma', 'Glioma', 'Pitutary']"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wubHPKqov61w",
        "colab_type": "text"
      },
      "source": [
        "Plot the confusion matrix"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3cfoG8Q0gMKZ",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 387
        },
        "outputId": "95a96f4c-bf40-4efd-a0f2-31d5606c657a"
      },
      "source": [
        "arr = confusion_matrix(pred.view(-1).cpu(), labels.view(-1).cpu())\n",
        "df_cm = pd.DataFrame(arr, LABELS, LABELS)\n",
        "plt.figure(figsize = (9,6))\n",
        "sns.heatmap(df_cm, annot=True, fmt=\"d\", cmap='viridis')\n",
        "plt.xlabel(\"Prediction\")\n",
        "plt.ylabel(\"Target\")\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAggAAAFzCAYAAABb8fH8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deZgU1b3/8fdnBlARFRFFBBWMaOIuIm5ZUOOeiMk1LknUGBPihonmRk28uS5JbtSYaBKN/lBQ1LjHBROXoLgmKqIiiMZIcGEQZBNwF5jv74+qkXaaGYZqqrun5/N6nnq66tSpOqdnGvo755w6RxGBmZmZWaG6SlfAzMzMqo8DBDMzMyviAMHMzMyKOEAwMzOzIg4QzMzMrIgDBDMzMyvSqdIVaEm/yy/y85dWZMBJT1W6CmbWToxtvFV5l9E4a4uSvqvqNvx37nXMqmoDBDMzs2rXSGNJ11dzM341183MzMwqxC0IZmZmGS2N0loQqvlLuJrrZmZmVtUaqd3hcg4QzMzMMip1DEI18xgEMzMzK+IWBDMzs4yW1vCKyA4QzMzMMvIYBDMzMyuy1AGCmZmZNVfLLQgepGhmZmZF3IJgZmaWkQcpmpmZWZHanQXBAYKZmVlmtTxI0WMQzMzMMloapW1tIWmUpNmSXmiWPlzSvyRNkXRhQfpPJU2V9LKk/QrS90/Tpko6c0XlugXBzMysul0DXApc25QgaU9gKLB9RHwkaYM0fSvgCGBrYCPgAUlbpJddBuwDNABPSxoTES+2VKgDBDMzs4zKMQYhIh6V1K9Z8gnA+RHxUZpndpo+FLgpTX9V0lRgcHpuakRMA5B0U5q3xQDBXQxmZmYZLUUlbSXYAviCpKckPSJp5zS9DzC9IF9DmtZSeovcgmBmZpZRY4ljFCUNA4YVJI2IiBFtuLQT0APYFdgZuEXSZqXVprgAMzMzq4A0GGhLQNBcA3B7RAQwXlIj0BOYAWxckK9vmkYr6cvlLgYzM7OMKtjFcCewJ0A6CLELMBcYAxwhaTVJ/YEBwHjgaWCApP6SupAMZBzTWgFuQTAzM8uoxC/5NpF0IzAE6CmpATgbGAWMSh99/Bg4Jm1NmCLpFpLBh0uAkyJiaXqfk4H7gXpgVERMaa1cBwhmZmYZNUb+AUJEHNnCqW+3kP9XwK+Wk34PcE9by3WAYGZmllE5WhAqxWMQzMzMrIhbEMzMzDJaWsN/ZztAMDMzy6gcYxAqxQGCmZlZRrU8BsEBgpmZWUZLo3a7GGr3nZmZmVlmbkEwMzPLqLGG/852gGBmZpaRxyCYmZlZEY9BMDMzsw7FLQhmZmYZNbqLwczMzJrzTIpmZmZWpJbHIDhAMDMzy6iWH3Os3XdmZmZmmbkFwczMLKOlXqzJzMzMmvMgRTMzMyvS6EGKZmZm1lwttyDU7jszMzOzzNyCYGZmlpEHKZqZmVmRWp4HwQGCmZlZRrU8k2LtvjMzMzPLzC0IZmZmGXk1R8vVcdvtxOGf25YAXp43h588dB8DN9yIs3YbQuf6el6Y8xanP3QfSyP4TPce/GbP/dl6/Q246KnHufL5CZWuvlVAXV0dlz19PnNnzOfnB59f6epYlRi03w6ceMmx1NXXce/IB7n5gjsrXaWa5y4Gy02vNbvxnW0H8tXbrme/m6+hTnUcPOBz/HavAxg+9q/sd/M1NLyziP/acmsAFnz0Iec8Po4rJzow6Mi+9sMDeeOlGZWuhlWRuro6hl96HD878Fd8b+tT2fOIPdjkc30rXa2at5S6krZqVt216yDq68TqnTpRL7FGp058sHgxi5c28urCtwF4vOE1DthsCwDmffA+k+bMYkljYyWrbBXUs08PdjlwIPeOfLDSVbEqsuXgzXlz6ixmvTqbJYuX8PDN/2D3oYMqXa2a1xgqaatmuXcxSDoI2BpYvSktIs7Lu9z24q333uXKiRP451HD+HDJEh6b/hp//c/LnLnbl9h2/V5MnvMWB262Bb27rVXpqlqVOOHiY7nyjOtZY63VV5zZOoyefXowp2HeJ8dzG+bz2V0GVLBG1t7l2oIg6QrgcGA4IOAbwKat5B8maYKkCe88/mSeVasaa3dZjX36b84Xrr+SXa69gq6dO3PIgM9xyti7+fkee3Ln17/Fu4s/pjGi0lW1KrDLQQNZMGchrzw7rdJVMTPK08UgaZSk2ZJeWM65H0sKST3TY0n6g6SpkiZJGliQ9xhJr6TbMSsqN+8WhN0jYjtJkyLiXEm/Be5tKXNEjABGAPS7/KIO8Y34+b6bMn3RQuZ/+AEA9017hZ027MOdr7zEYXfeBMAX+m5K/3V6VLKaViW23uOz7PbVQQw+YEe6rN6FrmuvwRnXDueCo/9Y6apZhc2dMZ/1+673yXHPvj2YO2NeK1fYqlCmxZquAS4Fri1MlLQxsC/wRkHyAcCAdNsFuBzYRVIP4GxgEBDAM5LGRMTbLRWa9zv7IH19X9JGwGKgd85ltitvvruIHXv1ZvVOSay2R99Nmfr2PNZboysAXerqOX7Hwfz5xYmVrKZViVE/u4FvbnI8R212Er868mImjnvBwYEB8PLTU+kzoDcb9tuATp07MeTwPXhijAcz520pKmlri4h4FJi/nFMXA6eTfOE3GQpcG4knge6SegP7AWMjYn4aFIwF9m+t3LxbEP4qqTvwG+BZkjdxVc5ltisTZ8/i3mn/5m+HHsWSCKbMeYsbX5zEj3f5PHtvuhmS+POUiTwxYzoA66/RlTGHHkW3Ll2ICL673U7sc9PVvLv44wq/EzOrpMaljVw6fCS/vu8s6urruP/qh3j9xYZKV6vmldqCIGkYMKwgaUTamr6i64YCMyLieelTgUYfYHrBcUOa1lJ6y2VEmfq2Ja0GrB4RC9uSv6N0MdjKGXDSU5Wugpm1E2Mbb839MYELXzygpO+q07e6t011lNQP+GtEbCOpK/AQsG9ELJT0GjAoIuZK+itwfkQ8nl73IHAGMITkO/iXafrPgQ8i4qKWysy1BUFSPXAQ0K+pLElExO/yLNfMzKwc2tpNsIp9BugPNLUe9AWelTQYmAFsXJC3b5o2gyRIKEx/uLVC8u5iuBv4EJgM+MF9MzOrKWUapPgpETEZ2KDpuFkLwhjgZEk3kQxSXBgRMyXdD/yfpHXTy/YFftpaOXkHCH0jYrucyzAzM6uIcky1LOlGkr/+e0pqAM6OiJEtZL8HOBCYCrwPHAsQEfMl/QJ4Os13XkQsb+DjJ/IOEO6VtG9E/D3ncszMzGpSRBy5gvP9CvYDOKmFfKOAUW0tN+8A4UngDkl1JI84iqT+a+dcrpmZWe68mmN2vwN2AyZHuR6XMDMzK5NaXs0x7wBhOvCCgwMzM6tF1b7gUinyDhCmAQ9Luhf4qCnRjzmamVktqPYlm0uRd4Dwarp1STczMzNrB3INECLiXABJ3dLjd/Msz8zMrJzcxZCRpG2A64Ae6fFc4OiImJJnuWZmZuXQ6C6GzEYAp0XEQwCShgBXArvnXK6ZmVnulroFIbM1m4IDgIh4WNKaOZdpZmZWFu5iyG5aumLUdenxt0mebDAzM7MqlneA8F3gXOD29PixNM3MzKzdq8RiTeWS91MMbwOn5FmGmZlZpVRoueeyyCVAkHRJRPxI0t1A0SyKEXFwHuWamZmVk8cgrLymMQcX5XR/MzMzy1EuAUJEPJO+PpLH/c3MzKqBxyBkJGkyxV0MC4EJwC8jYl6e5ZuZmeXJyz1ndy+wFLghPT4C6ArMAq4Bvppz+WZmZrnxREnZfTkiBhYcT5b0bEQMlPTtnMs2MzPLVS13MeT9zuolDW46kLQzUJ8eLsm5bDMzM8so7xaE7wGj0tUcBSwCvpdOt/zrnMs2MzPLlR9zzCginga2lbROeryw4PQteZZtZmaWNw9SzEjSasB/Af2ATlLyg4yI8/Is18zMrBzcgpDdXSSPNT4DfJRzWWZmZraK5B0g9I2I/XMuw8zMrCJq+SmGvAOEf0raNiIm51yOmZlZ2bmLIbvPA9+R9CpJF4OAiIjtci7XzMwsdx6kmN0BOd/fzMysYtyCsJIkrR0Ri4B38ri/mZmZ5SuvFoQbgK+QPL0Q8Kk2mAA2y6lcMzOzsqnlFoRchl9GxFfS1/4RsVn62rQ5ODAzs5rQGCppawtJoyTNlvRCQdpvJP1L0iRJd0jqXnDup5KmSnpZ0n4F6funaVMlnbmicnN/PkNSH0m7S/pi05Z3mWZmZuVQjgCBZPXj5lMGjAW2SQf9/xv4KYCkrUhWTt46veZPkuol1QOXkYwN3Ao4Ms3borxnUrwAOBx4kWTZZ0i6GB7Ns1wzM7NyKMdTDBHxqKR+zdL+XnD4JHBouj8UuCkiPgJelTQVaFo0cWpETAOQdFOa98WWys37KYZDgC3TipqZmdmq913g5nS/D0nA0KQhTQOY3ix9l9ZumneAMA3ojKdZNjOzGlTqIEVJw4BhBUkjImLESlx/FrAE+HNJFVmOvAOE94GJkh6kIEiIiFNyLtfMzCx3pQYIaTDQ5oCgkKTvkDwxuHdERJo8A9i4IFvfNI1W0pcr7wBhTLqZmZnVnEo95ihpf+B04EsR8X7BqTHADZJ+B2wEDADGk0w3MEBSf5LA4Ajgm62VkWuAEBGjJa0BbBIRL+dZlpmZWS2SdCMwBOgpqQE4m+SphdWAsZIAnoyI4yNiiqRbSAYfLgFOioil6X1OBu4H6oFRETGltXLzforhq8BFQBegv6QdgPMi4uA8yzUzMyuHcrQgRMSRy0ke2Ur+XwG/Wk76PcA9bS0373kQziF5vGIBQERMxLMomplZjYhQSVs1y3sMwuKIWJg2fzRpzLlMMzOzsvBqjtlNkfRNoF7SAOAU4J85l2lmZlYWXoshu+Ek0z1+BNwILAJ+lHOZZmZmVqK8n2J4Hzgr3czMzGpKtY8jKEUuAYKkVuc+8FMMZmZWC2q5iyGvFoTdSOZ8vhF4Cmp4FIeZmXVYbkFYeRsC+wBHkszU9DfgxhVNylBoi1Mm5FQ1a8/ue/P5SlfBqtB+G21f6SpYB1XLLQi5DFKMiKURcV9EHAPsCkwFHk5ncTIzM7Mql9sgRUmrAQeRtCL0A/4A3JFXeWZmZuX2yRJJNSivQYrXAtuQTOl4bkS8kEc5ZmZmleSJklbet4H3gB8CpxTMpCggImLtnMo1MzMrGw9SXEkRkfcETGZmZpajvKdaNjMzq1m1/BSDAwQzM7OMPEjRzMzMingMgpmZmRWp5QDBgwnNzMysiFsQzMzMMvIgRTMzMyviQYpmZmZWpJbHIDhAMDMzy6iWAwQPUjQzM7MibkEwMzPLqIaHIDhAMDMzy6qWuxgcIJiZmWVVw00IHoNgZmZmRdyCYGZmlpG7GMzMzKyIJ0oyMzOzIrXcguAxCGZmZlmFStvaQNIoSbMlvVCQ1kPSWEmvpK/rpumS9AdJUyVNkjSw4Jpj0vyvSDpmReU6QDAzM6tu1wD7N0s7E3gwIgYAD6bHAAcAA9JtGHA5JAEFcDawCzAYOLspqGiJAwQzM7OMIkrb2lZGPArMb5Y8FBid7o8GDilIvzYSTwLdJfUG9gPGRsT8iHgbGEtx0PEpDhDMzMyyitI2ScMkTSjYhrWx5F4RMTPdnwX0Svf7ANML8jWkaS2lt8iDFM3MzDIqdZBiRIwARpR4j5C0yp+ncAuCmZlZViW2IJTgrbTrgPR1dpo+A9i4IF/fNK2l9BY5QDAzM2t/xgBNTyIcA9xVkH50+jTDrsDCtCvifmBfSeumgxP3TdNa5C4GMzOzjMoxD4KkG4EhQE9JDSRPI5wP3CLpOOB14LA0+z3AgcBU4H3g2KSeMV/SL4Cn03znRUTzgY+f4gDBzMwsqzLMpBgRR7Zwau/l5A3gpBbuMwoY1dZyHSCYmZll5pkUzczMrANxC4KZmVlWXqzJzMzMijhAMDMzsyI1vJqjAwQzM7OM2rqeQnvkQYpmZmZWxC0IZmZmWdVwC4IDBDMzs6xqeAzCCrsYJF3QljQzM7OORlHaVs3aMgZhn+WkHbCqK2JmZtbuVG41x9y12MUg6QTgRGAzSZMKTq0F/CPvipmZmVnltDYG4QbgXuDXwJkF6e+saAUoMzOzDqEjjkGIiIUR8Vq6itTGwF4R8TpQJ6l/2WpoZmZWrTpiF0MTSWcDg4AtgauBLsD1wB75Vs3MzKzKVfmXfCnaMkjxa8DBwHsAEfEmyTgEMzMzq1FtmQfh44gIKXkgQ9KaOdfJzMysfejgLQi3SPp/QHdJ3wceAK7Mt1pmZmbtQKi0rYqtsAUhIi6StA+wiGQcwv9GxNjca9ZBnXblD9j1wIEsmL2IYTv+BICjfn4oBxy3FwvnLgJg1P/cxNP3TaxkNS0nZ50PDz8BPdaFu69J0k49B16bnuwvehfW7gZ3jEyOR1wPf7kH6urgrFPg84OT9Meegv/7IzQ2wqEHwfe/Ve53YpUwaL8dOPGSY6mrr+PekQ9y8wV3VrpKNa/aJzsqRZumWk4DAgcFZTB29COM+dP9nD7qpE+l3/77e7jt4r9WqFZWLoccAN/8Opz5f8vSLj5n2f4Fl0G3tJNv6mtwz7gkkJg9D757Gtx7fXLuF5fAyN9Cr/XhsB/AnnvA5v3K8x6sMurq6hh+6XGcse8vmNswn0vH/5onxkzgjZcaKl212lbDAUJbplp+R9KiZtt0SXdI2qwclexIJj/+L96Z/16lq2EVsvP20L2FIcARcN9DcNCXk+Nxj8OBe0GXLtC3N2zSBya9lGyb9IGNN4IunZM84x4v33uwythy8Oa8OXUWs16dzZLFS3j45n+w+9BBla6WtWNtGYNwCfAToA/QF/hvkkmUbgJG5Vc1K3TwiftxxbMXcNqVP6Bbd48T7YgmTIL1ekC/vsnxW3Nhww2Wne+1Psyem2zN09+aW966Wvn17NODOQ3zPjme2zCfnn3Wq2CNrL1rS4BwcET8v4h4JyIWRcQIYL+IuBlYd0UXS9pG0mGSjm7aSq51B3P3/xvLd7Y8hRN2OpP5Mxcw7DffrnSVrAL+9gActHela2FmhTr6Yk3vp1/wdel2GPBheq7Vt5dOsvTHdNsTuJBkToWW8g+TNEHShIbG/7TtHXQAC2YvpLExiAjuHTmOzw7avNJVsjJbsgQeeAwO2HNZWq+eMGv2suO35sAGPZOteXqvnuWrq1XG3BnzWb/vshaDnn17MHfGvFausFWihp9iaEuA8C3gKGA28Fa6/21JawAnr+DaQ4G9gVkRcSywPbBOS5kjYkREDIqIQX3rPtOW+ncIPTbs/sn+HofszGtTplewNlYJTzwD/Tf5dNfBnnskgxQ//hgaZsLrDbDd52Dbzyb7DTPh48VJnj0972nNe/npqfQZ0JsN+21Ap86dGHL4HjwxZkKlq1X7OupUy5LqgRMj4qstZFnR0KcPIqJR0hJJa5MEGRtnqGeH8dPrhrPdl7ZinZ5r8edXL+O6825juy9txWe235SI4K3X5vD7E6+qdDUtJz8+F8ZPhAULYcihcPKxyWOK94wr7l4Y0B/23xO+cgzU18PPf5S8AvzPj+B7/5085vj1A5O8VtsalzZy6fCR/Pq+s6irr+P+qx/i9Rf9BINlp4jWQxhJT0bErpluLv0J+BlwBPBj4F1gYtqa0Kp9Ox9R5bGVVcJ905+tdBWsCu230faVroJVobGNt+behr/Zxb8r6btq2qmnVW0/Q1vmQXhO0hjgVtL1GAAi4vYVXRgRJ6a7V0i6D1g7IiZlqqmZmVmVqfaBhqVoS4CwOjAP2KsgLYAVBggAkrYD+jWVJWnztgQXZmZmVa8jBwht6Q5oiaRRwHbAFKCx6Za0MbgwMzPr6CSdCnyP5PtzMnAs0JtkPqL1gGeAoyLiY0mrAdcCO5H8cX94RLyWpdwVBgiSVgeOA7YmaU0AICK+24b77xoRW2WpmJmZWdXLuQVBUh/gFGCriPhA0i0k4/oOBC6OiJskXUHyPX15+vp2RGwu6QjgAuDwLGW35THH64ANgf2AR0hmU3ynjfd/QpIDBDMzq0llmiipE7CGpE5AV2AmSbf/ben50cAh6f7Q9Jj0/N6SMg2EbDFASCsCsHlE/Bx4LyJGAwcBu7Tx/teSBAkvS5okabIkD1I0M7PaUOJESYUTBKbbsE/dPmIGcBHwBklgsJCkS2FBRCxJszWQLIdA+jo9vXZJmj/TnNutdTGMBwYCi9PjBZK2AWYBG7R41aeNJJlYaTLLxiCYmZnVhhK7GNLlC0a0dF7SuiStAv2BBSRPFO5fWqlt05anGEakFfwfYAzQDfh5G+8/JyLGZK2cmZlZB/dl4NWImAMg6XZgD6C7pE5pK0FfYEaafwbJhIQNaU/AOiSDFVdaawHCBpJOS/ebnmS4LH1t63KCz0m6Abgb+Kgp0Y85mplZLSjDPAhvALtK6gp8QLJ8wQTgIZLlDG4CjgHuSvOPSY+fSM+PixXNiNiC1gKEepLWguUNbmhrYWuQBAb7NrvWAYKZmbV/OQcIEfGUpNuAZ4ElwHMkXRJ/A26S9Ms0bWR6yUjgOklTgfkkTzxk0lqAMDMizst6YyhtDgUzM7NqV46ZFCPibODsZsnTgMHLyfsh8I1VUW5rjzmWPD+0pL6S7pA0O93+Iqlvqfc1MzOzfLUWIOzdyrm2upqkP2SjdLs7TTMzM2v/ani55xYDhIiYvwruv35EXB0RS9LtGmD9VXBfMzOzyuuIAcIqMk/StyXVp9u3yfi4hZmZWbUp00yKFZF3gPBd4DCSyZVmkjxy4YGLZmZmVa4tEyVlFhGvAwfnWYaZmZmterkECJJOj4gLJf2R5fSyRMQpeZRrZmZWVlXeTVCKvFoQXkpfJ+R0fzMzs4qr9nEEpcglQIiIu9PX0SvKa2Zm1m45QFg5ku6mlR9bRHhcgpmZtX8OEFbaRctJa/oxljxDo5mZmeUrrwChO9A3Ii4DkDSeZIKkAM7IqUwzM7OyquUxCHnNg3A6yRTLTboAg4AhwPE5lWlmZlZeNTyTYl4tCF0iYnrB8eMRMY9kZsU1cyrTzMysrNyCsPLWLTyIiJMLDr0Wg5mZWZXLK0B4StL3mydK+gEwPqcyzczMystdDCvtVOBOSd8Enk3TdgJWAw7JqUwzM7PyqvIv+VLkNVHSbGB3SXsBW6fJf4uIcXmUZ2ZmVgm1PAYh78WaxgEOCszMrDbVcICQ93LPZmZm1g7l2oJgZmZW02q4BcEBgpmZWUYeg2BmZmbFHCCYmZlZc7XcguBBimZmZlbELQhmZmZZ1XALggMEMzOzrBwgmJmZWXOqdAVy5DEIZmZmVsQBgpmZWVZlWM1RUndJt0n6l6SXJO0mqYeksZJeSV/XTfNK0h8kTZU0SdLArG/NAYKZmVlGitK2Nvo9cF9EfBbYHngJOBN4MCIGAA+mxwAHAAPSbRhwedb35gDBzMwsq5xbECStA3wRGAkQER9HxAJgKDA6zTYaOCTdHwpcG4knge6Semd5aw4QzMzMssq/i6E/MAe4WtJzkq6StCbQKyJmpnlmAb3S/T7A9ILrG9K0leYAwczMrEIkDZM0oWAb1ixLJ2AgcHlE7Ai8x7LuBAAiYiVGNLSdH3M0MzPLqNSpliNiBDCilSwNQENEPJUe30YSILwlqXdEzEy7EGan52cAGxdc3zdNW2luQTAzM8sq5y6GiJgFTJe0ZZq0N/AiMAY4Jk07Brgr3R8DHJ0+zbArsLCgK2KluAXBzMwsozIt1jQc+LOkLsA04FiSP/BvkXQc8DpwWJr3HuBAYCrwfpo3EwcIZmZmWZUhQIiIicCg5Zzaezl5AzhpVZTrLgYzMzMrUrUtCLF0aaWrYFVov422r3QVrAo1/GWbSlfBOqgydTFURNUGCGZmZlXPAYKZmZkVqeEAwWMQzMzMrIhbEMzMzDLyGAQzMzMr5gDBzMzMmlPUboTgAMHMzCyr2o0PPEjRzMzMirkFwczMLCMPUjQzM7NiDhDMzMysObcgmJmZWbEaDhA8SNHMzMyKuAXBzMwsI3cxmJmZWTEHCGZmZtZcLbcgeAyCmZmZFXELgpmZWVZei8HMzMyaq+UuBgcIZmZmWTlAMDMzs+bUWOka5MeDFM3MzKyIWxDMzMyycheDmZmZNedBimZmZlbMjzmamZlZc7XcguBBimZmZlbELQhmZmZZuQXBzMzMmlOUtrW5HKle0nOS/poe95f0lKSpkm6W1CVNXy09npqe75f1vTlAMDMzyyqitK3tfgi8VHB8AXBxRGwOvA0cl6YfB7ydpl+c5svEAYKZmVkVk9QXOAi4Kj0WsBdwW5plNHBIuj80PSY9v3eaf6U5QDAzM8uo1C4GScMkTSjYhi2nmEuA04GmiZ3XAxZExJL0uAHok+73AaYDpOcXpvlXmgcpmpmZZVXiIMWIGAGMaOm8pK8AsyPiGUlDSitt5ThAMDMzy6gM8yDsARws6UBgdWBt4PdAd0md0laCvsCMNP8MYGOgQVInYB1gXpaC3cVgZmaWVWOUtq1ARPw0IvpGRD/gCGBcRHwLeAg4NM12DHBXuj8mPSY9Py4i23SPDhDMzMzanzOA0yRNJRljMDJNHwmsl6afBpyZtQB3MZiZmWVVxomSIuJh4OF0fxoweDl5PgS+sSrKc4BgZmaWUS2vxeAAwczMLCuv5mhmZmbN1XILggcpmpmZWRG3IJiZmWVVwy0IDhDMzMwykscgmJmZWZHGFWdprzwGwczMzIq4BcHMzCwjdzGYmZlZsdqNDxwgmJmZZeYWBDMzM2uulidKcoBQ5dZcpyunXXkC/bbZGCK46LjLeenJf1e6WlZhg/bbgRMvOZa6+jruHfkgN19wZ6WrZDn65cCDGbLhFsz/6D0OfvByANbpvDq/G3wofbp2Z8b7Czh1/G0sWvwhADv33JSfbrs/nevqePvj9zn6sdGf3KsOceue32f2h+9wwhM3VuT9WPvgAKHKnXjJsUy4/zl+cdhv6dS5E6t17VLpKlmF1dXVMfzS4zhj318wt2E+l47/NU+MmcAbLzVUumqWkztfn8gN/xnP+YO+9kna97f4PE/MeZWr/v0PvrfFHnx/i8/z2ykPsFbn1fjf7Q9i2D+vZ+YHi+jRpeun7sBHqKgAABBdSURBVHXU5rsw7Z25dOu8WrnfRm2q4S4GP+ZYxbqu3ZVtv7gV944cB8CSxUt4b+H7Fa6VVdqWgzfnzamzmPXqbJYsXsLDN/+D3YcOqnS1LEcT5r3BgsUffCptr95bctfrzwNw1+vPs3fvLQH4St9teeDNl5j5wSIA5n+87P+MXquvxZd6DeC2154tU81rnxpL26pZrgGCpOGS1s2zjFrWu/8GLJyziJ+MOonLn7mQ0648ntW7Ourv6Hr26cGchnmfHM9tmE/PPutVsEZWCeut1o05H70LwJyP3mW91boB0K/beqzdZQ1Gf/4YbhvyfYZuvN0n1/x0u/25aMoDNNby0Ptyiyhtq2J5tyD0Ap6WdIuk/SWptcyShkmaIGlCQ0zLuWrVr75THQMG9ufuK+7nhJ1O58P3PuLwMw+pdLXMrApF+qVfrzq27t6b45+4ge/983pO+OwX6detB0M2HMD8j97jxQUzK1xTay9yDRAi4n+AAcBI4DvAK5L+T9JnWsg/IiIGRcSgvtosz6q1C3Ma5jOnYR7/Gj8VgEdve4IBO/rn0tHNnTGf9fsuazHo2bcHc2fMa+UKq0XzPnqX9dNWg/VX68b8j94DYNaHi3j8rf/wwdLFLPj4AybMfYMt196QHXtswp69t+SBfX/Ib3c+lF169ueCnb7WWhHWFlHiVsVyH4MQEQHMSrclwLrAbZIuzLvs9u7ttxYwZ/o8+m6xEQA77r0tr3sgWof38tNT6TOgNxv224BOnTsx5PA9eGLMhEpXy8ps3Kx/M3TT7QEYuun2jJv5cpI+82UGrrcx9RKr13diux59mPbOHC5+8UH2vO9ivvz33/Pjp2/jqbmvcsYzd1TyLdQERZS0VbNcn2KQ9EPgaGAucBXwk4hYLKkOeAU4Pc/ya8Flp4zip9efQqcunZg57S0u+u6fKl0lq7DGpY1cOnwkv77vLOrq67j/6od4/UUHjrXsokFfZ/D6/ejepSsP7X8ql770MFf9+3F+t/OhHLrpjrz5/kJOHX8rANPemcvjs//DnXudQBDc9tqzvPLOnAq/gxpW5V/ypVDk+OYknQNcHRGvL+fc5yLipZau3afuG7X7UzezVarhL9tUugpWhV762tmtjntbFfYdfF5J31V/H/+/udcxq9y6GCTVA0csLzgAaC04MDMzs8rKrYshIpZKelnSJhHxRl7lmJmZVUq1jyMoRd4zKa4LTJE0HnivKTEiDs65XDMzs/w5QMjs5znf38zMrHIcIGQTEY/keX8zM7OKqvLpkkuR91TLu0p6WtK7kj6WtFTSojzLNDMzs9Ll3cVwKXAEcCswiGROhC1yLtPMzKwsanmQYjlmUpwK1EfE0oi4Gtg/7zLNzMzKwos1Zfa+pC7AREkXSjq1DGWamZmVR84BgqSNJT0k6UVJU9IZipHUQ9JYSa+kr+um6ZL0B0lTJU2SNDDrW8v7y/qotIyTSR5z3Bj4es5lmpmZ1YolwI8jYitgV+AkSVsBZwIPRsQA4MH0GOAAkkUSBwDDgMuzFpx3gHBIRHwYEYsi4tyIOA34Ss5lmpmZlUfOLQgRMTMink333wFeAvoAQ4HRabbRwCHp/lDg2kg8CXSX1DvLW8s7QDhmOWnfyblMMzOz8mgsbZM0TNKEgm1YS0VJ6gfsCDwF9IqImempWUCvdL8PML3gsoY0baXl8hSDpCOBbwL9JY0pOLUWMD+PMs3MzMqt1KcYImIEMGKF5UjdgL8AP4qIRdKyNZ4iIiSt8hGPeT3m+E9gJtAT+G1B+jvApJzKNDMzK68yPIkgqTNJcPDniLg9TX5LUu+ImJl2IcxO02eQjPdr0jdNW2m5BAjpCo6vA7vlcX8zM7OOQElTwUjgpYj4XcGpMSTd+Oenr3cVpJ8s6SZgF2BhQVfESsl1oiRJ7wBN4VUXoDPwXkSsnWe5ZmZmZdGYewvCHiRPBE6WNDFN+xlJYHCLpONI/iA/LD13D3AgMBV4Hzg2a8F5r8WwVtN+GgUNJXlMw8zMrP3LuYshIh4H1MLpvZeTP4CTVkXZZZu0KH3k4k5gv3KVaWZmlqsankkx7y6GwkmR6kjWY/gwzzLNzMzKpsq/5EuR92JNXy3YXwK8RtLNYGZmZlUs7wDhqoj4R2GCpD1Y9jiGmZlZ+5X/IMWKyXsMwh/bmGZmZtb+RGNpWxXLaybF3YDdgfUlnVZwam2gPo8yzczMys5jEFZaF6Bbev+1CtIXAYfmVKaZmZmtInnNpPgI8Iika9JZFc3MzGpPDY9ByKuL4ZKI+BFw6fIWkIiIg/Mo18zMrKzcxbDSrktfL8rp/mZmZpXnAGGlTZH0I2BzYDIwMiKW5FSWmZlZZdRwgJDXY46jSWZNnAwcwKeXfDYzM7Mql1cLwlYRsS2ApJHA+JzKMTMzq5zG6p7LoBR5BQiLm3YiYkmykKOZmVmNqeEuhrwChO0lLUr3BayRHotkYce1cyrXzMysfBwgrJyI8GyJZmZW+2p4HoS812IwMzOzdijv1RzNzMxqVlT5gkulcIBgZmaWVQ13MThAMDMzy6qGByl6DIKZmZkVcQuCmZlZVp4oyczMzIrUcBeDAwQzM7OMwi0IZmZmVqSGWxA8SNHMzMyKuAXBzMwsK8+DYGZmZkU8k6KZmZk1FzXcguAxCGZmZllFY2lbG0jaX9LLkqZKOjPnd/QJBwhmZmZVSlI9cBlwALAVcKSkrcpRtrsYzMzMMipDF8NgYGpETAOQdBMwFHgx74IdIJiZmWWV/yDFPsD0guMGYJe8C4UqDhDGNt6qStehWkgaFhEjKl0Pqy7+XNjy+HNRXqV+V0kaBgwrSBpRLb8/j0FoH4atOIt1QP5c2PL4c9GORMSIiBhUsDUPDmYAGxcc903TcucAwczMrHo9DQyQ1F9SF+AIYEw5Cq7aLgYzM7OOLiKWSDoZuB+oB0ZFxJRylO0AoX2oiv4oqzr+XNjy+HNRYyLiHuCecperqOGVqMzMzCwbj0EwMzOzIg4QVoKkkHR9wXEnSXMk/bWEe94jqXvGawdJ+kPWsq2yJPWSdIOkaZKekfSEpK9JGtL0mZJ0cDmnVrXKkbRU0kRJL0i6VVLXwn/j6edi9zbcp5+kb+ZfY6t1DhBWznvANpLWSI/3ocTHTSLiwIhYkPHaCRFxSinlW2VIEnAn8GhEbBYRO5GMTu5bmC8ixkTE+ZWoo5XdBxGxQ0RsA3wMHN/s3/gQYIUBAtAPWKkAQZLHo1kRBwgr7x7goHT/SODGphOS1pQ0StJ4Sc9JGpqmf0fS7ZLuk/SKpAsLrnlNUs806n9J0pWSpkj6e1MgImlnSZPSvy5+I+mFNL3wL80eku5M8z0pabs0/RxJoyU9Jul1SV+XdKGkyWl9Oqf5/lfS0+lfLyPSLzDLz17AxxFxRVNCRLweEX8szJR+di5N9/tJGpf+jh+UtEmafo2ky9Pf+7T0czEq/TxdU3CvyyVNSD9f55bnbVpGjwGbN/0bl9QPOB44Nf1/4Avp7/3QpgskvZvung98Ic13avq5eUzSs+m2e5p/SJo+BnhR0nmSflRwv19J+mG53rBVHwcIK+8m4AhJqwPbAU8VnDsLGBcRg4E9gd9IWjM9twNwOLAtcLikwokvmgwALouIrYEFwH+l6VcDP4iIHYClLdTrXOC5iNgO+BlwbcG5z5B8IR0MXA88FBHbAh+wLNi5NCJ2Tv96WQP4yop/FFaCrYFnV/KaPwKj09/xn4HC7qV1gd2AU0mekb44LWNbSTukec6KiEEkn9svNQWRVl3Sv+YPACY3pUXEa8AVwMVpK8NjrdziTOCxNN/FwGxgn4gYSPJ/UOHnZiDww4jYAhgFHJ3WoY6kRet6rMNygLCSImISSRPekRQ/drIvcKakicDDwOrAJum5ByNiYUR8SLLIxqbLuf2rETEx3X8G6JeOT1grIp5I029ooWqfB65L6zgOWE/S2um5eyNiMcl/OPXAfWn65PS9AOwp6SlJk0mCia1b/CHYKifpMknPS3q6lWy7sez3fx3J77zJ3ZE8kjQZeCsiJkdEIzCFZb/jwyQ9CzxH8vsty4pw1mZrpP93TADeAEauovt2Bq5M/23fyqd/7+Mj4lX4JAiZJ2lHkv/LnouIeauoDtYOud8pmzHARSR9gusVpAv4r4h4uTCzpF2AjwqSlrL8n33zPGssJ08WHwFERKOkxbHs2dZGoFPaGvInYFBETJd0DklwY/mZwrIWIiLiJEk9Sb4csmj67DTy6c9R0++4P/DfwM4R8Xba9eDfcXX5IG0l/MQKevqWkP6Rl/7F36WFfKcCbwHbp/k/LDj3XrO8VwHfATYkaVGwDswtCNmMAs6NiMnN0u8Hhjf136eReEnSAYzvpEEGJM1+y/MY8K203CHA3IhY1MZimr4o5krqBhzaWmZbJcYBq0s6oSCt6wqu+SfLfv/fIvmdt9XaJF8GCyX1ImnCtvblHWCtguPXgJ3S/YNJWgqWl28dYGbaonQUSStiS+4A9gd2Jvn/zDowtyBkEBENfLofr8kvgEuASWlE/yqrpi//OJImwkbgEWDhcvKcA4ySNAl4HzimrTePiAWSrgReAGaRzP1tOYqIkHQIcLGk04E5JF/gZ7Ry2XDgakk/SfMfuxLlPS/pOeBfJEvH/iNz5a1S7gZuUzL4eThwJXCXpOdJug2bWgMmAUvT9GtIWgf/IunoZvmKRMTHkh4CFkRES+OdrIPwTIrtgKRuEfFuun8m0DsiPLrYzFap9A+bZ4FvRMQrla6PVZa7GNqHg9JHll4AvgD8stIVMrPaImkrYCrJgGoHB+YWBDMzMyvmFgQzMzMr4gDBzMzMijhAMDMzsyIOEMxyouWszlfCvT6Zd1/SVemAspbyfmrVP0nHp4+4mZm1mQMEs/wUrc5XeFIZV9CLiO9FxIutZBlCwap/EXFFRFzbcnYzs2IOEMzKo3B1vsIV9OqVrND5dLpK4w8gWQ5a0qWSXpb0ALBB040kPSxpULq/f7pC3/PpCo/9KF717xxJ/53m30HJqo+TJN0had2Ce16gZCXSf0v6Qll/OmZWdTyTolnOClbna1okayCwTUS8KmkYsDAidpa0GvAPSX8HdgS2JFlYpxfJAl+jmt13fZLZ9L6Y3qtHRMyXdAXwbkRclObbu+Cya4HhEfGIpPOAs4GmJX47RcRgSQem6V9e1T8LM2s/HCCY5adpdT5IWhBGkjT9f7KCHsmqeds1jS8gmTd/APBF4MZ0uts3JY1bzv13BR4tWI1vfmuVkbQO0D0iHkmTRpOs7tfk9vT1GZatAGlmHZQDBLP8tLQ6X+Fc+CL5i/7+ZvkOzL96RZpWgWxptVEz60A8BsGssu4HTpDUGUDSFpLWBB4FDk/HKPQG9lzOtU8CX0yXckZSjzS9+Wp+AETEQuDtgvEFR5Es/mVmVsR/JZhV1lUkzfnPpsuEzwEOIVl2dy+SsQdvAE80vzAi5qRjGG5PF9mZDexD8ap/hY4BrkgfuZzGSqwIaWYdi9diMDMzsyLuYjAzM7MiDhDMzMysiAMEMzMzK+IAwczMzIo4QDAzM7MiDhDMzMysiAMEMzMzK+IAwczMzIr8f6PoS6euiqP3AAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 648x432 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kHhrPo6Tv9Xc",
        "colab_type": "text"
      },
      "source": [
        "Print the classification report"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MNWrpWVXL_pQ",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 230
        },
        "outputId": "4dd25c3d-45e9-4c8c-cf56-11c58a41cecd"
      },
      "source": [
        "print(f\"Clasification Report\\n\\n{classification_report(pred.view(-1).cpu(), labels.view(-1).cpu())}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Clasification Report\n",
            "\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           1       0.98      1.00      0.99       895\n",
            "           2       1.00      0.99      0.99      1715\n",
            "           3       1.00      0.99      1.00      1070\n",
            "\n",
            "    accuracy                           0.99      3680\n",
            "   macro avg       0.99      0.99      0.99      3680\n",
            "weighted avg       0.99      0.99      0.99      3680\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N5xUvujSwBQc",
        "colab_type": "text"
      },
      "source": [
        "Print the Jaccard Similarity score / Index (Accuracy)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WXiYK3JxMC3F",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 125
        },
        "outputId": "f4288b40-ded1-491d-b242-6c052b3dcbc6"
      },
      "source": [
        "print(f\"Jaccard Index\\n\\n{round(jaccard_similarity_score(pred.view(-1).cpu(), labels.view(-1).cpu()), 2)}\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Jaccard Index\n",
            "\n",
            "0.99\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_classification.py:664: FutureWarning: jaccard_similarity_score has been deprecated and replaced with jaccard_score. It will be removed in version 0.23. This implementation has surprising behavior for binary and multiclass classification tasks.\n",
            "  FutureWarning)\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Qw0YkS8kbs8q",
        "colab_type": "text"
      },
      "source": [
        "We've got around 99% test accuracy which is really good. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HOzZqnNq2EyF",
        "colab_type": "text"
      },
      "source": [
        "## Future Scopes\n",
        "\n",
        "* The model will be deployed using python flask server as a website, so anyone could an upload MRI image of brain and find out what kinda tumor is present (if any).\n",
        "\n",
        "* Tumor detection will be added in future to locate where the tumor is present in the given MRI image of the brain.\n",
        "\n",
        "* Will add more sophisticated regularization techniques to prevent slight overfitting and increase accuracy upto 100 %."
      ]
    }
  ]
}