docs/tutorials/pytorch_lightning/ordinal-corn_mnist.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "e65499bc",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/jovyan/conda/lib/python3.8/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: /home/jovyan/conda/lib/python3.8/site-packages/torchvision/image.so: undefined symbol: _ZNK3c1010TensorImpl36is_contiguous_nondefault_policy_implENS_12MemoryFormatE\n",
      "  warn(f\"Failed to load image Python extension: {e}\")\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch            : 1.12.0\n",
      "pytorch_lightning: 1.6.5\n",
      "torchmetrics     : 0.9.2\n",
      "matplotlib       : 3.3.4\n",
      "coral_pytorch    : 1.3.0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%load_ext watermark\n",
    "%watermark -p torch,pytorch_lightning,torchmetrics,matplotlib,coral_pytorch"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "808da283",
   "metadata": {},
   "source": [
    "# A Convolutional Neural Net for Ordinal Regression using CORN -- MNIST Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d210e7af",
   "metadata": {},
   "source": [
    "In this tutorial, we implement a convolutional neural network for ordinal regression based on the CORN method. To learn more about CORN, please have a look at our preprint:\n",
    "\n",
    "- Xintong Shi, Wenzhi Cao, and Sebastian Raschka (2023). Deep Neural Networks for Rank-Consistent Ordinal Regression Based On Conditional Probabilities. Arxiv preprint;  [https://arxiv.org/abs/2111.08851](https://arxiv.org/abs/2111.08851)\n",
    "\n",
    "\n",
    "Please note that **MNIST is not an ordinal dataset**. The reason why we use MNIST in this tutorial is that it is included in the PyTorch's `torchvision` library and is thus easy to work with, since it doesn't require extra data downloading and preprocessing steps."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d47e374",
   "metadata": {},
   "source": [
    "## General settings and hyperparameters"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a5efc6a",
   "metadata": {},
   "source": [
    "- Here, we specify some general hyperparameter values and general settings\n",
    "- Note that for small datatsets, it is not necessary and better not to use multiple workers as it can sometimes cause issues with too many open files in PyTorch. So, if you have problems with the data loader later, try setting `NUM_WORKERS = 0` instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "315bf6bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "BATCH_SIZE = 256\n",
    "NUM_EPOCHS = 20\n",
    "LEARNING_RATE = 0.005\n",
    "NUM_WORKERS = 4\n",
    "\n",
    "DATA_BASEPATH = \"./\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea6e81b4",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Converting a regular classifier into a CORN ordinal regression model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28ca5d40",
   "metadata": {},
   "source": [
    "Changing a classifier to a CORN model for ordinal regression is actually really simple and only requires a few changes:\n",
    "\n",
    "**1)**\n",
    "\n",
    "Consider the following output layer used by a neural network classifier:\n",
    "\n",
    "```python\n",
    "output_layer = torch.nn.Linear(hidden_units[-1], num_classes)\n",
    "```\n",
    "\n",
    "In CORN we reduce the number of classes by 1:\n",
    "\n",
    "```python\n",
    "output_layer = torch.nn.Linear(hidden_units[-1], num_classes-1)\n",
    "```\n",
    "\n",
    "**2)** \n",
    "\n",
    "We swap the cross entropy loss from PyTorch,\n",
    "\n",
    "```python\n",
    "torch.nn.functional.cross_entropy(logits, true_labels)\n",
    "```\n",
    "\n",
    "with the CORN loss (also provided via `coral_pytorch`):\n",
    "\n",
    "```python\n",
    "loss = corn_loss(logits, true_labels,\n",
    "                 num_classes=num_classes)\n",
    "```\n",
    "\n",
    "Note that we pass `num_classes` instead of `num_classes-1` \n",
    "to the `corn_loss` as it takes care of the rest internally.\n",
    "\n",
    "\n",
    "**3)**\n",
    "\n",
    "In a regular classifier, we usually obtain the predicted class labels as follows:\n",
    "\n",
    "```python\n",
    "predicted_labels = torch.argmax(logits, dim=1)\n",
    "```\n",
    "\n",
    "In CORN, w replace this with the following code to convert the predicted probabilities into the predicted labels:\n",
    "\n",
    "```python\n",
    "predicted_labels = corn_label_from_logits(logits)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7c01b09",
   "metadata": {},
   "source": [
    "## Implementing a `ConvNet` using PyTorch Lightning's `LightningModule`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3bfceef2",
   "metadata": {},
   "source": [
    "- In this section, we set up the main model architecture using the `LightningModule` from PyTorch Lightning.\n",
    "- We start with defining our convolutional neural network `ConvNet` model in pure PyTorch, and then we use it in the `LightningModule` to get all the extra benefits that PyTorch Lightning provides."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "faa69b1f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "\n",
    "# Regular PyTorch Module\n",
    "class ConvNet(torch.nn.Module):\n",
    "    def __init__(self, in_channels, num_classes):\n",
    "        super().__init__()\n",
    "\n",
    "        # num_classes is used by the corn loss function\n",
    "        self.num_classes = num_classes\n",
    "\n",
    "        # Initialize CNN layers\n",
    "        all_layers = [\n",
    "            torch.nn.Conv2d(in_channels=in_channels, out_channels=3, \n",
    "                            kernel_size=(3, 3), stride=(1, 1), \n",
    "                            padding=1),\n",
    "            torch.nn.MaxPool2d(kernel_size=(2, 2),  stride=(2, 2)),\n",
    "            torch.nn.Conv2d(in_channels=3, out_channels=6, \n",
    "                            kernel_size=(3, 3), stride=(1, 1), \n",
    "                            padding=1),\n",
    "            torch.nn.MaxPool2d(kernel_size=(2, 2),  stride=(2, 2)),\n",
    "            torch.nn.Flatten()\n",
    "        ]\n",
    "        \n",
    "        # CORN output layer --------------------------------------\n",
    "        # Regular classifier would use num_classes instead of \n",
    "        # num_classes-1 below\n",
    "        output_layer = torch.nn.Linear(294, num_classes-1)\n",
    "        # ---------------------------------------------------------\n",
    "        \n",
    "        all_layers.append(output_layer)\n",
    "        self.model = torch.nn.Sequential(*all_layers)\n",
    "        \n",
    "    def forward(self, x):\n",
    "        x = self.model(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b76db8b",
   "metadata": {},
   "source": [
    "- In our `LightningModule` we use loggers to track mean absolute errors for both the training and validation set during training; this allows us to select the best model based on validation set performance later.\n",
    "- Given a CNN classifier with cross-entropy loss, it is very easy to change this classifier into a ordinal regression model using CORN. In essence, it only requires three changes:\n",
    "    1. Instead of using `num_classes` in the output layer, use `num_classes-1` as shown above\n",
    "    2. Change the loss from   \n",
    "    `loss = torch.nn.functional.cross_entropy(logits, y)` to  \n",
    "    `loss = corn_loss(logits, y, num_classes=self.num_classes)`\n",
    "    3. To obtain the class/rank labels from the logits, change  \n",
    "    `predicted_labels = torch.argmax(logits, dim=1)` to  \n",
    "    `predicted_labels = corn_label_from_logits(logits)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "81d3ae5e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from coral_pytorch.losses import corn_loss\n",
    "from coral_pytorch.dataset import corn_label_from_logits\n",
    "\n",
    "import pytorch_lightning as pl\n",
    "import torchmetrics\n",
    "\n",
    "\n",
    "# LightningModule that receives a PyTorch model as input\n",
    "class LightningCNN(pl.LightningModule):\n",
    "    def __init__(self, model, learning_rate):\n",
    "        super().__init__()\n",
    "\n",
    "        self.learning_rate = learning_rate\n",
    "        # The inherited PyTorch module\n",
    "        self.model = model\n",
    "\n",
    "        # Save settings and hyperparameters to the log directory\n",
    "        # but skip the model parameters\n",
    "        self.save_hyperparameters(ignore=['model'])\n",
    "\n",
    "        # Set up attributes for computing the MAE\n",
    "        self.train_mae = torchmetrics.MeanAbsoluteError()\n",
    "        self.valid_mae = torchmetrics.MeanAbsoluteError()\n",
    "        self.test_mae = torchmetrics.MeanAbsoluteError()\n",
    "        \n",
    "    # Defining the forward method is only necessary \n",
    "    # if you want to use a Trainer's .predict() method (optional)\n",
    "    def forward(self, x):\n",
    "        return self.model(x)\n",
    "        \n",
    "    # A common forward step to compute the loss and labels\n",
    "    # this is used for training, validation, and testing below\n",
    "    def _shared_step(self, batch):\n",
    "        features, true_labels = batch\n",
    "        logits = self(features)\n",
    "\n",
    "        # Use CORN loss --------------------------------------\n",
    "        # A regular classifier uses:\n",
    "        # loss = torch.nn.functional.cross_entropy(logits, y)\n",
    "        loss = corn_loss(logits, true_labels,\n",
    "                         num_classes=self.model.num_classes)\n",
    "        # ----------------------------------------------------\n",
    "        \n",
    "        # CORN logits to labels ------------------------------\n",
    "        # A regular classifier uses:\n",
    "        # predicted_labels = torch.argmax(logits, dim=1)\n",
    "        predicted_labels = corn_label_from_logits(logits)\n",
    "        # ----------------------------------------------------\n",
    "        \n",
    "        return loss, true_labels, predicted_labels\n",
    "\n",
    "    def training_step(self, batch, batch_idx):\n",
    "        loss, true_labels, predicted_labels = self._shared_step(batch)\n",
    "        self.log(\"train_loss\", loss)\n",
    "        self.train_mae(predicted_labels, true_labels)\n",
    "        self.log(\"train_mae\", self.train_mae, on_epoch=True, on_step=False)\n",
    "        return loss  # this is passed to the optimzer for training\n",
    "\n",
    "    def validation_step(self, batch, batch_idx):\n",
    "        loss, true_labels, predicted_labels = self._shared_step(batch)\n",
    "        self.log(\"valid_loss\", loss)\n",
    "        self.valid_mae(predicted_labels, true_labels)\n",
    "        self.log(\"valid_mae\", self.valid_mae,\n",
    "                 on_epoch=True, on_step=False, prog_bar=True)\n",
    "\n",
    "    def test_step(self, batch, batch_idx):\n",
    "        loss, true_labels, predicted_labels = self._shared_step(batch)\n",
    "        self.test_mae(predicted_labels, true_labels)\n",
    "        self.log(\"test_mae\", self.test_mae, on_epoch=True, on_step=False)\n",
    "\n",
    "    def configure_optimizers(self):\n",
    "        optimizer = torch.optim.Adam(self.parameters(), lr=self.learning_rate)\n",
    "        return optimizer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf9f26d6",
   "metadata": {},
   "source": [
    "## Setting up the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b5e03d5",
   "metadata": {},
   "source": [
    "- In this section, we are going to set up our dataset.\n",
    "- Please note that **MNIST is not an ordinal dataset**. The reason why we use MNIST in this tutorial is that it is included in the PyTorch's `torchvision` library and is thus easy to work with, since it doesn't require extra data downloading and preprocessing steps."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40691e02",
   "metadata": {},
   "source": [
    "### Inspecting the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "68b6367c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz\n",
      "Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to ./MNIST/raw/train-images-idx3-ubyte.gz\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4305da26845a4567bd6bf4952c123b2f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/9912422 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting ./MNIST/raw/train-images-idx3-ubyte.gz to ./MNIST/raw\n",
      "\n",
      "Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz\n",
      "Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz to ./MNIST/raw/train-labels-idx1-ubyte.gz\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4a7dbc69b20e473c9cac6866c48d85a3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/28881 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting ./MNIST/raw/train-labels-idx1-ubyte.gz to ./MNIST/raw\n",
      "\n",
      "Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz\n",
      "Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz to ./MNIST/raw/t10k-images-idx3-ubyte.gz\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "556e8b70e6b142bcac5a0d95dde4422a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1648877 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting ./MNIST/raw/t10k-images-idx3-ubyte.gz to ./MNIST/raw\n",
      "\n",
      "Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz\n",
      "Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz to ./MNIST/raw/t10k-labels-idx1-ubyte.gz\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "83555987def44783ba2328191b905799",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/4542 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting ./MNIST/raw/t10k-labels-idx1-ubyte.gz to ./MNIST/raw\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "from torchvision import datasets\n",
    "from torchvision import transforms\n",
    "from torch.utils.data import DataLoader\n",
    "\n",
    "\n",
    "train_dataset = datasets.MNIST(root=DATA_BASEPATH, \n",
    "                               train=True, \n",
    "                               transform=transforms.ToTensor(),\n",
    "                               download=True)\n",
    "\n",
    "train_loader = DataLoader(dataset=train_dataset, \n",
    "                          batch_size=BATCH_SIZE, \n",
    "                          num_workers=NUM_WORKERS,\n",
    "                          drop_last=True,\n",
    "                          shuffle=True)\n",
    "\n",
    "test_dataset = datasets.MNIST(root=DATA_BASEPATH, \n",
    "                              train=False,\n",
    "                              transform=transforms.ToTensor())\n",
    "\n",
    "test_loader = DataLoader(dataset=test_dataset, \n",
    "                         batch_size=BATCH_SIZE,\n",
    "                         num_workers=NUM_WORKERS,\n",
    "                         drop_last=False,\n",
    "                         shuffle=False)\n",
    "\n",
    "# Checking the dataset\n",
    "all_train_labels = []\n",
    "all_test_labels = []\n",
    "\n",
    "for images, labels in train_loader:  \n",
    "    all_train_labels.append(labels)\n",
    "all_train_labels = torch.cat(all_train_labels)\n",
    "    \n",
    "for images, labels in test_loader:  \n",
    "    all_test_labels.append(labels)\n",
    "all_test_labels = torch.cat(all_test_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "52beca34",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training labels: tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])\n",
      "Training label distribution: tensor([5911, 6730, 5949, 6125, 5832, 5410, 5911, 6254, 5841, 5941])\n",
      "\n",
      "Test labels: tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])\n",
      "Test label distribution: tensor([ 980, 1135, 1032, 1010,  982,  892,  958, 1028,  974, 1009])\n"
     ]
    }
   ],
   "source": [
    "print('Training labels:', torch.unique(all_train_labels))\n",
    "print('Training label distribution:', torch.bincount(all_train_labels))\n",
    "\n",
    "print('\\nTest labels:', torch.unique(all_test_labels))\n",
    "print('Test label distribution:', torch.bincount(all_test_labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91c9e7a0",
   "metadata": {},
   "source": [
    "- Above, we can see that the dataset consists of 8 features, and there are 998 examples in total.\n",
    "- The labels are in range from 1 (weakest) to 5 (strongest), and we normalize them to start at zero (hence, the normalized labels are in the range 0 to 4). \n",
    "- Notice also that the dataset is quite imbalanced."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4620218e",
   "metadata": {},
   "source": [
    "### Performance baseline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2905ec12",
   "metadata": {},
   "source": [
    "- Especially for imbalanced datasets, it's quite useful to compute a performance baseline.\n",
    "- In classification contexts, a useful baseline is to compute the accuracy for a scenario where the model always predicts the majority class -- you want your model to be better than that!\n",
    "- Note that if you are intersted in a single number that minimized the dataset mean squared error (MSE), that's the mean; similary, the median is a number that minimzes the mean absolute error (MAE).\n",
    "- So, if we use the mean absolute error, $\\mathrm{MAE}=\\frac{1}{N} \\sum_{i=1}^{N}\\left|y_{i}-\\hat{y}_{i}\\right|$, to evaluate the model, it is useful to compute the MAE pretending the predicted label is always the median:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "b6abc295",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Baseline MAE: 2.52\n"
     ]
    }
   ],
   "source": [
    "all_test_labels = all_test_labels.float()\n",
    "avg_prediction = torch.median(all_test_labels)  # median minimizes MAE\n",
    "baseline_mae = torch.mean(torch.abs(all_test_labels - avg_prediction))\n",
    "print(f'Baseline MAE: {baseline_mae:.2f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "022027fa",
   "metadata": {},
   "source": [
    "- In other words, a model that would always predict the dataset median would achieve a MAE of 2.52. A model that has an MAE of > 2.52 is certainly a bad model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2884d08",
   "metadata": {},
   "source": [
    "### Setting up a `DataModule`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b578add7",
   "metadata": {},
   "source": [
    "- There are three main ways we can prepare the dataset for Lightning. We can\n",
    "  1. make the dataset part of the model;\n",
    "  2. set up the data loaders as usual and feed them to the fit method of a Lightning Trainer -- the Trainer is introduced in the next subsection;\n",
    "  3. create a LightningDataModule.\n",
    "- Here, we are going to use approach 3, which is the most organized approach. The `LightningDataModule` consists of several self-explanatory methods as we can see below:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "2362cb42",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "from torch.utils.data.dataset import random_split\n",
    "from torch.utils.data import DataLoader\n",
    "\n",
    "\n",
    "class DataModule(pl.LightningDataModule):\n",
    "    def __init__(self, data_path='./'):\n",
    "        super().__init__()\n",
    "        self.data_path = data_path\n",
    "        \n",
    "    def prepare_data(self):\n",
    "        datasets.MNIST(root=self.data_path,\n",
    "                       download=True)\n",
    "        return\n",
    "\n",
    "    def setup(self, stage=None):\n",
    "        # Note transforms.ToTensor() scales input images\n",
    "        # to 0-1 range\n",
    "        train = datasets.MNIST(root=self.data_path, \n",
    "                               train=True, \n",
    "                               transform=transforms.ToTensor(),\n",
    "                               download=False)\n",
    "\n",
    "        self.test = datasets.MNIST(root=self.data_path, \n",
    "                                   train=False, \n",
    "                                   transform=transforms.ToTensor(),\n",
    "                                   download=False)\n",
    "\n",
    "        self.train, self.valid = random_split(train, lengths=[55000, 5000])\n",
    "\n",
    "    def train_dataloader(self):\n",
    "        train_loader = DataLoader(dataset=self.train, \n",
    "                                  batch_size=BATCH_SIZE, \n",
    "                                  drop_last=True,\n",
    "                                  shuffle=True,\n",
    "                                  num_workers=NUM_WORKERS)\n",
    "        return train_loader\n",
    "\n",
    "    def val_dataloader(self):\n",
    "        valid_loader = DataLoader(dataset=self.valid, \n",
    "                                  batch_size=BATCH_SIZE, \n",
    "                                  drop_last=False,\n",
    "                                  shuffle=False,\n",
    "                                  num_workers=NUM_WORKERS)\n",
    "        return valid_loader\n",
    "\n",
    "    def test_dataloader(self):\n",
    "        test_loader = DataLoader(dataset=self.test, \n",
    "                                 batch_size=BATCH_SIZE, \n",
    "                                 drop_last=False,\n",
    "                                 shuffle=False,\n",
    "                                 num_workers=NUM_WORKERS)\n",
    "        return test_loader"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5cf0ae36",
   "metadata": {},
   "source": [
    "- Note that the `prepare_data` method is usually used for steps that only need to be executed once, for example, downloading the dataset; the `setup` method defines the the dataset loading -- if you run your code in a distributed setting, this will be called on each node / GPU. \n",
    "- Next, lets initialize the `DataModule`; we use a random seed for reproducibility (so that the data set is shuffled the same way when we re-execute this code):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "ca66cd16",
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.manual_seed(1) \n",
    "data_module = DataModule(data_path=DATA_BASEPATH)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "075cda09",
   "metadata": {},
   "source": [
    "## Training the model using the PyTorch Lightning Trainer class"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69f27da4",
   "metadata": {},
   "source": [
    "- Next, we initialize our CNN (`ConvNet`) model.\n",
    "- Also, we define a call back so that we can obtain the model with the best validation set performance after training.\n",
    "- PyTorch Lightning offers [many advanced logging services](https://pytorch-lightning.readthedocs.io/en/latest/extensions/logging.html) like Weights & Biases. Here, we will keep things simple and use the `CSVLogger`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "84e29c51",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pytorch_lightning.callbacks import ModelCheckpoint\n",
    "from pytorch_lightning.loggers import CSVLogger\n",
    "\n",
    "\n",
    "pytorch_model = ConvNet(\n",
    "    in_channels=1,\n",
    "    num_classes=torch.unique(all_test_labels).shape[0])\n",
    "\n",
    "lightning_model = LightningCNN(\n",
    "    pytorch_model, learning_rate=LEARNING_RATE)\n",
    "\n",
    "callbacks = [ModelCheckpoint(\n",
    "    save_top_k=1, mode='min', monitor=\"valid_mae\")]  # save top 1 model \n",
    "logger = CSVLogger(save_dir=\"logs/\", name=\"cnn-corn-mnist\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ea7ec81",
   "metadata": {},
   "source": [
    "- Now it's time to train our model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "2fd44464",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/jovyan/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/callback_connector.py:96: LightningDeprecationWarning: Setting `Trainer(progress_bar_refresh_rate=50)` is deprecated in v1.5 and will be removed in v1.7. Please pass `pytorch_lightning.callbacks.progress.TQDMProgressBar` with `refresh_rate` directly to the Trainer's `callbacks` argument instead. Or, to disable the progress bar pass `enable_progress_bar = False` to the Trainer.\n",
      "  rank_zero_deprecation(\n",
      "GPU available: True, used: True\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "HPU available: False, using: 0 HPUs\n",
      "Missing logger folder: logs/cnn-corn-mnist\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "\n",
      "  | Name      | Type              | Params\n",
      "------------------------------------------------\n",
      "0 | model     | ConvNet           | 2.9 K \n",
      "1 | train_mae | MeanAbsoluteError | 0     \n",
      "2 | valid_mae | MeanAbsoluteError | 0     \n",
      "3 | test_mae  | MeanAbsoluteError | 0     \n",
      "------------------------------------------------\n",
      "2.9 K     Trainable params\n",
      "0         Non-trainable params\n",
      "2.9 K     Total params\n",
      "0.011     Total estimated model params size (MB)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Sanity Checking: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4a8a5666cc804b8cbc4e73b520a92cda",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Training: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Validation: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training took 1.38 min in total.\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "\n",
    "trainer = pl.Trainer(\n",
    "    max_epochs=NUM_EPOCHS,\n",
    "    callbacks=callbacks,\n",
    "    progress_bar_refresh_rate=50,  # recommended for notebooks\n",
    "    accelerator=\"auto\",  # Uses GPUs or TPUs if available\n",
    "    devices=\"auto\",  # Uses all available GPUs/TPUs if applicable\n",
    "    logger=logger,\n",
    "    deterministic=True,\n",
    "    log_every_n_steps=10)\n",
    "\n",
    "start_time = time.time()\n",
    "trainer.fit(model=lightning_model, datamodule=data_module)\n",
    "\n",
    "runtime = (time.time() - start_time)/60\n",
    "print(f\"Training took {runtime:.2f} min in total.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "372d16d0",
   "metadata": {},
   "source": [
    "## Evaluating the model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95060cfe",
   "metadata": {},
   "source": [
    "- After training, let's plot our training MAE and validation MAE using pandas, which, in turn, uses matplotlib for plotting (you may want to consider a [more advanced logger](https://pytorch-lightning.readthedocs.io/en/latest/extensions/logging.html) that does that for you):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "e2058839",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:xlabel='Epoch', ylabel='MAE'>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA2tElEQVR4nO3de3xcdZn48c+TyWVymUnatJmmTS8pLZdCW9rGtohAuYjlokVELIqIu9hlFxTYRYF1WdF1d/XnHa1UREBdFBEFCxaK1IaKQmlLL7QN2HtTeknaNJfJ/fL8/jgn6WQ6SSeXk4TM8369zmvO5XvmPHMyOc98zznf7xFVxRhjjImWNNgBGGOMGZosQRhjjInJEoQxxpiYLEEYY4yJyRKEMcaYmJIHO4D+NGrUKJ00aVKv1q2trSUzM7N/A+pHFl/fWHx9Y/H1zVCOb8OGDUdVdXTMhao6bIY5c+Zob61evbrX6w4Ei69vLL6+sfj6ZijHB6zXLo6pdorJGGNMTJ4mCBFZKCLviMhOEbk3xvIzReQ1EWkUkbujluWIyNMi8raIlIjIeV7GaowxpjPPrkGIiA9YCnwQOACsE5Hlqro9olgF8AXgmhhv8QPgRVW9TkRSgQyvYjXGGHMyLy9SzwV2qupuABF5ElgEdCQIVS0DykTkqsgVRSQIXAjc7JZrApo8jNUYMwQ1Nzdz4MABGhoaui2XnZ1NSUnJAEXVc0MhPr/fT0FBASkpKXGv42WCGAeURkwfAObFue5koBx4TERmAhuAO1S1NrqgiCwBlgCEQiGKi4t7FWw4HO71ugPB4usbi69vBiu+rKwsQqEQ48aNQ0S6LNfa2orP5xvAyHpmsONTVaqqqti8eTPhcDju9bxMELH+mvH2DJgMzAY+r6prReQHwL3A/Se9oerDwMMARUVFumDBgl4FW1xcTG/XHQgWX99YfH0zWPGVlJRQUFDQbXIAqKmpIRAIDFBUPTcU4gsEAoTDYYqKiuJex8uL1AeA8RHTBcDBHqx7QFXXutNP4yQMY0yCOVVyMPHpzX70MkGsA6aKSKF7kXkxsDyeFVX1MFAqIme4sy4l4tpFf1JVfrhqB2+Vt3jx9sYY857l2SkmVW0RkduBlYAPeFRVt4nIre7yZSIyBlgPBIE2EbkTmKaq1cDngSfc5LIb+KwXcYoID/9lN/PyvHh3Y4x57/K0HYSqrlDV01X1NFX9b3feMlVd5o4fVtUCVQ2qao47Xu0u26SqRao6Q1WvUdXjXsUZCvqpbLQHJxljOqusrOTHP/5xj9e78sorqays7PF6N998M08//XSP1/OKtaQG8gJpliCMMSfpKkG0trZ2u96KFSvIycnxKKqBM6w66+utUNDP3w9agjBmKPvqc9vYfrA65rLe3kY6bWyQr3z47C6X33vvvezatYtzzz2XlJQUsrKyyM/PZ9OmTWzfvp1rrrmG0tJSGhoauOOOO1iyZAkAkyZNYv369YTDYa644grmzZvHunXrGDduHH/4wx9IT08/ZWyrVq3i7rvvpqWlhfe973089NBDpKWlce+997J8+XKSk5O5/PLL+fa3v81vf/tbvvrVr+Lz+cjOzmbNmjU93hexWIIA8oJpVDY4nVPZHRPGmHbf+MY32Lp1K5s2baK4uJirrrqKrVu3UlhYCMCjjz7KyJEjqa+v533vex8f+9jHyM3N7fQeO3bs4JFHHuHxxx/n+uuv53e/+x033nhjt9ttaGjg5ptvZtWqVZx++uncdNNNPPTQQ9x0000888wzvP3224hIx2msr33ta6xcuZJx48b16tRWVyxBAHkBPy0KlXXNjMhMHexwjDExdPdLf6DaGcydO7cjOQA8+OCDPPPMMwCUlpayY8eOkxJEYWEhM2bMAGDOnDns3bv3lNt55513KCws5PTTTwfgM5/5DEuXLuX222/H7/dzyy23cNVVV3H11VcDcP7553PzzTdz/fXXc+211/bHRwXsGgQAoWAaAGU1jYMciTFmKIt8pkNxcTEvv/wyr732Gps3b2bWrFkxuwRJS0vrGPf5fLS0nPqWeqcX7pMlJyfzxhtv8LGPfYxnn32WhQsXArBs2TK+/vWvU1payrnnnsuxY8d6+tFib69f3uU9LhT0A3CkuoEzxgzd1pjGmIEVCASoqamJuayqqooRI0aQkZHB22+/zeuvv95v2z3zzDPZu3cvO3fuZMqUKfzyl7/koosuIhwOU1dXx5VXXsn8+fOZMmUKALt27WLevHnMmzeP5557jtLS0pNqMr1hCQLnLiZwEoQxxrTLzc3l/PPP55xzziE9PZ1QKNSxbOHChSxbtowZM2ZwxhlnMH/+/H7brt/v57HHHuPjH/94x0XqW2+9lYqKChYtWkRDQwOqyve+9z0AvvjFL7Jjxw5UlUsvvZSZM2f2SxyWIHCuQYCdYjLGnOxXv/pVzPlpaWm88MILMZe1X2cYNWoUW7du7aiF3H333THLt3v88cc7xi+99FI2btzYaXl+fj5vvPHGSev9/ve/7/Z9e8uuQQDpqT4ykqHMahDGGNPBahCunDThSLXVIIwx3rvtttv461//2mneHXfcwWc/60mPQr1mCcKV4xfKaqwGYYzx3tKlSwc7hLjYKSZXTlqS1SCMMSaCJQhXTppQXtPY5f3HxhiTaCxBuHLShKbWNirrmgc7FGOMGRIsQbhy/E4fTEfsOoQxxgCWIDrkpDkJosyuQxhjeikrKwuAgwcPct1118Uss2DBAtavX9/le0yaNImjR496El9PWYJwtScIa01tjOmrsWPHDqkH//SW3ebq6qhBWGtqY4amF+6Fw2/FXJTe2gK+XhzOxkyHK77R5eJ77rmHiRMn8i//8i8APPDAA4gIa9as4fjx4zQ3N/P1r3+dRYsWdVpv7969XH311WzdupX6+npuvvlmduzYwVlnnUV9fX3c4X33u9/l0UcfBeCWW27hzjvvpLa2luuvv54DBw7Q2trK/fffzyc+8YmYz4noK0sQrlSfEPQnW2tqY0yHxYsXc+edd3YkiKeeeooXX3yRu+66i2AwyNGjR5k/fz4f+chHunyWzEMPPURGRgZbtmxhy5YtzJ49O65tb9iwgccee4y1a9eiqsybN4+LLrqI3bt3M3bsWP74xz8CTqeBFRUVMZ8T0VeeJggRWQj8APABj6jqN6KWnwk8BswGvqyq345a7gPWA++q6tVexgpOr67WFsKYIaqbX/r1Hj0PYtasWZSVlXHw4EHKy8sZMWIE+fn53HXXXaxZs4akpCTeffddjhw5wpgxY2K+x5o1a7jlllsAmDFjRsezIU7l1Vdf5aMf/WhHF+PXXnstf/nLX1i4cCF3330399xzD1dffTUXXHABLS0tMZ8T0VeeXYNwD+5LgSuAacANIjItqlgF8AWgq7rQHUCJVzFGCwX9dheTMaaT6667jqeffprf/OY3LF68mCeeeILy8nI2bNjApk2bCIVCMZ8DEak3T6rsqk3W6aefzoYNG5g+fTr33XcfX/va17p8TkRfeXmRei6wU1V3q2oT8CTQ6USdqpap6jrgpMYHIlIAXAU84mGMneQF0uwuJmNMJ4sXL+bJJ5/k6aef5rrrrqOqqoq8vDxSUlJYvXo1+/bt63b9Cy+8kKeeegqArVu3smXLlri2e+GFF/Lss89SV1dHbW0tzzzzDBdccAEHDx4kIyODG2+8kbvvvps333yTcDhMVVUVV155Jd///vfZtGlTXz824O0ppnFAacT0AWBeD9b/PvAloNt6o4gsAZYAhEIhiouLexRku3A4TENlE4ermlm9evWQezZ1OBzu9WcbCBZf31h8sWVnZ3f5wJ5Ira2tcZXrjQkTJlBVVcWYMWPIyspi0aJFXH/99cyePZvp06dz+umnEw6HO7ZfU1NDOBymra2NmpoabrzxRl599VXOOeccpk+fzpw5c6itre0yXlUlHA4zdepUbrjhBoqKigC46aabmDJlCi+//DL3338/SUlJJCcn873vfY9Dhw6xePFiGhud3iD+53/+J+b7NzQ09OzvqKqeDMDHca47tE9/GvhhF2UfAO6OmL4a+LE7vgB4Pp5tzpkzR3tr9erV+uiru3XiPc/rsXBjr9/HK6tXrx7sELpl8fWNxRfb9u3b4ypXXV3tcSR9M1Tii7U/gfXaxTHVy1NMB4DxEdMFwME41z0f+IiI7MU5NXWJiPxf/4Z3shMPDrLrEMYY4+UppnXAVBEpBN4FFgOfjGdFVb0PuA9ARBbg1C5u9CbME0LB9kePNnJm7BsSjDGmX8ybN4/Gxs7XPH/5y18yffr0QYroZJ4lCFVtEZHbgZU4t7k+qqrbRORWd/kyERmDcxtrEGgTkTuBaapa7VVc3QkFnRqEtaY2ZuhQ1SF3TbA/rF27dkC3p73oqdrTdhCqugJYETVvWcT4YZxTT929RzFQ7EF4JxkdcGoQ5daa2pghwe/3c+zYMXJzc4dlkhgoqsqxY8fw+/09Ws9aUkfwp/jITk+xGoQxQ0RBQQEHDhygvLy823INDQ09PvgNpKEQn9/vp6Cg29/jJ7EEESUvkGYJwpghIiUlhcLCwlOWKy4uZtasWQMQUe8M9fi6Yr25RgkF/dZhnzHGYAniJHlBa01tjDFgCeIkeQE/ZTUN9mxqY0zCswQRJRRMo7lVOW7PpjbGJDhLEFGsLYQxxjgsQUTJC7S3prYEYYxJbJYgorTXIOxOJmNMorMEEaW9NbU9etQYk+gsQUQ50ZraahDGmMRmCSKGUDDNuvw2xiQ8SxAxhIJ+q0EYYxKeJYgYRgfSrEdXY0zCswQRg9MfUwNtbdaa2hiTuCxBxBAKtLembhrsUIwxZtBYgoghz9pCGGOMJYhYTjyb2u5kMsYkLksQMeQF3BqE3clkjElgniYIEVkoIu+IyE4RuTfG8jNF5DURaRSRuyPmjxeR1SJSIiLbROQOL+OM1tGa2tpCGGMSmGePHBURH7AU+CBwAFgnIstVdXtEsQrgC8A1Uau3AP+mqm+KSADYICJ/ilrXM/4UHzkZ1praGJPYvKxBzAV2qupuVW0CngQWRRZQ1TJVXQc0R80/pKpvuuM1QAkwzsNYTxIK+O0ahDEmoYlXT04TkeuAhap6izv9aWCeqt4eo+wDQFhVvx1j2SRgDXCOqlbHWL4EWAIQCoXmPPnkk72KNxwOk5WV1TH9rXX11LfAf56X3qv362/R8Q01Fl/fWHx9Y/H13sUXX7xBVYtiLfPsFBMgMeb1KBuJSBbwO+DOWMkBQFUfBh4GKCoq0gULFvQwTEdxcTGR6z5XtpnXdh2lt+/X36LjG2osvr6x+PrG4vOGl6eYDgDjI6YLgIPxriwiKTjJ4QlV/X0/x3ZKTod9jdaa2hiTsLxMEOuAqSJSKCKpwGJgeTwriogAPwNKVPW7HsbYpbxAGi1t1praGJO4PDvFpKotInI7sBLwAY+q6jYRudVdvkxExgDrgSDQJiJ3AtOAGcCngbdEZJP7lv+uqiu8ijfaiWdTN5KblTZQmzXGmCHDy2sQuAf0FVHzlkWMH8Y59RTtVWJfwxgw7d1tHKlpYBrBwQzFGGMGhbWk7kKe21iu3NpCGGMSlCWILuRZf0zGmARnCaILack+RmSkcMS62zDGJChLEN3IC/itwz5jTMKyBNGNvGAaR+yZEMaYBGUJohtODcJOMRljEpMliG6EgmmUW2tqY0yCsgTRjVDQT0ubUmGtqY0xCcgSRDfa20LYra7GmERkCaIb7a2py+xCtTEmAVmC6EbIbSxnF6qNMYnIEkQ3RnecYrIahDEm8ViC6EZ7a+oya01tjElAliBOIRT0Ww3CGJOQLEGcwuhAml2kNsYkJEsQpxAKWmtqY0xisgRxCtaa2hiTqCxBnEJewFpTG2MSkyWIUwjZg4OMMQnK0wQhIgtF5B0R2Ski98ZYfqaIvCYijSJyd0/WHSgdrantTiZjTILxLEGIiA9YClwBTANuEJFpUcUqgC8A3+7FugOivT8mawthjEk0XtYg5gI7VXW3qjYBTwKLIguoapmqrgOae7ruQLHW1MaYRJXs4XuPA0ojpg8A8/p7XRFZAiwBCIVCFBcX9zhQgHA43OW6gRTY+PZuin3v9uq9+0N38Q0FFl/fWHx9Y/F5w8sEITHmxXuvaNzrqurDwMMARUVFumDBgjg30VlxcTFdrTtu0xqSAxksWFDUq/fuD93FNxRYfH1j8fWNxecNL08xHQDGR0wXAAcHYN1+l2eN5YwxCcjLBLEOmCoihSKSCiwGlg/Auv0uFEizaxDGmITj2SkmVW0RkduBlYAPeFRVt4nIre7yZSIyBlgPBIE2EbkTmKaq1bHW9SrWU8kLplEedlpTJyXFOvtljDHDj5fXIFDVFcCKqHnLIsYP45w+imvdwRIK+mltU47VNnXc1WSMMcOdtaSOQ17AaSxnramNMYnEEkQc8tzuNsqt229jTAKxBBGHUNBqEMaYxGMJIg6js6w1tTEm8ViCiENqchIjM1OtPyZjTEKJK0GISKaIJLnjp4vIR0QkxdvQhpY8awthjEkw8dYg1gB+ERkHrAI+CzzuVVBDUV7QbzUIY0xCiTdBiKrWAdcCP1TVj+J0w50wQoE0eyaEMSahxJ0gROQ84FPAH915njayG2pCQT/l4UZa7dnUxpgEEW+CuBO4D3jG7S5jMrDas6iGoLxgmtua2moRxpjEEFctQFVfAV4BcC9WH1XVL3gZ2FDT3pq6rLqxY9wYY4azeO9i+pWIBEUkE9gOvCMiX/Q2tKElFLRHjxpjEku8p5imqWo1cA1OB3oTgE97FdRQlNfRmtpOMRljEkO8CSLFbfdwDfAHVW0m/qfDDQvtrantTiZjTKKIN0H8BNgLZAJrRGQiUO1VUENRanISuZmpHLFTTMaYBBHvReoHgQcjZu0TkYu9CWnoGm1tIYwxCSTei9TZIvJdEVnvDt/BqU0klJC1pjbGJJB4TzE9CtQA17tDNfCYV0ENVaFgmnX5bYxJGPG2hj5NVT8WMf1VEdnkQTxDWl7Az9FwE61tis+eTW2MGebirUHUi8gH2idE5Hyg/lQrichCEXlHRHaKyL0xlouIPOgu3yIisyOW3SUi20Rkq4j8WkQGvXVayFpTG2MSSLwJ4lZgqYjsFZG9wI+Af+puBRHxAUuBK3A69rtBRKI7+LsCmOoOS4CH3HXHAV8AilT1HMAHLI4zVs+0t4WwC9XGmEQQV4JQ1c2qOhOYAcxQ1VnAJadYbS6wU1V3q2oT8CSwKKrMIuAX6ngdyBGRfHdZMpAuIslABnAwvo/knbyAtaY2xiSOHvXI6rambvevwPe7KT4OKI2YPgDMi6PMOFVdLyLfBvbjnMp6SVVfirUREVmCU/sgFApRXFx86g8SQzgcPuW6x+rbAFizbgtJhwf2eUnxxDeYLL6+sfj6xuLzRl+67D7VVdpYy6NbX8csIyIjcGoXhUAl8FsRuVFV/++kwqoPAw8DFBUV6YIFC04RVmzFxcWcat3m1jb+7ZUXyMmfyIIFp/dqO70VT3yDyeLrG4uvbyw+b/TlmdSn6mrjADA+YrqAk08TdVXmMmCPqpa73Xr8Hnh/H2LtFyk+pzV1WY1dgzDGDH/dJggRqRGR6hhDDTD2FO+9DpgqIoUikopzkXl5VJnlwE3u3UzzgSpVPYRzamm+iGSIiACXAiW9+YD9LS/op8zaQhhjEkC3p5hUNdDbN1bVFhG5HViJcxfSo+7Dhm51ly/D6Rn2SmAnUIfzrGtUda2IPA28CbQAG3FPIw02p7Gc1SCMMcOfp48NVdUVOEkgct6yiHEFbuti3a8AX/Eyvt7IC6RRciih+ik0xiSovlyDSEihoJ/yGns2tTFm+LME0UN5QT9tCsfCdprJGDO8WYLooRON5SxBGGOGN0sQPRTqePSo3clkjBneLEH0UCjo1CDsTiZjzHBnCaKHRmWlIWL9MRljhj9LED3U3praahDGmOHOEkQvjA5Ya2pjzPBnCaIXQsE0u4vJGDPsWYLohVDAb3cxGWOGPUsQvZAXTONo2FpTG2OGN0sQvWCtqY0xicASRC+EAtYWwhgz/FmC6IU8a01tjEkAliB6ob01td3JZIwZzixB9EJ7a2qrQRhjhjNLEL1gz6Y2xiQCSxC9lGetqY0xw5ynCUJEForIOyKyU0TujbFcRORBd/kWEZkdsSxHRJ4WkbdFpEREzvMy1p4KBdM4Yh32GWOGMc8ShIj4gKXAFcA04AYRmRZV7ApgqjssAR6KWPYD4EVVPROYCZR4FWtvODUIO8VkjBm+vKxBzAV2qupuVW0CngQWRZVZBPxCHa8DOSKSLyJB4ELgZwCq2qSqlR7G2mMhtzV1S2vbYIdijDGe8DJBjANKI6YPuPPiKTMZKAceE5GNIvKIiGR6GGuPdbSmrm0a7FCMMcYTyR6+t8SYF915UVdlkoHZwOdVda2I/AC4F7j/pI2ILME5PUUoFKK4uLhXwYbD4R6tW3akBYAXVv+VSdm+Xm2zJ3oa30Cz+PrG4usbi88jqurJAJwHrIyYvg+4L6rMT4AbIqbfAfKBMcDeiPkXAH881TbnzJmjvbV69eoeld+0/7hOvOd5/dO2w73eZk/0NL6BZvH1jcXXNxZf7wHrtYtjqpenmNYBU0WkUERSgcXA8qgyy4Gb3LuZ5gNVqnpIVQ8DpSJyhlvuUmC7Z5E2hvG11PdolVB7dxt2J5MxZpjy7BSTqraIyO3ASsAHPKqq20TkVnf5MmAFcCWwE6gDPhvxFp8HnnCTy+6oZf2nvhKWzmV87kU4N1XFZ1RWqvNsaruTyRgzTHl5DQJVXYGTBCLnLYsYV+C2LtbdBBR5GR8A6Tkw8XzGl/wBwv8NWXlxrZbsSyI3M40yq0EYY4Ypa0kNcMl/kNTWDGu+1aPVQsE06/LbGDNsWYIAyD2NQ/kfhPWPQcWeuFfLC1gNwhgzfFmCcO2d9AlISobV/xP3OqGg32oQxphhyxKEqyktF+bfCm/9Fg6/Fdc6eUG/taY2xgxbliAinX8H+IOw6r/iKp4XSEOtNbUxZpiyBBEpfQR84C7YsRL2/e2UxUP26FFjzDBmCSLa3H+CQD68/ABodM8gnbU/etSuQxhjhiNLENFSM+Cie6B0Lfz9xW6L5gWcGoTdyWSMGY4sQcQy60YYeRq8/FVoa+2yWHtraqtBGGOGI0sQsfhS4JL/gPIS2PJUl8U6WlPbNQhjzDBkCaIr066B/HOddhEtXdcQQsE0ymqsBmGMGX4sQXQlKQku+wpU7Yf1j3ZZzGksZzUIY8zwYwmiO5MvhsILnT6aGmtiFskLWH9MxpjhyRJEd0Tgsgeg7hi8tjRmkbygn2O11praGDP8WII4lXFz4KyPwN9+COHykxaHgk5r6qNha01tjBleLEHE45L7obkO/vKdkxZZWwhjzHBlCSIeo0+Hcz8F638Gx/d1WmStqY0xw5UliHgtuA8QKP5Gp9nWH5MxZriyBBGv7HEwbwls/jUc2d4xOzfTaU1dWlE3iMEZY0z/8zRBiMhCEXlHRHaKyL0xlouIPOgu3yIis6OW+0Rko4g872WccfvAv0JaAP58ojvwZF8S5582ip+9uocX3jo0iMEZY0z/8ixBiIgPWApcAUwDbhCRaVHFrgCmusMS4KGo5XcAJV7F2GMZI+H8L8A7K2D/2o7ZD904m5njc7j91xt5fsvBQQzQGGP6j5c1iLnATlXdrapNwJPAoqgyi4BfqON1IEdE8gFEpAC4CnjEwxh7bv6/QGZep+7AA/4Ufv4Pc5kzYQRf+PVG/rDp3cGN0Rhj+oGXCWIcUBoxfcCdF2+Z7wNfAoZWC7TUTLjoS7D/b7DjTx2zs9KSefwf3sfcwpHc9ZtN/G7DgUEM0hhj+i7Zw/eWGPOin8ATs4yIXA2UqeoGEVnQ7UZEluCcniIUClFcXNzzSIFwOBz3utI2ibn+MbT+4YusL0oGOZFnP3uaUl2VxN2/3cy2khIuLEjpVTx9iW8wWHx9Y/H1jcXnEVX1ZADOA1ZGTN8H3BdV5ifADRHT7wD5wP/i1Cb2AoeBOuD/TrXNOXPmaG+tXr26Zyts+a3qV4Kqm39z0qL6pha98ZHXdeI9z+sTr+/rdUx9im+AWXx9Y/H1jcXXe8B67eKY6uUppnXAVBEpFJFUYDGwPKrMcuAm926m+UCVqh5S1ftUtUBVJ7nr/VlVb/Qw1p47+1oITYc/fx1aOnez4U/x8dObilhwxmj+/Zm3+OXr+7p4E2OMGbo8SxCq2gLcDqzEuRPpKVXdJiK3isitbrEVwG5gJ/BT4F+8iqfftXcHXrkP3vz5SYv9KT5+8uk5XHZWHvc/u5XH/7pnEII0xpje8/IaBKq6AicJRM5bFjGuwG2neI9ioNiD8PpuymUw8Xx45Zsw8wZIy+q0OC3Zx48/NYfbf/UmDzy3nZY25ZYLJg9SsMYY0zPWkrov2rsDry2H33/upH6aAFKTk1j6qdlcOX0MX/9jCcte2TXwcRpjTC9Yguir8XOdJLHrz/CjIlj5Zair6FQkxZfEg4tn8eGZY/nGC2+zdPXOwYnVGGN6wBJEf/jAXfD5N2H69c6DhR6c5Tw/IuJZ1sm+JL53/UyuOXcs31r5Dj94eccgBmyMMadmCaK/ZI+Da5bCra9CQRG89B9OjWLLb6HNaeuX7EviO9efy8dmF/C9l//Od196p/32XmOMGXIsQfS3MefAjb+DTz8L/mz4/S3w04thzxoAfEnCt66bwSeKxvPgn3fyrZWWJIwxQ5MlCK+cdjEsWQMf/QnUHoWffxieuB7KSkhKEv732uncMHcCPy7exf++8LYlCWPMkOPpba4JLykJZi6GaYtg7U/gL9+Fh94Ps24k6eIv89/XnENykvDwmt1s2l/JldPH8MGzxzAuJ32wIzfGGEsQAyIlHT5wJ8y+CdZ8C974Kbz1NEnv/zxfW3g7E0Zm8NT6Uh54bjsPPLed6eOy+dDZIS4/ewxT87IQidVllTHGeMsSxEDKGAkL/xfmfg5W/Re88k1k/WN8bsG9fO4LN7L7eDMvbT/Cym2H+fZLf+fbL/2dwlGZXH52iNENrVzYpiQlWbIwxgwMSxCDYeRk+PhjcN5t8NL98Md/hRfuYXLobG4dey63zpvFsQ9OY2X5SF4oOcbP/rKHljbl4e2r+OA0p2Zx3uRcUpPtEpIxxjuWIAZTQRF8doXTyG7PGji0CbY9AxseJxf4pC+VT4bOpnHuTFYdSmdT2iye3NjAE2v3E/Anc8mZeVw+bQwLzhhNZpr9KY0x/cuOKoNNBKZc6gzgPKXu+B44uAkOboRDm0greYYrG6u4ErgvJY3qUWewjcn86Z18frh5Inf7xnPW2BGcNjqL0/KynNfRmUwYmUGyz2oZxpjesQQx1Ig4p6BGToZzrnXmtbWx9sXfMG98CnJwI9mHNvP+g6t4v9ZAGjRLKkePjaKiPINjW9KpJpPXNJOXJBNfxgjSg7kERowiNzePUF6Isfn5ZAZznXYaSb7B/bzGmCHLEsR7QVIS9Rn5MH0BTL/OmdfWBhW74eBGUg5tIr/mEPkNVbTUHqeltgwaKklprsbX2ALlOEMMDUmZtKRm05YVInnkRNJHFyIjJkD2BMiZADnjnbuwzMlUobEG6o6CLxUC+ZZwzbBiCeK9KikJRk1xhhkf75idTMQfVRWa66ChiuZwBUfKDnOkrIzjx8qoqTxKfU0FrbXHyaitIVRbQUHZa4x95zlSpbXTpprTRyE5E0ke2Z40JkDORMge7ySQ1MwB+9ieUoWGSqdhY+1R58Df8Xos9nRrxMOikpIhOO7E/skZ74xnu6/BceCzfznz3mHf1uFMxDl4p2aSEhxLwdhzKIgqoqocqW5k99Ewfz1ay56yaiqOHKD52F6Sa0oZq+UUtJRTEC5nwqHXGctzpNDS+T0yRvE+MmDXWPAHIS3onL7qNJ4de35qphOn17QVqt6FqgNQVQrV7ePudLgM6o5BW0vs9VOzICMXMkdBYCyMmeGMZ4xy5rc2QuV+qCx1XnetgppDnd9DfBAc2zlpuEnEX3/IeTJhcqr3+wKgtcV52FXFbji2y/nsLQ1OwmtpjHhthJYmZhw9DLszO6ad14hymaOg8EJnmHSBc0u3ec+zBJHgRIQx2X7GZPt5/2mj3LnTAWhubWN/RR17ymt5+2gtK46G2VNWQ/XRd/HXvkuBlFMg5YyvKSdXwuTU1RCgjIDUEdBaMqkjmdauNw60kkSjL5PWlAD+rBGkZLQnkICTRNICJxJKWjDGsmzntam28wG/UwI4wEVV78IrbZ03npYN2QXOMHaWc7BvP+hn5naeTvH3fOe2NDrbr9zvDFWlJ5LI3leh5iCoE9N8gLX/fCKBdNRCImpt2QXgS4l/+22tzvYqdsGx3U4yqNjlJITKfVHJUCA5DXxpTpKKevW1NkBSEFJy3HKpnV+P74NNv4Z1jzjvNWa6kywmL4AJ5530MC3z3mAJwnQpxZfk3hF18j93uLGFPeW17D4aZs/RWlbv2EMofxwtbW00tSgtbW00t7QiLQ2kNFeT2lJLSksNaS01pLXW4m8Nk94axt9WS0ZbmKTaMIHaesb6axjjryAnqR5fcxgaqqGtuefBJ6U4PewGC2Di+eyvbGXijPNPJITgOCfZeCk5DXJPc4ZYWpudRFa5n7df/xNnjsk4kUz2vQZv/bYjgQAgSU7tJTJp5EyAEROdg33FbjcRuEng+N7O+y4lE3InOwfvs6+Bkac5N0PkngaZo7utyW0sLmbBggXdf97WZnj3TdjzinPb9hsPw2s/ck69jStyE8ZFUPA+Z98MBFVnQJ192TEe9epL7Vny7U+tLc7fKdk/MLXpHrAEYXolKy2Z6QXZTC/IBqA4+SALFpzT6/fbe7SWP751iO9sPsjbh2sQgfmFuVw9YwxXnDWSkb4G54JwQ5Xz2ljtTlc74ynpzmmb9gSQmedcp3HtKS5mYtGCvn7s/uVLgRGTYMQkDu9r48zoA3BEAjlp2PdXeOupzgkEIDndOejnnQlnXuUc/Ee6SSor5O0ByJcCE+Y5w0VfgqY6KF3rJIs9r8Bfvg1r/p9zIJwwHwovcob8mZ2vzTTVQX2Fc9qrrsIdr4D641HTJ14vbKyFV6IO+j2VGoD0EZAxwnlNHwHpI915I2NP+3NOxN7WBo1VbkyVneKetOdNqPvjibg7Pkulsw44pyBTs5zaVuTrSfMCJ5fx58D49/XlrxeTpwlCRBYCPwB8wCOq+o2o5eIuvxKoA25W1TdFZDzwC2AM0AY8rKo/8DJWM7gmjcrktouncNvFU9hZVsNzmw/x3JaDfPnZbfzncuH9p+Xy4Zlj+dC06WRnDNIvvYEWkUCiNba0UvJuBbt2vMOR0r+Tmuzj7OmzmHP2NFJThsjvvtQMp1fj0y52phuqYN/fYLdbw1j1VWd+WrZzLab94NnS0PV7pgUjDtAjnWSYPpLSw8eYOHGSmwClm1diz29tcg7aHQfu487pwfZ50Ym4U0zu7eINlV2WmwRwJOdE7Bm5kDv1RKLxpTqnSZvC0BiGphr3NezcGNEUPrGstfHkDWTmwRf7/yFknn2TRMQHLAU+CBwA1onIclXdHlHsCmCqO8wDHnJfW4B/c5NFANggIn+KWtcMU1PyAtz1wQB3XjaVkkM1PL/lIM9tOciXnt7Cl31vceHU0Xx45lgumxYiq4ctyFWVmsYWjtc2URExtLYpaSlJ+JN9nV7Tkn34U3ykJSc5r+6yFJ8MWCeKbW3K7qNhNpVWsbm0ks0HKik5VE1zq/MreVTWBOqbWqjdVkrQf4jLzgqx8JwxXHj6aPwpQ+i2W382nHGFMwAaLuPIlj9R+/Zq/I1HGTlxOunB0ScO/pGvGbnugTT2j4M9xcVMPNUpsN5qa3Nqqe2//OuPQ5372j6vtflEvNE1joyRFK/dyIKLL+2feFqbndpzRzKp7frmij7y8qfGXGCnqu4GEJEngUVA5EF+EfALdR6G8LqI5IhIvqoeAg4BqGqNiJQA46LWNcOciDBtbJBpY4N88UNnsOVAFc9vOcjzWw6x6u0y0pKTuPiMPK6emc+k3EyO15044B+vbaKironjtc1U1DZRWlZH419f5nhtEy1tfX/2hgidkkkwPZm8gJ/RgTRnyEojL+i8jg6kkRfwE0xPPmVSUVUOVzewubSyIyG89W4V4UbnAJCZ6mN6QTb/8IFCzi3IYeb4HPKz/TS2tPHqjqO8sPUwL5cc4fcb3yUj1cfFZ+TxoXPGcMmZeT1Opv2tsaWVtw5UsX7fcdbvPc6b+49TUZsDfNQpsA/OHZ/DB6eFuGxKiNNDQ6Qn46QkSM9xht6SfkzUvhQ3aXp/p5h49aAaEbkOWKiqt7jTnwbmqertEWWeB76hqq+606uAe1R1fUSZScAa4BxVrY6xnSXAEoBQKDTnySef7FW84XCYrKyhe6eFxXdCmyq7KttYe6iFNw63Ut108ndYgKwUyEoVAqlCelIrOekpBFKFrBQhkIoz7k4nJ0FTKzS3QXOr0tTpFZrbTow3tWlUOahtVqoalaompbJRaYlxpiFZIDtNOg+pQk6acKy2kXfrk9lT1UZlo/N5fALjA0lMzk6iMDuJyTk+8jOFpFMcNFvalLcr2lh/pIU3j7RQ3QTJSXB2ro+ikI9ZeclkpfbswNubv291k7LzeCs7KtvYcbyVvVVttLh/qlCGMCXHx9QRSUwd4aO1TdlU3srGslb2VDk7b3S6cG6eE+/pI5JI7qYn495+/9pUOVavqLs9rxLSUP7/vfjiizeoalGsZV7+pIi1p6P/k7stIyJZwO+AO2MlBwBVfRh4GKCoqEhPeadFF4rjuUtjEFl8nV0CfA5obVPW7a2gqr6ZkZmpjMhIZWRmKtnpKfgiDigDGZ+qUt3QQnlNI+U1jZTVNDjj4caOeeU1jeyvaORYbXtDO2HyaD8XT8thZkE2M8fncFZ+sNeniC5zX1vblA37jvPi1sOs3HaYn22tx5fUzHmTc1l4zhguPztEXuDUt/Ceav+pKrvKa9mwr4L1e4+zYd9xdh+tAyDFJ5wzLpvPTh/BnIkjmTNxBKMDJ9/FdJP7eqS6gVUlZawqOcKanUf5074GAv5kLj4jj8umhbjo9NFkp3c+1XSq+Fpa29hXUcfOsnDHsKOshl1ltdQ3O7dij8hIYdaEEcwan8OsCSOYMT6boL9/rncN9f/frniZIA4A4yOmC4CD8ZYRkRSc5PCEqv7ewzjNe5gvSZg/OXeww+hERMhOTyE7PYUped3/amxubeNYuIkNb7zGVR9c0O+x+JKEuYUjmVs4kvuvPou33q3iha2HeXHrYf7j2a3c/4etFE0cwdljs2lTpaVNaW1VWlVpbXOn29o4fKSBX+5dR0ubOuVa25e30dqm7K+o43idc0ttTkYKcyaM4LqiAoomjmRGQXaPEl0o6OeT8ybwyXkTqGtq4dUdR3m55AirSspYvvkgyUnCvMkjueysEJedFWL8yIyOdRuaW9lztNZNAGF2uYlg79E6mlpPVOvGZvs5LS+LG+bmdvyNNpUeZ+P+Sv78dhngnEacmpfFrPEjmDXBSRpT8rI6/fAY7rxMEOuAqSJSCLwLLAY+GVVmOXC7e31iHlClqofcu5t+BpSo6nc9jNGYQZXiS2JMtp/MFO8POiLCjIIcZhTk8KUPncHfj4R5cethXtx2mN9tOECyT/AlOadyfO7QPt5QrzRWN3RMJycl4UsSUpOT8SUJU0MBiiaOoGjSCCaPyuq3B1tlpCZz+dljuPzsMc5pqNJKXi45wsvbj/DV57bz1ee2c0YoQHpbAw+sW83+ijraLzElCUwYmcGUvCwuOTPElLwspuY5PR7Huh7zyXkTAKiqb2bLgUo27q9k4/7jrNx+mN+sLwWca0Azx+c4CWP8CM6dkMOorJNrQ00tbVTVN1NV30RVfTObylo4tuEAlfXNVNU1Oa/1zVTWNVNZ30xyknBWfoBp+dlMGxt0PlPq4N9g4FmCUNUWEbkdWIlzm+ujqrpNRG51ly8DVuDc4roT5zbXz7qrnw98GnhLRDa58/5dVVd4Fa8xiUREOGNMgDPGBLjjsqmnLO+cIrlgACLrmi9JmDNxBHMmjuCehWey71gtL5eU8fL2I5SWhZlRGOQj547rSASFozJ7dYouOz2FC6aO5oKpowHn9NneY3Vs3O/UMDaWHmfZK7tpdTPRhJEZjM3xU13f4h70m6htitGDwJubAadmEvSnkJORQk56CtkZqTQ0tfKHjQf5v9f3A05yKxyVybSx2UzLd27UOCs/ENfpwP7k6W0N7gF9RdS8ZRHjCtwWY71XiX19whhjAJiYm8k/fqCQf/xAoZvA5niyHRGhcFQmhaMyuXa205tZfVMrb71bxcb9zt1Yx8JNjM3xc1Z+MOLA75xmzMlIZVfJFi79wHxy0lMJ+JNj1rBUlQPH69l2sJqSQ9VsP1TNm/uO89zmE2fmR2WlOXf2uUljWn6AwlHenfYaIi1qjDHmvSM91ddxbSceetDHxNzuez0WEcaPzGD8yAwWnjOmY35VXTPb3YRRcqia7Qer+dmu3R3tYPwpSUwfl81T/3Rev9+FZQnCGGOGsOyMFM47LZfzTjtxM0ZTSxs7y8IdSaO2scWTW3QtQRhjzHtManJSRyNSL9kDi40xxsRkCcIYY0xMliCMMcbEZAnCGGNMTJYgjDHGxGQJwhhjTEyWIIwxxsRkCcIYY0xMnj0waDCISDmwr5erjwKO9mM4/c3i6xuLr28svr4ZyvFNVNXRsRYMqwTRFyKyvqunKg0FFl/fWHx9Y/H1zVCPryt2iskYY0xMliCMMcbEZAnihIcHO4BTsPj6xuLrG4uvb4Z6fDHZNQhjjDExWQ3CGGNMTJYgjDHGxJRQCUJEForIOyKyU0TujbFcRORBd/kWEZk9wPGNF5HVIlIiIttE5I4YZRaISJWIbHKH/xzgGPeKyFvuttfHWD5o+1BEzojYL5tEpFpE7owqM6D7T0QeFZEyEdkaMW+kiPxJRHa4ryO6WLfb76uH8X1LRN52/37PiEhOF+t2+13wML4HROTdiL/hlV2sO1j77zcRse0VkU1drOv5/uszVU2IAfABu4DJQCqwGZgWVeZK4AVAgPnA2gGOMR+Y7Y4HgL/HiHEB8Pwg7se9wKhulg/qPoz6ex/GaQQ0aPsPuBCYDWyNmPf/gHvd8XuBb3YRf7ffVw/juxxIdse/GSu+eL4LHsb3AHB3HH//Qdl/Ucu/A/znYO2/vg6JVIOYC+xU1d2q2gQ8CSyKKrMI+IU6XgdyRCR/oAJU1UOq+qY7XgOUAOMGavv9ZFD3YYRLgV2q2tuW9f1CVdcAFVGzFwE/d8d/DlwTY9V4vq+exKeqL6lqizv5OlDQ39uNVxf7Lx6Dtv/aifOQ6OuBX/f3dgdKIiWIcUBpxPQBTj74xlNmQIjIJGAWsDbG4vNEZLOIvCAiZw9sZCjwkohsEJElMZYPlX24mK7/MQdz/wGEVPUQOD8KgLwYZYbKfvwHnBphLKf6LnjpdvcU2KNdnKIbCvvvAuCIqu7oYvlg7r+4JFKCkBjzou/xjaeM50QkC/gdcKeqVkctfhPntMlM4IfAswMc3vmqOhu4ArhNRC6MWj7o+1BEUoGPAL+NsXiw91+8hsJ+/DLQAjzRRZFTfRe88hBwGnAucAjnNE60Qd9/wA10X3sYrP0Xt0RKEAeA8RHTBcDBXpTxlIik4CSHJ1T199HLVbVaVcPu+AogRURGDVR8qnrQfS0DnsGpykca9H2I8w/3pqoeiV4w2PvPdaT9tJv7WhajzKDuRxH5DHA18Cl1T5hHi+O74AlVPaKqraraBvy0i+0O9v5LBq4FftNVmcHafz2RSAliHTBVRArdX5iLgeVRZZYDN7l34swHqtpPBQwE95zlz4ASVf1uF2XGuOUQkbk4f8NjAxRfpogE2sdxLmZujSo2qPvQ1eUvt8HcfxGWA59xxz8D/CFGmXi+r54QkYXAPcBHVLWuizLxfBe8ii/ymtZHu9juoO0/12XA26p6INbCwdx/PTLYV8kHcsC5w+bvOHc3fNmddytwqzsuwFJ3+VtA0QDH9wGcavAWYJM7XBkV4+3ANpy7Ml4H3j+A8U12t7vZjWEo7sMMnAN+dsS8Qdt/OInqENCM86v2H4FcYBWww30d6ZYdC6zo7vs6QPHtxDl/3/4dXBYdX1ffhQGK75fud2sLzkE/fyjtP3f+4+3fuYiyA77/+jpYVxvGGGNiSqRTTMYYY3rAEoQxxpiYLEEYY4yJyRKEMcaYmCxBGGOMickShDE9ICKt0rnH2H7rJVREJkX2CmrMYEse7ACMeY+pV9VzBzsIYwaC1SCM6Qdu3/7fFJE33GGKO3+iiKxyO5ZbJSIT3Pkh91kLm93h/e5b+UTkp+I8D+QlEUkftA9lEp4lCGN6Jj3qFNMnIpZVq+pc4EfA9915P8Lp/nwGTqd3D7rzHwReUafTwNk4rWkBpgJLVfVsoBL4mKefxphuWEtqY3pARMKqmhVj/l7gElXd7Xa4eFhVc0XkKE5XEM3u/EOqOkpEyoECVW2MeI9JwJ9Udao7fQ+QoqpfH4CPZsxJrAZhTP/RLsa7KhNLY8R4K3ad0AwiSxDG9J9PRLy+5o7/DacnUYBPAa+646uAfwYQEZ+IBAcqSGPiZb9OjOmZ9KiH0L+oqu23uqaJyFqcH143uPO+ADwqIl8EyoHPuvPvAB4WkX/EqSn8M06voMYMGXYNwph+4F6DKFLVo4MdizH9xU4xGWOMiclqEMYYY2KyGoQxxpiYLEEYY4yJyRKEMcaYmCxBGGOMickShDHGmJj+P0MdESeYv2cUAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAuN0lEQVR4nO3deZxcZZ3v8c9Tay9V6e70lpCEbARCNgMdWWSZxBkxMEp0jJcwDuPljsMgZOTOHe+AL+flcieOehmVQVCueoER0aCMCFdRWUwIyprEkBVICCF7utNJJ1291vLcP87p7upOVe/VVfT5vl+v86qzPKfOr05Xn189Z3keY61FRES8y5fvAEREJL+UCEREPE6JQETE45QIREQ8TolARMTjAvkOYKiqqqrsjBkzhrVuS0sLpaWloxvQKCr0+KDwY1R8I6P4RqaQ49u0adNxa211xoXW2nfVUFdXZ4dr3bp1w153LBR6fNYWfoyKb2QU38gUcnzARpvluKpTQyIiHqdEICLicUoEIiIe9667WCwi40M8HufgwYO0t7cPep2ysjJ27dqVw6hGphDiKyoqYurUqQSDwUGvo0QgInlx8OBBotEoM2bMwBgzqHWam5uJRqM5jmz48h2ftZbGxkYOHjzIzJkzB72eTg2JSF60t7dTWVk56CQgAzPGUFlZOaRaFigRiEgeKQmMvuHsU88kgjeONvPom500tXbmOxQRkYLimUSwr7GFX+6Nc/BkW75DEREpKJ5JBNXRMAANzR15jkRECkFTUxPf+c53hrzeNddcQ1NT0+gHlEeeSQQ1biKobx7aRRQRGZ+yJYJkMtnvek8++STl5eU5iio/PHP7aFVENQKRQvXl/7eDnYdPD1gumUzi9/sH9Z7zzprAFz88P+vyO+64g7feeovFixcTDAaJRCJMnjyZLVu2sHPnTj7ykY9w4MAB2tvbue2227jpppsAmDFjBhs3biQWi3H11Vdz+eWX88ILLzBlyhR+9KMfZb19dOnSpVxwwQVs2rSJhoYGfvjDH/LVr36Vbdu2cd1117FmzRqArNt96qmn+OIXv0hHRwezZ8/mgQceIBKJDGpfDMQzNYKioJ/SINQrEYgI8LWvfY3Zs2ezZcsW7rzzTl555RW+8pWvsHPnTgDuv/9+Nm3axMaNG7n77rtpbGw84z12797Nrbfeyo4dOygvL+fxxx/vd5uhUIgNGzZw8803s2LFCu699162b9/Ogw8+2P3+mbZ7/Phx1qxZwzPPPMPmzZtZsmQJ3/zmN0dtX3imRgBQFjKqEYgUoP5+uafL5QNbF110Ua+HsO6++24ee+wxAA4cOMDu3buprKzstc7MmTNZvHgxAHV1dezfv7/fbVx77bUALFy4kPnz5zN58mQAZs2axYEDB6isrMy43ePHj7Nz504uu+wyADo7O7n00ktH/qFd3koEYaMagYhklN6PwPr163nmmWd48cUXKSkpYenSpRkf0gqHw93jfr+fRCLR7za6yvt8vl7r+nw+EolE1u1aa/nABz7AT37yk5F+zIw8c2oInESgGoGIAESjUZqbmzMuO3XqFBUVFZSUlPD666/z0ksvjUlM2bZ7ySWX8Ic//IE9e/YA0Nrayptvvjlq2/VUjaA8bNhy3MmueqJRxNsqKyu57LLLWLBgAcXFxdTW1nYvW758Offddx+LFi3ivPPO45JLLhmTmLJtt7q6mgcffJDrr7+ejg7nx+yaNWs499xzR2W7nkoEZWEf7fEEsY4E0aLBt8wnIuPTj3/844zzw+Ewv/71rzMu27dvHwBVVVVs3769e/5nP/vZrDUMcE43dVm6dClLly7NuCzbdt///vfz6quvZn3/kfDcqSHQnUMiIulylgiMMfcbY+qNMduzLDfGmLuNMXuMMVuNMRfmKpYu5W4i0HUCEcmVW2+9lcWLF/caHnjggXyH1a9cnhp6ELgH+GGW5VcDc9zhYuC77mvOlCkRiEiO3XvvvfkOYchyViOw1m4ATvRTZAXwQ+t4CSg3xkzOVTzgPEcAOjUkIpIunxeLpwAH0qYPuvOO9C1ojLkJuAmgtra214WVobAdLQSMYdOO3cxOvDOs98ilWCw27M82Vgo9RsU3MmMZX1lZWb8XVzNJJpNDXmcsFUp87e3tQ/o75jMRZLp/02YqaK39HvA9gCVLltj0q+1DsX79emrLUhRVTGTp0sXDeo9cWr9+PcP9bGOl0GNUfCMzlvHt2rVryE8J57sryIEUSnxFRUVccMEFgy6fz7uGDgLT0qanAodzvdGqaFjXCERE0uQzETwB/LV799AlwClr7RmnhUZbjRKBiAxDV0ufhw8fZuXKlRnLXHPNNWzcuHEswxoVOTs1ZIz5CbAUqDLGHAS+CAQBrLX3AU8C1wB7gFbgxlzFkq46GmbzOyfHYlMiMg6dddZZPProo/kOY1TlLBFYa68fYLkFbs3V9rOpiYZpbOkknkwR9HvqeTqRwvXrO+DotgGLFScT4B/kYWvSQrj6a1kX33777UyfPp1bbrkFgC996UsYY9iwYQMnT54kHo+zZs0aVqxY0Wu9ffv28aEPfYjt27fT1tbGjTfeyM6dOzn//PNpa+u/K9xIJMKtt97KM888Q0VFBf/6r//KP/3TP7F//37uuusurr32Wvbt28cNN9xAS0sLAPfccw/ve9/7ALjzzjv56U9/SkdHBx/96Ef58pe/PLh9MQDPHQm7uqxsjKkTexEvW7VqFY888kj39E9/+lNuvPFGHnvsMTZv3sy6dev4x3/8R5zfrJl997vfpaSkhK1bt/L5z3+eLVu29LvNlpYWli5dyqZNm4hGo/zzP/8zTz/9NI899hhf+MIXAKipqeHpp59m8+bNPPLII3zmM58BnI5pdu/ezSuvvMKWLVvYtGkTGzZsGPmOwGNtDQHURIsAp8vKSWVFeY5GRIB+f7mnaxvFu3IuuOAC6uvrOXz4MA0NDVRUVDB58mT+4R/+gQ0bNuDz+Th06BDHjh1j0qRJGd9jw4YN3QfqRYsWsWDBgn63GQqFWL58OeD0SRAOhwkGgyxcuLC7DaN4PM7q1avZsmULfr+/u5XRp556iqeeeqr7bqBYLMbu3bu58sorR7wvPJcI1Im9iHRZuXIljz76KEePHmXVqlU8/PDDNDQ0sGnTJoLBIDNmzMjYD0G6obRkHAwGu8un90nQ1R8BwLe+9S1qa2t57bXXSKVSFBU5P1ittXzuc5/j7/7u74bzUfvluVNDPZ3YKxGIeN2qVatYu3Ytjz76KCtXruTUqVPU1NQQDAZZt24d77zT/4OnV155JQ8//DAA27dv79Ua6XCdOnWKyZMn4/P5eOihh0gmkwB88IMf5P777ycWiwFw6NAh6uvrR7w98GCNQJ3Yi0iX+fPn09zczJQpU5g8eTKf+MQn+PCHP8ySJUtYvHgxc+fO7Xf9T3/609x4440sWrSIxYsXU1dXN+KYbrnlFj72sY/xs5/9jGXLlnX3nHbVVVexa9eu7i4qI5EIP/rRj6ipqRnxNj2XCEIBHxUlQSUCEQFg27aeu5Wqqqp48cUXM5br+iU+Y8aM7l/+xcXFrF27trvMQE8Wd70HOHcpZVo2Z84ctm7d2j3/q1/9avf4bbfdxm233TbQRxoyz50aAuc6QX1z/+f9RES8wnM1AnASgWoEIpIrF198cXeXkl0eeughFi5cmKeI+ufJRFATLeLVff21kC0iY2G89h/+8ssv523b/T33kI1nTw01NHcMa4eJyOgoKiqisbFR/4ejyFpLY2Nj9y2ng+XRGkGYjkSK0+0JyorVib1IPkydOpWDBw/S0NAw6HXa29uHfJAbS4UQX1FREVOnTh3SOp5MBOkPlSkRiORHMBhk5syZQ1pn/fr1Q2pnf6wVenzZePbUEKA7h0RE8GgiqFEzEyIi3TyZCKrdhueUCEREPJoIJhQFCAV8SgQiIng0ERhj1GWliIjLk4kAupqZUCIQEfFsIlCNQETE4dlEoIbnREQc3k0EkSJOtsbpTKTyHYqISF55NhHUTHCeJTge0+khEfE2zyaCavVUJiICeDgRdNUIdOeQiHidZxNBtZqZEBEBPJwIujqx151DIuJ1nk0EQb+PiaUh1QhExPM8mwjAeahM1whExOs8nQjUib2IiBKBEoGIeJ4SgTqxFxGP83YiiITpTKY43ZbIdygiInmT00RgjFlujHnDGLPHGHNHhuVlxpj/Z4x5zRizwxhzYy7j6atmgtNTmW4hFREvy1kiMMb4gXuBq4F5wPXGmHl9it0K7LTWvgdYCnzDGBPKVUx9qZkJEZHc1gguAvZYa/daazuBtcCKPmUsEDXGGCACnADG7DyNmpkQEQGTqwulxpiVwHJr7afc6RuAi621q9PKRIEngLlAFLjOWvurDO91E3ATQG1tbd3atWuHFVMsFiMSiXRPt8YttzzbynXnhbh6ZnBY7zma+sZXiAo9RsU3MopvZAo5vmXLlm2y1i7JuNBam5MB+Djwg7TpG4Bv9ymzEvgWYIBzgLeBCf29b11dnR2udevW9ZpOpVL2vH9+0q755Y5hv+do6htfISr0GBXfyCi+kSnk+ICNNstxNZenhg4C09KmpwKH+5S5Efi5G+ceNxHMzWFMvRhj9CyBiHheLhPBq8AcY8xM9wLwKpzTQOn2A38KYIypBc4D9uYwpjPURIt0jUBEPC2Qqze21iaMMauB3wJ+4H5r7Q5jzM3u8vuAfwEeNMZswzk9dLu19niuYsqkOhLmrYbYWG5SRKSg5CwRAFhrnwSe7DPvvrTxw8BVuYxhIDUTwrz0dmM+QxARyStPP1kMTo2gqTVORyKZ71BERPLC84mgpxP7zjxHIiKSH55PBF1dVtafVjMTIuJNSgQRp70h3UIqIl7l+USgZiZExOs8nwgqS0MYoxqBiHiX5xNBwO+jsjSkGoGIeJbnEwFAVUTNTIiIdykR4HRQ06DOaUTEo5QIcB4qU41ARLxKiQDnzqGGmDqxFxFvUiLAqRHEk5am1ni+QxERGXNKBPQ8S9AQ0+khEfEeJQJ6OrGvP61EICLeo0SAc9cQQENMdw6JiPcoEZDe8JxqBCLiPUoEQGnIT3HQr1tIRcSTlAhwOrGvmRBWMxMi4klKBC49VCYiXqVE4HJqBLpYLCLeo0TgUo1ARLxKicBVM6GI0+0J2uPqxF5EvEWJwNX1UJlqBSLiNUoErmo1MyEiHqVE4FIzEyLiVUoELjU8JyJepUTgqiwN4zPQcFq3kIqItygRuPw+Q2UkrBqBiHiOEkGa6khY1whExHOUCNJUR1UjEBHv6TcRGGMm9LPs7NEPJ79qoqoRiIj3DFQjWN81Yox5ts+yX4x2MPlWHQ1zPNZBKqVO7EXEOwZKBCZtfGI/yzKvbMxyY8wbxpg9xpg7spRZaozZYozZYYx5bqD3zKWaaJhEynKytTOfYYiIjKnAAMttlvFM070YY/zAvcAHgIPAq8aYJ6y1O9PKlAPfAZZba/cbY2oGG3guVEe7uqzsoNJ9wExEZLwbKBHUGGP+B86v/65x3OnqAda9CNhjrd0LYIxZC6wAdqaV+Uvg59ba/QDW2vohxj+quh8qa+5g7qR8RiIiMnaMtdl/2BtjvtjfytbaL/ez7kqcX/qfcqdvAC621q5OK3MXEATmA1Hg3621P8zwXjcBNwHU1tbWrV27tr+wsorFYkQikazLj7WkuP35Nv52YYjLpgSHtY2RGCi+QlDoMSq+kVF8I1PI8S1btmyTtXZJxoXW2mENwHsHWP5x4Adp0zcA3+5T5h7gJaAUqAJ2A+f29751dXV2uNatW9fv8lh73E6//Zf2u+v3DHsbIzFQfIWg0GNUfCOj+EamkOMDNtosx9WBTg31YoyZB6wCrgdOAZmzi+MgMC1teipwOEOZ49baFqDFGLMBeA/w5lDiGi2l4QClIb9uIRURTxkwERhjpuMc+K8HEsB0YIm1dt8Aq74KzDHGzAQO4SSQv+xT5nHgHmNMAAgBFwPfGsoHGG01E4r0UJmIeEq/icAY8wJQBqwFVlprdxtj3h5EEsBamzDGrAZ+C/iB+621O4wxN7vL77PW7jLG/AbYCqRwTiVtH9lHGhmnmQk1PCci3jFQjaAB55ROLc5dQrsZ4LbRdNbaJ4En+8y7r8/0ncCdg33PXKueEGbXkdP5DkNEZMz0+0CZtXYFsBDYDHzZGPM2UGGMuWgsgsuH6kiYBl0jEBEPGbDROWvtKWvt/dbaDwCXAF8E7jLGHMh5dHlQHQ3T3JGgrVOd2IuINwyp9VFr7TFr7d3W2vcBl+copryqiaoTexHxloEuFj8xwPrXjmIsBaG6KxHE2jm7siTP0YiI5N5AF4svBQ4APwFeZhANzb3b1bjtDelZAhHxioESwSScRuOux3kG4FfAT6y1O3IdWL701AiUCETEGwa6ayhprf2NtfaTOBeK9wDrjTF/PybR5cHE0hB+n9E1AhHxjME8WRwG/hynVjADuBv4eW7Dyh+/z1BZGtKpIRHxjIEuFv8HsAD4NfDlfD/1O1ZqJqjvYhHxjoFqBDcALcC5wGeM6b5WbABrrc3ap/G7WXUkTH2zmpkQEW/oNxFYa4f0nMF4URMtYqeamRARj/DkgX4gTif2nSTVib2IeIASQQbV0TBJdWIvIh6hRJBBVzMTunNIRLxAiSADPVQmIl6iRJBBTzMTunNIRMY/JYIMVCMQES9RIsigOOQnGg6omQkR8QQlgiyqo2HqlQhExAOUCLKojoZVIxART1AiyEKJQES8Qokgi5pokRKBiHiCEkEW1dEwsY4ErZ2JfIciIpJTSgRZqBN7EfEKJYIsup4l0J1DIjLeKRFkUa0agYh4hBJBFj0Nz6mZCREZ35QIsqgoCRHwGTUzISLjnhJBFj6foSoSVlPUIjLuKRH0ozqqTuxFZPxTIuhHjZ4uFhEPUCLohxqeExEvyGkiMMYsN8a8YYzZY4y5o59y7zXGJI0xK3MZz1DVRMM0xjrUib2IjGs5SwTGGD9wL3A1MA+43hgzL0u5rwO/zVUsw1UdDZOy0NiiWoGIjF+5rBFcBOyx1u611nYCa4EVGcr9PfCfQH0OYxmWarfLSl0nEJHxzFibm9Me7mme5dbaT7nTNwAXW2tXp5WZAvwYeD/wf4FfWmsfzfBeNwE3AdTW1tatXbt2WDHFYjEikcigy+85mWTNy+38j7owi6oDw9rmUAw1vnwo9BgV38govpEp5PiWLVu2yVq7JONCa21OBuDjwA/Spm8Avt2nzM+AS9zxB4GVA71vXV2dHa5169YNqfz+xhY7/fZf2kde3T/sbQ7FUOPLh0KPUfGNjOIbmUKOD9hosxxXc/kz9yAwLW16KnC4T5klwFpjDEAVcI0xJmGt/UUO4xo0tTckIl6Qy0TwKjDHGDMTOASsAv4yvYC1dmbXuDHmQZxTQ7/IYUxDUhT0Ey1SJ/YiMr7lLBFYaxPGmNU4dwP5gfuttTuMMTe7y+/L1bZHU000TH2zGp4TkfErp1dArbVPAk/2mZcxAVhr/2suYxku9V0sIuOdniwegPouFpHxTolgAGpmQkTGOyWCAdREw7R2JmnpUCf2IjI+KREMQH0Xi8h4p0QwgBo1MyEi45wSwQB6agS6hVRExiclggHU6OliERnnlAgGUF4SJOg3ukYgIuOWdxLBsR3M2/F16Gwd0mrGGKojeqhMRMYv7ySC1hPUNLwAv1sz5FX1LIGIjGfeSQQzr+DQWVfDS9+B/S8PaVU1MyEi45l3EgGwd9YnoWwaPH4LxNsGvV61mpkQkXHMU4kgGSiGa++Gxj2w7l8HvV51NExjSweJZCqH0YmI5IenEgEAs5fBhZ+EF++BgxsHtUpNNIy1cKKlM8fBiYiMPe8lAoCr/gWik+HxWyE+8INiamZCRMYzbyaCojL48N3Q8Do89/UBi+uhMhEZz7yZCADm/Bks/iv4w7/D4T/2W1TNTIjIeObdRADwwa9ApAZ+cSsksp//Vyf2IjKeeTsRFJfDh+6C+h3w/L9lLRYO+CkrDuoagYiMS95OBADnLYdFq+D5b8CRrVmL1eihMhEZp5QIAJZ/FUoqnQfNkvGMRdTMhIiMV0oEACUT4c+/CUe3we+/lbGIagQiMl4pEXQ5/0Ow4GPw3P+GYzvOWOzUCNqx1uYhOBGR3FEiSHf1nc4zBr+4BZK9O6uvjoZpj6eIqRN7ERlnlAjSlVbCn38DjmyBF/6916Kuvos37juZh8BERHJHiaCv+R+BeStg/deg/vXu2VfMqWJmVSk3PbSRn7yyP3/xiYiMMiWCTK75BoQiTltEqSQAlZEwv7jlMi6dXcXnfr6NLzy+nbhaIxWRcUCJIJNINVxzJxzaCC/e2z27rCTIA//1vfzdlbP44Yvv8Fc/eJnGmO4kEpF3NyWCbBZ8DOZ+yOna8vju7tl+n+Fz15zPXdctZsuBJq695w/sOHwqj4GKiIyMEkE2xjjPFgSLe50i6vKRC6bws5svJWUtK7/7Ir/aeiRPgYqIjIwSQX+itXD11+HAy/Dy/zlj8aKp5Ty++jLmnTWBW3+8mX/77RukUnrOQETeXZQIBrLoOpjzQXj2f0HjW2csrokW8eO/vZhV753GPev2cNNDG2luz9xMhYhIIcppIjDGLDfGvGGM2WOMuSPD8k8YY7a6wwvGmPfkMp5hMQY+fBf4Q/D4amg/83pAOODnq3+xkH9ZMZ/1bzTw0e+8wNvHW8Y+VhGRYchZIjDG+IF7gauBecD1xph5fYq9DfyJtXYR8C/A93IVz4hMOMs5RbT/BfjGXOeawcGNkNbchDGGGy6dwUN/czGNsQ5W3PN7nnuzIY9Bi4gMTi5rBBcBe6y1e621ncBaYEV6AWvtC9barkd1XwKm5jCekVl8PfztOlj4cdj+GPzgT+G+K+CV7/eqJVw6u5InVl/OWeXF3PjAK3xvw1tqn0hECprJ1UHKGLMSWG6t/ZQ7fQNwsbV2dZbynwXmdpXvs+wm4CaA2traurVr1w4rplgsRiQSGda66fyJVmrqN3DW4d8Sje0l6QtTX3MFh8+6iubouWAMHQnL97d1sPFYkksn+7lxQZiQ34xJfLlU6DEqvpFRfCNTyPEtW7Zsk7V2ScaF1tqcDMDHgR+kTd8AfDtL2WXALqByoPetq6uzw7Vu3bphr5vVwU3WPv731q6ZbO0XJ1j7ncusffl71rY12VQqZb/97Jt2+u2/tB/+9vP2cFPr2Mc3ygo9RsU3MopvZAo5PmCjzXJczeWpoYPAtLTpqcDhvoWMMYuAHwArrLWNOYwnN6ZcCNfeDZ99Az70Lefi8pOfhW/MxTyxmtXnnuL7N9Sxt6GFP7/793zpiR2se6Oets7kwO8tIjIGAjl871eBOcaYmcAhYBXwl+kFjDFnAz8HbrDWvpnDWHIvHIUl/80ZDv8RNj4A2x6FP/6ID9Qu5HdXruLL78xn7av7efCFfYQCPi6eOZE/ObeapefVMLu6NN+fQEQ8KmeJwFqbMMasBn4L+IH7rbU7jDE3u8vvA74AVALfMcYAJGy2c1jvJmddANdeAB/8Cmz7GWx8gJrnP8+9/jCpqfM4VnIOr3VO47eNVfz7r2pY86sSppQXMycap6P6KO+bXUm0KJjvTyEiHpHLGgHW2ieBJ/vMuy9t/FPAGReHx42+tYTt/4nvyFYmH/kdk1sbWQ5QBLHis9hjZvCHo5P4+cMv8jWmM2n6uVx53iT+5Nxqzp8cxU2UIiKjLqeJQNKcdYEzgPP8QfNRp4/kY9uIHN3O4mPbeU/rSxi/07R165FidhyaxqvPTOex8GwiZy9m9vw65k6uYFpFiLCxTvtHqUSfoe+8tGljoHKO03SGiIhLiSAfjIEJk53h3Ku6Zz//7G+58rwqOLaNkqPbec+h11hc/wLBxNPOo3dvj9L2S2tg0gKoXQCTFjpD5Rzw6+sg4kX6zy8gKX8YptY5AxACSKWg6R2SR7bRsG87jbFOjrcmqW9JUN+S5FgsQUfKRxIfCeunKByiakIJ1WWl1JSVMqk8wqSJEWomlOInAQ1vODWRo9vg5fsg2els3B+GmrlOUqhd2JMoisvztTsKT6LDaWrk3XCarrMVGnbB0e1wbAcc2+40p14+DaYsgSl1MHUJTJz17vg8klNKBIXO54OJM/FPnMmk+dcyqc/iZMpy6GQbbzXE3KGFVxpi7D0c4/ib7kGeFEF/jOmVpcyqei+zqpcxa0kpsytDnGOOUHbqDTi2zTlovPEb+OOPejZQdnZPUqiZS1XDbth5GmwqbbB9pjMN1jngRGqhbCqUTYPSqvwehOLt0HocWtyh9Ti0NLhDo/PaPe84xFvBF4SSSmcodV9LqjLM65o/EQLh3H0Ga+HUQfdgv815PbodTrzl7HdweturmQdzroKT+5y/7ytua7pF5U5ScIdgpzpa8iIlgnc5v89wdmUJZ1eWsGxuTa9lp1rjvHU8xlv1ToLoShbr3qgnnux5oryipIJZ1dcwq+rjzHpvKXOjrZzHPmpadhNocA8sb/4GbIoFADtGKfhAkZsUpvYkh/TxCVMgWDTw+1jrHKTbThJp3gt7DbSdgLaTztB6AtqanHkt7oG9tRE6Tmd+P3/IOYiXukPlOVBa7Rw0O2POul3D0W3Oa9vJzO8FEIp2J4j3tMThyDQIlWYZIs5rsKRnvGu+P+gc4LsO9sfcIb0hxIoZTtJe8DGone8k8fIZzg+KLqkkNLwOhzY5bWYd2gzP/xvYFJcB7Dy7d61h0iIIlQz8d5B3LSWCcaysJMiFZ1dw4dkVveYnkikOnGxjb0OMvQ0t7D3uJIp1bzTws00H3VJ+/L7zmVZxIbOqI5x7tp/5xY0cO7CPuefOIRQMEAoECAX9hINBggE/4VCQcDDQPfh8fjC+nsEmofmI8wv21EFo2t8zvvsZiB0980OU1vQkh+hkiLc4B/XWtAN924nuU1xLADb1eY9AsfPLvLjC+aU+pa7nIF9a7R70q3vmhScMvaaSTLhJp9GpRXQlipbGXonD13wImt6BzpaeIT6MlmqDpc6Bfv5fpNXY5kHRhIHX9fmddWvnw4V/7czrbIEjr/HWc48wu6gJDr4KO37uLDNu+SkXOokmUguRGohMcsZLKnsnmtFmrRNfexPBziZnPFhS+Ke0Uimnlvb287Dvedj/IiQ6nc6uQqXOa7DETfol7nRpP8tLoOZ8528xypQIPCjg9zGzqpSZVaX86fm9l51ujzvJIS1J7G1o4Q97WuhIpICz4M3BHbhCfh/hoI+ioJ+ioI+igJ+y4iDlJbVUlEylvOQKyqtDlE8PUlESoiKcoip1gomJo0TbjxFqOQynDjiJouF12Lve+QcproDiiVA52z24T+yet33vYRYsuTxtXoXzD5Vr/oDT13Wkut9if1y/nqVLl/aemUo5NZrOFqfGkT7eK2G0ub/450PFzNE9+IZKYfr7OHB2J7O74ms+5tQaDm1y+u/e8QtobzpzXeN3E0ONmyTShmhtT+IorXGus7SddN6nO5GfdGtsadO9ljdByunj4zKAFwBMT40p3FVzivaZjvQpE3V+UFTNcRL/aCeSVIrS2D546T7nwL/v9z37a+IsmLcCisqcv2Nnq/N3jrc6060nnNd4S+/l9GkL7vJ/UCKQ3JtQFGTxtHIWTyvvNT+VstQ3d7Du+RdYdGEd7fEUHfEkHYkU7fEk7Ykk7XF3PN4zryOeosNd1tqZ4HRbgoMnW9l+KE5TWyft8VSWSCoIByqpKKmjvCToDFNDlIYDRMJ+SsIBSkN+SkIBSsM9r2/6t0NoESU+P6UEKEn6KfFb/L4C/vXo8zkHqnAEKKBbe6O1MPcaZ+jS2QKxYxCrd16bj7nTacPRbc5yO8RmVMITnJsTisqdBF4zz5nuSuhFZbz5+i7OnT45LVHGoCMtYcaOwomWtHkxzjiYAoTLnB8SVXOcO+a6xifOHvxpMGvh+Jvw9obuA/97W91WcsrPdvo8n3kFzLgCyqYMbV90vX+i3U0MboIYTI1vGJQIZFB8PsOksiImR3zMP6ts1N63PZ6kqTXOydZOmlrjNLV20tTmTJ9y559sjXOqNc7e4zFaOpK0dCZo7UjSmcySRF75/RmzioI+SkIBgn5DwOfD7zMEfAa/OwT8Br/P1z2v96szP+A3RIsCTCgOUlYcZEKR89p3mFAcLOzEMxKhUufX7cRZ/ZdLpZzTYb2SRL1TOyuu6DnYF1e4B/8y5xrIAA43r+fcy5cOPt5UChLugbT9NDTtg+N7oHEPNO6GfX+ArY/0XqdsmpMYKue4ieIcZyibBiff7nXgJ3bMWWfCFDjnA7zeUc3c5X8LFdMHH2M2xrinh9xTmzmkRCB5VRT0M6nMz6SyQVwU7qMzkaKt00kMLR0JWjqTvPDKJuacv4DWzoSTNDoSTuLodMaTKUsiZdNeUySSts/8FPFkira4JZU2vzOZork9wam2OJ2JbDUZRyQc6E4KZcWB7sRxvL6D353a3p1kfGnJxm+6EpLBb3oSlM/0JKZQwEfI73Ne08aDfh/hwJnzu8bH/Ml0ny/tVNmCsd123zi6LrhHaqDqHDjnz3qX6Wx1LsIf3+0miD3O+NZHet9Q4As4D2aCc7prxhU9v/jd23CPrl/P3NFIAmNMiUDetboOdGUlPb8km97ys3Re7k+vtMeTnGqLc6otzmn3NdPQtezt4y2cbkvQ2p7gj42HSaYlo65kk0shN1GUlQSpLA1RGQlTWRpiYiREVWmYykiIiaUh9p1KcripjYmlIYqC/pzGVDBCJT0PVqaz1qnFdNUeTux1TvnMuNKpKRT6xeohUCIQGQbnArif2glDq8msz3SxGKdfkJQlLUGkuseTfWolnQl3SBvv6DPdmUgSTzrlO9x5XcnreKyDY6fb2Xn4NI0tHb1uJQb40ou/AyAaDjAxEupOHBNLQgDEk8624skU8aR1phOpnvkJe2YZN77ScIDyEufmgPLiIOUlISq6rgGVhJz57nTXeHHQn7VGk0xZWjsTtMWTtHUmaXUHZ7z3/PZEkkg4QKWb+KoiISpLw5QVB/FlOpVnjHOdJFoLMy4b0t/53UaJQKQAGGPwG9KuLYzNr3FrLc0dCRpjnZxo6WD9S5uZPONcTrR0cDzWSWOLM//AiVZeO9CEMXSfigr5ndeg3xD0+5yDfNf8gDO/p4yPgN/Q0pHoviZ05FQ7u46c5mRrnLZ49gvLoYCPihLn+ktzrBXz4rO0xp2D+0Cn6AbD7zNMLO1KeKG0RBHuqT25CTGRssTanVORzR0JZ7wzQXN7glhHgjf2dvDEsS29lsXanbItHQmKgn7Ki4OUlQS7E2FZcbBXAixz5zvLnVOKGRPVKFIiEPEwYwwTipyDzcyqUprfDrD04rPHPI6u2kqvmwZa45xMG29q6+SEaWP6lCpKQn6KQ36Kg353PEBJ0J0X8lMSdO4kS58uCvqJdSRobOlwmmqJOa+NLR2caOl0El+sg9dONtEY6yTWkRjSZwj4DEV+S0XsBKWhANGiAJWlIc6eWEK0KEBJKEBHwrk5wqmZdbK7Psap1jjN/WzLGJzkUBzkry6ZzqeuGOBC/TAoEYhI3g32VJtzau09w95OcchPdXRwTX60x5M0tjjJodGtHQX9hkg4QCQcoDTsHOy7xsMBH88991zGU38DiSdTnG6L09QWdxNFTyI85d5J19QaH3TsQ6VEICKSQVHQz5TyYqaU5/6BxKDf556CymG7VP3IZZ/FIiLyLqBEICLicUoEIiIep0QgIuJxSgQiIh6nRCAi4nFKBCIiHqdEICLiccba3LZ6ONqMMQ3AO8NcvQo4PorhjLZCjw8KP0bFNzKKb2QKOb7p1tqMXei96xLBSBhjNlprl+Q7jmwKPT4o/BgV38govpEp9Piy0akhERGPUyIQEfE4ryWC7+U7gAEUenxQ+DEqvpFRfCNT6PFl5KlrBCIiciav1QhERKQPJQIREY8bl4nAGLPcGPOGMWaPMeaODMuNMeZud/lWY8yFYxjbNGPMOmPMLmPMDmPMbRnKLDXGnDLGbHGHL4xVfO729xljtrnb3phheT7333lp+2WLMea0Mea/9ykz5vvPGHO/MabeGLM9bd5EY8zTxpjd7mtFlnX7/b7mML47jTGvu3/Dx4wx5VnW7ff7kMP4vmSMOZT2d7wmy7r52n+PpMW2zxizJcu6Od9/I2atHVcDTq/fbwGzgBDwGjCvT5lrgF8DBrgEeHkM45sMXOiOR4E3M8S3FPhlHvfhPqCqn+V5238Z/tZHcR6Uyev+A64ELgS2p83738Ad7vgdwNezfIZ+v685jO8qIOCOfz1TfIP5PuQwvi8Bnx3EdyAv+6/P8m8AX8jX/hvpMB5rBBcBe6y1e621ncBaYEWfMiuAH1rHS0C5MWbyWARnrT1ird3sjjcDu4ApY7HtUZS3/dfHnwJvWWuH+6T5qLHWbgBO9Jm9AvgPd/w/gI9kWHUw39ecxGetfcpa29Vr+kvA1NHe7mBl2X+Dkbf918UYY4D/AvxktLc7VsZjIpgCHEibPsiZB9rBlMk5Y8wM4ALg5QyLLzXGvGaM+bUxZv7YRoYFnjLGbDLG3JRheUHsP2AV2f/58rn/utRaa4+A8wMAqMlQplD25X/DqeVlMtD3IZdWu6eu7s9yaq0Q9t8VwDFr7e4sy/O5/wZlPCYCk2Fe33tkB1Mmp4wxEeA/gf9urT3dZ/FmnNMd7wG+DfxiLGMDLrPWXghcDdxqjLmyz/JC2H8h4FrgZxkW53v/DUUh7MvPAwng4SxFBvo+5Mp3gdnAYuAIzumXvvK+/4Dr6b82kK/9N2jjMREcBKalTU8FDg+jTM4YY4I4SeBha+3P+y631p621sbc8SeBoDGmaqzis9Yedl/rgcdwqt/p8rr/XFcDm621x/ouyPf+S3Os65SZ+1qfoUy+v4ufBD4EfMK6J7T7GsT3ISestcestUlrbQr4fpbt5nv/BYC/AB7JViZf+28oxmMieBWYY4yZ6f5qXAU80afME8Bfu3e/XAKc6qrC55p7PvH/Arustd/MUmaSWw5jzEU4f6fGMYqv1BgT7RrHuaC4vU+xvO2/NFl/heVz//XxBPBJd/yTwOMZygzm+5oTxpjlwO3Atdba1ixlBvN9yFV86dedPpplu3nbf64/A1631h7MtDCf+29I8n21OhcDzl0tb+LcTfB5d97NwM3uuAHudZdvA5aMYWyX41RdtwJb3OGaPvGtBnbg3AHxEvC+MYxvlrvd19wYCmr/udsvwTmwl6XNy+v+w0lKR4A4zq/UvwEqgWeB3e7rRLfsWcCT/X1fxyi+PTjn17u+h/f1jS/b92GM4nvI/X5txTm4Ty6k/efOf7Dre5dWdsz330gHNTEhIuJx4/HUkIiIDIESgYiIxykRiIh4nBKBiIjHKRGIiHicEoFIH8aYpOndwumotWhpjJmR3oKlSCEI5DsAkQLUZq1dnO8gRMaKagQig+S2K/91Y8wr7nCOO3+6MeZZt3G0Z40xZ7vza912/l9zh/e5b+U3xnzfOP1RPGWMKc7bhxJBiUAkk+I+p4auS1t22lp7EXAPcJc77x6cZrkX4TTcdrc7/27gOes0fnchzpOlAHOAe62184Em4GM5/TQiA9CTxSJ9GGNi1tpIhvn7gPdba/e6DQcetdZWGmOO4zR/EHfnH7HWVhljGoCp1tqOtPeYATxtrZ3jTt8OBK21a8bgo4lkpBqByNDYLOPZymTSkTaeRNfqJM+UCESG5rq01xfd8RdwWr0E+ATwe3f8WeDTAMYYvzFmwlgFKTIU+iUicqbiPh2R/8Za23ULadgY8zLOj6jr3XmfAe43xvxPoAG40Z1/G/A9Y8zf4Pzy/zROC5YiBUXXCEQGyb1GsMRaezzfsYiMJp0aEhHxONUIREQ8TjUCERGPUyIQEfE4JQIREY9TIhAR8TglAhERj/v/stUDiN+h4j8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "\n",
    "metrics = pd.read_csv(f\"{trainer.logger.log_dir}/metrics.csv\")\n",
    "\n",
    "aggreg_metrics = []\n",
    "agg_col = \"epoch\"\n",
    "for i, dfg in metrics.groupby(agg_col):\n",
    "    agg = dict(dfg.mean())\n",
    "    agg[agg_col] = i\n",
    "    aggreg_metrics.append(agg)\n",
    "\n",
    "df_metrics = pd.DataFrame(aggreg_metrics)\n",
    "df_metrics[[\"train_loss\", \"valid_loss\"]].plot(\n",
    "    grid=True, legend=True, xlabel='Epoch', ylabel='Loss')\n",
    "df_metrics[[\"train_mae\", \"valid_mae\"]].plot(\n",
    "    grid=True, legend=True, xlabel='Epoch', ylabel='MAE')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d566ed8",
   "metadata": {},
   "source": [
    "- As we can see from the loss plot above, the model starts overfitting pretty quickly; however the validation set MAE keeps improving. Based on the MAE plot, we can see that the best model, based on the validation set MAE, may be around epoch 16.\n",
    "- The `trainer` saved this model automatically for us, we which we can load from the checkpoint via the `ckpt_path='best'` argument; below we use the `trainer` instance to evaluate the best model on the test set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "c70e0b09",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Restoring states from the checkpoint path at logs/cnn-corn-mnist/version_0/checkpoints/epoch=17-step=3852.ckpt\n",
      "LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
      "Loaded model weights from checkpoint at logs/cnn-corn-mnist/version_0/checkpoints/epoch=17-step=3852.ckpt\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "62e0b4987a88403bafafc680c54b3ef4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Testing: 0it [00:00, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\">        Test metric        </span>┃<span style=\"font-weight: bold\">       DataLoader 0        </span>┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│<span style=\"color: #008080; text-decoration-color: #008080\">         test_mae          </span>│<span style=\"color: #800080; text-decoration-color: #800080\">    0.11959999799728394    </span>│\n",
       "└───────────────────────────┴───────────────────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1m       Test metric       \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m      DataLoader 0       \u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
       "│\u001b[36m \u001b[0m\u001b[36m        test_mae         \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m   0.11959999799728394   \u001b[0m\u001b[35m \u001b[0m│\n",
       "└───────────────────────────┴───────────────────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "[{'test_mae': 0.11959999799728394}]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainer.test(model=lightning_model, datamodule=data_module, ckpt_path='best')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03484f86",
   "metadata": {},
   "source": [
    "- The MAE of our model is quite good, especially compared to the 2.52 MAE baseline earlier."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5379fe0f",
   "metadata": {},
   "source": [
    "## Predicting labels of new data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20707635",
   "metadata": {},
   "source": [
    "- You can use the `trainer.predict` method on a new `DataLoader` or `DataModule` to apply the model to new data.\n",
    "- Alternatively, you can also manually load the best model from a checkpoint as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "e6116a03",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "logs/cnn-corn-mnist/version_0/checkpoints/epoch=17-step=3852.ckpt\n"
     ]
    }
   ],
   "source": [
    "path = trainer.checkpoint_callback.best_model_path\n",
    "print(path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "b81dabdd",
   "metadata": {},
   "outputs": [],
   "source": [
    "lightning_model = LightningCNN.load_from_checkpoint(path, model=pytorch_model)\n",
    "lightning_model.eval();"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03f1d045",
   "metadata": {},
   "source": [
    "- Note that our `ConvNet`, which is passed to `LightningCNN` requires input arguments. However, this is automatically being taken care of since we used `self.save_hyperparameters()` in `LightningCNN`'s `__init__` method.\n",
    "- Now, below is an example applying the model manually. Here, pretend that the `test_dataloader` is a new data loader."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "3cf7304f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([7, 2, 1, 0, 4])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_dataloader = data_module.test_dataloader()\n",
    "\n",
    "all_predicted_labels = []\n",
    "for batch in test_dataloader:\n",
    "    features, _ = batch\n",
    "    logits = lightning_model(features)\n",
    "    predicted_labels = corn_label_from_logits(logits)\n",
    "    all_predicted_labels.append(predicted_labels)\n",
    "    \n",
    "all_predicted_labels = torch.cat(all_predicted_labels)\n",
    "all_predicted_labels[:5]"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}