tutorials/introduction_pytorch_poutyne_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to PyTorch and Poutyne\n",
    "\n",
    "In this notebook, we train a simple fully-connected network and a simple convolutional network on MNIST. First, we train it by coding our own training loop as the PyTorch library expects of us to. Then, we use Poutyne to simplify our code.\n",
    "\n",
    "Let's import all the needed packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import math\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import torch\n",
    "from torch import nn\n",
    "import torch.optim as optim\n",
    "from torch.utils.data.dataset import Subset\n",
    "from torchvision import transforms, utils\n",
    "from torchvision.datasets.mnist import MNIST\n",
    "\n",
    "from poutyne import Model, ModelCheckpoint, CSVLogger, Callback, Experiment, torch_to_numpy, set_seeds, Callback"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Also, we need to set Pythons's, NumPy's and PyTorch's seeds by using Poutyne function so that our training is (almost) reproducible."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "set_seeds(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Basis of Training a Neural Network\n",
    "\n",
    "In **stochastic gradient descent**, a **batch** of `m` examples are drawn from the train dataset. In the so-called forward pass, these examples are passed through the neural network and an average of their loss values is done. In the backward pass, the average loss is backpropagated through the network to compute the gradient of each parameter. In practice, the `m` examples of a batch are drawn without replacement. Thus, we define one **epoch** of training being the number of batches needed to loop through the entire training dataset.\n",
    "\n",
    "In addition to the training dataset, a **validation dataset** is used to evaluate the neural network at the end of each epoch. This validation dataset can be used to select the best model during training and thus avoiding overfitting the training set. It also can have other uses such as selecting hyperparameters\n",
    "\n",
    "Finally, a **test dataset** is used at the end to evaluate the final model.\n",
    "\n",
    "## Training constants"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Now, let's set our training constants. We first have the CUDA device used for training if one is present. Second, we set the train_split to 0.8 (80%) to use 80% of the dataset for training and 20% for testing the trained model. Third, we set the number of classes (i.e. one for each number). Finally, we set the batch size (i.e. the number of elements to see before updating the model), the learning rate for the optimizer, and the epoch number (i.e. the number of times we see the full dataset)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "cuda_device = 0\n",
    "device = torch.device(\"cuda:%d\" % cuda_device if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "train_split_percent = 0.8\n",
    "\n",
    "num_classes = 10\n",
    "\n",
    "batch_size = 32\n",
    "learning_rate = 0.1\n",
    "num_epochs = 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the MNIST dataset\n",
    "\n",
    "The following loads the MNIST dataset and creates the PyTorch DataLoaders that split our datasets into batches. The train DataLoader shuffles the examples of the train dataset to draw the examples without replacement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_dataset = MNIST('./mnist/', train=True, download=True, transform=transforms.ToTensor())\n",
    "valid_dataset = MNIST('./mnist/', train=True, download=True, transform=transforms.ToTensor())\n",
    "test_dataset = MNIST('./mnist/', train=False, download=True, transform=transforms.ToTensor())\n",
    "\n",
    "num_data = len(train_dataset)\n",
    "indices = list(range(num_data))\n",
    "np.random.shuffle(indices)\n",
    "\n",
    "split = math.floor(train_split_percent * num_data)\n",
    "\n",
    "train_indices = indices[:split]\n",
    "train_dataset = Subset(train_dataset, train_indices)\n",
    "\n",
    "valid_indices = indices[split:]\n",
    "valid_dataset = Subset(valid_dataset, valid_indices)\n",
    "\n",
    "train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=batch_size, shuffle=True)\n",
    "valid_loader = torch.utils.data.DataLoader(valid_dataset, batch_size=batch_size)\n",
    "test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=batch_size)\n",
    "\n",
    "loaders = train_loader, valid_loader, test_loader"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at some examples of the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Get the first batch in our train DataLoader and\n",
    "# format it in grid.\n",
    "inputs = next(iter(train_loader))[0]\n",
    "input_grid = utils.make_grid(inputs)\n",
    "\n",
    "# Plot the images.\n",
    "fig = plt.figure(figsize=(10, 10))\n",
    "inp = input_grid.numpy().transpose((1, 2, 0))\n",
    "plt.imshow(inp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neural Network Architectures\n",
    "\n",
    "We train a fully-connected neural network and a convolutional neural network with approximately the same number of parameters.\n",
    "\n",
    "### Fully-connected Network\n",
    "In short, the fully-connected network follows this architecture: ``Input -> [Linear -> ReLU]*3 -> Linear``. The following table shows it in details:\n",
    "\n",
    "| Layer Type                  | Output Size |    # of Parameters   |\n",
    "|-----------------------------|:-----------:|:--------------------:|\n",
    "| Input                       |   1x28x28   |           0          |\n",
    "| Flatten                     |  1\\*28\\*28  |           0          |\n",
    "| **Linear with 256 neurons** |     256     | 28\\*28*256 = 200,704 |\n",
    "| ReLU                        |      *      |           0          |\n",
    "| **Linear with 128 neurons** |     128     |   256*128 = 32,768   |\n",
    "| ReLU                        |      *      |           0          |\n",
    "| **Linear with 64 neurons**  |     64      |    128*64 = 8,192    |\n",
    "| ReLU                        |      *      |           0          |\n",
    "| **Linear with 10 neurons**  |      10     |     64*10 = 640      |\n",
    "\n",
    "Total # of parameters of the fully-connected network: 242,304\n",
    "\n",
    "### Convolutional Network\n",
    "\n",
    "The convolutional neural network architecture starts with some convolution and max-pooling layers. These are then followed by fully-connected layers. We calculate the total number of parameters that the network needs. In short, the convolutional network follows this architecture: ``Input -> [Conv -> ReLU -> MaxPool]*2 -> Dropout -> Linear -> ReLU -> Dropout -> Linear``. The following table shows it in details:\n",
    "\n",
    "| Layer Type                                     | Output Size |     # of Parameters     |\n",
    "|------------------------------------------------|:-----------:|:-----------------------:|\n",
    "| Input                                          |   1x28x28   |            0            |\n",
    "| **Conv with 16 3x3 filters with padding of 1** |   16x28x28  |      16\\*3\\*3 = 144     |\n",
    "| ReLU                                           |   16x28x28  |            0            |\n",
    "| MaxPool 2x2                                    |   16x14x14  |            0            |\n",
    "| **Conv with 32 3x3 filters with padding of 1** |   32x14x14  |      32\\*3\\*3 = 288     |\n",
    "| ReLU                                           |   32x14x14  |            0            |\n",
    "| MaxPool 2x2                                    |    32x7x7   |            0            |\n",
    "| Dropout of 0.25                                |    32x7x7   |            0            |\n",
    "| Flatten                                        |   32\\*7\\*7  |            0            |\n",
    "| **Linear with 128 neurons**                    |     128     | 32\\*7\\*7\\*128 = 200,704 |\n",
    "| ReLU                                           |     128     |            0            |\n",
    "| Dropout of 0.5                                 |     128     |            0            |\n",
    "| **Linear with 10 neurons**                     |      10     |      128\\*10 = 1280     |\n",
    "\n",
    "Total # of parameters of the convolutional network: 202,416"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "def create_fully_connected_network():\n",
    "    \"\"\"\n",
    "    This function should return the fully-connected network layed out above.\n",
    "    \"\"\"\n",
    "    pass\n",
    "\n",
    "\n",
    "def create_convolutional_network():\n",
    "    \"\"\"\n",
    "    This function should return the convolutional network layed out above.\n",
    "    \"\"\"\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Training the PyTorch way \n",
    "\n",
    "That is, doing your own training loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def pytorch_accuracy(y_pred, y_true):\n",
    "    \"\"\"\n",
    "    Computes the accuracy for a batch of predictions\n",
    "\n",
    "    Args:\n",
    "        y_pred (torch.Tensor): the logit predictions of the neural network.\n",
    "        y_true (torch.Tensor): the ground truths.\n",
    "\n",
    "    Returns:\n",
    "        The average accuracy of the batch.\n",
    "    \"\"\"\n",
    "    y_pred = y_pred.argmax(1)\n",
    "    return (y_pred == y_true).float().mean() * 100\n",
    "\n",
    "\n",
    "def pytorch_train_one_epoch(network, optimizer, loss_function):\n",
    "    \"\"\"\n",
    "    This function should train the neural network for one epoch on the train DataLoader.\n",
    "\n",
    "    Args:\n",
    "        network (torch.nn.Module): The neural network to train.\n",
    "        optimizer (torch.optim.Optimizer): The optimizer of the neural network\n",
    "        loss_function: The loss function.\n",
    "\n",
    "    Returns:\n",
    "        A tuple (loss, accuracy) corresponding to an average of the losses and\n",
    "        an average of the accuracy, respectively, on the train DataLoader.\n",
    "    \"\"\"\n",
    "    network.train(True)\n",
    "    with torch.enable_grad():\n",
    "        pass\n",
    "\n",
    "\n",
    "def pytorch_test(network, loader, loss_function):\n",
    "    \"\"\"\n",
    "    This function should test the neural network on a DataLoader.\n",
    "\n",
    "    Args:\n",
    "        network (torch.nn.Module): The neural network to test.\n",
    "        loader (torch.utils.data.DataLoader): The DataLoader to test on.\n",
    "        loss_function: The loss function.\n",
    "\n",
    "    Returns:\n",
    "        A tuple (loss, accuracy) corresponding to an average of the losses and\n",
    "        an average of the accuracy, respectively, on the DataLoader.\n",
    "    \"\"\"\n",
    "    network.eval()\n",
    "    with torch.no_grad():\n",
    "        pass\n",
    "\n",
    "\n",
    "def pytorch_train(network):\n",
    "    \"\"\"\n",
    "    This function should transfer the neural network to the right device,\n",
    "    train it for a certain number of epochs, test at each epoch on the\n",
    "    validation set and output the results on the test set at the end of\n",
    "    training.\n",
    "\n",
    "    Args:\n",
    "        network (torch.nn.Module): The neural network to train.\n",
    "\n",
    "    Example:\n",
    "        This function should display something like this:\n",
    "\n",
    "        .. code-block:: python\n",
    "\n",
    "            Epoch 1/5: loss: 0.5026924496193726, acc: 84.26666259765625, val_loss: 0.17258917854229608, val_acc: 94.75\n",
    "            Epoch 2/5: loss: 0.13690324830015502, acc: 95.73332977294922, val_loss: 0.14024296019474666, val_acc: 95.68333435058594\n",
    "            Epoch 3/5: loss: 0.08836929737279813, acc: 97.29582977294922, val_loss: 0.10380942322810491, val_acc: 96.66666412353516\n",
    "            Epoch 4/5: loss: 0.06714504160980383, acc: 97.91874694824219, val_loss: 0.09626663728555043, val_acc: 97.18333435058594\n",
    "            Epoch 5/5: loss: 0.05063822727650404, acc: 98.42708587646484, val_loss: 0.10017542181412378, val_acc: 96.95833587646484\n",
    "            Test:\n",
    "                Loss: 0.09501855444908142\n",
    "                Accuracy: 97.12999725341797\n",
    "    \"\"\"\n",
    "    print(network)\n",
    "\n",
    "    pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "fc_net = create_fully_connected_network()\n",
    "pytorch_train(fc_net)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "conv_net = create_convolutional_network()\n",
    "pytorch_train(conv_net)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Training the Poutyne way\n",
    "\n",
    "That is, only 8 lines of code with a better output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def poutyne_train(pytorch_network):\n",
    "    \"\"\"\n",
    "    This function should create a Poutyne Model (see https://poutyne.org/model.html),\n",
    "    send the Model on the right device, and use the `fit_generator` method to train\n",
    "    the neural network. At the end, the `evaluate_generator` should be used on the test\n",
    "    set.\n",
    "\n",
    "    Args:\n",
    "        pytorch_network (torch.nn.Module): The neural network to train.\n",
    "    \"\"\"\n",
    "    print(pytorch_network)\n",
    "\n",
    "    pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "fc_net = create_fully_connected_network()\n",
    "poutyne_train(fc_net)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "conv_net = create_convolutional_network()\n",
    "poutyne_train(conv_net)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Poutyne Callbacks\n",
    "\n",
    "One nice feature of Poutyne is [callbacks](https://poutyne.org/callbacks.html). Callbacks allow to do actions during training of the neural network. In the following exercice, use 3 callbacks. One that saves the latest weights in a file to be able to continue the optimization at the end of training if more epochs are needed. Another one that saves the best weights according to the performance on the validation dataset. Finally, another one that saves the displayed logs into a TSV file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def train_with_callbacks(name, pytorch_network):\n",
    "    \"\"\"\n",
    "    In addition to the the `poutyne_train`, this function should save checkpoints and logs as described above.\n",
    "\n",
    "    Args:\n",
    "        name (str): a name used to save logs and checkpoints.\n",
    "        pytorch_network (torch.nn.Module): The neural network to train.\n",
    "    \"\"\"\n",
    "    print(pytorch_network)\n",
    "\n",
    "    pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "fc_net = create_fully_connected_network()\n",
    "train_with_callbacks('fc', fc_net)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "conv_net = create_convolutional_network()\n",
    "train_with_callbacks('conv', conv_net)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Making Your Own Callback\n",
    "\n",
    "While Poutyne provides a great number of [predefined callbacks](https://poutyne.org/callbacks.html), it is sometimes useful to make your own callback.\n",
    "\n",
    "In the following exercice, we want to see the effect of temperature on the optimization of our neural network. To do so, you should either increase or decrease the temperature during the optimization. As one will see in the result, temperature either as no effect or has detrimental effect on the performance of the neural network. This is so because the temperature has for effect to artificially changing the learning rates. Since we have found the right learning rate, increasing or decreasing it shows no improvement on the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "class CrossEntropyLossWithTemperature(nn.Module):\n",
    "    \"\"\"\n",
    "    This loss module is the cross-entropy loss function\n",
    "    with temperature. It divides the logits by a temperature\n",
    "    value before computing the cross-entropy loss.\n",
    "\n",
    "    Args:\n",
    "        initial_temperature (float): The initial value of the temperature.\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, initial_temperature):\n",
    "        super().__init__()\n",
    "        self.temperature = initial_temperature\n",
    "        self.celoss = nn.CrossEntropyLoss()\n",
    "\n",
    "    def forward(self, y_pred, y_true):\n",
    "        pass\n",
    "\n",
    "\n",
    "class TemperatureCallback(Callback):\n",
    "    \"\"\"\n",
    "    This callback multiply the loss temperature with a decay before\n",
    "    each batch.\n",
    "\n",
    "    Args:\n",
    "        celoss_with_temp (CrossEntropyLossWithTemperature): the loss module.\n",
    "        decay (float): The value of the temperature decay.\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, celoss_with_temp, decay):\n",
    "        super().__init__()\n",
    "        self.celoss_with_temp = celoss_with_temp\n",
    "        self.decay = decay\n",
    "\n",
    "    def on_train_batch_begin(self, batch, logs):\n",
    "        pass\n",
    "\n",
    "\n",
    "def train_with_temperature(network, initial_temperature, temperature_decay):\n",
    "    \"\"\"\n",
    "    In addition to the the `poutyne_train`, this function should use a cross-entropy\n",
    "    loss with temperature and should decay the temperature at each batch.\n",
    "\n",
    "    Args:\n",
    "        network (torch.nn.Module): The neural network to train.\n",
    "        initial_temperature (float): The initial value of the temperature.\n",
    "        decay (float): The value of the temperature decay.\n",
    "    \"\"\"\n",
    "    print(network)\n",
    "\n",
    "    pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "conv_net = create_convolutional_network()\n",
    "# Initial temperature = 0.1\n",
    "# Final temperature ≈ 0.1 * 1.0008^7500 ≈ 40.25\n",
    "train_with_temperature(conv_net, initial_temperature=0.1, temperature_decay=1.0008)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "conv_net = create_convolutional_network()\n",
    "# Initial temperature = 40.25\n",
    "# Final temperature ≈ 40.25 * 0.9992^7500 ≈ 0.1\n",
    "train_with_temperature(conv_net, initial_temperature=4.25, temperature_decay=0.9995)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Poutyne ModelBundle\n",
    "\n",
    "Most of the time when using Poutyne (or even Pytorch in general), we will find ourselves in an iterative model hyperparameters finetuning loop. For efficient model search, we will usually wish to save our best performing models, their training and testing statistics and even sometimes wish to retrain an already trained model for further tuning. All of the above can be easily implemented with the flexibility of Poutyne Callbacks, but having to define and initialize each and every Callback object we wish for our model quickly feels cumbersome.\n",
    "\n",
    "This is why Poutyne provides a [ModelBundle class](https://poutyne.org/experiment.html#poutyne.ModelBundle), which aims specifically at enabling quick model iteration search, while not sacrifying on the quality of a single experiment - statistics logging, best models saving, etc. ModelBundle is actually a simple wrapper between a PyTorch network and Poutyne's core Callback objects for logging and saving. Given a working directory where to output the various logging files and a PyTorch network, the ModelBundle class reduces the whole training loop to a single line.\n",
    "\n",
    "The following code should use [Poutyne's ModelBundle class](https://poutyne.org/experiment.html#poutyne.ModelBundle) to train a network for 5 epochs. The code should be quite simpler than the code in the Poutyne Callbacks section while doing more (only 3 lines). Once trained for 5 epochs, it is then possible to resume the optimization at the 5th epoch for 5 more epochs until the 10th epoch using the same function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def experiment_train(network, working_directory, epochs=5):\n",
    "    \"\"\"\n",
    "    This function should create a Poutyne Experiment, and use it to train the input\n",
    "    module on the train loader and test its performance on the test loader.\n",
    "    All training and testing statistics will be saved by the Experiment class, as well\n",
    "    as best model checkpoints.\n",
    "\n",
    "    Args:\n",
    "        network (torch.nn.Module): The neural network to train.\n",
    "        working_directory (str): The directory where to output files to save.\n",
    "        epochs (int): The number of epochs. (Default: 5)\n",
    "    \"\"\"\n",
    "    print(network)\n",
    "\n",
    "    pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "conv_net = create_convolutional_network()\n",
    "experiment_train(conv_net, './conv_net_model_bundle')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "conv_net = create_convolutional_network()\n",
    "experiment_train(conv_net, './conv_net_model_bundle', epochs=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Next Step\n",
    "We have plenty more examples for you, see [this](https://github.com/GRAAL-Research/poutyne/blob/master/examples/tips_and_tricks.ipynb) notebook for tips and tricks using Poutyne, this [one](https://github.com/GRAAL-Research/poutyne/blob/master/examples/policy_cifar_example.ipynb) and this [one](https://github.com/GRAAL-Research/poutyne/blob/master/examples/policy_interface.ipynb) for policy examples or this [one](https://github.com/GRAAL-Research/poutyne/blob/master/examples/transfer_learning.ipynb) for a transfer learning example."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}