extras/exercises/04_pytorch_custom_datasets_exercises.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "04_pytorch_custom_datasets_exercises.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyO06uacGGDzw0TkyZ8mqgSU",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/mrdbourke/pytorch-deep-learning/blob/main/extras/exercises/04_pytorch_custom_datasets_exercises.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# 04. PyTorch Custom Datasets Exercises Template\n",
        "\n",
        "Welcome to the 04. PyTorch Custom Datasets exercise template.\n",
        "\n",
        "The best way to practice PyTorch code is to write more PyTorch code.\n",
        "\n",
        "So read the original notebook and try to complete the exercises by writing code where it's required.\n",
        "\n",
        "Feel free to reference the original resources whenever you need but should practice writing all of the code yourself.\n",
        "\n",
        "## Resources\n",
        "\n",
        "1. These exercises/solutions are based on [notebook 04 of the Learn PyTorch for Deep Learning course](https://www.learnpytorch.io/04_pytorch_custom_datasets/).\n",
        "2. See a live [walkthrough of the solutions (errors and all) on YouTube](https://youtu.be/vsFMF9wqWx0).\n",
        "3. See [other solutions on the course GitHub](https://github.com/mrdbourke/pytorch-deep-learning/tree/main/extras/solutions)."
      ],
      "metadata": {
        "id": "Vex99np2wFVt"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Check for GPU\n",
        "!nvidia-smi"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GaeYzOTLwWh2",
        "outputId": "14673108-7705-4e73-b7bc-d94bf9c96423",
        "collapsed": true
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mon Apr 18 22:14:23 2022       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                               |                      |               MIG M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla P100-PCIE...  Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   37C    P0    28W / 250W |      0MiB / 16280MiB |      0%      Default |\n",
            "|                               |                      |                  N/A |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                                  |\n",
            "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
            "|        ID   ID                                                   Usage      |\n",
            "|=============================================================================|\n",
            "|  No running processes found                                                 |\n",
            "+-----------------------------------------------------------------------------+\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Import torch\n",
        "import torch\n",
        "from torch import nn\n",
        "\n",
        "# Exercises require PyTorch > 1.10.0\n",
        "print(torch.__version__)\n",
        "\n",
        "# Setup device agnostic code\n",
        "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
        "device"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "id": "DNwZLMbCzJLk",
        "outputId": "1a7b73e2-ec4b-41b0-b4da-c3216a8a29ac"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "1.10.0+cu111\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'cuda'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 1. Our models are underperforming (not fitting the data well). What are 3 methods for preventing underfitting? Write them down and explain each with a sentence."
      ],
      "metadata": {
        "id": "FSFX7tc1w-en"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        ""
      ],
      "metadata": {
        "id": "-90Qvx9gtrLj"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 2. Recreate the data loading functions we built in [sections 1, 2, 3 and 4 of notebook 04](https://www.learnpytorch.io/04_pytorch_custom_datasets/). You should have train and test `DataLoader`'s ready to use."
      ],
      "metadata": {
        "id": "oBK-WI6YxDYa"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# 1. Get data\n"
      ],
      "metadata": {
        "id": "MZkCPJBR3lw4"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# 2. Become one with the data\n",
        "import os\n",
        "def walk_through_dir(dir_path):\n",
        "  \"\"\"Walks through dir_path returning file counts of its contents.\"\"\"\n",
        "  for dirpath, dirnames, filenames in os.walk(dir_path):\n",
        "    print(f\"There are {len(dirnames)} directories and {len(filenames)} images in '{dirpath}'.\")"
      ],
      "metadata": {
        "id": "TYmhAX7J52VX"
      },
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Setup train and testing paths\n"
      ],
      "metadata": {
        "id": "3A9ZmOn-7Jhh"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Visualize an image"
      ],
      "metadata": {
        "id": "51ywNKkN7WOl"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Do the image visualization with matplotlib\n"
      ],
      "metadata": {
        "id": "Qe4LoASC9sQ-"
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We've got some images in our folders.\n",
        "\n",
        "Now we need to make them compatible with PyTorch by:\n",
        "1. Transform the data into tensors.\n",
        "2. Turn the tensor data into a `torch.utils.data.Dataset` and later a `torch.utils.data.DataLoader`."
      ],
      "metadata": {
        "id": "2AU4FGYC_KBz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# 3.1 Transforming data with torchvision.transforms\n"
      ],
      "metadata": {
        "id": "KbGMaYGT-vwq"
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Write transform for turning images into tensors\n"
      ],
      "metadata": {
        "id": "gnvUSYYW_ohN"
      },
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Write a function to plot transformed images\n"
      ],
      "metadata": {
        "id": "vp8I2cpMAxcT"
      },
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Load image data using `ImageFolder`"
      ],
      "metadata": {
        "id": "FKgfqPArChVR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Use ImageFolder to create dataset(s)\n"
      ],
      "metadata": {
        "id": "8OFgwQF1CkOu"
      },
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Get class names as a list\n",
        "class_names = train_data.classes\n",
        "class_names"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MbT0fhXHEQyJ",
        "outputId": "ad2b0da9-285c-493d-c362-a6b00e5d2197"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['pizza', 'steak', 'sushi']"
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Can also get class names as a dict\n",
        "class_dict = train_data.class_to_idx\n",
        "class_dict"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uCcWk7NDEay1",
        "outputId": "88718fd6-c6b2-4132-9383-0a64a0a7bcee"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'pizza': 0, 'steak': 1, 'sushi': 2}"
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Check the lengths of each dataset\n",
        "len(train_data), len(test_data)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "W7H7bX4HEgie",
        "outputId": "2ff1fdd2-f990-461d-c2e9-27435e7744c9"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(225, 75)"
            ]
          },
          "metadata": {},
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Turn train and test Datasets into DataLoaders\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nskNr5YCEoRl",
        "outputId": "cd825b1d-095f-4080-86ce-7d17aafef7e8"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(<torch.utils.data.dataloader.DataLoader at 0x7fce57ec08d0>,\n",
              " <torch.utils.data.dataloader.DataLoader at 0x7fce57ec0dd0>)"
            ]
          },
          "metadata": {},
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# How many batches of images are in our data loaders?\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "z8vJxmxAFqw6",
        "outputId": "e5a8bd88-1b05-4109-de5a-8183a85c7872"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(225, 75)"
            ]
          },
          "metadata": {},
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 3. Recreate `model_0` we built in section 7 of notebook 04."
      ],
      "metadata": {
        "id": "XeYFEqw8xK26"
      }
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "MBErjcUCyDzE"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 4. Create training and testing functions for `model_0`."
      ],
      "metadata": {
        "id": "DKdEEFEqxM-8"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def train_step(model: torch.nn.Module,\n",
        "               dataloader: torch.utils.data.DataLoader,\n",
        "               loss_fn: torch.nn.Module,\n",
        "               optimizer: torch.optim.Optimizer):\n",
        "  \n",
        "  # Put the model in train mode\n",
        "  model.train()\n",
        "\n",
        "  # Setup train loss and train accuracy values\n",
        "  train_loss, train_acc = 0, 0\n",
        "\n",
        "  # Loop through data loader and data batches\n",
        " \n",
        "    # Send data to target device\n",
        "\n",
        "    # 1. Forward pass\n",
        "    \n",
        "    # 2. Calculate and accumulate loss\n",
        "    \n",
        "\n",
        "    # 3. Optimizer zero grad \n",
        "    \n",
        "\n",
        "    # 4. Loss backward \n",
        "    \n",
        "\n",
        "    # 5. Optimizer step\n",
        "    \n",
        "\n",
        "    # Calculate and accumualte accuracy metric across all batches\n",
        "   \n",
        "\n",
        "  # Adjust metrics to get average loss and average accuracy per batch\n",
        "  "
      ],
      "metadata": {
        "id": "rnUox1qayDes"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def test_step(model: torch.nn.Module,\n",
        "              dataloader: torch.utils.data.DataLoader,\n",
        "              loss_fn: torch.nn.Module):\n",
        "  \n",
        "  # Put model in eval mode\n",
        "  model.eval()\n",
        "\n",
        "  # Setup the test loss and test accuracy values\n",
        "  test_loss, test_acc = 0, 0\n",
        "\n",
        "  # Turn on inference context manager\n",
        "  \n",
        "    # Loop through DataLoader batches\n",
        "    \n",
        "      # Send data to target device\n",
        "      \n",
        "\n",
        "      # 1. Forward pass\n",
        "      \n",
        "\n",
        "      # 2. Calculuate and accumulate loss\n",
        "\n",
        "\n",
        "      # Calculate and accumulate accuracy\n",
        "\n",
        "    \n",
        "  # Adjust metrics to get average loss and accuracy per batch\n"
      ],
      "metadata": {
        "id": "O7_EVPpHNKUP"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from tqdm.auto import tqdm\n",
        "\n",
        "def train(model: torch.nn.Module,\n",
        "          train_dataloader: torch.utils.data.DataLoader,\n",
        "          test_dataloader: torch.utils.data.DataLoader,\n",
        "          optimizer: torch.optim.Optimizer,\n",
        "          loss_fn: torch.nn.Module = nn.CrossEntropyLoss(),\n",
        "          epochs: int = 5):\n",
        "  \n",
        "  # Create results dictionary\n",
        "  results = {\"train_loss\": [],\n",
        "             \"train_acc\": [],\n",
        "             \"test_loss\": [],\n",
        "             \"test_acc\": []}\n",
        "\n",
        "  # Loop through the training and testing steps for a number of epochs\n",
        "  for epoch in tqdm(range(epochs)):\n",
        "    # Train step\n",
        "    train_loss, train_acc = train_step(model=model, \n",
        "                                       dataloader=train_dataloader,\n",
        "                                       loss_fn=loss_fn,\n",
        "                                       optimizer=optimizer)\n",
        "    # Test step\n",
        "    test_loss, test_acc = test_step(model=model, \n",
        "                                    dataloader=test_dataloader,\n",
        "                                    loss_fn=loss_fn)\n",
        "    \n",
        "    # Print out what's happening\n",
        "    print(f\"Epoch: {epoch+1} | \"\n",
        "          f\"train_loss: {train_loss:.4f} | \"\n",
        "          f\"train_acc: {train_acc:.4f} | \"\n",
        "          f\"test_loss: {test_loss:.4f} | \"\n",
        "          f\"test_acc: {test_acc:.4f}\"\n",
        "    )\n",
        "\n",
        "    # Update the results dictionary\n",
        "    results[\"train_loss\"].append(train_loss)\n",
        "    results[\"train_acc\"].append(train_acc)\n",
        "    results[\"test_loss\"].append(test_loss)\n",
        "    results[\"test_acc\"].append(test_acc)\n",
        "\n",
        "  # Return the results dictionary\n",
        "  return results"
      ],
      "metadata": {
        "id": "zXxTIh9tOh68"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 5. Try training the model you made in exercise 3 for 5, 20 and 50 epochs, what happens to the results?\n",
        "* Use `torch.optim.Adam()` with a learning rate of 0.001 as the optimizer. "
      ],
      "metadata": {
        "id": "lvf-3pODxXYI"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Train for 5 epochs\n",
        "torch.manual_seed(42)\n",
        "torch.cuda.manual_seed(42)\n",
        "\n",
        "loss_fn = nn.CrossEntropyLoss()\n",
        "optimizer = torch.optim.Adam(#TODO, \n",
        "                             lr=0.001)"
      ],
      "metadata": {
        "id": "rV7s2qtIyDIZ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Train for 20 epochs\n",
        "torch.manual_seed(42)\n",
        "torch.cuda.manual_seed(42)\n",
        "\n",
        "loss_fn = nn.CrossEntropyLoss()\n",
        "optimizer = torch.optim.Adam(#TODO, \n",
        "                             lr=0.001)"
      ],
      "metadata": {
        "id": "UEtZzyF1QGTq"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Train for 50 epochs\n",
        "torch.manual_seed(42)\n",
        "torch.cuda.manual_seed(42)\n",
        "\n",
        "loss_fn = nn.CrossEntropyLoss()\n",
        "optimizer = torch.optim.Adam(#TODO, \n",
        "                             lr=0.001)"
      ],
      "metadata": {
        "id": "Dwvg40qAQGP9"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "It looks like our model is starting to overfit towards the end (performing far better on the training data than on the testing data).\n",
        "\n",
        "In order to fix this, we'd have to introduce ways of preventing overfitting."
      ],
      "metadata": {
        "id": "fn8_fDGzQGMn"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 6. Double the number of hidden units in your model and train it for 20 epochs, what happens to the results?"
      ],
      "metadata": {
        "id": "qxZW-uAbxe_F"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Double the number of hidden units and train for 20 epochs\n",
        "torch.manual_seed(42)\n",
        "torch.cuda.manual_seed(42)\n"
      ],
      "metadata": {
        "id": "HdRM86voyC0x"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "It looks like the model is still overfitting, even when changing the number of hidden units.\n",
        "\n",
        "To fix this, we'd have to look at ways to prevent overfitting with our model."
      ],
      "metadata": {
        "id": "THYGHbxyTfzM"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 7. Double the data you're using with your model from step 6 and train it for 20 epochs, what happens to the results?\n",
        "* **Note:** You can use the [custom data creation notebook](https://github.com/mrdbourke/pytorch-deep-learning/blob/main/extras/04_custom_data_creation.ipynb) to scale up your Food101 dataset.\n",
        "* You can also find the [already formatted double data (20% instead of 10% subset) dataset on GitHub](https://github.com/mrdbourke/pytorch-deep-learning/blob/main/data/pizza_steak_sushi_20_percent.zip), you will need to write download code like in exercise 2 to get it into this notebook."
      ],
      "metadata": {
        "id": "JAPDzW0wxhi3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Download 20% data for Pizza/Steak/Sushi from GitHub\n",
        "import requests\n",
        "import zipfile\n",
        "from pathlib import Path\n",
        "\n",
        "# Setup path to data folder\n",
        "data_path = Path(\"data/\")\n",
        "image_path = data_path / \"pizza_steak_sushi_20_percent\"\n",
        "\n",
        "# If the image folder doesn't exist, download it and prepare it... \n",
        "if image_path.is_dir():\n",
        "    print(f\"{image_path} directory exists.\")\n",
        "else:\n",
        "    print(f\"Did not find {image_path} directory, creating one...\")\n",
        "    image_path.mkdir(parents=True, exist_ok=True)\n",
        "    \n",
        "# Download pizza, steak, sushi data\n",
        "with open(data_path / \"pizza_steak_sushi_20_percent.zip\", \"wb\") as f:\n",
        "    request = requests.get(\"https://github.com/mrdbourke/pytorch-deep-learning/raw/main/data/pizza_steak_sushi_20_percent.zip\")\n",
        "    print(\"Downloading pizza, steak, sushi 20% data...\")\n",
        "    f.write(request.content)\n",
        "\n",
        "# Unzip pizza, steak, sushi data\n",
        "with zipfile.ZipFile(data_path / \"pizza_steak_sushi_20_percent.zip\", \"r\") as zip_ref:\n",
        "    print(\"Unzipping pizza, steak, sushi 20% data...\") \n",
        "    zip_ref.extractall(image_path)"
      ],
      "metadata": {
        "id": "8tWfa7Y0yCkX"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# See how many images we have\n",
        "walk_through_dir(image_path)"
      ],
      "metadata": {
        "id": "DrFK2ScnVg4q"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Excellent, we now have double the training and testing images... "
      ],
      "metadata": {
        "id": "WhlWd-z-Vk22"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Create the train and test paths\n",
        "train_data_20_percent_path = image_path / \"train\"\n",
        "test_data_20_percent_path = image_path / \"test\"\n",
        "\n",
        "train_data_20_percent_path, test_data_20_percent_path"
      ],
      "metadata": {
        "id": "hNzXRfO1Tt1Q"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Turn the 20 percent datapaths into Datasets and DataLoaders\n",
        "from torchvision.datasets import ImageFolder\n",
        "from torchvision import transforms\n",
        "from torch.utils.data import DataLoader\n",
        "\n",
        "simple_transform = transforms.Compose([\n",
        "  transforms.Resize((64, 64)),                                     \n",
        "  transforms.ToTensor()\n",
        "])\n",
        "\n",
        "# Create datasets\n",
        "\n",
        "\n",
        "# Create dataloaders\n"
      ],
      "metadata": {
        "id": "R1_xU3FQUPkN"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Train a model with increased amount of data\n",
        "torch.manual_seed(42)\n",
        "torch.cuda.manual_seed(42)"
      ],
      "metadata": {
        "id": "BuJ9YpRCVXRm"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 8. Make a prediction on your own custom image of pizza/steak/sushi (you could even download one from the internet) with your trained model from exercise 7 and share your prediction. \n",
        "* Does the model you trained in exercise 7 get it right? \n",
        "* If not, what do you think you could do to improve it?"
      ],
      "metadata": {
        "id": "bCCVfXk5xjYS"
      }
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "Q1X-33t0vT20"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}