notebooks/Hugging_Face_Finetuning.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "dea00c9e-ab6a-4a56-8e91-782b06002f6c",
      "metadata": {
        "id": "dea00c9e-ab6a-4a56-8e91-782b06002f6c"
      },
      "source": [
        "# Fine-tuning a BERT model with skorch and Hugging Face"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "11b4d0cc-40c5-48a9-bd52-fdd522498acf",
      "metadata": {
        "id": "11b4d0cc-40c5-48a9-bd52-fdd522498acf"
      },
      "source": [
        "In this notebook, we follow the fine-tuning guideline from [Hugging Face documentation](https://huggingface.co/docs/transformers/training). Please check it out if we you want to know more about BERT and fine-tuning. Here, we assume that you're familiar with the general ideas.\n",
        "\n",
        "You will learn how to:\n",
        "- integrate the [Hugging Face transformers](https://huggingface.co/docs/transformers/index) library with skorch\n",
        "- use skorch to fine-tune a BERT model on a text classification task\n",
        "- use skorch with the [Hugging Face accelerate](https://huggingface.co/docs/accelerate/index) library for automatic mixed precision (AMP) training"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "922bfcd7",
      "metadata": {
        "id": "922bfcd7"
      },
      "source": [
        "<table align=\"left\"><td>\n",
        "<a target=\"_blank\" href=\"https://colab.research.google.com/github/skorch-dev/skorch/blob/master/notebooks/Hugging_Face_Finetuning.ipynb\">\n",
        "    <img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>  \n",
        "</td><td>\n",
        "<a target=\"_blank\" href=\"https://github.com/skorch-dev/skorch/blob/master/notebooks/Hugging_Face_Finetuning.ipynb\"><img width=32px src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a></td></table>"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "9737e20e",
      "metadata": {
        "id": "9737e20e"
      },
      "source": [
        "The first part of the notebook requires hugginface `transformers` as an additional dependency. If you have not already installed it, you can do so like this:\n",
        "\n",
        "`python -m pip install transformers`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "id": "465f48cc",
      "metadata": {
        "id": "465f48cc"
      },
      "outputs": [],
      "source": [
        "import subprocess\n",
        "\n",
        "# Installation on Google Colab\n",
        "try:\n",
        "    import google.colab\n",
        "    subprocess.run(['python', '-m', 'pip', 'install', 'skorch', 'transformers'])\n",
        "except ImportError:\n",
        "    pass"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f9e0f846-ea36-4a07-835d-c2b8c69c27fd",
      "metadata": {
        "id": "f9e0f846-ea36-4a07-835d-c2b8c69c27fd"
      },
      "source": [
        "## Imports"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "id": "5164c02c-9d4f-4b1c-bf72-0e8f21a23b89",
      "metadata": {
        "id": "5164c02c-9d4f-4b1c-bf72-0e8f21a23b89"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "import torch\n",
        "from sklearn.datasets import fetch_20newsgroups\n",
        "from sklearn.metrics import accuracy_score\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.pipeline import Pipeline\n",
        "from skorch import NeuralNetClassifier\n",
        "from skorch.callbacks import LRScheduler, ProgressBar\n",
        "from skorch.hf import HuggingfacePretrainedTokenizer\n",
        "from torch import nn\n",
        "from torch.optim.lr_scheduler import LambdaLR\n",
        "from transformers import AutoModelForSequenceClassification\n",
        "from transformers import AutoTokenizer"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "380c4fee-8c1d-42e2-933f-27570d1c7ea3",
      "metadata": {
        "id": "380c4fee-8c1d-42e2-933f-27570d1c7ea3"
      },
      "source": [
        "## Parameters"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "83c77bdf-9929-47b7-8e19-39b623fd4a52",
      "metadata": {
        "id": "83c77bdf-9929-47b7-8e19-39b623fd4a52"
      },
      "source": [
        "Change the values below if you want to try out different model architectures and hyper-parameters."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "id": "e6f17a28-52e3-4f0c-8c3e-308ecc9288f3",
      "metadata": {
        "id": "e6f17a28-52e3-4f0c-8c3e-308ecc9288f3"
      },
      "outputs": [],
      "source": [
        "# Choose a tokenizer and BERT model that work together\n",
        "TOKENIZER = \"distilbert-base-uncased\"\n",
        "PRETRAINED_MODEL = \"distilbert-base-uncased\"\n",
        "\n",
        "# model hyper-parameters\n",
        "OPTMIZER = torch.optim.AdamW\n",
        "LR = 5e-5\n",
        "MAX_EPOCHS = 3\n",
        "CRITERION = nn.CrossEntropyLoss\n",
        "BATCH_SIZE = 8\n",
        "\n",
        "# device\n",
        "DEVICE = 'cuda' if torch.cuda.is_available() else 'cpu'"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "556ed4ac-27e8-4b80-bcc6-2431f1d9ec12",
      "metadata": {
        "id": "556ed4ac-27e8-4b80-bcc6-2431f1d9ec12"
      },
      "source": [
        "## Data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "id": "8ecfd881-f589-493a-a028-68becdf240ea",
      "metadata": {
        "id": "8ecfd881-f589-493a-a028-68becdf240ea"
      },
      "outputs": [],
      "source": [
        "dataset = fetch_20newsgroups()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "27e33d0d-5bfb-49ce-82e7-d0e05d10c0e6",
      "metadata": {
        "id": "27e33d0d-5bfb-49ce-82e7-d0e05d10c0e6"
      },
      "source": [
        "For this notebook, we're making use the 20 newsgroups dataset. It is a text classification dataset with 20 classes. A decent score would be to reach 89% accuracy out of sample. For more details, read the description below:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "id": "a9d84f34-b506-40c1-a9da-2d40c0a2feca",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "a9d84f34-b506-40c1-a9da-2d40c0a2feca",
        "outputId": "de9fd9e4-4b32-49e2-9284-08000c15c71e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            ".. _20newsgroups_dataset:\n",
            "\n",
            "The 20 newsgroups text dataset\n",
            "------------------------------\n",
            "\n",
            "The 20 newsgroups dataset comprises around 18000 newsgroups posts on\n",
            "20 topics split in two subsets: one for training (or development)\n",
            "and the other one for testing (or for performance evaluation). The split\n",
            "between the train and test set is based upon a messages posted before\n",
            "and after a specific date.\n",
            "\n",
            "This module contains two loaders. The first one,\n",
            ":func:`sklearn.datasets.fetch_20newsgroups`,\n",
            "returns a list of the raw texts that can be fed to text feature\n",
            "extractors such as :class:`~sklearn.feature_extraction.text.CountVectorizer`\n",
            "with custom parameters so as to extract feature vectors.\n",
            "The second one, :func:`sklearn.datasets.fetch_20newsgroups_vectorized`,\n",
            "returns ready-to-use features, i.e., it is not necessary to use a feature\n",
            "extractor.\n",
            "\n",
            "**Data Set Characteristics:**\n",
            "\n",
            "    =================   ==========\n",
            "    Classes                     20\n",
            "    Samples total            18846\n",
            "    Dimensionality               1\n",
            "    Features                  text\n",
            "    =================   ==========\n",
            "\n",
            "\n"
          ]
        }
      ],
      "source": [
        "print(dataset.DESCR.split('Usage')[0])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "id": "cec3f23b-7906-4804-a517-db17001c22e6",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cec3f23b-7906-4804-a517-db17001c22e6",
        "outputId": "8301ef90-6246-462a-9ceb-62eb481daf9f"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['alt.atheism',\n",
              " 'comp.graphics',\n",
              " 'comp.os.ms-windows.misc',\n",
              " 'comp.sys.ibm.pc.hardware',\n",
              " 'comp.sys.mac.hardware',\n",
              " 'comp.windows.x',\n",
              " 'misc.forsale',\n",
              " 'rec.autos',\n",
              " 'rec.motorcycles',\n",
              " 'rec.sport.baseball',\n",
              " 'rec.sport.hockey',\n",
              " 'sci.crypt',\n",
              " 'sci.electronics',\n",
              " 'sci.med',\n",
              " 'sci.space',\n",
              " 'soc.religion.christian',\n",
              " 'talk.politics.guns',\n",
              " 'talk.politics.mideast',\n",
              " 'talk.politics.misc',\n",
              " 'talk.religion.misc']"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ],
      "source": [
        "dataset.target_names"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "id": "061eb1d4-4805-4a7f-91fe-b64b43939905",
      "metadata": {
        "id": "061eb1d4-4805-4a7f-91fe-b64b43939905"
      },
      "outputs": [],
      "source": [
        "X = dataset.data\n",
        "y = dataset.target"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "id": "b491705e-95b0-4699-86b0-71128e7d54e3",
      "metadata": {
        "id": "b491705e-95b0-4699-86b0-71128e7d54e3"
      },
      "outputs": [],
      "source": [
        "X_train, X_test, y_train, y_test, = train_test_split(X, y, stratify=y, random_state=0)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "id": "d4ec04ba-57fd-47bf-9528-1dc56fcad7ea",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "d4ec04ba-57fd-47bf-9528-1dc56fcad7ea",
        "outputId": "77d2d8c4-89c5-47c1-edc3-5d7309f53438"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['From: hilmi-er@dsv.su.se (Hilmi Eren)\\nSubject: Re: ARMENIA SAYS IT COULD SHOOT DOWN TURKISH PLANES (Henrik)\\nLines: 53\\nNntp-Posting-Host: alban.dsv.su.se\\nReply-To: hilmi-er@dsv.su.se (Hilmi Eren)\\nOrganization: Dept. of Computer and Systems Sciences, Stockholm University\\n\\n\\n  \\n|>      henrik@quayle.kpc.com writes:\\n\\n\\n|>\\tThe Armenians in Nagarno-Karabagh are simply DEFENDING their RIGHTS\\n|>        to keep their homeland and it is the AZERIS that are INVADING their \\n|>        territorium...\\n\\t\\n\\n\\tHomeland? First Nagarno-Karabagh was Armenians homeland today\\n\\tFizuli, Lacin and several villages (in Azerbadjan)\\n\\tare their homeland. Can\\'t you see the\\n\\tthe  \"Great Armenia\" dream in this? With facist methods like\\n\\tkilling, raping and bombing villages. The last move was the \\n\\tblast of a truck with 60 kurdish refugees, trying to\\n\\tescape the from Lacin, a city that was \"given\" to the Kurds\\n\\tby the Armenians. \\n\\n\\n|>       However, I hope that the Armenians WILL force a TURKISH airplane \\n|>       to LAND for purposes of SEARCHING for ARMS similar to the one\\n|>       that happened last SUMMER. Turkey searched an AMERICAN plane\\n|>       (carrying humanitarian aid) bound to ARMENIA.\\n|>\\n\\n\\tDon\\'t speak about things you don\\'t know: 8 American Cargo planes\\n\\twere heading to Armenia. When the Turkish authorities\\n\\tannounced that they were going to search these cargo \\n\\tplanes 3 of these planes returned to it\\'s base in Germany.\\n\\t5 of these planes were searched in Turkey. The content of\\n\\tof the other 3 planes? Not hard to guess, is it? It was sure not\\n\\thumanitarian aid.....\\n\\n\\tSearch Turkish planes? You don\\'t know what you are talking about.\\n\\tTurkey\\'s government has announced that it\\'s giving weapons\\n\\tto Azerbadjan since Armenia started to attack Azerbadjan\\n\\tit self, not the Karabag province. So why search a plane for weapons\\n\\tsince it\\'s content is announced to be weapons? \\n\\n\\n\\n\\n\\n\\nHilmi Eren\\nDept. of Computer and Systems Sciences, Stockholm University\\nSweden\\nHilmi-er@dsv.su.se\\n\\n\\n\\n\\n',\n",
              " 'Subject: VHS movie for sale\\nFrom: koutd@hirama.hiram.edu (DOUGLAS KOU)\\nOrganization: Hiram College\\nNntp-Posting-Host: hirama.hiram.edu\\nLines: 13\\n\\nVHS movie for sale.\\n\\nDance with Wovies\\t($12.00)\\n\\nThe tape is new and just open, buyer pay shipping cost.\\nIf you are interested, please send your offer to\\nkoutd@hirama.hiram.edu\\n\\nthanks,\\n\\nDouglas Kou\\nHiram College\\n\\n']"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ],
      "source": [
        "X_train[:2]"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ad5a8f51-98bc-4f09-b14d-5708a1788b80",
      "metadata": {
        "id": "ad5a8f51-98bc-4f09-b14d-5708a1788b80"
      },
      "source": [
        "## Prepare the training"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "9f7c0f71-4c62-4c3c-b411-02075d81169f",
      "metadata": {
        "id": "9f7c0f71-4c62-4c3c-b411-02075d81169f"
      },
      "source": [
        "We want to use a linear learning rate schedule that linearly decreases the learning rate during training."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "id": "acde3950-a9dc-434f-b76c-587e98fa8d35",
      "metadata": {
        "id": "acde3950-a9dc-434f-b76c-587e98fa8d35"
      },
      "outputs": [],
      "source": [
        "num_training_steps = MAX_EPOCHS * (len(X_train) // BATCH_SIZE + 1)\n",
        "\n",
        "def lr_schedule(current_step):\n",
        "    factor = float(num_training_steps - current_step) / float(max(1, num_training_steps))\n",
        "    assert factor > 0\n",
        "    return factor"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "3207018e-f281-4fff-b394-b35d41694bad",
      "metadata": {
        "id": "3207018e-f281-4fff-b394-b35d41694bad"
      },
      "source": [
        "Next we wrap the BERT module itself inside a simple `nn.Module`. The only real work for us here is to load the pretrained model and to return the _logits_ from the model output. The rest of the outputs is not needed."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "id": "57e0924d-65d4-4fd0-af48-338acf40ec53",
      "metadata": {
        "id": "57e0924d-65d4-4fd0-af48-338acf40ec53"
      },
      "outputs": [],
      "source": [
        "class BertModule(nn.Module):\n",
        "    def __init__(self, name, num_labels):\n",
        "        super().__init__()\n",
        "        self.name = name\n",
        "        self.num_labels = num_labels\n",
        "        \n",
        "        self.reset_weights()\n",
        "        \n",
        "    def reset_weights(self):\n",
        "        self.bert = AutoModelForSequenceClassification.from_pretrained(\n",
        "            self.name, num_labels=self.num_labels\n",
        "        )\n",
        "        \n",
        "    def forward(self, **kwargs):\n",
        "        pred = self.bert(**kwargs)\n",
        "        return pred.logits"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ac6234ba",
      "metadata": {
        "id": "ac6234ba"
      },
      "source": [
        "### Tokenizer"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "37f08038",
      "metadata": {
        "id": "37f08038"
      },
      "source": [
        "We make use of `HuggingfacePretrainedTokenizer`, which is a wrapper that skorch provides to use the tokenizers from Hugging Face. In this instance, we use a tokenizer that was pretrained in conjunction with BERT. The tokenizer is automatically downloaded if not already present. More on Hugging Face tokenizers can be found [here](https://huggingface.co/docs/tokenizers/index)."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "87512ddc-8e1e-4db1-b203-f26e6e8f3730",
      "metadata": {
        "id": "87512ddc-8e1e-4db1-b203-f26e6e8f3730"
      },
      "source": [
        "## Training"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "82675bea-7f6e-470b-a468-e803a94cfcf3",
      "metadata": {
        "id": "82675bea-7f6e-470b-a468-e803a94cfcf3"
      },
      "source": [
        "### Putting it all togther"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6a626328-b4a1-445c-9fad-5347263e889e",
      "metadata": {
        "id": "6a626328-b4a1-445c-9fad-5347263e889e"
      },
      "source": [
        "Now we can put together all the parts from above. There is nothing special going on here, we simply use an sklearn `Pipeline` to chain the `HuggingfacePretrainedTokenizer` and the neural net. Using skorch's `NeuralNetClassifier`, we make sure to pass the `BertModule` as the first argument and to set the number of labels based on `y_train`. The criterion is `CrossEntropyLoss` because we return the logits. Moreover, we make use of the learning rate schedule we defined above, and we add the `ProgressBar` callback to monitor our progress."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "id": "b798511a-c389-4a77-a77a-35ecc7c30e45",
      "metadata": {
        "id": "b798511a-c389-4a77-a77a-35ecc7c30e45"
      },
      "outputs": [],
      "source": [
        "pipeline = Pipeline([\n",
        "    ('tokenizer', HuggingfacePretrainedTokenizer(TOKENIZER)),\n",
        "    ('net', NeuralNetClassifier(\n",
        "        BertModule,\n",
        "        module__name=PRETRAINED_MODEL,\n",
        "        module__num_labels=len(set(y_train)),\n",
        "        optimizer=OPTMIZER,\n",
        "        lr=LR,\n",
        "        max_epochs=MAX_EPOCHS,\n",
        "        criterion=CRITERION,\n",
        "        batch_size=BATCH_SIZE,\n",
        "        iterator_train__shuffle=True,\n",
        "        device=DEVICE,\n",
        "        callbacks=[\n",
        "            LRScheduler(LambdaLR, lr_lambda=lr_schedule, step_every='batch'),\n",
        "            ProgressBar(),\n",
        "        ],\n",
        "    )),\n",
        "])"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "be19feb6-aeaf-43f7-8409-1baebe9a4c77",
      "metadata": {
        "id": "be19feb6-aeaf-43f7-8409-1baebe9a4c77"
      },
      "source": [
        "Since we are using skorch, we could now take this pipeline to run a grid search or other kind of hyper-parameter sweep to figure out the best hyper-parameters for this model. E.g. we could try out a different BERT model or a different `max_length`."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "744c78c9-6941-4de9-a43c-d4f154bdd13e",
      "metadata": {
        "id": "744c78c9-6941-4de9-a43c-d4f154bdd13e"
      },
      "source": [
        "### Fitting"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "id": "3159ab90-1c5e-411f-937a-5ac9addbedbc",
      "metadata": {
        "id": "3159ab90-1c5e-411f-937a-5ac9addbedbc"
      },
      "outputs": [],
      "source": [
        "torch.manual_seed(0)\n",
        "torch.cuda.manual_seed(0)\n",
        "torch.cuda.manual_seed_all(0)\n",
        "np.random.seed(0)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "id": "9e8d0e55-dd20-4f13-ac3f-4ff9b807cf33",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 860,
          "referenced_widgets": [
            "7737eb5f4b0147fca1d57fcff3b94310",
            "731489d30da74c22a941c2c40eac2f9e",
            "24c355db75d6402fb3d592b979efd97d",
            "97ede09e513643dbbe06a7a495d0e338",
            "5e48eaee9fec42439607252b57abf573",
            "2032954a84d64dc88d011ff804fc159e",
            "f47d171dfb1a49b0be475751052b9dd6",
            "9c74749827474c259b3de86723fa458a",
            "01aebe710a0448f298b8ee7cf168f158",
            "e0957d3f9d4f41499eb8fd103a858941",
            "d963b1ec70254a83aa9b7a6158cee474",
            "a351265b57e748df9bd4e3385bd82df1",
            "6a9891abe74043488b320c0721180e85",
            "8431059e4ad847279ab27f799eb47e7e",
            "3919403b5b4a422ab9a2f8b17004a8f8",
            "f67d5799925545e8beb0a5e138dc338d",
            "0d4b749541ba470ebdaaa8276cf63519",
            "e4d9c0c46410454988e56e6bd374d0dd",
            "05527a2144d046ddaf80ba6ea7d05da4",
            "cb45242338134614a9383354bb5e3d15",
            "cc5eeb0ad24c46818273525abe40754c",
            "257643eab22642a5b6d3930836dacdc9",
            "4e281b399814454cb89e687b3cba3a76",
            "77ef0a692bcc48018722b6d428a9523f",
            "641592e304264892a87f030243c7bfaa",
            "9d7acf7bb42c4cd68db0f906429243c4",
            "d3ea83287e134c0383d5ea9ecc3bd66d",
            "ae73e7099644414eb26fe669ae30edc2",
            "5e1d2c6e1dd34078a2a65f6fdedb2120",
            "7c954494c5b743318fe151cb212ec143",
            "7fd494a95eb442b2a07f490c8475685f",
            "c7f268cf4e144fd9a78b7390f7906b12",
            "42286df0e74b475d8701e8538e9f13ab",
            "1d9803aa96634b1fb07c06805fc258f5",
            "69d65608f1e94268b74e1992ece58318",
            "3889afc887b3425b8a4fe11984a15825",
            "c6e07ba089694552997dc3b85992ff14",
            "3dd66f23558e49afa6c2d552c9bdc886",
            "fc6f07f74486405e8d012bf4a114c3d3",
            "d685d3ad9c2c46648b07e8c8c66268ce",
            "45f636f84742412bb9aaad4edf9a87aa",
            "789b6dde0c844dcdacf197bb1133115b",
            "ba3a2b60fbb8426c95a48eb4c1581fc2",
            "de7b7750d12a4f9cbc12f94c31abd3d1",
            "ee66c812582e4892b5cb77c12af5befe",
            "77e92bb077da47299f9d40665a3e3ee3",
            "3198d6e184d647889319b76f5b03e44c",
            "bf9383b18e1f491984984189c75f6c60",
            "9a761ec322c4412aaeb6609e84e0cadc",
            "a69f39b12d754641b34316abd6c20467",
            "5a0ab8b67c7a4e22a2b812103809fba9",
            "17d766cdf6034912bc2b367f0e5510b4",
            "3457c65dafb440ec87f701c6a8f62f92",
            "59feac5daa344bcd8ab6389bcd1157e0",
            "b1936e7d43164d92b07a5ddb9d2bfba0"
          ]
        },
        "id": "9e8d0e55-dd20-4f13-ac3f-4ff9b807cf33",
        "outputId": "a9fa6552-a6ed-47b6-ef64-67e73c17c8cb"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/28.0 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "7737eb5f4b0147fca1d57fcff3b94310"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/483 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "a351265b57e748df9bd4e3385bd82df1"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/232k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "4e281b399814454cb89e687b3cba3a76"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/466k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "1d9803aa96634b1fb07c06805fc258f5"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading:   0%|          | 0.00/268M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "ee66c812582e4892b5cb77c12af5befe"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForSequenceClassification: ['vocab_layer_norm.weight', 'vocab_projector.bias', 'vocab_projector.weight', 'vocab_transform.weight', 'vocab_layer_norm.bias', 'vocab_transform.bias']\n",
            "- This IS expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['pre_classifier.weight', 'classifier.weight', 'pre_classifier.bias', 'classifier.bias']\n",
            "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
            "                                                                     "
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "  epoch    train_loss    valid_acc    valid_loss       dur\n",
            "-------  ------------  -----------  ------------  --------\n",
            "      1        \u001b[36m1.1628\u001b[0m       \u001b[32m0.8338\u001b[0m        \u001b[35m0.5839\u001b[0m  179.8571\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "                                                                   "
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "      2        \u001b[36m0.3709\u001b[0m       \u001b[32m0.8751\u001b[0m        \u001b[35m0.4214\u001b[0m  178.7779\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "                                                                     "
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "      3        \u001b[36m0.1523\u001b[0m       \u001b[32m0.8910\u001b[0m        \u001b[35m0.3945\u001b[0m  178.4507\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\r"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 7min 17s, sys: 1min 56s, total: 9min 14s\n",
            "Wall time: 9min 29s\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Pipeline(steps=[('tokenizer',\n",
              "                 HuggingfacePretrainedTokenizer(tokenizer='distilbert-base-uncased')),\n",
              "                ('net',\n",
              "                 <class 'skorch.classifier.NeuralNetClassifier'>[initialized](\n",
              "  module_=BertModule(\n",
              "    (bert): DistilBertForSequenceClassification(\n",
              "      (distilbert): DistilBertModel(\n",
              "        (embeddings): Embeddings(\n",
              "          (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
              "          (position_embeddin...\n",
              "                (lin1): Linear(in_features=768, out_features=3072, bias=True)\n",
              "                (lin2): Linear(in_features=3072, out_features=768, bias=True)\n",
              "                (activation): GELUActivation()\n",
              "              )\n",
              "              (output_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            )\n",
              "          )\n",
              "        )\n",
              "      )\n",
              "      (pre_classifier): Linear(in_features=768, out_features=768, bias=True)\n",
              "      (classifier): Linear(in_features=768, out_features=20, bias=True)\n",
              "      (dropout): Dropout(p=0.2, inplace=False)\n",
              "    )\n",
              "  ),\n",
              "))])"
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ],
      "source": [
        "%time pipeline.fit(X_train, y_train)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "4385305a-af65-4c19-b5a3-809f933e7794",
      "metadata": {
        "id": "4385305a-af65-4c19-b5a3-809f933e7794"
      },
      "source": [
        "### Evaluation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "id": "cc50f549-ce59-4440-a1e4-c5b30c4e8e4b",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cc50f549-ce59-4440-a1e4-c5b30c4e8e4b",
        "outputId": "404bc0ac-ed6e-4aa9-d063-7512bdf4266f"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 26.8 s, sys: 68.5 ms, total: 26.9 s\n",
            "Wall time: 24.6 s\n"
          ]
        }
      ],
      "source": [
        "%%time\n",
        "with torch.inference_mode():\n",
        "    y_pred = pipeline.predict(X_test)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "id": "eb04fc96-baa3-4592-a236-a022c96d1e2a",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eb04fc96-baa3-4592-a236-a022c96d1e2a",
        "outputId": "b14fb031-e3de-458e-b3f7-43b4efeb0aa4"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.8985507246376812"
            ]
          },
          "metadata": {},
          "execution_count": 16
        }
      ],
      "source": [
        "accuracy_score(y_test, y_pred)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "9799b81f-57d5-4239-ba7b-82cd2f76ed0d",
      "metadata": {
        "id": "9799b81f-57d5-4239-ba7b-82cd2f76ed0d"
      },
      "source": [
        "We can be happy with the results. We set ourselves the goal to reach or exceed 89% accuracy on the test set and we managed to do that."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "707bd2da-89ef-4684-a45c-7ccc44fb7596",
      "metadata": {
        "id": "707bd2da-89ef-4684-a45c-7ccc44fb7596"
      },
      "source": [
        "## Training with automatic mixed precision (AMP)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "a8376427-6d9d-421b-9078-3357f041726d",
      "metadata": {
        "id": "a8376427-6d9d-421b-9078-3357f041726d"
      },
      "source": [
        "For this to work, you need:\n",
        "- A GPU that is capable of mixed precision training\n",
        "- The [accelerate library](https://huggingface.co/docs/accelerate/index), which you can install as: `python -m pip install 'accelerate>=0.11'`.\n",
        "- skorch version 0.12 or installed from the current master branch (`python -m pip install git+https://github.com/skorch-dev/skorch.git`)\n",
        "\n",
        "Again, we assume that you're familiar with the general concept of mixed precision training. For more information on how skorch integrates with accelerate, please consult the [skorch docs](https://skorch.readthedocs.io/en/latest/user/huggingface.html#accelerate)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "id": "a6981f1d",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "a6981f1d",
        "outputId": "80f42c00-b23f-41f4-976d-8648239d8755"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "CompletedProcess(args=['python', '-m', 'pip', 'install', 'accelerate>=0.11'], returncode=0)"
            ]
          },
          "metadata": {},
          "execution_count": 17
        }
      ],
      "source": [
        "import subprocess\n",
        "\n",
        "subprocess.run(['python', '-m', 'pip', 'install', 'accelerate>=0.11'])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "id": "c47aa1a6-f466-4a2c-84ab-034e4d6bdbcd",
      "metadata": {
        "id": "c47aa1a6-f466-4a2c-84ab-034e4d6bdbcd"
      },
      "outputs": [],
      "source": [
        "from accelerate import Accelerator\n",
        "from skorch.hf import AccelerateMixin"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "id": "63dfa5a1-c75b-4b61-93b6-cedc203ce9a6",
      "metadata": {
        "id": "63dfa5a1-c75b-4b61-93b6-cedc203ce9a6"
      },
      "outputs": [],
      "source": [
        "class AcceleratedNet(AccelerateMixin, NeuralNetClassifier):\n",
        "    \"\"\"NeuralNetClassifier with accelerate support\"\"\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "id": "2eeac984-cc13-48e7-979f-9e9f208211df",
      "metadata": {
        "id": "2eeac984-cc13-48e7-979f-9e9f208211df"
      },
      "outputs": [],
      "source": [
        "accelerator = Accelerator(mixed_precision='fp16')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "id": "783df165-369c-448b-8867-12a8eb85daa2",
      "metadata": {
        "id": "783df165-369c-448b-8867-12a8eb85daa2"
      },
      "outputs": [],
      "source": [
        "pipeline2 = Pipeline([\n",
        "    ('tokenizer', HuggingfacePretrainedTokenizer(TOKENIZER)),\n",
        "    ('net', AcceleratedNet(                   # <= changed\n",
        "        BertModule,\n",
        "        accelerator=accelerator,              # <= changed\n",
        "        module__name=PRETRAINED_MODEL,\n",
        "        module__num_labels=len(set(y_train)),\n",
        "        optimizer=OPTMIZER,\n",
        "        lr=LR,\n",
        "        max_epochs=MAX_EPOCHS,\n",
        "        criterion=CRITERION,\n",
        "        batch_size=BATCH_SIZE,\n",
        "        iterator_train__shuffle=True,\n",
        "        # device=DEVICE,                      # <= changed\n",
        "        callbacks=[\n",
        "            LRScheduler(LambdaLR, lr_lambda=lr_schedule, step_every='batch'),\n",
        "            ProgressBar(),\n",
        "        ],\n",
        "    )),\n",
        "])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "id": "123f7967-e2d5-4bff-877d-cef395af9f14",
      "metadata": {
        "id": "123f7967-e2d5-4bff-877d-cef395af9f14"
      },
      "outputs": [],
      "source": [
        "torch.manual_seed(0)\n",
        "torch.cuda.manual_seed(0)\n",
        "torch.cuda.manual_seed_all(0)\n",
        "np.random.seed(0)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "id": "4f7e22db-eced-4360-ab2d-92ea9d1dbe79",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4f7e22db-eced-4360-ab2d-92ea9d1dbe79",
        "outputId": "2e39249e-7072-4de2-e7b6-87a3375c4f89"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForSequenceClassification: ['vocab_layer_norm.weight', 'vocab_projector.bias', 'vocab_projector.weight', 'vocab_transform.weight', 'vocab_layer_norm.bias', 'vocab_transform.bias']\n",
            "- This IS expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
            "Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['pre_classifier.weight', 'classifier.weight', 'pre_classifier.bias', 'classifier.bias']\n",
            "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "  epoch    train_loss    valid_acc    valid_loss      dur\n",
            "-------  ------------  -----------  ------------  -------\n",
            "      1        \u001b[36m1.1681\u001b[0m       \u001b[32m0.8397\u001b[0m        \u001b[35m0.5697\u001b[0m  79.4182\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": []
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "      2        \u001b[36m0.3479\u001b[0m       \u001b[32m0.8863\u001b[0m        \u001b[35m0.4011\u001b[0m  78.2318\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "                                                                     "
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "      3        \u001b[36m0.1438\u001b[0m       \u001b[32m0.8933\u001b[0m        \u001b[35m0.3853\u001b[0m  77.5366\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\r"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Pipeline(steps=[('tokenizer',\n",
              "                 HuggingfacePretrainedTokenizer(tokenizer='distilbert-base-uncased')),\n",
              "                ('net',\n",
              "                 <class '__main__.AcceleratedNet'>[initialized](\n",
              "  module_=BertModule(\n",
              "    (bert): DistilBertForSequenceClassification(\n",
              "      (distilbert): DistilBertModel(\n",
              "        (embeddings): Embeddings(\n",
              "          (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
              "          (position_embeddings): Embedding(...\n",
              "                (lin1): Linear(in_features=768, out_features=3072, bias=True)\n",
              "                (lin2): Linear(in_features=3072, out_features=768, bias=True)\n",
              "                (activation): GELUActivation()\n",
              "              )\n",
              "              (output_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
              "            )\n",
              "          )\n",
              "        )\n",
              "      )\n",
              "      (pre_classifier): Linear(in_features=768, out_features=768, bias=True)\n",
              "      (classifier): Linear(in_features=768, out_features=20, bias=True)\n",
              "      (dropout): Dropout(p=0.2, inplace=False)\n",
              "    )\n",
              "  ),\n",
              "))])"
            ]
          },
          "metadata": {},
          "execution_count": 23
        }
      ],
      "source": [
        "pipeline2.fit(X_train, y_train)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "id": "8fda6530-49b4-4423-b6fa-cf941dbf91db",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8fda6530-49b4-4423-b6fa-cf941dbf91db",
        "outputId": "c840cf7b-dc4a-4e12-a985-1ab5bc17794a"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 23.6 s, sys: 219 ms, total: 23.8 s\n",
            "Wall time: 24.7 s\n"
          ]
        }
      ],
      "source": [
        "%%time\n",
        "with torch.inference_mode():\n",
        "    y_pred = pipeline2.predict(X_test)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "id": "ab4ce9f3-85ab-4b6a-af6c-b7926320d4b0",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ab4ce9f3-85ab-4b6a-af6c-b7926320d4b0",
        "outputId": "ac94dfbd-6e3c-4195-f3be-921c5279bd57"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.9070342877341817"
            ]
          },
          "metadata": {},
          "execution_count": 25
        }
      ],
      "source": [
        "accuracy_score(y_test, y_pred)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "c4163616-3b62-46b5-976d-355378209fa7",
      "metadata": {
        "id": "c4163616-3b62-46b5-976d-355378209fa7"
      },
      "source": [
        "Using AMP, we could reduce our training and prediction time by half, while attaining the same scores."
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": []
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "base",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.13 (default, Mar 28 2022, 08:03:21) [MSC v.1916 64 bit (AMD64)]"
    },
    "vscode": {
      "interpreter": {
        "hash": "bd97b8bffa4d3737e84826bc3d37be3046061822757ce35137ab82ad4c5a2016"
      }
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "7737eb5f4b0147fca1d57fcff3b94310": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_731489d30da74c22a941c2c40eac2f9e",
              "IPY_MODEL_24c355db75d6402fb3d592b979efd97d",
              "IPY_MODEL_97ede09e513643dbbe06a7a495d0e338"
            ],
            "layout": "IPY_MODEL_5e48eaee9fec42439607252b57abf573"
          }
        },
        "731489d30da74c22a941c2c40eac2f9e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_2032954a84d64dc88d011ff804fc159e",
            "placeholder": "​",
            "style": "IPY_MODEL_f47d171dfb1a49b0be475751052b9dd6",
            "value": "Downloading: 100%"
          }
        },
        "24c355db75d6402fb3d592b979efd97d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_9c74749827474c259b3de86723fa458a",
            "max": 28,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_01aebe710a0448f298b8ee7cf168f158",
            "value": 28
          }
        },
        "97ede09e513643dbbe06a7a495d0e338": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e0957d3f9d4f41499eb8fd103a858941",
            "placeholder": "​",
            "style": "IPY_MODEL_d963b1ec70254a83aa9b7a6158cee474",
            "value": " 28.0/28.0 [00:00&lt;00:00, 1.38kB/s]"
          }
        },
        "5e48eaee9fec42439607252b57abf573": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2032954a84d64dc88d011ff804fc159e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f47d171dfb1a49b0be475751052b9dd6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "9c74749827474c259b3de86723fa458a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "01aebe710a0448f298b8ee7cf168f158": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "e0957d3f9d4f41499eb8fd103a858941": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d963b1ec70254a83aa9b7a6158cee474": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "a351265b57e748df9bd4e3385bd82df1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_6a9891abe74043488b320c0721180e85",
              "IPY_MODEL_8431059e4ad847279ab27f799eb47e7e",
              "IPY_MODEL_3919403b5b4a422ab9a2f8b17004a8f8"
            ],
            "layout": "IPY_MODEL_f67d5799925545e8beb0a5e138dc338d"
          }
        },
        "6a9891abe74043488b320c0721180e85": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_0d4b749541ba470ebdaaa8276cf63519",
            "placeholder": "​",
            "style": "IPY_MODEL_e4d9c0c46410454988e56e6bd374d0dd",
            "value": "Downloading: 100%"
          }
        },
        "8431059e4ad847279ab27f799eb47e7e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_05527a2144d046ddaf80ba6ea7d05da4",
            "max": 483,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_cb45242338134614a9383354bb5e3d15",
            "value": 483
          }
        },
        "3919403b5b4a422ab9a2f8b17004a8f8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_cc5eeb0ad24c46818273525abe40754c",
            "placeholder": "​",
            "style": "IPY_MODEL_257643eab22642a5b6d3930836dacdc9",
            "value": " 483/483 [00:00&lt;00:00, 27.6kB/s]"
          }
        },
        "f67d5799925545e8beb0a5e138dc338d": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0d4b749541ba470ebdaaa8276cf63519": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e4d9c0c46410454988e56e6bd374d0dd": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "05527a2144d046ddaf80ba6ea7d05da4": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "cb45242338134614a9383354bb5e3d15": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "cc5eeb0ad24c46818273525abe40754c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "257643eab22642a5b6d3930836dacdc9": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "4e281b399814454cb89e687b3cba3a76": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_77ef0a692bcc48018722b6d428a9523f",
              "IPY_MODEL_641592e304264892a87f030243c7bfaa",
              "IPY_MODEL_9d7acf7bb42c4cd68db0f906429243c4"
            ],
            "layout": "IPY_MODEL_d3ea83287e134c0383d5ea9ecc3bd66d"
          }
        },
        "77ef0a692bcc48018722b6d428a9523f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ae73e7099644414eb26fe669ae30edc2",
            "placeholder": "​",
            "style": "IPY_MODEL_5e1d2c6e1dd34078a2a65f6fdedb2120",
            "value": "Downloading: 100%"
          }
        },
        "641592e304264892a87f030243c7bfaa": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_7c954494c5b743318fe151cb212ec143",
            "max": 231508,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_7fd494a95eb442b2a07f490c8475685f",
            "value": 231508
          }
        },
        "9d7acf7bb42c4cd68db0f906429243c4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_c7f268cf4e144fd9a78b7390f7906b12",
            "placeholder": "​",
            "style": "IPY_MODEL_42286df0e74b475d8701e8538e9f13ab",
            "value": " 232k/232k [00:00&lt;00:00, 204kB/s]"
          }
        },
        "d3ea83287e134c0383d5ea9ecc3bd66d": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ae73e7099644414eb26fe669ae30edc2": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5e1d2c6e1dd34078a2a65f6fdedb2120": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "7c954494c5b743318fe151cb212ec143": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "7fd494a95eb442b2a07f490c8475685f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "c7f268cf4e144fd9a78b7390f7906b12": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "42286df0e74b475d8701e8538e9f13ab": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "1d9803aa96634b1fb07c06805fc258f5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_69d65608f1e94268b74e1992ece58318",
              "IPY_MODEL_3889afc887b3425b8a4fe11984a15825",
              "IPY_MODEL_c6e07ba089694552997dc3b85992ff14"
            ],
            "layout": "IPY_MODEL_3dd66f23558e49afa6c2d552c9bdc886"
          }
        },
        "69d65608f1e94268b74e1992ece58318": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_fc6f07f74486405e8d012bf4a114c3d3",
            "placeholder": "​",
            "style": "IPY_MODEL_d685d3ad9c2c46648b07e8c8c66268ce",
            "value": "Downloading: 100%"
          }
        },
        "3889afc887b3425b8a4fe11984a15825": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_45f636f84742412bb9aaad4edf9a87aa",
            "max": 466062,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_789b6dde0c844dcdacf197bb1133115b",
            "value": 466062
          }
        },
        "c6e07ba089694552997dc3b85992ff14": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ba3a2b60fbb8426c95a48eb4c1581fc2",
            "placeholder": "​",
            "style": "IPY_MODEL_de7b7750d12a4f9cbc12f94c31abd3d1",
            "value": " 466k/466k [00:01&lt;00:00, 500kB/s]"
          }
        },
        "3dd66f23558e49afa6c2d552c9bdc886": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "fc6f07f74486405e8d012bf4a114c3d3": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d685d3ad9c2c46648b07e8c8c66268ce": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "45f636f84742412bb9aaad4edf9a87aa": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "789b6dde0c844dcdacf197bb1133115b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "ba3a2b60fbb8426c95a48eb4c1581fc2": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "de7b7750d12a4f9cbc12f94c31abd3d1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "ee66c812582e4892b5cb77c12af5befe": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_77e92bb077da47299f9d40665a3e3ee3",
              "IPY_MODEL_3198d6e184d647889319b76f5b03e44c",
              "IPY_MODEL_bf9383b18e1f491984984189c75f6c60"
            ],
            "layout": "IPY_MODEL_9a761ec322c4412aaeb6609e84e0cadc"
          }
        },
        "77e92bb077da47299f9d40665a3e3ee3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_a69f39b12d754641b34316abd6c20467",
            "placeholder": "​",
            "style": "IPY_MODEL_5a0ab8b67c7a4e22a2b812103809fba9",
            "value": "Downloading: 100%"
          }
        },
        "3198d6e184d647889319b76f5b03e44c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_17d766cdf6034912bc2b367f0e5510b4",
            "max": 267967963,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_3457c65dafb440ec87f701c6a8f62f92",
            "value": 267967963
          }
        },
        "bf9383b18e1f491984984189c75f6c60": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_59feac5daa344bcd8ab6389bcd1157e0",
            "placeholder": "​",
            "style": "IPY_MODEL_b1936e7d43164d92b07a5ddb9d2bfba0",
            "value": " 268M/268M [00:03&lt;00:00, 75.0MB/s]"
          }
        },
        "9a761ec322c4412aaeb6609e84e0cadc": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "a69f39b12d754641b34316abd6c20467": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5a0ab8b67c7a4e22a2b812103809fba9": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "17d766cdf6034912bc2b367f0e5510b4": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3457c65dafb440ec87f701c6a8f62f92": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "59feac5daa344bcd8ab6389bcd1157e0": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b1936e7d43164d92b07a5ddb9d2bfba0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}