demo.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ae8ceee0"
      },
      "source": [
        "# 🤖💬❓nlp-uncertainty-zoo Demo\n",
        "\n",
        "This is a quick demo for the nlp-uncertainty-zoo, detailing how to jump in quickly with package. We will do this by training two different models on the Rotten Tomatoes sentiment analysis dataset, where want to classify where a movie review is positive or negative. \n",
        "\n",
        "For that purpose, we first start by importing all necessary packages as well as loading and preprocessing the dataset. Even though the first model we are using is LSTM-based, we will still use the BERT tokenizer here for the sake of simplicity."
      ],
      "id": "ae8ceee0"
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-5P3e2BLIayC",
        "outputId": "4ec8b511-0ccb-40a7-9f36-a41b30a06ef5"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Overwriting requirements.txt\n"
          ]
        }
      ],
      "source": [
        "%%writefile requirements.txt\n",
        "\n",
        "torch>=1.9.0\n",
        "numpy>=1.19.5\n",
        "wandb>=0.12.5\n",
        "scikit-learn>=0.24.1\n",
        "transformers>=4.5.1\n",
        "einops>=0.3.0\n",
        "datasets>=1.6.2\n",
        "tqdm>=4.49.0\n",
        "blitz-bayesian-pytorch>=0.2.7\n",
        "gpytorch>=1.5.0\n",
        "scipy>=1.5.4\n",
        "dill>=0.3.3\n",
        "joblib>=1.0.1\n",
        "alpaca-ml>=0.8.2\n",
        "frozendict==2.3.4\n",
        "protobuf==3.20.0"
      ],
      "id": "-5P3e2BLIayC"
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "F72lTwK8IhlA",
        "outputId": "60202e55-8557-43f2-f1b5-96eec35fa7d7"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: torch>=1.9.0 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 2)) (1.13.1+cu116)\n",
            "Requirement already satisfied: numpy>=1.19.5 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 3)) (1.22.4)\n",
            "Requirement already satisfied: wandb>=0.12.5 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 4)) (0.13.10)\n",
            "Requirement already satisfied: scikit-learn>=0.24.1 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 5)) (1.0.2)\n",
            "Requirement already satisfied: transformers>=4.5.1 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 6)) (4.26.1)\n",
            "Requirement already satisfied: einops>=0.3.0 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 7)) (0.6.0)\n",
            "Requirement already satisfied: datasets>=1.6.2 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 8)) (2.10.0)\n",
            "Requirement already satisfied: tqdm>=4.49.0 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 9)) (4.64.1)\n",
            "Requirement already satisfied: blitz-bayesian-pytorch>=0.2.7 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 10)) (0.2.8)\n",
            "Requirement already satisfied: gpytorch>=1.5.0 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 11)) (1.9.1)\n",
            "Requirement already satisfied: scipy>=1.5.4 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 12)) (1.7.3)\n",
            "Requirement already satisfied: dill>=0.3.3 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 13)) (0.3.6)\n",
            "Requirement already satisfied: joblib>=1.0.1 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 14)) (1.2.0)\n",
            "Requirement already satisfied: alpaca-ml>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 15)) (0.9.2)\n",
            "Requirement already satisfied: frozendict==2.3.4 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 16)) (2.3.4)\n",
            "Requirement already satisfied: protobuf==3.20.0 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 17)) (3.20.0)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.8/dist-packages (from torch>=1.9.0->-r requirements.txt (line 2)) (4.5.0)\n",
            "Requirement already satisfied: pathtools in /usr/local/lib/python3.8/dist-packages (from wandb>=0.12.5->-r requirements.txt (line 4)) (0.1.2)\n",
            "Requirement already satisfied: GitPython>=1.0.0 in /usr/local/lib/python3.8/dist-packages (from wandb>=0.12.5->-r requirements.txt (line 4)) (3.1.31)\n",
            "Requirement already satisfied: psutil>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from wandb>=0.12.5->-r requirements.txt (line 4)) (5.4.8)\n",
            "Requirement already satisfied: sentry-sdk>=1.0.0 in /usr/local/lib/python3.8/dist-packages (from wandb>=0.12.5->-r requirements.txt (line 4)) (1.15.0)\n",
            "Requirement already satisfied: setproctitle in /usr/local/lib/python3.8/dist-packages (from wandb>=0.12.5->-r requirements.txt (line 4)) (1.3.2)\n",
            "Requirement already satisfied: PyYAML in /usr/local/lib/python3.8/dist-packages (from wandb>=0.12.5->-r requirements.txt (line 4)) (6.0)\n",
            "Requirement already satisfied: requests<3,>=2.0.0 in /usr/local/lib/python3.8/dist-packages (from wandb>=0.12.5->-r requirements.txt (line 4)) (2.25.1)\n",
            "Requirement already satisfied: appdirs>=1.4.3 in /usr/local/lib/python3.8/dist-packages (from wandb>=0.12.5->-r requirements.txt (line 4)) (1.4.4)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.8/dist-packages (from wandb>=0.12.5->-r requirements.txt (line 4)) (57.4.0)\n",
            "Requirement already satisfied: Click!=8.0.0,>=7.0 in /usr/local/lib/python3.8/dist-packages (from wandb>=0.12.5->-r requirements.txt (line 4)) (8.1.3)\n",
            "Requirement already satisfied: docker-pycreds>=0.4.0 in /usr/local/lib/python3.8/dist-packages (from wandb>=0.12.5->-r requirements.txt (line 4)) (0.4.0)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.8/dist-packages (from scikit-learn>=0.24.1->-r requirements.txt (line 5)) (3.1.0)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.8/dist-packages (from transformers>=4.5.1->-r requirements.txt (line 6)) (23.0)\n",
            "Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /usr/local/lib/python3.8/dist-packages (from transformers>=4.5.1->-r requirements.txt (line 6)) (0.13.2)\n",
            "Requirement already satisfied: huggingface-hub<1.0,>=0.11.0 in /usr/local/lib/python3.8/dist-packages (from transformers>=4.5.1->-r requirements.txt (line 6)) (0.12.1)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.8/dist-packages (from transformers>=4.5.1->-r requirements.txt (line 6)) (2022.6.2)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.8/dist-packages (from transformers>=4.5.1->-r requirements.txt (line 6)) (3.9.0)\n",
            "Requirement already satisfied: aiohttp in /usr/local/lib/python3.8/dist-packages (from datasets>=1.6.2->-r requirements.txt (line 8)) (3.8.4)\n",
            "Requirement already satisfied: fsspec[http]>=2021.11.1 in /usr/local/lib/python3.8/dist-packages (from datasets>=1.6.2->-r requirements.txt (line 8)) (2023.1.0)\n",
            "Requirement already satisfied: xxhash in /usr/local/lib/python3.8/dist-packages (from datasets>=1.6.2->-r requirements.txt (line 8)) (3.2.0)\n",
            "Requirement already satisfied: responses<0.19 in /usr/local/lib/python3.8/dist-packages (from datasets>=1.6.2->-r requirements.txt (line 8)) (0.18.0)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.8/dist-packages (from datasets>=1.6.2->-r requirements.txt (line 8)) (1.3.5)\n",
            "Requirement already satisfied: pyarrow>=6.0.0 in /usr/local/lib/python3.8/dist-packages (from datasets>=1.6.2->-r requirements.txt (line 8)) (9.0.0)\n",
            "Requirement already satisfied: multiprocess in /usr/local/lib/python3.8/dist-packages (from datasets>=1.6.2->-r requirements.txt (line 8)) (0.70.14)\n",
            "Requirement already satisfied: torchvision>=0.5.0 in /usr/local/lib/python3.8/dist-packages (from blitz-bayesian-pytorch>=0.2.7->-r requirements.txt (line 10)) (0.14.1+cu116)\n",
            "Requirement already satisfied: pillow>=7.1 in /usr/local/lib/python3.8/dist-packages (from blitz-bayesian-pytorch>=0.2.7->-r requirements.txt (line 10)) (7.1.2)\n",
            "Requirement already satisfied: linear-operator>=0.2.0 in /usr/local/lib/python3.8/dist-packages (from gpytorch>=1.5.0->-r requirements.txt (line 11)) (0.3.0)\n",
            "Requirement already satisfied: pytest in /usr/local/lib/python3.8/dist-packages (from alpaca-ml>=0.8.2->-r requirements.txt (line 15)) (3.6.4)\n",
            "Requirement already satisfied: pylint in /usr/local/lib/python3.8/dist-packages (from alpaca-ml>=0.8.2->-r requirements.txt (line 15)) (2.16.2)\n",
            "Requirement already satisfied: dppy>=0.2.0 in /usr/local/lib/python3.8/dist-packages (from alpaca-ml>=0.8.2->-r requirements.txt (line 15)) (0.3.2)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.8/dist-packages (from alpaca-ml>=0.8.2->-r requirements.txt (line 15)) (3.5.3)\n",
            "Requirement already satisfied: pydocstyle in /usr/local/lib/python3.8/dist-packages (from alpaca-ml>=0.8.2->-r requirements.txt (line 15)) (6.3.0)\n",
            "Requirement already satisfied: pytest-html in /usr/local/lib/python3.8/dist-packages (from alpaca-ml>=0.8.2->-r requirements.txt (line 15)) (1.22.1)\n",
            "Requirement already satisfied: black in /usr/local/lib/python3.8/dist-packages (from alpaca-ml>=0.8.2->-r requirements.txt (line 15)) (23.1.0)\n",
            "Requirement already satisfied: six>=1.4.0 in /usr/local/lib/python3.8/dist-packages (from docker-pycreds>=0.4.0->wandb>=0.12.5->-r requirements.txt (line 4)) (1.15.0)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets>=1.6.2->-r requirements.txt (line 8)) (22.2.0)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets>=1.6.2->-r requirements.txt (line 8)) (1.3.1)\n",
            "Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets>=1.6.2->-r requirements.txt (line 8)) (3.0.1)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets>=1.6.2->-r requirements.txt (line 8)) (6.0.4)\n",
            "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets>=1.6.2->-r requirements.txt (line 8)) (1.8.2)\n",
            "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets>=1.6.2->-r requirements.txt (line 8)) (4.0.2)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets>=1.6.2->-r requirements.txt (line 8)) (1.3.3)\n",
            "Requirement already satisfied: gitdb<5,>=4.0.1 in /usr/local/lib/python3.8/dist-packages (from GitPython>=1.0.0->wandb>=0.12.5->-r requirements.txt (line 4)) (4.0.10)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests<3,>=2.0.0->wandb>=0.12.5->-r requirements.txt (line 4)) (2022.12.7)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests<3,>=2.0.0->wandb>=0.12.5->-r requirements.txt (line 4)) (2.10)\n",
            "Requirement already satisfied: chardet<5,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from requests<3,>=2.0.0->wandb>=0.12.5->-r requirements.txt (line 4)) (4.0.0)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests<3,>=2.0.0->wandb>=0.12.5->-r requirements.txt (line 4)) (1.26.14)\n",
            "Requirement already satisfied: pathspec>=0.9.0 in /usr/local/lib/python3.8/dist-packages (from black->alpaca-ml>=0.8.2->-r requirements.txt (line 15)) (0.11.0)\n",
            "Requirement already satisfied: platformdirs>=2 in /usr/local/lib/python3.8/dist-packages (from black->alpaca-ml>=0.8.2->-r requirements.txt (line 15)) (3.0.0)\n",
            "Requirement already satisfied: tomli>=1.1.0 in /usr/local/lib/python3.8/dist-packages (from black->alpaca-ml>=0.8.2->-r requirements.txt (line 15)) (2.0.1)\n",
            "Requirement already satisfied: mypy-extensions>=0.4.3 in /usr/local/lib/python3.8/dist-packages (from black->alpaca-ml>=0.8.2->-r requirements.txt (line 15)) (1.0.0)\n",
            "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.8/dist-packages (from matplotlib->alpaca-ml>=0.8.2->-r requirements.txt (line 15)) (4.38.0)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.8/dist-packages (from matplotlib->alpaca-ml>=0.8.2->-r requirements.txt (line 15)) (1.4.4)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.8/dist-packages (from matplotlib->alpaca-ml>=0.8.2->-r requirements.txt (line 15)) (0.11.0)\n",
            "Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.8/dist-packages (from matplotlib->alpaca-ml>=0.8.2->-r requirements.txt (line 15)) (2.8.2)\n",
            "Requirement already satisfied: pyparsing>=2.2.1 in /usr/local/lib/python3.8/dist-packages (from matplotlib->alpaca-ml>=0.8.2->-r requirements.txt (line 15)) (3.0.9)\n",
            "Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas->datasets>=1.6.2->-r requirements.txt (line 8)) (2022.7.1)\n",
            "Requirement already satisfied: snowballstemmer>=2.2.0 in /usr/local/lib/python3.8/dist-packages (from pydocstyle->alpaca-ml>=0.8.2->-r requirements.txt (line 15)) (2.2.0)\n",
            "Requirement already satisfied: astroid<=2.16.0-dev0,>=2.14.2 in /usr/local/lib/python3.8/dist-packages (from pylint->alpaca-ml>=0.8.2->-r requirements.txt (line 15)) (2.14.2)\n",
            "Requirement already satisfied: isort<6,>=4.2.5 in /usr/local/lib/python3.8/dist-packages (from pylint->alpaca-ml>=0.8.2->-r requirements.txt (line 15)) (5.12.0)\n",
            "Requirement already satisfied: tomlkit>=0.10.1 in /usr/local/lib/python3.8/dist-packages (from pylint->alpaca-ml>=0.8.2->-r requirements.txt (line 15)) (0.11.6)\n",
            "Requirement already satisfied: mccabe<0.8,>=0.6 in /usr/local/lib/python3.8/dist-packages (from pylint->alpaca-ml>=0.8.2->-r requirements.txt (line 15)) (0.7.0)\n",
            "Requirement already satisfied: pluggy<0.8,>=0.5 in /usr/local/lib/python3.8/dist-packages (from pytest->alpaca-ml>=0.8.2->-r requirements.txt (line 15)) (0.7.1)\n",
            "Requirement already satisfied: atomicwrites>=1.0 in /usr/local/lib/python3.8/dist-packages (from pytest->alpaca-ml>=0.8.2->-r requirements.txt (line 15)) (1.4.1)\n",
            "Requirement already satisfied: more-itertools>=4.0.0 in /usr/local/lib/python3.8/dist-packages (from pytest->alpaca-ml>=0.8.2->-r requirements.txt (line 15)) (9.0.0)\n",
            "Requirement already satisfied: py>=1.5.0 in /usr/local/lib/python3.8/dist-packages (from pytest->alpaca-ml>=0.8.2->-r requirements.txt (line 15)) (1.11.0)\n",
            "Requirement already satisfied: pytest-metadata in /usr/local/lib/python3.8/dist-packages (from pytest-html->alpaca-ml>=0.8.2->-r requirements.txt (line 15)) (2.0.4)\n",
            "Requirement already satisfied: wrapt<2,>=1.11 in /usr/local/lib/python3.8/dist-packages (from astroid<=2.16.0-dev0,>=2.14.2->pylint->alpaca-ml>=0.8.2->-r requirements.txt (line 15)) (1.14.1)\n",
            "Requirement already satisfied: lazy-object-proxy>=1.4.0 in /usr/local/lib/python3.8/dist-packages (from astroid<=2.16.0-dev0,>=2.14.2->pylint->alpaca-ml>=0.8.2->-r requirements.txt (line 15)) (1.9.0)\n",
            "Requirement already satisfied: smmap<6,>=3.0.1 in /usr/local/lib/python3.8/dist-packages (from gitdb<5,>=4.0.1->GitPython>=1.0.0->wandb>=0.12.5->-r requirements.txt (line 4)) (5.0.0)\n",
            "Found existing installation: nlp-uncertainty-zoo 0.9.0\n",
            "Uninstalling nlp-uncertainty-zoo-0.9.0:\n",
            "  Would remove:\n",
            "    /usr/local/lib/python3.8/dist-packages/nlp_uncertainty_zoo-0.9.0.dist-info/*\n",
            "    /usr/local/lib/python3.8/dist-packages/nlp_uncertainty_zoo/*\n",
            "Proceed (Y/n)? Y\n",
            "  Successfully uninstalled nlp-uncertainty-zoo-0.9.0\n",
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting git+https://github.com/Kaleidophon/nlp-uncertainty-zoo@demo\n",
            "  Cloning https://github.com/Kaleidophon/nlp-uncertainty-zoo (to revision demo) to /tmp/pip-req-build-i2h164nu\n",
            "  Running command git clone --filter=blob:none --quiet https://github.com/Kaleidophon/nlp-uncertainty-zoo /tmp/pip-req-build-i2h164nu\n",
            "  Running command git checkout -b demo --track origin/demo\n",
            "  Switched to a new branch 'demo'\n",
            "  Branch 'demo' set up to track remote branch 'demo' from 'origin'.\n",
            "  Resolved https://github.com/Kaleidophon/nlp-uncertainty-zoo to commit 76e9b0856b97b30c4947d3ef718e6bc58972f9f2\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: torch>=1.9.0 in /usr/local/lib/python3.8/dist-packages (from nlp-uncertainty-zoo==0.9.0) (1.13.1+cu116)\n",
            "Requirement already satisfied: numpy>=1.19.5 in /usr/local/lib/python3.8/dist-packages (from nlp-uncertainty-zoo==0.9.0) (1.22.4)\n",
            "Requirement already satisfied: wandb>=0.12.5 in /usr/local/lib/python3.8/dist-packages (from nlp-uncertainty-zoo==0.9.0) (0.13.10)\n",
            "Requirement already satisfied: scikit-learn>=0.24.1 in /usr/local/lib/python3.8/dist-packages (from nlp-uncertainty-zoo==0.9.0) (1.0.2)\n",
            "Requirement already satisfied: transformers>=4.5.1 in /usr/local/lib/python3.8/dist-packages (from nlp-uncertainty-zoo==0.9.0) (4.26.1)\n",
            "Requirement already satisfied: einops>=0.3.0 in /usr/local/lib/python3.8/dist-packages (from nlp-uncertainty-zoo==0.9.0) (0.6.0)\n",
            "Requirement already satisfied: datasets>=1.6.2 in /usr/local/lib/python3.8/dist-packages (from nlp-uncertainty-zoo==0.9.0) (2.10.0)\n",
            "Requirement already satisfied: tqdm>=4.49.0 in /usr/local/lib/python3.8/dist-packages (from nlp-uncertainty-zoo==0.9.0) (4.64.1)\n",
            "Requirement already satisfied: blitz-bayesian-pytorch>=0.2.7 in /usr/local/lib/python3.8/dist-packages (from nlp-uncertainty-zoo==0.9.0) (0.2.8)\n",
            "Requirement already satisfied: gpytorch>=1.5.0 in /usr/local/lib/python3.8/dist-packages (from nlp-uncertainty-zoo==0.9.0) (1.9.1)\n",
            "Requirement already satisfied: scipy>=1.5.4 in /usr/local/lib/python3.8/dist-packages (from nlp-uncertainty-zoo==0.9.0) (1.7.3)\n",
            "Requirement already satisfied: dill>=0.3.3 in /usr/local/lib/python3.8/dist-packages (from nlp-uncertainty-zoo==0.9.0) (0.3.6)\n",
            "Requirement already satisfied: joblib>=1.0.1 in /usr/local/lib/python3.8/dist-packages (from nlp-uncertainty-zoo==0.9.0) (1.2.0)\n",
            "Requirement already satisfied: alpaca-ml>=0.8.2 in /usr/local/lib/python3.8/dist-packages (from nlp-uncertainty-zoo==0.9.0) (0.9.2)\n",
            "Requirement already satisfied: frozendict==2.3.4 in /usr/local/lib/python3.8/dist-packages (from nlp-uncertainty-zoo==0.9.0) (2.3.4)\n",
            "Requirement already satisfied: protobuf==3.20.0 in /usr/local/lib/python3.8/dist-packages (from nlp-uncertainty-zoo==0.9.0) (3.20.0)\n",
            "Requirement already satisfied: pydocstyle in /usr/local/lib/python3.8/dist-packages (from alpaca-ml>=0.8.2->nlp-uncertainty-zoo==0.9.0) (6.3.0)\n",
            "Requirement already satisfied: pytest in /usr/local/lib/python3.8/dist-packages (from alpaca-ml>=0.8.2->nlp-uncertainty-zoo==0.9.0) (3.6.4)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.8/dist-packages (from alpaca-ml>=0.8.2->nlp-uncertainty-zoo==0.9.0) (2.25.1)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.8/dist-packages (from alpaca-ml>=0.8.2->nlp-uncertainty-zoo==0.9.0) (3.5.3)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.8/dist-packages (from alpaca-ml>=0.8.2->nlp-uncertainty-zoo==0.9.0) (1.3.5)\n",
            "Requirement already satisfied: pylint in /usr/local/lib/python3.8/dist-packages (from alpaca-ml>=0.8.2->nlp-uncertainty-zoo==0.9.0) (2.16.2)\n",
            "Requirement already satisfied: dppy>=0.2.0 in /usr/local/lib/python3.8/dist-packages (from alpaca-ml>=0.8.2->nlp-uncertainty-zoo==0.9.0) (0.3.2)\n",
            "Requirement already satisfied: black in /usr/local/lib/python3.8/dist-packages (from alpaca-ml>=0.8.2->nlp-uncertainty-zoo==0.9.0) (23.1.0)\n",
            "Requirement already satisfied: pytest-html in /usr/local/lib/python3.8/dist-packages (from alpaca-ml>=0.8.2->nlp-uncertainty-zoo==0.9.0) (1.22.1)\n",
            "Requirement already satisfied: torchvision>=0.5.0 in /usr/local/lib/python3.8/dist-packages (from blitz-bayesian-pytorch>=0.2.7->nlp-uncertainty-zoo==0.9.0) (0.14.1+cu116)\n",
            "Requirement already satisfied: pillow>=7.1 in /usr/local/lib/python3.8/dist-packages (from blitz-bayesian-pytorch>=0.2.7->nlp-uncertainty-zoo==0.9.0) (7.1.2)\n",
            "Requirement already satisfied: xxhash in /usr/local/lib/python3.8/dist-packages (from datasets>=1.6.2->nlp-uncertainty-zoo==0.9.0) (3.2.0)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from datasets>=1.6.2->nlp-uncertainty-zoo==0.9.0) (23.0)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.8/dist-packages (from datasets>=1.6.2->nlp-uncertainty-zoo==0.9.0) (6.0)\n",
            "Requirement already satisfied: multiprocess in /usr/local/lib/python3.8/dist-packages (from datasets>=1.6.2->nlp-uncertainty-zoo==0.9.0) (0.70.14)\n",
            "Requirement already satisfied: huggingface-hub<1.0.0,>=0.2.0 in /usr/local/lib/python3.8/dist-packages (from datasets>=1.6.2->nlp-uncertainty-zoo==0.9.0) (0.12.1)\n",
            "Requirement already satisfied: fsspec[http]>=2021.11.1 in /usr/local/lib/python3.8/dist-packages (from datasets>=1.6.2->nlp-uncertainty-zoo==0.9.0) (2023.1.0)\n",
            "Requirement already satisfied: responses<0.19 in /usr/local/lib/python3.8/dist-packages (from datasets>=1.6.2->nlp-uncertainty-zoo==0.9.0) (0.18.0)\n",
            "Requirement already satisfied: aiohttp in /usr/local/lib/python3.8/dist-packages (from datasets>=1.6.2->nlp-uncertainty-zoo==0.9.0) (3.8.4)\n",
            "Requirement already satisfied: pyarrow>=6.0.0 in /usr/local/lib/python3.8/dist-packages (from datasets>=1.6.2->nlp-uncertainty-zoo==0.9.0) (9.0.0)\n",
            "Requirement already satisfied: linear-operator>=0.2.0 in /usr/local/lib/python3.8/dist-packages (from gpytorch>=1.5.0->nlp-uncertainty-zoo==0.9.0) (0.3.0)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.8/dist-packages (from scikit-learn>=0.24.1->nlp-uncertainty-zoo==0.9.0) (3.1.0)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.8/dist-packages (from torch>=1.9.0->nlp-uncertainty-zoo==0.9.0) (4.5.0)\n",
            "Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /usr/local/lib/python3.8/dist-packages (from transformers>=4.5.1->nlp-uncertainty-zoo==0.9.0) (0.13.2)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.8/dist-packages (from transformers>=4.5.1->nlp-uncertainty-zoo==0.9.0) (2022.6.2)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.8/dist-packages (from transformers>=4.5.1->nlp-uncertainty-zoo==0.9.0) (3.9.0)\n",
            "Requirement already satisfied: GitPython>=1.0.0 in /usr/local/lib/python3.8/dist-packages (from wandb>=0.12.5->nlp-uncertainty-zoo==0.9.0) (3.1.31)\n",
            "Requirement already satisfied: pathtools in /usr/local/lib/python3.8/dist-packages (from wandb>=0.12.5->nlp-uncertainty-zoo==0.9.0) (0.1.2)\n",
            "Requirement already satisfied: setproctitle in /usr/local/lib/python3.8/dist-packages (from wandb>=0.12.5->nlp-uncertainty-zoo==0.9.0) (1.3.2)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.8/dist-packages (from wandb>=0.12.5->nlp-uncertainty-zoo==0.9.0) (57.4.0)\n",
            "Requirement already satisfied: docker-pycreds>=0.4.0 in /usr/local/lib/python3.8/dist-packages (from wandb>=0.12.5->nlp-uncertainty-zoo==0.9.0) (0.4.0)\n",
            "Requirement already satisfied: psutil>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from wandb>=0.12.5->nlp-uncertainty-zoo==0.9.0) (5.4.8)\n",
            "Requirement already satisfied: sentry-sdk>=1.0.0 in /usr/local/lib/python3.8/dist-packages (from wandb>=0.12.5->nlp-uncertainty-zoo==0.9.0) (1.15.0)\n",
            "Requirement already satisfied: Click!=8.0.0,>=7.0 in /usr/local/lib/python3.8/dist-packages (from wandb>=0.12.5->nlp-uncertainty-zoo==0.9.0) (8.1.3)\n",
            "Requirement already satisfied: appdirs>=1.4.3 in /usr/local/lib/python3.8/dist-packages (from wandb>=0.12.5->nlp-uncertainty-zoo==0.9.0) (1.4.4)\n",
            "Requirement already satisfied: six>=1.4.0 in /usr/local/lib/python3.8/dist-packages (from docker-pycreds>=0.4.0->wandb>=0.12.5->nlp-uncertainty-zoo==0.9.0) (1.15.0)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets>=1.6.2->nlp-uncertainty-zoo==0.9.0) (22.2.0)\n",
            "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets>=1.6.2->nlp-uncertainty-zoo==0.9.0) (4.0.2)\n",
            "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets>=1.6.2->nlp-uncertainty-zoo==0.9.0) (1.8.2)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets>=1.6.2->nlp-uncertainty-zoo==0.9.0) (1.3.1)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets>=1.6.2->nlp-uncertainty-zoo==0.9.0) (6.0.4)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets>=1.6.2->nlp-uncertainty-zoo==0.9.0) (1.3.3)\n",
            "Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /usr/local/lib/python3.8/dist-packages (from aiohttp->datasets>=1.6.2->nlp-uncertainty-zoo==0.9.0) (3.0.1)\n",
            "Requirement already satisfied: gitdb<5,>=4.0.1 in /usr/local/lib/python3.8/dist-packages (from GitPython>=1.0.0->wandb>=0.12.5->nlp-uncertainty-zoo==0.9.0) (4.0.10)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests->alpaca-ml>=0.8.2->nlp-uncertainty-zoo==0.9.0) (1.26.14)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests->alpaca-ml>=0.8.2->nlp-uncertainty-zoo==0.9.0) (2022.12.7)\n",
            "Requirement already satisfied: chardet<5,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from requests->alpaca-ml>=0.8.2->nlp-uncertainty-zoo==0.9.0) (4.0.0)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests->alpaca-ml>=0.8.2->nlp-uncertainty-zoo==0.9.0) (2.10)\n",
            "Requirement already satisfied: pathspec>=0.9.0 in /usr/local/lib/python3.8/dist-packages (from black->alpaca-ml>=0.8.2->nlp-uncertainty-zoo==0.9.0) (0.11.0)\n",
            "Requirement already satisfied: mypy-extensions>=0.4.3 in /usr/local/lib/python3.8/dist-packages (from black->alpaca-ml>=0.8.2->nlp-uncertainty-zoo==0.9.0) (1.0.0)\n",
            "Requirement already satisfied: platformdirs>=2 in /usr/local/lib/python3.8/dist-packages (from black->alpaca-ml>=0.8.2->nlp-uncertainty-zoo==0.9.0) (3.0.0)\n",
            "Requirement already satisfied: tomli>=1.1.0 in /usr/local/lib/python3.8/dist-packages (from black->alpaca-ml>=0.8.2->nlp-uncertainty-zoo==0.9.0) (2.0.1)\n",
            "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.8/dist-packages (from matplotlib->alpaca-ml>=0.8.2->nlp-uncertainty-zoo==0.9.0) (4.38.0)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.8/dist-packages (from matplotlib->alpaca-ml>=0.8.2->nlp-uncertainty-zoo==0.9.0) (1.4.4)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.8/dist-packages (from matplotlib->alpaca-ml>=0.8.2->nlp-uncertainty-zoo==0.9.0) (0.11.0)\n",
            "Requirement already satisfied: pyparsing>=2.2.1 in /usr/local/lib/python3.8/dist-packages (from matplotlib->alpaca-ml>=0.8.2->nlp-uncertainty-zoo==0.9.0) (3.0.9)\n",
            "Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.8/dist-packages (from matplotlib->alpaca-ml>=0.8.2->nlp-uncertainty-zoo==0.9.0) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.8/dist-packages (from pandas->alpaca-ml>=0.8.2->nlp-uncertainty-zoo==0.9.0) (2022.7.1)\n",
            "Requirement already satisfied: snowballstemmer>=2.2.0 in /usr/local/lib/python3.8/dist-packages (from pydocstyle->alpaca-ml>=0.8.2->nlp-uncertainty-zoo==0.9.0) (2.2.0)\n",
            "Requirement already satisfied: tomlkit>=0.10.1 in /usr/local/lib/python3.8/dist-packages (from pylint->alpaca-ml>=0.8.2->nlp-uncertainty-zoo==0.9.0) (0.11.6)\n",
            "Requirement already satisfied: isort<6,>=4.2.5 in /usr/local/lib/python3.8/dist-packages (from pylint->alpaca-ml>=0.8.2->nlp-uncertainty-zoo==0.9.0) (5.12.0)\n",
            "Requirement already satisfied: astroid<=2.16.0-dev0,>=2.14.2 in /usr/local/lib/python3.8/dist-packages (from pylint->alpaca-ml>=0.8.2->nlp-uncertainty-zoo==0.9.0) (2.14.2)\n",
            "Requirement already satisfied: mccabe<0.8,>=0.6 in /usr/local/lib/python3.8/dist-packages (from pylint->alpaca-ml>=0.8.2->nlp-uncertainty-zoo==0.9.0) (0.7.0)\n",
            "Requirement already satisfied: py>=1.5.0 in /usr/local/lib/python3.8/dist-packages (from pytest->alpaca-ml>=0.8.2->nlp-uncertainty-zoo==0.9.0) (1.11.0)\n",
            "Requirement already satisfied: more-itertools>=4.0.0 in /usr/local/lib/python3.8/dist-packages (from pytest->alpaca-ml>=0.8.2->nlp-uncertainty-zoo==0.9.0) (9.0.0)\n",
            "Requirement already satisfied: atomicwrites>=1.0 in /usr/local/lib/python3.8/dist-packages (from pytest->alpaca-ml>=0.8.2->nlp-uncertainty-zoo==0.9.0) (1.4.1)\n",
            "Requirement already satisfied: pluggy<0.8,>=0.5 in /usr/local/lib/python3.8/dist-packages (from pytest->alpaca-ml>=0.8.2->nlp-uncertainty-zoo==0.9.0) (0.7.1)\n",
            "Requirement already satisfied: pytest-metadata in /usr/local/lib/python3.8/dist-packages (from pytest-html->alpaca-ml>=0.8.2->nlp-uncertainty-zoo==0.9.0) (2.0.4)\n",
            "Requirement already satisfied: lazy-object-proxy>=1.4.0 in /usr/local/lib/python3.8/dist-packages (from astroid<=2.16.0-dev0,>=2.14.2->pylint->alpaca-ml>=0.8.2->nlp-uncertainty-zoo==0.9.0) (1.9.0)\n",
            "Requirement already satisfied: wrapt<2,>=1.11 in /usr/local/lib/python3.8/dist-packages (from astroid<=2.16.0-dev0,>=2.14.2->pylint->alpaca-ml>=0.8.2->nlp-uncertainty-zoo==0.9.0) (1.14.1)\n",
            "Requirement already satisfied: smmap<6,>=3.0.1 in /usr/local/lib/python3.8/dist-packages (from gitdb<5,>=4.0.1->GitPython>=1.0.0->wandb>=0.12.5->nlp-uncertainty-zoo==0.9.0) (5.0.0)\n",
            "Building wheels for collected packages: nlp-uncertainty-zoo\n",
            "  Building wheel for nlp-uncertainty-zoo (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for nlp-uncertainty-zoo: filename=nlp_uncertainty_zoo-0.9.0-py3-none-any.whl size=76116 sha256=fa50ece8f69fa73e52785cf42c6eba99aed318785530ed1991be3abce6b1b2ab\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-pqntnkc5/wheels/e5/89/c4/63092c98950e7924cd2142ba3cf5b8bc766b6f41d438de30e8\n",
            "Successfully built nlp-uncertainty-zoo\n",
            "Installing collected packages: nlp-uncertainty-zoo\n",
            "Successfully installed nlp-uncertainty-zoo-0.9.0\n"
          ]
        }
      ],
      "source": [
        "!pip install -r requirements.txt\n",
        "!pip install git+https://github.com/Kaleidophon/nlp-uncertainty-zoo\n"
      ],
      "id": "F72lTwK8IhlA"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a1e8f0d1"
      },
      "source": [
        "## Loading the dataset & preprocessing"
      ],
      "id": "a1e8f0d1"
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "bf435d06"
      },
      "outputs": [],
      "source": [
        "import random\n",
        "from string import ascii_lowercase\n",
        "\n",
        "from datasets import load_dataset, ReadInstruction\n",
        "import numpy as np\n",
        "import torch\n",
        "from torch.utils.data import DataLoader\n",
        "from transformers import BertTokenizer, BertModel\n",
        "from nlp_uncertainty_zoo.models import LSTMEnsemble, VariationalBert  # We will test these two models in this demo!\n",
        "\n",
        "# CONST\n",
        "BATCH_SIZE = 16"
      ],
      "id": "bf435d06"
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "1f61486f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 168,
          "referenced_widgets": [
            "dc0bf6ba55cb4908a492f845dbfd7d71",
            "fca4e1a0c4274428ae247c4690430a5b",
            "958614ddda6547b682f2e111d21b2350",
            "f5ae790b4ba949e6af819aa71102bed6",
            "88cae46fe0b241d4b20223ba674391a4",
            "f8c92cfafa1742c18ebb6b0d9e646284",
            "b5f515ea68b3424fa95a0e1258e54b41",
            "bebca6f8fe6048dda13e13bf993709f1",
            "6b8563ada68f4945a7adffce7713880d",
            "3935bb98a26647e48941e547ed5c0779",
            "b402d95e1bf841dea521a1404739e415",
            "de6e9320ced14aaaa10f81fe9c501fec",
            "c9a9fab793d34ad49cbf2908350e4d04",
            "601ce7d9813a40a88a2ef58e99345637",
            "6b5d1f908b6b48f9b4baa16b1d285cfe",
            "6315a9343de9468cb06689d42dcc3b5a",
            "9a23e58d21494b66886f7b5a84a604b6",
            "82407042963b489cb513653dc79819cf",
            "02be4776a3fb4f229fa517ee6ec99ceb",
            "fbe909a8434b471eab77addd48f61b5c",
            "4072731cf4be4ab198abe60bfb2c5ea1",
            "9343a66615774849b2e397ad91e87134",
            "dea997b3e5804bc883454734633455ed",
            "485f592d05354006b4d7d24b6e9a5ad3",
            "27f3dc7712664547ba51bc1058791571",
            "985286ce89ad406590d1c7bf594ad17b",
            "3e02992cb3614a1b832028130f95c1d4",
            "6c3767f6a1264f2ab86e52b53aab6840",
            "5fae15cddeef466489d6298408bd0308",
            "256342f0909d47d8b4d7b88fb5097e81",
            "f57f6ab74522424a8d8183698cd9bf84",
            "1a3aaad7c01d43e6a770824e198b0fa1",
            "40d1fd61b2024defa993a9fdc7f2cc64",
            "7bf14949226544e49d39c4744b3ef6b8",
            "dc4a4bc2dbb34b91a42cfbe08f5ef149",
            "2c7338f42527405c9f4aceb609107e69",
            "cd227304c23f46a080147864356702cc",
            "186f5742fd6d4b5b98932708d49a3af1",
            "11cc1b8150964e0b90f47496d535b5f8",
            "1b2d49d63d8a48a4a674310d44ef28e3",
            "ca05a94bcf2e4fb88df94e6a1fda74ee",
            "0da955052cf44f2c9cfea4dfc27cb8c1",
            "4d74cbb284ff4b179ef04d0aa2ea3047",
            "cd2a422de522448a948b29d90b8264d1",
            "3c6e475832c6469c911a2c05c0500424",
            "57e4798293824e418af084e0c5351e26",
            "0effe1422f634fa58c4c0c5b5e59e3b5",
            "55c845e610ff4e11a41b8a54979f0fc1",
            "1c62eccf29764231a20c44f5fe0da5dc",
            "369ec0ff10f043d08a11833a291dcf67",
            "c405831192504dae990029ccef180e85",
            "f71beb50b901486c8b67c1cc48575a8c",
            "a086444e398a46af9c77e2c26a18306f",
            "2b98db052a2f4fb7a6a695ddeb312360",
            "2f5ae230ff5745969c7723d1dcbd38d4",
            "f5ba6a38ff4f4e3aa274a06d374e0d88",
            "1d44e548de84443da34888fb24c7412e",
            "b1e3d682c7434e6f9f9f515a269ca32c",
            "d7dc4d22a77847d28555fadf94199e22",
            "f9a9456a47894a0691b32450ea42947e",
            "3728f25b38ec45aea4b0846b9352df34",
            "165108ca8e8f4f6cb68ec2cc9374da63",
            "6bc3814a734640a591615b2ba1ef2380",
            "fac8479b0b7a439b854c681feefb8c07",
            "d20e2ffcca174ab091b258aadd27968b",
            "61b0c52c76594e51b070fd3de1a3f6e4"
          ]
        },
        "outputId": "cd6b6a52-d47d-4db9-d00f-ab9c238a49cc"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:datasets.builder:Found cached dataset rotten_tomatoes (/root/.cache/huggingface/datasets/rotten_tomatoes/default/1.0.0/40d411e45a6ce3484deed7cc15b82a53dad9a72aafd9f86f8f227134bec5ca46)\n",
            "WARNING:datasets.builder:Found cached dataset rotten_tomatoes (/root/.cache/huggingface/datasets/rotten_tomatoes/default/1.0.0/40d411e45a6ce3484deed7cc15b82a53dad9a72aafd9f86f8f227134bec5ca46)\n",
            "WARNING:datasets.builder:Found cached dataset twitter-sentiment-analysis (/root/.cache/huggingface/datasets/carblacac___twitter-sentiment-analysis/default/1.0.0/cd65e23e456de6a4f7264e305380b0ffe804d6f5bfd361c0ec0f68d8d1fab95b)\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading (…)solve/main/vocab.txt:   0%|          | 0.00/232k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "dc0bf6ba55cb4908a492f845dbfd7d71"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading (…)okenizer_config.json:   0%|          | 0.00/28.0 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "de6e9320ced14aaaa10f81fe9c501fec"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading (…)lve/main/config.json:   0%|          | 0.00/570 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "dea997b3e5804bc883454734633455ed"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Map:   0%|          | 0/100 [00:00<?, ? examples/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "7bf14949226544e49d39c4744b3ef6b8"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Map:   0%|          | 0/25 [00:00<?, ? examples/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "3c6e475832c6469c911a2c05c0500424"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Map:   0%|          | 0/25 [00:00<?, ? examples/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "f5ba6a38ff4f4e3aa274a06d374e0d88"
            }
          },
          "metadata": {}
        }
      ],
      "source": [
        "def preprocess_with(tokenizer):\n",
        "    def preprocess(input_):\n",
        "        return tokenizer(\n",
        "            input_[\"text\"],\n",
        "            truncation=True,\n",
        "            padding=\"max_length\",\n",
        "            max_length=50\n",
        "        )\n",
        "    \n",
        "    return preprocess\n",
        "\n",
        "# We only use a subset of the data here for demonstration purposes\n",
        "train_split = load_dataset(\"rotten_tomatoes\", split='train[:100]')\n",
        "test_split = load_dataset(\"rotten_tomatoes\", split='test[:25]')\n",
        "ood_test_split = load_dataset(\"carblacac/twitter-sentiment-analysis\", split='test[:25]')\n",
        "tokenizer = BertTokenizer.from_pretrained(\"bert-base-uncased\")\n",
        "\n",
        "\n",
        "train_split = train_split.map(preprocess_with(tokenizer), batched=True)\n",
        "train_split.set_format(type=\"torch\", columns=[\"input_ids\", \"attention_mask\", \"label\"])\n",
        "train_split = train_split.rename_column(\"label\", \"labels\")\n",
        "\n",
        "test_split = test_split.map(preprocess_with(tokenizer), batched=True)\n",
        "test_split.set_format(type=\"torch\", columns=[\"input_ids\", \"attention_mask\", \"label\"])\n",
        "test_split = test_split.rename_column(\"label\", \"labels\")\n",
        "\n",
        "ood_test_split = ood_test_split.map(preprocess_with(tokenizer), batched=True)\n",
        "ood_test_split = ood_test_split.rename_column(\"feeling\", \"labels\")\n",
        "ood_test_split.set_format(type=\"torch\", columns=[\"input_ids\", \"attention_mask\", \"labels\"])\n",
        "\n",
        "\n",
        "train_loader = DataLoader(train_split, batch_size=BATCH_SIZE)\n",
        "test_loader = DataLoader(test_split, batch_size=BATCH_SIZE)\n",
        "ood_test_loader = DataLoader(ood_test_split, batch_size=BATCH_SIZE)"
      ],
      "id": "1f61486f"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4751bac9"
      },
      "source": [
        "## Training"
      ],
      "id": "4751bac9"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "18aec99c"
      },
      "source": [
        "We now start by training an ensemble of LSTMs. Due to fact that all members of an ensemble are randomly initialized, models tend to converge to different solutions, making the ensemble very robust to unseen data points (see paper by Lakshminarayanan et al., 2017). This is also a very useful property for uncertainty quantification, as we will see later. "
      ],
      "id": "18aec99c"
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "752e25e3"
      },
      "outputs": [],
      "source": [
        "SEED = 1234\n",
        "np.random.seed(SEED)\n",
        "torch.manual_seed(SEED)\n",
        "\n",
        "vocab_size = len(tokenizer.vocab)"
      ],
      "id": "752e25e3"
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "1121b0e4"
      },
      "outputs": [],
      "source": [
        "ensemble = LSTMEnsemble(vocab_size=vocab_size, output_size=2, ensemble_size=5)"
      ],
      "id": "1121b0e4"
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1f79190e",
        "outputId": "4f409ec3-d6bb-4534-a4ec-f7f9ec071a8f"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Step 7: Train Loss 1.9173: 100%|██████████| 7/7 [03:43<00:00, 31.90s/it]\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'model_name': 'lstm_ensemble',\n",
              " 'train_loss': 1.9173227548599243,\n",
              " 'best_val_loss': inf}"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ],
      "source": [
        "ensemble.fit(train_loader, num_training_steps=7)"
      ],
      "id": "1f79190e"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b3651728"
      },
      "source": [
        "Next up, we will fine-tune a BERT model. For uncertainty quantification, we will use Monte Carlo Dropout (Gal & Ghahramani, 2016a,b; Xiao et al., 2020): By using multiple different dropout masks during inference, we can create different predictions for the same data point. "
      ],
      "id": "b3651728"
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 124,
          "referenced_widgets": [
            "72a3bbfb710d417192a4e31a99cf49e5",
            "d3def39d079f4cd8b68c3bb388d06884",
            "b5d3bab4339242be9c70f24cd21bc2bc",
            "5d300bed22be405b9170920885a9447f",
            "a6f9b42637dc417a92533a577278c03a",
            "f7f1fb0dce524099913abb524ae10308",
            "27b43b848c254cfc8fbd70dfd4a92d2b",
            "c6b2f20955d24aa885c880345acd0cfb",
            "5f8835b2096f4b849c0f100d9e18fb9b",
            "8cc8c4f84c974be4a07cc73b5a79b3d4",
            "0a4b3f5002664de99fe30fc4a0d159a6"
          ]
        },
        "id": "95d41f7d",
        "outputId": "81583b69-c0d1-4b72-ead7-0cb18ac035b5"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading (…)\"pytorch_model.bin\";:   0%|          | 0.00/440M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "72a3bbfb710d417192a4e31a99cf49e5"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.seq_relationship.bias', 'cls.seq_relationship.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.bias']\n",
            "- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
            "- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
          ]
        }
      ],
      "source": [
        "variational_bert = VariationalBert(\n",
        "    bert_name=\"bert-base-uncased\", \n",
        "    output_size=2,\n",
        ")"
      ],
      "id": "95d41f7d"
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "8d16390f",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b585d91b-ca22-4c84-9681-ac6c68ce3c67"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Step 2: Train Loss 0.0897: 100%|██████████| 2/2 [00:21<00:00, 10.59s/it]\n"
          ]
        }
      ],
      "source": [
        "variational_bert.fit(train_loader, num_training_steps=2)"
      ],
      "id": "8d16390f"
    },
    {
      "cell_type": "markdown",
      "source": [
        "Note: We set the `num_training_steps` purposefully low here to slightly undertrain our models. This makes the demo easier to run and leads to more informative uncertainty values in the rest of the demo - otherwise these models would fit our dataset perfectly, and that would be a bit boring."
      ],
      "metadata": {
        "id": "Wd7cUi01k99H"
      },
      "id": "Wd7cUi01k99H"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "88d6cafc"
      },
      "source": [
        "## Evaluating task performance & calibration"
      ],
      "id": "88d6cafc"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "34c2b303"
      },
      "source": [
        "Before we continue, let us first evaluate the models to reassure ourselves that the training was successful:"
      ],
      "id": "34c2b303"
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "c65a8c50"
      },
      "outputs": [],
      "source": [
        "from nlp_uncertainty_zoo.utils.task_eval import evaluate_task"
      ],
      "id": "c65a8c50"
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "d5fe480a",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "84b78155-41b6-4371-c325-d1000fbe1fa4"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Evaluating batch 2/2...: 100%|██████████| 2/2 [00:05<00:00,  2.70s/it]\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "defaultdict(float, {'accuracy': 1.0, 'macro_f1_scores': 1.0})"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ],
      "source": [
        "evaluate_task(ensemble, eval_split=test_loader)"
      ],
      "id": "d5fe480a"
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "2FzBImfoT6z6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c107a754-a66f-4c08-9d27-41a1a7186969"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Evaluating batch 2/2...: 100%|██████████| 2/2 [00:44<00:00, 22.21s/it]\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "defaultdict(float, {'accuracy': 1.0, 'macro_f1_scores': 1.0})"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ],
      "source": [
        "evaluate_task(variational_bert, eval_split=test_loader)"
      ],
      "id": "2FzBImfoT6z6"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7a1be622"
      },
      "source": [
        "We can also evaluate to what extend the probability of a predicted class actually corresponds to the chance of the model actually predicting the correct class, also called *calibration* (Guo et al., 2017). One way to evaluate this propery is the expected calibration error (ECE): By binning predictions with similar confidence scores, we can evaluate if the mean confidence per bin corresponds to the accuracy on the binned samples. Another approach is evaluation using *prediction sets* (Kompa et al., 2020). The idea here is to sort predictings descendingly and add classes to a set until a certain amount of probability mass - for instance 90 % in the example below - is reached. If the model is well calibrated, these prediction sets should be small and contain the correct class (on average). Using the functions implemented in the package, we evaluate these properties below: "
      ],
      "id": "7a1be622"
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "id": "0748a7cb"
      },
      "outputs": [],
      "source": [
        "from nlp_uncertainty_zoo.utils.calibration_eval import evaluate_calibration"
      ],
      "id": "0748a7cb"
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "id": "Z3py0MFgUFRg",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "80089980-2a6e-4da9-ff70-25d081267133"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Evaluating batch 2/2...: 100%|██████████| 2/2 [00:05<00:00,  2.62s/it]\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "defaultdict(float,\n",
              "            {'ece': 0.30536604166030884,\n",
              "             'sce': 0.1526830017566681,\n",
              "             'ace': 0.15749655961990355,\n",
              "             'coverage_percentage': 1.0,\n",
              "             'coverage_width': 2.0})"
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ],
      "source": [
        "evaluate_calibration(ensemble, eval_split=test_loader)"
      ],
      "id": "Z3py0MFgUFRg"
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "id": "DLdcRIayUHgS",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "788b0a0a-69b6-4fac-c57e-b76472379308"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Evaluating batch 2/2...: 100%|██████████| 2/2 [00:47<00:00, 23.73s/it]\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "defaultdict(float,\n",
              "            {'ece': 0.0012063980102539062,\n",
              "             'sce': 0.0,\n",
              "             'ace': 0.0006239920854568482,\n",
              "             'coverage_percentage': 1.0,\n",
              "             'coverage_width': 1.0})"
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ],
      "source": [
        "evaluate_calibration(variational_bert, eval_split=test_loader)"
      ],
      "id": "DLdcRIayUHgS"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ea4ab469"
      },
      "source": [
        "## Uncertainty quantification\n",
        "\n",
        "Next, we want to use the model to actually quantify their uncertainty in a prediction. For this purpose, we manually define some sequences which should seem suspicious to the models. "
      ],
      "id": "ea4ab469"
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "id": "38e5a9ac",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1b0798e5-2985-4042-9c52-d15e58cebcec"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "the gorgeously elaborate continuation of \" the lord of the rings \" trilogy is so huge that a column of words cannot adequately describe co - writer / director peter jackson's expanded vision of j. r. r. tolkien's middle -\n"
          ]
        }
      ],
      "source": [
        "original_sentence = tokenizer.batch_decode(list(train_loader)[0][\"input_ids\"], skip_special_tokens=True)[1]\n",
        "print(original_sentence)"
      ],
      "id": "38e5a9ac"
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "id": "f9222448",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8ff83bf3-a87e-40ae-d73d-b0d81e8e30da"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/ huge the column \" the middle lord the - continuation describe trilogy peter adequately of j. r. rings expanded vision words \" of - r. jackson's cannot of is tolkien's gorgeously so co writer elaborate that a of director\n",
            "the grzgzeousbulyo eldeabnorate kcfonrktminuadgio of\" the lo of tee igngs \" rilgy ispsao pus tthat ppa columwn uosfp odrs uanznot adequatelywh demsclorbejco -k writrd jl/fdhiecntior pemte jackon's expajnded visiovo j.r. rp tolkipslnen'easg rmiaddlez -\n"
          ]
        }
      ],
      "source": [
        "# The model hasn't been finetuned on German, so this should be weird\n",
        "sentence1 = (\n",
        "    \"Die umwerfend aufwendige Fortsetzung der „Der Herr der Ringe“-Trilogie ist so umfangreich,\"\n",
        "    \"dass eine Kolonne von Worten die erweiterte Vision von Co-Autor/Regisseur Peter Jackson \"\n",
        "    \"von j. r . r . Tolkiens Mittelerde nicht angemessen beschreiben kann.\"\n",
        ").lower()\n",
        "# Now we scramble the contents of the sentence randomly\n",
        "tokens = original_sentence.split(\" \")\n",
        "sentence2 = \" \".join(random.sample(tokens, len(tokens)))\n",
        "print(sentence2)\n",
        "\n",
        "# Add noise to the sentence\n",
        "delete_chars = 30\n",
        "add_noise_chars = 60\n",
        "\n",
        "sentence3 = str(original_sentence)\n",
        "\n",
        "for _ in range(delete_chars):\n",
        "    idx = random.choice(range(len(sentence3)))\n",
        "    sentence3 = sentence3[:idx] + sentence3[idx + 1:]\n",
        "    \n",
        "for _ in range(add_noise_chars):\n",
        "    idx = random.choice(range(len(sentence3)))\n",
        "    char = random.choice(ascii_lowercase)\n",
        "    \n",
        "    sentence3 = sentence3[:idx] + char + sentence3[idx:]\n",
        "    \n",
        "print(sentence3)"
      ],
      "id": "f9222448"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "831fc2b0"
      },
      "source": [
        "We first check the predictions for the sentence above. The original sentence had a positive sentiment, so we first whether our model come to the same conclusion:"
      ],
      "id": "831fc2b0"
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "id": "89fe259a"
      },
      "outputs": [],
      "source": [
        "def make_single_prediction(model, input_, tokenizer):\n",
        "  \"\"\"\n",
        "  Make a prediction for a single sentence for the IMDB sentiment classification task.\n",
        "  \"\"\"\n",
        "  tokenized_input = tokenizer(input_, return_tensors=\"pt\", return_attention_mask=True)\n",
        "\n",
        "  with torch.no_grad():\n",
        "    prediction = torch.argmax(model.predict(tokenized_input[\"input_ids\"], attention_mask=tokenized_input[\"attention_mask\"])).cpu().numpy()\n",
        "\n",
        "  predicted_label = \"positive\" if prediction == 1 else \"negative\"\n",
        "\n",
        "  return predicted_label"
      ],
      "id": "89fe259a"
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "id": "_XbbNHLcVSbm",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e30c8042-e6f6-41d6-a76e-b672de346bce"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "positive\n",
            "positive\n",
            "positive\n",
            "positive\n"
          ]
        }
      ],
      "source": [
        "# Predictions for LSTM Ensemble\n",
        "print(make_single_prediction(ensemble, original_sentence, tokenizer))\n",
        "print(make_single_prediction(ensemble, sentence1, tokenizer))\n",
        "print(make_single_prediction(ensemble, sentence2, tokenizer))\n",
        "print(make_single_prediction(ensemble, sentence3, tokenizer))"
      ],
      "id": "_XbbNHLcVSbm"
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "id": "3DCjYLO_VYfq",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "50d6fabd-0543-4da2-c9c5-4a596f281ebf"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "positive\n",
            "positive\n",
            "positive\n",
            "positive\n"
          ]
        }
      ],
      "source": [
        "# Predictions for Variational BERT\n",
        "print(make_single_prediction(variational_bert, original_sentence, tokenizer))\n",
        "print(make_single_prediction(variational_bert, sentence1, tokenizer))\n",
        "print(make_single_prediction(variational_bert, sentence2, tokenizer))\n",
        "print(make_single_prediction(variational_bert, sentence3, tokenizer))"
      ],
      "id": "3DCjYLO_VYfq"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4272fda4"
      },
      "source": [
        "Here, all models still predict the right label. This could be because they learned underlying features - or because they are confidently incorrect. Since the sentences are very different from the training sentences, we can now measure the uncertainty. Since the inputs above are pretty different from the inputs the models were trained on, we would hope the models to be more uncertain on the noisy sentences. \n",
        "\n",
        "In this demo, we will explore three different uncertainty matrix: Maximum softmax probability, predictive entropy, and mutual information. Depending on the model, there might be different metrics available. You can check that by inspecting the ``available_uncertainty_metrics`` attribute:"
      ],
      "id": "4272fda4"
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "id": "29e90ad6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "49f3224c-443a-4d6f-b051-fdf1f1297c89"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'max_prob': <function nlp_uncertainty_zoo.utils.metrics.max_prob(logits: torch.FloatTensor) -> torch.FloatTensor>,\n",
              " 'predictive_entropy': <function nlp_uncertainty_zoo.utils.metrics.predictive_entropy(logits: torch.FloatTensor, eps: float = 1e-05) -> torch.FloatTensor>,\n",
              " 'dempster_shafer': <function nlp_uncertainty_zoo.utils.metrics.dempster_shafer(logits: torch.FloatTensor) -> torch.FloatTensor>,\n",
              " 'softmax_gap': <function nlp_uncertainty_zoo.utils.metrics.softmax_gap(logits: torch.FloatTensor) -> torch.FloatTensor>,\n",
              " 'variance': <function nlp_uncertainty_zoo.utils.metrics.variance(logits: torch.FloatTensor) -> torch.FloatTensor>,\n",
              " 'mutual_information': <function nlp_uncertainty_zoo.utils.metrics.mutual_information(logits: torch.FloatTensor, eps: float = 1e-05) -> torch.FloatTensor>}"
            ]
          },
          "metadata": {},
          "execution_count": 21
        }
      ],
      "source": [
        "ensemble.available_uncertainty_metrics"
      ],
      "id": "29e90ad6"
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "id": "rcHR0STuXsr7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3d41c963-7ece-462a-efdc-c05c3c0ceb85"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'max_prob': <function nlp_uncertainty_zoo.utils.metrics.max_prob(logits: torch.FloatTensor) -> torch.FloatTensor>,\n",
              " 'predictive_entropy': <function nlp_uncertainty_zoo.utils.metrics.predictive_entropy(logits: torch.FloatTensor, eps: float = 1e-05) -> torch.FloatTensor>,\n",
              " 'dempster_shafer': <function nlp_uncertainty_zoo.utils.metrics.dempster_shafer(logits: torch.FloatTensor) -> torch.FloatTensor>,\n",
              " 'softmax_gap': <function nlp_uncertainty_zoo.utils.metrics.softmax_gap(logits: torch.FloatTensor) -> torch.FloatTensor>,\n",
              " 'variance': <function nlp_uncertainty_zoo.utils.metrics.variance(logits: torch.FloatTensor) -> torch.FloatTensor>,\n",
              " 'mutual_information': <function nlp_uncertainty_zoo.utils.metrics.mutual_information(logits: torch.FloatTensor, eps: float = 1e-05) -> torch.FloatTensor>}"
            ]
          },
          "metadata": {},
          "execution_count": 22
        }
      ],
      "source": [
        "variational_bert.available_uncertainty_metrics"
      ],
      "id": "rcHR0STuXsr7"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3996f260"
      },
      "source": [
        "But back to metrics here. An easy and intuitive metric is the maximum softmax probability (Hendrycks & Gimpel, 2016)\n",
        "\n",
        "$$1 - \\max_k p_{\\theta}(y=k|x)$$\n",
        "\n",
        "Intuitively, when the model is uncertain, the distribution over classes should be uniform, thus yielding a low maximum probability over classes. We substract the value from 1 here in order to have small values correspond to high certainty. \n",
        "\n",
        "Another way to measure uncertainty is to use the Shannon entropy of the predictive distribution: For a uniform distribution, the entropy will be maximal:\n",
        "\n",
        "$$-\\sum_{k=1}^K p_{\\theta}(y=k|x) \\log p_{\\theta}(y=k|x)$$\n",
        "\n",
        "Lastly, Smith & Gal (2017) propose mutual information as a way to exlusively measure the *model uncertainty*:\n",
        "\n",
        "$$\\text{H}\\bigg[\\mathbb{E}_{q(\\theta)}\\Big[p_{\\theta}(y|x)\\Big]\\bigg] - \\mathbb{E}_{q(\\theta)}\\bigg[\\text{H}\\Big[p_{\\theta}(y|x)\\Big]\\bigg]$$\n",
        "\n",
        "Here, the first term denotes the total uncertainty, from which the second term, the *data uncertainty*, is subtracted, leaving only the model uncertainty. Usually, the expectation in both terms would over the weight posterior $p(\\theta|\\mathcal{D})$ of the model, which is generally intractable to evaluate for neural networks, which is why we model an approximate posterior $q(\\theta)$ instead. To evaluate this expectation, we use monte carlo sampling, by simply averaging the predictions coming from different sets of weights - in the case of the LSTM ensemble, these come from different ensemble members, for the Variational BERT, this corresponds to predictions using different dropout masks."
      ],
      "id": "3996f260"
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "id": "SJMs9I-f1Bqx"
      },
      "outputs": [],
      "source": [
        "def make_single_uncertainty(model, input_, tokenizer, metric):\n",
        "  \"\"\"\n",
        "  Make a prediction for a single sentence for the IMDB sentiment classification task.\n",
        "  \"\"\"\n",
        "  tokenized_input = tokenizer(input_, return_tensors=\"pt\", return_attention_mask=True)\n",
        "  uncertainty = model.get_uncertainty(tokenized_input[\"input_ids\"], attention_mask=tokenized_input[\"attention_mask\"], metric=metric).cpu().numpy()[0][0]\n",
        "\n",
        "  return uncertainty"
      ],
      "id": "SJMs9I-f1Bqx"
    },
    {
      "cell_type": "markdown",
      "source": [
        "As you can see from the snippet below, you can get the uncertainty from an input using the `get_uncertainty()` function of a model object. You can specify the metric specifying the `metric` argument. If no metric name was given, models fall back onto the metric defined in the `default_uncertainty_metric` attribute, which usually corresponds to the predictive entropy metric we just discussed."
      ],
      "metadata": {
        "id": "Qg5tIb7kkL3r"
      },
      "id": "Qg5tIb7kkL3r"
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "id": "e21e8401",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5c05c480-8ae7-4faf-f8f8-71af36095805"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Maximum softmax probability | LSTM Ensemble: Original sentence 0.588 / corrupted sentence 0.597 / German sentence 0.620\n",
            "Mutual information | LSTM Ensemble: Original sentence 0.647 / corrupted sentence 0.654 / German sentence 0.583\n"
          ]
        }
      ],
      "source": [
        "# Uncertainty estimates from the ensemble\n",
        "max_prob_orig_ensemble = make_single_uncertainty(ensemble, original_sentence, tokenizer, metric=\"max_prob\")\n",
        "max_prob_corrupted_ensemble = make_single_uncertainty(ensemble, sentence3, tokenizer, metric=\"max_prob\")\n",
        "max_prob_german_ensemble = make_single_uncertainty(ensemble, sentence1, tokenizer, metric=\"max_prob\")\n",
        "\n",
        "print(f\"Maximum softmax probability | LSTM Ensemble: Original sentence {max_prob_orig_ensemble:.3f} / corrupted sentence {max_prob_corrupted_ensemble:.3f} / German sentence {max_prob_german_ensemble:.3f}\")\n",
        "\n",
        "mutual_info_orig_ensemble = make_single_uncertainty(ensemble, original_sentence, tokenizer, metric=\"mutual_information\")\n",
        "mutual_info_corrupted_ensemble = make_single_uncertainty(ensemble, sentence3, tokenizer, metric=\"mutual_information\")\n",
        "mutual_info_german_ensemble = make_single_uncertainty(ensemble, sentence1, tokenizer, metric=\"mutual_information\")\n",
        "\n",
        "print(f\"Mutual information | LSTM Ensemble: Original sentence {mutual_info_orig_ensemble:.3f} / corrupted sentence {mutual_info_corrupted_ensemble:.3f} / German sentence {mutual_info_german_ensemble:.3f}\")"
      ],
      "id": "e21e8401"
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {
        "id": "82hmVmtP0jI5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "67c087dd-984d-469f-f2b7-13ed0731e807"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Maximum softmax probability | Variational BERT: Original sentence 0.009 / corrupted sentence 0.009 / German sentence 0.009 \n",
            "Mutual information | Variational BERT: Original sentence 0.009 / corrupted sentence 0.008 / German sentence 0.009\n"
          ]
        }
      ],
      "source": [
        "# Uncertainty estimates from the variational BERT\n",
        "max_prob_orig_variational_bert = make_single_uncertainty(variational_bert, original_sentence, tokenizer, metric=\"max_prob\")\n",
        "max_prob_corrupted_variational_bert = make_single_uncertainty(variational_bert, sentence3, tokenizer, metric=\"max_prob\")\n",
        "max_prob_german_variational_bert = make_single_uncertainty(variational_bert, sentence1, tokenizer, metric=\"max_prob\")\n",
        "\n",
        "print(f\"Maximum softmax probability | Variational BERT: Original sentence {max_prob_orig_variational_bert:.3f} / corrupted sentence {max_prob_corrupted_variational_bert:.3f} / German sentence {max_prob_german_variational_bert:.3f} \")\n",
        "\n",
        "mutual_info_orig_variational_bert = make_single_uncertainty(variational_bert, original_sentence, tokenizer, metric=\"mutual_information\")\n",
        "mutual_info_corrupted_variational_bert = make_single_uncertainty(variational_bert, sentence3, tokenizer, metric=\"mutual_information\")\n",
        "mutual_info_german_variational_bert = make_single_uncertainty(variational_bert, sentence1, tokenizer, metric=\"mutual_information\")\n",
        "\n",
        "print(f\"Mutual information | Variational BERT: Original sentence {mutual_info_orig_variational_bert:.3f} / corrupted sentence {mutual_info_corrupted_variational_bert:.3f} / German sentence {mutual_info_german_variational_bert:.3f}\")"
      ],
      "id": "82hmVmtP0jI5"
    },
    {
      "cell_type": "markdown",
      "source": [
        "Here we can see that both the ensemble and Variational BERT indicate either the same or higher uncertainty on the German sentence with the MSP score, but not the with Mutual information on the corrupted sentence! This is an open problem in the uncertainty quantification research: It is sometimes unclear, when our uncertainty estimates will be reliable. For this reason, we show how to evaluate them in detail next."
      ],
      "metadata": {
        "id": "4U1uVBMYndKV"
      },
      "id": "4U1uVBMYndKV"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7575f64c"
      },
      "source": [
        "## Evaluating the quality of uncertainty estimates"
      ],
      "id": "7575f64c"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f4f5e3ae"
      },
      "source": [
        "As we have done before with the raw probalities, we also want to know how reliable the uncertainty estimates for our models are. The package also provides several ways to do this: Firstly, we can evaluate them using an OOD detection task - the model should be more uncertain on data points that are unlike the ones in the training set. By using the uncertainty scores, we can use binary classification metrics like the area under the precision-recall curve (AUPR) and the area under the receiver-operator characteristic (AUROC) to evaluate this. In our Rotten tomatoes example, we will use Tweets from a different data set as an OOD data."
      ],
      "id": "f4f5e3ae"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "29fe9353"
      },
      "source": [
        "The other way introduced by Ulmer et al. (2022) is to measure how much high uncertainty corresponds to the model making wrong predictions. This is quantified by collecting the model loss and uncertainty for all points in the test set, and measuring their correlation using the [Kendall's $\\tau$ correlation coefficient](https://en.wikipedia.org/wiki/Kendall_rank_correlation_coefficient). The values range from -1 to 1, which 1 indicating that high uncertainty perfectly correlates with high model loss."
      ],
      "id": "29fe9353"
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {
        "id": "edcea9db"
      },
      "outputs": [],
      "source": [
        "from nlp_uncertainty_zoo.utils.uncertainty_eval import evaluate_uncertainty"
      ],
      "id": "edcea9db"
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {
        "id": "zoImAxB2X7Zj",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d2d7e3d1-6063-4deb-dfc7-9b2da0462612"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Evaluating batch 2/4...: 100%|██████████| 4/4 [01:09<00:00, 17.47s/it]\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "defaultdict(float,\n",
              "            {'kendalls_tau_id_max_prob_seq': KendalltauResult(correlation=0.3199999999999999, pvalue=0.025387066467633498),\n",
              "             'kendalls_tau_id_predictive_entropy_seq': KendalltauResult(correlation=0.24666666666666665, pvalue=0.0882790830212813),\n",
              "             'kendalls_tau_id_dempster_shafer_seq': KendalltauResult(correlation=0.3199999999999999, pvalue=0.025387066467633498),\n",
              "             'kendalls_tau_id_softmax_gap_seq': KendalltauResult(correlation=0.3666666666666666, pvalue=0.009856596254745291),\n",
              "             'kendalls_tau_id_variance_seq': KendalltauResult(correlation=0.23999999999999996, pvalue=0.09754603408321157),\n",
              "             'kendalls_tau_id_mutual_information_seq': KendalltauResult(correlation=-0.006666666666666665, pvalue=0.981568217609005),\n",
              "             'kendalls_tau_ood_max_prob_seq': KendalltauResult(correlation=0.14666666666666664, pvalue=0.3187493522432472),\n",
              "             'kendalls_tau_ood_predictive_entropy_seq': KendalltauResult(correlation=0.17333333333333328, pvalue=0.23629671774050431),\n",
              "             'kendalls_tau_ood_dempster_shafer_seq': KendalltauResult(correlation=-0.17999999999999997, pvalue=0.21823005846562027),\n",
              "             'kendalls_tau_ood_softmax_gap_seq': KendalltauResult(correlation=-0.07333333333333332, pvalue=0.6272676336274836),\n",
              "             'kendalls_tau_ood_variance_seq': KendalltauResult(correlation=-0.22666666666666663, pvalue=0.11834341824544657),\n",
              "             'kendalls_tau_ood_mutual_information_seq': KendalltauResult(correlation=-0.09999999999999998, pvalue=0.5021679313285716),\n",
              "             'aupr_max_prob': 0.48978360936213544,\n",
              "             'auroc_max_prob': 0.4576,\n",
              "             'aupr_predictive_entropy': 0.5479417715362291,\n",
              "             'auroc_predictive_entropy': 0.4752,\n",
              "             'aupr_dempster_shafer': 0.525810666092867,\n",
              "             'auroc_dempster_shafer': 0.4928,\n",
              "             'aupr_softmax_gap': 0.5053083371847207,\n",
              "             'auroc_softmax_gap': 0.4976,\n",
              "             'aupr_variance': 0.5718652682417142,\n",
              "             'auroc_variance': 0.5088,\n",
              "             'aupr_mutual_information': 0.6169532682724577,\n",
              "             'auroc_mutual_information': 0.608})"
            ]
          },
          "metadata": {},
          "execution_count": 27
        }
      ],
      "source": [
        "evaluate_uncertainty(ensemble, id_eval_split=test_loader, ood_eval_split=ood_test_loader)"
      ],
      "id": "zoImAxB2X7Zj"
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {
        "id": "g5iqay_LYHJ9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9e8a9129-c314-49a9-e7d6-a0ab5e45b53b"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Evaluating batch 2/4...: 100%|██████████| 4/4 [10:31<00:00, 149.47s/it]"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "defaultdict(float,\n",
              "            {'kendalls_tau_id_max_prob_seq': KendalltauResult(correlation=-0.14666666666666664, pvalue=0.3187493522432472),\n",
              "             'kendalls_tau_id_predictive_entropy_seq': KendalltauResult(correlation=-0.1533333333333333, pvalue=0.29659240311553436),\n",
              "             'kendalls_tau_id_dempster_shafer_seq': KendalltauResult(correlation=0.06666666666666665, pvalue=0.6604714750997261),\n",
              "             'kendalls_tau_id_softmax_gap_seq': KendalltauResult(correlation=-0.21999999999999997, pvalue=0.12994145940142376),\n",
              "             'kendalls_tau_id_variance_seq': KendalltauResult(correlation=-0.10666666666666665, pvalue=0.4730453015604536),\n",
              "             'kendalls_tau_id_mutual_information_seq': KendalltauResult(correlation=0.03999999999999999, pvalue=0.7993481021210237),\n",
              "             'kendalls_tau_ood_max_prob_seq': KendalltauResult(correlation=-0.35999999999999993, pvalue=0.011371729355231645),\n",
              "             'kendalls_tau_ood_predictive_entropy_seq': KendalltauResult(correlation=-0.11999999999999998, pvalue=0.4176309114337012),\n",
              "             'kendalls_tau_ood_dempster_shafer_seq': KendalltauResult(correlation=-0.17333333333333328, pvalue=0.23629671774050431),\n",
              "             'kendalls_tau_ood_softmax_gap_seq': KendalltauResult(correlation=0.01333333333333333, pvalue=0.9447431182315829),\n",
              "             'kendalls_tau_ood_variance_seq': KendalltauResult(correlation=0.05333333333333332, pvalue=0.7288211329394639),\n",
              "             'kendalls_tau_ood_mutual_information_seq': KendalltauResult(correlation=0.05333333333333332, pvalue=0.7288211329394639),\n",
              "             'aupr_max_prob': 0.539034474126265,\n",
              "             'auroc_max_prob': 0.48640000000000005,\n",
              "             'aupr_predictive_entropy': 0.42829551387764725,\n",
              "             'auroc_predictive_entropy': 0.3328,\n",
              "             'aupr_dempster_shafer': 0.4232158691371806,\n",
              "             'auroc_dempster_shafer': 0.3312,\n",
              "             'aupr_softmax_gap': 0.5145271321818373,\n",
              "             'auroc_softmax_gap': 0.4848,\n",
              "             'aupr_variance': 0.4833822051640597,\n",
              "             'auroc_variance': 0.48,\n",
              "             'aupr_mutual_information': 0.5916907930097783,\n",
              "             'auroc_mutual_information': 0.5968})"
            ]
          },
          "metadata": {},
          "execution_count": 28
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\rEvaluating batch 2/4...: 100%|██████████| 4/4 [10:31<00:00, 157.81s/it]"
          ]
        }
      ],
      "source": [
        "evaluate_uncertainty(variational_bert, id_eval_split=test_loader, ood_eval_split=ood_test_loader)"
      ],
      "id": "g5iqay_LYHJ9"
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4e71193f"
      },
      "source": [
        "Thanks for reading through this demo! We only showcase the most useful, but not all, of the functionalities of the package here. If you would like to know more about the different models and functionalities in the package, consult [the documentation](http://dennisulmer.eu/nlp-uncertainty-zoo/). If you find any bugs or have requests for missing features, please [open an issue on the Github repository](https://github.com/Kaleidophon/nlp-uncertainty-zoo/issues). Below you can find the papers that were referenced in this demo:\n",
        "\n",
        "Gal, Yarin, and Zoubin Ghahramani. \"A theoretically grounded application of dropout in recurrent neural networks.\" Advances in neural information processing systems 29 (2016).\n",
        "\n",
        "Gal, Yarin, and Zoubin Ghahramani. \"Dropout as a bayesian approximation: Representing model uncertainty in deep learning.\" international conference on machine learning. PMLR, 2016.\n",
        "\n",
        "Hendrycks, Dan, and Kevin Gimpel. \"A baseline for detecting misclassified and out-of-distribution examples in neural networks.\" arXiv preprint arXiv:1610.02136 (2016).\n",
        "\n",
        "Kompa, Benjamin, Jasper Snoek, and Andrew L. Beam. \"Empirical frequentist coverage of deep learning uncertainty quantification procedures.\" Entropy 23.12 (2021): 1608.\n",
        "\n",
        "Lakshminarayanan, Balaji, Alexander Pritzel, and Charles Blundell. \"Simple and scalable predictive uncertainty estimation using deep ensembles.\" Advances in neural information processing systems 30 (2017).\n",
        "\n",
        "Smith, Lewis, and Yarin Gal. \"Understanding measures of uncertainty for adversarial example detection.\" arXiv preprint arXiv:1803.08533 (2018).\n",
        "\n",
        "Ulmer, Dennis Thomas, Jes Frellsen, and Christian Hardmeier. \"Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method & Data Scarcity.\" Findings of 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2022.\n",
        "\n",
        "Xiao, Tim Z., Aidan N. Gomez, and Yarin Gal. \"Wat zei je? detecting out-of-distribution translations with variational transformers.\" arXiv preprint arXiv:2006.08344 (2020)."
      ],
      "id": "4e71193f"
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {
        "id": "b9b991e8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "661e59ca-434a-4b35-c1ba-dbe2fa9aca26"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "\n"
          ]
        }
      ],
      "source": [],
      "id": "b9b991e8"
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.13"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "dc0bf6ba55cb4908a492f845dbfd7d71": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_fca4e1a0c4274428ae247c4690430a5b",
              "IPY_MODEL_958614ddda6547b682f2e111d21b2350",
              "IPY_MODEL_f5ae790b4ba949e6af819aa71102bed6"
            ],
            "layout": "IPY_MODEL_88cae46fe0b241d4b20223ba674391a4"
          }
        },
        "fca4e1a0c4274428ae247c4690430a5b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f8c92cfafa1742c18ebb6b0d9e646284",
            "placeholder": "​",
            "style": "IPY_MODEL_b5f515ea68b3424fa95a0e1258e54b41",
            "value": "Downloading (…)solve/main/vocab.txt: 100%"
          }
        },
        "958614ddda6547b682f2e111d21b2350": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_bebca6f8fe6048dda13e13bf993709f1",
            "max": 231508,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_6b8563ada68f4945a7adffce7713880d",
            "value": 231508
          }
        },
        "f5ae790b4ba949e6af819aa71102bed6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_3935bb98a26647e48941e547ed5c0779",
            "placeholder": "​",
            "style": "IPY_MODEL_b402d95e1bf841dea521a1404739e415",
            "value": " 232k/232k [00:00&lt;00:00, 2.33MB/s]"
          }
        },
        "88cae46fe0b241d4b20223ba674391a4": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f8c92cfafa1742c18ebb6b0d9e646284": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b5f515ea68b3424fa95a0e1258e54b41": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "bebca6f8fe6048dda13e13bf993709f1": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "6b8563ada68f4945a7adffce7713880d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "3935bb98a26647e48941e547ed5c0779": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b402d95e1bf841dea521a1404739e415": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "de6e9320ced14aaaa10f81fe9c501fec": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_c9a9fab793d34ad49cbf2908350e4d04",
              "IPY_MODEL_601ce7d9813a40a88a2ef58e99345637",
              "IPY_MODEL_6b5d1f908b6b48f9b4baa16b1d285cfe"
            ],
            "layout": "IPY_MODEL_6315a9343de9468cb06689d42dcc3b5a"
          }
        },
        "c9a9fab793d34ad49cbf2908350e4d04": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_9a23e58d21494b66886f7b5a84a604b6",
            "placeholder": "​",
            "style": "IPY_MODEL_82407042963b489cb513653dc79819cf",
            "value": "Downloading (…)okenizer_config.json: 100%"
          }
        },
        "601ce7d9813a40a88a2ef58e99345637": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_02be4776a3fb4f229fa517ee6ec99ceb",
            "max": 28,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_fbe909a8434b471eab77addd48f61b5c",
            "value": 28
          }
        },
        "6b5d1f908b6b48f9b4baa16b1d285cfe": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_4072731cf4be4ab198abe60bfb2c5ea1",
            "placeholder": "​",
            "style": "IPY_MODEL_9343a66615774849b2e397ad91e87134",
            "value": " 28.0/28.0 [00:00&lt;00:00, 637B/s]"
          }
        },
        "6315a9343de9468cb06689d42dcc3b5a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9a23e58d21494b66886f7b5a84a604b6": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "82407042963b489cb513653dc79819cf": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "02be4776a3fb4f229fa517ee6ec99ceb": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "fbe909a8434b471eab77addd48f61b5c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "4072731cf4be4ab198abe60bfb2c5ea1": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9343a66615774849b2e397ad91e87134": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "dea997b3e5804bc883454734633455ed": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_485f592d05354006b4d7d24b6e9a5ad3",
              "IPY_MODEL_27f3dc7712664547ba51bc1058791571",
              "IPY_MODEL_985286ce89ad406590d1c7bf594ad17b"
            ],
            "layout": "IPY_MODEL_3e02992cb3614a1b832028130f95c1d4"
          }
        },
        "485f592d05354006b4d7d24b6e9a5ad3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_6c3767f6a1264f2ab86e52b53aab6840",
            "placeholder": "​",
            "style": "IPY_MODEL_5fae15cddeef466489d6298408bd0308",
            "value": "Downloading (…)lve/main/config.json: 100%"
          }
        },
        "27f3dc7712664547ba51bc1058791571": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_256342f0909d47d8b4d7b88fb5097e81",
            "max": 570,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_f57f6ab74522424a8d8183698cd9bf84",
            "value": 570
          }
        },
        "985286ce89ad406590d1c7bf594ad17b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_1a3aaad7c01d43e6a770824e198b0fa1",
            "placeholder": "​",
            "style": "IPY_MODEL_40d1fd61b2024defa993a9fdc7f2cc64",
            "value": " 570/570 [00:00&lt;00:00, 13.6kB/s]"
          }
        },
        "3e02992cb3614a1b832028130f95c1d4": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "6c3767f6a1264f2ab86e52b53aab6840": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5fae15cddeef466489d6298408bd0308": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "256342f0909d47d8b4d7b88fb5097e81": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f57f6ab74522424a8d8183698cd9bf84": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "1a3aaad7c01d43e6a770824e198b0fa1": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "40d1fd61b2024defa993a9fdc7f2cc64": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "7bf14949226544e49d39c4744b3ef6b8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_dc4a4bc2dbb34b91a42cfbe08f5ef149",
              "IPY_MODEL_2c7338f42527405c9f4aceb609107e69",
              "IPY_MODEL_cd227304c23f46a080147864356702cc"
            ],
            "layout": "IPY_MODEL_186f5742fd6d4b5b98932708d49a3af1"
          }
        },
        "dc4a4bc2dbb34b91a42cfbe08f5ef149": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_11cc1b8150964e0b90f47496d535b5f8",
            "placeholder": "​",
            "style": "IPY_MODEL_1b2d49d63d8a48a4a674310d44ef28e3",
            "value": "Map: 100%"
          }
        },
        "2c7338f42527405c9f4aceb609107e69": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ca05a94bcf2e4fb88df94e6a1fda74ee",
            "max": 100,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_0da955052cf44f2c9cfea4dfc27cb8c1",
            "value": 100
          }
        },
        "cd227304c23f46a080147864356702cc": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_4d74cbb284ff4b179ef04d0aa2ea3047",
            "placeholder": "​",
            "style": "IPY_MODEL_cd2a422de522448a948b29d90b8264d1",
            "value": " 100/100 [00:00&lt;00:00, 793.07 examples/s]"
          }
        },
        "186f5742fd6d4b5b98932708d49a3af1": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": "hidden",
            "width": null
          }
        },
        "11cc1b8150964e0b90f47496d535b5f8": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1b2d49d63d8a48a4a674310d44ef28e3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "ca05a94bcf2e4fb88df94e6a1fda74ee": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0da955052cf44f2c9cfea4dfc27cb8c1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "4d74cbb284ff4b179ef04d0aa2ea3047": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "cd2a422de522448a948b29d90b8264d1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "3c6e475832c6469c911a2c05c0500424": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_57e4798293824e418af084e0c5351e26",
              "IPY_MODEL_0effe1422f634fa58c4c0c5b5e59e3b5",
              "IPY_MODEL_55c845e610ff4e11a41b8a54979f0fc1"
            ],
            "layout": "IPY_MODEL_1c62eccf29764231a20c44f5fe0da5dc"
          }
        },
        "57e4798293824e418af084e0c5351e26": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_369ec0ff10f043d08a11833a291dcf67",
            "placeholder": "​",
            "style": "IPY_MODEL_c405831192504dae990029ccef180e85",
            "value": "Map:   0%"
          }
        },
        "0effe1422f634fa58c4c0c5b5e59e3b5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f71beb50b901486c8b67c1cc48575a8c",
            "max": 25,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_a086444e398a46af9c77e2c26a18306f",
            "value": 25
          }
        },
        "55c845e610ff4e11a41b8a54979f0fc1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_2b98db052a2f4fb7a6a695ddeb312360",
            "placeholder": "​",
            "style": "IPY_MODEL_2f5ae230ff5745969c7723d1dcbd38d4",
            "value": " 0/25 [00:00&lt;?, ? examples/s]"
          }
        },
        "1c62eccf29764231a20c44f5fe0da5dc": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": "hidden",
            "width": null
          }
        },
        "369ec0ff10f043d08a11833a291dcf67": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c405831192504dae990029ccef180e85": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "f71beb50b901486c8b67c1cc48575a8c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "a086444e398a46af9c77e2c26a18306f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "2b98db052a2f4fb7a6a695ddeb312360": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2f5ae230ff5745969c7723d1dcbd38d4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "f5ba6a38ff4f4e3aa274a06d374e0d88": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_1d44e548de84443da34888fb24c7412e",
              "IPY_MODEL_b1e3d682c7434e6f9f9f515a269ca32c",
              "IPY_MODEL_d7dc4d22a77847d28555fadf94199e22"
            ],
            "layout": "IPY_MODEL_f9a9456a47894a0691b32450ea42947e"
          }
        },
        "1d44e548de84443da34888fb24c7412e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_3728f25b38ec45aea4b0846b9352df34",
            "placeholder": "​",
            "style": "IPY_MODEL_165108ca8e8f4f6cb68ec2cc9374da63",
            "value": "Map:   0%"
          }
        },
        "b1e3d682c7434e6f9f9f515a269ca32c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_6bc3814a734640a591615b2ba1ef2380",
            "max": 25,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_fac8479b0b7a439b854c681feefb8c07",
            "value": 25
          }
        },
        "d7dc4d22a77847d28555fadf94199e22": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d20e2ffcca174ab091b258aadd27968b",
            "placeholder": "​",
            "style": "IPY_MODEL_61b0c52c76594e51b070fd3de1a3f6e4",
            "value": " 0/25 [00:00&lt;?, ? examples/s]"
          }
        },
        "f9a9456a47894a0691b32450ea42947e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": "hidden",
            "width": null
          }
        },
        "3728f25b38ec45aea4b0846b9352df34": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "165108ca8e8f4f6cb68ec2cc9374da63": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "6bc3814a734640a591615b2ba1ef2380": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "fac8479b0b7a439b854c681feefb8c07": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "d20e2ffcca174ab091b258aadd27968b": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "61b0c52c76594e51b070fd3de1a3f6e4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "72a3bbfb710d417192a4e31a99cf49e5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_d3def39d079f4cd8b68c3bb388d06884",
              "IPY_MODEL_b5d3bab4339242be9c70f24cd21bc2bc",
              "IPY_MODEL_5d300bed22be405b9170920885a9447f"
            ],
            "layout": "IPY_MODEL_a6f9b42637dc417a92533a577278c03a"
          }
        },
        "d3def39d079f4cd8b68c3bb388d06884": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f7f1fb0dce524099913abb524ae10308",
            "placeholder": "​",
            "style": "IPY_MODEL_27b43b848c254cfc8fbd70dfd4a92d2b",
            "value": "Downloading (…)&quot;pytorch_model.bin&quot;;: 100%"
          }
        },
        "b5d3bab4339242be9c70f24cd21bc2bc": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_c6b2f20955d24aa885c880345acd0cfb",
            "max": 440473133,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_5f8835b2096f4b849c0f100d9e18fb9b",
            "value": 440473133
          }
        },
        "5d300bed22be405b9170920885a9447f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_8cc8c4f84c974be4a07cc73b5a79b3d4",
            "placeholder": "​",
            "style": "IPY_MODEL_0a4b3f5002664de99fe30fc4a0d159a6",
            "value": " 440M/440M [00:05&lt;00:00, 85.4MB/s]"
          }
        },
        "a6f9b42637dc417a92533a577278c03a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f7f1fb0dce524099913abb524ae10308": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "27b43b848c254cfc8fbd70dfd4a92d2b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "c6b2f20955d24aa885c880345acd0cfb": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5f8835b2096f4b849c0f100d9e18fb9b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "8cc8c4f84c974be4a07cc73b5a79b3d4": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0a4b3f5002664de99fe30fc4a0d159a6": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}