ML-Intro-Course/15_toxicity_detection.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WW1udtWLYTaK"
      },
      "source": [
        "# Text classification with Transformers\n",
        "\n",
        "<a href=\"https://colab.research.google.com/drive/1ylzigj4yN0qNQjVVuVHzsrOkrq1FpDBy\" target=\"_blank\">\n",
        "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\">\n",
        "</a>\n",
        "\n",
        "Return to the [castle](https://github.com/Nkluge-correa/TeenyTinyCastle).\n",
        "\n",
        "Text classification is the process of categorizing text into predefined classes or categories based on its content. One common classification task in NLP is toxicity detection.\n",
        "\n",
        "This is a common application of text classification in natural language processing, where the objective is to identify whether a piece of text contains language that is offensive, harmful, or threatening. To accomplish this, machine learning models are trained on large datasets of labeled toxic and non-toxic text examples and then used to predict the toxicity of new, unlabeled text.\n",
        "\n",
        "We can define toxicity as:\n",
        "\n",
        "> \"_abusive speech targeting specific group characteristics, such as ethnic origin, religion, gender, or sexual orientation._\"\n",
        "\n",
        "For an in-depth discussion on toxicity detection as a machine learning problem, we recommend \"_[Learning from the worst: Dynamically generated datasets to improve online hate detection](https://scholar.google.com/scholar_url?url=https://arxiv.org/abs/2012.15761&hl=pt-BR&sa=T&oi=gsb&ct=res&cd=0&d=7265559494033067667&ei=QUJYY6TJL4iKmgHXk5LgDQ&scisig=AAGBfm3gsyOD5eqcUPLFvWmVm8PlLcMr3g)_\".\n",
        "\n",
        "<img src=\"https://miro.medium.com/max/1400/1*d4k-PRw-warACDpklCh1mw.png\" alt=\"toxic-image\" width=\"800\"/>\n",
        "\n",
        "_Source: [Toxicity in AI Text Generation](https://towardsdatascience.com/toxicity-in-ai-text-generation-9e9d9646e68f)._\n",
        "\n",
        "In this notebook, we will be using a dataset created from the [Toxic Comment Classification Challenge Dataset](https://github.com/tianqwang/Toxic-Comment-Classification-Challenge), created by the [Conversation AI](https://conversationai.github.io/) team, a research initiative founded by [Jigsaw](https://jigsaw.google.com/) and Google (both a part of Alphabet).\n",
        "\n",
        "## ☣️ DISCLAIMER/WARNING ☣️\n",
        "\n",
        "_This dataset contains text that may be considered profane, vulgar, or offensive. The vocabulary of the trained model also contains this type of language._\n",
        "\n",
        "The original dataset contains an unequal distribution of “_hate_” and “_not hate_” samples for multi-classification. However, we created a smaller version of the original dataset and made it available on [Hugging Face](https://huggingface.co/datasets/dieineb/toxic_content_dataset).\n",
        "\n",
        "But before starting, let us install the required libraries for this tutorial by running the cell below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Vop-t27gIjWg",
        "outputId": "e05a95ce-83e8-4296-d4e0-242e75165976"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Note: you may need to restart the kernel to use updated packages.\n"
          ]
        }
      ],
      "source": [
        "# Install the required libraries\n",
        "%pip install datasets -q"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ESBRm3JynA-4"
      },
      "source": [
        "Now, we can download our dataset directly from the hub. 🤗"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 334,
          "referenced_widgets": [
            "1ca8ce3e01ba42948a1e4493ec2e1a3a",
            "b2a9d77d113943f08a10560bb2454ee7",
            "e9a67b62a8da40289eedde858138984e",
            "193d60122b014a72a3a1608cbefcec10",
            "c34f8d6666794bb7a26cde0b9afcc6df",
            "75525b3760744d1d9620cd74815a4936",
            "83c822d4f81d4bc38f44659049b280dc",
            "e22945e6a43b48f681eb7805edb0f8f7",
            "ee23a3ea57de44d69636a030dbedbc3e",
            "23d0a0b2bbeb4327817609e21b1d6363",
            "7c572c6f7a634f97bdf53f78159d4fbd",
            "8269280765314b7a9c854f6859d60bef",
            "b13a5a530d2c42fb96fa8f6025e186de",
            "bf56868c2b734528930c6a7a8a8473ca",
            "57fe3fc671a34c63b1458e64cd0b5d91",
            "66edae1cc2704e29b360433d7daaf88f",
            "73923227531f43ea8afc94e4fc0cd176",
            "722d08759d3a4c8b8362b3210012a43d",
            "2e3b21a596cd4404888f72098b0750dd",
            "36deacf6e5294414b3ca37af271c4419",
            "1561b8c5dd544402b0a76cd3b458d5dc",
            "b6bc31f1f36443c9987fba4f4367240c",
            "d2294fdc3bd5451d94473f09306cb215",
            "c1bc3c9b1b6444c6887980230c9d254a",
            "5c3878a63ff346dc9cd06ec3bd1c9cb0",
            "3d310d773c9b44b5a68b60b1e465c08f",
            "7eb5b288160b4760bc1ebf291878114a",
            "98d0896622454db68579abe0fb09781e",
            "e5465f649db645abb8b18c2f8564fb2e",
            "ad474ef8888f42e8b5b59c1fed78fee1",
            "5afe88be584a4671b73389106f917f47",
            "cda4da6ab570438db82f4c5140da3079",
            "565ab0d44b5040f0a94abaeabd488e67",
            "496f6eb8158d4ed5b11d2f75fdd14c6d",
            "a207cf549e934a0387c910e1499d1c05",
            "2d444946e6b3418caad9b8b22489a6b7",
            "493ab53d65594c39a6fc0408f7d95353",
            "bd19ea0e14ba42b1babff7664189ef3e",
            "5a87b28067c04e70b8636fe3361f5782",
            "321b102134ef408385f3156b52f14296",
            "aa061a3603e04520a80f2e662382e970",
            "da141a2bcb88463490ccb93f1481aa5f",
            "35c59acf797d46d1be105336898beb9b",
            "c221e567eefa42e5acc9d7416445ca07"
          ]
        },
        "id": "jnxVX43qrdvE",
        "outputId": "9239e310-ad1d-4b4b-c13a-60110abba147"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Downloading and preparing dataset csv/dieineb--toxic_content_dataset to C:/Users/CWLINK/.cache/huggingface/datasets/dieineb___csv/dieineb--toxic_content_dataset-6e7143093a907c22/0.0.0/6954658bab30a358235fa864b05cf819af0e179325c740e4bc853bcc7ec513e1...\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "867f5dfc4dda4ce5b790292648381b64",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading data files:   0%|          | 0/1 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "376ddde1d446428090c9521ca79fe4c0",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading data:   0%|          | 0.00/15.5M [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "9a42ca5caecd47d6851897b466b48db0",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Extracting data files:   0%|          | 0/1 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "2282ed4268654e70877568ab03e81cc0",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Generating train split: 0 examples [00:00, ? examples/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Dataset csv downloaded and prepared to C:/Users/CWLINK/.cache/huggingface/datasets/dieineb___csv/dieineb--toxic_content_dataset-6e7143093a907c22/0.0.0/6954658bab30a358235fa864b05cf819af0e179325c740e4bc853bcc7ec513e1. Subsequent calls will reuse this data.\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "f43f055ed40448eda68b76139af76452",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "  0%|          | 0/1 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>comment_text</th>\n",
              "      <th>toxic</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>explanation edits made username hardcore metal...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>aww matches background colour seemingly stuck ...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>hey man really trying edit war guy constantly ...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>make real suggestions improvement wondered sec...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>sir hero chance remember page</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                        comment_text  toxic\n",
              "0  explanation edits made username hardcore metal...      1\n",
              "1  aww matches background colour seemingly stuck ...      1\n",
              "2  hey man really trying edit war guy constantly ...      1\n",
              "3  make real suggestions improvement wondered sec...      1\n",
              "4                      sir hero chance remember page      1"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "from datasets import load_dataset\n",
        "\n",
        "# load the dataset from the hub\n",
        "dataset = load_dataset(\"AiresPucrs/toxic-comments\", split='train')\n",
        "\n",
        "# turn the dataset into a pandas.DataFrame\n",
        "df = dataset.to_pandas()\n",
        "\n",
        "display(df.head())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uoigR3O-o0FA"
      },
      "source": [
        "Text is one of the most widespread forms of sequence data and discrete signals (as opposed to continuous signals, like _images_ or _audio_). These sequences can be sequences of characters, syllables, words, or tokens.\n",
        "\n",
        "Deep learning for NLP is pattern recognition applied to paragraphs, sentences, and words, just as computer vision is pattern recognition applied to videos, images, and pixels.\n",
        "\n",
        "Like all neural networks, language models based on deep-learning architectures don't take as input raw text, i.e., you _can not multiply a word by a weight matrix, add a bias, and apply a ReLU function at the end_. Neural networks only work with numeric tensors. Thus, we need to _vectorize_ our text data, i.e., transform the text into numeric tensors.\n",
        "\n",
        "For a comprehensive guide on how to vectorize text data, we recommend Chapter 6: Deep learning for text and sequences, in [_Deep Learning with Python_](https://tanthiamhuat.files.wordpress.com/2018/03/deeplearningwithpython.pdf). Below we will be using the `TextVectorization` layer from the [Keras](https://keras.io/) library.\n",
        "\n",
        "We will define two hyperparameters for this class: our vocabulary size (`vocab_size`) and the maximum sequence length (`sequence_length`), which truncates a sequence after this limit is reached."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "yC5p_tWuo8LG"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "\n",
        "# number of words in our vocabulary\n",
        "vocab_size = 20000\n",
        "\n",
        "# sentences lengthier than 100 will be truncated\n",
        "sequence_length = 100\n",
        "\n",
        "vectorization_layer = tf.keras.layers.TextVectorization(\n",
        "    max_tokens=vocab_size,\n",
        "    output_mode=\"int\",\n",
        "    output_sequence_length=sequence_length\n",
        "    )\n",
        "\n",
        "vectorization_layer.adapt(df.comment_text)\n",
        "vocabulary = vectorization_layer.get_vocabulary()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S_TKigVWpulC"
      },
      "source": [
        "You can save your vocabulary as a `.txt` file for later use. Don't forget to save it on your drive, or to download it before finishing this session.\n",
        "\n",
        "If you are using Colab, run the following to save in your Drive:\n",
        "\n",
        "```python\n",
        "from google.colab import drive\n",
        "drive.mount('/content/drive')\n",
        "\n",
        "with open(r'/content/drive/MyDrive/toxic-vocabulary.txt', 'w', encoding='utf-8') as fp:\n",
        "    for word in vocabulary:\n",
        "        fp.write(\"%s\\n\" % word)\n",
        "    fp.close()\n",
        "```\n",
        "\n",
        "☣️ DISCLAIMER - THIS VOCABULARY CONTAINS TOXIC LANGUAGE ☣️"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "l72L2xcQqJ2t"
      },
      "outputs": [],
      "source": [
        "with open(r'toxic-vocabulary.txt', 'w', encoding='utf-8') as fp:\n",
        "    for word in vocabulary:\n",
        "        fp.write(\"%s\\n\" % word)\n",
        "    fp.close()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EheCSinmYTaQ"
      },
      "source": [
        "\n",
        "Now, we can split our dataset for training, validation, and testing (the usual ML split). For this we will use the classic `sklearn.model_selection.train_test_split`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LWsg_nhpsRgc",
        "outputId": "b268d258-575d-406f-f136-368e7dd988fe"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Training split:  (56125, 100) (56125,)\n",
            "Test split:  (14032, 100) (14032,)\n"
          ]
        }
      ],
      "source": [
        "import numpy as np\n",
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "# We are doing a 80/20 split by setting `test_size=0.2`\n",
        "# A constant random_state value will guarantee that the split is reproducible.\n",
        "x_train, x_test, y_train, y_test = train_test_split(\n",
        "    df.comment_text, df.toxic, test_size=0.2, random_state=42)\n",
        "\n",
        "# Turn our target and labels into numpy.arrays\n",
        "x_train = vectorization_layer(x_train)\n",
        "y_train = np.array(y_train).astype(float)\n",
        "x_test = vectorization_layer(x_test)\n",
        "y_test = np.array(y_test).astype(float)\n",
        "\n",
        "print(\"Training split: \", x_train.shape, y_train.shape)\n",
        "print(\"Test split: \", x_test.shape, y_test.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JPddcv5eYTaR"
      },
      "source": [
        "[Recurrent neural networks](https://en.wikipedia.org/wiki/Recurrent_neural_networks \"Recurrent neural networks\"), like `LSTM` and `GRU`, and convolutional networks, like `1D Convnets`, are great options for dealing with problems involving NLP. In [other notebooks](https://github.com/Nkluge-correa/TeenyTinyCastle/blob/master/ML-Explainability/NLP/model_maker.ipynb) of our repository, you will see many examples of how to build these networks for tasks like sentiment analysis.\n",
        "\n",
        "However, in this notebook, we will be using a `transformer` model, an extremely versatile and scalable architecture proposed by Vaswani et al. in [Attention Is All You Need](https://arxiv.org/abs/1706.03762).\n",
        "\n",
        "<img src=\"https://machinelearningmastery.com/wp-content/uploads/2021/08/attention_research_1.png\" alt=\"drawing\" height=\"450\"/>\n",
        "\n",
        "_Source: [Attention Is All You Need](https://arxiv.org/abs/1706.03762)._\n",
        "\n",
        "A `transformer` is a  deep learning model that uses a mechanism called [self-attention](https://en.wikipedia.org/wiki/Attention_(machine_learning)), differentially weighting the significance of each part of the input data. Like RNNs, transformers are designed to process sequential input data. However, unlike RNNs, transformers can process an entire sequence (as long as it fits on its context head) all at once (_not sequencially_). This allows for more parallelization, thus reducing training times and also allowing the training on larger datasets.\n",
        "\n",
        "For an extremely _comprehensive_ and _ilustrated_ explanation of what is \"_attention_\" or how a \"_transformer works_\", we recommend the work of _Jay Alammar_:\n",
        "\n",
        "- [The Illustrated Transformer](https://jalammar.github.io/illustrated-transformer/).\n",
        "- [The Illustrated GPT-2](https://jalammar.github.io/illustrated-gpt2/).\n",
        "\n",
        "And, for an interective journey trough the transformer arquitecture, we recommend the [LLM Visualization](https://bbycroft.net/llm) tool. \n",
        "\n",
        "Using only the _encoder_ component of the original transformer architecture, we will implement a small transformer model in this notebook (a.k.a. a encoder-only transformer). The original transformer architecture, which consists of both an encoder and a decoder transformer block, was (originally) designed for _sequence-to-sequence_ tasks like translation.\n",
        "\n",
        "If you are interested in learning about the transformer architecture, check our _[sequence-to-sequence](https://github.com/Nkluge-correa/TeenyTinyCastle/blob/master/ML-Intro-Course/16_seuqnece_to_sequence.ipynb)_ tutorial.\n",
        "\n",
        "In general, text classification can be done using the encoder component. The encoder it's a very generic module that learns to transform a sequence into a more useful representation after ingesting it.\n",
        "\n",
        "Our model only has 4 attention heads with a capacity of 100 tokens and 4 transformer blocks. Our embedding layer's size is also restricted to embeddings with 128 dimensions and a vocabulary of 20.000 tokens (where the dense word vectors will be created)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "m-RsRVBnYTaR",
        "outputId": "dc2461e6-ab0b-4210-dee6-18659788c4d0"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Model: \"ToxicityModel\"\n",
            "__________________________________________________________________________________________________\n",
            " Layer (type)                   Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            " input_1 (InputLayer)           [(None, 100)]        0           []                               \n",
            "                                                                                                  \n",
            " embedding (Embedding)          (None, 100, 128)     2560000     ['input_1[0][0]']                \n",
            "                                                                                                  \n",
            " layer_normalization (LayerNorm  (None, 100, 128)    256         ['embedding[0][0]']              \n",
            " alization)                                                                                       \n",
            "                                                                                                  \n",
            " multi_head_attention (MultiHea  (None, 100, 128)    527488      ['layer_normalization[0][0]',    \n",
            " dAttention)                                                      'layer_normalization[0][0]']    \n",
            "                                                                                                  \n",
            " dropout (Dropout)              (None, 100, 128)     0           ['multi_head_attention[0][0]']   \n",
            "                                                                                                  \n",
            " tf.__operators__.add (TFOpLamb  (None, 100, 128)    0           ['dropout[0][0]',                \n",
            " da)                                                              'embedding[0][0]']              \n",
            "                                                                                                  \n",
            " layer_normalization_1 (LayerNo  (None, 100, 128)    256         ['tf.__operators__.add[0][0]']   \n",
            " rmalization)                                                                                     \n",
            "                                                                                                  \n",
            " conv1d (Conv1D)                (None, 100, 4)       516         ['layer_normalization_1[0][0]']  \n",
            "                                                                                                  \n",
            " dropout_1 (Dropout)            (None, 100, 4)       0           ['conv1d[0][0]']                 \n",
            "                                                                                                  \n",
            " conv1d_1 (Conv1D)              (None, 100, 128)     640         ['dropout_1[0][0]']              \n",
            "                                                                                                  \n",
            " tf.__operators__.add_1 (TFOpLa  (None, 100, 128)    0           ['conv1d_1[0][0]',               \n",
            " mbda)                                                            'tf.__operators__.add[0][0]']   \n",
            "                                                                                                  \n",
            " layer_normalization_2 (LayerNo  (None, 100, 128)    256         ['tf.__operators__.add_1[0][0]'] \n",
            " rmalization)                                                                                     \n",
            "                                                                                                  \n",
            " multi_head_attention_1 (MultiH  (None, 100, 128)    527488      ['layer_normalization_2[0][0]',  \n",
            " eadAttention)                                                    'layer_normalization_2[0][0]']  \n",
            "                                                                                                  \n",
            " dropout_2 (Dropout)            (None, 100, 128)     0           ['multi_head_attention_1[0][0]'] \n",
            "                                                                                                  \n",
            " tf.__operators__.add_2 (TFOpLa  (None, 100, 128)    0           ['dropout_2[0][0]',              \n",
            " mbda)                                                            'tf.__operators__.add_1[0][0]'] \n",
            "                                                                                                  \n",
            " layer_normalization_3 (LayerNo  (None, 100, 128)    256         ['tf.__operators__.add_2[0][0]'] \n",
            " rmalization)                                                                                     \n",
            "                                                                                                  \n",
            " conv1d_2 (Conv1D)              (None, 100, 4)       516         ['layer_normalization_3[0][0]']  \n",
            "                                                                                                  \n",
            " dropout_3 (Dropout)            (None, 100, 4)       0           ['conv1d_2[0][0]']               \n",
            "                                                                                                  \n",
            " conv1d_3 (Conv1D)              (None, 100, 128)     640         ['dropout_3[0][0]']              \n",
            "                                                                                                  \n",
            " tf.__operators__.add_3 (TFOpLa  (None, 100, 128)    0           ['conv1d_3[0][0]',               \n",
            " mbda)                                                            'tf.__operators__.add_2[0][0]'] \n",
            "                                                                                                  \n",
            " layer_normalization_4 (LayerNo  (None, 100, 128)    256         ['tf.__operators__.add_3[0][0]'] \n",
            " rmalization)                                                                                     \n",
            "                                                                                                  \n",
            " multi_head_attention_2 (MultiH  (None, 100, 128)    527488      ['layer_normalization_4[0][0]',  \n",
            " eadAttention)                                                    'layer_normalization_4[0][0]']  \n",
            "                                                                                                  \n",
            " dropout_4 (Dropout)            (None, 100, 128)     0           ['multi_head_attention_2[0][0]'] \n",
            "                                                                                                  \n",
            " tf.__operators__.add_4 (TFOpLa  (None, 100, 128)    0           ['dropout_4[0][0]',              \n",
            " mbda)                                                            'tf.__operators__.add_3[0][0]'] \n",
            "                                                                                                  \n",
            " layer_normalization_5 (LayerNo  (None, 100, 128)    256         ['tf.__operators__.add_4[0][0]'] \n",
            " rmalization)                                                                                     \n",
            "                                                                                                  \n",
            " conv1d_4 (Conv1D)              (None, 100, 4)       516         ['layer_normalization_5[0][0]']  \n",
            "                                                                                                  \n",
            " dropout_5 (Dropout)            (None, 100, 4)       0           ['conv1d_4[0][0]']               \n",
            "                                                                                                  \n",
            " conv1d_5 (Conv1D)              (None, 100, 128)     640         ['dropout_5[0][0]']              \n",
            "                                                                                                  \n",
            " tf.__operators__.add_5 (TFOpLa  (None, 100, 128)    0           ['conv1d_5[0][0]',               \n",
            " mbda)                                                            'tf.__operators__.add_4[0][0]'] \n",
            "                                                                                                  \n",
            " layer_normalization_6 (LayerNo  (None, 100, 128)    256         ['tf.__operators__.add_5[0][0]'] \n",
            " rmalization)                                                                                     \n",
            "                                                                                                  \n",
            " multi_head_attention_3 (MultiH  (None, 100, 128)    527488      ['layer_normalization_6[0][0]',  \n",
            " eadAttention)                                                    'layer_normalization_6[0][0]']  \n",
            "                                                                                                  \n",
            " dropout_6 (Dropout)            (None, 100, 128)     0           ['multi_head_attention_3[0][0]'] \n",
            "                                                                                                  \n",
            " tf.__operators__.add_6 (TFOpLa  (None, 100, 128)    0           ['dropout_6[0][0]',              \n",
            " mbda)                                                            'tf.__operators__.add_5[0][0]'] \n",
            "                                                                                                  \n",
            " layer_normalization_7 (LayerNo  (None, 100, 128)    256         ['tf.__operators__.add_6[0][0]'] \n",
            " rmalization)                                                                                     \n",
            "                                                                                                  \n",
            " conv1d_6 (Conv1D)              (None, 100, 4)       516         ['layer_normalization_7[0][0]']  \n",
            "                                                                                                  \n",
            " dropout_7 (Dropout)            (None, 100, 4)       0           ['conv1d_6[0][0]']               \n",
            "                                                                                                  \n",
            " conv1d_7 (Conv1D)              (None, 100, 128)     640         ['dropout_7[0][0]']              \n",
            "                                                                                                  \n",
            " tf.__operators__.add_7 (TFOpLa  (None, 100, 128)    0           ['conv1d_7[0][0]',               \n",
            " mbda)                                                            'tf.__operators__.add_6[0][0]'] \n",
            "                                                                                                  \n",
            " global_average_pooling1d (Glob  (None, 100)         0           ['tf.__operators__.add_7[0][0]'] \n",
            " alAveragePooling1D)                                                                              \n",
            "                                                                                                  \n",
            " dense (Dense)                  (None, 128)          12928       ['global_average_pooling1d[0][0]'\n",
            "                                                                 ]                                \n",
            "                                                                                                  \n",
            " dropout_8 (Dropout)            (None, 128)          0           ['dense[0][0]']                  \n",
            "                                                                                                  \n",
            " dense_1 (Dense)                (None, 1)            129         ['dropout_8[0][0]']              \n",
            "                                                                                                  \n",
            "==================================================================================================\n",
            "Total params: 4,689,681\n",
            "Trainable params: 4,689,681\n",
            "Non-trainable params: 0\n",
            "__________________________________________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "def transformer_encoder(inputs, head_size, num_heads, ff_dim, dropout=0):\n",
        "    \"\"\"\n",
        "    Apply a Transformer Encoder block to the input tensor.\n",
        "\n",
        "    Arguments:\n",
        "        - inputs : tensor of shape (batch_size, sequence_length, hidden_size)\n",
        "            representing the input sequence.\n",
        "        - head_size : integer representing the dimensionality of the keys\n",
        "            and values in each attention head.\n",
        "        - num_heads : integer representing the number of attention heads.\n",
        "        - ff_dim : integer representing the number of filters in the feedforward layer.\n",
        "        - dropout : float between 0 and 1 representing the dropout rate.\n",
        "\n",
        "    Returns:\n",
        "        - tensor of shape (batch_size, sequence_length, hidden_size) representing\n",
        "            the output of the Transformer Encoder block.\n",
        "    \"\"\"\n",
        "    x = tf.keras.layers.LayerNormalization(epsilon=1e-6)(inputs)\n",
        "    x = tf.keras.layers.MultiHeadAttention(\n",
        "        key_dim=head_size, num_heads=num_heads, dropout=dropout\n",
        "    )(x, x)\n",
        "    x = tf.keras.layers.Dropout(dropout)(x)\n",
        "    res = x + inputs\n",
        "\n",
        "    # Feed Forward Part\n",
        "    x = tf.keras.layers.LayerNormalization(epsilon=1e-6)(res)\n",
        "    x = tf.keras.layers.Conv1D(filters=ff_dim, kernel_size=1, activation=\"relu\")(x)\n",
        "    x = tf.keras.layers.Dropout(dropout)(x)\n",
        "    x = tf.keras.layers.Conv1D(filters=inputs.shape[-1], kernel_size=1)(x)\n",
        "    return x + res\n",
        "\n",
        "def build_model(\n",
        "    input_shape,\n",
        "    head_size,\n",
        "    num_heads,\n",
        "    ff_dim,\n",
        "    num_transformer_blocks,\n",
        "    mlp_units,\n",
        "    dropout=0,\n",
        "    mlp_dropout=0):\n",
        "\n",
        "    vocab_size = 20000\n",
        "    sequence_length = 100\n",
        "    embed_size = 128\n",
        "\n",
        "    inputs = tf.keras.Input(shape=input_shape, dtype=\"int32\")\n",
        "    x = tf.keras.layers.Embedding(input_dim=vocab_size,\n",
        "                              output_dim=embed_size,\n",
        "                              input_length=sequence_length)(inputs)\n",
        "    for _ in range(num_transformer_blocks):\n",
        "        x = transformer_encoder(x, head_size, num_heads, ff_dim, dropout)\n",
        "\n",
        "    x = tf.keras.layers.GlobalAveragePooling1D(data_format=\"channels_first\")(x)\n",
        "    for dim in mlp_units:\n",
        "        x = tf.keras.layers.Dense(dim, activation=\"relu\")(x)\n",
        "        x = tf.keras.layers.Dropout(mlp_dropout)(x)\n",
        "    outputs = tf.keras.layers.Dense(1, activation=\"sigmoid\")(x)\n",
        "    return tf.keras.Model(inputs, outputs)\n",
        "\n",
        "\n",
        "model = build_model(\n",
        "    (x_train.shape[1]),\n",
        "    head_size=256,\n",
        "    num_heads=4,\n",
        "    ff_dim=4,\n",
        "    num_transformer_blocks=4,\n",
        "    mlp_units=[128],\n",
        "    mlp_dropout=0.4,\n",
        "    dropout=0.2,\n",
        ")\n",
        "\n",
        "model.compile(loss=tf.losses.BinaryCrossentropy(),\n",
        "              optimizer='adam',\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "model._name = 'ToxicityModel'\n",
        "model.summary()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pw9l2aNJzGIC"
      },
      "source": [
        "Below, we will train this model for one epoch using a T4 GPU (on Colab). However, you can also download a model trained on 20 epochs (+ learned vocabulary) by cloning the available repo on the hub 🤗.\n",
        "\n",
        "```bash\n",
        "!git lfs install\n",
        "!git clone https://huggingface.co/dieineb/toxicity-classifier\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xY06MeGauycE",
        "outputId": "43d1850a-ec5f-45ef-a88c-d80079923796"
      },
      "outputs": [],
      "source": [
        "print(\"Version: \", tf.__version__)\n",
        "print(\"Eager mode: \", tf.executing_eagerly())\n",
        "print(\"GPU is\", \"available\" if tf.config.list_physical_devices('GPU') else \"NOT AVAILABLE\")\n",
        "\n",
        "\n",
        "callbacks = [tf.keras.callbacks.ModelCheckpoint(\"toxicity-model.keras\",\n",
        "                                                save_best_only=True),\n",
        "            tf.keras.callbacks.EarlyStopping(monitor=\"val_loss\",\n",
        "                                            patience=5,\n",
        "                                            verbose=1,\n",
        "                                            mode=\"auto\",\n",
        "                                            baseline=None,\n",
        "                                            restore_best_weights=True)]\n",
        "model.fit(x_train,\n",
        "          y_train,\n",
        "          validation_split = 0.2,\n",
        "          epochs=1,\n",
        "          batch_size=16,\n",
        "          verbose=1,\n",
        "          callbacks=callbacks)\n",
        "\n",
        "test_loss_score, test_acc_score = model.evaluate(x_test, y_test)\n",
        "\n",
        "print(f'Final Loss: {test_loss_score:.2f}.')\n",
        "print(f'Final Performance: {test_acc_score * 100:.2f} %.')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XdsvZOVn0Udu"
      },
      "source": [
        "To clone the pre-trained model from huggingface.co, run this line on your terminal\n",
        "\n",
        "```bash\n",
        "git clone https://huggingface.co/AiresPucrs/toxicity-classifier\n",
        "```\n",
        "\n",
        "Or run the cell below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Cloned repo!\n"
          ]
        }
      ],
      "source": [
        "import os\n",
        "\n",
        "os.system(\"git clone https://huggingface.co/AiresPucrs/toxicity-classifier\")\n",
        "print(\"Cloned repo!\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "To load the model, we simply create a new vectorization layer and load the saved vocabulary, and with the `tf.keras.models.load_model`, you can load the weights and architecture of any in-built model that uses the layers from the Keras API."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xGn1SBIEYTaS",
        "outputId": "7569ce7b-a3c9-4d6f-e060-f909fbba1412"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "I think you should shut up your big mouth\n",
            "\n",
            "Toxic 🤬 95.73% | Not toxic 😊 4.27\n",
            "\n",
            "__________________________________________________\n",
            "I do not agree with you\n",
            "\n",
            "Toxic 🤬 0.99% | Not toxic 😊 99.01\n",
            "\n",
            "__________________________________________________\n"
          ]
        }
      ],
      "source": [
        "# If you cloned the model repo, the path is toxicity_model/toxicity_model.keras\n",
        "\n",
        "toxicity_model = tf.keras.models.load_model('./toxicity-classifier/toxicity-model.keras')\n",
        "\n",
        "# If you cloned the model repo, the path is toxicity_model/toxic_vocabulary.txt\n",
        "\n",
        "with open('toxic-vocabulary.txt', encoding='utf-8') as fp:\n",
        "    vocabulary = [line.strip() for line in fp]\n",
        "    fp.close()\n",
        "\n",
        "vectorization_layer = tf.keras.layers.TextVectorization(max_tokens=20000,\n",
        "                                        output_mode=\"int\",\n",
        "                                        output_sequence_length=100,\n",
        "                                        vocabulary=vocabulary)\n",
        "\n",
        "strings = [\n",
        "    'I think you should shut up your big mouth',\n",
        "    'I do not agree with you'\n",
        "]\n",
        "\n",
        "preds = toxicity_model.predict(vectorization_layer(strings),verbose=0)\n",
        "\n",
        "for i, string in enumerate(strings):\n",
        "    print(f'{string}\\n')\n",
        "    print(f'Toxic 🤬 {(1 - preds[i][0]) * 100:.2f)}% | Not toxic 😊 {preds[i][0] * 100:.2f}\\n')\n",
        "    print(\"_\" * 50)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EkczWCgdYTaT"
      },
      "source": [
        "Moderation services use ML models, like the one we just made, to help make online forums safe spaces. Our model seems to discern between a \"_polite reply_\" and a \"_rude interjection_\", but a more careful analysis made reveal its vulnerabilities (something beyond the point of this tutorial).\n",
        "\n",
        "You can also try to repurpose this architecture for other applications and tasks, like multiclass classification (e.g., by changing the function of the last layer to a `softmax` with the intended number of classes to your problem) instead of binary classification.\n",
        "\n",
        "---\n",
        "\n",
        "Return to the [castle](https://github.com/Nkluge-correa/TeenyTinyCastle)."
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "T4",
      "machine_shape": "hm",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.13"
    },
    "orig_nbformat": 4,
    "vscode": {
      "interpreter": {
        "hash": "aca09746cf57686f00a55ae76e987247ecfb5dd0b3b2e2474d8dbbf0c5e3377e"
      }
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "1561b8c5dd544402b0a76cd3b458d5dc": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "193d60122b014a72a3a1608cbefcec10": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_23d0a0b2bbeb4327817609e21b1d6363",
            "placeholder": "​",
            "style": "IPY_MODEL_7c572c6f7a634f97bdf53f78159d4fbd",
            "value": " 1/1 [00:06&lt;00:00,  6.24s/it]"
          }
        },
        "1ca8ce3e01ba42948a1e4493ec2e1a3a": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_b2a9d77d113943f08a10560bb2454ee7",
              "IPY_MODEL_e9a67b62a8da40289eedde858138984e",
              "IPY_MODEL_193d60122b014a72a3a1608cbefcec10"
            ],
            "layout": "IPY_MODEL_c34f8d6666794bb7a26cde0b9afcc6df"
          }
        },
        "23d0a0b2bbeb4327817609e21b1d6363": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2d444946e6b3418caad9b8b22489a6b7": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_aa061a3603e04520a80f2e662382e970",
            "max": 1,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_da141a2bcb88463490ccb93f1481aa5f",
            "value": 1
          }
        },
        "2e3b21a596cd4404888f72098b0750dd": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "321b102134ef408385f3156b52f14296": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "35c59acf797d46d1be105336898beb9b": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "36deacf6e5294414b3ca37af271c4419": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "3d310d773c9b44b5a68b60b1e465c08f": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_cda4da6ab570438db82f4c5140da3079",
            "placeholder": "​",
            "style": "IPY_MODEL_565ab0d44b5040f0a94abaeabd488e67",
            "value": " 1/1 [00:00&lt;00:00, 58.75it/s]"
          }
        },
        "493ab53d65594c39a6fc0408f7d95353": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_35c59acf797d46d1be105336898beb9b",
            "placeholder": "​",
            "style": "IPY_MODEL_c221e567eefa42e5acc9d7416445ca07",
            "value": " 70157/0 [00:00&lt;00:00, 260804.20 examples/s]"
          }
        },
        "496f6eb8158d4ed5b11d2f75fdd14c6d": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_a207cf549e934a0387c910e1499d1c05",
              "IPY_MODEL_2d444946e6b3418caad9b8b22489a6b7",
              "IPY_MODEL_493ab53d65594c39a6fc0408f7d95353"
            ],
            "layout": "IPY_MODEL_bd19ea0e14ba42b1babff7664189ef3e"
          }
        },
        "565ab0d44b5040f0a94abaeabd488e67": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "57fe3fc671a34c63b1458e64cd0b5d91": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_1561b8c5dd544402b0a76cd3b458d5dc",
            "placeholder": "​",
            "style": "IPY_MODEL_b6bc31f1f36443c9987fba4f4367240c",
            "value": " 15.5M/15.5M [00:06&lt;00:00, 2.16MB/s]"
          }
        },
        "5a87b28067c04e70b8636fe3361f5782": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5afe88be584a4671b73389106f917f47": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "5c3878a63ff346dc9cd06ec3bd1c9cb0": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ad474ef8888f42e8b5b59c1fed78fee1",
            "max": 1,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_5afe88be584a4671b73389106f917f47",
            "value": 1
          }
        },
        "66edae1cc2704e29b360433d7daaf88f": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "722d08759d3a4c8b8362b3210012a43d": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "73923227531f43ea8afc94e4fc0cd176": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "75525b3760744d1d9620cd74815a4936": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "7c572c6f7a634f97bdf53f78159d4fbd": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "7eb5b288160b4760bc1ebf291878114a": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8269280765314b7a9c854f6859d60bef": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_b13a5a530d2c42fb96fa8f6025e186de",
              "IPY_MODEL_bf56868c2b734528930c6a7a8a8473ca",
              "IPY_MODEL_57fe3fc671a34c63b1458e64cd0b5d91"
            ],
            "layout": "IPY_MODEL_66edae1cc2704e29b360433d7daaf88f"
          }
        },
        "83c822d4f81d4bc38f44659049b280dc": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "98d0896622454db68579abe0fb09781e": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "a207cf549e934a0387c910e1499d1c05": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_5a87b28067c04e70b8636fe3361f5782",
            "placeholder": "​",
            "style": "IPY_MODEL_321b102134ef408385f3156b52f14296",
            "value": "Generating train split: "
          }
        },
        "aa061a3603e04520a80f2e662382e970": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": "20px"
          }
        },
        "ad474ef8888f42e8b5b59c1fed78fee1": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b13a5a530d2c42fb96fa8f6025e186de": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_73923227531f43ea8afc94e4fc0cd176",
            "placeholder": "​",
            "style": "IPY_MODEL_722d08759d3a4c8b8362b3210012a43d",
            "value": "Downloading data: 100%"
          }
        },
        "b2a9d77d113943f08a10560bb2454ee7": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_75525b3760744d1d9620cd74815a4936",
            "placeholder": "​",
            "style": "IPY_MODEL_83c822d4f81d4bc38f44659049b280dc",
            "value": "Downloading data files: 100%"
          }
        },
        "b6bc31f1f36443c9987fba4f4367240c": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "bd19ea0e14ba42b1babff7664189ef3e": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "bf56868c2b734528930c6a7a8a8473ca": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_2e3b21a596cd4404888f72098b0750dd",
            "max": 15526478,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_36deacf6e5294414b3ca37af271c4419",
            "value": 15526478
          }
        },
        "c1bc3c9b1b6444c6887980230c9d254a": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_98d0896622454db68579abe0fb09781e",
            "placeholder": "​",
            "style": "IPY_MODEL_e5465f649db645abb8b18c2f8564fb2e",
            "value": "Extracting data files: 100%"
          }
        },
        "c221e567eefa42e5acc9d7416445ca07": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "c34f8d6666794bb7a26cde0b9afcc6df": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "cda4da6ab570438db82f4c5140da3079": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d2294fdc3bd5451d94473f09306cb215": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_c1bc3c9b1b6444c6887980230c9d254a",
              "IPY_MODEL_5c3878a63ff346dc9cd06ec3bd1c9cb0",
              "IPY_MODEL_3d310d773c9b44b5a68b60b1e465c08f"
            ],
            "layout": "IPY_MODEL_7eb5b288160b4760bc1ebf291878114a"
          }
        },
        "da141a2bcb88463490ccb93f1481aa5f": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "e22945e6a43b48f681eb7805edb0f8f7": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e5465f649db645abb8b18c2f8564fb2e": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "e9a67b62a8da40289eedde858138984e": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e22945e6a43b48f681eb7805edb0f8f7",
            "max": 1,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_ee23a3ea57de44d69636a030dbedbc3e",
            "value": 1
          }
        },
        "ee23a3ea57de44d69636a030dbedbc3e": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}