NLP/Text_Summarization_ BART _T5_Pegasus.ipynb

{
  "cells": [
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "5tJrazWj0mJH"
      },
      "source": [
        "<a href=\"https://www.youtube.com/watch?v=HDSNjrxSwqw&list=PLxqBkZuBynVQEvXfJpq3smfuKq3AiNW-N&index=14\"><h1 style=\"font-size:250%; font-family:cursive; color:#ff6666;\"><b>Link to my YouTube Video - Text Summarization with Transformer - BART + T5 + Pegasus</b></h1></a>\n",
        "\n",
        "[![IMAGE ALT TEXT](https://imgur.com/YNWfb8Q.png)](https://www.youtube.com/watch?v=HDSNjrxSwqw&list=PLxqBkZuBynVQEvXfJpq3smfuKq3AiNW-N&index=14)\n"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## First What is BERT?\n",
        "\n",
        "BERT stands for Bidirectional Encoder Representations from Transformers. The name itself gives us several clues to what BERT is all about.\n",
        "\n",
        "BERT architecture consists of several Transformer encoders stacked together. Each Transformer encoder encapsulates two sub-layers: a self-attention layer and a feed-forward layer.\n",
        "\n",
        "### There are two different BERT models:\n",
        "\n",
        "- BERT base, which is a BERT model consists of 12 layers of Transformer encoder, 12 attention heads, 768 hidden size, and 110M parameters.\n",
        "\n",
        "- BERT large, which is a BERT model consists of 24 layers of Transformer encoder,16 attention heads, 1024 hidden size, and 340 parameters.\n",
        "\n",
        "\n",
        "\n",
        "BERT Input and Output\n",
        "BERT model expects a sequence of tokens (words) as an input. In each sequence of tokens, there are two special tokens that BERT would expect as an input:\n",
        "\n",
        "- [CLS]: This is the first token of every sequence, which stands for classification token.\n",
        "- [SEP]: This is the token that makes BERT know which token belongs to which sequence. This special token is mainly important for a next sentence prediction task or question-answering task. If we only have one sequence, then this token will be appended to the end of the sequence.\n",
        "\n",
        "\n",
        "It is also important to note that the maximum size of tokens that can be fed into BERT model is 512. If the tokens in a sequence are less than 512, we can use padding to fill the unused token slots with [PAD] token. If the tokens in a sequence are longer than 512, then we need to do a truncation.\n",
        "\n",
        "And that’s all that BERT expects as input.\n",
        "\n",
        "BERT model then will output an embedding vector of size 768 in each of the tokens. We can use these vectors as an input for different kinds of NLP applications, whether it is text classification, next sentence prediction, Named-Entity-Recognition (NER), or question-answering.\n",
        "\n",
        "\n",
        "------------\n",
        "\n",
        "**For a text classification task**, we focus our attention on the embedding vector output from the special [CLS] token. This means that we’re going to use the embedding vector of size 768 from [CLS] token as an input for our classifier, which then will output a vector of size the number of classes in our classification task.\n",
        "\n",
        "-----------------------\n",
        "\n",
        "![Imgur](https://imgur.com/NpeB9vb.png)\n",
        "\n",
        "-------------------------"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {
        "id": "4rQG3XA0vnS3"
      },
      "outputs": [],
      "source": [
        "!pip install transformers[sentencepiece] datasets sacrebleu rouge_score py7zr -q"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JyGMPbRXvGUL",
        "outputId": "98f7fb96-fb8c-449a-d346-85763d1a1b6b"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Package punkt is already up-to-date!\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "execution_count": 29,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "\n",
        "from transformers import pipeline, set_seed\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "import pandas as pd\n",
        "from datasets import load_dataset, load_metric\n",
        "from transformers import AutoModelForSeq2SeqLM, AutoTokenizer\n",
        "\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "\n",
        "import nltk\n",
        "from nltk.tokenize import sent_tokenize\n",
        "\n",
        "nltk.download(\"punkt\")"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "vQGzessMvGUN"
      },
      "source": [
        "## [The CNN/DailyMail Dataset](https://huggingface.co/datasets/cnn_dailymail)\n",
        "\n",
        "##  An important aspect of the dataset is that the summaries are abstractive and not extractive, which means that they consist of new sentences instead of simple excerpts.\n",
        "\n",
        "Extractive Summarization: the extractive approach selects the most important phrases and lines from the documents. It then combines all the important lines to create the summary. So, in this case, every line and word of the summary actually belongs to the original document which is summarized.\n",
        "\n",
        "Abstractive Summarization: The abstractive approach uses new phrases and terms that are different from the original document, keeping the meaning the same, just like how humans do in summarization. So, it is much harder than the extractive approach."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 121,
          "referenced_widgets": [
            "ae50112f680e446e87368ad81147235e",
            "68cd6ca7aa40454195ccc26f9d870cb8",
            "d7e6521389644dcaabe20c16fc20a3eb",
            "13c185bb27074738998f4eb99c007505",
            "5ccbc79e76f44dcf86ccc02be27b4910",
            "3ff1d017bfda485bb965e897f14e3f70",
            "ae75682934dc4077967b76fdbd194233",
            "ef044cc6232249d4a80cb876dc889395",
            "4008c0bbaea645a4bd95c26672cce12e",
            "7597e26176f643a4873a9b11a5778af8",
            "c02fa16b6d0a497e89e55c63b0427ddf"
          ]
        },
        "id": "2cJKBLSCvGUO",
        "outputId": "dc143cad-954c-4573-fe96-cc2761228956"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "WARNING:datasets.builder:Using custom data configuration default\n",
            "WARNING:datasets.builder:Reusing dataset cnn_dailymail (/root/.cache/huggingface/datasets/cnn_dailymail/default/3.0.0/1b3c71476f6d152c31c1730e83ccb08bcf23e348233f4fcc11e182248e6bf7de)\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "ae50112f680e446e87368ad81147235e",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "  0%|          | 0/3 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Features in cnn_dailymail : ['article', 'highlights', 'id']\n"
          ]
        }
      ],
      "source": [
        "from datasets import load_dataset\n",
        "\n",
        "dataset = load_dataset(\"cnn_dailymail\", version=\"3.0.0\")\n",
        "\n",
        "print(f\"Features in cnn_dailymail : {dataset['train'].column_names}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NpbiUHKDvGUO",
        "outputId": "9698dac0-5c92-4f3a-89dc-a8996b14d4f3"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "Article (excerpt of 500 characters, total length: 4051):\n",
            "\n",
            "Editor's note: In our Behind the Scenes series, CNN correspondents share their experiences in covering news and analyze the stories behind the events. Here, Soledad O'Brien takes users inside a jail where many of the inmates are mentally ill. An inmate housed on the \"forgotten floor,\" where many mentally ill inmates are housed in Miami before trial. MIAMI, Florida (CNN) -- The ninth floor of the Miami-Dade pretrial detention facility is dubbed the \"forgotten floor.\" Here, inmates with the most s\n",
            "\n",
            "Summary (length: 281):\n",
            "Mentally ill inmates in Miami are housed on the \"forgotten floor\"\n",
            "Judge Steven Leifman says most are there as a result of \"avoidable felonies\"\n",
            "While CNN tours facility, patient shouts: \"I am the son of the president\"\n",
            "Leifman says the system is unjust and he's fighting for change .\n"
          ]
        }
      ],
      "source": [
        "sample = dataset[\"train\"][1]\n",
        "print(f\"\"\"\n",
        "Article (excerpt of 500 characters, total length: {len(sample[\"article\"])}):\n",
        "\"\"\")\n",
        "print(sample[\"article\"][:500])\n",
        "print(f'\\nSummary (length: {len(sample[\"highlights\"])}):')\n",
        "print(sample[\"highlights\"])"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "YZNdu7bcvGUP"
      },
      "source": [
        "-------------------\n",
        "\n",
        "## Text Summarization Pipelines\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "metadata": {
        "id": "VR4hgH04vGUQ"
      },
      "outputs": [],
      "source": [
        "sample_text = dataset[\"train\"][1][\"article\"][:1000]\n",
        "\n",
        "# We'll collect the generated summaries of each model in a dictionary\n",
        "summaries = {}"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "Tr1g3imGvGUR"
      },
      "source": [
        "### Summarization Baseline\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 33,
      "metadata": {
        "id": "9aUXF1bQ0geQ"
      },
      "outputs": [],
      "source": [
        "def baseline_summary_three_sent(text):\n",
        "    return \"\\n\".join(sent_tokenize(text)[:3])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 34,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 70
        },
        "id": "cBIan-VI0gbU",
        "outputId": "22fda354-c05b-418e-c8b7-70bf0a153504"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'Editor\\'s note: In our Behind the Scenes series, CNN correspondents share their experiences in covering news and analyze the stories behind the events.\\nHere, Soledad O\\'Brien takes users inside a jail where many of the inmates are mentally ill. An inmate housed on the \"forgotten floor,\" where many mentally ill inmates are housed in Miami before trial.\\nMIAMI, Florida (CNN) -- The ninth floor of the Miami-Dade pretrial detention facility is dubbed the \"forgotten floor.\"'"
            ]
          },
          "execution_count": 34,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "summaries['baseline'] = baseline_summary_three_sent(sample_text)\n",
        "\n",
        "summaries['baseline']"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# huggingface pipeline\n",
        "\n",
        "The pipelines are a great and easy way to use models for inference.\n",
        "\n",
        "Stating ”summarization”: will return a `SummarizationPipeline`\n",
        "\n",
        "”text-generation”: will return a TextGenerationPipeline\n",
        "\n",
        "-----------------------\n",
        "\n",
        "# GPT-2\n",
        "\n",
        "We can use GPT-2 it to generate summaries by simply appending “TL;DR” at the end of the input text.\n",
        "\n",
        "The expression “TL;DR” (too long; didn’t read) is often used on platforms like\n",
        "Reddit to indicate a short version of a long post. We will start our\n",
        "summarization experiment by re-creating the procedure of the original paper\n",
        "with the pipeline() function from Transformers\n",
        "\n",
        "We create a text generation pipeline and load the GPT-2 model:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 35,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DccrU9QR0gYY",
        "outputId": "f11bb1f5-8061-47aa-cc23-5e8101865c05"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
          ]
        }
      ],
      "source": [
        "from transformers import pipeline, set_seed\n",
        "\n",
        "set_seed(42)\n",
        "\n",
        "pipe = pipeline('text-generation', model = 'gpt2-medium' )\n",
        "\n",
        "gpt2_query = sample_text + \"\\nTL;DR:\\n\"\n",
        "\n",
        "pipe_out = pipe(gpt2_query, max_length = 512, clean_up_tokenization_spaces = True)\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 36,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eMrXmX0l0gVd",
        "outputId": "6241794e-4899-424e-edcf-42bc1c3599a1"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[{'generated_text': 'Editor\\'s note: In our Behind the Scenes series, CNN correspondents share their experiences in covering news and analyze the stories behind the events. Here, Soledad O\\'Brien takes users inside a jail where many of the inmates are mentally ill. An inmate housed on the \"forgotten floor,\" where many mentally ill inmates are housed in Miami before trial. MIAMI, Florida (CNN) -- The ninth floor of the Miami-Dade pretrial detention facility is dubbed the \"forgotten floor.\" Here, inmates with the most severe mental illnesses are incarcerated until they\\'re ready to appear in court. Most often, they face drug charges or charges of assaulting an officer --charges that Judge Steven Leifman says are usually \"avoidable felonies.\" He says the arrests often result from confrontations with police. Mentally ill people often won\\'t do what they\\'re told when police arrive on the scene -- confrontation seems to exacerbate their illness and they become more paranoid, delusional, and less likely to follow dir\\nTL;DR:\\nMIAMI-DADE, Florida | April 13, 2012 -- Some inmates are locked up in solitary confinement, where they must be isolated from the world for six months before they\\'re released onto the general jail population to participate in other inmates. Others are housed in \"theforgotten floor,\" the top floor of the pretrial facility that is also used by criminal offenders. What makes these inmates separate is their medical needs. They\\'re given special medications to treat their mental illness, but it\\'s only in those cases where necessary to take certain medications themselves for the very dangerous condition. These medications can lead to anaphylaxis or dangerous reactions if taken by people who are on them alone. A mentally ill person like John Brown, a convicted felon with severe mental illness. He spends four months in the \"forgotten floor\" and is charged with being a felon, disorderly conduct, assault with a deadly weapon (a gun), possession of child pornography, resisting arrest, and possession of marijuana. Brown was in the second floor when he fell asleep in bed. On April 11, his bed fell over and he was arrested and held until jail officials could check on him. After his detention, Brown refused to return to court the next morning to answer the charges. \"I won\\'t be back at all,\" Brown said. \"There will be no new hearings or there will be no court.\" This episode will air on January 18, 2013 on CNN\\'s Inside Story.'}]"
            ]
          },
          "execution_count": 36,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "pipe_out"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 37,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 140
        },
        "id": "nZtMSEro0gSu",
        "outputId": "a4de6061-86bc-4e67-e843-81b7b310b8d5"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'MIAMI-DADE, Florida | April 13, 2012 -- Some inmates are locked up in solitary confinement, where they must be isolated from the world for six months before they\\'re released onto the general jail population to participate in other inmates. Others are housed in \"theforgotten floor,\" the top floor of the pretrial facility that is also used by criminal offenders. What makes these inmates separate is their medical needs. They\\'re given special medications to treat their mental illness, but it\\'s only in those cases where necessary to take certain medications themselves for the very dangerous condition. These medications can lead to anaphylaxis or dangerous reactions if taken by people who are on them alone. A mentally ill person like John Brown, a convicted felon with severe mental illness. He spends four months in the \"forgotten floor\" and is charged with being a felon, disorderly conduct, assault with a deadly weapon (a gun), possession of child pornography, resisting arrest, and possession of marijuana. Brown was in the second floor when he fell asleep in bed. On April 11, his bed fell over and he was arrested and held until jail officials could check on him. After his detention, Brown refused to return to court the next morning to answer the charges. \"I won\\'t be back at all,\" Brown said. \"There will be no new hearings or there will be no court.\" This episode will air on January 18, 2013 on CNN\\'s Inside Story.'"
            ]
          },
          "execution_count": 37,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "pipe_out[0][\"generated_text\"][len(gpt2_query) :]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 38,
      "metadata": {
        "id": "K5jwOJpA0gPg"
      },
      "outputs": [],
      "source": [
        "summaries['gpt2'] = \"\\n\".join(sent_tokenize(pipe_out[0][\"generated_text\"][len(gpt2_query) :]))"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "qjLeUfVW5-48"
      },
      "source": [
        "# T5\n",
        "\n",
        "T5 (Text-To-Text Transfer Transformer) is a transformer model that is trained in an end-to-end manner with text as input and modified text as output, in contrast to BERT-style models that can only output either a class label or a span of the input. This text-to-text formatting makes the T5 model fit for multiple NLP tasks like Summarization, Question-Answering, Machine Translation, and Classification problems.\n",
        "\n",
        "How T5 is different from BERT?\n",
        "Both T5 and BERT are trained with MLM (Masked Language Model) approach. \n",
        "\n",
        "What is MLM?\n",
        "\n",
        "The MLM is a fill-in-the-blank task, where the model masks part of the input text and tries to predict what that masked word should be.\n",
        "\n",
        "Example:\n",
        "\n",
        "“I like to eat peanut butter and <MASK> sandwiches,”\n",
        "“I like to eat peanut butter and jelly sandwiches,”\n",
        "\n",
        "\n",
        "The only difference is that T5 replaces multiple consecutive tokens with the single Mask Keyword, unlike, BERT which uses Mask token for each word. This illustration is shown below.\n",
        "\n",
        "\n",
        "### T5 expects a prefix before the input text to understand the task given by the user. For example,\n",
        "\n",
        "- “summarize:” for the summarization, \n",
        "- “cola sentence:” for the classification, \n",
        "- “translate English to Spanish:” for the machine translation, etc., \n",
        "\n",
        "\n",
        "--------------\n",
        "\n",
        "\n",
        "But here in this case, I can directly load T5 for summarization with the pipeline() function, which also takes care of formatting the inputs in the text-to-text format so we don’t\n",
        "need to prepend them with \"summarize\":\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 39,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "h6j9VqdQ0gM5",
        "outputId": "1b101ada-d3a1-4e1d-a9c4-268812790d89"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/transformers/models/t5/tokenization_t5_fast.py:166: FutureWarning: This tokenizer was incorrectly instantiated with a model max length of 512 which will be corrected in Transformers v5.\n",
            "For now, this behavior is kept to avoid breaking backwards compatibility when padding/encoding with `truncation is True`.\n",
            "- Be aware that you SHOULD NOT rely on t5-small automatically truncating your input to 512 when padding/encoding.\n",
            "- If you want to encode/pad to sequences longer than 512 you can either instantiate this tokenizer with `model_max_length` or pass `max_length` when encoding/padding.\n",
            "- To avoid this warning, please instantiate this tokenizer with `model_max_length` set to your preferred value.\n",
            "  FutureWarning,\n"
          ]
        }
      ],
      "source": [
        "pipe = pipeline('summarization', model = 't5-small' )\n",
        "\n",
        "pipe_out = pipe(sample_text)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 40,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lreykJlF0gJp",
        "outputId": "bf1f56f0-f251-44c5-d0bf-24d9bb000eb1"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[{'summary_text': \"inmates with the most severe mental illnesses are incarcerated until they're ready to appear in court . most often, they face drug charges or charges of assaulting an officer . mentally ill people become more paranoid, delusional, and less likely to follow dir .\"}]"
            ]
          },
          "execution_count": 40,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "pipe_out"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 41,
      "metadata": {
        "id": "veS7tqCk0gGT"
      },
      "outputs": [],
      "source": [
        "summaries['t5'] = 'n'.join(sent_tokenize(pipe_out[0]['summary_text']))"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "cYlign897Sd1"
      },
      "source": [
        "# BART\n",
        "\n",
        "BART is a denoising autoencoder for pretraining sequence-to-sequence models. It is trained by (1) corrupting text with an arbitrary noising function, and (2) learning a model to reconstruct the original text. It uses a standard Transformer-based neural machine translation architecture. \n",
        "\n",
        "That means, It uses a standard seq2seq/NMT architecture with a bidirectional encoder (like BERT) and a left-to-right decoder (like GPT). This means the encoder's attention mask is fully visible, like BERT, and the decoder's attention mask is causal, like GPT2.\n",
        "\n",
        "\n",
        "This means that a fine-tuned BART model can take a text sequence (for example, English) as input and produce a different text sequence at the output (for example, French). \n",
        "\n",
        "This type of model is relevant for machine translation, question-answering , text summarization, or sequence classification (categorizing input text sentences or tokens). \n",
        "\n",
        "Another task is sentence entailment which, given two or more sentences, evaluates whether the sentences are logical extensions or are logically related to a given statement."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 42,
      "metadata": {
        "id": "t5OQKD6H6BSJ"
      },
      "outputs": [],
      "source": [
        "pipe = pipeline(\"summarization\", model=\"facebook/bart-large-cnn\")\n",
        "pipe_out = pipe(sample_text)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 43,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SoZO5EUR6BLr",
        "outputId": "b387e275-040e-4c7d-c482-95b0e536c7bf"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[{'summary_text': 'Miami-Dade pretrial detention facility is dubbed the \"forgotten floor\" Here, inmates with the most severe mental illnesses are incarcerated. Most often, they face drug charges or charges of assaulting an officer. Judge Steven Leifman says the arrests often result from confrontations with police.'}]"
            ]
          },
          "execution_count": 43,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "pipe_out"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 44,
      "metadata": {
        "id": "WLOJIUyF6BIt"
      },
      "outputs": [],
      "source": [
        "summaries[\"bart\"] = \"\\n\".join(sent_tokenize(pipe_out[0][\"summary_text\"]))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 45,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "id": "ZbGxe1q1MIWM",
        "outputId": "345193a7-36c8-40e9-e3ae-8f3cc7768435"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'Miami-Dade pretrial detention facility is dubbed the \"forgotten floor\" Here, inmates with the most severe mental illnesses are incarcerated.\\nMost often, they face drug charges or charges of assaulting an officer.\\nJudge Steven Leifman says the arrests often result from confrontations with police.'"
            ]
          },
          "execution_count": 45,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "summaries[\"bart\"]"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "jmY2JBiG8PlM"
      },
      "source": [
        "# PEGASUS\n",
        "\n",
        "The PEGASUS model’s pre-training task is very similar to summarization, i.e. important sentences are removed and masked from an input document and are later generated together as one output sequence from the remaining sentences, which is fairly similar to a summary. In PEGASUS, several whole sentences are removed from documents during pre-training, and the model is tasked with recovering them. The Input for such pre-training is a document with missing sentences, while the output consists of the missing sentences being concatenated together. The advantage of this self-supervision is that you can create as many examples as there are documents without any human intervention, which often becomes a bottleneck problem in purely supervised systems."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 46,
      "metadata": {
        "id": "Lox7RRHt6BF3"
      },
      "outputs": [],
      "source": [
        "pipe = pipeline('summarization', model=\"google/pegasus-cnn_dailymail\"  )\n",
        "\n",
        "pipe_out = pipe(sample_text)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 47,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-WZQyrM36BCn",
        "outputId": "51aab94d-94ce-4789-bffc-5d2253a6d617"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[{'summary_text': 'Mentally ill inmates are housed on the \"forgotten floor\" of a Miami jail .<n>Judge Steven Leifman says the charges are usually \"avoidable felonies\"<n>He says the arrests often result from confrontations with police .<n>Mentally ill people often won\\'t do what they\\'re told when police arrive on the scene .'}]"
            ]
          },
          "execution_count": 47,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "pipe_out"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 48,
      "metadata": {
        "id": "8w49uN5u6A-z"
      },
      "outputs": [],
      "source": [
        "summaries[\"pegasus\"] = pipe_out[0][\"summary_text\"].replace(\" .<n>\", \".\\n\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 49,
      "metadata": {
        "id": "OIdFjtn49b1I"
      },
      "outputs": [],
      "source": [
        "## Comparing Different Summaries"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 50,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ullRF6UO6A79",
        "outputId": "fbe54ffa-27d2-4020-8c3e-0befc4a5931b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "GROUND TRUTH\n",
            "Mentally ill inmates in Miami are housed on the \"forgotten floor\"\n",
            "Judge Steven Leifman says most are there as a result of \"avoidable felonies\"\n",
            "While CNN tours facility, patient shouts: \"I am the son of the president\"\n",
            "Leifman says the system is unjust and he's fighting for change .\n",
            "BASELINE\n",
            "Editor's note: In our Behind the Scenes series, CNN correspondents share their experiences in covering news and analyze the stories behind the events.\n",
            "Here, Soledad O'Brien takes users inside a jail where many of the inmates are mentally ill. An inmate housed on the \"forgotten floor,\" where many mentally ill inmates are housed in Miami before trial.\n",
            "MIAMI, Florida (CNN) -- The ninth floor of the Miami-Dade pretrial detention facility is dubbed the \"forgotten floor.\"\n",
            "GPT2\n",
            "MIAMI-DADE, Florida | April 13, 2012 -- Some inmates are locked up in solitary confinement, where they must be isolated from the world for six months before they're released onto the general jail population to participate in other inmates.\n",
            "Others are housed in \"theforgotten floor,\" the top floor of the pretrial facility that is also used by criminal offenders.\n",
            "What makes these inmates separate is their medical needs.\n",
            "They're given special medications to treat their mental illness, but it's only in those cases where necessary to take certain medications themselves for the very dangerous condition.\n",
            "These medications can lead to anaphylaxis or dangerous reactions if taken by people who are on them alone.\n",
            "A mentally ill person like John Brown, a convicted felon with severe mental illness.\n",
            "He spends four months in the \"forgotten floor\" and is charged with being a felon, disorderly conduct, assault with a deadly weapon (a gun), possession of child pornography, resisting arrest, and possession of marijuana.\n",
            "Brown was in the second floor when he fell asleep in bed.\n",
            "On April 11, his bed fell over and he was arrested and held until jail officials could check on him.\n",
            "After his detention, Brown refused to return to court the next morning to answer the charges.\n",
            "\"I won't be back at all,\" Brown said.\n",
            "\"There will be no new hearings or there will be no court.\"\n",
            "This episode will air on January 18, 2013 on CNN's Inside Story.\n",
            "T5\n",
            "inmates with the most severe mental illnesses are incarcerated until they're ready to appear in court .nmost often, they face drug charges or charges of assaulting an officer .nmentally ill people become more paranoid, delusional, and less likely to follow dir .\n",
            "BART\n",
            "Miami-Dade pretrial detention facility is dubbed the \"forgotten floor\" Here, inmates with the most severe mental illnesses are incarcerated.\n",
            "Most often, they face drug charges or charges of assaulting an officer.\n",
            "Judge Steven Leifman says the arrests often result from confrontations with police.\n",
            "PEGASUS\n",
            "Mentally ill inmates are housed on the \"forgotten floor\" of a Miami jail.\n",
            "Judge Steven Leifman says the charges are usually \"avoidable felonies\"<n>He says the arrests often result from confrontations with police.\n",
            "Mentally ill people often won't do what they're told when police arrive on the scene .\n"
          ]
        }
      ],
      "source": [
        "print(\"GROUND TRUTH\")\n",
        "\n",
        "print(dataset['train'][1]['highlights'])\n",
        "\n",
        "\n",
        "for model_name in summaries:\n",
        "    print(model_name.upper())\n",
        "    print(summaries[model_name])\n",
        "    "
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "dxP1r4t5-QOP"
      },
      "source": [
        "\n",
        "# SacreBLEU\n",
        "\n",
        "The bleu_metric object is an instance of the Metric class, and works like an\n",
        "aggregator: you can add single instances with add() or whole batches via\n",
        "add_batch(). Once you have added all the samples you need to evaluate, you\n",
        "then call compute() and the metric is calculated. This returns a dictionary with\n",
        "several values, such as the precision for each n-gram, the length penalty, as\n",
        "well as the final BLEU score. Let’s look at the example from before:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 51,
      "metadata": {
        "id": "7S--atct6A42"
      },
      "outputs": [],
      "source": [
        "\n",
        "from datasets import load_metric\n",
        "\n",
        "bleu_metric = load_metric(\"sacrebleu\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 52,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 300
        },
        "id": "m4SyzdJ66A1y",
        "outputId": "af2983a5-a930-4633-fcb6-c915ba7f77ab"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-e35236e0-6ce8-4510-b068-0955044cee18\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Value</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>score</th>\n",
              "      <td>18.73841</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>counts</th>\n",
              "      <td>[27, 14, 10, 6]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>totals</th>\n",
              "      <td>[67, 66, 65, 64]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>precisions</th>\n",
              "      <td>[40.298507462686565, 21.21212121212121, 15.384...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>bp</th>\n",
              "      <td>1.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>sys_len</th>\n",
              "      <td>67</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>ref_len</th>\n",
              "      <td>57</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>precision</th>\n",
              "      <td>[40.3, 21.21, 15.38, 9.38]</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-e35236e0-6ce8-4510-b068-0955044cee18')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-e35236e0-6ce8-4510-b068-0955044cee18 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-e35236e0-6ce8-4510-b068-0955044cee18');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "                                                        Value\n",
              "score                                                18.73841\n",
              "counts                                        [27, 14, 10, 6]\n",
              "totals                                       [67, 66, 65, 64]\n",
              "precisions  [40.298507462686565, 21.21212121212121, 15.384...\n",
              "bp                                                        1.0\n",
              "sys_len                                                    67\n",
              "ref_len                                                    57\n",
              "precision                          [40.3, 21.21, 15.38, 9.38]"
            ]
          },
          "execution_count": 52,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "bleu_metric.add(prediction = [summaries[\"pegasus\"]], reference = [dataset['train'][1]['highlights'] ])\n",
        "\n",
        "results = bleu_metric.compute(smooth_method = 'floor', smooth_value = 0 )\n",
        "\n",
        "results['precision'] = [np.round(p , 2) for p in results['precisions'] ]\n",
        "\n",
        "pd.DataFrame.from_dict(results, orient = 'index', columns = ['Value'] )"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "evBnRNmCNU4H"
      },
      "source": [
        "# ROUGE\n",
        "\n",
        "# ROUGE vs BLEU\n",
        "\n",
        "Bleu measures precision: how much the words (and/or n-grams) in the machine generated summaries appeared in the human reference summaries.\n",
        "\n",
        "Rouge measures recall: how much the words (and/or n-grams) in the human reference summaries appeared in the machine generated summaries.\n",
        "\n",
        "### Interpretation of Rouge Score\n",
        "\n",
        "ROUGE-n recall=40% means that 40% of the n-grams in the reference summary are also present in the generated summary.\n",
        "\n",
        "--------\n",
        "\n",
        "The ROUGE score was specifically developed for applications like\n",
        "summarization where high recall is more important than just precision.5 \n",
        "\n",
        "The approach is very similar to the BLEU score in that we look at different n-grams\n",
        "and compare their occurrences in the generated text and the reference texts.\n",
        "\n",
        "\n",
        "The difference is that with ROUGE we check how many n-grams in the\n",
        "reference text also occur in the generated text. For BLEU we looked at how\n",
        "many n-grams in the generated text appear in the reference"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 53,
      "metadata": {
        "id": "1rJy54Bd9OkP"
      },
      "outputs": [],
      "source": [
        "rouge_metric = load_metric('rouge')"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "GqZPSwDONnq8"
      },
      "source": [
        "## ROUGE-N\n",
        "\n",
        "With ROUGE-N, the N represents the n-gram that we are using. For ROUGE-1 we would be measuring the match-rate of unigrams between our model output and reference.\n",
        "\n",
        "ROUGE-2 and ROUGE-3 would use bigrams and trigrams respectively.\n",
        "\n",
        "\n",
        "## ROUGE-L\n",
        "\n",
        "ROUGE-L measures the longest common subsequence (LCS) between our model output and reference. All this means is that we count the longest sequence of tokens that is shared between both:\n",
        "\n",
        "\n",
        "In the HF Datasets implementation, two variations of ROUGE are\n",
        "calculated: one calculates the score per sentence and averages it for the\n",
        "summaries (ROUGE-L), and the other calculates it directly over the whole\n",
        "summary (ROUGE-Lsum).\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 56,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 293
        },
        "id": "CCNutX-cNnNv",
        "outputId": "4b31402a-bb93-4765-8244-2d6415ee51e7"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "rouge_dict  {'rouge1': 0.365079365079365, 'rouge2': 0.14516129032258066, 'rougeL': 0.20634920634920634, 'rougeLsum': 0.2857142857142857}\n",
            "rouge_dict  {'rouge1': 0.1836734693877551, 'rouge2': 0.0410958904109589, 'rougeL': 0.10204081632653061, 'rougeLsum': 0.17006802721088435}\n",
            "rouge_dict  {'rouge1': 0.1758241758241758, 'rouge2': 0.0, 'rougeL': 0.13186813186813187, 'rougeLsum': 0.15384615384615383}\n",
            "rouge_dict  {'rouge1': 0.3655913978494624, 'rouge2': 0.13186813186813184, 'rougeL': 0.2150537634408602, 'rougeLsum': 0.3225806451612903}\n",
            "rouge_dict  {'rouge1': 0.5, 'rouge2': 0.24489795918367346, 'rougeL': 0.36000000000000004, 'rougeLsum': 0.46}\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-7279dded-a102-4eda-a78b-1e6fc87df40a\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>rouge1</th>\n",
              "      <th>rouge2</th>\n",
              "      <th>rougeL</th>\n",
              "      <th>rougeLsum</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>baseline</th>\n",
              "      <td>0.365079</td>\n",
              "      <td>0.145161</td>\n",
              "      <td>0.206349</td>\n",
              "      <td>0.285714</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>gpt2</th>\n",
              "      <td>0.183673</td>\n",
              "      <td>0.041096</td>\n",
              "      <td>0.102041</td>\n",
              "      <td>0.170068</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>t5</th>\n",
              "      <td>0.175824</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.131868</td>\n",
              "      <td>0.153846</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>bart</th>\n",
              "      <td>0.365591</td>\n",
              "      <td>0.131868</td>\n",
              "      <td>0.215054</td>\n",
              "      <td>0.322581</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>pegasus</th>\n",
              "      <td>0.500000</td>\n",
              "      <td>0.244898</td>\n",
              "      <td>0.360000</td>\n",
              "      <td>0.460000</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-7279dded-a102-4eda-a78b-1e6fc87df40a')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-7279dded-a102-4eda-a78b-1e6fc87df40a button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-7279dded-a102-4eda-a78b-1e6fc87df40a');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "            rouge1    rouge2    rougeL  rougeLsum\n",
              "baseline  0.365079  0.145161  0.206349   0.285714\n",
              "gpt2      0.183673  0.041096  0.102041   0.170068\n",
              "t5        0.175824  0.000000  0.131868   0.153846\n",
              "bart      0.365591  0.131868  0.215054   0.322581\n",
              "pegasus   0.500000  0.244898  0.360000   0.460000"
            ]
          },
          "execution_count": 56,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "rouge_names = [\"rouge1\", \"rouge2\", \"rougeL\", \"rougeLsum\"]\n",
        "\n",
        "reference = dataset['train'][1]['highlights']\n",
        "\n",
        "records = []\n",
        "\n",
        "for model_name in summaries:\n",
        "    rouge_metric.add(prediction = summaries[model_name], reference = reference )\n",
        "    score = rouge_metric.compute()\n",
        "    rouge_dict = dict((rn, score[rn].mid.fmeasure ) for rn in rouge_names )\n",
        "    print('rouge_dict ', rouge_dict )\n",
        "    records.append(rouge_dict)\n",
        "\n",
        "pd.DataFrame.from_records(records, index = summaries.keys() )"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "2n10Vi-s2cKl"
      },
      "source": [
        "# Evaluationg on the TEST set of the CNN/DailyMail Dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 57,
      "metadata": {
        "id": "k_yjJmEu9OhV"
      },
      "outputs": [],
      "source": [
        "def calculate_metric_on_baseline_test_ds(dataset, metric, column_text = 'article', column_summary = 'highlights' ):\n",
        "    \"\"\"\n",
        "    This function calculates a specified metric on a baseline test dataset for a Natural Language Processing (NLP) task.\n",
        "    It assumes the task is a text summarization task, where the goal is to generate a summary (e.g., highlights) from a text (e.g., article).\n",
        "\n",
        "    Parameters:\n",
        "    dataset (pandas.DataFrame): The test dataset. It should contain a column for the text and a column for the true summary.\n",
        "    metric (datasets.Metric): The metric to calculate. This should be a metric object from the Hugging Face datasets library.\n",
        "    column_text (str, optional): The name of the column in the dataset that contains the text. Defaults to 'article'.\n",
        "    column_summary (str, optional): The name of the column in the dataset that contains the true summary. Defaults to 'highlights'.\n",
        "\n",
        "    Returns:\n",
        "    score (float): The calculated score of the metric on the test dataset.\n",
        "    \"\"\"\n",
        "    summaries = [baseline_summary_three_sent(text) for text in dataset[column_text] ]\n",
        "\n",
        "    metric.add_batch(predictions = summaries, references = dataset[column_summary] )\n",
        "\n",
        "    score = metric.compute()\n",
        "    return score"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 61,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 81
        },
        "id": "e8LVc-_F9Oek",
        "outputId": "80c0818b-630d-4873-bf3e-bda3c071dd0c"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "\n",
              "  <div id=\"df-de1b1962-e2b7-44c2-bd14-e6b759ef826c\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>rouge1</th>\n",
              "      <th>rouge2</th>\n",
              "      <th>rougeL</th>\n",
              "      <th>rougeLsum</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>baseline</th>\n",
              "      <td>0.253995</td>\n",
              "      <td>0.100642</td>\n",
              "      <td>0.165754</td>\n",
              "      <td>0.231571</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-de1b1962-e2b7-44c2-bd14-e6b759ef826c')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-de1b1962-e2b7-44c2-bd14-e6b759ef826c button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-de1b1962-e2b7-44c2-bd14-e6b759ef826c');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ],
            "text/plain": [
              "            rouge1    rouge2    rougeL  rougeLsum\n",
              "baseline  0.253995  0.100642  0.165754   0.231571"
            ]
          },
          "execution_count": 61,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "test_sampled = dataset['train'].shuffle(seed = 42).select(range(1000))\n",
        "\n",
        "score = calculate_metric_on_baseline_test_ds(test_sampled, rouge_metric )\n",
        "\n",
        "rouge_dict = dict((rn, score[rn].mid.fmeasure ) for rn in rouge_names )\n",
        "\n",
        "pd.DataFrame.from_dict(rouge_dict, orient = 'index' , columns = ['baseline'] ).T"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "opOiKp2J455L"
      },
      "source": [
        "## Strategy to calculate the ROUGE Metric on test dataset for the other Models"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 62,
      "metadata": {
        "id": "dQlqJ-6cNW5H"
      },
      "outputs": [],
      "source": [
        "from tqdm import tqdm\n",
        "import torch\n",
        "\n",
        "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
        "\n",
        "def generate_batch_sized_chunks(list_of_elements, batch_size):\n",
        "    \"\"\"split the dataset into smaller batches that we can process simultaneously\n",
        "    Yield successive batch-sized chunks from list_of_elements.\n",
        "    \n",
        "    Generator function to yield successive batch-sized chunks from list_of_elements.\n",
        "\n",
        "    Parameters:\n",
        "    list_of_elements (list): List of elements to be divided into chunks.\n",
        "    batch_size (int): The size of each chunk.\n",
        "\n",
        "    Yields:\n",
        "    list: Batch-sized chunk from list_of_elements.\n",
        "    \n",
        "    \"\"\"\n",
        "    for i in range(0, len(list_of_elements), batch_size):\n",
        "        yield list_of_elements[i : i + batch_size]\n",
        "\n",
        "def calculate_metric_on_test_ds(dataset, metric, model, tokenizer, \n",
        "                               batch_size=16, device=device, \n",
        "                               column_text=\"article\", \n",
        "                               column_summary=\"highlights\"):\n",
        "    \"\"\"\n",
        "    Function to calculate a specified metric on a test dataset for a Natural Language Processing (NLP) task.\n",
        "    It assumes the task is a text summarization task, where the goal is to generate a summary from a text.\n",
        "\n",
        "    Parameters:\n",
        "    dataset (pandas.DataFrame): The test dataset. It should contain a column for the text and a column for the true summary.\n",
        "    metric (datasets.Metric): The metric to calculate. This should be a metric object from the Hugging Face datasets library.\n",
        "    model (transformers.PreTrainedModel): The transformer model to use for text generation.\n",
        "    tokenizer (transformers.PreTrainedTokenizer): The tokenizer corresponding to the model.\n",
        "    batch_size (int, optional): The size of the batches to use for processing. Defaults to 16.\n",
        "    device (str, optional): The device to run the model on. Defaults to the output of torch.cuda.is_available().\n",
        "    column_text (str, optional): The name of the column in the dataset that contains the text. Defaults to 'article'.\n",
        "    column_summary (str, optional): The name of the column in the dataset that contains the true summary. Defaults to 'highlights'.\n",
        "\n",
        "    Returns:\n",
        "    score (float): The calculated score of the metric on the test dataset.\n",
        "    \"\"\"\n",
        "    article_batches = list(generate_batch_sized_chunks(dataset[column_text], batch_size))\n",
        "    target_batches = list(generate_batch_sized_chunks(dataset[column_summary], batch_size))\n",
        "\n",
        "    for article_batch, target_batch in tqdm(\n",
        "        zip(article_batches, target_batches), total=len(article_batches)):\n",
        "        \n",
        "        inputs = tokenizer(article_batch, max_length=1024,  truncation=True, \n",
        "                        padding=\"max_length\", return_tensors=\"pt\")\n",
        "        \n",
        "        summaries = model.generate(input_ids=inputs[\"input_ids\"].to(device),\n",
        "                         attention_mask=inputs[\"attention_mask\"].to(device), \n",
        "                         length_penalty=0.8, num_beams=8, max_length=128)\n",
        "        ''' parameter for length penalty ensures that the model does not generate sequences that are too long. '''\n",
        "        \n",
        "        # Finally, we decode the generated texts, \n",
        "        # replace the <n> token, and add the decoded texts with the references to the metric.\n",
        "        decoded_summaries = [tokenizer.decode(s, skip_special_tokens=True, \n",
        "                                clean_up_tokenization_spaces=True) \n",
        "               for s in summaries]      \n",
        "        \n",
        "        decoded_summaries = [d.replace(\"<n>\", \" \") for d in decoded_summaries]\n",
        "        \n",
        "        \n",
        "        metric.add_batch(predictions=decoded_summaries, references=target_batch)\n",
        "        \n",
        "    #  Finally compute and return the ROUGE scores.\n",
        "    score = metric.compute()\n",
        "    return score"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Hud5crl3NW2Y"
      },
      "outputs": [],
      "source": [
        "\n",
        "from transformers import AutoModelForSeq2SeqLM, AutoTokenizer\n",
        "\n",
        "model_ckpt = \"google/pegasus-cnn_dailymail\"\n",
        "\n",
        "tokenizer = AutoTokenizer.from_pretrained(model_ckpt)\n",
        "\n",
        "model_pegasus = AutoModelForSeq2SeqLM.from_pretrained(model_ckpt).to(device)\n",
        "\n",
        "score = calculate_metric_on_test_ds(test_sampled, rouge_metric, \n",
        "                                   model_pegasus, tokenizer, batch_size=8)\n",
        "\n",
        "rouge_dict = dict((rn, score[rn].mid.fmeasure) for rn in rouge_names)\n",
        "\n",
        "# At the end, we compute and return the ROUGE scores.\n",
        "pd.DataFrame(rouge_dict, index=[\"pegasus\"])"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3.9.14 64-bit",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.14"
    },
    "vscode": {
      "interpreter": {
        "hash": "36cf16204b8548560b1c020c4e8fb5b57f0e4c58016f52f2d4be01e192833930"
      }
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "13c185bb27074738998f4eb99c007505": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_7597e26176f643a4873a9b11a5778af8",
            "placeholder": "​",
            "style": "IPY_MODEL_c02fa16b6d0a497e89e55c63b0427ddf",
            "value": " 3/3 [00:00&lt;00:00,  2.75it/s]"
          }
        },
        "3ff1d017bfda485bb965e897f14e3f70": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "4008c0bbaea645a4bd95c26672cce12e": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "5ccbc79e76f44dcf86ccc02be27b4910": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "68cd6ca7aa40454195ccc26f9d870cb8": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_3ff1d017bfda485bb965e897f14e3f70",
            "placeholder": "​",
            "style": "IPY_MODEL_ae75682934dc4077967b76fdbd194233",
            "value": "100%"
          }
        },
        "7597e26176f643a4873a9b11a5778af8": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ae50112f680e446e87368ad81147235e": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_68cd6ca7aa40454195ccc26f9d870cb8",
              "IPY_MODEL_d7e6521389644dcaabe20c16fc20a3eb",
              "IPY_MODEL_13c185bb27074738998f4eb99c007505"
            ],
            "layout": "IPY_MODEL_5ccbc79e76f44dcf86ccc02be27b4910"
          }
        },
        "ae75682934dc4077967b76fdbd194233": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "c02fa16b6d0a497e89e55c63b0427ddf": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "d7e6521389644dcaabe20c16fc20a3eb": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ef044cc6232249d4a80cb876dc889395",
            "max": 3,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_4008c0bbaea645a4bd95c26672cce12e",
            "value": 3
          }
        },
        "ef044cc6232249d4a80cb876dc889395": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}