2024-01-Summarization_with_Transformers.ipynb

{"cells":[{"cell_type":"markdown","metadata":{"id":"QVemECCPfYg0"},"source":["# Summarization with Transformers\n","\n","![](https://i.imgur.com/7SXKckD.png)\n","\n","Transfer Learning is the power of leveraging already trained models and tune \\ adapt them to our own downstream tasks."]},{"cell_type":"markdown","metadata":{"id":"YHruWvqUgO-x"},"source":["# Automated document summarization methodologies\n","\n","- __Extractive techniques:__ These methods use mathematical and statistical concepts to extract a key subset of content from the original document such that this subset contains the core information of the entire document. This content could be words, phrases or even sentences. The end result from this approach is a short executive summary of a couple of lines which are taken or extracted from the original document. No new content is generated in this technique hence the name \"extractive\".\n","\n","\n","- __Abstractive techniques:__ These methods are more complex and sophisticated and leverage language semantics to create representations and also make use of natural language generation (NLG) techniques where the machine makes use of knowledge bases and semantic representations to generate text on its own and create summaries just like a human would write them."]},{"cell_type":"markdown","metadata":{"id":"x3I_xy4pwcjZ"},"source":["# Abstractive Summarization by Fine-tuning Transformers\n","\n","In this section we’ll take a look at how Transformer models can be used to condense long documents into summaries, a task known as abstractive text summarization.\n","\n","This is one of the most challenging NLP tasks as it requires a range of abilities, such as understanding long passages and generating coherent text that captures the main topics in a document.\n","\n","However, when done well, text summarization is a powerful tool that can speed up various business processes by relieving the burden of domain experts to read long documents in detail.\n","\n","Remember this is a sequence to sequence problem and requires past documents with the large and short (summary) form to make the model learn enough patterns to take in a new large document in the future and summarize it\n","\n"]},{"cell_type":"markdown","metadata":{"id":"kh6e2tRDw1NO"},"source":["## Install Relevant Libraries\n","\n"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"Ulm5qCWdwQ1p","outputId":"789ab7ca-2e05-4d32-d926-e7f598407ecc","tags":[]},"outputs":[{"name":"stdout","output_type":"stream","text":["huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n","To disable this warning, you can either:\n","\t- Avoid using `tokenizers` before the fork if possible\n","\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n","Requirement already satisfied: datasets in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (2.12.0)\n","Requirement already satisfied: requests>=2.19.0 in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from datasets) (2.28.1)\n","Requirement already satisfied: xxhash in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from datasets) (3.2.0)\n","Requirement already satisfied: aiohttp in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from datasets) (3.8.3)\n","Requirement already satisfied: responses<0.19 in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from datasets) (0.18.0)\n","Requirement already satisfied: pandas in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from datasets) (1.4.4)\n","Requirement already satisfied: fsspec[http]>=2021.11.1 in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from datasets) (2023.5.0)\n","Requirement already satisfied: pyarrow>=8.0.0 in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from datasets) (12.0.0)\n","Requirement already satisfied: tqdm>=4.62.1 in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from datasets) (4.64.1)\n","Requirement already satisfied: pyyaml>=5.1 in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from datasets) (6.0)\n","Requirement already satisfied: numpy>=1.17 in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from datasets) (1.22.2)\n","Requirement already satisfied: dill<0.3.7,>=0.3.0 in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from datasets) (0.3.6)\n","Requirement already satisfied: huggingface-hub<1.0.0,>=0.11.0 in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from datasets) (0.14.1)\n","Requirement already satisfied: packaging in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from datasets) (22.0)\n","Requirement already satisfied: multiprocess in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from datasets) (0.70.14)\n","Requirement already satisfied: aiosignal>=1.1.2 in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from aiohttp->datasets) (1.3.1)\n","Requirement already satisfied: charset-normalizer<3.0,>=2.0 in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from aiohttp->datasets) (2.1.1)\n","Requirement already satisfied: frozenlist>=1.1.1 in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from aiohttp->datasets) (1.3.3)\n","Requirement already satisfied: multidict<7.0,>=4.5 in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from aiohttp->datasets) (6.0.2)\n","Requirement already satisfied: attrs>=17.3.0 in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from aiohttp->datasets) (22.1.0)\n","Requirement already satisfied: yarl<2.0,>=1.0 in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from aiohttp->datasets) (1.8.1)\n","Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from aiohttp->datasets) (4.0.2)\n","Requirement already satisfied: filelock in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from huggingface-hub<1.0.0,>=0.11.0->datasets) (3.12.0)\n","Requirement already satisfied: typing-extensions>=3.7.4.3 in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from huggingface-hub<1.0.0,>=0.11.0->datasets) (3.7.4.3)\n","Requirement already satisfied: certifi>=2017.4.17 in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from requests>=2.19.0->datasets) (2022.12.7)\n","Requirement already satisfied: idna<4,>=2.5 in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from requests>=2.19.0->datasets) (3.4)\n","Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from requests>=2.19.0->datasets) (1.26.13)\n","Requirement already satisfied: python-dateutil>=2.8.1 in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from pandas->datasets) (2.8.2)\n","Requirement already satisfied: pytz>=2020.1 in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from pandas->datasets) (2022.6)\n","Requirement already satisfied: six>=1.5 in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from python-dateutil>=2.8.1->pandas->datasets) (1.15.0)\n"]}],"source":["!pip install datasets"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"QIal0OzCWQzn","outputId":"ab43bb78-e54d-41c9-ec2a-ac66bc2b834d","tags":[]},"outputs":[{"name":"stdout","output_type":"stream","text":["huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n","To disable this warning, you can either:\n","\t- Avoid using `tokenizers` before the fork if possible\n","\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n","Requirement already satisfied: transformers in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (4.29.2)\n","Requirement already satisfied: regex!=2019.12.17 in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from transformers) (2023.5.5)\n","Requirement already satisfied: requests in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from transformers) (2.28.1)\n","Requirement already satisfied: tqdm>=4.27 in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from transformers) (4.64.1)\n","Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from transformers) (0.13.3)\n","Requirement already satisfied: huggingface-hub<1.0,>=0.14.1 in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from transformers) (0.14.1)\n","Requirement already satisfied: pyyaml>=5.1 in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from transformers) (6.0)\n","Requirement already satisfied: packaging>=20.0 in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from transformers) (22.0)\n","Requirement already satisfied: numpy>=1.17 in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from transformers) (1.22.2)\n","Requirement already satisfied: filelock in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from transformers) (3.12.0)\n","Requirement already satisfied: typing-extensions>=3.7.4.3 in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from huggingface-hub<1.0,>=0.14.1->transformers) (3.7.4.3)\n","Requirement already satisfied: fsspec in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from huggingface-hub<1.0,>=0.14.1->transformers) (2023.5.0)\n","Requirement already satisfied: idna<4,>=2.5 in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from requests->transformers) (3.4)\n","Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from requests->transformers) (1.26.13)\n","Requirement already satisfied: charset-normalizer<3,>=2 in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from requests->transformers) (2.1.1)\n","Requirement already satisfied: certifi>=2017.4.17 in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from requests->transformers) (2022.12.7)\n"]}],"source":["!pip install transformers"]},{"cell_type":"markdown","metadata":{"id":"s00mEGVvCPTj"},"source":["You will be leveraging 🤗 Transformers and 🤗 Datasets as well as other dependencies"]},{"cell_type":"markdown","metadata":{"id":"hBwJ9VevCaUL"},"source":["## Load Dataset\n","\n","Here we load The CNN / DailyMail Dataset is an English-language dataset containing just over 300k unique news articles as written by journalists at CNN and the Daily Mail. The current version supports both extractive and abstractive summarization, though the original version was created for machine reading and comprehension and abstractive question answering.\n","\n","You can find the dataset in [HuggingFace Datasets](https://huggingface.co/datasets/cnn_dailymail)\n","\n","For each instance, there is a string for the article, a string for the highlights, and a string for the id.\n","\n","![](rhRwLI1.png)"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":87,"referenced_widgets":["b00737288ab04fe384293bb7947d54b4","896fa12e861c4ded971314ec0e9501ec","5cb76281cb4a46b9bcf10f87785e1da7","5063986ff6fa413799b1399f4792d4b0","7b6cbef80de34612a9b060456fcd4931","3cbed1cfde0e4dd2b9f28ed082ca0f3d","b0d1b8a72fb44208a04d258911676704","be5bce3b7a8346bfa3e845720fe0a1a9","9569b244a5d342ad963642e0970b3208","2f2e8344f76f42b8a981d5c381210d56","8342186574c4420e9f6fb2beaf3351f2"]},"id":"t3YrFfK8w35v","outputId":"edf40e2f-d20e-4228-f272-678fb271078b"},"outputs":[{"name":"stderr","output_type":"stream","text":["WARNING:datasets.builder:Found cached dataset cnn_dailymail (/home/hp/.cache/huggingface/datasets/cnn_dailymail/3.0.0/3.0.0/1b3c71476f6d152c31c1730e83ccb08bcf23e348233f4fcc11e182248e6bf7de)\n","100%|██████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 238.15it/s]\n"]}],"source":["from datasets import load_dataset, load_metric\n","\n","cnn_data = load_dataset(\"cnn_dailymail\", '3.0.0')"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"j-lxRB0dWAYU","outputId":"97ffe9fa-0d0e-4ab1-daba-dd8d8128f285"},"outputs":[{"data":{"text/plain":["dict_keys(['train', 'validation', 'test'])"]},"execution_count":45,"metadata":{},"output_type":"execute_result"}],"source":["cnn_data.keys()"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"2e-X6QCwxGbW","outputId":"82b8ae58-0a4d-4eed-d730-5f12b5925c81"},"outputs":[{"data":{"text/plain":["{'article': ['LONDON, England (Reuters) -- Harry Potter star Daniel Radcliffe gains access to a reported £20 million ($41.1 million) fortune as he turns 18 on Monday, but he insists the money won\\'t cast a spell on him. Daniel Radcliffe as Harry Potter in \"Harry Potter and the Order of the Phoenix\" To the disappointment of gossip columnists around the world, the young actor says he has no plans to fritter his cash away on fast cars, drink and celebrity parties. \"I don\\'t plan to be one of those people who, as soon as they turn 18, suddenly buy themselves a massive sports car collection or something similar,\" he told an Australian interviewer earlier this month. \"I don\\'t think I\\'ll be particularly extravagant. \"The things I like buying are things that cost about 10 pounds -- books and CDs and DVDs.\" At 18, Radcliffe will be able to gamble in a casino, buy a drink in a pub or see the horror film \"Hostel: Part II,\" currently six places below his number one movie on the UK box office chart. Details of how he\\'ll mark his landmark birthday are under wraps. His agent and publicist had no comment on his plans. \"I\\'ll definitely have some sort of party,\" he said in an interview. \"Hopefully none of you will be reading about it.\" Radcliffe\\'s earnings from the first five Potter films have been held in a trust fund which he has not been able to touch. Despite his growing fame and riches, the actor says he is keeping his feet firmly on the ground. \"People are always looking to say \\'kid star goes off the rails,\\'\" he told reporters last month. \"But I try very hard not to go that way because it would be too easy for them.\" His latest outing as the boy wizard in \"Harry Potter and the Order of the Phoenix\" is breaking records on both sides of the Atlantic and he will reprise the role in the last two films.  Watch I-Reporter give her review of Potter\\'s latest » . There is life beyond Potter, however. The Londoner has filmed a TV movie called \"My Boy Jack,\" about author Rudyard Kipling and his son, due for release later this year. He will also appear in \"December Boys,\" an Australian film about four boys who escape an orphanage. Earlier this year, he made his stage debut playing a tortured teenager in Peter Shaffer\\'s \"Equus.\" Meanwhile, he is braced for even closer media scrutiny now that he\\'s legally an adult: \"I just think I\\'m going to be more sort of fair game,\" he told Reuters. E-mail to a friend . Copyright 2007 Reuters. All rights reserved.This material may not be published, broadcast, rewritten, or redistributed.',\n","  'Editor\\'s note: In our Behind the Scenes series, CNN correspondents share their experiences in covering news and analyze the stories behind the events. Here, Soledad O\\'Brien takes users inside a jail where many of the inmates are mentally ill. An inmate housed on the \"forgotten floor,\" where many mentally ill inmates are housed in Miami before trial. MIAMI, Florida (CNN) -- The ninth floor of the Miami-Dade pretrial detention facility is dubbed the \"forgotten floor.\" Here, inmates with the most severe mental illnesses are incarcerated until they\\'re ready to appear in court. Most often, they face drug charges or charges of assaulting an officer --charges that Judge Steven Leifman says are usually \"avoidable felonies.\" He says the arrests often result from confrontations with police. Mentally ill people often won\\'t do what they\\'re told when police arrive on the scene -- confrontation seems to exacerbate their illness and they become more paranoid, delusional, and less likely to follow directions, according to Leifman. So, they end up on the ninth floor severely mentally disturbed, but not getting any real help because they\\'re in jail. We toured the jail with Leifman. He is well known in Miami as an advocate for justice and the mentally ill. Even though we were not exactly welcomed with open arms by the guards, we were given permission to shoot videotape and tour the floor.  Go inside the \\'forgotten floor\\' » . At first, it\\'s hard to determine where the people are. The prisoners are wearing sleeveless robes. Imagine cutting holes for arms and feet in a heavy wool sleeping bag -- that\\'s kind of what they look like. They\\'re designed to keep the mentally ill patients from injuring themselves. That\\'s also why they have no shoes, laces or mattresses. Leifman says about one-third of all people in Miami-Dade county jails are mentally ill. So, he says, the sheer volume is overwhelming the system, and the result is what we see on the ninth floor. Of course, it is a jail, so it\\'s not supposed to be warm and comforting, but the lights glare, the cells are tiny and it\\'s loud. We see two, sometimes three men -- sometimes in the robes, sometimes naked, lying or sitting in their cells. \"I am the son of the president. You need to get me out of here!\" one man shouts at me. He is absolutely serious, convinced that help is on the way -- if only he could reach the White House. Leifman tells me that these prisoner-patients will often circulate through the system, occasionally stabilizing in a mental hospital, only to return to jail to face their charges. It\\'s brutally unjust, in his mind, and he has become a strong advocate for changing things in Miami. Over a meal later, we talk about how things got this way for mental patients. Leifman says 200 years ago people were considered \"lunatics\" and they were locked up in jails even if they had no charges against them. They were just considered unfit to be in society. Over the years, he says, there was some public outcry, and the mentally ill were moved out of jails and into hospitals. But Leifman says many of these mental hospitals were so horrible they were shut down. Where did the patients go? Nowhere. The streets. They became, in many cases, the homeless, he says. They never got treatment. Leifman says in 1955 there were more than half a million people in state mental hospitals, and today that number has been reduced 90 percent, and 40,000 to 50,000 people are in mental hospitals. The judge says he\\'s working to change this. Starting in 2008, many inmates who would otherwise have been brought to the \"forgotten floor\"  will instead be sent to a new mental health facility -- the first step on a journey toward long-term treatment, not just punishment. Leifman says it\\'s not the complete answer, but it\\'s a start. Leifman says the best part is that it\\'s a win-win solution. The patients win, the families are relieved, and the state saves money by simply not cycling these prisoners through again and again. And, for Leifman, justice is served. E-mail to a friend .'],\n"," 'highlights': [\"Harry Potter star Daniel Radcliffe gets £20M fortune as he turns 18 Monday .\\nYoung actor says he has no plans to fritter his cash away .\\nRadcliffe's earnings from first five Potter films have been held in trust fund .\",\n","  'Mentally ill inmates in Miami are housed on the \"forgotten floor\"\\nJudge Steven Leifman says most are there as a result of \"avoidable felonies\"\\nWhile CNN tours facility, patient shouts: \"I am the son of the president\"\\nLeifman says the system is unjust and he\\'s fighting for change .'],\n"," 'id': ['42c027e4ff9730fbb3de84c1af0d2c506e41c3e4',\n","  'ee8871b15c50d0db17b0179a6d2beab35065f1e9']}"]},"execution_count":46,"metadata":{},"output_type":"execute_result"}],"source":["cnn_data['train'][:2]"]},{"cell_type":"markdown","metadata":{"id":"8NIPmfO4C5F3"},"source":["Given infrastructure constraints we subset our dataset and limit training on only 10000 records."]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"nJ1nqvjlnBrk","outputId":"ad62b006-5dc7-4c4e-ff76-967689a5e7b6"},"outputs":[{"name":"stderr","output_type":"stream","text":["Loading cached shuffled indices for dataset at /home/hp/.cache/huggingface/datasets/cnn_dailymail/3.0.0/3.0.0/1b3c71476f6d152c31c1730e83ccb08bcf23e348233f4fcc11e182248e6bf7de/cache-380ef232430d96da.arrow\n","Loading cached shuffled indices for dataset at /home/hp/.cache/huggingface/datasets/cnn_dailymail/3.0.0/3.0.0/1b3c71476f6d152c31c1730e83ccb08bcf23e348233f4fcc11e182248e6bf7de/cache-fe8ce4e4283768ec.arrow\n","Loading cached shuffled indices for dataset at /home/hp/.cache/huggingface/datasets/cnn_dailymail/3.0.0/3.0.0/1b3c71476f6d152c31c1730e83ccb08bcf23e348233f4fcc11e182248e6bf7de/cache-914896ee6a47e4aa.arrow\n"]}],"source":["cnn_data['train'] = cnn_data['train'].shuffle(seed=42).select(range(10000))\n","cnn_data['validation'] = cnn_data['validation'].shuffle(seed=42).select(range(2000))\n","cnn_data['test'] = cnn_data['test'].shuffle(seed=42).select(range(2000))"]},{"cell_type":"markdown","metadata":{"id":"W71mMauIDMdb"},"source":["The `dataset` object itself is [`DatasetDict`](https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasetdict), which contains one key for the training, validation and test set:"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"DJatnnGjnQms","outputId":"42a81ac6-d363-4612-c3d4-45dcbea5b38a"},"outputs":[{"data":{"text/plain":["DatasetDict({\n","    train: Dataset({\n","        features: ['article', 'highlights', 'id'],\n","        num_rows: 10000\n","    })\n","    validation: Dataset({\n","        features: ['article', 'highlights', 'id'],\n","        num_rows: 2000\n","    })\n","    test: Dataset({\n","        features: ['article', 'highlights', 'id'],\n","        num_rows: 2000\n","    })\n","})"]},"execution_count":7,"metadata":{},"output_type":"execute_result"}],"source":["cnn_data"]},{"cell_type":"markdown","metadata":{"id":"FxgedRlSDPUY"},"source":["To access an actual element, you need to select a split first, then give an index:"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"GZINx7tNnSqh","outputId":"4be3b767-5960-431d-a989-f70c5748422d"},"outputs":[{"data":{"text/plain":["{'article': \"By . Anthony Bond . PUBLISHED: . 07:03 EST, 2 March 2013 . | . UPDATED: . 08:07 EST, 2 March 2013 . Three members of the same family who died in a static caravan from carbon monoxide poisoning would have been unconscious 'within minutes', investigators said today. The bodies of married couple John and Audrey Cook were discovered alongside their daughter, Maureen, at the mobile home they shared on Tremarle Home Park in Camborne, west Cornwall. The inquests have now opened into the deaths last Saturday, with investigators saying the three died along with the family's pet dog, of carbon monoxide poisoning from a cooker. Tragic: The inquests have opened into the deaths of three members of the same family who were found in their static caravan last weekend. John and Audrey Cook are pictured . Awful: The family died following carbon monoxide poisoning at this caravan at the Tremarle Home Park in Camborne, Cornwall . It is also believed there was no working carbon monoxide detector in the static caravan. Cornwall Fire and Rescue Service said this would have resulted in the three being unconscious 'within minutes', . A spokesman for Cornwall coroner Dr Emma Carlyon confirmed the inquests were opened and adjourned yesterday afternoon. They will resume at a later date. Devon and Cornwall Police confirmed on Monday that carbon monoxide poisoning had been established as the cause of death. A police spokesman said the source of the poisoning was 'believed to be from incorrect operation of the gas cooker'. Poisoning: This woman left flowers outside the caravan following the deaths. It has emerged that the trio would have been unconscious 'within minutes' Touching: This tribute was left outside the caravan following news of the deaths . Early readings from experts at the site revealed a potentially lethal level of carbon monoxide present within the caravan at the time it was taken, shortly after the discovery of the bodies. Friends and neighbours have paid tribute to the trio. One . neighbour, Sonya Owen, 53, said: 'It's very distressing. I knew the . daughter, she was living her with her mum and dad. Everybody is really . upset.' Margaret Holmes, 65, who lived near the couple and their . daughter, said: 'They had lived here for around 40 years and they kept . themselves to themselves. 'I just can’t believe this has . happened, it is so sad and I am so shocked, I think we all are, you just . don’t expect this sort of thing to happen on your doorstep. 'Everyone will miss them, we used to chat a lot when we were both in the garden. 'I would just like to send my condolences to their family, I can’t imagine what they’re going through.' Nic Clark, 52, who was good friends with daughter Maureen, added: 'They were a lovely kind family, a great trio. 'Maureen . used to go out and walk her dog, a little Jack Russell, it is so sad . what has happened, I understand the dog went with them. 'They . will be sorely missed and I think everyone is just in shock at the . moment, I would like to send my condolences to the Cook family.'\",\n"," 'highlights': 'John and .\\nAudrey Cook were discovered alongside their daughter, Maureen .\\nThey were found at Tremarle Home Park in Cornwall .\\nInvestigators say the three died of carbon monoxide .\\npoisoning .',\n"," 'id': '08cf276c9eadb638e0c7fdc83ce0229c8af5d09b'}"]},"execution_count":8,"metadata":{},"output_type":"execute_result"}],"source":["cnn_data[\"train\"][0]"]},{"cell_type":"markdown","metadata":{"id":"lGxERC19DSOo"},"source":["To get a sense of what the data looks like, the following function will show some examples picked randomly in the dataset."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"pn3EOeG6nYmU"},"outputs":[],"source":["import datasets\n","import random\n","import pandas as pd\n","from IPython.display import display, HTML\n","\n","\n","def show_random_elements(dataset, num_examples=5):\n","    assert num_examples <= len(\n","        dataset\n","    ), \"Can't pick more elements than there are in the dataset.\"\n","    picks = []\n","    for _ in range(num_examples):\n","        pick = random.randint(0, len(dataset) - 1)\n","        while pick in picks:\n","            pick = random.randint(0, len(dataset) - 1)\n","        picks.append(pick)\n","\n","    df = pd.DataFrame(dataset[picks])\n","    for column, typ in dataset.features.items():\n","        if isinstance(typ, datasets.ClassLabel):\n","            df[column] = df[column].transform(lambda i: typ.names[i])\n","    display(HTML(df.to_html()))"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":1000},"id":"_ppQPBfRnapX","outputId":"be776fe2-41bd-4973-f147-639b32526e5a","tags":[]},"outputs":[{"data":{"text/html":["<table border=\"1\" class=\"dataframe\">\n","  <thead>\n","    <tr style=\"text-align: right;\">\n","      <th></th>\n","      <th>article</th>\n","      <th>highlights</th>\n","      <th>id</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <th>0</th>\n","      <td>The United States has no plans to send troops back into Iraq despite the bloody resurgence of an al-Qaeda faction that has captured major cities, seized hundreds of millions of dollars and forced more than a half-million people to flee their homes this week. America has 35,000 troops station around the Middle East, a Pentagon official confirmed on Wednesday, including 10,000 in nearby Kuwait. A State Department official told MailOnline on background that there are no plans to use them. The Sunni-led group called the Islamic State of Iraq and the Levant (ISIL) – also known as the Islamic State of Iraq in Syria (ISIS) –  was formerly known as Al-Qaeda in Iraq. On Tuesday the White House touted its past support of the Iraqi government with copious amounts of military hardware, but hinted that a return of armed personnel would be out of the question. SCROLL DOWN FOR VIDEO . U.S.Special Forces participated in the 'Eager Lion' joint military exercises in the Gulf of Aqaba on June 5, along with Kuwait, Jordan and France; the US has 2,000 troops stations there, along with a contingent of F-16 fighter jets, and left a Patriot missile battery behind after the war games concluded . Warlord Abu Bakr al-Baghdadi has seized control of the Iraqi provincial capital of Tikrit just a day of gaining power in the country's second biggest city Mosul. ISIS militants gathered in this photo in Iraq's Nineveh province . Iraqi soldiers were no match for the group formerly known as Al-Qaeda in Iraq as Jihadists seized all of Mosul and Nineveh province and also took areas in Kirkuk province, to its east, and Salaheddin to the south . 'Our shipments, in terms of assistance to Iraq, have included the delivery of 300 Hellfire missiles, millions of rounds of small arms fire, thousands of rounds of tank ammunition, helicopter-fired rockets, machine guns, grenades, flares, sniper rifles, M16s and M4 rifles to the Iraqi security forces,' Deputy White House Press Secretary Joshua Earnest eagerly cataloged. And a Pentagon source told MailOnline that the U.S. expected to speed up a planned transfer of Apache helicopters – a sale that was put on the back burner after Iraq made a large purchase of guns and ammunition from neighboring Iran. America's support of Nouri al-Maliki's Shia Muslim-led government, Earnest said Tuesday, has been 'rapid, comprehensive, and is continuing.' But Maliki, he said, must 'step up to the plate' and 'better meet the needs of the Iraqi people,' rather than counting on America to ride to the rescue. Separately, Earnest praised former Secretary of State Hillary Clinton for 'ending the war in Iraq, responsibly winding down the war in Afghanistan, and decimating and destroying core al-Qaeda.' Clinton is a likely Democratic front-runner for the presidency in 2016. Aboard Air Force One en route to Massachusetts on Wednesday, Earnest stayed away from any suggestion that President Obama and Defense Secretary Chuck Hagel might intervene with boots on the ground. 'The United States is deeply concerned about the continued aggression of ISIL in Iraq,' he said, referring to the deterioration of security there as a humanitarian issue. 'The situation in Iraq is grave,' Earnest said, according to a White House pool reporter. There is no doubt that the situation has deteriorated over the last 24 hours.' Earnest, the pool reporter wrote, 'said Washington was continuing to work with the Iraqi government to see how it could help.' The United States has considerable forces at its disposal that could be sent into Iraq, or maneuvered to patrol its waters, if Obama should decide to intervene. A Pentagon official told MailOnline that the U.S. maintains a fighting force of approximately 10,000 troops in Kuwait and 2,000 in Jordan. The units in Jordan include a detachment of F-16 fighter jets and a Patriot missile battery that remained behind after the 2013 joint 'Eager Lion' drills with the Jordanian military. The U.S. also maintains a Combined Air Operations Center in Qatar, and the Navy's 5th fleet in Bahrain. Asked for a comprehensive list of forces in the area, . U.S. Central Command spokesman Commander Bill Speaks told MailOnline: . 'That's kind of a loaded question. Certainly we have a significant . military presence there.' 'Of course, we will not provide details of military assets,' Cmdr. Speaks said in a followup email, 'but there are roughly 35,000 total US forces in the Middle East region.' The U.S. Navy's Fifth Fleet is based in the island nation of Bahrain, just 300 miles from Iraq's shores on the Persian Gulf . America has an estimated 35,000 troops in the Middle East, many of whom could quickly reach Iraq if the White House should decide to intervene against ISIS . EXODUS: As many as 500,000 Iraqis have been forced to flee the country's second biggest city of Mosul after militants from an al-Qaeda splinter group seized control . The State Department said Tuesday that the U.S. 'supports a strong, coordinated response to push back against this aggression in Mosul,' Iraq's oil-rich and second largest city, which is now in ISIS hands. Abu Bakr al-Baghdadi, the head of the so called Islamic State of Iraq and the Levant, is a warlord considered more dangerous than the late Osama bin Laden . But it made no suggestion that American troops should be part of that response. On Wednesday, ISIS and its terrorist warlord Abu Bakr al-Baghdadi took control of Tikrit, another Iraqi city. Maliki has asked his parliament to declare martial law throughout the country. But the U.S. government, like Britain's, has signaled that moral support and armaments will be the limit of its help. The U.S. pulled its last ground troops out of Iraq in December 2011, following nearly nine years of costly and controversial deployments involving 1.5 million troops. More than 30,000 Americans were wounded in the conflict, and nearly 4,500 were killed. The Obama administration took a victory lap at the time of the final pullout, with the president declaring in a speech at Fort Bragg, N.C. that from that day forward 'Iraqis future will be in the hands of its people. America's war in Iraq will be over.' 'It's harder to end a war than begin one,' Obama said, presaging the slogan that has marked his military withdrawal from Afghanistan. 'Indeed, everything that American troops have done in Iraq – all the fighting and all the dying, the bleeding and the building, and the training and the partnering – all of it has led to this moment of success.' ISIS now controls Mosul, Tikrit and parts of Syria . He also claimed then that U.S. forces had 'broken the momentum of the Taliban' in Afghanistan, and had 'gone after al-Qaeda so that terrorists who threaten America will have no safe haven and Osama bin Laden will never again walk the face of this Earth.' But according to a senior U.S. intelligence official who spoke with The Washington Post, Abu Bakr al-Baghdadi, the ISIS leader who also goes by the moniker 'Abu Dua,' is 'more violent, more virulent, [and] more anti-American' than bin Laden. He claims to be a direct descendant of the Muslim prophet Muhammad. The U.S. currently has a $10 million bounty on his head. Republican Senators John McCain, Lindsey Graham and Kelly Ayotte said Tuesday that a 'growing threat to our national security interests is the cost of President Obama’s decision to withdraw all of our troops from Iraq in 2011, against the advice of our commanders and regardless of conditions on the ground.' 'Unfortunately,' they said in a statement, the president is now making the same disastrous mistake in Afghanistan, increasing the risk that al-Qaeda and its terrorist allies will return there just as they are in Iraq.'</td>\n","      <td>The US has 35,000 troops stationed in the Middle East including 10,000 in Kuwait – plus 10,000 troops, an F-16 detachment and a .\\nPatriot missile battery in Jordan .\\nPresident Obama completed his troop withdrawal from Iraq in December 2011, leaving the country in the .\\nhands of government security forces .\\nThe White House said Tuesday that Hillary Clinton deserves credit for 'ending the war in Iraq, responsibly winding down the war in Afghanistan, and decimating and destroying core al-Qaeda'\\nThree GOP senators warned that the Iraq mess is a preview of Afghanistan once the U.S. completes the Obama-led troop draw-down there .\\nThe Islamic State of Iraq in Syria (ISIS), formerly known as Al-Qaeda in Iraq Islamic State of Iraq, is capturing cities, seizing money and oil, and displacing hundreds of thousands of people .\\nAmerica has provided the Iraqi government with copious military materiel but doesn't plan to respond to ISIS's advances with troops .\\nInstead, Washington has told Baghdad to 'step up to the plate' and help its people in ways that freeze out terror groups .</td>\n","      <td>27e95991703b8af7e4ab4cf003792c5c33a35851</td>\n","    </tr>\n","    <tr>\n","      <th>1</th>\n","      <td>A bomb squad has been called to a residential street when police discovered possible explosives in the back of a vehicle. However, after 10 hours of investigations the squad was still unable to confirm whether or not the suspicious parcel contained explosives. A woman had been stopped by a Random Breath and Drug Testing Unit at 2.50am in inner northern Perth suburb Mount Lawley. The driver 'aroused suspicion' leading to a search of the car and police discovering the questionable package, according to ABC News. The female driver has been detained following the discovery of 'possible explosives' in her vehicle . It was at this point that the possible explosives were detected. Police  cordoned off Lawley Crescent and for much of the day residents were advised to stay inside. The woman in question has been detained and is assisting police with their enquiries. The vehicle has now been taken away for testing as police continue investigations to determine the contents of the package.</td>\n","      <td>Police discovered possible explosives during a routine RBT in Perth .\\nThe bomb squad was called in and the residential street was cordoned off .\\nThe female driver's behaviour 'aroused suspicion', leading police to search her vehicle and uncover the suspicious package .\\nAfter 10 hours,  squad was still unable to determine the package's contents .\\nThe driver has been detained and is helping police with investigations .\\nThe car has been taken away for testing as police investigations continue .</td>\n","      <td>dcdc1d0bcdc21ddd0ed9c7b362ba9d714661b914</td>\n","    </tr>\n","    <tr>\n","      <th>2</th>\n","      <td>A homeless man was shot dead by police after he hit two officers with rocks and refused to put down other stones, authorities have said. Police Chief Bob Metzger told a news conference that officers had used a stun gun on Antonio Zambrano-Montes in Pasco, Washington, but it had no effect. He added because of Zambrano-Montes's 'threatening behavior', police fired their guns. Metzger said he did not know whether a weapon was found. But multiple witnesses say the man was running away from the scene when he was killed at about 5pm on Tuesday. Scroll down for video . Erika Zambrano holds a photo of shooting victim Antonio Zambrano-Montes outside the city hall building in Pasco, Washington. He was shot and killed by Pasco police officers during a confrontation on Tuesday . They told the Tri-City Herald the man had run about half a block when he was killed about 5pm on Tuesday near the Fiesta Foods store. The 35-year-old's last address was a Pasco homeless shelter, according to Franklin County Coroner Dan Blasdel. He was an orchard worker raised in Michoacan, Mexico, who had lived in Pasco for the last ten years and didn't speak any English, the Tri-City Herald reported. His cousin Blanca Zambrano told the newspaper: 'He was a kind person, family-oriented. He was hardworking.' The shooting occurred after officers responded to a report of a man throwing rocks at cars at a busy intersection near a grocery store. Dario Infante, 21, of Pasco, recorded video from a vehicle about 50 feet away as the scene unfolded. He said he decided to start recording when he saw an officer trying to use a stun gun on the man. Infante said he saw the man throw a few rocks at police officers but he didn't see him hit any officers. Five 'pops' are audible shortly after the video begins, and the man can be seen running away, across a street and down a sidewalk, pursued by three officers. Police investigate the scene of an officer involved shooting at the intersection of 10th Avenue and Lewis Street in Pasco, Washington . As the officers draw closer to the running man, he stops, turns around and faces them. Multiple 'pops' are heard and the man falls to the ground. 'He didn't throw any rocks after he started running,' Infante said. Several dozen people gathered at Pasco City Hall yesterday afternoon to raise concerns about the shooting. The ACLU of Washington also issued a statement, calling the incident 'very disturbing.' The group's executive director, Kathleen Taylor said: 'Fleeing from police and not following an officer's command should not be sufficient for a person to get shot,' She added deadly force should be used only as a last resort. Pasco residents, pictured from left,  Angel Morgan, five, and his brother Jose Morgan, six, and Alex Gonzalez, four, and his brother Angel Gonzalez, eight, gather around a candlelit vigil yesterday in memory of Antonio Zambrano-Montes . Ben Patrick told the newspaper police fired at the man as his back was turned. 'I really thought they were just going walk up and tackle or tase him,' he said. 'But they opened fire. His back was turned.' Patrick's wife, Shannon, also said the man was running away. The shooting happened in front of her young children. 'He turned around to take off running and the cops just shot him,' she said. 'All he was trying to do was walk away.' Other witnesses heard officers give the man orders to stop and drop the rock. They said the man refused to listen. Metzger has identified the three officers involved in the shooting. They were placed on leave for the investigation, a standard practice. The Tri-City Special Investigation Unit, which will not include Pasco police, will investigate. Investigators are looking at cellphone video of the scene that has been posted online. Carlos Sanchez, who witnessed the shooting from the grocery store parking lot, also said it looked like the man was running away from officers when he was killed. 'They started shooting and they kept on shooting him,' he said. The case is the fourth fatal shooting involving a Tri-City police officer in Pasco in the last six months. Officers have been cleared of any wrongdoing in all three previous cases.</td>\n","      <td>Antonio Zambrano-Montes, 35, was shot dead for his 'threatening behavior'\\nBut multiple witnesses report he was running away when he was killed .\\nIt's the fourth fatal shooting involving a Tri-City police officer in Pasco in the last six months .\\nOfficers have been cleared of any wrongdoing in all three previous cases .</td>\n","      <td>07e6e604c46097159d01a4d8a4b32fe96ec467eb</td>\n","    </tr>\n","    <tr>\n","      <th>3</th>\n","      <td>By . Sophie Jane Evans . A Chinese meat factory has been shut down following allegations that it supplied out-of-date meat to American fast-food chains across the country. Shanghai Husi Food Co, a unit of U.S.-based food supplier OSI Group, was temporarily closed after allegedly selling expired chicken and beef to Chinese branches of McDonald's and KFC. A TV report showed workers apparently picking up meat from the factory floor, as well as mixing meat beyond its expiration date with fresh produce. Scroll down for video . Shut down: Shanghai Husi Food Co has been shut down following allegations that it supplied out-of-date meat to American fast-food chains across China. Above, employees work at the factory prior to its closure . Shockingly, employees were even heard saying that if their clients knew what they were doing, the firm would lose its contracts. McDonald's and Yum Brands Inc - owner of KFC, Pizza Hut and Taco Bell, with over 6,200 Chinese branches collectively - immediately stopped using the supplier after the Dragon TV report aired. Meanwhile, the Shanghai office of China's food and drug agency said it was . investigating the allegations, and told customers to suspend use of the supplier's . products. 'At present, the company has been sealed and suspect products seized,' the Shanghai Municipal Food and Drug Administration said on its website. Under investigation: The factory allegedly sold expired chicken and beef to branches of McDonald's and KFC . Angry: McDonald's (pictured) and Yum Brands Inc - owner of KFC, Pizza Hut and Taco Bell, with over 6,200 Chinese branches collectively - immediately stopped using the supplier after the allegations became public . McDonald's . sealed 4,500 cases of beef, pork, chicken and other products supplied . by Husi for investigation, the city government said in a statement. The Communist Party secretary of . Shanghai, Han Zheng, has reportedly called for 'severe punishment' of any wrongdoing. It is the latest food safety scare for McDonald's and KFC, which were hurt by a safety scandal in 2012 involving chicken allegedly pumped with unapproved antibiotic drugs and growth hormones. Today, the chains apologised to customers following the TV report, adding that the factory had served restaurants in the Shanghai area. 'We will not tolerate any violations of government laws and regulations from our suppliers,' said Yum China, which ordered all of its KFC and Pizza Hut restaurants to seal up and stop using all meat materials supplied by the Husi factory. Meanwhile a spokesman for McDonald's, which was provided with chicken, beef and lettuce by Husi, told Reuters: 'If proven, the practices outlined in the . reports are completely unacceptable to McDonald's anywhere in the . world. The fast-food branches also said they were conducting their . own investigations. China is McDonald's third-biggest market as measured in number of restaurants, while Yum's KFC, based in Louisville, Kentucky, is China's biggest . restaurant chain, with more than 4,000 outlets and plans to open 700 . more this year. 'I think this is going to be really challenging for both these firms,' said Benjamin Cavender, Shanghai-based principal at China Market Research Group. Scandal: It is the latest food safety scare for McDonald's and KFC (pictured), which were hurt by a safety scandal in 2012 involving chicken allegedly pumped with unapproved antibiotic drugs and growth hormones . 'I don't know that this is something an apology can fix so easily, because at this point people don't have a whole lot of trust that they have good systems in place.' Yum shares were down 3.5 percent at $74.72 and McDonald's shares were down 0.9 percent at $98.13 on Monday afternoon on the New York Stock Exchange. The Shanghai Municipal Food and Drug Administration shut down Husi on Sunday after the local Chinese TV broadcast aired. OSI said on its Chinese website that management was 'appalled by the report.' The company has formed its own investigation team, is fully cooperating with government inspectors and will take all necessary actions based on results of the investigation. 'Management believes this to be an isolated event, but takes full responsibility for the situation,' OSI said. OSI, which has close to 60 manufacturing facilities worldwide and had revenue of more than $5 billion in 2012, has been supplying McDonald's in China since 1992 and KFC and Pizza Hut parent Yum since 2008, according to its website. News of the scare spread quickly to diners negotiating Shanghai's lunch-hour rush today. 'For now I won't go to eat at McDonald's or KFC, at least until this whole thing settles down,' said Xu Xinyu, 24, a financial services worker, eating at a noodle shop near a McDonald's outlet in downtown Shanghai. Yet some Chinese consumers appear to have developed a comparatively thick skin when it comes to food scandals. 'Isn't everywhere like this?' asked student Li Xiaoye, 20, eating a beef burger in a Shanghai McDonald's outlet. 'I'll keep going because wherever I eat, the issues are all the same.' The incident highlights the difficulty in ensuring quality and safety along the supply chain in China. Wal-Mart Stores Inc came under the spotlight this year after a supplier's donkey meat product was found to contain fox meat. It also came under fire for selling expired duck meat in 2011. OSI is one of McDonald's key meat suppliers and has a good reputation, according to an industry insider speaking on condition of anonymity. He added the incident highlighted the issue firms faced enforcing strict processes with local staff. As well as Yum and McDonald's, OSI listed Starbucks Corp , Japan's Saizeriya Co Ltd, Papa John's International Inc, Burger King Worldwide Inc and Doctor's Associates Inc's Subway brand as clients in China, according to a 2012 press release. A Starbucks spokesman told Reuters that the company does not now have any direct business dealings with Husi Food. Burger King, Subway, Papa John's and Saizeriya did not immediately respond to requests for comment. A woman who answered the phone at Husi's headquarters said no one was available to comment. But a company manager, Yang Liqun, told Xinhua News Agency that Husi has a strict . quality control system and will cooperate in the investigation.</td>\n","      <td>Shanghai Husi Food Co Ltd temporarily shut down by Chinese authorities .\\nAllegedly supplied out-of-date meat to U.S. fast food chains across China .\\nTV report also showed workers apparently picking up meat from the floor .\\nMcDonald's and KFC immediately stopped using supplier following report .\\nChina's food and drug agency is investigating allegations against factory .\\nShanghai Husi is the Chinese unit of U.S.-based food supplier OSI Group .</td>\n","      <td>7a2c0ba18336842e1137d24c04338b6c67a4d724</td>\n","    </tr>\n","    <tr>\n","      <th>4</th>\n","      <td>By . Steve Robson . PUBLISHED: . 03:37 EST, 12 August 2013 . | . UPDATED: . 04:57 EST, 12 August 2013 . A cost-cutting local council has sparked anger after spending £76,000 on a bespoke 3D sign welcoming visitors to the town. Bournemouth Borough Council, which has cut millions from its budget and shed scores of staff, believes the new signage will 'promote a sense of arrival for visitors'. The 'Welcome to Bournemouth' sign, which sits above the A338 road, has been criticised by councillors as a waste of taxpayers' money. Costly: The £76,000 'Welcome to Bournemouth' sign which has been erected on the A338 . Controversial: The council said the new sign will make visitors feel more welcome and more likely to return to Bournemouth . Labour councillor Ben Grower described it as a 'flight of fancy'. He told The Sun newspaper: 'Jobs are being cut and services not expanded. This is the biggest two fingers to the people I have seen in many years.' Tourism bosses at the Conservative-led authority, which has plans to save £76million over five years, believe it will make visitors feel more welcome and more likely to return to the coastal town. Councillor Lawrence Williams said: . 'First impressions are everything and this is why it's important to make . the gateway into our town as welcoming as possible. This type of . signage makes a statement about our town and the community it represents . as well as a significant contribution to the way an area is perceived. 'Tourism is worth over £600m to the . local economy, so the more welcoming our town is, the more the . likelihood is that visitors will make return trips in the future, . hopefully staying for longer and spending more time and money in the . Borough. Hot spot: Tourists enjoy warm weather on Bournemouth beach over the weekend . 'This in turn continues to boost the local economy. We are proud of our town and the new signage is a great way to greet people coming to spend time here.' Mike Francis, president of the Bournemouth Tourism Management Board, also backed the project. 'It demonstrates that we value the immeasurable contribution tourism makes to the local economy, and the signage is something the industry has been working with the Council to provide for a long time, to further reflect the significance of Bournemouth as one of the UK’s premier resorts,' he said. A spokesman for the council added that . the £76,000 accounts for the design, installation, traffic management . and power supply for the sign.</td>\n","      <td>Bournemouth Borough Council erects costly sign above A338 road .\\nCritics brand it a 'flight of fancy' and a waste of taxpayers' money .\\nAuthority says it will make visitors feel more welcome .</td>\n","      <td>8d34a2d497a007435170115f043f38dfcc3eb7c1</td>\n","    </tr>\n","  </tbody>\n","</table>"],"text/plain":["<IPython.core.display.HTML object>"]},"metadata":{},"output_type":"display_data"}],"source":["show_random_elements(cnn_data[\"train\"])"]},{"cell_type":"markdown","metadata":{"id":"pr6XL_M3Da1V"},"source":["We will use the [🤗 Datasets](https://github.com/huggingface/datasets) library to download the metric we need to use for evaluation (to compare our model to the benchmark). This can be easily done with the function `load_metric`.  "]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"EeLsNeganmap","outputId":"0461df63-3bd7-4851-ca45-efc514a58e15","tags":[]},"outputs":[{"name":"stdout","output_type":"stream","text":["huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n","To disable this warning, you can either:\n","\t- Avoid using `tokenizers` before the fork if possible\n","\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n","Requirement already satisfied: rouge_score in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (0.1.2)\n","Requirement already satisfied: absl-py in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from rouge_score) (0.15.0)\n","Requirement already satisfied: numpy in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from rouge_score) (1.22.2)\n","Requirement already satisfied: six>=1.14.0 in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from rouge_score) (1.15.0)\n","Requirement already satisfied: nltk in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from rouge_score) (3.8.1)\n","Requirement already satisfied: click in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from nltk->rouge_score) (8.1.3)\n","Requirement already satisfied: joblib in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from nltk->rouge_score) (1.2.0)\n","Requirement already satisfied: regex>=2021.8.3 in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from nltk->rouge_score) (2023.5.5)\n","Requirement already satisfied: tqdm in /home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages (from nltk->rouge_score) (4.64.1)\n"]}],"source":["!pip install rouge_score"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"GORGPJWDndsw","outputId":"70276169-e73b-49e1-af4a-46ec3f04d541","tags":[]},"outputs":[{"data":{"text/plain":["Metric(name: \"rouge\", features: {'predictions': Value(dtype='string', id='sequence'), 'references': Value(dtype='string', id='sequence')}, usage: \"\"\"\n","Calculates average rouge scores for a list of hypotheses and references\n","Args:\n","    predictions: list of predictions to score. Each prediction\n","        should be a string with tokens separated by spaces.\n","    references: list of reference for each prediction. Each\n","        reference should be a string with tokens separated by spaces.\n","    rouge_types: A list of rouge types to calculate.\n","        Valid names:\n","        `\"rouge{n}\"` (e.g. `\"rouge1\"`, `\"rouge2\"`) where: {n} is the n-gram based scoring,\n","        `\"rougeL\"`: Longest common subsequence based scoring.\n","        `\"rougeLSum\"`: rougeLsum splits text using `\"\n","\"`.\n","        See details in https://github.com/huggingface/datasets/issues/617\n","    use_stemmer: Bool indicating whether Porter stemmer should be used to strip word suffixes.\n","    use_aggregator: Return aggregates if this is set to True\n","Returns:\n","    rouge1: rouge_1 (precision, recall, f1),\n","    rouge2: rouge_2 (precision, recall, f1),\n","    rougeL: rouge_l (precision, recall, f1),\n","    rougeLsum: rouge_lsum (precision, recall, f1)\n","Examples:\n","\n","    >>> rouge = datasets.load_metric('rouge')\n","    >>> predictions = [\"hello there\", \"general kenobi\"]\n","    >>> references = [\"hello there\", \"general kenobi\"]\n","    >>> results = rouge.compute(predictions=predictions, references=references)\n","    >>> print(list(results.keys()))\n","    ['rouge1', 'rouge2', 'rougeL', 'rougeLsum']\n","    >>> print(results[\"rouge1\"])\n","    AggregateScore(low=Score(precision=1.0, recall=1.0, fmeasure=1.0), mid=Score(precision=1.0, recall=1.0, fmeasure=1.0), high=Score(precision=1.0, recall=1.0, fmeasure=1.0))\n","    >>> print(results[\"rouge1\"].mid.fmeasure)\n","    1.0\n","\"\"\", stored examples: 0)"]},"execution_count":48,"metadata":{},"output_type":"execute_result"}],"source":["metric = load_metric(\"rouge\")\n","metric"]},{"cell_type":"markdown","metadata":{"id":"UHGujJzBDghX"},"source":["The metric is an instance of [`datasets.Metric`](https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.Metric):"]},{"cell_type":"markdown","metadata":{"id":"u9CaEexlDnLy"},"source":["You can call its `compute` method with your predictions and labels, which need to be list of decoded strings:"]},{"cell_type":"markdown","metadata":{"id":"yKcyUMwgEZPK"},"source":["For summarization, one of the most commonly used metrics is the ROUGE score (short for Recall-Oriented Understudy for Gisting Evaluation).\n","\n","The basic idea behind this metric is to compare a generated summary against a set of reference summaries that are typically created by humans.\n","\n","To make this more precise, suppose we want to compare the following two summaries:\n","\n","```\n","generated_summary = \"I absolutely loved reading the Hunger Games\"\n","reference_summary = \"I loved reading the Hunger Games\"\n","```\n","\n","One way to compare them could be to count the number of overlapping words, which in this case would be 6.\n","\n","However, this is a bit crude, so instead ROUGE is based on computing the precision and recall scores for the overlap.\n","\n","For ROUGE, recall measures how much of the reference summary is captured by the generated one. If we are just comparing words, recall can be calculated according to the following formula:\n","\n","![](iYgPhYB.png)\n","\n","For our simple example above, this formula gives a perfect recall of 6/6 = 1; i.e., all the words in the reference summary have been produced by the model.\n","\n","\n","This may sound great, but imagine if our generated summary had been “I really really loved reading the Hunger Games all night”. This would also have perfect recall, but is arguably a worse summary since it is verbose.\n","\n","To deal with these scenarios we also compute the precision, which in the ROUGE context measures how much of the generated summary was relevant:\n","\n","![](4aadAXM.png)\n","\n","\n","Applying this to our verbose summary gives a precision of 6/10 = 0.6, which is considerably worse than the precision of 6/7 = 0.86 obtained by our shorter one.\n","\n","In practice, both precision and recall are usually computed, and then the F1-score (the harmonic mean of precision and recall) is reported.\n","\n"]},{"cell_type":"code","execution_count":1,"metadata":{"id":"uSbSh1hHeZcK","executionInfo":{"status":"ok","timestamp":1706202908746,"user_tz":-60,"elapsed":4,"user":{"displayName":"Ekaterina Butyugina","userId":"10324182844490961884"}}},"outputs":[],"source":["#!wget http://i.imgur.com/rhRwLI1.png"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"N9CGr0tineFX","outputId":"7bdfdebf-40a3-414e-99e0-d14449480a36"},"outputs":[{"data":{"text/plain":["{'rouge1': AggregateScore(low=Score(precision=0.8571428571428571, recall=1.0, fmeasure=0.923076923076923), mid=Score(precision=0.8571428571428571, recall=1.0, fmeasure=0.923076923076923), high=Score(precision=0.8571428571428571, recall=1.0, fmeasure=0.923076923076923)),\n"," 'rouge2': AggregateScore(low=Score(precision=0.6666666666666666, recall=0.8, fmeasure=0.7272727272727272), mid=Score(precision=0.6666666666666666, recall=0.8, fmeasure=0.7272727272727272), high=Score(precision=0.6666666666666666, recall=0.8, fmeasure=0.7272727272727272)),\n"," 'rougeL': AggregateScore(low=Score(precision=0.8571428571428571, recall=1.0, fmeasure=0.923076923076923), mid=Score(precision=0.8571428571428571, recall=1.0, fmeasure=0.923076923076923), high=Score(precision=0.8571428571428571, recall=1.0, fmeasure=0.923076923076923)),\n"," 'rougeLsum': AggregateScore(low=Score(precision=0.8571428571428571, recall=1.0, fmeasure=0.923076923076923), mid=Score(precision=0.8571428571428571, recall=1.0, fmeasure=0.923076923076923), high=Score(precision=0.8571428571428571, recall=1.0, fmeasure=0.923076923076923))}"]},"execution_count":14,"metadata":{},"output_type":"execute_result"}],"source":["generated_summary = \"I absolutely loved reading the Hunger Games\"\n","reference_summary = \"I loved reading the Hunger Games\"\n","scores = metric.compute(\n","    predictions=[generated_summary], references=[reference_summary]\n",")\n","scores"]},{"cell_type":"markdown","metadata":{"id":"mxWdd6ydFG_c"},"source":["🤗 Datasets actually computes confidence intervals for precision, recall, and F1-score; these are the low, mid, and high attributes you can see here.\n","\n","Moreover, 🤗 Datasets computes a variety of ROUGE scores which are based on different types of text granularity when comparing the generated and reference summaries.\n","\n","- rouge1 is the overlap of unigrams — this is just a fancy way of saying the overlap of words\n","\n","- rouge2 measures the overlap between bigrams (think the overlap of pairs of words)\n","\n","- rougeL and rougeLsum measure the longest matching sequences of words by looking for the longest common substrings in the generated and reference summaries\n","\n","- The “sum” in rougeLsum refers to the fact that this metric is computed over a whole summary, while rougeL is computed as the average over individual sentences"]},{"cell_type":"markdown","metadata":{"id":"4pE8mP-dFy71","tags":[]},"source":["## Preprocessing the data\n","\n","Before we can feed those texts to our model, we need to preprocess them. This is done by a 🤗 Transformers `Tokenizer` which will (as the name indicates) tokenize the inputs (including converting the tokens to their corresponding IDs in the pretrained vocabulary) and put it in a format the model expects, as well as generate the other inputs that the model requires.\n","\n","To do all of this, we instantiate our tokenizer with the `AutoTokenizer.from_pretrained` method, which will ensure:\n","\n","- we get a tokenizer that corresponds to the model architecture we want to use,\n","- we download the vocabulary used when pretraining this specific checkpoint.\n","\n","That vocabulary will be cached, so it's not downloaded again the next time we run the cell."]},{"cell_type":"markdown","metadata":{"id":"4SrW0hSZF2iw"},"source":["This notebook is built to run  with any model checkpoint from the [Model Hub](https://huggingface.co/models) as long as that model has a sequence-to-sequence version in the Transformers library.\n","\n","Here we picked the [`t5-small`](https://huggingface.co/t5-small) checkpoint.\n","\n","![](MFE2vfu.png)\n","\n","T5 can be used for a variety of tasks and we will fine-tune it for summarization."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"wkaEn0VgneHM"},"outputs":[],"source":["from transformers import AutoTokenizer\n","\n","model_checkpoint = \"t5-small\"\n","tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)"]},{"cell_type":"markdown","metadata":{"id":"cLh--lgHGKut"},"source":["By default, the call above will use one of the fast tokenizers (backed by Rust) from the 🤗 Tokenizers library."]},{"cell_type":"markdown","metadata":{"id":"AY-lVe6kGLs0"},"source":["You can directly call this tokenizer on one sentence or a pair of sentences:"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"iVWVSuXNneJR","outputId":"ac7713fb-b715-4bf9-c7bc-3c62c5609074"},"outputs":[{"data":{"text/plain":["{'input_ids': [8774, 6, 48, 19, 3, 9, 7142, 55, 1], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1]}"]},"execution_count":16,"metadata":{},"output_type":"execute_result"}],"source":["tokenizer(\"Hello, this is a sentence!\")"]},{"cell_type":"markdown","metadata":{"id":"mJiTiMpqGOP0"},"source":["To prepare the targets for our model, we need to tokenize them inside the `as_target_tokenizer` context manager. This will make sure the tokenizer uses the special tokens corresponding to the targets:"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"Btk0KZBdV8ND","outputId":"9db1df9b-9e5a-4d86-eabc-8845fbc62f9d"},"outputs":[{"name":"stdout","output_type":"stream","text":["{'input_ids': [[8774, 6, 48, 19, 3, 9, 7142, 55, 1], [100, 19, 430, 7142, 5, 1]], 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1]]}\n"]},{"name":"stderr","output_type":"stream","text":["/home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages/transformers/tokenization_utils_base.py:3606: UserWarning: `as_target_tokenizer` is deprecated and will be removed in v5 of Transformers. You can tokenize your labels by using the argument `text_target` of the regular `__call__` method (either in the same call as your input texts if you use the same keyword arguments, or in a separate call.\n","  warnings.warn(\n"]}],"source":["with tokenizer.as_target_tokenizer():\n","    print(tokenizer([\"Hello, this is a sentence!\", \"This is another sentence.\"]))"]},{"cell_type":"markdown","metadata":{"id":"BIti0LStGXR7"},"source":["If you are using the T5 model, we have to prefix the inputs with \"summarize:\" (the model can also translate and it needs the prefix to know which task it has to perform)."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"EHyZTes5Vjns"},"outputs":[],"source":["prefix = \"summarize: \""]},{"cell_type":"markdown","metadata":{"id":"63BuW7ewGc9F"},"source":["We can then write the function that will preprocess our samples. We just feed them to the `tokenizer` with the argument `truncation=True`.\n","\n","This will ensure that an input longer that what the model selected can handle will be truncated to the maximum length accepted by the model.\n","\n","The padding will be dealt with later on (in a data collator) so we pad examples to the longest length in the batch and not the whole dataset."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"-QXwyXQjVgDZ"},"outputs":[],"source":["max_input_length = 1024\n","max_target_length = 128"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"QcK8X3MMoS0-"},"outputs":[],"source":["def preprocess_function(examples):\n","    inputs = [prefix + doc for doc in examples[\"article\"]]\n","    model_inputs = tokenizer(inputs, max_length=max_input_length, truncation=True)\n","\n","    # Setup the tokenizer for targets\n","    with tokenizer.as_target_tokenizer():\n","        labels = tokenizer(\n","            examples[\"highlights\"], max_length=max_target_length, truncation=True\n","        )\n","\n","    model_inputs[\"labels\"] = labels[\"input_ids\"]\n","    return model_inputs"]},{"cell_type":"markdown","metadata":{"id":"1DOnwyKMGkDu"},"source":["This function works with one or several examples. In the case of several examples, the tokenizer will return a list of lists for each key:"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"vWka-P0foag6","outputId":"441cef79-654c-4ccd-f97d-401f39b93839","tags":[]},"outputs":[{"data":{"text/plain":["{'input_ids': [[21603, 10, 938, 3, 5, 11016, 12528, 3, 5, 3, 10744, 8775, 20619, 2326, 10, 3, 5, 10668, 10, 4928, 3, 6038, 6, 204, 1332, 2038, 3, 5, 1820, 3, 5, 3, 6880, 4296, 11430, 10, 3, 5, 12046, 10, 4560, 3, 6038, 6, 204, 1332, 2038, 3, 5, 5245, 724, 13, 8, 337, 384, 113, 3977, 16, 3, 9, 14491, 22133, 45, 4146, 1911, 6778, 15, 14566, 53, 133, 43, 118, 25429, 3, 31, 4065, 77, 676, 31, 6, 16273, 7, 243, 469, 5, 37, 5678, 13, 4464, 1158, 1079, 11, 31423, 6176, 130, 3883, 5815, 70, 3062, 6, 7758, 60, 35, 6, 44, 8, 1156, 234, 79, 2471, 30, 4691, 1635, 109, 1210, 1061, 16, 5184, 12940, 6, 4653, 26334, 5, 37, 16, 10952, 7, 43, 230, 2946, 139, 8, 14319, 336, 1856, 6, 28, 16273, 7, 2145, 8, 386, 3977, 590, 28, 8, 384, 31, 7, 3947, 1782, 6, 13, 4146, 1911, 6778, 15, 14566, 53, 45, 3, 9, 21859, 5, 21902, 447, 10, 37, 16, 10952, 7, 43, 2946, 139, 8, 14319, 13, 386, 724, 13, 8, 337, 384, 113, 130, 435, 16, 70, 14491, 22133, 336, 1851, 5, 1079, 11, 31423, 6176, 33, 3, 22665, 3, 5, 71, 210, 1329, 10, 37, 384, 3977, 826, 4146, 1911, 6778, 15, 14566, 53, 44, 48, 22133, 44, 8, 4691, 1635, 109, 1210, 1061, 16, 5184, 12940, 6, 26334, 3, 5, 94, 19, 92, 6141, 132, 47, 150, 464, 4146, 1911, 6778, 15, 19199, 16, 8, 14491, 22133, 5, 26334, 3655, 11, 22175, 1387, 243, 48, 133, 43, 741, 15, 26, 16, 8, 386, 271, 25429, 3, 31, 4065, 77, 676, 31, 6, 3, 5, 71, 3, 7, 18461, 21, 26334, 4301, 106, 49, 707, 15325, 1184, 120, 106, 5899, 8, 16, 10952, 7, 130, 2946, 11, 19181, 1211, 29, 15, 26, 4981, 3742, 5, 328, 56, 4258, 44, 3, 9, 865, 833, 5, 23130, 11, 26334, 5076, 5899, 30, 2089, 24, 4146, 1911, 6778, 15, 14566, 53, 141, 118, 2127, 38, 8, 1137, 13, 1687, 5, 71, 2095, 3, 7, 18461, 243, 8, 1391, 13, 8, 14566, 53, 47, 3, 31, 29958, 26, 12, 36, 45, 12153, 2986, 13, 8, 1807, 21859, 31, 5, 1908, 23, 739, 53, 10, 100, 2335, 646, 3652, 1067, 8, 22133, 826, 8, 14319, 5, 94, 65, 13999, 24, 8, 17204, 133, 43, 118, 25429, 3, 31, 4065, 77, 676, 31, 11031, 53, 10, 100, 14886, 47, 646, 1067, 8, 22133, 826, 1506, 13, 8, 14319, 3, 5, 8840, 1183, 7, 45, 2273, 44, 8, 353, 5111, 3, 9, 6149, 90, 13958, 593, 13, 4146, 1911, 6778, 15, 915, 441, 8, 22133, 44, 8, 97, 34, 47, 1026, 6, 10545, 227, 8, 9087, 13, 8, 5678, 5, 9779, 11, 14245, 7, 43, 1866, 14886, 12, 8, 17204, 5, 555, 3, 5, 14245, 6, 8357, 9, 20412, 6, 12210, 6, 243, 10, 3, 31, 196, 17, 31, 7, 182, 19285, 53, 5, 27, 2124, 8, 3, 5, 3062, 6, 255, 47, 840, 160, 28, 160, 15480, 11, 7370, 5, 2181, 6965, 19, 310, 3, 5, 13423, 5, 31, 16689, 22960, 6, 7123, 6, 113, 4114, 1084, 8, 1158, 11, 70, 3, 5, 3062, 6, 243, 10, 3, 31, 10273, 141, 4114, 270, 21, 300, 1283, 203, 11, 79, 2697, 3, 5, 1452, 12, 1452, 5, 3, 31, 196, 131, 54, 22, 17, 857, 48, 65, 3, 5, 2817, 6, 34, 19, 78, 6819, 11, 27, 183, 78, 19170, 6, 27, 317, 62, 66, 33, 6, 25, 131, 3, 5, 278, 22, 17, 1672, 48, 1843, 13, 589, 12, 1837, 30, 39, 26929, 5, 3, 31, 18403, 782, 56, 3041, 135, 6, 62, 261, 12, 3582, 3, 9, 418, 116, 62, 130, 321, 16, 8, 2004, 5, 3, 31, 196, 133, 131, 114, 12, 1299, 82, 9681, 40, 1433, 7, 12, 70, 384, 6, 27, 54, 22, 17, 3034, 125, 79, 22, 60, 352, 190, 5, 31, 20310, 8265, 6, 9065, 6, 113, 47, 207, 803, 28, 3062, 7758, 60, 35, 6, 974, 10, 3, 31, 10273, 130, 3, 9, 3061, 773, 384, 6, 3, 9, 248, 17204, 5, 3, 31, 19059, 15, 35, 3, 5, 261, 12, 281, 91, 11, 1482, 160, 1782, 6, 3, 9, 385, 4496, 14010, 6, 34, 19, 78, 6819, 3, 5, 125, 65, 2817, 6, 27, 734, 8, 1782, 877, 28, 135, 5, 3, 31, 10273, 3, 5, 56, 36, 78, 4610, 4785, 11, 27, 317, 921, 19, 131, 16, 8700, 44, 8, 3, 5, 798, 6, 27, 133, 114, 12, 1299, 82, 9681, 40, 1433, 7, 12, 8, 6176, 384, 5, 31, 1], [21603, 10, 3, 14284, 11430, 3, 25271, 22164, 41, 254, 17235, 61, 1636, 71, 3400, 1338, 13, 412, 5, 567, 5, 3684, 2063, 7701, 13, 538, 6, 2237, 21, 8, 166, 97, 57, 3, 9, 412, 5, 134, 5, 2753, 6, 7546, 3, 9, 3161, 2937, 30, 10847, 8, 3060, 13, 6414, 7749, 2721, 5, 1661, 4534, 19, 8, 166, 412, 5, 134, 5, 2488, 12, 819, 3, 9, 907, 9638, 3684, 2063, 1338, 5, 1661, 4534, 15514, 8, 7241, 1636, 84, 1285, 2440, 13, 6414, 11552, 379, 4623, 6, 1473, 6, 1651, 7190, 11, 1410, 1636, 12, 8269, 3, 75, 63, 29, 20231, 581, 8, 1288, 13, 5413, 26, 53, 8, 4345, 13, 6414, 6026, 5, 96, 1326, 22006, 150, 19178, 7, 81, 8, 8565, 13, 3, 3770, 81, 3, 9, 296, 406, 6414, 7749, 976, 4534, 243, 6, 2651, 24, 2721, 31, 7, 1338, 3240, 15, 26, 3, 9, 1516, 1147, 1039, 16, 20270, 1252, 1041, 5, 37, 3161, 6, 84, 47, 7546, 25141, 120, 6, 3088, 21, 4095, 49, 7415, 30, 6414, 1397, 12, 1709, 135, 45, 271, 261, 21, 2716, 3659, 42, 14244, 5, 94, 92, 2454, 7, 7961, 13, 1038, 2665, 725, 11, 412, 5, 567, 5, 3161, 7, 1918, 6414, 529, 18, 1409, 4597, 2661, 6, 1989, 116, 9352, 224, 38, 7449, 11, 1117, 7054, 33, 16, 12374, 5, 96, 634, 296, 398, 1518, 544, 976, 4534, 243, 5, 96, 1326, 398, 5970, 24, 1038, 973, 19, 59, 46, 6364, 5712, 535, 94, 47, 8, 166, 3684, 2063, 13385, 3533, 15, 26, 57, 3, 9, 412, 5, 134, 5, 2753, 6, 11, 163, 8, 8486, 97, 24, 3684, 2063, 7701, 13, 538, 43, 1736, 5, 4534, 2237, 8, 1338, 250, 8, 907, 1323, 4532, 8, 3, 60, 4571, 3745, 27405, 13, 8, 3684, 2063, 16, 1600, 5, 2867, 4534, 11, 4263, 1661, 18150, 155, 651, 8067, 162, 9776, 6, 113, 5468, 865, 6, 243, 8, 296, 31, 7, 192, 6414, 1355, 6740, 7, 130, 464, 30, 3, 5503, 70, 1519, 102, 699, 7, 16, 3245, 13, 3, 9, 1252, 6414, 13385, 5018, 21, 416, 215, 5, 96, 196, 17, 31, 7, 4813, 24, 46, 1231, 1127, 3, 233, 5619, 30, 8, 21479, 3813, 13, 66, 2251, 976, 8067, 162, 9776, 243, 6, 2651, 24, 3, 88, 1644, 8, 4623, 18, 1265, 5, 134, 5, 2843, 30, 8, 962, 12, 36, 3, 9485, 57, 96, 1748, 6414, 11552, 535, 37, 3613, 2804, 2721, 1636, 3684, 2063, 3161, 507, 4225, 1636, 3088, 21, 96, 4029, 53, 323, 9930, 6414, 7749, 1397, 16, 662, 203, 121, 298, 3, 28807, 14705, 169, 13, 7749, 18, 6801, 3, 2414, 29, 2552, 5, 94, 2454, 7, 423, 5856, 28, 66, 1038, 2665, 725, 11, 3161, 7, 30, 6414, 529, 18, 1409, 4597, 2661, 11, 1028, 12764, 297, 6, 379, 3684, 2063, 3161, 7, 24, 3, 8464, 1117, 7054, 11, 7449, 21, 12385, 12, 13859, 12, 1038, 3148, 7, 5, 86, 811, 6, 8, 3161, 4951, 8, 9257, 169, 13, 6414, 827, 6, 3, 9, 3800, 3, 9485, 57, 2830, 1661, 3455, 20500, 17, 9, 32, 5, 1473, 2762, 7, 12, 4405, 165, 6414, 827, 2614, 38, 294, 13, 46, 1456, 606, 515, 24, 4532, 323, 17572, 1807, 9830, 24, 1137, 3298, 483, 5, 2390, 5923, 3271, 14626, 3899, 92, 3, 11675, 8, 1075, 13, 6414, 579, 38, 3, 9, 1349, 827, 1391, 24, 225, 36, 294, 13, 8, 1127, 12, 1252, 14811, 3, 13403, 334, 2982, 5, 3899, 11, 119, 2440, 5468, 13, 8, 174, 12, 1709, 6414, 7749, 42, 8, 1397, 12, 143, 135, 45, 7784, 95, 16, 8, 1780, 13, 10287, 1637, 5, 96, 634, 2630, 1020, 8, 296, 8519, 469, 19, 13, 5273, 343, 7, 652, 1520, 13, 6414, 1397, 976, 243, 1290, 1483, 11374, 1289, 14851, 9, 221, 23, 6, 2090, 879, 13, 8, 1331, 20474, 447, 4654, 7038, 5, 24533, 29, 2488, 1290, 265, 1635, 2776, 26, 1024, 89, 23, 6, 113, 3, 27624, 8, 3684, 2063, 30, 2875, 16, 3, 9, 3, 2375, 7428, 5023, 12, 8, 412, 5, 567, 5, 2146, 11993, 6, 47, 14101, 21, 8, 1338, 5, 86, 24533, 31, 7, 919, 12, 2516, 44, 8, 13385, 6, 165, 534, 6978, 12, 8, 907, 9638, 718, 21, 8, 27, 14611, 188, 12, 15614, 66, 6414, 1356, 6, 379, 3352, 31, 7, 6, 1446, 13, 3, 7388, 30, 529, 18, 29, 76, 2482, 291, 2315, 5, 282, 5100, 865, 81, 2776, 26, 1024, 89, 23, 31, 7, 8605, 6, 1945, 1384, 3, 7, 18461, 2715, 17223, 115, 7, 243, 24533, 228, 1716, 38, 46, 677, 12, 7449, 11, 1117, 7054, 38, 3, 9, 2982, 24, 1891, 95, 165, 6414, 7749, 478, 11, 230, 141, 25058, 15, 26, 8, 423, 1038, 573, 5, 96, 196, 317, 8, 1569, 24, 8, 24533, 29, 7, 141, 44, 8, 3684, 2063, 19, 3, 9, 1569, 24, 62, 897, 24, 8, 19143, 7, 11, 8, 1117, 9677, 7, 56, 240, 12, 842, 6, 11, 24, 19, 1517, 95, 3, 9, 6414, 7749, 478, 11, 8, 1393, 24, 34, 54, 43, 21, 1038, 8494, 11, 21, 70, 3, 60, 8576, 257, 139, 46, 1038, 573, 976, 17223, 115, 7, 243, 5, 3899, 2283, 718, 21, 17210, 581, 7449, 21, 165, 2231, 12, 1344, 3, 9, 6414, 10931, 5, 461, 2875, 6, 8067, 162, 9776, 3240, 15, 26, 24, 4623, 1636, 84, 3150, 65, 8560, 224, 888, 1636, 92, 228, 380, 17210, 581, 7449, 5, 96, 29613, 31, 7, 1102, 19, 964, 10, 17210, 8207, 991, 12, 8946, 772, 6, 68, 16, 128, 1488, 17210, 33, 17508, 976, 8067, 162, 9776, 243, 16, 3, 9, 4494, 3179, 28, 4534, 227, 79, 1736, 16, 368, 1060, 5, 37, 2493, 3, 25422, 7, 412, 5, 134, 5, 18, 1361, 2231, 21, 1936, 1038, 1666, 30, 7449, 12, 13859, 12, 529, 1409, 4597, 2661, 10663, 16, 3, 14651, 8, 606, 13, 6414, 827, 5, 7449, 3213, 165, 6414, 478, 19, 3855, 21, 9257, 3659, 6, 68, 8, 1038, 573, 17316, 15, 7, 34, 13, 6168, 12, 653, 12, 1344, 6414, 7749, 9004, 5, 37, 19143, 2253, 12, 8, 907, 9638, 1883, 3, 9, 2493, 227, 1]], 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]], 'labels': [[1079, 11, 3, 5, 31423, 6176, 130, 3883, 5815, 70, 3062, 6, 7758, 60, 35, 3, 5, 328, 130, 435, 44, 4691, 1635, 109, 1210, 1061, 16, 26334, 3, 5, 24724, 7, 497, 8, 386, 3977, 13, 4146, 1911, 6778, 15, 3, 5, 14566, 53, 3, 5, 1], [8747, 10, 24533, 54, 1716, 38, 677, 13, 8494, 6, 1945, 1384, 3, 7, 18461, 845, 3, 5, 19957, 3088, 21, 3, 13494, 6414, 7749, 45, 271, 14244, 6, 261, 57, 2716, 3, 5, 4534, 6, 4263, 1661, 18150, 155, 651, 8067, 162, 9776, 464, 12, 1428, 1519, 102, 699, 7, 3, 5, 17841, 29, 2753, 11560, 32, 4004, 3618, 30, 96, 434, 291, 651, 2671, 3306, 121, 8988, 6, 668, 10104, 3, 5, 1]]}"]},"execution_count":21,"metadata":{},"output_type":"execute_result"}],"source":["preprocess_function(cnn_data[\"train\"][:2])"]},{"cell_type":"markdown","metadata":{"id":"EOPRAI7YGm6r"},"source":["To apply this function on all the pairs of sentences in our dataset, we just use the `map` method of our `dataset` object we created earlier.\n","\n","This will apply the function on all the elements of all the splits in `dataset`, so our training, validation and testing data will be preprocessed in one single command."]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":151,"referenced_widgets":["5af9673f6d3a4c92aad07826fcbbb540","d8068067d8f34edc95f1fed90f117ecb","18d16a6df7f74898869e4747fe8d9381","7976af16629249fbb502fc2060bd6b49","a7ec5ac603004099813a830095efc15f","18894bcc9e314b90a2d85f0b5dcf57cd","707afd63edaf4d6a8c3b8913ec72c4fa","93d1740edeae473c9a85726f7323b7ee","33d6e730b93c46878845754c9b72caa3","e63218e0ff114ac6aa73ef802ac875cb","432a4f9cfe3d4d15a57b85ae171919fa","a820bd6ee2df4704b281b706e1c85ceb","6faa6b2fe4c54fc28e8be338282c36be","21e188bdc1e2413da1b342a738e60560","f85384c50de14a36890de11e044093ce","ce484e67b0914e39aced8c230d4cf279","6a74a7447f5943938111211efbfdfb7f","bb2ec489de364ad8880a7573a7646203","bed5d8daa4fe4f078034e5c72e4f1ba0","e21fc3b168ac439e8b552e5d5a975cd8","51f0be58c4464cba8a6015a15dcf9cd6","287e0d722daa4548a78789af52e494ef","3063a7a9a3c34e938c696c45a66c1eef","1161e035055348808f698906b6a52148","3448a99fe1f94a0c85cf528d68f76998","e2a02fb91de04a8d8ff9968cd26bb7c3","69ab1b9e393b44d4be6b07ffdd94f4cd","521ac63555c14ffb99cdc6071e6483d1","765648acfe434374a3af20f101ef556b","d887c5cdb2244ccfb7d51a888af74fb2","dd3b803800fc4eb58c71100502f25a8c","5de13a0eab8b4306ba648cec703b84e0","7bf67db043e347df9db239d42454aac9"]},"id":"tzJFvBvUoazt","outputId":"f2762fd4-5f63-4880-f708-d99ac9e73844"},"outputs":[{"name":"stderr","output_type":"stream","text":["WARNING:datasets.arrow_dataset:Loading cached processed dataset at /home/hp/.cache/huggingface/datasets/cnn_dailymail/3.0.0/3.0.0/1b3c71476f6d152c31c1730e83ccb08bcf23e348233f4fcc11e182248e6bf7de/cache-0dc3f57604238959.arrow\n","WARNING:datasets.arrow_dataset:Loading cached processed dataset at /home/hp/.cache/huggingface/datasets/cnn_dailymail/3.0.0/3.0.0/1b3c71476f6d152c31c1730e83ccb08bcf23e348233f4fcc11e182248e6bf7de/cache-5864a55615beeef4.arrow\n","WARNING:datasets.arrow_dataset:Loading cached processed dataset at /home/hp/.cache/huggingface/datasets/cnn_dailymail/3.0.0/3.0.0/1b3c71476f6d152c31c1730e83ccb08bcf23e348233f4fcc11e182248e6bf7de/cache-97012e3b1fe4e94c.arrow\n"]}],"source":["tokenized_datasets = cnn_data.map(preprocess_function, batched=True)\n","no_deprecation_warning=True"]},{"cell_type":"markdown","metadata":{"id":"nS2KiQayGtPP"},"source":["Even better, the results are automatically cached by the 🤗 Datasets library to avoid spending time on this step the next time you run your notebook.\n","\n","The 🤗 Datasets library is normally smart enough to detect when the function you pass to map has changed (and thus requires to not use the cache data).\n","\n","For instance, it will properly detect if you change the task in the first cell and rerun the notebook."]},{"cell_type":"markdown","metadata":{"id":"8Lc1xOvtGwHr"},"source":["## Fine-tuning the Transformer Model"]},{"cell_type":"markdown","metadata":{"id":"zzJG5rBwGy_N"},"source":["Now that our data is ready, we can download the pretrained model and fine-tune it.\n","\n","Since our task is of the sequence-to-sequence kind, we use the `AutoModelForSeq2SeqLM` class.\n","\n","Like with the tokenizer, the `from_pretrained` method will download and cache the model for us."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"YAKkfTyUoa2g"},"outputs":[],"source":["from transformers import AutoModelForSeq2SeqLM, DataCollatorForSeq2Seq, Seq2SeqTrainingArguments, Seq2SeqTrainer\n","\n","model = AutoModelForSeq2SeqLM.from_pretrained(model_checkpoint)"]},{"cell_type":"markdown","metadata":{"id":"Tf5sEF0NG8Xs"},"source":["To instantiate a `Seq2SeqTrainer`, we will need to define three more things.\n","\n","The most important is the [`Seq2SeqTrainingArguments`](https://huggingface.co/transformers/main_classes/trainer.html#transformers.Seq2SeqTrainingArguments), which is a class that contains all the attributes to customize the training.\n","\n","It requires one folder name, which will be used to save the checkpoints of the model, and all other arguments are optional:"]},{"cell_type":"code","execution_count":null,"metadata":{"tags":[],"id":"7I8OsQcieZcM"},"outputs":[],"source":["#!pip install --upgrade accelerate\n","#!pip uninstall -y transformers accelerate\n","#!pip install transformers accelerate"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"FT535rBzoa5N"},"outputs":[],"source":["batch_size = 16\n","model_name = model_checkpoint.split(\"/\")[-1]\n","\n","args = Seq2SeqTrainingArguments(\n","    f\"{model_name}-finetuned-xsum\",\n","    evaluation_strategy = \"epoch\",\n","    learning_rate=2e-5,\n","    per_device_train_batch_size=batch_size,\n","    per_device_eval_batch_size=batch_size,\n","    weight_decay=0.01,\n","    save_total_limit=3,\n","    num_train_epochs=3,\n","    predict_with_generate=True,\n","    fp16=True,\n","    push_to_hub=False,\n",")"]},{"cell_type":"markdown","metadata":{"id":"FQEUrN9MHDUX"},"source":["Here,\n","\n","- we set the evaluation to be done at the end of each epoch\n","- tweak the learning rate\n","- use the `batch_size` defined at the top of the cell\n","- customize the weight decay\n","\n","Since the `Seq2SeqTrainer` will save the model regularly and our dataset is quite large, we tell it to make three saves maximum.\n","\n","Lastly, we use the `predict_with_generate` option (to properly generate summaries) and activate mixed precision training (to go a bit faster)."]},{"cell_type":"markdown","metadata":{"id":"hMj1-YBEHOp0"},"source":["Then, we need a special kind of data collator, which will not only pad the inputs to the maximum length in the batch, but also the labels:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"1Rni16BFqGt5"},"outputs":[],"source":["data_collator = DataCollatorForSeq2Seq(tokenizer, model=model)"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"zI26UHgZqJDx"},"outputs":[],"source":["import numpy as np\n","import nltk"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"-LTx5pixqJFW","outputId":"7e6aab73-6331-49af-858a-9624a64d88f9"},"outputs":[{"name":"stderr","output_type":"stream","text":["[nltk_data] Downloading package punkt to /home/hp/nltk_data...\n","[nltk_data]   Package punkt is already up-to-date!\n"]},{"data":{"text/plain":["True"]},"execution_count":28,"metadata":{},"output_type":"execute_result"}],"source":["nltk.download('punkt')"]},{"cell_type":"markdown","metadata":{"id":"ScsvA8TJHYQS"},"source":["The last thing to define for our `Seq2SeqTrainer` is how to compute the metrics from the predictions.\n","\n","We need to define a function for this, which will just use the `metric` we loaded earlier, and we have to do a bit of pre-processing to decode the predictions into texts:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"J62wvh8OqJHZ"},"outputs":[],"source":["def compute_metrics(eval_pred):\n","    predictions, labels = eval_pred\n","    decoded_preds = tokenizer.batch_decode(predictions, skip_special_tokens=True)\n","    # Replace -100 in the labels as we can't decode them.\n","    labels = np.where(labels != -100, labels, tokenizer.pad_token_id)\n","    decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)\n","\n","    # Rouge expects a newline after each sentence\n","    decoded_preds = [\"\\n\".join(nltk.sent_tokenize(pred.strip())) for pred in decoded_preds]\n","    decoded_labels = [\"\\n\".join(nltk.sent_tokenize(label.strip())) for label in decoded_labels]\n","\n","    result = metric.compute(predictions=decoded_preds, references=decoded_labels, use_stemmer=True)\n","    # Extract a few results\n","    result = {key: value.mid.fmeasure * 100 for key, value in result.items()}\n","\n","    # Add mean generated length\n","    prediction_lens = [np.count_nonzero(pred != tokenizer.pad_token_id) for pred in predictions]\n","    result[\"gen_len\"] = np.mean(prediction_lens)\n","\n","    return {k: round(v, 4) for k, v in result.items()}"]},{"cell_type":"markdown","metadata":{"id":"4imiRnNQHdSB"},"source":["Then we just need to pass all of this along with our datasets to the `Seq2SeqTrainer`:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"nmZkZiWEq2Di"},"outputs":[],"source":["trainer = Seq2SeqTrainer(\n","    model,\n","    args,\n","    train_dataset=tokenized_datasets[\"train\"],\n","    eval_dataset=tokenized_datasets[\"validation\"],\n","    data_collator=data_collator,\n","    tokenizer=tokenizer,\n","    compute_metrics=compute_metrics\n",")"]},{"cell_type":"markdown","metadata":{"id":"0QC2rtukHgZS"},"source":["We can now finetune our model by just calling the `train` method:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"5rFFpwBkeZcM"},"outputs":[],"source":["import time"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":957},"id":"NaiqBN6ptEAk","outputId":"ae036f0f-b783-4f90-971a-4e1b212c5c59","tags":[]},"outputs":[{"name":"stderr","output_type":"stream","text":["/home/hp/.conda/envs/GenerataX/lib/python3.9/site-packages/transformers/optimization.py:407: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n","  warnings.warn(\n"]},{"data":{"text/html":["\n","    <div>\n","      \n","      <progress value='1875' max='1875' style='width:300px; height:20px; vertical-align: middle;'></progress>\n","      [1875/1875 26:48, Epoch 3/3]\n","    </div>\n","    <table border=\"1\" class=\"dataframe\">\n","  <thead>\n"," <tr style=\"text-align: left;\">\n","      <th>Epoch</th>\n","      <th>Training Loss</th>\n","      <th>Validation Loss</th>\n","      <th>Rouge1</th>\n","      <th>Rouge2</th>\n","      <th>Rougel</th>\n","      <th>Rougelsum</th>\n","      <th>Gen Len</th>\n","    </tr>\n","  </thead>\n","  <tbody>\n","    <tr>\n","      <td>1</td>\n","      <td>1.872800</td>\n","      <td>1.694398</td>\n","      <td>24.369100</td>\n","      <td>11.758400</td>\n","      <td>20.091100</td>\n","      <td>22.911900</td>\n","      <td>19.000000</td>\n","    </tr>\n","    <tr>\n","      <td>2</td>\n","      <td>1.875000</td>\n","      <td>1.692535</td>\n","      <td>24.347200</td>\n","      <td>11.719000</td>\n","      <td>20.029200</td>\n","      <td>22.901600</td>\n","      <td>19.000000</td>\n","    </tr>\n","    <tr>\n","      <td>3</td>\n","      <td>1.865200</td>\n","      <td>1.690754</td>\n","      <td>24.429200</td>\n","      <td>11.803600</td>\n","      <td>20.106900</td>\n","      <td>22.962900</td>\n","      <td>19.000000</td>\n","    </tr>\n","  </tbody>\n","</table><p>"],"text/plain":["<IPython.core.display.HTML object>"]},"metadata":{},"output_type":"display_data"},{"name":"stdout","output_type":"stream","text":["CPU times: user 26min 37s, sys: 12.7 s, total: 26min 50s\n","Wall time: 26min 49s\n"]},{"data":{"text/plain":["TrainOutput(global_step=1875, training_loss=1.8704330403645832, metrics={'train_runtime': 1609.1057, 'train_samples_per_second': 18.644, 'train_steps_per_second': 1.165, 'total_flos': 8120055539171328.0, 'train_loss': 1.8704330403645832, 'epoch': 3.0})"]},"execution_count":51,"metadata":{},"output_type":"execute_result"}],"source":["%%time\n","# train\n","trainer.train()"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"Xop3rvjJeZcN"},"outputs":[],"source":[]},{"cell_type":"code","execution_count":null,"metadata":{"id":"cAghpundeZcN"},"outputs":[],"source":[]},{"cell_type":"markdown","metadata":{"id":"AGwxjpnQHoqp"},"source":["# Using your fine-tuned model for Summarization\n","\n","Once you’ve fine-tuned the model you can use it with a pipeline object, for inference as follows:"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"FbwU9peyxbwX"},"outputs":[],"source":["from transformers import pipeline"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"collapsed":true,"id":"v3jgg5Y5zd0N","jupyter":{"outputs_hidden":true},"outputId":"c0e9114a-4fde-42ac-9420-313e59b2805e","tags":[]},"outputs":[{"data":{"text/plain":["T5ForConditionalGeneration(\n","  (shared): Embedding(32128, 512)\n","  (encoder): T5Stack(\n","    (embed_tokens): Embedding(32128, 512)\n","    (block): ModuleList(\n","      (0): T5Block(\n","        (layer): ModuleList(\n","          (0): T5LayerSelfAttention(\n","            (SelfAttention): T5Attention(\n","              (q): Linear(in_features=512, out_features=512, bias=False)\n","              (k): Linear(in_features=512, out_features=512, bias=False)\n","              (v): Linear(in_features=512, out_features=512, bias=False)\n","              (o): Linear(in_features=512, out_features=512, bias=False)\n","              (relative_attention_bias): Embedding(32, 8)\n","            )\n","            (layer_norm): T5LayerNorm()\n","            (dropout): Dropout(p=0.1, inplace=False)\n","          )\n","          (1): T5LayerFF(\n","            (DenseReluDense): T5DenseActDense(\n","              (wi): Linear(in_features=512, out_features=2048, bias=False)\n","              (wo): Linear(in_features=2048, out_features=512, bias=False)\n","              (dropout): Dropout(p=0.1, inplace=False)\n","              (act): ReLU()\n","            )\n","            (layer_norm): T5LayerNorm()\n","            (dropout): Dropout(p=0.1, inplace=False)\n","          )\n","        )\n","      )\n","      (1-5): 5 x T5Block(\n","        (layer): ModuleList(\n","          (0): T5LayerSelfAttention(\n","            (SelfAttention): T5Attention(\n","              (q): Linear(in_features=512, out_features=512, bias=False)\n","              (k): Linear(in_features=512, out_features=512, bias=False)\n","              (v): Linear(in_features=512, out_features=512, bias=False)\n","              (o): Linear(in_features=512, out_features=512, bias=False)\n","            )\n","            (layer_norm): T5LayerNorm()\n","            (dropout): Dropout(p=0.1, inplace=False)\n","          )\n","          (1): T5LayerFF(\n","            (DenseReluDense): T5DenseActDense(\n","              (wi): Linear(in_features=512, out_features=2048, bias=False)\n","              (wo): Linear(in_features=2048, out_features=512, bias=False)\n","              (dropout): Dropout(p=0.1, inplace=False)\n","              (act): ReLU()\n","            )\n","            (layer_norm): T5LayerNorm()\n","            (dropout): Dropout(p=0.1, inplace=False)\n","          )\n","        )\n","      )\n","    )\n","    (final_layer_norm): T5LayerNorm()\n","    (dropout): Dropout(p=0.1, inplace=False)\n","  )\n","  (decoder): T5Stack(\n","    (embed_tokens): Embedding(32128, 512)\n","    (block): ModuleList(\n","      (0): T5Block(\n","        (layer): ModuleList(\n","          (0): T5LayerSelfAttention(\n","            (SelfAttention): T5Attention(\n","              (q): Linear(in_features=512, out_features=512, bias=False)\n","              (k): Linear(in_features=512, out_features=512, bias=False)\n","              (v): Linear(in_features=512, out_features=512, bias=False)\n","              (o): Linear(in_features=512, out_features=512, bias=False)\n","              (relative_attention_bias): Embedding(32, 8)\n","            )\n","            (layer_norm): T5LayerNorm()\n","            (dropout): Dropout(p=0.1, inplace=False)\n","          )\n","          (1): T5LayerCrossAttention(\n","            (EncDecAttention): T5Attention(\n","              (q): Linear(in_features=512, out_features=512, bias=False)\n","              (k): Linear(in_features=512, out_features=512, bias=False)\n","              (v): Linear(in_features=512, out_features=512, bias=False)\n","              (o): Linear(in_features=512, out_features=512, bias=False)\n","            )\n","            (layer_norm): T5LayerNorm()\n","            (dropout): Dropout(p=0.1, inplace=False)\n","          )\n","          (2): T5LayerFF(\n","            (DenseReluDense): T5DenseActDense(\n","              (wi): Linear(in_features=512, out_features=2048, bias=False)\n","              (wo): Linear(in_features=2048, out_features=512, bias=False)\n","              (dropout): Dropout(p=0.1, inplace=False)\n","              (act): ReLU()\n","            )\n","            (layer_norm): T5LayerNorm()\n","            (dropout): Dropout(p=0.1, inplace=False)\n","          )\n","        )\n","      )\n","      (1-5): 5 x T5Block(\n","        (layer): ModuleList(\n","          (0): T5LayerSelfAttention(\n","            (SelfAttention): T5Attention(\n","              (q): Linear(in_features=512, out_features=512, bias=False)\n","              (k): Linear(in_features=512, out_features=512, bias=False)\n","              (v): Linear(in_features=512, out_features=512, bias=False)\n","              (o): Linear(in_features=512, out_features=512, bias=False)\n","            )\n","            (layer_norm): T5LayerNorm()\n","            (dropout): Dropout(p=0.1, inplace=False)\n","          )\n","          (1): T5LayerCrossAttention(\n","            (EncDecAttention): T5Attention(\n","              (q): Linear(in_features=512, out_features=512, bias=False)\n","              (k): Linear(in_features=512, out_features=512, bias=False)\n","              (v): Linear(in_features=512, out_features=512, bias=False)\n","              (o): Linear(in_features=512, out_features=512, bias=False)\n","            )\n","            (layer_norm): T5LayerNorm()\n","            (dropout): Dropout(p=0.1, inplace=False)\n","          )\n","          (2): T5LayerFF(\n","            (DenseReluDense): T5DenseActDense(\n","              (wi): Linear(in_features=512, out_features=2048, bias=False)\n","              (wo): Linear(in_features=2048, out_features=512, bias=False)\n","              (dropout): Dropout(p=0.1, inplace=False)\n","              (act): ReLU()\n","            )\n","            (layer_norm): T5LayerNorm()\n","            (dropout): Dropout(p=0.1, inplace=False)\n","          )\n","        )\n","      )\n","    )\n","    (final_layer_norm): T5LayerNorm()\n","    (dropout): Dropout(p=0.1, inplace=False)\n","  )\n","  (lm_head): Linear(in_features=512, out_features=32128, bias=False)\n",")"]},"execution_count":91,"metadata":{},"output_type":"execute_result"}],"source":["model.to('cpu')"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"pzYEqoGkyUtF"},"outputs":[],"source":["summarize = pipeline(task='summarization', model=model, tokenizer=tokenizer)"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"Ahvm36LQysp4"},"outputs":[],"source":["# An official announcement https://blog.google/technology/ai/gemini-collection/\n","\n","document = \"\"\"\n","Learn more about Gemini, our most capable AI model\n","Dec 06, 2023 3 articles\n","Gemini is a multimodal AI model, meaning it can process and generate different formats of data, including text, code, audio, images, and video.\n","This sets it apart from previous models, which were primarily focused on text-based tasks\n","Today we introduced Gemini, our largest and most capable AI model — and the next step on our journey toward making AI helpful for everyone. Built from the ground up to be multimodal, Gemini can generalize and seamlessly understand, operate across and combine different types of information, including text, images, audio, video and code. This means it has sophisticated multimodal reasoning and advanced coding capabilities. And with three different sizes — Ultra, Pro and Nano — Gemini has the flexibility to run on everything from data centers to mobile devices. We trained Gemini at scale on our AI-optimized infrastructure using Google's Tensor Processing Units (TPUs) v4 and v5e. Today, we also announced our most powerful and scalable TPU system to date, Cloud TPU v5p.\n","Gemini is available in some of our core products starting today: Bard is using a fine-tuned version of Gemini Pro for more advanced reasoning, planning, understanding and more. Pixel 8 Pro is the first smartphone engineered for Gemini Nano, using it in features like Summarize in Recorder and Smart Reply in Gboard. And we’re already starting to experiment with Gemini in Search, where it's making our Search Generative Experience (SGE) faster. Early next year, we’ll bring Gemini Ultra to a new Bard Advanced experience. And in the coming months, Gemini will power features in more of our products and services like Ads, Chrome and Duet AI.\n","Android developers who want to build Gemini-powered apps on-device can now sign up for an early preview of Gemini Nano, our most efficient model, via Android AICore. Starting December 13, developers and enterprise customers will be able to access Gemini Pro via the Gemini API in Vertex AI or Google AI Studio, our free web-based developer tool. And as we continue to refine Gemini Ultra, including completing extensive trust and safety checks, we’ll make it available to select groups before opening it up broadly to developers and enterprise customers early next year.\n","Explore the collection to learn more about our newest model, and the start of the Gemini era.\n","\"\"\""]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":553},"id":"B4dSWU9Y0PuO","outputId":"9883944e-4c9f-407b-db9a-f7d8c63ad3bf"},"outputs":[{"data":{"text/plain":["\"\\nLearn more about Gemini, our most capable AI model\\nDec 06, 2023 3 articles\\nGemini is a multimodal AI model, meaning it can process and generate different formats of data, including text, code, audio, images, and video. \\nThis sets it apart from previous models, which were primarily focused on text-based tasks\\nToday we introduced Gemini, our largest and most capable AI model — and the next step on our journey toward making AI helpful for everyone. Built from the ground up to be multimodal, Gemini can generalize and seamlessly understand, operate across and combine different types of information, including text, images, audio, video and code. This means it has sophisticated multimodal reasoning and advanced coding capabilities. And with three different sizes — Ultra, Pro and Nano — Gemini has the flexibility to run on everything from data centers to mobile devices. We trained Gemini at scale on our AI-optimized infrastructure using Google's Tensor Processing Units (TPUs) v4 and v5e. Today, we also announced our most powerful and scalable TPU system to date, Cloud TPU v5p.\\nGemini is available in some of our core products starting today: Bard is using a fine-tuned version of Gemini Pro for more advanced reasoning, planning, understanding and more. Pixel 8 Pro is the first smartphone engineered for Gemini Nano, using it in features like Summarize in Recorder and Smart Reply in Gboard. And we’re already starting to experiment with Gemini in Search, where it's making our Search Generative Experience (SGE) faster. Early next year, we’ll bring Gemini Ultra to a new Bard Advanced experience. And in the coming months, Gemini will power features in more of our products and services like Ads, Chrome and Duet AI.\\nAndroid developers who want to build Gemini-powered apps on-device can now sign up for an early preview of Gemini Nano, our most efficient model, via Android AICore. Starting December 13, developers and enterprise customers will be able to access Gemini Pro via the Gemini API in Vertex AI or Google AI Studio, our free web-based developer tool. And as we continue to refine Gemini Ultra, including completing extensive trust and safety checks, we’ll make it available to select groups before opening it up broadly to developers and enterprise customers early next year.\\nExplore the collection to learn more about our newest model, and the start of the Gemini era.\\n\""]},"execution_count":94,"metadata":{},"output_type":"execute_result"}],"source":["document"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"7Gh2qqWmeZcN"},"outputs":[],"source":["authors_summary = \"\"\"Gemini is a multimodal AI model, meaning it can process and generate different formats of data, including text, code, audio, images, and video.\n","This sets it apart from previous models, which were primarily focused on text-based tasks\"\"\"\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"HEq6c-fgztMX"},"outputs":[],"source":["summary = summarize(document)[0]['summary_text']"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"2ALmeY0d9n6V","outputId":"a68eed31-b603-4730-e8dd-d9da177a7e3f"},"outputs":[{"name":"stdout","output_type":"stream","text":["Gemini is a multimodal AI model, meaning it can process and generate different formats of data, including text, code, audio, images, and video .\n","Built from the ground up to be multimodal, Gemini can generalize and seamlessly understand, operate across and combine different types of information .\n","This means it has sophisticated multimodal reasoning and advanced coding capabilities .\n"]}],"source":["print('\\n'.join(nltk.sent_tokenize(summary)))"]},{"cell_type":"markdown","metadata":{"id":"fingp88-H1cd"},"source":["We can feed some examples from the test set (which the model has not seen) to our pipeline to get a feel for the quality of the summaries."]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"ncKGARDa9-ye","outputId":"4a766f4e-06c5-4900-f50a-b0344d72505c","tags":[]},"outputs":[{"name":"stderr","output_type":"stream","text":["WARNING:datasets.arrow_dataset:Loading cached shuffled indices for dataset at /home/hp/.cache/huggingface/datasets/cnn_dailymail/3.0.0/3.0.0/1b3c71476f6d152c31c1730e83ccb08bcf23e348233f4fcc11e182248e6bf7de/cache-e66693a00b034ce9.arrow\n"]},{"name":"stdout","output_type":"stream","text":["Acutal Headline:-\n"," Battle between lenders has intensified recently, causing rates to plummet .\n","HSBC have now announced 1.99% interest deal on a five-year fix mortgage .\n","Offer expected to spark flood of rate cuts by banks and building societies .\n","Experts have described cheapest deal ever of its kind as 'astonishing'\n","\n","Summarized Headline:-\n"," The battle between lenders has intensified in recent months, plunging home loan rates to their lowest in history .\n","But the mortgage wars will erupt again next week after HSBC announced a 1.99 per cent interest rate on a five-year fix .\n","Fifteen lenders had already cut rates across their ranges in the past week, but more are likely to follow .\n","\n","\n","Acutal Headline:-\n"," BBC's Back In Time For Dinner claims that grubs are the future of food .\n","The Robshaw family dig into cricket tacos, worm tarts and insect burgers .\n","Meat will become scarce or more expensive as demand for it grows .\n","Insects are full of protein, low in fat and packed full of nutrients .\n","\n","Summarized Headline:-\n"," The Robshaws sample insects in tonight's episode of BBC Two's Back for Dinner .\n","The family are serving a radical dinner of Mexican spiced cricket tacos, Asian worm stir fry, buffalo worm tart and cricket kebabs .\n","They also try burgers made with a mixture of insect and beef protein .\n","\n","\n","Acutal Headline:-\n"," The $39.99 (£26) gadget is called the Nanoheat Wireless Heated Mug .\n","It keeps hot drinks at between 68 and 71°C (155 and 160°F) for 45 minutes .\n","Mug can be used more than seven times before it needs to be charged .\n","And it can be charged wirelessly or using a traditional USB cord .\n","\n","Summarized Headline:-\n"," The Nanoheat Wireless Heated Mug is fitted with a rchargeable battery, nanoheater and heat sleeve .\n","It can maintain the temperature of hot drinks at between 68°C and 71°C (155 and 160°F) It weighs a little over one pound (454g) and can be charged wirelessly or using a traditional USB cord .\n","The mug is only available in white, but wireless charging plates are available in both white and black .\n","\n","\n","Acutal Headline:-\n"," Cyclone strength winds have lashed parts of NSW as heavy rain continues to fall .\n","Commuters using umbrellas have abandoned them in bins and gutters after the wind destroyed them .\n","Workers have posted pictures of broken brollies around Sydney with the hashtag #umbrellageddon .\n","Manufacturers say if damage is caused by the wind that does not constitute a faulty frame .\n","'Always carry your umbrella into the wind and not over your head,' umbrella expert advised .\n","\n","Summarized Headline:-\n"," Workers have begun posting photos of their broken umbrellas scattered in gutters and discarded in bins along with the hashtag #umbrellageddon .\n","'This is #sydney.\n","The #umbrella #cemetery.\n","Mine #died 2mins after leaving the #house,' one user posted on Instagram .\n","A broken umbrella is abandoned in Sydney's CBD on April 21 .\n","\n","\n","Acutal Headline:-\n"," Australia have seen sense by revamping their overseas selection policy .\n","It leaves England looking old-fashioned and they need to react .\n","Matt Giteau and Drew Mitchell are top-class performers for Toulon .\n","Steffon Armitage and Nick Abendanon should come into the selection mix .\n","England need the best squad possible for the forthcoming World Cup .\n","\n","Summarized Headline:-\n"," Stuart Lancaster has remained steadfast in his policy not to select overseas players .\n","Toulon’s Steffon Armitage came off the bench against Leinster on Sunday .\n","He turned the game on its head and his turnover work and ball-carrying are second to none .\n","\n","\n","Acutal Headline:-\n"," Manchester United will embark on a short tour of America this summer .\n","They are likely to spend 12 days on the west coast, staying in one place .\n","United will defend their International Champions Cup title .\n","Louis van Gaal has avoided a lengthy, energy-sapping tour this year .\n","CLICK HERE for all the latest Manchester United news .\n","\n","Summarized Headline:-\n"," Manchester United have turned down offers from Asia and Australia .\n","The trip will last in the region of 12 days – a whole week shorter than last summer - and will be over by August 1 .\n","United beat Real Madrid in the final to win the International Champions Cup .\n","Van Gaal believes his team will travel to match venues from one base .\n","\n","\n","Acutal Headline:-\n"," The boy's mom Michelle Schwab is charged with child endangerment .\n","She is assistant director of KinderCare in Columbus, Ohio, and has 3 sons .\n","The company confirmed Schwab is taking a leave of absence .\n","Schwab was allegedly holding the child when he slipped and fell 10ft into the pit on Saturday around 3pm at the Cleveland Metroparks Zoo .\n","He was rescued by his parents before emergency responders arrived on the scene; he suffered from minor bruises and bumps .\n","The cheetahs seemed to ignore the boy and his parents while in the pit .\n","\n","Summarized Headline:-\n"," Michelle Schwab, 38, charged with child endangerment for allegedly dangling her son .\n","She is assistant director at Cleveland Metroparks Zoo and faces six months in jail and $1,000 fine .\n","Visitors heard a scream at 3pm on Saturday and looked to see Schwab and her husband leaping into the pit to retrieve the child, who was treated for a leg injury .\n","\n","\n","Acutal Headline:-\n"," Agnese Klavina, 30, vanished from Puerto Banus resort on September 6 .\n","Had been on night out at Aqwa Mist nightclub popular with rich and famous .\n","Westley Capper, 37, has already admitted driving her away in a Mercedes .\n","He declined to answer questions in appearance before Spanish judge .\n","Co-accused Craig Porter, 33, was in the car and also said nothing in court .\n","Capper's dad John specialises in buying and selling luxury properties .\n","\n","Summarized Headline:-\n"," Privately-educated Westley Capper, 37, has already admitted driving Agnese Klavina, 30, away from the nightspot in the upmarket Costa del Sol resort of Puerto Banus .\n","Police believe the overweight expat - who has never been publicly named - showed he was in little mood for co-operating by refusing to be questioned after being summoned to court yesterday .\n","A Cessna private jet owned by John Capper has had travel restrictions imposed on him while the investigation is now .\n","\n","\n","Acutal Headline:-\n"," Miss Wendy and owner Marc Métral put through to Semi Finals by panel .\n","But viewers said act was 'cruel' while RSPCA said it would work to 'ascertain what methods were used'\n","Cowell called act 'incredible' - Amanda Holden said: 'You made TV history'\n","But similar act was seen on America's Got Talent in United States in 2012 .\n","\n","Summarized Headline:-\n"," The RSPCA said it would be contacting Britain’s Got Talent ‘to ascertain what methods were used’ in the performance .\n","It is believed that a false mouth was placed on Miss Wendy's face, which is triggered by a remote control .\n","Some viewers said the act was cruel and others said it was cruel .\n","\n","\n","Acutal Headline:-\n"," Australian Warren Rodwell was held hostage for 472 days and feared his Abu Sayyaf captors would behead him .\n","He was freed in March 2013 after his family successfully managed to raise a ransom .\n","Now he fears he will not receive Victims of Terrorism Overseas compensation .\n","Unless Prime Minister Tony Abbott decides otherwise, his kidnapping is not listed as a 'declared terrorist event'\n","ASIO has officially declared Abu Sayyaf as a terrorist organisation .\n","\n","Summarized Headline:-\n"," Warren Rodwell was held against his will in a foreign jungle for 472 days .\n","He fears the Federal government will not award him Victims of Terror Overseas compensation .\n","Under Australian law, survivors and families of victims of overseas attacks can claim up to $75,000 .\n","But for that to happen, Prime Minister Tony Abbott would have to declare his kidnapping a terrorist event .\n","A government media blackout was enforced during his captivity .\n","\n","\n"]}],"source":["for item in cnn_data['test'].shuffle(seed=42).select(range(10)):\n","  print('Acutal Headline:-\\n', item['highlights'])\n","  print()\n","  summary = summarize(item['article'])[0]['summary_text']\n","  print('Summarized Headline:-\\n', '\\n'.join(nltk.sent_tokenize(summary)))\n","  print('\\n')"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"yNmP6SlFeZcN"},"outputs":[],"source":[]},{"cell_type":"code","execution_count":null,"metadata":{"id":"UjCTXAOYeZcN"},"outputs":[],"source":[]}],"metadata":{"accelerator":"GPU","colab":{"machine_shape":"hm","provenance":[],"toc_visible":true},"gpuClass":"standard","kernelspec":{"display_name":"Python 3 (ipykernel)","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.9.15"},"widgets":{"application/vnd.jupyter.widget-state+json":{"1161e035055348808f698906b6a52148":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HTMLModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HTMLModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HTMLView","description":"","description_tooltip":null,"layout":"IPY_MODEL_521ac63555c14ffb99cdc6071e6483d1","placeholder":"​","style":"IPY_MODEL_765648acfe434374a3af20f101ef556b","value":"100%"}},"18894bcc9e314b90a2d85f0b5dcf57cd":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"18d16a6df7f74898869e4747fe8d9381":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"FloatProgressModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"FloatProgressModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"ProgressView","bar_style":"success","description":"","description_tooltip":null,"layout":"IPY_MODEL_93d1740edeae473c9a85726f7323b7ee","max":10,"min":0,"orientation":"horizontal","style":"IPY_MODEL_33d6e730b93c46878845754c9b72caa3","value":10}},"21e188bdc1e2413da1b342a738e60560":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"FloatProgressModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"FloatProgressModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"ProgressView","bar_style":"success","description":"","description_tooltip":null,"layout":"IPY_MODEL_bed5d8daa4fe4f078034e5c72e4f1ba0","max":2,"min":0,"orientation":"horizontal","style":"IPY_MODEL_e21fc3b168ac439e8b552e5d5a975cd8","value":2}},"287e0d722daa4548a78789af52e494ef":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"DescriptionStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"DescriptionStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","description_width":""}},"2f2e8344f76f42b8a981d5c381210d56":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"3063a7a9a3c34e938c696c45a66c1eef":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HBoxModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HBoxModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HBoxView","box_style":"","children":["IPY_MODEL_1161e035055348808f698906b6a52148","IPY_MODEL_3448a99fe1f94a0c85cf528d68f76998","IPY_MODEL_e2a02fb91de04a8d8ff9968cd26bb7c3"],"layout":"IPY_MODEL_69ab1b9e393b44d4be6b07ffdd94f4cd"}},"33d6e730b93c46878845754c9b72caa3":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"ProgressStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"ProgressStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","bar_color":null,"description_width":""}},"3448a99fe1f94a0c85cf528d68f76998":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"FloatProgressModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"FloatProgressModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"ProgressView","bar_style":"success","description":"","description_tooltip":null,"layout":"IPY_MODEL_d887c5cdb2244ccfb7d51a888af74fb2","max":2,"min":0,"orientation":"horizontal","style":"IPY_MODEL_dd3b803800fc4eb58c71100502f25a8c","value":2}},"3cbed1cfde0e4dd2b9f28ed082ca0f3d":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"432a4f9cfe3d4d15a57b85ae171919fa":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"DescriptionStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"DescriptionStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","description_width":""}},"5063986ff6fa413799b1399f4792d4b0":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HTMLModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HTMLModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HTMLView","description":"","description_tooltip":null,"layout":"IPY_MODEL_2f2e8344f76f42b8a981d5c381210d56","placeholder":"​","style":"IPY_MODEL_8342186574c4420e9f6fb2beaf3351f2","value":" 3/3 [00:00&lt;00:00, 52.55it/s]"}},"51f0be58c4464cba8a6015a15dcf9cd6":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"521ac63555c14ffb99cdc6071e6483d1":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"5af9673f6d3a4c92aad07826fcbbb540":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HBoxModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HBoxModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HBoxView","box_style":"","children":["IPY_MODEL_d8068067d8f34edc95f1fed90f117ecb","IPY_MODEL_18d16a6df7f74898869e4747fe8d9381","IPY_MODEL_7976af16629249fbb502fc2060bd6b49"],"layout":"IPY_MODEL_a7ec5ac603004099813a830095efc15f"}},"5cb76281cb4a46b9bcf10f87785e1da7":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"FloatProgressModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"FloatProgressModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"ProgressView","bar_style":"success","description":"","description_tooltip":null,"layout":"IPY_MODEL_be5bce3b7a8346bfa3e845720fe0a1a9","max":3,"min":0,"orientation":"horizontal","style":"IPY_MODEL_9569b244a5d342ad963642e0970b3208","value":3}},"5de13a0eab8b4306ba648cec703b84e0":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"69ab1b9e393b44d4be6b07ffdd94f4cd":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"6a74a7447f5943938111211efbfdfb7f":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"6faa6b2fe4c54fc28e8be338282c36be":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HTMLModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HTMLModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HTMLView","description":"","description_tooltip":null,"layout":"IPY_MODEL_6a74a7447f5943938111211efbfdfb7f","placeholder":"​","style":"IPY_MODEL_bb2ec489de364ad8880a7573a7646203","value":"100%"}},"707afd63edaf4d6a8c3b8913ec72c4fa":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"DescriptionStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"DescriptionStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","description_width":""}},"765648acfe434374a3af20f101ef556b":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"DescriptionStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"DescriptionStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","description_width":""}},"7976af16629249fbb502fc2060bd6b49":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HTMLModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HTMLModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HTMLView","description":"","description_tooltip":null,"layout":"IPY_MODEL_e63218e0ff114ac6aa73ef802ac875cb","placeholder":"​","style":"IPY_MODEL_432a4f9cfe3d4d15a57b85ae171919fa","value":" 10/10 [00:13&lt;00:00,  1.35s/ba]"}},"7b6cbef80de34612a9b060456fcd4931":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"7bf67db043e347df9db239d42454aac9":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"DescriptionStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"DescriptionStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","description_width":""}},"8342186574c4420e9f6fb2beaf3351f2":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"DescriptionStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"DescriptionStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","description_width":""}},"896fa12e861c4ded971314ec0e9501ec":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HTMLModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HTMLModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HTMLView","description":"","description_tooltip":null,"layout":"IPY_MODEL_3cbed1cfde0e4dd2b9f28ed082ca0f3d","placeholder":"​","style":"IPY_MODEL_b0d1b8a72fb44208a04d258911676704","value":"100%"}},"93d1740edeae473c9a85726f7323b7ee":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"9569b244a5d342ad963642e0970b3208":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"ProgressStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"ProgressStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","bar_color":null,"description_width":""}},"a7ec5ac603004099813a830095efc15f":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"a820bd6ee2df4704b281b706e1c85ceb":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HBoxModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HBoxModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HBoxView","box_style":"","children":["IPY_MODEL_6faa6b2fe4c54fc28e8be338282c36be","IPY_MODEL_21e188bdc1e2413da1b342a738e60560","IPY_MODEL_f85384c50de14a36890de11e044093ce"],"layout":"IPY_MODEL_ce484e67b0914e39aced8c230d4cf279"}},"b00737288ab04fe384293bb7947d54b4":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HBoxModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HBoxModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HBoxView","box_style":"","children":["IPY_MODEL_896fa12e861c4ded971314ec0e9501ec","IPY_MODEL_5cb76281cb4a46b9bcf10f87785e1da7","IPY_MODEL_5063986ff6fa413799b1399f4792d4b0"],"layout":"IPY_MODEL_7b6cbef80de34612a9b060456fcd4931"}},"b0d1b8a72fb44208a04d258911676704":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"DescriptionStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"DescriptionStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","description_width":""}},"bb2ec489de364ad8880a7573a7646203":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"DescriptionStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"DescriptionStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","description_width":""}},"be5bce3b7a8346bfa3e845720fe0a1a9":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"bed5d8daa4fe4f078034e5c72e4f1ba0":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"ce484e67b0914e39aced8c230d4cf279":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"d8068067d8f34edc95f1fed90f117ecb":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HTMLModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HTMLModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HTMLView","description":"","description_tooltip":null,"layout":"IPY_MODEL_18894bcc9e314b90a2d85f0b5dcf57cd","placeholder":"​","style":"IPY_MODEL_707afd63edaf4d6a8c3b8913ec72c4fa","value":"100%"}},"d887c5cdb2244ccfb7d51a888af74fb2":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"dd3b803800fc4eb58c71100502f25a8c":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"ProgressStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"ProgressStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","bar_color":null,"description_width":""}},"e21fc3b168ac439e8b552e5d5a975cd8":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"ProgressStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"ProgressStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"StyleView","bar_color":null,"description_width":""}},"e2a02fb91de04a8d8ff9968cd26bb7c3":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HTMLModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HTMLModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HTMLView","description":"","description_tooltip":null,"layout":"IPY_MODEL_5de13a0eab8b4306ba648cec703b84e0","placeholder":"​","style":"IPY_MODEL_7bf67db043e347df9db239d42454aac9","value":" 2/2 [00:02&lt;00:00,  1.44s/ba]"}},"e63218e0ff114ac6aa73ef802ac875cb":{"model_module":"@jupyter-widgets/base","model_module_version":"1.2.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.2.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.2.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"justify_items":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"object_fit":null,"object_position":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"f85384c50de14a36890de11e044093ce":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.5.0","model_name":"HTMLModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.5.0","_model_name":"HTMLModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.5.0","_view_name":"HTMLView","description":"","description_tooltip":null,"layout":"IPY_MODEL_51f0be58c4464cba8a6015a15dcf9cd6","placeholder":"​","style":"IPY_MODEL_287e0d722daa4548a78789af52e494ef","value":" 2/2 [00:02&lt;00:00,  1.25s/ba]"}}}}},"nbformat":4,"nbformat_minor":0}