ch09/Text_Summarization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[**Blueprints for Text Analysis Using Python**](https://github.com/blueprints-for-text-analytics-python/blueprints-text)  \n",
    "Jens Albrecht, Sidharth Ramachandran, Christian Winkler\n",
    "\n",
    "**If you like the book or the code examples here, please leave a friendly comment on [Amazon.com](https://www.amazon.com/Blueprints-Text-Analytics-Using-Python/dp/149207408X)!**\n",
    "<img src=\"../rating.png\" width=\"100\"/>\n",
    "\n",
    "\n",
    "# Chapter 9:<div class='tocSkip'/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Remark<div class='tocSkip'/>\n",
    "\n",
    "The code in this notebook differs slightly from the printed book. For example we frequently use pretty print (`pp.pprint`) instead of `print` and `tqdm`'s `progress_apply` instead of Pandas' `apply`. \n",
    "\n",
    "Moreover, several layout and formatting commands, like `figsize` to control figure size or subplot commands are removed in the book.\n",
    "\n",
    "You may also find some lines marked with three hashes ###. Those are not in the book as well as they don't contribute to the concept.\n",
    "\n",
    "All of this is done to simplify the code in the book and put the focus on the important parts instead of formatting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup<div class='tocSkip'/>\n",
    "\n",
    "Set directory locations. If working on Google Colab: copy files and install required libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "You are working on a local system.\n",
      "Files will be searched relative to \"..\".\n"
     ]
    }
   ],
   "source": [
    "import sys, os\n",
    "ON_COLAB = 'google.colab' in sys.modules\n",
    "\n",
    "if ON_COLAB:\n",
    "    GIT_ROOT = 'https://github.com/blueprints-for-text-analytics-python/blueprints-text/raw/master'\n",
    "    os.system(f'wget {GIT_ROOT}/ch09/setup.py')\n",
    "\n",
    "%run -i setup.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Python Settings<div class=\"tocSkip\"/>\n",
    "\n",
    "Common imports, defaults for formatting in Matplotlib, Pandas etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run \"$BASE_DIR/settings.py\"\n",
    "\n",
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%config InlineBackend.figure_format = 'png'\n",
    "\n",
    "# to print output of all statements and not just the last\n",
    "from IPython.core.interactiveshell import InteractiveShell\n",
    "InteractiveShell.ast_node_interactivity = \"all\"\n",
    "\n",
    "# otherwise text between $ signs will be interpreted as formula and printed in italic\n",
    "pd.set_option('display.html.use_mathjax', False)\n",
    "\n",
    "# path to import blueprints packages\n",
    "sys.path.append(BASE_DIR + '/packages')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# adjust matplotlib resolution for book version\n",
    "matplotlib.rcParams.update({'figure.dpi': 200 })"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from rouge_score import rouge_scorer\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from nltk import tokenize\n",
    "import matplotlib.pyplot as plt\n",
    "import html\n",
    "import re\n",
    "import random\n",
    "import rouge_score\n",
    "import wikipediaapi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to create short summaries of long textual information for quick reading"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Summarization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extractive Methods\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data pre-processing \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "import os.path\n",
    "from dateutil import parser\n",
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def download_article(url):\n",
    "    # check if article already there\n",
    "    filename = url.split(\"/\")[-1] + \".html\"\n",
    "    filename = f\"{BASE_DIR}/ch09/\" + filename\n",
    "    if not os.path.isfile(filename):\n",
    "        r = requests.get(url)\n",
    "        with open(filename, \"w+\") as f:\n",
    "            f.write(r.text)\n",
    "    return filename"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def parse_article(article_file):\n",
    "    with open(article_file, \"r\") as f:\n",
    "        html = f.read()\n",
    "    r = {}\n",
    "    soup = BeautifulSoup(html, 'html.parser')\n",
    "    r['id'] = soup.select_one(\"div.StandardArticle_inner-container\")['id']\n",
    "    r['url'] = soup.find(\"link\", {'rel': 'canonical'})['href']\n",
    "    r['headline'] = soup.h1.text\n",
    "    r['section'] = soup.select_one(\"div.ArticleHeader_channel a\").text\n",
    "    \n",
    "    r['text'] = soup.select_one(\"div.StandardArticleBody_body\").text\n",
    "    r['authors'] = [a.text \n",
    "                    for a in soup.select(\"div.BylineBar_first-container.ArticleHeader_byline-bar\\\n",
    "                                          div.BylineBar_byline span\")]\n",
    "    r['time'] = soup.find(\"meta\", { 'property': \"og:article:published_time\"})['content']\n",
    "    return r"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Article Published on '2018-03-15T11:36:28+0000'\n",
      "'LONDON/SAN FRANCISCO (Reuters) - U.S. President Donald Trump has blocked microchip maker Broadcom Ltd’s (AVGO.O) $117 billion takeover of rival Qualcomm (QCOM.O) amid concerns that it would give China the upper hand in the next generation of mobile communications, or 5G. A 5G sign is seen at the Mobile World Congress in Barcelona, Spain February 28, 2018. REUTERS/Yves HermanBelow are some facts... 4G wireless and looks set to top the list of patent holders heading into the 5G cycle. Huawei, Nokia, Ericsson and others are also vying to amass 5G patents, which has helped spur complex cross-licensing agreements like the deal struck late last year Nokia and Huawei around handsets. Editing by Kim Miyoung in Singapore and Jason Neely in LondonOur Standards:The Thomson Reuters Trust Principles.'\n"
     ]
    }
   ],
   "source": [
    "import reprlib\n",
    "r = reprlib.Repr()\n",
    "r.maxstring = 800\n",
    "\n",
    "url1 = \"https://www.reuters.com/article/us-qualcomm-m-a-broadcom-5g/what-is-5g-and-who-are-the-major-players-idUSKCN1GR1IN\"\n",
    "article_name1 = download_article(url1)\n",
    "article1 = parse_article(article_name1)\n",
    "print ('Article Published on', r.repr(article1['time']))\n",
    "print (r.repr(article1['text']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Blueprint - Summarizing text using topic representation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Identifying important words with TF-IDF values\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from nltk import tokenize\n",
    "\n",
    "sentences = tokenize.sent_tokenize(article1['text'])\n",
    "tfidfVectorizer = TfidfVectorizer()\n",
    "words_tfidf = tfidfVectorizer.fit_transform(sentences)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LONDON/SAN FRANCISCO (Reuters) - U.S. President Donald Trump has blocked microchip maker Broadcom Ltd’s (AVGO.O) $117 billion takeover of rival Qualcomm (QCOM.O) amid concerns that it would give China the upper hand in the next generation of mobile communications, or 5G.\n",
      "5G networks, now in the final testing stage, will rely on denser arrays of small antennas and the cloud to offer data speeds up to 50 or 100 times faster than current 4G networks and serve as critical infrastructure for a range of industries.\n",
      "The concern is that a takeover by Singapore-based Broadcom could see the firm cut research and development spending by Qualcomm or hive off strategically important parts of the company to other buyers, including in China, U.S. officials and analysts have said.\n"
     ]
    }
   ],
   "source": [
    "# Parameter to specify number of summary sentences required\n",
    "num_summary_sentence = 3\n",
    "\n",
    "# Sort the sentences in descending order by the sum of TF-IDF values\n",
    "sent_sum = words_tfidf.sum(axis=1)\n",
    "important_sent = np.argsort(sent_sum, axis=0)[::-1]\n",
    "\n",
    "# Print three most important sentences in the order they appear in the article\n",
    "for i in range(0, len(sentences)):\n",
    "    if i in important_sent[:num_summary_sentence]:\n",
    "        print (sentences[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from nltk import tokenize\n",
    "\n",
    "def tfidf_summary(text, num_summary_sentence):\n",
    "    summary_sentence = []\n",
    "    sentences = tokenize.sent_tokenize(text)\n",
    "    tfidfVectorizer = TfidfVectorizer()\n",
    "    words_tfidf = tfidfVectorizer.fit_transform(sentences)\n",
    "    sentence_sum = words_tfidf.sum(axis=1)\n",
    "    important_sentences = np.argsort(sentence_sum, axis=0)[::-1]\n",
    "    for i in range(0, len(sentences)):\n",
    "        if i in important_sentences[:num_summary_sentence]:\n",
    "            summary_sentence.append(sentences[i])\n",
    "    return summary_sentence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LSA algorithm\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LONDON/SAN FRANCISCO (Reuters) - U.S. President Donald Trump has blocked microchip maker Broadcom Ltd’s (AVGO.O) $117 billion takeover of rival Qualcomm (QCOM.O) amid concerns that it would give China the upper hand in the next generation of mobile communications, or 5G.\n",
      "Moving to new networks promises to enable new mobile services and even whole new business models, but could pose challenges for countries and industries unprepared to invest in the transition.\n",
      "The concern is that a takeover by Singapore-based Broadcom could see the firm cut research and development spending by Qualcomm or hive off strategically important parts of the company to other buyers, including in China, U.S. officials and analysts have said.\n"
     ]
    }
   ],
   "source": [
    "from sumy.parsers.plaintext import PlaintextParser\n",
    "from sumy.nlp.tokenizers import Tokenizer\n",
    "from sumy.nlp.stemmers import Stemmer\n",
    "from sumy.utils import get_stop_words\n",
    "\n",
    "from sumy.summarizers.lsa import LsaSummarizer\n",
    "\n",
    "LANGUAGE = \"english\"\n",
    "stemmer = Stemmer(LANGUAGE)\n",
    "\n",
    "parser = PlaintextParser.from_string(article1['text'], Tokenizer(LANGUAGE))\n",
    "summarizer = LsaSummarizer(stemmer)\n",
    "summarizer.stop_words = get_stop_words(LANGUAGE)\n",
    "\n",
    "for sentence in summarizer(parser.document, num_summary_sentence):\n",
    "    print (str(sentence))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sumy.parsers.plaintext import PlaintextParser\n",
    "from sumy.nlp.tokenizers import Tokenizer\n",
    "from sumy.nlp.stemmers import Stemmer\n",
    "from sumy.utils import get_stop_words\n",
    "from sumy.summarizers.lsa import LsaSummarizer\n",
    "\n",
    "def lsa_summary(text, num_summary_sentence):\n",
    "    summary_sentence = []\n",
    "    LANGUAGE = \"english\"\n",
    "    stemmer = Stemmer(LANGUAGE)\n",
    "    parser = PlaintextParser.from_string(text, Tokenizer(LANGUAGE))\n",
    "    summarizer = LsaSummarizer(stemmer)\n",
    "    summarizer.stop_words = get_stop_words(LANGUAGE)\n",
    "    for sentence in summarizer(parser.document, num_summary_sentence):\n",
    "        summary_sentence.append(str(sentence))\n",
    "    return summary_sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Article Published '2018-03-15T11:36:28+0000'\n",
      "'NEW YORK A protracted trade war between China and the United States, the world’s largest economies, and a deteriorating global growth outlook has left investors apprehensive about the end to the longest expansion in American history. FILE PHOTO: Ships and shipping containers are pictured at the port of Long Beach in Long Beach, California, U.S., January 30, 2019.   REUTERS/Mike BlakeThe recent ...hton wrote in the June Cass Freight Index report.  12. MISERY INDEX The so-called Misery Index adds together the unemployment rate and the inflation rate. It typically rises during recessions and sometimes prior to downturns. It has slipped lower in 2019 and does not look very miserable.  Reporting by Saqib Iqbal Ahmed; Editing by Chizu NomiyamaOur Standards:The Thomson Reuters Trust Principles.'\n"
     ]
    }
   ],
   "source": [
    "r.maxstring = 800\n",
    "url2 = \"https://www.reuters.com/article/us-usa-economy-watchlist-graphic/predicting-the-next-u-s-recession-idUSKCN1V31JE\"\n",
    "article_name2 = download_article(url2)\n",
    "article2 = parse_article(article_name2)\n",
    "print ('Article Published', r.repr(article1['time']))\n",
    "print (r.repr(article2['text']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "REUTERS/Mike BlakeThe recent rise in U.S.-China trade war tensions has brought forward the next U.S. recession, according to a majority of economists polled by Reuters who now expect the Federal Reserve to cut rates again in September and once more next year.\n",
      "On Tuesday, U.S. stocks jumped sharply higher and safe-havens like the Japanese yen and Gold retreated after the U.S. Trade Representative said additional tariffs on some Chinese goods, including cell phones and laptops, will be delayed to Dec. 15.\n",
      "ISM said its index of national factory activity slipped to 51.2 last month, the lowest reading since August 2016, as U.S. manufacturing activity slowed to a near three-year low in July and hiring at factories shifted into lower gear, suggesting a further loss of momentum in economic growth early in the third quarter.\n"
     ]
    }
   ],
   "source": [
    "summary_sentence = tfidf_summary(article2['text'], num_summary_sentence)\n",
    "for sentence in summary_sentence:\n",
    "    print (sentence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NEW YORK A protracted trade war between China and the United States, the world’s largest economies, and a deteriorating global growth outlook has left investors apprehensive about the end to the longest expansion in American history.\n",
      "REUTERS/Mike BlakeThe recent rise in U.S.-China trade war tensions has brought forward the next U.S. recession, according to a majority of economists polled by Reuters who now expect the Federal Reserve to cut rates again in September and once more next year.\n",
      "Trade tensions have pulled corporate confidence and global growth to multi-year lows and U.S. President Donald Trump’s announcement of more tariffs have raised downside risks significantly, Morgan Stanley analysts said in a recent note.\n"
     ]
    }
   ],
   "source": [
    "summary_sentence = lsa_summary(article2['text'], num_summary_sentence)\n",
    "for sentence in summary_sentence:\n",
    "    print (sentence)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Blueprint - Summarizing text using an indicator representation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "REUTERS/Mike BlakeThe recent rise in U.S.-China trade war tensions has brought forward the next U.S. recession, according to a majority of economists polled by Reuters who now expect the Federal Reserve to cut rates again in September and once more next year.\n",
      "As recession signals go, this so-called inversion in the yield curve has a solid track record as a predictor of recessions.\n",
      "Markets turned down before the 2001 recession and tumbled at the start of the 2008 recession.\n"
     ]
    }
   ],
   "source": [
    "from sumy.summarizers.text_rank import TextRankSummarizer\n",
    "\n",
    "parser = PlaintextParser.from_string(article2['text'], Tokenizer(LANGUAGE))\n",
    "summarizer = TextRankSummarizer(stemmer)\n",
    "summarizer.stop_words = get_stop_words(LANGUAGE)\n",
    "\n",
    "for sentence in summarizer(parser.document, num_summary_sentence):\n",
    "    print (str(sentence))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sumy.parsers.plaintext import PlaintextParser\n",
    "from sumy.nlp.tokenizers import Tokenizer\n",
    "from sumy.nlp.stemmers import Stemmer\n",
    "from sumy.utils import get_stop_words\n",
    "from sumy.summarizers.text_rank import TextRankSummarizer\n",
    "\n",
    "def textrank_summary(text, num_summary_sentence):\n",
    "    summary_sentence = []\n",
    "    LANGUAGE = \"english\"\n",
    "    stemmer = Stemmer(LANGUAGE)\n",
    "    parser = PlaintextParser.from_string(text, Tokenizer(LANGUAGE))\n",
    "    summarizer = TextRankSummarizer(stemmer)\n",
    "    summarizer.stop_words = get_stop_words(LANGUAGE)\n",
    "    for sentence in summarizer(parser.document, num_summary_sentence):\n",
    "        summary_sentence.append(str(sentence))\n",
    "    return summary_sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Acquiring Qualcomm would represent the jewel in the crown of Broadcom’s portfolio of communications chips, which supply wi-fi, power management, video and other features in smartphones alongside Qualcomm’s core baseband chips - radio modems that wirelessly connect phones to networks.\n",
      "Qualcomm (QCOM.O) is the dominant player in smartphone communications chips, making half of all core baseband radio chips in smartphones.\n",
      "Slideshow (2 Images)The standards are set by a global body to ensure all phones work across different mobile networks, and whoever’s essential patents end up making it into the standard stands to reap huge royalty licensing revenue streams.\n"
     ]
    }
   ],
   "source": [
    "parser = PlaintextParser.from_string(article1['text'], Tokenizer(LANGUAGE))\n",
    "summarizer = TextRankSummarizer(stemmer)\n",
    "summarizer.stop_words = get_stop_words(LANGUAGE)\n",
    "\n",
    "for sentence in summarizer(parser.document, num_summary_sentence):\n",
    "    print (str(sentence))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "import wikipediaapi\n",
    "\n",
    "wiki_wiki = wikipediaapi.Wikipedia(\n",
    "        language='en',\n",
    "        extract_format=wikipediaapi.ExtractFormat.WIKI\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "r.maxstring = 500"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'The Mongol invasion of Europe in the 13th century occurred from the 1220s into the 1240s. In Eastern Europe, the Mongols destroyed Volga Bulgaria, Cumania, Alania, and the Kievan Rus\\' federation. In Central Europe, the Mongol armies launched a tw...tnotes\\nReferences\\nSverdrup, Carl (2010). \"Numbers in Mongol Warfare\". Journal of Medieval Military History. Boydell Press. 8: 109–17 [p. 115]. ISBN 978-1-84383-596-7.\\n\\nFurther reading\\nExternal links\\nThe Islamic World to 1600: The Golden Horde'\n"
     ]
    }
   ],
   "source": [
    "p_wiki = wiki_wiki.page('Mongol_invasion_of_Europe')\n",
    "print (r.repr(p_wiki.text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In Central Europe, the Mongol armies launched a two-pronged invasion of fragmented Poland, culminating in the Battle of Legnica (9 April 1241), and the Kingdom of Hungary, culminating in the Battle of Mohi (11 April 1241).\n",
      "Warring European princes realized they had to cooperate in the face of a Mongol invasion, so local wars and conflicts were suspended in parts of central Europe, only to be resumed after the Mongols had withdrawn.\n",
      "Under Wenceslaus' leadership during the Mongol invasion, Bohemia remained one of a few European kingdoms that was never conquered and molested by the Mongols even though most kingdoms around it such as Poland and Moravia were ravaged.\n",
      "A major Hungarian loss was imminent, and the Mongols intentionally left a gap in their formation to permit the wavering Hungarian forces to flee and spread out in doing so, leaving them unable to effectively resist the Mongols as they picked off the retreating Hungarian remnants.\n",
      "European tactics against Mongols The traditional European method of warfare of melee combat between knights ended in catastrophe when it was deployed against the Mongol forces as the Mongols were able to keep a distance and advance with superior numbers.\n",
      "Austrian knights under Duke Frederick also fared better in fighting the Mongol invasion in Vienna.King Béla IV hired the help of the Knights of St. John, as well as training his own better-armed local knights, in preparation for the Second Mongol invasion of Hungary.\n",
      "After the division of the Mongol Empire into four fragments, when the Golden Horde attempted the next invasion of Hungary, Hungary had increased their proportion of knights (led by Ladislaus IV of Hungary) and they quickly defeated the main Golden Horde Army in the hills of western Transylvania.By this time as well, many Eastern and Central European countries had ended their hostilities with one another and united to finally drive out the remnants of the Golden Horde.\n",
      "An analysis of tree rings there found that Hungary had cold wet weather in early 1242, which likely turned Hungary's central plain into a huge swamp; so, lacking pastures for their horses, the Mongols would have had to fall back to Rus' in search of better grasslands.Regardless of their reasons, the Mongols had completely withdrawn from Central Europe by mid-1242, though they still launched military operations in the west at this time, most notably the 1241–1243 Mongol invasion of Anatolia.\n",
      "They have sometimes been collectively referred to as \"the second Mongol invasion of Europe\", \"the second Tatar-Mongol invasion of central and south-eastern Europe\", or \"the second Mongol invasion of central Europe.\"\n",
      "In Caffa in Crimea for example, when the Mongols under Janibeg besieged it after a large fight between Christians and Muslims began, a relief force of an Italian army came and defeated the Mongols, killing 15,000 of their troops and destroying their siege engines.\n"
     ]
    }
   ],
   "source": [
    "r.maxstring = 200\n",
    "\n",
    "num_summary_sentence = 10\n",
    "\n",
    "summary_sentence = textrank_summary(p_wiki.text, num_summary_sentence)\n",
    "\n",
    "for sentence in summary_sentence:\n",
    "    print (sentence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In more detail, our contributions are as follows: \\begin{itemize}[noitemsep] \\item{We introduce a new dataset for summarisation of scientific publications consisting of over 10k documents} \\item{Following the approach of \\cite{hermann2015teaching} in the news domain, we introduce a method, \\textit{HighlightROUGE}, which can be used to automatically extend this dataset %extractive summarisation datasets% and show empirically that this improves summarisation performance} \\item{Taking inspiration from previous work in summarising scientific literature \\citep{kupiec1995trainable, papers_citationSaggion2016}, we introduce a %further metric we use as a feature, \\textit{AbstractROUGE}, which can be used to extract summaries by exploiting the abstract of a paper} \\item{We benchmark several neural as well traditional summarisation methods on the dataset and use simple features to model the global context of a summary statement, which contribute most to the overall score} \\item{We compare our best performing system to several well-established baseline methods, some of which use more elaborate methods to model the global context than we do, and show that our best performing model outperforms them on this extractive summarisation task by a considerable margin} \\item{We analyse to what degree different sections in scientific papers contribute to a summary} \\end{itemize} We expect the research documented in this paper to be relevant beyond the document summarisation community, for other tasks in the space of automatically understand scientific publications, such as keyphrase extraction~\\cite{kim-EtAl:2010:SemEval,sterckx2016supervised,augenstein2017scienceie,augenstein2017multitask}, semantic relation extraction~\\cite{gupta-manning:2011:IJCNLP-2011,marsi-ozturk:2015:EMNLP} or topic classification of scientific articles~\\cite{oseaghdha-teufel:2014:Coling}.\n",
      "Therefore we assign sentences an integer location for 7 different sections: Highlight, Abstract, Introduction, Results / Discussion / Analysis, Method, Conclusion, all else.\\footnote{based on a small manually created gazetteer of alternative names} Location features have been used in other ways in previous work on summarising scientific literature; \\newcite{Visser2009} extract sentence location features based on the headings they occurred beneath while \\newcite{Teufel2002} divide the paper into 20 equal parts and assign each sentence a location based on which segment it occurred in - an attempt to capture distinct zones of the paper.\n",
      "\\paragraph{Single Feature Models} The simplest class of summarisers use a single feature from Section \\ref{sec:handcrafted_feats} (Sentence Length, Numeric Count and Section are excluded due to lack of granularity when sorting by these).% The sentences are sorted in descending order by the feature being used and the top $n$ sentences taken as the summary, then sorted back into the order they appear in in the paper.\n",
      "Word2Vec takes as input the sentence represented as an averaged word vector of 100 numbers.\\footnote{Word embeddings are obtained by training a Word2Vec skip-gram model on the 10000 papers with dimensionality 100, minimum word count 5, a context window of 20 words and downsample setting of 0.001} Word2VecAF takes the sentence average vector, abstract average vector and handcrafted features, giving a 208-dimensional vector for classification.\n",
      "Hence, although some sections are slightly more likely to contain good summary sentences, and assuming that we do not take summary sentences from the abstract which is already a summary, then Figure \\ref{fig:rouge_by_section} suggests that there is no definitive section from which summary sentences should be extracted.\n"
     ]
    }
   ],
   "source": [
    "parser = PlaintextParser.from_file(f\"{BASE_DIR}/ch09/acl2017.tex\", Tokenizer(LANGUAGE))\n",
    "summarizer = TextRankSummarizer(stemmer)\n",
    "summarizer.stop_words = get_stop_words(LANGUAGE)\n",
    "\n",
    "for sentence in summarizer(parser.document, 5):\n",
    "    print (str(sentence))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Measuring the performance of Text Summarization methods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_rouge_score(rouge_score):\n",
    "    for k,v in rouge_score.items():\n",
    "        print (k, 'Precision:', \"{:.2f}\".format(v.precision), 'Recall:', \"{:.2f}\".format(v.recall), 'fmeasure:', \"{:.2f}\".format(v.fmeasure))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rouge1 Precision: 0.06 Recall: 0.83 fmeasure: 0.11\n"
     ]
    }
   ],
   "source": [
    "num_summary_sentence = 3 ##\n",
    "gold_standard = article2['headline']\n",
    "summary = \"\"\n",
    "\n",
    "summary = ''.join(textrank_summary(article2['text'], num_summary_sentence))\n",
    "scorer = rouge_scorer.RougeScorer(['rouge1'], use_stemmer=True)\n",
    "scores = scorer.score(gold_standard, summary)\n",
    "print_rouge_score(scores)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rouge1 Precision: 0.04 Recall: 0.83 fmeasure: 0.08\n"
     ]
    }
   ],
   "source": [
    "summary = ''.join(lsa_summary(article2['text'], num_summary_sentence))\n",
    "scores = scorer.score(gold_standard, summary)\n",
    "print_rouge_score(scores)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rouge2 Precision: 0.18 Recall: 0.46 fmeasure: 0.26\n",
      "rougeL Precision: 0.16 Recall: 0.40 fmeasure: 0.23\n"
     ]
    }
   ],
   "source": [
    "num_summary_sentence = 10 ##\n",
    "gold_standard = p_wiki.summary\n",
    "\n",
    "summary = ''.join(textrank_summary(p_wiki.text, num_summary_sentence))\n",
    "\n",
    "scorer = rouge_scorer.RougeScorer(['rouge2','rougeL'], use_stemmer=True)\n",
    "scores = scorer.score(gold_standard, summary)\n",
    "print_rouge_score(scores)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rouge2 Precision: 0.04 Recall: 0.08 fmeasure: 0.05\n",
      "rougeL Precision: 0.12 Recall: 0.25 fmeasure: 0.16\n"
     ]
    }
   ],
   "source": [
    "summary = ''.join(lsa_summary(p_wiki.text, num_summary_sentence))\n",
    "\n",
    "scorer = rouge_scorer.RougeScorer(['rouge2','rougeL'], use_stemmer=True)\n",
    "scores = scorer.score(gold_standard, summary)\n",
    "print_rouge_score(scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Blueprint - Summarizing text using machine learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1 - Creating target labels\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe tex2jax_ignore\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>850</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Filename</th>\n",
       "      <td>60763_5_3122150</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ThreadID</th>\n",
       "      <td>60763_5_3122150</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Title</th>\n",
       "      <td>which attractions need to be pre booked?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>userID</th>\n",
       "      <td>musicqueenLon...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <td>29 September 2009, 1:41</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>postNum</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>text</th>\n",
       "      <td>Hi    I am coming to NY in Oct! So excited&amp;quot; Have wanted to visit for years.    We are planning on doing all the usual stuff so wont list it all but wondered which attractions should be pre booked and which can you just turn up at&gt;    I am plannin on booking ESB but what else?    thanks x</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>summary</th>\n",
       "      <td>A woman was planning to travel NYC in October and needed some suggestions about attractions in the NYC. She was planning on booking ESB.Someone suggested that the TOTR was much better compared to ESB. The other suggestion was to prebook the show to avoid wasting time in line.Someone also suggested her New York Party Shuttle tours.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                   850\n",
       "Filename                                                                                                                                                                                                                                                                                                                               60763_5_3122150\n",
       "ThreadID                                                                                                                                                                                                                                                                                                                               60763_5_3122150\n",
       "Title                                                                                                                                                                                                                                                                                                         which attractions need to be pre booked?\n",
       "userID                                                                                                                                                                                                                                                                                                                                musicqueenLon...\n",
       "Date                                                                                                                                                                                                                                                                                                                           29 September 2009, 1:41\n",
       "postNum                                                                                                                                                                                                                                                                                                                                              1\n",
       "text                                             Hi    I am coming to NY in Oct! So excited&quot; Have wanted to visit for years.    We are planning on doing all the usual stuff so wont list it all but wondered which attractions should be pre booked and which can you just turn up at>    I am plannin on booking ESB but what else?    thanks x\n",
       "summary   A woman was planning to travel NYC in October and needed some suggestions about attractions in the NYC. She was planning on booking ESB.Someone suggested that the TOTR was much better compared to ESB. The other suggestion was to prebook the show to avoid wasting time in line.Someone also suggested her New York Party Shuttle tours."
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "pd.set_option('display.max_colwidth', 500)\n",
    "pd.set_option('display.max_rows',1000)\n",
    "\n",
    "file = \"travel_threads.csv.gz\"\n",
    "file = f\"{BASE_DIR}/data/travel-forum-threads/travel_threads.csv.gz\" ### real location\n",
    "df = pd.read_csv(file, sep='|', dtype={'ThreadID': 'object'})\n",
    "df[df['ThreadID']=='60763_5_3122150'].head(1).T\n",
    "\n",
    "# You can view the actual post here ###\n",
    "# URL - https://www.tripadvisor.com/ShowTopic-g60763-i5-k3122150-Which_attractions_need_to_be_pre_booked-New_York_City_New_York.html ###"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cleaning up the data to remove special characters\n",
    "# Re-using the blueprint from Chapter 4\n",
    "# Renamed as regex_clean to differentiate with the next clean function\n",
    "from blueprints.preparation import clean as regex_clean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Re-using the blueprint from Chapter 4 but adapting to add additional steps specific to this dataset\n",
    "import re ###\n",
    "import spacy ###\n",
    "from spacy.tokenizer import Tokenizer\n",
    "from spacy.util import compile_prefix_regex, \\\n",
    "                       compile_infix_regex, compile_suffix_regex\n",
    "from textacy.preprocessing.replace import replace_urls\n",
    "import textacy\n",
    "\n",
    "def custom_tokenizer(nlp):\n",
    "    \n",
    "    # use default patterns except the ones matched by re.search\n",
    "    prefixes = [pattern for pattern in nlp.Defaults.prefixes \n",
    "                if pattern not in ['-', '_', '#']]\n",
    "    suffixes = [pattern for pattern in nlp.Defaults.suffixes\n",
    "                if pattern not in ['_']]\n",
    "    infixes  = [pattern for pattern in nlp.Defaults.infixes\n",
    "                if not re.search(pattern, 'xx-xx')]\n",
    "\n",
    "    return Tokenizer(vocab          = nlp.vocab, \n",
    "                     rules          = nlp.Defaults.tokenizer_exceptions,\n",
    "                     prefix_search  = compile_prefix_regex(prefixes).search,\n",
    "                     suffix_search  = compile_suffix_regex(suffixes).search,\n",
    "                     infix_finditer = compile_infix_regex(infixes).finditer,\n",
    "                     token_match    = nlp.Defaults.token_match)\n",
    "\n",
    "nlp = spacy.load('en')\n",
    "nlp.tokenizer = custom_tokenizer(nlp)\n",
    "\n",
    "def extract_lemmas(doc, **kwargs):\n",
    "    return [t.lemma_ for t in textacy.extract.words(doc, **kwargs)]\n",
    "\n",
    "def extract_noun_chunks(doc, include_pos=['NOUN'], sep='_'):\n",
    "    chunks = []\n",
    "    for noun_chunk in doc.noun_chunks:\n",
    "        chunk = [token.lemma_ for token in noun_chunk\n",
    "                 if token.pos_ in include_pos]\n",
    "        if len(chunk) >= 2:\n",
    "            chunks.append(sep.join(chunk))\n",
    "    return chunks\n",
    "\n",
    "def extract_entities(doc, include_types=None, sep='_'):\n",
    "\n",
    "    ents = textacy.extract.entities(doc, \n",
    "             include_types=include_types, \n",
    "             exclude_types=None, \n",
    "             drop_determiners=True, \n",
    "             min_freq=1)\n",
    "    \n",
    "    return [re.sub('\\s+', sep, e.lemma_)+'/'+e.label_ for e in ents]\n",
    "\n",
    "def clean(text):\n",
    "    # Replace URLs\n",
    "    text = replace_urls(text)\n",
    "    \n",
    "    # Replace semi-colons (relevant in Java code ending)\n",
    "    text = text.replace(';','')\n",
    "    \n",
    "    # Replace character tabs (present as literal in description field)\n",
    "    text = text.replace('\\t','')\n",
    "    \n",
    "    # Find and remove any stack traces - doesn't fix all code fragments but removes many exceptions\n",
    "    start_loc = text.find(\"Stack trace:\")\n",
    "    text = text[:start_loc]\n",
    "    \n",
    "    # Remove Hex Code\n",
    "    text = re.sub(r'(\\w+)0x\\w+', '', text)\n",
    "    \n",
    "    # Initialize Spacy\n",
    "    doc = nlp(text)\n",
    "    \n",
    "    # From Blueprint function\n",
    "    lemmas          = extract_lemmas(doc, \n",
    "                                     exclude_pos = ['PART', 'PUNCT', 'DET', 'PRON', 'SYM', 'SPACE', 'NUM'],\n",
    "                                     filter_stops = True,\n",
    "                                     filter_nums = True,\n",
    "                                     filter_punct = True)\n",
    "\n",
    "    return lemmas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Applying regex based cleaning function\n",
    "df['text'] = df['text'].apply(regex_clean)\n",
    "# Extracting lemmas using spacy pipeline\n",
    "df['lemmas'] = df['text'].apply(clean)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GroupShuffleSplit\n",
    "\n",
    "gss = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=42)\n",
    "train_split, test_split = next(gss.split(df, groups=df['ThreadID']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of threads for Training  559\n",
      "Number of threads for Testing  140\n"
     ]
    }
   ],
   "source": [
    "train_df = df.iloc[train_split]\n",
    "test_df = df.iloc[test_split]\n",
    "\n",
    "print ('Number of threads for Training ', train_df['ThreadID'].nunique())\n",
    "print ('Number of threads for Testing ', test_df['ThreadID'].nunique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "import textdistance\n",
    "\n",
    "compression_factor = 0.3\n",
    "\n",
    "train_df['similarity'] = train_df.apply(\n",
    "    lambda x: textdistance.jaro_winkler(x.text, x.summary), axis=1)\n",
    "train_df[\"rank\"] = train_df.groupby(\"ThreadID\")[\"similarity\"].rank(\n",
    "    \"max\", ascending=False)\n",
    "\n",
    "topN = lambda x: x <= np.ceil(compression_factor * x.max())\n",
    "train_df['summaryPost'] = train_df.groupby('ThreadID')['rank'].apply(topN)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe tex2jax_ignore\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>summaryPost</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>850</th>\n",
       "      <td>Hi I am coming to NY in Oct! So excited\" Have wanted to visit for years. We are planning on doing all the usual stuff so wont list it all but wondered which attractions should be pre booked and which can you just turn up at&gt; I am plannin on booking ESB but what else? thanks x</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>851</th>\n",
       "      <td>I wouldnt bother doing the ESB if I was you TOTR is much better. What other attractions do you have in mind?</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>852</th>\n",
       "      <td>The Statue of Liberty, if you plan on going to the statue itself or to Ellis Island (as opposed to taking a boat past): http://www.statuecruises.com/ Also, we prefer to book shows and plays in advance rather than trying for the same-day tickets, as that allows us to avoid wasting time in line. If that sounds appealing to you, have a look at http://www.broadwaybox.com/</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                   text  \\\n",
       "850                                                                                                Hi I am coming to NY in Oct! So excited\" Have wanted to visit for years. We are planning on doing all the usual stuff so wont list it all but wondered which attractions should be pre booked and which can you just turn up at> I am plannin on booking ESB but what else? thanks x   \n",
       "851                                                                                                                                                                                                                                                                        I wouldnt bother doing the ESB if I was you TOTR is much better. What other attractions do you have in mind?   \n",
       "852  The Statue of Liberty, if you plan on going to the statue itself or to Ellis Island (as opposed to taking a boat past): http://www.statuecruises.com/ Also, we prefer to book shows and plays in advance rather than trying for the same-day tickets, as that allows us to avoid wasting time in line. If that sounds appealing to you, have a look at http://www.broadwaybox.com/   \n",
       "\n",
       "     summaryPost  \n",
       "850         True  \n",
       "851        False  \n",
       "852         True  "
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df[['text','summaryPost']][train_df['ThreadID']=='60763_5_3122150'].head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2 - Adding features to assist model prediction\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df['titleSimilarity'] = train_df.apply(\n",
    "    lambda x: textdistance.jaro_winkler(x.text, x.Title), axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Adding post length as a feature\n",
    "train_df['textLength'] = train_df['text'].str.len()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df.loc[train_df['textLength'] <= 20, 'summaryPost'] = False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_cols = ['titleSimilarity','textLength','postNum']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df['combined'] = [\n",
    "    ' '.join(map(str, l)) for l in train_df['lemmas'] if l is not '']\n",
    "tfidf = TfidfVectorizer(min_df=10, ngram_range=(1, 2), stop_words=\"english\")\n",
    "tfidf_result = tfidf.fit_transform(train_df['combined']).toarray()\n",
    "\n",
    "tfidf_df = pd.DataFrame(tfidf_result, columns=tfidf.get_feature_names())\n",
    "tfidf_df.columns = [\"word_\" + str(x) for x in tfidf_df.columns]\n",
    "tfidf_df.index = train_df.index\n",
    "train_df_tf = pd.concat([train_df[feature_cols], tfidf_df], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_df['similarity'] = test_df.apply(lambda x: textdistance.jaro_winkler(x.text, x.summary), axis=1)\n",
    "test_df[\"rank\"] = test_df.groupby(\"ThreadID\")[\"similarity\"].rank(\"max\", ascending=False)\n",
    "\n",
    "topN = lambda x: x <= np.ceil(compression_factor * x.max())\n",
    "test_df['summaryPost'] = test_df.groupby('ThreadID')['rank'].apply(topN)\n",
    "\n",
    "test_df['titleSimilarity'] = test_df.apply(lambda x: textdistance.jaro_winkler(x.text, x.Title), axis=1)\n",
    "\n",
    "test_df['textLength'] = test_df['text'].str.len()\n",
    "\n",
    "test_df.loc[test_df['textLength'] <= 20, 'summaryPost'] = False\n",
    "\n",
    "test_df['combined'] = [' '.join(map(str, l)) for l in test_df['lemmas'] if l is not '']\n",
    "\n",
    "tfidf_result = tfidf.transform(test_df['combined']).toarray()\n",
    "\n",
    "tfidf_df = pd.DataFrame(tfidf_result, columns = tfidf.get_feature_names())\n",
    "tfidf_df.columns = [\"word_\" + str(x) for x in tfidf_df.columns]\n",
    "tfidf_df.index = test_df.index\n",
    "test_df_tf = pd.concat([test_df[feature_cols], tfidf_df], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3 - Build a machine learning model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier(random_state=20)"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "model1 = RandomForestClassifier(random_state=20)\n",
    "model1.fit(train_df_tf, train_df['summaryPost'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function to calculate rouge_score for each thread\n",
    "def calculate_rouge_score(x, column_name):\n",
    "    # Get the original summary - only first value since they are repeated\n",
    "    ref_summary = x['summary'].values[0]\n",
    "    \n",
    "    # Join all posts that have been predicted as summary\n",
    "    predicted_summary = ''.join(x['text'][x[column_name]])\n",
    "    \n",
    "    # Return the rouge score for each ThreadID\n",
    "    scorer = rouge_scorer.RougeScorer(['rouge1'], use_stemmer=True)\n",
    "    scores = scorer.score(ref_summary, predicted_summary)\n",
    "    return scores['rouge1'].fmeasure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean ROUGE-1 Score for test threads 0.3454255443194838\n"
     ]
    }
   ],
   "source": [
    "test_df['predictedSummaryPost'] = model1.predict(test_df_tf)\n",
    "print('Mean ROUGE-1 Score for test threads',\n",
    "      test_df.groupby('ThreadID')[['summary','text','predictedSummaryPost']] \\\n",
    "      .apply(calculate_rouge_score, column_name='predictedSummaryPost').mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['60763_5_3139646']"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random.seed(2)\n",
    "random.sample(test_df['ThreadID'].unique().tolist(), 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of posts 9\n",
      "Number of summary posts 2\n",
      "Title:  What's fun for kids?\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe tex2jax_ignore\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>postNum</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>813</th>\n",
       "      <td>4</td>\n",
       "      <td>Well, you're really in luck, because there's a lot going on, including the Elmwood Avenue Festival of the Arts (http://www.elmwoodartfest.org), with special activities for youngsters, performances (including one by Nikki Hicks, one of my favorite local vocalists), and food of all kinds. Elmwood Avenue is one of the area's most colorful and thriving neighborhoods, and very walkable. The Buffalo Irish Festival is also going on that weekend in Hamburg, as it happens, at the fairgrounds: www.buf...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>814</th>\n",
       "      <td>5</td>\n",
       "      <td>Depending on your time frame, a quick trip to Niagara Falls would be great. It is a 45 minute drive from Hamburg and well worth the investment of time. Otherwise you have some beaches in Angola to enjoy. If the girls like to shop you have the Galleria, which is a great expansive Mall. If you enjoy a more eclectic afternoon, lunch on Elmwood Avenue, a stroll through the Albright Know Art gallery, and hitting some of the hip shops would be a cool afternoon. Darien Lake Theme Park is 40 minutes...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     postNum  \\\n",
       "813        4   \n",
       "814        5   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    text  \n",
       "813  Well, you're really in luck, because there's a lot going on, including the Elmwood Avenue Festival of the Arts (http://www.elmwoodartfest.org), with special activities for youngsters, performances (including one by Nikki Hicks, one of my favorite local vocalists), and food of all kinds. Elmwood Avenue is one of the area's most colorful and thriving neighborhoods, and very walkable. The Buffalo Irish Festival is also going on that weekend in Hamburg, as it happens, at the fairgrounds: www.buf...  \n",
       "814  Depending on your time frame, a quick trip to Niagara Falls would be great. It is a 45 minute drive from Hamburg and well worth the investment of time. Otherwise you have some beaches in Angola to enjoy. If the girls like to shop you have the Galleria, which is a great expansive Mall. If you enjoy a more eclectic afternoon, lunch on Elmwood Avenue, a stroll through the Albright Know Art gallery, and hitting some of the hip shops would be a cool afternoon. Darien Lake Theme Park is 40 minutes...  "
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example_df = test_df[test_df['ThreadID'] == '60974_588_2180141']\n",
    "print('Total number of posts', example_df['postNum'].max())\n",
    "print('Number of summary posts',\n",
    "      example_df[example_df['predictedSummaryPost']].count().values[0])\n",
    "print('Title: ', example_df['Title'].values[0])\n",
    "example_df[['postNum', 'text']][example_df['predictedSummaryPost']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Closing Remarks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Further reading\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}