nltk-similarity-matrix-extractive.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "7afdac34",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to\n",
      "[nltk_data]     C:\\Users\\subha\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import nltk\n",
    "nltk.download('punkt')\n",
    "import re\n",
    "from nltk.corpus import stopwords"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18bdc1a5",
   "metadata": {},
   "source": [
    "dataset-"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "34e5d5c6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>article_id</th>\n",
       "      <th>article_text</th>\n",
       "      <th>source</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Maria Sharapova has basically no friends as te...</td>\n",
       "      <td>https://www.tennisworldusa.org/tennis/news/Mar...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>BASEL, Switzerland (AP), Roger Federer advance...</td>\n",
       "      <td>http://www.tennis.com/pro-game/2018/10/copil-s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>Roger Federer has revealed that organisers of ...</td>\n",
       "      <td>https://scroll.in/field/899938/tennis-roger-fe...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Kei Nishikori will try to end his long losing ...</td>\n",
       "      <td>http://www.tennis.com/pro-game/2018/10/nishiko...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>Federer, 37, first broke through on tour over ...</td>\n",
       "      <td>https://www.express.co.uk/sport/tennis/1036101...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   article_id                                       article_text  \\\n",
       "0           1  Maria Sharapova has basically no friends as te...   \n",
       "1           2  BASEL, Switzerland (AP), Roger Federer advance...   \n",
       "2           3  Roger Federer has revealed that organisers of ...   \n",
       "3           4  Kei Nishikori will try to end his long losing ...   \n",
       "4           5  Federer, 37, first broke through on tour over ...   \n",
       "\n",
       "                                              source  \n",
       "0  https://www.tennisworldusa.org/tennis/news/Mar...  \n",
       "1  http://www.tennis.com/pro-game/2018/10/copil-s...  \n",
       "2  https://scroll.in/field/899938/tennis-roger-fe...  \n",
       "3  http://www.tennis.com/pro-game/2018/10/nishiko...  \n",
       "4  https://www.express.co.uk/sport/tennis/1036101...  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('https://raw.githubusercontent.com/amankharwal/Website-data/master/tennis.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "d44d3af4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"BASEL, Switzerland (AP), Roger Federer advanced to the 14th Swiss Indoors final of his career by beating seventh-seeded Daniil Medvedev 6-1, 6-4 on Saturday. Seeking a ninth title at his hometown event, and a 99th overall, Federer will play 93th-ranked Marius Copil on Sunday. Federer dominated the 20th-ranked Medvedev and had his first match-point chance to break serve again at 5-1. He then dropped his serve to love, and let another match point slip in Medvedev's next service game by netting a backhand. He clinched on his fourth chance when Medvedev netted from the baseline. Copil upset expectations of a Federer final against Alexander Zverev in a 6-3, 6-7 (6), 6-4 win over the fifth-ranked German in the earlier semifinal. The Romanian aims for a first title after arriving at Basel without a career win over a top-10 opponent. Copil has two after also beating No. 6 Marin Cilic in the second round. Copil fired 26 aces past Zverev and never dropped serve, clinching after 2 1/2 hours with a forehand volley winner to break Zverev for the second time in the semifinal. He came through two rounds of qualifying last weekend to reach the Basel main draw, including beating Zverev's older brother, Mischa. Federer had an easier time than in his only previous match against Medvedev, a three-setter at Shanghai two weeks ago.\""
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['article_text'][1]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a829498",
   "metadata": {},
   "source": [
    "### Now, I will split the sequences into the data by tokenizing them using a list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "45f914b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.tokenize import sent_tokenize\n",
    "sentences = []\n",
    "for s in df['article_text']:\n",
    "    sentences.append(sent_tokenize(s))\n",
    "\n",
    "sentences = [y for x in sentences for y in x]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24f51311",
   "metadata": {},
   "source": [
    "### Now I am going to use the Glove method for word representation, it is an unsupervised learning algorithm developed by Stanford University to generate word integrations by aggregating the global word-to-word co-occurrence matrix from a corpus. To implement this method you have to download a file from here and store it into the same directory where your python file is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "9b5d1ec5",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\subha\\AppData\\Local\\Temp\\ipykernel_2924\\2999140426.py:10: FutureWarning: The default value of regex will change from True to False in a future version.\n",
      "  clean_sentences = pd.Series(sentences).str.replace(\"[^a-zA-Z]\", \" \")\n"
     ]
    }
   ],
   "source": [
    "word_embeddings = {}\n",
    "f = open('glove.6B.100d.txt', encoding='utf-8')\n",
    "for line in f:\n",
    "    values = line.split()\n",
    "    word = values[0]\n",
    "    coefs = np.asarray(values[1:], dtype='float32')\n",
    "    word_embeddings[word] = coefs\n",
    "f.close()\n",
    "\n",
    "clean_sentences = pd.Series(sentences).str.replace(\"[^a-zA-Z]\", \" \")\n",
    "clean_sentences = [s.lower() for s in clean_sentences]\n",
    "stop_words = stopwords.words('english')\n",
    "def remove_stopwords(sen):\n",
    "    sen_new = \" \".join([i for i in sen if i not in stop_words])\n",
    "    return sen_new\n",
    "clean_sentences = [remove_stopwords(r.split()) for r in clean_sentences]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "371f62f4",
   "metadata": {},
   "source": [
    "## Now, I will create vectors for the sentences:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "7afbf681",
   "metadata": {},
   "outputs": [],
   "source": [
    "sentence_vectors = []\n",
    "for i in clean_sentences:\n",
    "    if len(i) != 0:\n",
    "        v = sum([word_embeddings.get(w, np.zeros((100,))) for w in i.split()])/(len(i.split())+0.001)\n",
    "    else:\n",
    "        v = np.zeros((100,))\n",
    "    sentence_vectors.append(v)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ece89195",
   "metadata": {},
   "source": [
    "## Finding Similarities to Summarize Text\n",
    "\n",
    "### The next step is to find similarities between the sentences, and I will use the cosine similarity approach for this task. Let’s create an empty similarity matrix for this task and fill it with cosine similarities of sentences:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "68ef9108",
   "metadata": {},
   "outputs": [],
   "source": [
    "sim_mat = np.zeros([len(sentences), len(sentences)])\n",
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "for i in range(len(sentences)):\n",
    "    for j in range(len(sentences)):\n",
    "        if i != j:\n",
    "            sim_mat[i][j] = cosine_similarity(sentence_vectors[i].reshape(1,100), sentence_vectors[j].reshape(1,100))[0,0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "477acc44",
   "metadata": {},
   "source": [
    "### Now I am going to convert the sim_mat similarity matrix into the graph, the nodes in this graph will represent the sentences and the edges will represent the similarity scores between the sentences:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "cd84ce1c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import networkx as nx\n",
    "\n",
    "nx_graph = nx.from_numpy_array(sim_mat)\n",
    "scores = nx.pagerank(nx_graph)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd8bd34d",
   "metadata": {},
   "source": [
    "### Now, let’s summarize text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "73812c62",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ARTICLE:\n",
      "Maria Sharapova has basically no friends as tennis players on the WTA Tour. The Russian player has no problems in openly speaking about it and in a recent interview she said: 'I don't really hide any feelings too much. I think everyone knows this is my job here. When I'm on the courts or when I'm on the court playing, I'm a competitor and I want to beat every single person whether they're in the locker room or across the net.So I'm not the one to strike up a conversation about the weather and know that in the next few minutes I have to go and try to win a tennis match. I'm a pretty competitive girl. I say my hellos, but I'm not sending any players flowers as well. Uhm, I'm not really friendly or close to many players. I have not a lot of friends away from the courts.' When she said she is not really close to a lot of players, is that something strategic that she is doing? Is it different on the men's tour than the women's tour? 'No, not at all. I think just because you're in the same sport doesn't mean that you have to be friends with everyone just because you're categorized, you're a tennis player, so you're going to get along with tennis players. I think every person has different interests. I have friends that have completely different jobs and interests, and I've met them in very different parts of my life. I think everyone just thinks because we're tennis players we should be the greatest of friends. But ultimately tennis is just a very small part of what we do. There are so many other things that we're interested in, that we do.'\n",
      "\n",
      "\n",
      "SUMMARY:\n",
      "When I'm on the courts or when I'm on the court playing, I'm a competitor and I want to beat every single person whether they're in the locker room or across the net.So I'm not the one to strike up a conversation about the weather and know that in the next few minutes I have to go and try to win a tennis match.\n",
      "\n",
      "\n",
      "ARTICLE:\n",
      "BASEL, Switzerland (AP), Roger Federer advanced to the 14th Swiss Indoors final of his career by beating seventh-seeded Daniil Medvedev 6-1, 6-4 on Saturday. Seeking a ninth title at his hometown event, and a 99th overall, Federer will play 93th-ranked Marius Copil on Sunday. Federer dominated the 20th-ranked Medvedev and had his first match-point chance to break serve again at 5-1. He then dropped his serve to love, and let another match point slip in Medvedev's next service game by netting a backhand. He clinched on his fourth chance when Medvedev netted from the baseline. Copil upset expectations of a Federer final against Alexander Zverev in a 6-3, 6-7 (6), 6-4 win over the fifth-ranked German in the earlier semifinal. The Romanian aims for a first title after arriving at Basel without a career win over a top-10 opponent. Copil has two after also beating No. 6 Marin Cilic in the second round. Copil fired 26 aces past Zverev and never dropped serve, clinching after 2 1/2 hours with a forehand volley winner to break Zverev for the second time in the semifinal. He came through two rounds of qualifying last weekend to reach the Basel main draw, including beating Zverev's older brother, Mischa. Federer had an easier time than in his only previous match against Medvedev, a three-setter at Shanghai two weeks ago.\n",
      "\n",
      "\n",
      "SUMMARY:\n",
      "Major players feel that a big event in late November combined with one in January before the Australian Open will mean too much tennis and too little rest.\n",
      "\n",
      "\n",
      "ARTICLE:\n",
      "Roger Federer has revealed that organisers of the re-launched and condensed Davis Cup gave him three days to decide if he would commit to the controversial competition. Speaking at the Swiss Indoors tournament where he will play in Sundays final against Romanian qualifier Marius Copil, the world number three said that given the impossibly short time frame to make a decision, he opted out of any commitment. \"They only left me three days to decide\", Federer said. \"I didn't to have time to consult with all the people I had to consult. \"I could not make a decision in that time, so I told them to do what they wanted.\" The 20-time Grand Slam champion has voiced doubts about the wisdom of the one-week format to be introduced by organisers Kosmos, who have promised the International Tennis Federation up to $3 billion in prize money over the next quarter-century. The competition is set to feature 18 countries in the November 18-24 finals in Madrid next year, and will replace the classic home-and-away ties played four times per year for decades. Kosmos is headed by Barcelona footballer Gerard Pique, who is hoping fellow Spaniard Rafael Nadal will play in the upcoming event. Novak Djokovic has said he will give precedence to the ATP's intended re-launch of the defunct World Team Cup in January 2020, at various Australian venues. Major players feel that a big event in late November combined with one in January before the Australian Open will mean too much tennis and too little rest. Federer said earlier this month in Shanghai in that his chances of playing the Davis Cup were all but non-existent. \"I highly doubt it, of course. We will see what happens,\" he said. \"I do not think this was designed for me, anyhow. This was designed for the future generation of players.\" Argentina and Britain received wild cards to the new-look event, and will compete along with the four 2018 semi-finalists and the 12 teams who win qualifying rounds next February. \"I don't like being under that kind of pressure,\" Federer said of the deadline Kosmos handed him.\n",
      "\n",
      "\n",
      "SUMMARY:\n",
      "Speaking at the Swiss Indoors tournament where he will play in Sundays final against Romanian qualifier Marius Copil, the world number three said that given the impossibly short time frame to make a decision, he opted out of any commitment.\n",
      "\n",
      "\n",
      "ARTICLE:\n",
      "Kei Nishikori will try to end his long losing streak in ATP finals and Kevin Anderson will go for his second title of the year at the Erste Bank Open on Sunday. The fifth-seeded Nishikori reached his third final of 2018 after beating Mikhail Kukushkin of Kazakhstan 6-4, 6-3 in the semifinals. A winner of 11 ATP events, Nishikori hasn't triumphed since winning in Memphis in February 2016. He has lost eight straight finals since. The second-seeded Anderson defeated Fernando Verdasco 6-3, 3-6, 6-4. Anderson has a shot at a fifth career title and second of the year after winning in New York in February. Nishikori leads Anderson 4-2 on career matchups, but the South African won their only previous meeting this year. With a victory on Sunday, Anderson will qualify for the ATP Finals. Currently in ninth place, Nishikori with a win could move to within 125 points of the cut for the eight-man event in London next month. Nishikori held serve throughout against Kukushkin, who came through qualifying. He used his first break point to close out the first set before going up 3-0 in the second and wrapping up the win on his first match point. Against Verdasco, Anderson hit nine of his 19 aces in the opening set. The Spaniard broke Anderson twice in the second but didn't get another chance on the South African's serve in the final set.\n",
      "\n",
      "\n",
      "SUMMARY:\n",
      "\"I felt like the best weeks that I had to get to know players when I was playing were the Fed Cup weeks or the Olympic weeks, not necessarily during the tournaments.\n",
      "\n",
      "\n",
      "ARTICLE:\n",
      "Federer, 37, first broke through on tour over two decades ago and he has since gone on to enjoy a glittering career. The 20-time Grand Slam winner is chasing his 99th ATP title at the Swiss Indoors this week and he faces Jan-Lennard Struff in the second round on Thursday (6pm BST). Davenport enjoyed most of her success in the late 1990s and her third and final major tournament win came at the 2000 Australian Open. But she claims the mentality of professional tennis players slowly began to change after the new millennium. \"It seems pretty friendly right now,\" said Davenport. \"I think there is a really nice environment and a great atmosphere, especially between some of the veteran players helping some of the younger players out. \"It's a very pleasant atmosphere, I'd have to say, around the locker rooms. \"I felt like the best weeks that I had to get to know players when I was playing were the Fed Cup weeks or the Olympic weeks, not necessarily during the tournaments. \"And even though maybe we had smaller teams, I still think we kept to ourselves quite a bit. \"Not always, but I really feel like in the mid-2000 years there was a huge shift of the attitudes of the top players and being more friendly and being more giving, and a lot of that had to do with players like Roger coming up. \"I just felt like it really kind of changed where people were a little bit, definitely in the 90s, a lot more quiet, into themselves, and then it started to become better.\" Meanwhile, Federer is hoping he can improve his service game as he hunts his ninth Swiss Indoors title this week. \"I didn't serve very well [against first-round opponent Filip Kranjovic,\" Federer said. \"I think I was misfiring the corners, I was not hitting the lines enough. \"Clearly you make your life more difficult, but still I was up 6-2, 3-1, break points, so things could have ended very quickly today, even though I didn't have the best serve percentage stats. \"But maybe that's exactly what caught up to me eventually. It's just getting used to it. This is where the first rounds can be tricky.\"\n",
      "\n",
      "\n",
      "SUMMARY:\n",
      "Currently in ninth place, Nishikori with a win could move to within 125 points of the cut for the eight-man event in London next month.\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "ranked_sentences = sorted(((scores[i],s) for i,s in enumerate(sentences)), reverse=True)\n",
    "for i in range(5):\n",
    "    print(\"ARTICLE:\")\n",
    "    print(df['article_text'][i])\n",
    "    print('\\n')\n",
    "    print(\"SUMMARY:\")\n",
    "    print(ranked_sentences[i][1])\n",
    "    print('\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dcbbac38",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}