NLP-feature-engineering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# NLP Feature Engineering\n",
    "\n",
    "Contents:\n",
    "\n",
    "* Extracting Basic Features\n",
    "    - Number of Characters\n",
    "    - Number of Words\n",
    "    - Average Word Length\n",
    "    - Number of Hastags and Mentions (Social Media)\n",
    "    - Number of Sentences\n",
    "    - Number of Paragraphs\n",
    "    - Words Starting with an Uppercase\n",
    "    - All-capital Words\n",
    "    - Numeric quantities\n",
    "* Text Preprocessing\n",
    "    - Tokenization/Segmentation\n",
    "    - Lemmatization (and Stemming)\n",
    "    - Converting to Lowercase\n",
    "    - Text Cleaning\n",
    "        - Removing Unnecessary Whitespaces and Escape Squences\n",
    "        - Removing Punctuations\n",
    "        - Removing Stopwords or Commonly Occuring Words/Tokens\n",
    "        - Removing Special Characters (emojis, numbers...)\n",
    "        - Expanding Contractions (don't, etc.)\n",
    "* Extracting Word Features\n",
    "    - Parts-of-Speech (POS) Tagging\n",
    "    - Named Entity Recognition (NEG)\n",
    "* Dependency Parsing (Not in the Notebook)\n",
    "* Vectorization (Convert documents into a set of numerical features)\n",
    "    - Bag of Words (Bag of n-grams)\n",
    "    - tf-idf\n",
    "\n",
    "_Dataset: [troll-tweets](https://github.com/fivethirtyeight/russian-troll-tweets)_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import necessary modules\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import spacy\n",
    "\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "from sklearn.metrics.pairwise import linear_kernel\n",
    "\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#df = pd.read_csv(\"data/IRAhandle_tweets_1.csv\")\n",
    "df = pd.read_pickle(\"data/tweets1.pkl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>content</th>\n",
       "      <th>author</th>\n",
       "      <th>language</th>\n",
       "      <th>retweet</th>\n",
       "      <th>followers</th>\n",
       "      <th>following</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>\"We have a sitting Democrat US Senator on tria...</td>\n",
       "      <td>10_GOP</td>\n",
       "      <td>English</td>\n",
       "      <td>0</td>\n",
       "      <td>9636</td>\n",
       "      <td>1052</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Marshawn Lynch arrives to game in anti-Trump s...</td>\n",
       "      <td>10_GOP</td>\n",
       "      <td>English</td>\n",
       "      <td>0</td>\n",
       "      <td>9637</td>\n",
       "      <td>1054</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Daughter of fallen Navy Sailor delivers powerf...</td>\n",
       "      <td>10_GOP</td>\n",
       "      <td>English</td>\n",
       "      <td>1</td>\n",
       "      <td>9637</td>\n",
       "      <td>1054</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>JUST IN: President Trump dedicates Presidents ...</td>\n",
       "      <td>10_GOP</td>\n",
       "      <td>English</td>\n",
       "      <td>0</td>\n",
       "      <td>9642</td>\n",
       "      <td>1062</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>19,000 RESPECTING our National Anthem! #StandF...</td>\n",
       "      <td>10_GOP</td>\n",
       "      <td>English</td>\n",
       "      <td>1</td>\n",
       "      <td>9645</td>\n",
       "      <td>1050</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             content  author language  \\\n",
       "0  \"We have a sitting Democrat US Senator on tria...  10_GOP  English   \n",
       "1  Marshawn Lynch arrives to game in anti-Trump s...  10_GOP  English   \n",
       "2  Daughter of fallen Navy Sailor delivers powerf...  10_GOP  English   \n",
       "3  JUST IN: President Trump dedicates Presidents ...  10_GOP  English   \n",
       "4  19,000 RESPECTING our National Anthem! #StandF...  10_GOP  English   \n",
       "\n",
       "   retweet  followers  following  \n",
       "0        0       9636       1052  \n",
       "1        0       9637       1054  \n",
       "2        1       9637       1054  \n",
       "3        0       9642       1062  \n",
       "4        1       9645       1050  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['content', 'author', 'language', 'retweet', 'followers', 'following'], dtype='object')"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df[['content', 'author', 'language', 'retweet', 'followers', 'following']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>content</th>\n",
       "      <th>author</th>\n",
       "      <th>language</th>\n",
       "      <th>retweet</th>\n",
       "      <th>followers</th>\n",
       "      <th>following</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>243886</th>\n",
       "      <td>BREAKING: Killer avalanche sweeps three skiers...</td>\n",
       "      <td>AUSTINLOVESBEER</td>\n",
       "      <td>English</td>\n",
       "      <td>1</td>\n",
       "      <td>34</td>\n",
       "      <td>41</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>243887</th>\n",
       "      <td>Why men should support International Women’s D...</td>\n",
       "      <td>AUSTINLOVESBEER</td>\n",
       "      <td>English</td>\n",
       "      <td>1</td>\n",
       "      <td>34</td>\n",
       "      <td>41</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>243888</th>\n",
       "      <td>How we can rebuild trust in a UK divided by in...</td>\n",
       "      <td>AUSTINLOVESBEER</td>\n",
       "      <td>English</td>\n",
       "      <td>1</td>\n",
       "      <td>34</td>\n",
       "      <td>41</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>243889</th>\n",
       "      <td>John Humphrys accused of patronising Angela Ra...</td>\n",
       "      <td>AUSTINLOVESBEER</td>\n",
       "      <td>English</td>\n",
       "      <td>1</td>\n",
       "      <td>34</td>\n",
       "      <td>41</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>243890</th>\n",
       "      <td>Fossilized poop found in 180-million-year-old ...</td>\n",
       "      <td>AUSTINLOVESBEER</td>\n",
       "      <td>English</td>\n",
       "      <td>1</td>\n",
       "      <td>34</td>\n",
       "      <td>41</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                  content           author  \\\n",
       "243886  BREAKING: Killer avalanche sweeps three skiers...  AUSTINLOVESBEER   \n",
       "243887  Why men should support International Women’s D...  AUSTINLOVESBEER   \n",
       "243888  How we can rebuild trust in a UK divided by in...  AUSTINLOVESBEER   \n",
       "243889  John Humphrys accused of patronising Angela Ra...  AUSTINLOVESBEER   \n",
       "243890  Fossilized poop found in 180-million-year-old ...  AUSTINLOVESBEER   \n",
       "\n",
       "       language  retweet  followers  following  \n",
       "243886  English        1         34         41  \n",
       "243887  English        1         34         41  \n",
       "243888  English        1         34         41  \n",
       "243889  English        1         34         41  \n",
       "243890  English        1         34         41  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic Feature Extraction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Character Count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "104.59038258894343\n"
     ]
    }
   ],
   "source": [
    "# Create a feature char_count\n",
    "df['char_count'] = df['content'].apply(len)\n",
    "\n",
    "# Print the average character count\n",
    "print(df['char_count'].mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The average character count of a tweet is 33-34 characters. However, the average value of this dataset is 104, which is significantly higher than the average. This might prove an important feature.\n",
    "> The most common length of a tweet back when Twitter only allowed 140 characters was 34 characters. Now that the limit is 280 characters, the most common length of a tweet is 33 characters. Historically, only 9% of tweets hit Twitter’s 140-character limit, now it’s 1%. [source](https://techcrunch.com/2018/10/30/twitters-doubling-of-character-count-from-140-to-280-had-little-impact-on-length-of-tweets/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Word Counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def count_words(string):\n",
    "    '''Returns number of words in a string'''\n",
    "    \n",
    "    # Split the string into words\n",
    "    words = string.split()\n",
    "    \n",
    "    # Return the number of words\n",
    "    return len(words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "13.396980618391003\n"
     ]
    }
   ],
   "source": [
    "# Create a new feature word_count\n",
    "df['word_count'] = df.content.apply(count_words)\n",
    "\n",
    "# Print the average word count of the talks\n",
    "print(df['word_count'].mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can compute the correlation of `word_count` with other features and derive interesting insights."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sentence Counts "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def count_sentences(string):\n",
    "    '''Returns number of sentences in a string'''\n",
    "    \n",
    "    # Split the string into sentences\n",
    "    words = string.split('.')\n",
    "    \n",
    "    # Return the number of sentences\n",
    "    return len(words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.604028028914556\n"
     ]
    }
   ],
   "source": [
    "# Create a new feature word_count\n",
    "df['sentence_count'] = df.content.apply(count_sentences)\n",
    "\n",
    "# Print the average word count of the talks\n",
    "print(df['sentence_count'].mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hashtag and Mention Counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def count_hashtags(string):\n",
    "    '''Returns number of hashtags in a string.'''\n",
    "    \n",
    "    # Split the string into words\n",
    "    words = string.split()\n",
    "    \n",
    "    # Create a list of words that are hashtags\n",
    "    hashtags = [word for word in words if word.startswith(\"#\")]\n",
    "    \n",
    "    # Return number of hashtags\n",
    "    return(len(hashtags))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAEICAYAAACj2qi6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAW/UlEQVR4nO3de5SlVX3m8e8jaETuDLEHudiasDISXCr2AGtMYiVEBE0Gs5YXWEQawyyiAzGZ9KwRLwkOhIQ4wSRgZMShh0ZRQzQOZASxQ6xRZ7wADtoQYuiFLbTdA0Ij0GAurb/54+yCQ3Gqa1dVN6e76/tZ66xzar/73ZfzNuepd79vHVJVSJLU4xnjHoAkaedhaEiSuhkakqRuhoYkqZuhIUnqZmhIkroZGnpaJPmvSX5nG7V1WJLNSXZrP08m+Xfbou3W3vVJlm+r9ubQ7+8luT/J/3u6++6R5PQkXxr3ODRehoYWLMm6JD9I8kiS7yf5P0nemuTxf19V9daqOr+zrV/cWp2quruq9qqqH26Dsb83yUentX9iVa1aaNtzHMehwArgiKr6l7PUPSLJ/26vz0vy9qFtE0nWj9hnmwbrDOPa7n1o/AwNbSu/XFV7A88HLgTeAVy+rTtJsvu2bnMH8Xzggaq6r6Puy4Fbhl5/fbuNSprG0NA2VVUPVdW1wJuA5UmOBEhyRZLfa68PTPI/21nJpiRfTPKMJB8BDgP+qi0//ackS5NUkjOS3A38zVDZcID8RJKvJXkoyTVJDmh9PeU376mzmSQnAO8C3tT6+0bb/vhvzG1c70nynST3Jbkyyb5t29Q4lie5uy0tvXum9ybJvm3/77X23tPa/0VgNfC8No4rZnmbl/FEaLwMuHWW+tPHsX97/7+X5MH2+pCh7acnuaudOX47yanT9v+jtt+3k5zYyi4Afhb4QJvDB1r5nya5J8nDSW5J8rND7eyRZFVr6452vJ9ylqQdi6Gh7aKqvgasZ/BBMt2Ktu3HgSUMPrirqt4M3M3grGWvqnrf0D6vBF4EvHqGLk8Dfg14HrAFuLhjjJ8Ffh/489bfS0ZUO709fh54IbAX8IFpdX4G+CngOOB3k7xohi4vAfZt7byyjfktVfXXwInAhjaO00ftnGR1ku8DZwGXJHmYwfu3Psn1s813yDOA/87g7OYw4AdTc0qyJ4P37sR25vhveHIoHQN8CzgQeB9weZJU1buBLwJntzmc3erfBLwUOAD4GPAXSZ7dtp0LLG3vx6uAX53DHDQmhoa2pw0MPiym+2fgIOD5VfXPVfXFmv1L0N5bVY9W1Q9m2P6Rqrqtqh4Ffgd449SF8gU6FXh/Vd1VVZuBdwInTzvL+c9V9YOq+gbwDeAp4dPG8ibgnVX1SFWtAy4C3tw7kKp6FXA0cGtV7cNgGfCcqtqvqk4cqvq8dhb3+INBsE2180BVfaqqHquqR4ALGITYlB8BRybZo6o2VtXtQ9u+U1UfbteTVjE4jku2MuaPtv62VNVFwI8xCFiANwK/X1UPVtV6OoJe42doaHs6GNg0ovy/AGuBz7VlkHM62rpnDtu/AzyTwW/DC/W81t5w27vz5A/K4budHmNwNjLdgcCzRrR1cM8gkpzdPvy/Afx0e30+8J4WDM8dqr6hBcnjD+BLQ209J8mH2hLZw8AXgP2S7NZC903AW4GNST6T5F+NmmtVPdZejprvVF8r2tLTQ23M+/LEcXkeTz5usx1j7QAMDW0XSf41gw/Ep9yi2X7TXlFVLwR+GfjtJMdNbZ6hydnORA4den0Yg7OZ+4FHgecMjWs3Bstive1uYLCMM9z2FuDeWfab7v42pultfbdn56r6QPvw/1/AL7R2vltV+7Zg6LmAPmUFg9/2j2lnLD/XytP6uqGd1RwE/B3w4c52n/RetusX72BwRrF/G/9DU/0AG4FDhnYZPobaQRka2qaS7JPkl4BPAB+tqjUj6vxSkp9MEuBh4IftAYMP4xfOo+tfzeBW1OcA5wGfbEsofw88O8lrkzwTeA+DJZIp9wJLM3R78DQfB/5Dkhck2YsnroFsmcvg2liuBi5IsneS5wO/DXx063s+xUsYnG0cxfzvmtqbwXWM77cbBs6d2pBkSZJ/265t/COwmSeOzWymH7u9GQTs94Ddk/wusM/Q9quBd7YL8wcDZ6MdnqGhbeWvkjzCYInh3cD7gbfMUPdw4K8ZfCB9GfhgVU22bX/AE0su/3EO/X8EuILB8smzgbfD4G4u4N8D/43Bb/WPMrgIP+Uv2vMDSUZ9CK9sbX8B+DbwD8BvzGFcw36j9X8XgzOwj7X2uyQ5DNjUloWO4ok7qObqT4A9GJz9fAX47NC2ZzA4E9nAYGnxlQzevx5/Cry+3Q11MXADcD2D4P4Og/dueAnqPAbH4tsM/j18kkFQaQcW/ydMknYESd4GnFxVr5y1ssbGMw1JY5HkoCSvaH+r8lMMznA+Pe5xaet21b+ulbTjexbwIeAFwPcZXAf74FhHpFm5PCVJ6ubylCSp2y63PHXggQfW0qVL57Xvo48+yp577rltB7QTWczzd+6Lc+6wuOc/PPdbbrnl/qr68Vl22fVCY+nSpdx8883z2ndycpKJiYltO6CdyGKev3OfGPcwxmYxz3947km+s/XaAy5PSZK6GRqSpG6GhiSpm6EhSepmaEiSuhkakqRuhoYkqZuhIUnqZmhIkrrtcn8RvhBrvvsQp5/zmbH0ve7C146lX0maC880JEndDA1JUjdDQ5LUzdCQJHUzNCRJ3QwNSVI3Q0OS1M3QkCR1MzQkSd0MDUlSN0NDktTN0JAkdTM0JEndDA1JUjdDQ5LUzdCQJHUzNCRJ3QwNSVI3Q0OS1M3QkCR1MzQkSd0MDUlSN0NDktRt1tBIcmiSzye5I8ntSX6zlR+QZHWSO9vz/q08SS5OsjbJN5McNdTW8lb/ziTLh8pfnmRN2+fiJNlaH5Kk8eg509gCrKiqFwHHAmclOQI4B7ixqg4Hbmw/A5wIHN4eZwKXwiAAgHOBY4CjgXOHQuDSVndqvxNa+Ux9SJLGYNbQqKqNVfX19voR4A7gYOAkYFWrtgp4XXt9EnBlDXwF2C/JQcCrgdVVtamqHgRWAye0bftU1ZerqoArp7U1qg9J0hjsPpfKSZYCLwO+Ciypqo0wCJYkz23VDgbuGdptfSvbWvn6EeVspY/p4zqTwZkKS5YsYXJyci7TetySPWDFi7fMa9+Fmu+Yt6XNmzfvEOMYB+c+Oe5hjM1inv985t4dGkn2Aj4F/FZVPdwuO4ysOqKs5lHeraouAy4DWLZsWU1MTMxl98ddctU1XLRmTjm6zaw7dWIs/Q6bnJxkvu/dzs65T4x7GGOzmOc/n7l33T2V5JkMAuOqqvrLVnxvW1qiPd/XytcDhw7tfgiwYZbyQ0aUb60PSdIY9Nw9FeBy4I6qev/QpmuBqTuglgPXDJWf1u6iOhZ4qC0x3QAcn2T/dgH8eOCGtu2RJMe2vk6b1taoPiRJY9CzFvMK4M3AmiS3trJ3ARcCVyc5A7gbeEPbdh3wGmAt8BjwFoCq2pTkfOCmVu+8qtrUXr8NuALYA7i+PdhKH5KkMZg1NKrqS4y+7gBw3Ij6BZw1Q1srgZUjym8GjhxR/sCoPiRJ4+FfhEuSuhkakqRuhoYkqZuhIUnqZmhIkroZGpKkboaGJKmboSFJ6mZoSJK6GRqSpG6GhiSpm6EhSepmaEiSuhkakqRuhoYkqZuhIUnqZmhIkroZGpKkboaGJKmboSFJ6mZoSJK6GRqSpG6GhiSpm6EhSepmaEiSuhkakqRuhoYkqZuhIUnqZmhIkroZGpKkboaGJKmboSFJ6mZoSJK6GRqSpG6GhiSpm6EhSeo2a2gkWZnkviS3DZW9N8l3k9zaHq8Z2vbOJGuTfCvJq4fKT2hla5OcM1T+giRfTXJnkj9P8qxW/mPt57Vt+9JtNWlJ0vz0nGlcAZwwovyPq+ql7XEdQJIjgJOBn277fDDJbkl2A/4MOBE4Ajil1QX4w9bW4cCDwBmt/Azgwar6SeCPWz1J0hjNGhpV9QVgU2d7JwGfqKp/rKpvA2uBo9tjbVXdVVX/BHwCOClJgF8APtn2XwW8bqitVe31J4HjWn1J0pjsvoB9z05yGnAzsKKqHgQOBr4yVGd9KwO4Z1r5McC/AL5fVVtG1D94ap+q2pLkoVb//ukDSXImcCbAkiVLmJycnNeEluwBK168ZfaK28F8x7wtbd68eYcYxzg498lxD2NsFvP85zP3+YbGpcD5QLXni4BfA0adCRSjz2hqK/WZZduTC6suAy4DWLZsWU1MTGxl6DO75KpruGjNQnJ0/tadOjGWfodNTk4y3/duZ+fcJ8Y9jLFZzPOfz9zndfdUVd1bVT+sqh8BH2aw/ASDM4VDh6oeAmzYSvn9wH5Jdp9W/qS22vZ96V8mkyRtB/MKjSQHDf34K8DUnVXXAie3O59eABwOfA24CTi83Sn1LAYXy6+tqgI+D7y+7b8cuGaoreXt9euBv2n1JUljMutaTJKPAxPAgUnWA+cCE0leymC5aB3w6wBVdXuSq4G/BbYAZ1XVD1s7ZwM3ALsBK6vq9tbFO4BPJPk94P8Cl7fyy4GPJFnL4Azj5AXPVpK0ILOGRlWdMqL48hFlU/UvAC4YUX4dcN2I8rt4YnlruPwfgDfMNj5J0tPHvwiXJHUzNCRJ3QwNSVI3Q0OS1M3QkCR1MzQkSd0MDUlSN0NDktTN0JAkdTM0JEndDA1JUjdDQ5LUzdCQJHUzNCRJ3QwNSVI3Q0OS1M3QkCR1MzQkSd0MDUlSN0NDktTN0JAkdTM0JEndDA1JUjdDQ5LUzdCQJHUzNCRJ3QwNSVI3Q0OS1M3QkCR1MzQkSd0MDUlSN0NDktTN0JAkdTM0JEndDA1JUjdDQ5LUbdbQSLIyyX1JbhsqOyDJ6iR3tuf9W3mSXJxkbZJvJjlqaJ/lrf6dSZYPlb88yZq2z8VJsrU+JEnj03OmcQVwwrSyc4Abq+pw4Mb2M8CJwOHtcSZwKQwCADgXOAY4Gjh3KAQubXWn9jthlj4kSWMya2hU1ReATdOKTwJWtdergNcNlV9ZA18B9ktyEPBqYHVVbaqqB4HVwAlt2z5V9eWqKuDKaW2N6kOSNCa7z3O/JVW1EaCqNiZ5bis/GLhnqN76Vra18vUjyrfWx1MkOZPB2QpLlixhcnJyfpPaA1a8eMu89l2o+Y55W9q8efMOMY5xcO6T4x7G2Czm+c9n7vMNjZlkRFnNo3xOquoy4DKAZcuW1cTExFybAOCSq67hojXb+i3ps+7UibH0O2xycpL5vnc7O+c+Me5hjM1inv985j7fu6fubUtLtOf7Wvl64NCheocAG2YpP2RE+db6kCSNyXxD41pg6g6o5cA1Q+WntbuojgUeaktMNwDHJ9m/XQA/HrihbXskybHtrqnTprU1qg9J0pjMuhaT5OPABHBgkvUM7oK6ELg6yRnA3cAbWvXrgNcAa4HHgLcAVNWmJOcDN7V651XV1MX1tzG4Q2sP4Pr2YCt9SJLGZNbQqKpTZth03Ii6BZw1QzsrgZUjym8GjhxR/sCoPiRJ4+NfhEuSuhkakqRuhoYkqZuhIUnqZmhIkroZGpKkboaGJKmboSFJ6mZoSJK6GRqSpG6GhiSpm6EhSepmaEiSuhkakqRuhoYkqZuhIUnqZmhIkroZGpKkboaGJKmboSFJ6mZoSJK6GRqSpG6GhiSpm6EhSepmaEiSuhkakqRuhoYkqZuhIUnqZmhIkroZGpKkboaGJKmboSFJ6mZoSJK6GRqSpG6GhiSpm6EhSeq2oNBIsi7JmiS3Jrm5lR2QZHWSO9vz/q08SS5OsjbJN5McNdTO8lb/ziTLh8pf3tpf2/bNQsYrSVqYbXGm8fNV9dKqWtZ+Pge4saoOB25sPwOcCBzeHmcCl8IgZIBzgWOAo4Fzp4Km1TlzaL8TtsF4JUnztD2Wp04CVrXXq4DXDZVfWQNfAfZLchDwamB1VW2qqgeB1cAJbds+VfXlqirgyqG2JEljsPsC9y/gc0kK+FBVXQYsqaqNAFW1MclzW92DgXuG9l3fyrZWvn5E+VMkOZPBGQlLlixhcnJyXpNZsgesePGWee27UPMd87a0efPmHWIc4+DcJ8c9jLFZzPOfz9wXGhqvqKoNLRhWJ/m7rdQddT2i5lH+1MJBWF0GsGzZspqYmNjqoGdyyVXXcNGahb4l87Pu1Imx9DtscnKS+b53OzvnPjHuYYzNYp7/fOa+oOWpqtrQnu8DPs3gmsS9bWmJ9nxfq74eOHRo90OADbOUHzKiXJI0JvMOjSR7Jtl76jVwPHAbcC0wdQfUcuCa9vpa4LR2F9WxwENtGesG4Pgk+7cL4McDN7RtjyQ5tt01ddpQW5KkMVjIWswS4NPtLtjdgY9V1WeT3ARcneQM4G7gDa3+dcBrgLXAY8BbAKpqU5LzgZtavfOqalN7/TbgCmAP4Pr2kCSNybxDo6ruAl4yovwB4LgR5QWcNUNbK4GVI8pvBo6c7xglSduWfxEuSepmaEiSuhkakqRuhoYkqZuhIUnqZmhIkroZGpKkboaGJKmboSFJ6mZoSJK6GRqSpG6GhiSpm6EhSepmaEiSuhkakqRuhoYkqZuhIUnqZmhIkroZGpKkboaGJKmboSFJ6mZoSJK6GRqSpG6GhiSpm6EhSepmaEiSuhkakqRuhoYkqZuhIUnqZmhIkrrtPu4BaGDpOZ8ZS7/rLnztWPqVtHPyTEOS1M3QkCR1MzQkSd0MDUlSNy+EL3LDF+BXvHgLpz9NF+S9AC/tnHb4M40kJyT5VpK1Sc4Z93gkaTHboUMjyW7AnwEnAkcApyQ5YryjkqTFa0dfnjoaWFtVdwEk+QRwEvC3Yx2VFmxcf5cCLo1JC7Gjh8bBwD1DP68HjpleKcmZwJntx81JvjXP/g4E7p/nvju9ty+S+ecPRxYvirnPYDHPHRb3/Ifn/vyeHXb00MiIsnpKQdVlwGUL7iy5uaqWLbSdndVinr9zX5xzh8U9//nMfYe+psHgzOLQoZ8PATaMaSyStOjt6KFxE3B4khckeRZwMnDtmMckSYvWDr08VVVbkpwN3ADsBqysqtu3Y5cLXuLayS3m+Tv3xWsxz3/Oc0/VUy4RSJI00o6+PCVJ2oEYGpKkboZGs5i/riTJuiRrktya5OZxj2d7S7IyyX1JbhsqOyDJ6iR3tuf9xznG7WWGub83yXfb8b81yWvGOcbtJcmhST6f5I4ktyf5zVa+WI79TPOf0/H3mgaPf13J3wOvYnCb703AKVW1KP7yPMk6YFlVLYo/cEryc8Bm4MqqOrKVvQ/YVFUXtl8a9q+qd4xznNvDDHN/L7C5qv5onGPb3pIcBBxUVV9PsjdwC/A64HQWx7Gfaf5vZA7H3zONgce/rqSq/gmY+roS7YKq6gvApmnFJwGr2utVDP5j2uXMMPdFoao2VtXX2+tHgDsYfOvEYjn2M81/TgyNgVFfVzLnN3MnVsDnktzSvpJlMVpSVRth8B8X8Nwxj+fpdnaSb7blq11yeWZYkqXAy4CvsgiP/bT5wxyOv6Ex0PV1JbuwV1TVUQy+TfistoShxeNS4CeAlwIbgYvGO5ztK8lewKeA36qqh8c9nqfbiPnP6fgbGgOL+utKqmpDe74P+DSD5brF5t625ju19nvfmMfztKmqe6vqh1X1I+DD7MLHP8kzGXxgXlVVf9mKF82xHzX/uR5/Q2Ng0X5dSZI920UxkuwJHA/ctvW9dknXAsvb6+XANWMcy9Nq6gOz+RV20eOfJMDlwB1V9f6hTYvi2M80/7kef++eatptZn/CE19XcsGYh/S0SPJCBmcXMPhamY/t6nNP8nFggsHXQt8LnAv8D+Bq4DDgbuANVbXLXTCeYe4TDJYmClgH/PrUGv+uJMnPAF8E1gA/asXvYrCuvxiO/UzzP4U5HH9DQ5LUzeUpSVI3Q0OS1M3QkCR1MzQkSd0MDUlSN0NDktTN0JAkdfv/PDO+9hysKnIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Create a feature hashtag_count\n",
    "df['hashtag_count'] = df['content'].apply(count_hashtags)\n",
    "\n",
    "# Plot the distribution\n",
    "df['hashtag_count'].hist()\n",
    "plt.title('Distribution of #Hashtag')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def count_mentions(string):\n",
    "    '''Returns number of mentions in a string.'''\n",
    "    \n",
    "    # Split the string into words\n",
    "    words = string.split()\n",
    "    \n",
    "    # Create a list of words that are mentions\n",
    "    mentions = [word for word in words if word.startswith(\"@\")]\n",
    "    \n",
    "    # Return number of mentions\n",
    "    return(len(mentions))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAAEICAYAAACqMQjAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAZZUlEQVR4nO3df9jldV3n8ecrUCMRAYlZBBTMqUBdSSbgWvtxJy0OamHX5Q+4SCZjL8qFso220GwxlV3dXXQvTNmlYBkIRdYfQYbhhEzmpggqOhAZszTBOATBADJY2th7//h+bjnenM/9c+a+mbmfj+s61znn8/18Pz/OnDmv8/1xvneqCkmSxvmepR6AJOmJy5CQJHUZEpKkLkNCktRlSEiSugwJSVKXIaGdKsn/TPI7O6itZyXZlmSP9nx9kn+3I9pu7X0iyZod1d4c+n1HkvuT/P1i971QSX48yVeXehzaeQwJzVuSTUn+MckjSR5K8pdJfjnJd95XVfXLVfX2Wbb109PVqaq7qmrvqvr2Dhj7W5P84ZT2T6yqtQtte47jOBQ4Gziyqv7VDHWPTPJ/2+O3JfnVkWUTSSrJR6es88JWvn4HjbeSPHfyeVX9RVX90I5oW09MhoQW6meq6mnAs4F3Ar8FXLyjO0my545u8wni2cADVXXfLOoeDXxh5PEXpyz/B+DfJHnGSNka4G8WPEotW4aEdoiqeriqrgFeC6xJ8nyAJJcmeUd7fECSj7etjq1J/iLJ9yS5HHgW8Mdtd9JvJjmsfWs9PcldwKdGykYD4weSfD7Jw0muTrJ/62siyebRMU5urSRZDbwZeG3r78tt+Xd2X7VxvSXJ3yW5L8llSZ7elk2OY02Su9quot/uvTZJnt7W/4fW3lta+z8NrAOe2cZx6Qwv8yoeC4kfAW6ZsvxbwB8BJ7d+9wBeA1wxZTw/nGRd+zf4apLXjCy7NMn7kvxJ20K8MckPtGWfbtW+3Mb72qmvc5Ij2uv4UJLbkvzsLNtOkve01/rhJF+ZfA9paRkS2qGq6vPAZuDHxyw+uy37fmAFwwd1VdXrgLsYtkr2rqr/OrLOTwJHAC/tdHka8IvAM4HtwAWzGOOfAv8Z+FDr74Vjqv1Cu/0U8Bxgb+D3ptT5MeCHgOOB/5TkiE6X7wWe3tr5yTbm11fVnwEnAlvaOH5h3MrtA/0h4EzgvUm+zvD6bU7yiSnVL2vtw/Ca3QZsGWnrqQzB9AHgQOAU4P1JnjfSxinA7wL7ARuB8wCq6ifa8he28X5oyjifBPwx8MnW9q8AVyQZ3R01tm3gBOAngB8E9mX4svHAuNdDi8uQ0M6wBdh/TPk/AwcBz66qf277s2e6eNhbq+rRqvrHzvLLq+rWqnoU+B3gNe0b9EKdCry7qu6sqm3Am4CTp2zF/G5V/WNVfRn4MvC4sGljeS3wpqp6pKo2AecDr5vtQKrq3wLHALdU1T4Mu/XOqap9q+rEKXX/Eti/fTCfxhAao14BbKqq/11V26vqi8BHgFeN1PloVX2+qrYzbIUcNcuhHscQpu+sqm9V1aeAjzMEw0xt/zPwNOCHgVTV7VV1zyz71U5kSGhnOBjYOqb8vzF8e/xkkjuTnDOLtu6ew/K/A54EHDCrUU7vma290bb3ZPgGP2n0bKRvMHxATnUA8OQxbR08m0EkOattRXwZeF57/HbgLW2XzoFjVrscOIthK+hjU5Y9Gzi2rftQa+9UYPSg+WzmNc4zgbur6l9GyqbOdWzbLVB+D3gfcG+Si5LsM8t+tRMZEtqhkvwow4fCZ6Yua9+kz66q5wA/A/x6kuMnF3eanGlL49CRx89i+EZ6P/Ao8H0j49qDYTfXbNvdwvCBOtr2duDeGdab6v42pqltfW02K1fV71XVvsCfAy9p7Xytqp7etiTGHfC+HPj3wLVV9Y0py+4G/rytO3nbu6reMMd5jbMFODQjZ7cxt7leUFVHA89j2O30H3fAmLRAhoR2iCT7JHkFcCXwh1W1YUydVyR5bpIAXwe+3W4wfPg+Zx5d/3yGU0O/D3gb8OF2iuzfAN+b5OVtX/lbgKeMrHcvcNiUD7RRHwT+Q5LDk+zNY8cwts9lcG0sVwHnJXlakmcDvw784fRrPs4LGbYmXsTjz2qa2uffMhz7GHcw/ePADyZ5XZIntduPTnM8Zarp/p1uZAjn32ztTjB8GbhypkbbGI5t/1aPAv/EY+8NLSFDQgv1x0keYfiG+tvAu4HXd+quBP4M2AZ8Fnh/Va1vy/4Lj+1C+Y059H85cCnDbozvBX4VhrOtGL5N/wHDN9lHGQ6aT/o/7f6BJOM+dC9pbX8a+FuGD61fmcO4Rv1K6/9Ohi2sD7T2ZyXJs4CtbavgRTx2hlNXVX2mqraMKX+E4SDxyQzf/P8eeBffHaDTeSuwtv07vWZ0QVV9C/hZhoPx9wPvB06rqr+eRbv7AL8PPMiwi+oB4L/PckzaieIfHZIk9bglIUnqMiQkSV2GhCSpy5CQJHXtdhdNO+CAA+qwww6b17qPPvooT33qU3fsgHYBznv5Wa5zd959X/jCF+6vqu9/3IKqmvbG8GOlG4DbGa4D88ZW/laGUwtvabeXjazzJoZf1n4VeOlI+epWtpHhsgKT5YcznGN9B/Ah4Mmt/Cnt+ca2/LCZxnv00UfXfN1www3zXndX5ryXn+U6d+fdB9xcYz5TZ7O7aTtwdlUdwXBtljOTHNmWvaeqjmq3a2G45j3DOdjPa6Hw/iR7tF+8vo/hHOojgVNG2nlXa2slw3nSp7fy04EHq+q5wHtaPUnSIpkxJKrqnhouAjb5Q5zbmf66MycBV1bVN2v45edGhouTHQNsrOGCad9i+BXmSe3Xty8BPtzWXwu8cqStyT8C82Hg+FZfkrQI5nRMIslhDNexvxF4MXBWktOAmxm2Nh5kCJDPjay2mcdC5e4p5ccCzwAeqscudzBa/+DJdapqe5KHW/37p4zrDOAMgBUrVrB+/fq5TOs7tm3bNu91d2XOe/lZrnN33nM365Bo16/5CPBrVfX1JBcyXI2y2v35DNf1H/dNvxi/1VLT1GeGZY8VVF0EXASwatWqmpiYmHYuPevXr2e+6+7KnPfys1zn7rznblanwLaLbn0EuKKqPgpQVfdW1bdruCzw7zPsToJhS2D0ypyHMFwjpld+P7DvyHX6J8u/q622/OmMvwS1JGknmDEk2jGAi4Hbq+rdI+UHjVT7OeDW9vgahj/O8pQkhzNc1O3zwE3AynZVzSczHNy+ph1Vv4HH/ujJGuDqkbbWtMevAj7V6kuSFsFsdje9mOGvaG1IMvk3dd/McHbSUQy7fzYBvwRQVbcluQr4K4Yzo86s4XLJJDkLuA7YA7ikqm5r7f0WcGWGv4X8JYZQot1fnmQjwxbEyQuYqyRpjmYMiar6DOOPDVw7zTrn8djfrh0tv3bcelV1J4/trhot/yfg1TONUZK0c3hZDklS1253WY6F2PC1h/mFc/5kSfre9M6XL0m/kjQdtyQkSV2GhCSpy5CQJHUZEpKkLkNCktRlSEiSugwJSVKXISFJ6jIkJEldhoQkqcuQkCR1GRKSpC5DQpLUZUhIkroMCUlSlyEhSeoyJCRJXYaEJKnLkJAkdRkSkqQuQ0KS1GVISJK6DAlJUpchIUnqMiQkSV2GhCSpy5CQJHUZEpKkLkNCktRlSEiSugwJSVKXISFJ6poxJJIcmuSGJLcnuS3JG1v5/knWJbmj3e/XypPkgiQbk3wlyYtG2lrT6t+RZM1I+dFJNrR1LkiS6fqQJC2O2WxJbAfOrqojgOOAM5McCZwDXF9VK4Hr23OAE4GV7XYGcCEMH/jAucCxwDHAuSMf+he2upPrrW7lvT4kSYtgxpCoqnuq6ovt8SPA7cDBwEnA2lZtLfDK9vgk4LIafA7YN8lBwEuBdVW1taoeBNYBq9uyfarqs1VVwGVT2hrXhyRpEew5l8pJDgN+BLgRWFFV98AQJEkObNUOBu4eWW1zK5uufPOYcqbpY+q4zmDYEmHFihWsX79+LtP6jhV7wdkv2D6vdRdqvmPeEbZt27ak/S+V5TpvWL5zd95zN+uQSLI38BHg16rq6+2wwdiqY8pqHuWzVlUXARcBrFq1qiYmJuay+ne894qrOX/DnHJzh9l06sSS9AtDQM33NduVLdd5w/Kdu/Oeu1md3ZTkSQwBcUVVfbQV39t2FdHu72vlm4FDR1Y/BNgyQ/khY8qn60OStAhmc3ZTgIuB26vq3SOLrgEmz1BaA1w9Un5aO8vpOODhtsvoOuCEJPu1A9YnANe1ZY8kOa71ddqUtsb1IUlaBLPZt/Ji4HXAhiS3tLI3A+8ErkpyOnAX8Oq27FrgZcBG4BvA6wGqamuStwM3tXpvq6qt7fEbgEuBvYBPtBvT9CFJWgQzhkRVfYbxxw0Ajh9Tv4AzO21dAlwypvxm4Pljyh8Y14ckaXH4i2tJUpchIUnqMiQkSV2GhCSpy5CQJHUZEpKkLkNCktRlSEiSugwJSVKXISFJ6jIkJEldhoQkqcuQkCR1GRKSpC5DQpLUZUhIkroMCUlSlyEhSeoyJCRJXYaEJKnLkJAkdRkSkqQuQ0KS1GVISJK6DAlJUpchIUnqMiQkSV2GhCSpy5CQJHUZEpKkLkNCktRlSEiSugwJSVKXISFJ6poxJJJckuS+JLeOlL01ydeS3NJuLxtZ9qYkG5N8NclLR8pXt7KNSc4ZKT88yY1J7kjyoSRPbuVPac83tuWH7ahJS5JmZzZbEpcCq8eUv6eqjmq3awGSHAmcDDyvrfP+JHsk2QN4H3AicCRwSqsL8K7W1krgQeD0Vn468GBVPRd4T6snSVpEM4ZEVX0a2DrL9k4Crqyqb1bV3wIbgWPabWNV3VlV3wKuBE5KEuAlwIfb+muBV460tbY9/jBwfKsvSVokey5g3bOSnAbcDJxdVQ8CBwOfG6mzuZUB3D2l/FjgGcBDVbV9TP2DJ9epqu1JHm717586kCRnAGcArFixgvXr189rQiv2grNfsH3mijvBfMe8I2zbtm1J+18qy3XesHzn7rznbr4hcSHwdqDa/fnALwLjvukX47dYapr6zLDsuwurLgIuAli1alVNTExMM/S+915xNedvWEhuzt+mUyeWpF8YAmq+r9mubLnOG5bv3J333M3r7Kaqureqvl1V/wL8PsPuJBi2BA4dqXoIsGWa8vuBfZPsOaX8u9pqy5/O7Hd7SZJ2gHmFRJKDRp7+HDB55tM1wMntzKTDgZXA54GbgJXtTKYnMxzcvqaqCrgBeFVbfw1w9Uhba9rjVwGfavUlSYtkxn0rST4ITAAHJNkMnAtMJDmKYffPJuCXAKrqtiRXAX8FbAfOrKpvt3bOAq4D9gAuqarbWhe/BVyZ5B3Al4CLW/nFwOVJNjJsQZy84NlKkuZkxpCoqlPGFF88pmyy/nnAeWPKrwWuHVN+J4/trhot/yfg1TONT5K08/iLa0lSlyEhSeoyJCRJXYaEJKnLkJAkdRkSkqQuQ0KS1GVISJK6DAlJUpchIUnqMiQkSV2GhCSpy5CQJHUZEpKkLkNCktRlSEiSugwJSVKXISFJ6jIkJEldhoQkqcuQkCR1GRKSpC5DQpLUZUhIkroMCUlSlyEhSeoyJCRJXYaEJKnLkJAkdRkSkqQuQ0KS1GVISJK6DAlJUteMIZHkkiT3Jbl1pGz/JOuS3NHu92vlSXJBko1JvpLkRSPrrGn170iyZqT86CQb2joXJMl0fUiSFs9stiQuBVZPKTsHuL6qVgLXt+cAJwIr2+0M4EIYPvCBc4FjgWOAc0c+9C9sdSfXWz1DH5KkRTJjSFTVp4GtU4pPAta2x2uBV46UX1aDzwH7JjkIeCmwrqq2VtWDwDpgdVu2T1V9tqoKuGxKW+P6kCQtkj3nud6KqroHoKruSXJgKz8YuHuk3uZWNl355jHl0/XxOEnOYNgaYcWKFaxfv35+k9oLzn7B9nmtu1DzHfOOsG3btiXtf6ks13nD8p278567+YZET8aU1TzK56SqLgIuAli1alVNTEzMtQkA3nvF1Zy/YUe/JLOz6dSJJekXhoCa72u2K1uu84blO3fnPXfzPbvp3rariHZ/XyvfDBw6Uu8QYMsM5YeMKZ+uD0nSIplvSFwDTJ6htAa4eqT8tHaW03HAw22X0XXACUn2awesTwCua8seSXJcO6vptCltjetDkrRIZty3kuSDwARwQJLNDGcpvRO4KsnpwF3Aq1v1a4GXARuBbwCvB6iqrUneDtzU6r2tqiYPhr+B4QyqvYBPtBvT9CFJWiQzhkRVndJZdPyYugWc2WnnEuCSMeU3A88fU/7AuD4kSYvHX1xLkroMCUlSlyEhSeoyJCRJXYaEJKnLkJAkdRkSkqQuQ0KS1GVISJK6DAlJUpchIUnqMiQkSV2GhCSpy5CQJHUZEpKkLkNCktRlSEiSugwJSVKXISFJ6jIkJEldhoQkqcuQkCR1GRKSpC5DQpLUZUhIkroMCUlSlyEhSeoyJCRJXYaEJKnLkJAkdRkSkqQuQ0KS1GVISJK6DAlJUteCQiLJpiQbktyS5OZWtn+SdUnuaPf7tfIkuSDJxiRfSfKikXbWtPp3JFkzUn50a39jWzcLGa8kaW52xJbET1XVUVW1qj0/B7i+qlYC17fnACcCK9vtDOBCGEIFOBc4FjgGOHcyWFqdM0bWW70DxitJmqWdsbvpJGBte7wWeOVI+WU1+Bywb5KDgJcC66pqa1U9CKwDVrdl+1TVZ6uqgMtG2pIkLYI9F7h+AZ9MUsD/qqqLgBVVdQ9AVd2T5MBW92Dg7pF1N7ey6co3jyl/nCRnMGxxsGLFCtavXz+vyazYC85+wfZ5rbtQ8x3zjrBt27Yl7X+pLNd5w/Kdu/Oeu4WGxIuraksLgnVJ/nqauuOOJ9Q8yh9fOITTRQCrVq2qiYmJaQfd894rrub8DQt9SeZn06kTS9IvDAE139dsV7Zc5w3Ld+7Oe+4WtLupqra0+/uAjzEcU7i37Sqi3d/Xqm8GDh1Z/RBgywzlh4wplyQtknmHRJKnJnna5GPgBOBW4Bpg8gylNcDV7fE1wGntLKfjgIfbbqnrgBOS7NcOWJ8AXNeWPZLkuHZW02kjbUmSFsFC9q2sAD7WzkrdE/hAVf1pkpuAq5KcDtwFvLrVvxZ4GbAR+AbweoCq2prk7cBNrd7bqmpre/wG4FJgL+AT7SZJWiTzDomquhN44ZjyB4Djx5QXcGanrUuAS8aU3ww8f75jlCQtjL+4liR1GRKSpC5DQpLUZUhIkroMCUlSlyEhSeoyJCRJXYaEJKnLkJAkdRkSkqQuQ0KS1GVISJK6DAlJUpchIUnqMiQkSV2GhCSpy5CQJHUZEpKkLkNCktRlSEiSugwJSVKXISFJ6jIkJEldhoQkqcuQkCR1GRKSpC5DQpLUZUhIkroMCUlSlyEhSeoyJCRJXYaEJKnLkJAkdRkSkqSuJ3xIJFmd5KtJNiY5Z6nHI0nLyRM6JJLsAbwPOBE4EjglyZFLOypJWj72XOoBzOAYYGNV3QmQ5ErgJOCvlnRUO8Fh5/zJkvS76Z0vX5J+Je0anughcTBw98jzzcCxUyslOQM4oz3dluSr8+zvAOD+ea67S8q7gGU472a5zhuW79ydd9+zxxU+0UMiY8rqcQVVFwEXLbiz5OaqWrXQdnY1znv5Wa5zd95z94Q+JsGw5XDoyPNDgC1LNBZJWnae6CFxE7AyyeFJngycDFyzxGOSpGXjCb27qaq2JzkLuA7YA7ikqm7biV0ueJfVLsp5Lz/Lde7Oe45S9bhd/JIkAU/83U2SpCVkSEiSugyJZrlc/iPJJUnuS3LrSNn+SdYluaPd77eUY9wZkhya5IYktye5LckbW/luPfck35vk80m+3Ob9u6388CQ3tnl/qJ0YsttJskeSLyX5eHu+2887yaYkG5LckuTmVjbv97khwbK7/MelwOopZecA11fVSuD69nx3sx04u6qOAI4Dzmz/xrv73L8JvKSqXggcBaxOchzwLuA9bd4PAqcv4Rh3pjcCt488Xy7z/qmqOmrktxHzfp8bEoPvXP6jqr4FTF7+Y7dTVZ8Gtk4pPglY2x6vBV65qINaBFV1T1V9sT1+hOGD42B287nXYFt7+qR2K+AlwIdb+W43b4AkhwAvB/6gPQ/LYN4d836fGxKDcZf/OHiJxrIUVlTVPTB8mAIHLvF4dqokhwE/AtzIMph72+VyC3AfsA74f8BDVbW9Vdld3+//A/hN4F/a82ewPOZdwCeTfKFdsggW8D5/Qv9OYhHN6vIf2vUl2Rv4CPBrVfX14cvl7q2qvg0clWRf4GPAEeOqLe6odq4krwDuq6ovJJmYLB5Tdbead/PiqtqS5EBgXZK/XkhjbkkMlvvlP+5NchBAu79vicezUyR5EkNAXFFVH23Fy2LuAFX1ELCe4ZjMvkkmvyTuju/3FwM/m2QTw+7jlzBsWezu86aqtrT7+xi+FBzDAt7nhsRguV/+4xpgTXu8Brh6CceyU7T90RcDt1fVu0cW7dZzT/L9bQuCJHsBP81wPOYG4FWt2m4376p6U1UdUlWHMfx//lRVncpuPu8kT03ytMnHwAnArSzgfe4vrpskL2P4pjF5+Y/zlnhIO0WSDwITDJcOvhc4F/gj4CrgWcBdwKuraurB7V1akh8D/gLYwGP7qN/McFxit517kn/NcKByD4YvhVdV1duSPIfhG/b+wJeAn6+qby7dSHeetrvpN6rqFbv7vNv8Ptae7gl8oKrOS/IM5vk+NyQkSV3ubpIkdRkSkqQuQ0KS1GVISJK6DAlJUpchIUnqMiQkSV3/H0iTVOGmzeu+AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Create a feature mention_count\n",
    "df['mention_count'] = df['content'].apply(count_mentions)\n",
    "\n",
    "# Plot the distribution\n",
    "df['mention_count'].hist()\n",
    "plt.title('Distribution of #Mentions')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Readability Test (reading-ease score)\n",
    "\n",
    "Readability tests or readability metrics are formulae for evaluating the readability of text, usually by counting syllables, words, and sentences. [wikipedia](https://en.wikipedia.org/wiki/Readability_test)\n",
    "\n",
    "The Flesch–Kincaid readability tests are readability tests designed to indicate how difficult a passage in English is to understand. In the Flesch reading-ease test, higher scores indicate material that is easier to read; lower numbers mark passages that are more difficult to read. [wikipedia](https://en.wikipedia.org/wiki/Flesch–Kincaid_readability_tests)\n",
    "\n",
    "![flesch](https://wikimedia.org/api/rest_v1/media/math/render/svg/bd4916e193d2f96fa3b74ee258aaa6fe242e110e)\n",
    "\n",
    "Score|Difficulty\n",
    "-----|:---------\n",
    "90-100|Very Easy\n",
    "80-89|Easy\n",
    "70-79|Fairly Easy\n",
    "60-69|Standard\n",
    "50-59|Fairly Difficult\n",
    "30-49|Difficult\n",
    "0-29|Very Confusing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Read Flesch reading-ease score from wikipedia\n",
    "# url = 'https://en.wikipedia.org/wiki/Flesch–Kincaid_readability_tests'\n",
    "# flesch = pd.read_html(url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install textstat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import textstat\n",
    "\n",
    "# df.content.apply(textstat.flesch_reading_ease)\n",
    "\n",
    "# ### While the maximum score is 121.22, there is no limit on how low the score can be. A negative score is valid."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "#flesch_score = 206.835 - 1.015 * (df.word_count / df.sentence_count) - 84.6 * (num_syllables / df.word_count)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text Preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Hello', 'world', '.', 'How', 'are', 'you', '?']\n"
     ]
    }
   ],
   "source": [
    "#import spacy\n",
    "\n",
    "# Load the en_core_web_sm (or en) model\n",
    "#nlp = spacy.load('en')\n",
    "nlp = spacy.load('en_core_web_sm')\n",
    "\n",
    "# Create a Doc object\n",
    "doc = nlp(\"Hello world. How are you?\")\n",
    "\n",
    "# Generate the tokens\n",
    "tokens = [token.text for token in doc]\n",
    "print(tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lemmatization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hello world . how be -PRON- ?\n"
     ]
    }
   ],
   "source": [
    "# Generate lemmas\n",
    "lemmas = [token.lemma_ for token in doc]\n",
    "\n",
    "# Convert lemmas into a string\n",
    "print(' '.join(lemmas))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Text Cleaning\n",
    "\n",
    "Several _Text Cleaning_ steps are performed if certain information does not additional value for text vectorization. It is better to do _Text Cleaning_  after the _tokenization_ step, since doing it before might cause undesirable effects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hello world\n"
     ]
    }
   ],
   "source": [
    "from spacy.lang.en.stop_words import STOP_WORDS\n",
    "\n",
    "# Remove stopwords and non-alphabetic tokens\n",
    "a_lemmas = [lemma for lemma in lemmas \n",
    "            if lemma.isalpha() and lemma not in STOP_WORDS]\n",
    "\n",
    "# Print string after text cleaning\n",
    "print(' '.join(a_lemmas))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that removing all non-alpha words or using a pre-defined stop words may not be ideal in many cases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function to preprocess text\n",
    "def preprocess(text):\n",
    "    # Create Doc object\n",
    "    doc = nlp(text, disable=['ner', 'parser'])\n",
    "    # Generate lemmas\n",
    "    lemmas = [token.lemma_ for token in doc]\n",
    "    # Remove stopwords and non-alphabetic characters\n",
    "    a_lemmas = [lemma for lemma in lemmas \n",
    "            if lemma.isalpha() and lemma not in STOP_WORDS]\n",
    "    \n",
    "    return ' '.join(a_lemmas)\n",
    "  \n",
    "# # Apply preprocess to df['content']\n",
    "# df['content'] = df['content'].apply(preprocess)\n",
    "# print(df['content'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extracting Word Features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Spacy has pre-trained neural models for tagging, parsing and entity recognition. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!python -m spacy download en_core_web_sm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "nlp = spacy.load(\"en_core_web_sm\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### POS Tagging"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('Hello', 'INTJ'), ('world', 'NOUN'), ('.', 'PUNCT'), ('How', 'ADV'), ('are', 'AUX'), ('you', 'PRON'), ('?', 'PUNCT')]\n"
     ]
    }
   ],
   "source": [
    "# Create a Doc object\n",
    "doc = nlp(\"Hello world. How are you?\")\n",
    "\n",
    "# Generate tokens and pos tags\n",
    "pos = [(token.text, token.pos_) for token in doc]\n",
    "print(pos)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3\n"
     ]
    }
   ],
   "source": [
    "# Returns number of proper nouns\n",
    "def num_proper_nouns(text, model=nlp):\n",
    "    # Create doc object\n",
    "    doc = model(text)\n",
    "    # Generate list of POS tags\n",
    "    pos = [token.pos_ for token in doc]\n",
    "    \n",
    "    # Return number of proper nouns\n",
    "    return pos.count(\"PROPN\")\n",
    "\n",
    "print(num_proper_nouns(\"Turgut, Kahn, and Jakob went to the university.\", nlp))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n"
     ]
    }
   ],
   "source": [
    "# Returns number of other nouns\n",
    "def num_nouns(text, model=nlp):\n",
    "    # Create doc object\n",
    "    doc = model(text)\n",
    "    # Generate list of POS tags\n",
    "    pos = [token.pos_ for token in doc]\n",
    "    \n",
    "    # Return number of other nouns\n",
    "    return pos.count(\"NOUN\")\n",
    "\n",
    "print(num_nouns(\"Turgut, Kahn, and Jakob went to the university.\", nlp))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compute number of proper nouns\n",
    "df['num_propn'] = df['content'].apply(num_proper_nouns)\n",
    "\n",
    "# Compute number of nouns\n",
    "df['num_noun'] = df['content'].apply(num_nouns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Named Entity Recognition (NER)\n",
    "\n",
    "Possible Uses:\n",
    "\n",
    "- Classifying/tagging articles by the people mentioned in them\n",
    "- Detecting people, location, year, etc. in a text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Kyle PERSON\n",
      "Khan PERSON\n",
      "Turgut GPE\n",
      "San Jose GPE\n",
      "Google ORG\n",
      "2015 DATE\n"
     ]
    }
   ],
   "source": [
    "# Create a Doc instance \n",
    "text = 'Kyle, Khan, and Turgut went to the university together. They are working in San Jose at Google since 2015.'\n",
    "doc = nlp(text)\n",
    "\n",
    "# Print all named entities and their labels\n",
    "for ent in doc.ents:\n",
    "    print(ent.text, ent.label_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model correctly predicted the labels of Kyle, Khan, Google, San Jose, and 2015; but mislabeled Turgut as GPE. The reason is that the predictions of the model depend strongly on the data it is trained on. We can train spaCy models on our custom data if required."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Identifying People"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Kyle', 'Khan']\n"
     ]
    }
   ],
   "source": [
    "def find_persons(text):\n",
    "  # Create Doc object\n",
    "  doc = nlp(text)\n",
    "  \n",
    "  # Identify the persons\n",
    "  persons = [ent.text for ent in doc.ents if ent.label_ == 'PERSON']\n",
    "  \n",
    "  # Return persons\n",
    "  return persons\n",
    "\n",
    "print(find_persons(text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vectorization\n",
    "\n",
    "Vectorization is the general process of turning a collection of text documents into numerical feature vectors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc-hr-collapsed": false
   },
   "source": [
    "### Bag of Words (Bag of n-Grams) (BoW)\n",
    "\n",
    "It is better to perform preprocessing before applying bag-of-words model, since it leads to smaller vocabulary list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus = pd.Series([\n",
    "    'Two roads diverged in a yellow wood',\n",
    "    'And sorry I could not travel both', \n",
    "    'And be one traveller, long I stood',\n",
    "    'And looked down one as far as I could',\n",
    "    'To where it bent in the undergrowth.'\n",
    "])\n",
    "# Poem by Frost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Count Vectorizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[CountVectorizer](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html) converts a collection of text documents to a matrix of token counts. It implements both tokenization and occurrence counting. \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CountVectorizer(analyzer='word', binary=False, decode_error='strict',\n",
       "                dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',\n",
       "                lowercase=True, max_df=1.0, max_features=None, min_df=1,\n",
       "                ngram_range=(1, 1), preprocessor=None, stop_words=None,\n",
       "                strip_accents=None, token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       "                tokenizer=None, vocabulary=None)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create CountVectorizer object\n",
    "vectorizer = CountVectorizer()\n",
    "vectorizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generating a bag of words representation for a given corpus of documents: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(5, 27)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<5x27 sparse matrix of type '<class 'numpy.int64'>'\n",
       "\twith 32 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Generate matrix of word vectors\n",
    "bow_matrix = vectorizer.fit_transform(corpus)\n",
    "\n",
    "# Print the shape of bow_matrix\n",
    "print(bow_matrix.shape)\n",
    "\n",
    "bow_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0,\n",
       "        1, 0, 0, 1, 1],\n",
       "       [1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 1, 0,\n",
       "        0, 0, 0, 0, 0],\n",
       "       [1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 1, 0, 0, 0, 1,\n",
       "        0, 0, 0, 0, 0],\n",
       "       [1, 2, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0,\n",
       "        0, 0, 0, 0, 0],\n",
       "       [0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0,\n",
       "        0, 1, 1, 0, 0]])"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bow_matrix.toarray()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generating a bag of words representation for a given corpus of documents: After some preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    road diverge yellow wood\n",
       "1                sorry travel\n",
       "2        traveller long stand\n",
       "3                    look far\n",
       "4            bend undergrowth\n",
       "dtype: object"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from spacy.lang.en.stop_words import STOP_WORDS\n",
    "\n",
    "# Function to preprocess text\n",
    "def preprocess(text):\n",
    "    # Create a Doc object\n",
    "    doc = nlp(text, disable=['ner', 'parser'])\n",
    "    # Generate lemmas\n",
    "    lemmas = [token.lemma_ for token in doc]\n",
    "    # Remove stopwords and non-alphabetic characters\n",
    "    a_lemmas = [lemma for lemma in lemmas \n",
    "            if lemma.isalpha() and lemma not in STOP_WORDS]\n",
    "    \n",
    "    return ' '.join(a_lemmas)\n",
    "  \n",
    "# Apply lemmatize to corpus\n",
    "corpus_prep = corpus.apply(preprocess)\n",
    "\n",
    "# Print corpus_prep\n",
    "corpus_prep"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(5, 13)\n"
     ]
    }
   ],
   "source": [
    "# Create CountVectorizer object\n",
    "vectorizer = CountVectorizer()\n",
    "\n",
    "# Generate matrix of word vectors\n",
    "bow_prep_matrix = vectorizer.fit_transform(corpus_prep)\n",
    "\n",
    "# Print the shape of bow_lem_matrix\n",
    "print(bow_prep_matrix.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The number of features reduced significantly from 27 to 13 after preprocessing. While this may have some disadvantages, preprocessing text before vectorizing reduces the dimension and may likely lead to a better performance when applying ML."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Preprocessing Using `CountVectorizer`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some argumens of scikit-learn's `CountVectorizer` allows us to perform basic preprocessing. These are as follows:\n",
    "\n",
    "- lowercase : False , True\n",
    "- strip_accents : 'unciode' , 'ascii' , None \n",
    "- stop_words : 'english' , list , None \n",
    "- token_pattern : regex\n",
    "- tokenizer : function\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Mapping Indices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>bend</th>\n",
       "      <th>diverge</th>\n",
       "      <th>far</th>\n",
       "      <th>long</th>\n",
       "      <th>look</th>\n",
       "      <th>road</th>\n",
       "      <th>sorry</th>\n",
       "      <th>stand</th>\n",
       "      <th>travel</th>\n",
       "      <th>traveller</th>\n",
       "      <th>undergrowth</th>\n",
       "      <th>wood</th>\n",
       "      <th>yellow</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   bend  diverge  far  long  look  road  sorry  stand  travel  traveller  \\\n",
       "0     0        1    0     0     0     1      0      0       0          0   \n",
       "1     0        0    0     0     0     0      1      0       1          0   \n",
       "2     0        0    0     1     0     0      0      1       0          1   \n",
       "3     0        0    1     0     1     0      0      0       0          0   \n",
       "4     1        0    0     0     0     0      0      0       0          0   \n",
       "\n",
       "   undergrowth  wood  yellow  \n",
       "0            0     1       1  \n",
       "1            0     0       0  \n",
       "2            0     0       0  \n",
       "3            0     0       0  \n",
       "4            1     0       0  "
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create CountVectorizer object\n",
    "vectorizer = CountVectorizer()\n",
    "\n",
    "# Generate matrix of word vectors\n",
    "bow_prep_matrix = vectorizer.fit_transform(corpus_prep)\n",
    "\n",
    "# Convert bow_matrix into a DataFrame\n",
    "bow_df = pd.DataFrame(bow_prep_matrix.toarray())\n",
    "\n",
    "# Map the column names to vocabulary \n",
    "bow_df.columns = vectorizer.get_feature_names()\n",
    "\n",
    "# Print bow_df\n",
    "bow_df\n",
    "\n",
    "# # Features as a DataFrame\n",
    "# feature_names = count_vectorizer.get_feature_names()\n",
    "# df_features = pd.DataFrame(count_train.toarray(), columns = feature_names)\n",
    "# df_features.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The column names refer to the tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>corpus</th>\n",
       "      <th>bend</th>\n",
       "      <th>diverge</th>\n",
       "      <th>far</th>\n",
       "      <th>long</th>\n",
       "      <th>look</th>\n",
       "      <th>road</th>\n",
       "      <th>sorry</th>\n",
       "      <th>stand</th>\n",
       "      <th>travel</th>\n",
       "      <th>traveller</th>\n",
       "      <th>undergrowth</th>\n",
       "      <th>wood</th>\n",
       "      <th>yellow</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Two roads diverged in a yellow wood</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>And sorry I could not travel both</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>And be one traveller, long I stood</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>And looked down one as far as I could</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>To where it bent in the undergrowth.</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                  corpus  bend  diverge  far  long  look  \\\n",
       "0    Two roads diverged in a yellow wood     0        1    0     0     0   \n",
       "1      And sorry I could not travel both     0        0    0     0     0   \n",
       "2     And be one traveller, long I stood     0        0    0     1     0   \n",
       "3  And looked down one as far as I could     0        0    1     0     1   \n",
       "4   To where it bent in the undergrowth.     1        0    0     0     0   \n",
       "\n",
       "   road  sorry  stand  travel  traveller  undergrowth  wood  yellow  \n",
       "0     1      0      0       0          0            0     1       1  \n",
       "1     0      1      0       1          0            0     0       0  \n",
       "2     0      0      1       0          1            0     0       0  \n",
       "3     0      0      0       0          0            0     0       0  \n",
       "4     0      0      0       0          0            1     0       0  "
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Merge corpus with bow df\n",
    "pd.DataFrame(corpus.rename(\"corpus\")).join(bow_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### BoW Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split into training and test sets\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=.25)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### BoW Model (Representation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a CountVectorizer object\n",
    "vectorizer = CountVectorizer(lowercase=True, stop_words=\"english\")\n",
    "\n",
    "# Fit and transform X_train\n",
    "X_train_bow = vectorizer.fit_transform(X_train)\n",
    "\n",
    "# Transform X_test\n",
    "X_test_bow = vectorizer.transform(X_test)\n",
    "\n",
    "# Print shape of X_train_bow and X_test_bow\n",
    "print(X_train_bow.shape)\n",
    "print(X_test_bow.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Classification Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a MultinomialNB object\n",
    "clf = MultinomialNB()\n",
    "\n",
    "# Fit the classifier\n",
    "clf.fit(X_train_bow, y_train)\n",
    "\n",
    "# Measure the accuracy\n",
    "accuracy = clf.score(X_test_bow, y_test)\n",
    "print(\"The accuracy of the classifier on the test set is %.3f\" % accuracy)\n",
    "\n",
    "# Predict the sentiment of a negative review\n",
    "review = \"The movie was terrible. The music was underwhelming and the acting mediocre.\"\n",
    "prediction = clf.predict(vectorizer.transform([review]))[0]\n",
    "print(\"The sentiment predicted by the classifier is %i\" % (prediction))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### n-gram Models\n",
    "\n",
    "Applications:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ng1, ng2 and ng3 have 27, 55 and 78 features respectively.\n"
     ]
    }
   ],
   "source": [
    "# Generate n-grams upto n=1\n",
    "vectorizer_ng1 = CountVectorizer(ngram_range=(1,1))\n",
    "ng1 = vectorizer_ng1.fit_transform(corpus)\n",
    "\n",
    "# Generate n-grams upto n=2\n",
    "vectorizer_ng2 = CountVectorizer(ngram_range=(1,2))\n",
    "ng2 = vectorizer_ng2.fit_transform(corpus)\n",
    "\n",
    "# Generate n-grams upto n=3\n",
    "vectorizer_ng3 = CountVectorizer(ngram_range=(1, 3))\n",
    "ng3 = vectorizer_ng3.fit_transform(corpus)\n",
    "\n",
    "# Print the number of features for each model\n",
    "print(\"ng1, ng2 and ng3 have %i, %i and %i features respectively.\" % (ng1.shape[1], ng2.shape[1], ng3.shape[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define an instance of MultinomialNB \n",
    "clf_ng = MultinomialNB()\n",
    "\n",
    "# Fit the classifier \n",
    "clf_ng.fit(X_train_ng, y_train)\n",
    "\n",
    "# Measure the accuracy \n",
    "accuracy = clf_ng.score(X_test_ng, y_test)\n",
    "print(\"The accuracy of the classifier on the test set is %.3f\" % accuracy)\n",
    "\n",
    "# Predict the sentiment of a negative review\n",
    "review = \"The movie was not good. The plot had several holes and the acting lacked panache.\"\n",
    "prediction = clf_ng.predict(ng_vectorizer.transform([review]))[0]\n",
    "print(\"The sentiment predicted by the classifier is %i\" % (prediction))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Comparing Performance\n",
    "\n",
    "I will compare the performance using three criteria: accuracy of the model on the test set, time taken to execute and the number of features created when generating the n-gram representation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start_time = time.time()\n",
    "# Splitting the data into training and test sets\n",
    "train_X, test_X, train_y, test_y = train_test_split(df['review'], df['sentiment'], test_size=0.5, \n",
    "                                                    random_state=42, stratify=df['sentiment'])\n",
    "\n",
    "# Generating ngrams\n",
    "vectorizer = CountVectorizer()\n",
    "train_X = vectorizer.fit_transform(train_X)\n",
    "test_X = vectorizer.transform(test_X)\n",
    "\n",
    "# Fit classifier\n",
    "clf = MultinomialNB()\n",
    "clf.fit(train_X, train_y)\n",
    "\n",
    "# Print accuracy, time and number of dimensions\n",
    "print(\"The program took %.3f seconds to complete. The accuracy on the test set is %.2f. The ngram representation had %i features.\" % (time.time() - start_time, clf.score(test_X, test_y), train_X.shape[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start_time = time.time()\n",
    "# Splitting the data into training and test sets\n",
    "train_X, test_X, train_y, test_y = train_test_split(df['review'], df['sentiment'], test_size=0.5, \n",
    "                                                    random_state=42, stratify=df['sentiment'])\n",
    "\n",
    "# Generating ngrams\n",
    "vectorizer = CountVectorizer(ngram_range=(1,3))\n",
    "train_X = vectorizer.fit_transform(train_X)\n",
    "test_X = vectorizer.transform(test_X)\n",
    "\n",
    "# Fit classifier\n",
    "clf = MultinomialNB()\n",
    "clf.fit(train_X, train_y)\n",
    "\n",
    "# Print accuracy, time and number of dimensions\n",
    "print(\"The program took %.3f seconds to complete. The accuracy on the test set is %.2f. The ngram representation had %i features.\" % (time.time() - start_time, clf.score(test_X, test_y), train_X.shape[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Despite taking higher computation time and generating more features, the classifier only performs marginally better in the latter case, producing an accuracy of 77% in comparison to the 75% for the unigram model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### tf-idf (term frequency - inverse document frequency)\n",
    "\n",
    "Possible applications:\n",
    "- Automatically detect stopwords\n",
    "- Search\n",
    "- Recommender systems\n",
    "- Better performance in predictive modeling for some cases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "ted = pd.read_csv(\"data/fe/ted.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    We're going to talk — my — a new lecture, just...\n",
       "1    This is a representation of your brain, and yo...\n",
       "2    It's a great honor today to share with you The...\n",
       "Name: transcript, dtype: object"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ted = ted.transcript\n",
    "ted.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(500, 29158)\n"
     ]
    }
   ],
   "source": [
    "# Create TfidfVectorizer object\n",
    "vectorizer = TfidfVectorizer()\n",
    "\n",
    "# Generate matrix of word vectors\n",
    "tfidf_matrix = vectorizer.fit_transform(ted)\n",
    "\n",
    "# Print the shape of tfidf_matrix\n",
    "print(tfidf_matrix.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['tyrannous',\n",
       " 'extraordinarily',\n",
       " 'plumbers',\n",
       " 'camara',\n",
       " 'orientation',\n",
       " 'dot',\n",
       " 'kyunki',\n",
       " 'preconceptions',\n",
       " 'submerge',\n",
       " 'hamburger']"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "indices = np.random.randint(29158, size=10)\n",
    "np.asarray(vectorizer.get_feature_names())[indices].tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Similarity Scores (Cosine Similarity)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0      Two roads diverged in a yellow wood\n",
       "1        And sorry I could not travel both\n",
       "2       And be one traveller, long I stood\n",
       "3    And looked down one as far as I could\n",
       "4     To where it bent in the undergrowth.\n",
       "dtype: object"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1.         0.         0.         0.         0.10617563]\n",
      " [0.         1.         0.0879537  0.16458645 0.        ]\n",
      " [0.         0.0879537  1.         0.16458645 0.        ]\n",
      " [0.         0.16458645 0.16458645 1.         0.        ]\n",
      " [0.10617563 0.         0.         0.         1.        ]]\n"
     ]
    }
   ],
   "source": [
    "# Initialize an instance of tf-idf Vectorizer\n",
    "tfidf_vectorizer = TfidfVectorizer()\n",
    "\n",
    "# Generate the tf-idf vectors for the corpus\n",
    "tfidf_matrix = tfidf_vectorizer.fit_transform(corpus)\n",
    "\n",
    "# Compute and print the cosine similarity matrix\n",
    "cosine_sim = cosine_similarity(tfidf_matrix, tfidf_matrix)\n",
    "print(cosine_sim)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "    [[1.         0.36413198 0.18314713 0.18435251 0.16336438]\n",
    "     [0.36413198 1.         0.15054075 0.21704584 0.11203887]\n",
    "     [0.18314713 0.15054075 1.         0.21318602 0.07763512]\n",
    "     [0.18435251 0.21704584 0.21318602 1.         0.12960089]\n",
    "     [0.16336438 0.11203887 0.07763512 0.12960089 1.        ]]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the similarity matrix, we see that the second and the fourth sentences (+ the third and the fourth) are the most similar. Also the first and the fifth sentences have, on average, the lowest pairwise cosine scores. This is intuitive as they contain entities that are not present in the other sentences."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Recommender Systems Using tf-idf and Cosine Similarity Matrix\n",
    "\n",
    "Steps:\n",
    "- Text Preprocessing\n",
    "- Generating Tf-idf Vectors\n",
    "- Generating Cosine Similarity Matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we're working with large data, it is a good practice to use  `linear_kernel` instead of `cosine_similarity` to improve performance, since both give same result when the vectors are in the `tf-idf` representation. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "movie = pd.read_csv(\"data/fe/movie_overviews.csv\", usecols=[1,2])\n",
    "movie = movie.dropna().reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>overview</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Toy Story</td>\n",
       "      <td>Led by Woody, Andy's toys live happily in his ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jumanji</td>\n",
       "      <td>When siblings Judy and Peter discover an encha...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Grumpier Old Men</td>\n",
       "      <td>A family wedding reignites the ancient feud be...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Waiting to Exhale</td>\n",
       "      <td>Cheated on, mistreated and stepped on, the wom...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Father of the Bride Part II</td>\n",
       "      <td>Just when George Banks has recovered from his ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         title  \\\n",
       "0                    Toy Story   \n",
       "1                      Jumanji   \n",
       "2             Grumpier Old Men   \n",
       "3            Waiting to Exhale   \n",
       "4  Father of the Bride Part II   \n",
       "\n",
       "                                            overview  \n",
       "0  Led by Woody, Andy's toys live happily in his ...  \n",
       "1  When siblings Judy and Peter discover an encha...  \n",
       "2  A family wedding reignites the ancient feud be...  \n",
       "3  Cheated on, mistreated and stepped on, the wom...  \n",
       "4  Just when George Banks has recovered from his ...  "
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movie.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate mapping between titles and index\n",
    "indices = pd.Series(movie.index, index=movie['title']).drop_duplicates()\n",
    "\n",
    "def get_recommendations(title, cosine_sim, indices):\n",
    "    # Get index of movie that matches title\n",
    "    idx = indices[title]\n",
    "    # Sort the movies based on the similarity scores\n",
    "    sim_scores = list(enumerate(cosine_sim[idx]))\n",
    "    sim_scores = sorted(sim_scores, key=lambda x: x[1], reverse=True)\n",
    "    # Get the scores for 10 most similar movies\n",
    "    sim_scores = sim_scores[1:11]\n",
    "    # Get the movie indices\n",
    "    movie_indices = [i[0] for i in sim_scores]\n",
    "    # Return the top 10 most similar movies\n",
    "    return movie['title'].iloc[movie_indices]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "132                              Batman Forever\n",
      "6902                            The Dark Knight\n",
      "1113                             Batman Returns\n",
      "7567                 Batman: Under the Red Hood\n",
      "523                                      Batman\n",
      "7901                           Batman: Year One\n",
      "8164    Batman: The Dark Knight Returns, Part 1\n",
      "2576               Batman: Mask of the Phantasm\n",
      "8225    Batman: The Dark Knight Returns, Part 2\n",
      "6145                              Batman Begins\n",
      "Name: title, dtype: object\n"
     ]
    }
   ],
   "source": [
    "# Initialize the TfidfVectorizer \n",
    "tfidf = TfidfVectorizer(stop_words = \"english\")\n",
    "\n",
    "# Construct the TF-IDF matrix\n",
    "tfidf_matrix = tfidf.fit_transform(movie.overview)\n",
    "\n",
    "# Generate the cosine similarity matrix\n",
    "cosine_sim = linear_kernel(tfidf_matrix, tfidf_matrix)\n",
    " \n",
    "# Generate recommendations \n",
    "print(get_recommendations('The Dark Knight Rises', cosine_sim, indices))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Word Embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Generating Word Vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I I 1.0\n",
      "I like 0.18428169\n",
      "I bananas 0.14788893\n",
      "I and -0.03217984\n",
      "I apples 0.052373044\n",
      "like I 0.18428169\n",
      "like like 1.0\n",
      "like bananas 0.10241573\n",
      "like and 0.1815342\n",
      "like apples 0.12026104\n",
      "bananas I 0.14788893\n",
      "bananas like 0.10241573\n",
      "bananas bananas 1.0\n",
      "bananas and -0.11511948\n",
      "bananas apples 0.6352397\n",
      "and I -0.03217984\n",
      "and like 0.1815342\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/stb/anaconda3/lib/python3.7/runpy.py:193: ModelsWarning: [W007] The model you're using has no word vectors loaded, so the result of the Token.similarity method will be based on the tagger, parser and NER, which may not give useful similarity judgements. This may happen if you're using one of the small models, e.g. `en_core_web_sm`, which don't ship with word vectors and only use context-sensitive tensors. You can always add your own word vectors, or use one of the larger models instead if available.\n",
      "  \"__main__\", mod_spec)\n",
      "/Users/stb/anaconda3/lib/python3.7/runpy.py:193: ModelsWarning: [W007] The model you're using has no word vectors loaded, so the result of the Token.similarity method will be based on the tagger, parser and NER, which may not give useful similarity judgements. This may happen if you're using one of the small models, e.g. `en_core_web_sm`, which don't ship with word vectors and only use context-sensitive tensors. You can always add your own word vectors, or use one of the larger models instead if available.\n",
      "  \"__main__\", mod_spec)\n",
      "/Users/stb/anaconda3/lib/python3.7/runpy.py:193: ModelsWarning: [W007] The model you're using has no word vectors loaded, so the result of the Token.similarity method will be based on the tagger, parser and NER, which may not give useful similarity judgements. This may happen if you're using one of the small models, e.g. `en_core_web_sm`, which don't ship with word vectors and only use context-sensitive tensors. You can always add your own word vectors, or use one of the larger models instead if available.\n",
      "  \"__main__\", mod_spec)\n",
      "/Users/stb/anaconda3/lib/python3.7/runpy.py:193: ModelsWarning: [W007] The model you're using has no word vectors loaded, so the result of the Token.similarity method will be based on the tagger, parser and NER, which may not give useful similarity judgements. This may happen if you're using one of the small models, e.g. `en_core_web_sm`, which don't ship with word vectors and only use context-sensitive tensors. You can always add your own word vectors, or use one of the larger models instead if available.\n",
      "  \"__main__\", mod_spec)\n",
      "/Users/stb/anaconda3/lib/python3.7/runpy.py:193: ModelsWarning: [W007] The model you're using has no word vectors loaded, so the result of the Token.similarity method will be based on the tagger, parser and NER, which may not give useful similarity judgements. This may happen if you're using one of the small models, e.g. `en_core_web_sm`, which don't ship with word vectors and only use context-sensitive tensors. You can always add your own word vectors, or use one of the larger models instead if available.\n",
      "  \"__main__\", mod_spec)\n",
      "/Users/stb/anaconda3/lib/python3.7/runpy.py:193: ModelsWarning: [W007] The model you're using has no word vectors loaded, so the result of the Token.similarity method will be based on the tagger, parser and NER, which may not give useful similarity judgements. This may happen if you're using one of the small models, e.g. `en_core_web_sm`, which don't ship with word vectors and only use context-sensitive tensors. You can always add your own word vectors, or use one of the larger models instead if available.\n",
      "  \"__main__\", mod_spec)\n",
      "/Users/stb/anaconda3/lib/python3.7/runpy.py:193: ModelsWarning: [W007] The model you're using has no word vectors loaded, so the result of the Token.similarity method will be based on the tagger, parser and NER, which may not give useful similarity judgements. This may happen if you're using one of the small models, e.g. `en_core_web_sm`, which don't ship with word vectors and only use context-sensitive tensors. You can always add your own word vectors, or use one of the larger models instead if available.\n",
      "  \"__main__\", mod_spec)\n",
      "/Users/stb/anaconda3/lib/python3.7/runpy.py:193: ModelsWarning: [W007] The model you're using has no word vectors loaded, so the result of the Token.similarity method will be based on the tagger, parser and NER, which may not give useful similarity judgements. This may happen if you're using one of the small models, e.g. `en_core_web_sm`, which don't ship with word vectors and only use context-sensitive tensors. You can always add your own word vectors, or use one of the larger models instead if available.\n",
      "  \"__main__\", mod_spec)\n",
      "/Users/stb/anaconda3/lib/python3.7/runpy.py:193: ModelsWarning: [W007] The model you're using has no word vectors loaded, so the result of the Token.similarity method will be based on the tagger, parser and NER, which may not give useful similarity judgements. This may happen if you're using one of the small models, e.g. `en_core_web_sm`, which don't ship with word vectors and only use context-sensitive tensors. You can always add your own word vectors, or use one of the larger models instead if available.\n",
      "  \"__main__\", mod_spec)\n",
      "/Users/stb/anaconda3/lib/python3.7/runpy.py:193: ModelsWarning: [W007] The model you're using has no word vectors loaded, so the result of the Token.similarity method will be based on the tagger, parser and NER, which may not give useful similarity judgements. This may happen if you're using one of the small models, e.g. `en_core_web_sm`, which don't ship with word vectors and only use context-sensitive tensors. You can always add your own word vectors, or use one of the larger models instead if available.\n",
      "  \"__main__\", mod_spec)\n",
      "/Users/stb/anaconda3/lib/python3.7/runpy.py:193: ModelsWarning: [W007] The model you're using has no word vectors loaded, so the result of the Token.similarity method will be based on the tagger, parser and NER, which may not give useful similarity judgements. This may happen if you're using one of the small models, e.g. `en_core_web_sm`, which don't ship with word vectors and only use context-sensitive tensors. You can always add your own word vectors, or use one of the larger models instead if available.\n",
      "  \"__main__\", mod_spec)\n",
      "/Users/stb/anaconda3/lib/python3.7/runpy.py:193: ModelsWarning: [W007] The model you're using has no word vectors loaded, so the result of the Token.similarity method will be based on the tagger, parser and NER, which may not give useful similarity judgements. This may happen if you're using one of the small models, e.g. `en_core_web_sm`, which don't ship with word vectors and only use context-sensitive tensors. You can always add your own word vectors, or use one of the larger models instead if available.\n",
      "  \"__main__\", mod_spec)\n",
      "/Users/stb/anaconda3/lib/python3.7/runpy.py:193: ModelsWarning: [W007] The model you're using has no word vectors loaded, so the result of the Token.similarity method will be based on the tagger, parser and NER, which may not give useful similarity judgements. This may happen if you're using one of the small models, e.g. `en_core_web_sm`, which don't ship with word vectors and only use context-sensitive tensors. You can always add your own word vectors, or use one of the larger models instead if available.\n",
      "  \"__main__\", mod_spec)\n",
      "/Users/stb/anaconda3/lib/python3.7/runpy.py:193: ModelsWarning: [W007] The model you're using has no word vectors loaded, so the result of the Token.similarity method will be based on the tagger, parser and NER, which may not give useful similarity judgements. This may happen if you're using one of the small models, e.g. `en_core_web_sm`, which don't ship with word vectors and only use context-sensitive tensors. You can always add your own word vectors, or use one of the larger models instead if available.\n",
      "  \"__main__\", mod_spec)\n",
      "/Users/stb/anaconda3/lib/python3.7/runpy.py:193: ModelsWarning: [W007] The model you're using has no word vectors loaded, so the result of the Token.similarity method will be based on the tagger, parser and NER, which may not give useful similarity judgements. This may happen if you're using one of the small models, e.g. `en_core_web_sm`, which don't ship with word vectors and only use context-sensitive tensors. You can always add your own word vectors, or use one of the larger models instead if available.\n",
      "  \"__main__\", mod_spec)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "and bananas -0.11511948\n",
      "and and 1.0\n",
      "and apples 0.0039500236\n",
      "apples I 0.052373044\n",
      "apples like 0.12026104\n",
      "apples bananas 0.6352397\n",
      "apples and 0.0039500236\n",
      "apples apples 1.0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/stb/anaconda3/lib/python3.7/runpy.py:193: ModelsWarning: [W007] The model you're using has no word vectors loaded, so the result of the Token.similarity method will be based on the tagger, parser and NER, which may not give useful similarity judgements. This may happen if you're using one of the small models, e.g. `en_core_web_sm`, which don't ship with word vectors and only use context-sensitive tensors. You can always add your own word vectors, or use one of the larger models instead if available.\n",
      "  \"__main__\", mod_spec)\n",
      "/Users/stb/anaconda3/lib/python3.7/runpy.py:193: ModelsWarning: [W007] The model you're using has no word vectors loaded, so the result of the Token.similarity method will be based on the tagger, parser and NER, which may not give useful similarity judgements. This may happen if you're using one of the small models, e.g. `en_core_web_sm`, which don't ship with word vectors and only use context-sensitive tensors. You can always add your own word vectors, or use one of the larger models instead if available.\n",
      "  \"__main__\", mod_spec)\n",
      "/Users/stb/anaconda3/lib/python3.7/runpy.py:193: ModelsWarning: [W007] The model you're using has no word vectors loaded, so the result of the Token.similarity method will be based on the tagger, parser and NER, which may not give useful similarity judgements. This may happen if you're using one of the small models, e.g. `en_core_web_sm`, which don't ship with word vectors and only use context-sensitive tensors. You can always add your own word vectors, or use one of the larger models instead if available.\n",
      "  \"__main__\", mod_spec)\n",
      "/Users/stb/anaconda3/lib/python3.7/runpy.py:193: ModelsWarning: [W007] The model you're using has no word vectors loaded, so the result of the Token.similarity method will be based on the tagger, parser and NER, which may not give useful similarity judgements. This may happen if you're using one of the small models, e.g. `en_core_web_sm`, which don't ship with word vectors and only use context-sensitive tensors. You can always add your own word vectors, or use one of the larger models instead if available.\n",
      "  \"__main__\", mod_spec)\n",
      "/Users/stb/anaconda3/lib/python3.7/runpy.py:193: ModelsWarning: [W007] The model you're using has no word vectors loaded, so the result of the Token.similarity method will be based on the tagger, parser and NER, which may not give useful similarity judgements. This may happen if you're using one of the small models, e.g. `en_core_web_sm`, which don't ship with word vectors and only use context-sensitive tensors. You can always add your own word vectors, or use one of the larger models instead if available.\n",
      "  \"__main__\", mod_spec)\n"
     ]
    }
   ],
   "source": [
    "# Load the en_core_web_lg\n",
    "nlp = spacy.load('en_core_web_sm')\n",
    "\n",
    "# Create the doc object\n",
    "doc = nlp('I like bananas and apples')\n",
    "\n",
    "# Compute pairwise similarity scores\n",
    "for token1 in doc:\n",
    "  for token2 in doc:\n",
    "    print(token1.text, token2.text, token1.similarity(token2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Computing Similarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create Doc objects\n",
    "mother_doc = nlp(mother)\n",
    "hopes_doc = nlp(hopes)\n",
    "hey_doc = nlp(hey)\n",
    "\n",
    "# Print similarity between mother and hopes\n",
    "print(mother_doc.similarity(hopes_doc))\n",
    "\n",
    "# Print similarity between mother and hey\n",
    "print(mother_doc.similarity(hey_doc))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}