03-NLP/helloworld-nlp.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text processing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can do many interesting things with text!\n",
    "It even exists an entire area in Machine Learning called Natural Language Processing (NLP) which cover any kind of machine manipulation of natural human languages.   \n",
    "I want to show here a couple of pre-processing examples that are normally the base for more sophisticated models and algorithms."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start with a simple text in a Python string:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sampleText = \" The Elephant's 4 legs: this is THE Pub. Hi, you, my super_friend! You can't believe what happened to our * common friend *, the butcher. How are you?\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The basic atomic part of each text are the tokens. \n",
    "A **token** is the NLP name for a sequence of characters that we want to treat as a group.  \n",
    "For example we can consider group of characters separated by blank spaces, therefore forming words.  \n",
    "Tokens can be extracted by splitting the text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "textTokens = sampleText.split() # by default split by spaces"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['The',\n",
       " \"Elephant's\",\n",
       " '4',\n",
       " 'legs:',\n",
       " 'this',\n",
       " 'is',\n",
       " 'THE',\n",
       " 'Pub.',\n",
       " 'Hi,',\n",
       " 'you,',\n",
       " 'my',\n",
       " 'super_friend!',\n",
       " 'You',\n",
       " \"can't\",\n",
       " 'believe',\n",
       " 'what',\n",
       " 'happened',\n",
       " 'to',\n",
       " 'our',\n",
       " '*',\n",
       " 'common',\n",
       " 'friend',\n",
       " '*,',\n",
       " 'the',\n",
       " 'butcher.',\n",
       " 'How',\n",
       " 'are',\n",
       " 'you?']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "textTokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The sample text has 28 tokens\n"
     ]
    }
   ],
   "source": [
    "print (\"The sample text has {} tokens\".format (len(textTokens)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, tokens are words but also symbols like *.\n",
    "This is because we have split the string simply by using blank spaces.\n",
    "But you can pass other separators to the function split() such as commas."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokens frequency"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I have the number of tokens for this sample text. Let's say now that I want to count the frequency for each token.  \n",
    "This can be done quickly by using the Python package Counter from collections."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from collections import Counter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({'*': 1,\n",
       "         '*,': 1,\n",
       "         '4': 1,\n",
       "         \"Elephant's\": 1,\n",
       "         'Hi,': 1,\n",
       "         'How': 1,\n",
       "         'Pub.': 1,\n",
       "         'THE': 1,\n",
       "         'The': 1,\n",
       "         'You': 1,\n",
       "         'are': 1,\n",
       "         'believe': 1,\n",
       "         'butcher.': 1,\n",
       "         \"can't\": 1,\n",
       "         'common': 1,\n",
       "         'friend': 1,\n",
       "         'happened': 1,\n",
       "         'is': 1,\n",
       "         'legs:': 1,\n",
       "         'my': 1,\n",
       "         'our': 1,\n",
       "         'super_friend!': 1,\n",
       "         'the': 1,\n",
       "         'this': 1,\n",
       "         'to': 1,\n",
       "         'what': 1,\n",
       "         'you,': 1,\n",
       "         'you?': 1})"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "totalWords = Counter(textTokens); \n",
    "totalWords"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are a number of problems:\n",
    "* some word (like The/THE or You/you) is in two different tokens because of capital vs. small letter\n",
    "* some token contains punctuation marks which makes the same word counted twice\n",
    "* same token consists of symbols (like *)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Remove capital letters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is very easy to do in Python by using the lower() function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "loweredText = sampleText.lower()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\" the elephant's 4 legs: this is the pub. hi, you, my super_friend! you can't believe what happened to our * common friend *, the butcher. how are you?\""
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loweredText"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({'*': 1,\n",
       "         '*,': 1,\n",
       "         '4': 1,\n",
       "         'are': 1,\n",
       "         'believe': 1,\n",
       "         'butcher.': 1,\n",
       "         \"can't\": 1,\n",
       "         'common': 1,\n",
       "         \"elephant's\": 1,\n",
       "         'friend': 1,\n",
       "         'happened': 1,\n",
       "         'hi,': 1,\n",
       "         'how': 1,\n",
       "         'is': 1,\n",
       "         'legs:': 1,\n",
       "         'my': 1,\n",
       "         'our': 1,\n",
       "         'pub.': 1,\n",
       "         'super_friend!': 1,\n",
       "         'the': 3,\n",
       "         'this': 1,\n",
       "         'to': 1,\n",
       "         'what': 1,\n",
       "         'you': 1,\n",
       "         'you,': 1,\n",
       "         'you?': 1})"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "textTokens = loweredText.split()\n",
    "totalWords = Counter(textTokens); \n",
    "totalWords"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the token \"the\" is counted correctly 3 times !  \n",
    "But other words like you are still wrongly counted because of the punctuation such as comma or question mark."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Remove punctuation and trailing spaces"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Removing the extra spaces is very easy, by using the string function strip():"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"the elephant's 4 legs: this is the pub. hi, you, my super_friend! you can't believe what happened to our * common friend *, the butcher. how are you?\""
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "strippedText = loweredText.strip()\n",
    "strippedText"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To remove punctuaction we can use regular expressions.  \n",
    "Regular expression are very powerful for match patterns in a sequence of characters.\n",
    "\n",
    "The way in RE to match specific characters is to list them inside square brackets. For example [abc] will match only a single a,b or c letter an nothing else. A shorthand for matching sequential characters is to use the dash, for example [a-z] will match any single lowercase letter and [0-9] any single digit character.  \n",
    "To exclude characters from the matching we use the ^ (hat) symbol, for example [^abc] will match any single character except the letters a, b or c.  \n",
    "Finally there is also a special symbol for the whitespaces as they are so ubiquitous in text. Whitespaces are blank spaces, tabs, newlines, carriage returns. The symbol \\s will match any of them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function re.sub() takes as input a starting string, a string to substitute and a pattern to match and returns the string with the matches replaced with the given substring using the given pattern. \n",
    "for example re.sub(r'[s]', 'z', \"whatsapp\") will return \"whatzapp\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So one way to remove the punctuation is to replace any character that is NOT a letter, a number or a whitespace with an empty substring:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'the elephants 4 legs this is the pub hi you my superfriend you cant believe what happened to our  common friend  the butcher how are you'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "processedText = re.sub(r'[^a-z0-9\\s]', '', strippedText) # keep only numbers and letters\n",
    "processedText"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another useful symbol is \\w which match ANY alphanumeric character or \\W which matches any NON alphanumeric character.  \n",
    "So, an alternative way could be:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'the elephants 4 legs this is the pub hi you my super_friend you cant believe what happened to our  common friend  the butcher how are you'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "processedText = re.sub(r'[^\\s\\w]', '', strippedText) # remove punctuation\n",
    "processedText"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'the elephants 4 legs this is the pub hi you my super_friend you cant believe what happened to our  common friend  the butcher how are you'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "processedText4 = re.sub(r'^\\s+', r'', processedText) # remove spaces at the beginning\n",
    "processedText4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({'4': 1,\n",
       "         'are': 1,\n",
       "         'believe': 1,\n",
       "         'butcher': 1,\n",
       "         'cant': 1,\n",
       "         'common': 1,\n",
       "         'elephants': 1,\n",
       "         'friend': 1,\n",
       "         'happened': 1,\n",
       "         'hi': 1,\n",
       "         'how': 1,\n",
       "         'is': 1,\n",
       "         'legs': 1,\n",
       "         'my': 1,\n",
       "         'our': 1,\n",
       "         'pub': 1,\n",
       "         'super_friend': 1,\n",
       "         'the': 3,\n",
       "         'this': 1,\n",
       "         'to': 1,\n",
       "         'what': 1,\n",
       "         'you': 3})"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "textTokens = processedText.split()\n",
    "totalWords = Counter(textTokens); \n",
    "totalWords"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the token *you* is also counted correctly 3 times. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A collection can be sorted easily: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_items([('believe', 1), ('pub', 1), ('this', 1), ('my', 1), ('legs', 1), ('to', 1), ('what', 1), ('are', 1), ('is', 1), ('common', 1), ('you', 3), ('hi', 1), ('4', 1), ('happened', 1), ('our', 1), ('super_friend', 1), ('the', 3), ('how', 1), ('friend', 1), ('butcher', 1), ('cant', 1), ('elephants', 1)])\n"
     ]
    }
   ],
   "source": [
    "print (totalWords.items())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('you', 3),\n",
       " ('the', 3),\n",
       " ('believe', 1),\n",
       " ('pub', 1),\n",
       " ('this', 1),\n",
       " ('my', 1),\n",
       " ('legs', 1),\n",
       " ('to', 1),\n",
       " ('what', 1),\n",
       " ('are', 1),\n",
       " ('is', 1),\n",
       " ('common', 1),\n",
       " ('hi', 1),\n",
       " ('4', 1),\n",
       " ('happened', 1),\n",
       " ('our', 1),\n",
       " ('super_friend', 1),\n",
       " ('how', 1),\n",
       " ('friend', 1),\n",
       " ('butcher', 1),\n",
       " ('cant', 1),\n",
       " ('elephants', 1)]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sorted(totalWords.items(), key=lambda x:x[1],reverse=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An alternative way (without lambda functions) to sort a collection:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('you', 3),\n",
       " ('the', 3),\n",
       " ('believe', 1),\n",
       " ('pub', 1),\n",
       " ('this', 1),\n",
       " ('my', 1),\n",
       " ('legs', 1),\n",
       " ('to', 1),\n",
       " ('what', 1),\n",
       " ('are', 1),\n",
       " ('is', 1),\n",
       " ('common', 1),\n",
       " ('hi', 1),\n",
       " ('4', 1),\n",
       " ('happened', 1),\n",
       " ('our', 1),\n",
       " ('super_friend', 1),\n",
       " ('how', 1),\n",
       " ('friend', 1),\n",
       " ('butcher', 1),\n",
       " ('cant', 1),\n",
       " ('elephants', 1)]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from operator import itemgetter\n",
    "sorted(totalWords.items(), key=itemgetter(1), reverse=True)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's put these results into functions we can re-use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def tokenise(text):\n",
    "    return text.split() # split by space; return a list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def removePunctuation(text):\n",
    "    processedText = re.sub(r'([^\\s\\w_]|_)+', '', text.strip()) # remove punctuation\n",
    "\n",
    "    return processedText     "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def preprocessText(text, lowercase=True):\n",
    "    if lowercase:\n",
    "        processedText = removePunctuation(text.lower())\n",
    "    else:\n",
    "        processedText = removePunctuation(text)\n",
    "\n",
    "    return tokenise(processedText)       "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def getMostCommonWords(tokens, n=10):\n",
    "    wordsCount = Counter(tokens) # count the occurrences\n",
    "    \n",
    "    return wordsCount.most_common()[:n]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's process a text file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Python makes working with files pretty simple.   \n",
    "Let's use one of the books from the Project Gutenberg, which I have already dowloaded from ...."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "filePath = \"../datasets/theprince.txt\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First of all, we need to open the file using the function *open()* that returns a file object.  \n",
    "We can then read the file content using the file object method *read()*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "f = open(filePath)\n",
    "try:\n",
    "    theText = f.read()  # this is a giant String\n",
    "finally:\n",
    "    f.close()  # we should always close the file once finished"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*** Analysing book The Prince:\n",
      "The book is 300814 chars long\n",
      "The text has 52536 tokens\n"
     ]
    }
   ],
   "source": [
    "print (\"*** Analysing book The Prince:\")  \n",
    "print (\"The book is {} chars long\".format (len(theText)))\n",
    "\n",
    "tokens = preprocessText(theText)\n",
    "\n",
    "print (\"The text has {} tokens\".format (len(tokens)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is an easy way to read the entire text but:  \n",
    "1. It's quite memory inefficient\n",
    "2. It's slower than processing data as it is read, because it defers any processing done on read data until after all data has been read into memory, rather than processing as data is read.  \n",
    "\n",
    "A better way is to read the file line by line.  \n",
    "We can do this with a simple loop which will go through each line.  \n",
    "Note that the block keyword *with* will automatically close the file at the end of the block."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Title: The Project Gutenberg EBook of The Prince, by Nicolo Machiavelli\n",
      "\n"
     ]
    }
   ],
   "source": [
    "textTokens = [] # tokens will be added here\n",
    "lines = 0\n",
    "\n",
    "with open(filePath) as f:\n",
    "    for line in f:\n",
    "               # I know that the very first line is the book title\n",
    "        if lines == 0:\n",
    "            print (\"Title: {}\".format(line))\n",
    "            \n",
    "               # every line gets processed\n",
    "        lineTokens = preprocessText(line)               \n",
    "                # append the tokens to my list\n",
    "        textTokens.extend(lineTokens)\n",
    "        \n",
    "        lines += 1  # finally move to the next line"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The text has 5064 lines\n",
      "The text has 52536 tokens\n"
     ]
    }
   ],
   "source": [
    "print (\"The text has {} lines\".format (lines))\n",
    "print (\"The text has {} tokens\".format (len(textTokens)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['the',\n",
       " 'project',\n",
       " 'gutenberg',\n",
       " 'ebook',\n",
       " 'of',\n",
       " 'the',\n",
       " 'prince',\n",
       " 'by',\n",
       " 'nicolo',\n",
       " 'machiavelli']"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "textTokens[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Unique tokens\n",
    "\n",
    "Another useful data structure in Python is the set which is an unordered collection of distinct objects\n",
    "Arranging the tokens in a set means that they will put only once,  and could be a smaller data collection useful to see how many distinct tokens are in a text or to see if a specific token is in the text or not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The text has 5630 unique tokens\n",
      " -> lexical diversity: each token in average is repeated 9.331438721136767 times\n"
     ]
    }
   ],
   "source": [
    "uniqueTokens = set(textTokens)\n",
    "print (\"The text has {} unique tokens\".format (len(uniqueTokens)))\n",
    "print (\" -> lexical diversity: each token in average is repeated {} times\".format(len(textTokens) / len(uniqueTokens)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['accumulate', 'accuse', 'accused', 'accustomed', 'accustoms']"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sorted(uniqueTokens)[200:205]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'accuse' in uniqueTokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'phone' in uniqueTokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Remove stopwords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('the', 3109), ('to', 2107), ('and', 1935), ('of', 1802), ('in', 993)]"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "getMostCommonWords(textTokens, 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see the most common words are not really meaningful but we can remove them"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Stopwords (https://en.wikipedia.org/wiki/Stop_words) are common (English) words that do not contribute much to the content or meaning of a document (e.g., \"the\", \"a\", \"is\", \"to\", etc.). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "f = open(\"stopwords.txt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "stopWordsText = f.read().splitlines()  # splitlines is used to remove newlines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "f.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "stopWords = set(stopWordsText)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "betterTokens = [token for token in textTokens if token not in stopWords]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['project',\n",
       " 'gutenberg',\n",
       " 'ebook',\n",
       " 'prince',\n",
       " 'nicolo',\n",
       " 'machiavelli',\n",
       " 'ebook',\n",
       " 'use',\n",
       " 'anyone',\n",
       " 'anywhere']"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "betterTokens[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('prince', 222),\n",
       " ('men', 161),\n",
       " ('castruccio', 136),\n",
       " ('people', 115),\n",
       " ('many', 101),\n",
       " ('others', 96),\n",
       " ('time', 93),\n",
       " ('great', 89),\n",
       " ('duke', 88),\n",
       " ('project', 87)]"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "getMostCommonWords(betterTokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate a words cloud from a text\n",
    "\n",
    "Small example: we generate a words cloud from a text.  \n",
    "To make it more interesting we take as text a novel directly from the web (Project Gutenberg)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from urllib.request import urlopen"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def getWordCloud(filename):\n",
    "    textTokens = [] # tokens will be added here\n",
    "    lines = 0\n",
    "    path = \"http://www.gutenberg.org/files/\"\n",
    "\n",
    "    url = path + filename + \"/\" + filename + \"-0.txt\"\n",
    "    f = urlopen(url)\n",
    "    \n",
    "    for line in f:            \n",
    "               # every line gets processed\n",
    "        lineTokens = preprocessText(line.decode('utf-8'))               \n",
    "                # append the tokens to my list\n",
    "        textTokens.extend(lineTokens)\n",
    "        \n",
    "        lines += 1  # finally move to the next line\n",
    "            \n",
    "    fs = open(\"stopwords.txt\")\n",
    "    stopWordsText = fs.read().splitlines()\n",
    "    stopWords = set(stopWordsText)\n",
    "    fs.close()\n",
    "    betterTokens = [token for token in textTokens if token not in stopWords]\n",
    "    \n",
    "    wordsCount = Counter(betterTokens) # count the occurences\n",
    "    \n",
    "          # put each token and its occurrence in a file\n",
    "    with open(\"wordcloud_\"+filename+\".txt\", 'a') as fw:\n",
    "        for line in wordsCount.most_common():\n",
    "    #        freq = (line[1] + 10 // 2) // 10\n",
    "     #       if freq > 2:\n",
    "            fw.write(str(line[1]) + ' ' + line[0] + '\\n')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "getWordCloud(\"1342\") # Jane Austen Pride and Prejudice"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Jane Austen](wordcloud1342.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "getWordCloud(\"804\") # Laurence Sterne A sentimental journey through France and Italy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Laurence Sterne](wordcloud804.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}