nlp_nltk_intro.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text mining intro"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "You are currently looking at **version 1.0** of this notebook.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text Mining areas in short:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Working text\n",
    "Working with text needs a tool box that is quite different from working with numerical data. Generally characters, words, sentences need to be cleaned and (pre)processed before doing actual analyses. Luckily their are very valuable frameworks and toolboxes around, like NLTK:\n",
    " - NLTK documentation link: http://www.nltk.org/api/nltk.html\n",
    " - NLTK cheat sheet: https://blogs.princeton.edu/etc/files/2014/03/Text-Analysis-with-NLTK-Cheatsheet.pdf\n",
    " - NLTK book: http://www.nltk.org/book/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Sentiment Analysis\n",
    "Sentiment analysis is generally a starting point in analyzing a text and is then coupled with other techniques (e.g., topic analysis). Sentiment analysis is usually done using a corpus of positive and negative words.\n",
    "It identifies entities and emotions in a sentence and use these to determine if the entity is being viewed positively or negatively"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Easy example sentiment analyses\n",
    "<li>I had an <b style=\"color:green\">excellent</b> souffle at the restaurant Cavity Maker</li>\n",
    "<li>Excellent is a positive word for both the souffle as well as for the restaurant</li>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Not so easy examples\n",
    "Often, looking at words alone is not enough to figure out the sentiment:  \n",
    "<li><i>The Girl on the Train is an <span style=\"color:green\">excellent</span> book for a ‘stuck at home’ snow day</i></li> This one is easy since it includes an explicit positive opinion using a positive word\n",
    "<li><i>The Girl on the Train is an <span style=\"color:green\">excellent</span> book for using as a liner for your cat’s litter box</i></li> Not so simple! The positive word \"excellent\" is used with a negative connotation. \n",
    "<li><i>The Girl on the Train is <span style=\"color:green\">better</span> than Gone Girl</i></li> The positive word is used as a comparator. Whether the writer likes The Girl on the Train or not depends on what he or she thinks of Gone Girl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sources of sentiment coded words\n",
    "<ol>\n",
    "<li>Hu and Liu's sentiment analysis lexicon: words coded as either positive or negative</li>\n",
    "<ul>\n",
    "<li>http://ptrckprry.com/course/ssd/data/positive-words.txt\n",
    "<li>http://ptrckprry.com/course/ssd/data/negative-words.txt\n",
    "</ul>\n",
    "<li>NRC Emotion Lexicon: words coded into emotional categories (many languages)</li>\n",
    "<ul>\n",
    "<li>http://saifmohammad.com/WebPages/NRC-Emotion-Lexicon.htm</li>\n",
    "</ul>\n",
    "<li>SentiWordNet: Lists of words weighted by positive or negative sentiment. Includes guidance on how to use the words</li>\n",
    "<ul>\n",
    "<li>http://sentiwordnet.isti.cnr.it/</li>\n",
    "</ul>\n",
    "<li>Vadar Sentiment tool: 7800 words with positive or negative polarity</li>\n",
    "<ul>\n",
    "<li>Included with python nltk</li>\n",
    "</ul>\n",
    "</ol>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Topic modeling\n",
    "The goal of topic modeling is to identify the major concepts underlying a piece of text.  \n",
    "Topic modeling uses \"Unsupervised Learning\". No apriori knowledge is necessary.  \n",
    "Though it is helpful in cleaning up results!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Setup notebook\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import the generic libraries used in this notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import string\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import requests\n",
    "import json\n",
    "import re\n",
    "from collections import OrderedDict, Counter\n",
    "import pprint\n",
    "\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "matplotlib.rc('xtick', labelsize=14) \n",
    "matplotlib.rc('ytick', labelsize=14)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Manage warnings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set defaults and constants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set pandas defaults\n",
    "pd.set_option('max_rows', 10)                                # Show max 10 rows: head(5) ... tail(5)\n",
    "pd.set_option('display.float_format', lambda x: '%.3f' % x)  # Set precision of DataFrames/Series"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check current working directory and file structure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pwd\n",
    "# !ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 1. Working text\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text1 = \"Ethics are built right into the ideals and objectives of the United Nations \"\n",
    "n_chars = len(text1) # The length of text1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text2 = text1.split(' ') # Return a list of the words in text2, separating by ' '.\n",
    "n_words = len(text2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(text2)\n",
    "n_chars, n_words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list('abcdefghijklm'), list('1234567890')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### List comprehension allows us to find specific words:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "[w for w in text2 if len(w) > 3] # Words that are greater than 3 letters long in text2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "[w for w in text2 if w.istitle()] # Capitalized words in text2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "[w for w in text2 if w.endswith('s')] # Words in text2 that end in 's'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "We can find unique words using `set()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text3 = 'To be or not to be'\n",
    "text4 = text3.split(' ')\n",
    "len(text4), len(set(text4))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "set(text4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "set([w.lower() for w in text4])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Processing free-text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text5 = '\"Ethics are built right into the ideals and objectives of the United Nations\" \\\n",
    "#UNSG @ NY Society for Ethical Culture bit.ly/2guVelr'\n",
    "text6 = text5.split(' ')\n",
    "text6;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Finding hastags:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "[w for w in text6 if w.startswith('#')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Finding callouts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "[w for w in text6 if w.startswith('@')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text7 = '@UN @UN_Women \"Ethics are built right into the ideals and objectives of the United Nations\" \\\n",
    "#UNSG @ NY Society for Ethical Culture bit.ly/2guVelr'\n",
    "text8 = text7.split(' ')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Regular expressions help us with more complex parsing\n",
    "For example `'@[A-Za-z0-9_]+'` will return all words that: \n",
    "* start with `'@'` and are followed by at least one: \n",
    "* capital letter (`'A-Z'`)\n",
    "* lowercase letter (`'a-z'`) \n",
    "* number (`'0-9'`)\n",
    "* or underscore (`'_'`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "[w for w in text8 if re.search('@[A-Za-z0-9_]+', w)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read a labeled data set; [(text, label)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"data/sentiment_labelled_sentences/full_set.txt\") as f:\n",
    "    content = f.readlines()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### First look at data structure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "content[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Split sentences and labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Remove leading and trailing white spaces before splitting labels\n",
    "content = [x.strip() for x in content]\n",
    "\n",
    "## Separate the sentences from the labels; '\\t1\\n' => 1 is the label\n",
    "sentences = [x.split(\"\\t\")[0] for x in content]\n",
    "labels = [x.split(\"\\t\")[1] for x in content]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocessing the text data\n",
    "\n",
    "To transform this prediction problem into one amenable to linear classification, we will first need to preprocess the text data. We will do four transformations:\n",
    "\n",
    "1. Remove punctuation and numbers.\n",
    "2. Transform all words to lower-case.\n",
    "3. Remove _stop words_.\n",
    "4. Convert the sentences into vectors, using a bag-of-words representation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def full_remove(x, removal_list):\n",
    "    # Replace chars from removal list with spaces\n",
    "    for remove_item in removal_list:\n",
    "        x = x.replace(remove_item, ' ')\n",
    "    # Return without superfluous spaces\n",
    "    return ' '.join(x.split(None))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Remove digits\n",
    "digit_less = [full_remove(x, list('1234567890')) for x in sentences]\n",
    "\n",
    "## Remove punctuation\n",
    "punc_less = [full_remove(x, list(string.punctuation)) for x in digit_less]\n",
    "\n",
    "## Make everything lower-case\n",
    "sents_lower = [x.lower() for x in punc_less]\n",
    "type(sents_lower), sents_lower[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Stop words\n",
    " - Stop words are words that are filtered out because they are believed to contain no useful information for the task at hand. You can create your own arbitrary stop word list or use a generic one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.corpus import stopwords"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus = ' '.join([sent_words for sent_words in sents_lower])\n",
    "dictionary = set(corpus.split())\n",
    "\n",
    "# Use predefined stop words set\n",
    "stop_words = set(stopwords.words('english'))\n",
    "\n",
    "# Define our own unwanted words set\n",
    "unwanted_words = set(['the', 'a', 'an', 'i', 'he', 'she', 'they', 'to', 'of', 'it', 'from'])\n",
    "\n",
    "# Get short words\n",
    "MIN_LENGTH = 3\n",
    "short_words = set([word for word in dictionary if len(word) < MIN_LENGTH])\n",
    "\n",
    "# Define set of words to clear from text/sentences\n",
    "clear_set = stop_words | unwanted_words | short_words\n",
    "\n",
    "# Clear text from unwanted words\n",
    "sents_split = [x.split() for x in sents_lower]\n",
    "sents_processed = [' '.join(list(filter(lambda word: word not in clear_set, sent_words))) for sent_words in sents_split]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What do the sentences look like so far?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sents_processed[0:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Basic NLP Tasks with NLTK\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NLTK sources\n",
    " - ntlk documentation link: http://www.nltk.org/api/nltk.html\n",
    " - Commands cheat sheet: https://blogs.princeton.edu/etc/files/2014/03/Text-Analysis-with-NLTK-Cheatsheet.pdf\n",
    " - nltk book: http://www.nltk.org/book/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "from nltk.book import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Counting vocabulary of words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "'no words in text:', len(text7), text7"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "'no words in sentence:', len(sent7), sent7"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "'no unique words:', len(set(text7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "'first 10 unique words:', list(set(text7))[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Frequency of words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dist = FreqDist(text7)\n",
    "dist2 = Counter(text7)\n",
    "len(dist), dist == dist2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab1 = dist.keys()\n",
    "# vocab1[:10] # can't slice in python 3\n",
    "\n",
    "# Python 3 dict.keys() returns an iterable view instead of a list\n",
    "list(vocab1)[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "'frequency of key in text:', dist['four']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "freqwords = [w for w in vocab1 if len(w) > 5 and dist[w] > 100]\n",
    "'words with more than 5 characters and frequency higher than 100:', freqwords"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Normalization and stemming\n",
    "Stemming is the process for reducing inflected/derived words to their stem/base/root. The stem need not be identical to the morphological root of the word."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input1 = 'List listed lists listing listings'\n",
    "words1 = input1.lower().split(' ')\n",
    "words1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "porter = nltk.PorterStemmer()\n",
    "[porter.stem(t) for t in words1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lemmatization\n",
    "Lemmatisation is the process of grouping together the different inflected forms.\n",
    "For example, in English, the verb ‘to walk’ may appear as ‘walk’, ‘walked’, ‘walks’, ‘walking’. The base form, ‘walk’, that one might look up in a dictionary, is called the lemma for the word."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputl = 'Walk walked walks walking walker Walkers'\n",
    "wordsl = inputl.lower().split(' ')\n",
    "\n",
    "WNlemma = nltk.WordNetLemmatizer()\n",
    "'walks => walk ', [WNlemma.lemmatize(t) for t in wordsl], [WNlemma.lemmatize(t) for t in wordsl] == wordsl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "udhr = nltk.corpus.udhr.words('English-Latin1')\n",
    "'Universal declaration of human rights corpus:', udhr[:20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "[porter.stem(t) for t in udhr[:20]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "WNlemma = nltk.WordNetLemmatizer()\n",
    "lemmatized = [WNlemma.lemmatize(t) for t in udhr[:20]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Lexical diversity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(set(lemmatized)) / len(lemmatized)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text11 = \"Children shouldn't drink a sugary drink before bed.\"\n",
    "text11.split(' ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_tokens = nltk.word_tokenize(text11)\n",
    "text_nltk = nltk.Text(text_tokens)\n",
    "text_tokens, text_nltk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nltk.word_tokenize(text11), '-'*50, 'no of words:', len(nltk.word_tokenize(text11))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text12 = \"This is the first sentence. A gallon of milk in the U.S. costs $2.99. Is this the third sentence? Yes, it is!\"\n",
    "sentences = nltk.sent_tokenize(text12)\n",
    "len(sentences)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(nltk.word_tokenize(moby_raw))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text1[:10], nltk.Text(text1[:10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "words = ' '.join(text1).lower().split(' ')\n",
    "dist = FreqDist(words)#.most_common() in ['whale', 'Whale']\n",
    "dist['whale'] * 100 / len(nltk.word_tokenize(' '.join(text1)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "FreqDist(text1).most_common(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# word length > 5, frequency > 150\n",
    "dist = FreqDist(text1).most_common()\n",
    "sorted([k for k, v in dist if len(k) > 5 and v > 150])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Longest word + length\n",
    "from collections import OrderedDict\n",
    "dist = FreqDist(text1).most_common()\n",
    "\n",
    "# dictionary sorted by length of the key string\n",
    "longest_word = OrderedDict(sorted(dist, key=lambda t: len(t[0]), reverse=True)).popitem(last=False)\n",
    "longest_word[0], len(longest_word[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.Series({len(w):w for w in text1})[-1:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# unique words with frequency of more than 2000 and their frequency\n",
    "dist = FreqDist(text1).most_common(50)\n",
    "result = sorted([(f, w) for w, f in dist if f > 2000 and w.isalpha()])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Average # tokens per sentance\n",
    "sentences = nltk.sent_tokenize(' '.join(text1))\n",
    "np.mean([len(nltk.word_tokenize(s)) for s in sentences])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Advanced NLP Tasks with NLTK\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### POS tagging"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nltk.help.upenn_tagset('NN'), nltk.help.upenn_tagset('DT'), nltk.help.upenn_tagset('VB'), nltk.help.upenn_tagset('MD')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text13 = nltk.word_tokenize(text11)\n",
    "nltk.pos_tag(text13)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text14 = nltk.word_tokenize(\"Visiting aunts can be a nuisance\")\n",
    "nltk.pos_tag(text14)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Parsing sentence structure\n",
    "text15 = nltk.word_tokenize(\"Alice loves Bob\")\n",
    "grammar = nltk.CFG.fromstring(\"\"\"\n",
    "S -> NP VP\n",
    "VP -> V NP\n",
    "NP -> 'Alice' | 'Bob'\n",
    "V -> 'loves'\n",
    "\"\"\")\n",
    "\n",
    "parser = nltk.ChartParser(grammar)\n",
    "trees = parser.parse_all(text15)\n",
    "for tree in trees:\n",
    "    print(tree)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.corpus import treebank\n",
    "text17 = treebank.parsed_sents('wsj_0001.mrg')[0]\n",
    "print(text17)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### POS tagging and parsing ambiguity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text18 = nltk.word_tokenize(\"The old man the boat\")\n",
    "nltk.pos_tag(text18)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text19 = nltk.word_tokenize(\"Colorless green ideas sleep furiously\")\n",
    "nltk.pos_tag(text19)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Named Entities: People, places, organizations\n",
    " - Named entities are often the subject of sentiments so identifying them can be very useful"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Named entity detection - Part-of-speech tagging\n",
    " - tokenize sentences with sentence detector (english)\n",
    " - tokenize words in each sentence\n",
    " - chunk them; ne_chunk identifies likely chunked candidates (ne = named entity)\n",
    " - build chunks using nltk's guess on what members of chunk represent (people, place, organization)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "en={}\n",
    "try:\n",
    "    sent_detector = nltk.data.load('tokenizers/punkt/english.pickle')\n",
    "    sentences = sent_detector.tokenize(community_data.raw().strip())\n",
    "    for sentence in sentences:\n",
    "            tokenized = nltk.word_tokenize(sentence)\n",
    "            tagged = nltk.pos_tag(tokenized)\n",
    "            chunked = nltk.ne_chunk(tagged)\n",
    "            for tree in chunked:\n",
    "                if hasattr(tree, 'label'):\n",
    "                    ne = ' '.join(c[0] for c in tree.leaves())\n",
    "                    en[ne] = [tree.label(), ' '.join(c[1] for c in tree.leaves())]\n",
    "except Exception as e:\n",
    "    print(str(e))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pp = pprint.PrettyPrinter(indent=4)\n",
    "pp.pprint(en)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# most frequent parts of speech in this text? What is their frequency?\n",
    "df = pd.DataFrame(nltk.pos_tag(text1))\n",
    "df.columns = ['word', 'pos']\n",
    "df = df.groupby('pos')['pos'].count().sort_values(ascending=False)\n",
    "list(zip(df.head(5).index, df.head(5)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2 - Spelling Recommenders\n",
    "\n",
    "For this part of the assignment you will create three different spelling recommenders, that each take a list of misspelled words and recommends a correctly spelled word for every word in the list.\n",
    "\n",
    "For every misspelled word, the recommender should find find the word in `correct_spellings` that has the shortest distance*, and starts with the same letter as the misspelled word, and return that word as a recommendation.\n",
    "\n",
    "*Each of the three different recommenders will use a different distance measure (outlined below).\n",
    "\n",
    "Each of the recommenders should provide recommendations for the three default words provided: `['cormulent', 'incendenece', 'validrate']`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.corpus import words\n",
    "from nltk.metrics.distance import jaccard_distance, edit_distance\n",
    "from nltk.util import ngrams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "correct_spellings = words.words()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Spelling recommender - n-gram of 2 words\n",
    "\n",
    "For this recommender, your function should provide recommendations for the three default words provided above using the following distance metric:\n",
    "\n",
    "**[Jaccard distance](https://en.wikipedia.org/wiki/Jaccard_index) on the n-grams of the two words.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def spelling_recommender_ngram(entries, n_gram, n=1):\n",
    "    best = []\n",
    "    f_dist = lambda entry, word: jaccard_distance(set(ngrams(entry, n_gram)), set(ngrams(word, n_gram)))\n",
    "    for entry in entries:\n",
    "        # fast scan of matching first letter\n",
    "        spellings_check = [w for w in correct_spellings if w[:n] == entry[:n]]\n",
    "\n",
    "        # series with distance as key and word as value - 'auto sort on key'\n",
    "        df = pd.Series({f_dist(entry, word):word for word in spellings_check})\n",
    "        best.append(df.iloc[0])\n",
    "\n",
    "    return best"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %%timeit\n",
    "spelling_recommender_ngram(['cormulent', 'incendenece', 'validrate'], 3, 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def spelling_recommender_ngram2(entries, n_gram, n=1):\n",
    "    best = []\n",
    "    for i, entry in enumerate(entries):\n",
    "        # fast scan of matching first letter\n",
    "        spellings_check = [w for w in correct_spellings if w[:n] == entry[:n]]\n",
    "        # tuples with (entry, word, distance)\n",
    "        distances = [(entry, word, jaccard_distance(set(ngrams(entry, n_gram)), set(ngrams(word, n_gram))))\n",
    "                                   for word in spellings_check]\n",
    "        distances.sort(key=lambda tup: tup[2])\n",
    "        best.append(distances[0])\n",
    "\n",
    "    return [word for _, word, _ in best]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %%timeit\n",
    "spelling_recommender_ngram2(['cormulent', 'incendenece', 'validrate'], 2, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%timeit\n",
    "spelling_recommender_ngram(['cormulent', 'incendenece', 'validrate'], 2, 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%timeit\n",
    "spelling_recommender_ngram(['cormulent', 'incendenece', 'validrate'], 2, 4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "spelling_recommender_ngram(['cormulent', 'incendenece', 'validrate'], 3, 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Spelling recommender - levenshtein of 3 words\n",
    "\n",
    "For this recommender, your function should provide recommendations for the three default words provided above using the following distance metric:\n",
    "\n",
    "**[Edit distance on the two words with transpositions.](https://en.wikipedia.org/wiki/Damerau%E2%80%93Levenshtein_distance)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def spelling_recommender_edit(entries, n=1):\n",
    "    best = []\n",
    "    f_dist = lambda entry, word: edit_distance(entry, word)\n",
    "    \n",
    "    for i, entry in enumerate(entries):\n",
    "        spellings_check = [w for w in correct_spellings if w[:n] == entry[:n]]\n",
    "\n",
    "        # series with distance as key and word as value - 'auto sort on key'\n",
    "        df = pd.Series({f_dist(entry, word):word for word in spellings_check})\n",
    "        best.append(df.iloc[0])\n",
    "\n",
    "    return best"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%timeit\n",
    "spelling_recommender_edit(['cormulent', 'incendenece', 'validrate'], 4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "spelling_recommender_edit(['cormulent', 'incendenece', 'validrate'], 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}