Assignment_2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "_You are currently looking at **version 1.0** of this notebook. To download notebooks and datafiles, as well as get help on Jupyter notebooks in the Coursera platform, visit the [Jupyter Notebook FAQ](https://www.coursera.org/learn/python-text-mining/resources/d9pwm) course resource._\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment 2 - Introduction to NLTK\n",
    "\n",
    "In part 1 of this assignment you will use nltk to explore the Herman Melville novel Moby Dick. Then in part 2 you will create a spelling recommender function that uses nltk to find words similar to the misspelling. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1 - Analyzing Moby Dick"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "nltk.download('popular')\n",
    "\n",
    "nltk.download('nps_chat')\n",
    "\n",
    "nltk.download('webtext')\n",
    "\n",
    "from nltk.book import *\n",
    "\n",
    "# If you would like to work with the raw text you can use 'moby_raw'\n",
    "with open('moby.txt', 'r') as f:\n",
    "    moby_raw = f.read()\n",
    "    \n",
    "# If you would like to work with the novel in nltk.Text format you can use 'text1'\n",
    "moby_tokens = nltk.word_tokenize(moby_raw)\n",
    "text1 = nltk.Text(moby_tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 1\n",
    "\n",
    "How many tokens (words and punctuation symbols) are in text1?\n",
    "\n",
    "*This function should return an integer.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "254989"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def example_one():\n",
    "    \n",
    "    return len(nltk.word_tokenize(moby_raw)) # or alternatively len(text1)\n",
    "\n",
    "example_one()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 2\n",
    "\n",
    "How many unique tokens (unique words and punctuation) does text1 have?\n",
    "\n",
    "*This function should return an integer.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20755"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def example_two():\n",
    "    \n",
    "    return len(set(nltk.word_tokenize(moby_raw))) # or alternatively len(set(text1))\n",
    "\n",
    "example_two()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 3\n",
    "\n",
    "After lemmatizing the verbs, how many unique tokens does text1 have?\n",
    "\n",
    "*This function should return an integer.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "16900"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.stem import WordNetLemmatizer\n",
    "\n",
    "def example_three():\n",
    "\n",
    "    lemmatizer = WordNetLemmatizer()\n",
    "    lemmatized = [lemmatizer.lemmatize(w,'v') for w in text1]\n",
    "\n",
    "    return len(set(lemmatized))\n",
    "\n",
    "example_three()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 1\n",
    "\n",
    "What is the lexical diversity of the given text input? (i.e. ratio of unique tokens to the total number of tokens)\n",
    "\n",
    "*This function should return a float.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.08139566804842562"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_one():\n",
    "    \n",
    "    \n",
    "    return example_two() / example_one()\n",
    "\n",
    "answer_one()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 2\n",
    "\n",
    "What percentage of tokens is 'whale'or 'Whale'?\n",
    "\n",
    "*This function should return a float.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.4125668166077752"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_two():\n",
    "    \n",
    "    dist = FreqDist(text1)\n",
    "    one = dist[u'whale']\n",
    "    two = dist[u'Whale']\n",
    "    return ((one + two) / example_one())*100\n",
    "\n",
    "answer_two()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 3\n",
    "\n",
    "What are the 20 most frequently occurring (unique) tokens in the text? What is their frequency?\n",
    "\n",
    "*This function should return a list of 20 tuples where each tuple is of the form `(token, frequency)`. The list should be sorted in descending order of frequency.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(',', 19204),\n",
       " ('the', 13715),\n",
       " ('.', 7308),\n",
       " ('of', 6513),\n",
       " ('and', 6010),\n",
       " ('a', 4545),\n",
       " ('to', 4515),\n",
       " (';', 4173),\n",
       " ('in', 3908),\n",
       " ('that', 2978),\n",
       " ('his', 2459),\n",
       " ('it', 2196),\n",
       " ('I', 2097),\n",
       " ('!', 1767),\n",
       " ('is', 1722),\n",
       " ('--', 1713),\n",
       " ('with', 1659),\n",
       " ('he', 1658),\n",
       " ('was', 1639),\n",
       " ('as', 1620)]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_three():\n",
    "    dist = FreqDist(text1)\n",
    "    return dist.most_common(20)\n",
    "\n",
    "answer_three()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 4\n",
    "\n",
    "What tokens have a length of greater than 5 and frequency of more than 150?\n",
    "\n",
    "*This function should return an alphabetically sorted list of the tokens that match the above constraints. To sort your list, use `sorted()`*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Captain',\n",
       " 'Pequod',\n",
       " 'Queequeg',\n",
       " 'Starbuck',\n",
       " 'almost',\n",
       " 'before',\n",
       " 'himself',\n",
       " 'little',\n",
       " 'seemed',\n",
       " 'should',\n",
       " 'though',\n",
       " 'through',\n",
       " 'whales',\n",
       " 'without']"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_four():\n",
    "    \n",
    "    dist = FreqDist(text1)\n",
    "    keys = dist.keys()\n",
    "    res = [w for w in keys if len(w) > 5 and dist[w] > 150]\n",
    "    return sorted(res)\n",
    "\n",
    "answer_four()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 5\n",
    "\n",
    "Find the longest word in text1 and that word's length.\n",
    "\n",
    "*This function should return a tuple `(longest_word, length)`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(\"twelve-o'clock-at-night\", 23)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_five():\n",
    "    \n",
    "    words = list(set(text1))\n",
    "    sortedwords = sorted(words, key=len)\n",
    "    return sortedwords[-1], len(sortedwords[-1])\n",
    "\n",
    "answer_five()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 6\n",
    "\n",
    "What unique words have a frequency of more than 2000? What is their frequency?\n",
    "\n",
    "\"Hint:  you may want to use `isalpha()` to check if the token is a word and not punctuation.\"\n",
    "\n",
    "*This function should return a list of tuples of the form `(frequency, word)` sorted in descending order of frequency.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(13715, 'the'),\n",
       " (6513, 'of'),\n",
       " (6010, 'and'),\n",
       " (4545, 'a'),\n",
       " (4515, 'to'),\n",
       " (3908, 'in'),\n",
       " (2978, 'that'),\n",
       " (2459, 'his'),\n",
       " (2196, 'it'),\n",
       " (2097, 'I')]"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_six():\n",
    "    \n",
    "    from operator import itemgetter\n",
    "    dist = FreqDist(text1)\n",
    "    keys = dist.keys()\n",
    "    res = [(dist[w], w) for w in keys if w.isalpha() and dist[w] > 2000]\n",
    "    return sorted(res, key=itemgetter(0), reverse=True)\n",
    "\n",
    "answer_six()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 7\n",
    "\n",
    "What is the average number of tokens per sentence?\n",
    "\n",
    "*This function should return a float.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "25.881952902963864"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_seven():\n",
    "    sentences = nltk.sent_tokenize(moby_raw)\n",
    "    summation = 0\n",
    "    for sen in sentences:\n",
    "        words = nltk.word_tokenize(sen)\n",
    "        summation += len(words)\n",
    "    return summation / len(sentences)\n",
    "\n",
    "answer_seven()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 8\n",
    "\n",
    "What are the 5 most frequent parts of speech in this text? What is their frequency?\n",
    "\n",
    "*This function should return a list of tuples of the form `(part_of_speech, frequency)` sorted in descending order of frequency.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('NN', 33152), ('IN', 28601), ('DT', 25975), (',', 19204), ('JJ', 17420)]"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_eight():\n",
    "    \n",
    "    from operator import itemgetter\n",
    "    sentences = nltk.sent_tokenize(moby_raw)\n",
    "    dicts = {}\n",
    "    for sen in sentences:\n",
    "        words = nltk.word_tokenize(sen)\n",
    "        pair = nltk.pos_tag(words)\n",
    "        for (x,y) in pair:\n",
    "            if y in dicts:\n",
    "                dicts[y] += 1\n",
    "            else:\n",
    "                dicts[y] = 1\n",
    "    lst = [(pos,freq) for pos, freq in dicts.items()]\n",
    "    return sorted(lst, key=itemgetter(1), reverse=True)[:5]\n",
    "\n",
    "answer_eight()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2 - Spelling Recommender\n",
    "\n",
    "For this part of the assignment you will create three different spelling recommenders, that each take a list of misspelled words and recommends a correctly spelled word for every word in the list.\n",
    "\n",
    "For every misspelled word, the recommender should find find the word in `correct_spellings` that has the shortest distance*, and starts with the same letter as the misspelled word, and return that word as a recommendation.\n",
    "\n",
    "*Each of the three different recommenders will use a different distance measure (outlined below).\n",
    "\n",
    "Each of the recommenders should provide recommendations for the three default words provided: `['cormulent', 'incendenece', 'validrate']`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['abalienate',\n",
       " 'abalienation',\n",
       " 'abalone',\n",
       " 'Abama',\n",
       " 'abampere',\n",
       " 'abandon',\n",
       " 'abandonable',\n",
       " 'abandoned',\n",
       " 'abandonedly',\n",
       " 'abandonee',\n",
       " 'abandoner',\n",
       " 'abandonment',\n",
       " 'Abanic',\n",
       " 'Abantes',\n",
       " 'abaptiston',\n",
       " 'Abarambo',\n",
       " 'Abaris',\n",
       " 'abarthrosis',\n",
       " 'abarticular',\n",
       " 'abarticulation',\n",
       " 'abas',\n",
       " 'abase',\n",
       " 'abased',\n",
       " 'abasedly',\n",
       " 'abasedness',\n",
       " 'abasement',\n",
       " 'abaser',\n",
       " 'Abasgi',\n",
       " 'abash',\n",
       " 'abashed']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.corpus import words\n",
    "\n",
    "correct_spellings = words.words()\n",
    "correct_spellings[40:70]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 9\n",
    "\n",
    "For this recommender, your function should provide recommendations for the three default words provided above using the following distance metric:\n",
    "\n",
    "**[Jaccard distance](https://en.wikipedia.org/wiki/Jaccard_index) on the trigrams of the two words.**\n",
    "\n",
    "*This function should return a list of length three:\n",
    "`['cormulent_reccomendation', 'incendenece_reccomendation', 'validrate_reccomendation']`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.6/site-packages/ipykernel_launcher.py:12: DeprecationWarning: generator 'ngrams' raised StopIteration\n",
      "  if sys.path[0] == '':\n",
      "/opt/conda/lib/python3.6/site-packages/ipykernel_launcher.py:18: DeprecationWarning: generator 'ngrams' raised StopIteration\n",
      "/opt/conda/lib/python3.6/site-packages/ipykernel_launcher.py:24: DeprecationWarning: generator 'ngrams' raised StopIteration\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['corpulent', 'indecence', 'validate']"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_nine(entries=['cormulent', 'incendenece', 'validrate']):\n",
    "    \n",
    "    min1 = min2 = min3 = 999999999\n",
    "    \n",
    "    set1 = set(nltk.ngrams(entries[0], n=3))\n",
    "    set2 = set(nltk.ngrams(entries[1], n=3))\n",
    "    set3 = set(nltk.ngrams(entries[2], n=3))\n",
    "    \n",
    "    for i in range(len(correct_spellings)):\n",
    "        string = correct_spellings[i]\n",
    "        if string[0] == 'c':\n",
    "            set11 = set(nltk.ngrams(string, n=3))\n",
    "            ed = nltk.jaccard_distance(set1, set11)\n",
    "            if ed < min1:\n",
    "                min1 = ed\n",
    "                id1 = i\n",
    "        elif string[0] == 'i':\n",
    "            set12 = set(nltk.ngrams(string, n=3))\n",
    "            ed = nltk.jaccard_distance(set2, set12)\n",
    "            if ed < min2:\n",
    "                min2 = ed\n",
    "                id2 = i\n",
    "        elif string[0] == 'v':\n",
    "            set13 = set(nltk.ngrams(string, n=3))\n",
    "            ed = nltk.jaccard_distance(set3, set13)\n",
    "            if ed < min3:\n",
    "                min3 = ed\n",
    "                id3 = i\n",
    "        else:\n",
    "            continue\n",
    "        \n",
    "    return [correct_spellings[id1], correct_spellings[id2], correct_spellings[id3]]\n",
    "    \n",
    "answer_nine()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 10\n",
    "\n",
    "For this recommender, your function should provide recommendations for the three default words provided above using the following distance metric:\n",
    "\n",
    "**[Jaccard distance](https://en.wikipedia.org/wiki/Jaccard_index) on the 4-grams of the two words.**\n",
    "\n",
    "*This function should return a list of length three:\n",
    "`['cormulent_reccomendation', 'incendenece_reccomendation', 'validrate_reccomendation']`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.6/site-packages/ipykernel_launcher.py:12: DeprecationWarning: generator 'ngrams' raised StopIteration\n",
      "  if sys.path[0] == '':\n",
      "/opt/conda/lib/python3.6/site-packages/ipykernel_launcher.py:18: DeprecationWarning: generator 'ngrams' raised StopIteration\n",
      "/opt/conda/lib/python3.6/site-packages/ipykernel_launcher.py:24: DeprecationWarning: generator 'ngrams' raised StopIteration\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['cormus', 'incendiary', 'valid']"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_ten(entries=['cormulent', 'incendenece', 'validrate']):\n",
    "    \n",
    "    min1 = min2 = min3 = 999999999\n",
    "    \n",
    "    set1 = set(nltk.ngrams(entries[0], n=4))\n",
    "    set2 = set(nltk.ngrams(entries[1], n=4))\n",
    "    set3 = set(nltk.ngrams(entries[2], n=4))\n",
    "    \n",
    "    for i in range(len(correct_spellings)):\n",
    "        string = correct_spellings[i]\n",
    "        if string[0] == 'c':\n",
    "            set11 = set(nltk.ngrams(string, n=4))\n",
    "            ed = nltk.jaccard_distance(set1, set11)\n",
    "            if ed < min1:\n",
    "                min1 = ed\n",
    "                id1 = i\n",
    "        elif string[0] == 'i':\n",
    "            set12 = set(nltk.ngrams(string, n=4))\n",
    "            ed = nltk.jaccard_distance(set2, set12)\n",
    "            if ed < min2:\n",
    "                min2 = ed\n",
    "                id2 = i\n",
    "        elif string[0] == 'v':\n",
    "            set13 = set(nltk.ngrams(string, n=4))\n",
    "            ed = nltk.jaccard_distance(set3, set13)\n",
    "            if ed < min3:\n",
    "                min3 = ed\n",
    "                id3 = i\n",
    "        else:\n",
    "            continue\n",
    "        \n",
    "    return [correct_spellings[id1], correct_spellings[id2], correct_spellings[id3]]\n",
    "    \n",
    "answer_ten()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 11\n",
    "\n",
    "For this recommender, your function should provide recommendations for the three default words provided above using the following distance metric:\n",
    "\n",
    "**[Edit distance on the two words with transpositions.](https://en.wikipedia.org/wiki/Damerau%E2%80%93Levenshtein_distance)**\n",
    "\n",
    "*This function should return a list of length three:\n",
    "`['cormulent_reccomendation', 'incendenece_reccomendation', 'validrate_reccomendation']`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['corpulent', 'intendence', 'validate']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def answer_eleven(entries=['cormulent', 'incendenece', 'validrate']):\n",
    "    \n",
    "    min1 = min2 = min3 = 999999999\n",
    "    \n",
    "    for i in range(len(correct_spellings)):\n",
    "        string = correct_spellings[i]\n",
    "        if string[0] == 'c':\n",
    "            ed = nltk.edit_distance(entries[0], string)\n",
    "            if ed < min1:\n",
    "                min1 = ed\n",
    "                id1 = i\n",
    "        elif string[0] == 'i':\n",
    "            ed = nltk.edit_distance(entries[1], string)\n",
    "            if ed < min2:\n",
    "                min2 = ed\n",
    "                id2 = i\n",
    "        elif string[0] == 'v':\n",
    "            ed = nltk.edit_distance(entries[2], string)\n",
    "            if ed < min3:\n",
    "                min3 = ed\n",
    "                id3 = i\n",
    "        else:\n",
    "            continue\n",
    "        \n",
    "    return [correct_spellings[id1], correct_spellings[id2], correct_spellings[id3]] \n",
    "    \n",
    "answer_eleven()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "coursera": {
   "course_slug": "python-text-mining",
   "graded_item_id": "r35En",
   "launcher_item_id": "tCVfW",
   "part_id": "NTVgL"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}