Weekly/Week_3/text-analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Analysis with NLTK\n",
    "\n",
    "The [Natural Language Toolkit](http://www.nltk.org/) (NLTK) and is a fully featured library for doing everything from basic text analysis to sophisticated computational linguistics.\n",
    "\n",
    "We're just scratch the surface here:\n",
    "- Parsing\n",
    "- Tokenization\n",
    "- Stopword removal\n",
    "- Frequency counting\n",
    "- Stemming\n",
    "- Basic sentiment analysis\n",
    "\n",
    "For more detail and to go deeper in learning about Natural Language Processing a good resources is: [Natural Langauge Processing with Python](http://www.nltk.org/book/). There are also APIs out there that allow you to do quite sophisticated text analysis, for example [AlchemyAPI](http://www.alchemyapi.com/). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# We're going to use the Natural Language Toolkit here (NLTK) - http://www.nltk.org/\n",
    "import nltk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's first look at some basic text operations using the 2016 State of the Union (SOTU) address. Here's the [sotu_2016.txt](https://www.dropbox.com/s/nume1fwb2g3frpf/sotu_2016.txt?dl=0) file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Load in transcript file\n",
    "file = open('Data/sotu_2016.txt', 'r')\n",
    "transcript = unicode(file.read().decode(\"utf8\"))\n",
    "\n",
    "# Print it out\n",
    "print transcript"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Converting to lowercase text is very easy with the `.lower()` function. Sometimes is makes sense to convert everything because then capitals at the beginning of sentences or in names aren't treated differently. Other functions in the same vein include `.upper()` (converts to upper case) and `.title()` (converts to title case)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "transcript_lower = transcript.lower()\n",
    "print transcript_lower"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you look *really* closely at the output you can see some inconsistencies. For instance sometimes it looks like the apostrophe's are angled and other times they're not. We want to normalize the text so that it's always using the same version of the apostrophe, as well as quotes and a few other special characters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "table = {\n",
    "    ord(u'\\u2018') : u\"'\",\n",
    "    ord(u'\\u2019') : u\"'\",\n",
    "    ord(u'\\u201C') : u'\"',\n",
    "    ord(u'\\u201d') : u'\"',\n",
    "    ord(u'\\u2026') : u'', \n",
    "    ord(u'\\u2014') : u'', # get rid of em dashes\n",
    "}\n",
    "transcript_lower = transcript_lower.translate(table)\n",
    "print transcript_lower"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we want to strip out all of the punctuation so that we're left with just the words. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import string\n",
    "\n",
    "def remove_punctuation(text):\n",
    "    # Grab the list of standard punctuation symbols that are provided in the string library\n",
    "    punctuations = string.punctuation # includes following characters: !\"#$%&\\'()*+,-./:;<=>?@[\\\\]^_`{|}~\n",
    "\n",
    "    # But don't strip out apostrophes, as we want to preserve possessives and contractions, an alternative would be to expand contractions\n",
    "    excluded_punctuations = [\"'\"]\n",
    "    for p in punctuations:\n",
    "        if p not in excluded_punctuations:\n",
    "            # replace each punctuation symbol by a space\n",
    "            text = text.replace(p, \" \") \n",
    "\n",
    "    return text\n",
    "\n",
    "transcript_lower = remove_punctuation(transcript_lower)\n",
    "print transcript_lower"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In some cases multiple punctuations in a row were removed, resulting in larger gaps of 2 or more spaces. We want to condense these down to only a single space which will result in a solid block of text with each word separated by only one space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "\" \".join(transcript_lower.split())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenization\n",
    "Tokenization is the process by which we split up a text into individual tokens. In this case we're interested in words as tokens. In other cases we might be interested in tokenizing at the level of paragraphs or sentences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from nltk.tokenize import WhitespaceTokenizer\n",
    "\n",
    "# Create a tokenizer from NLTK which will create tokens based on the whitespace in between words\n",
    "tokenizer = WhitespaceTokenizer()\n",
    "\n",
    "# Use the tokenizer on the text string for the transcript\n",
    "transcript_tokens = tokenizer.tokenize(transcript_lower)\n",
    "\n",
    "# And the output is a python list of tokens (in unicode)\n",
    "print transcript_tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We might wonder how long (i.e. how many tokens) were in the speech. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "len(transcript_tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have a list of tokens we can have NLTK parse it into a Text object which lets us do some exploratory operations on it, like search it and see concordances (i.e. context around the search term). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "text_transcript = nltk.text.Text(transcript_tokens)\n",
    "text_transcript.concordance(\"americans\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Basic visualization options, like a dispersion plot can help understand how different terms are distributed throughout the text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "text_transcript.dispersion_plot([\"education\", \"technology\", \"security\", \"economy\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stopword Removal\n",
    "We may also want to remove stopwords, words that are often considered to be so frequent that they don't contain much information. In general you should carefully consider what words are included in your stoplist, but of course there is a default list of terms built into NLTK. To get that list we first have to initiate the download of the NLTK resources (the next cell should initiate a dialog box where you need to click the \"download\" button)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "nltk.download()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from nltk.corpus import stopwords\n",
    "stopword_list = stopwords.words('english')\n",
    "print stopword_list "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To remove the words in the stopword list we can use a [python list comprehension](https://docs.python.org/2/tutorial/datastructures.html#list-comprehensions)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "transcript_tokens = [w for w in transcript_tokens if w not in stopword_list]\n",
    "print transcript_tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "len(transcript_tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Frequencies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And finally we're ready to count up the frequency of tokens in the text. Below we print just the top 25 most frequent terms. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from nltk.probability import FreqDist\n",
    "\n",
    "# Construct the distribution (it will count the number of occurrances of each unique token)\n",
    "frequency_distribution = FreqDist(transcript_tokens)\n",
    "frequency_distribution.most_common(25)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that a very common terms is \"applause\", because the text included a transcription of when the audience applauded during the speech. We might want to remove that term."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "del frequency_distribution[\"applause\"]\n",
    "frequency_distribution.most_common(25)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Can you start to see some of the themes of the speech based on the frequency of different words chosen? It looks like \"work\" and the \"economy\" came up fairly often. Maybe we want to screen the tokens for other economic types terms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "econ_terms = [\"debt\", \"economy\", \"budget\", \"spending\", \"tax\", \"deficit\", \"work\", \"working\", \"workers\", \"workforce\", \"companies\", \"worked\", \"economics\", \"businesses\", \"jobs\"]\n",
    "\n",
    "econ_transcript_tokens = [t for t in transcript_tokens if t in econ_terms]\n",
    "econ_transcript_freq_dist = FreqDist(econ_transcript_tokens)\n",
    "econ_transcript_freq_dist"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need a chart! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "# This magic function allows you to see the charts directly within the notebook. \n",
    "%matplotlib inline\n",
    "\n",
    "# This command will make the plots more attractive by adopting the commone style of of a different library called ggplot\n",
    "matplotlib.style.use(\"ggplot\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's wrangle the data into a format where we can create a Pandas DataFrame and then chart it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Create a simple list\n",
    "econ_data = []\n",
    "for item in econ_transcript_freq_dist:\n",
    "    econ_data.append([item, econ_transcript_freq_dist[item]])\n",
    "\n",
    "# Use the list to intitlize a pandas dataframe\n",
    "econ_df = pd.DataFrame(econ_data, columns=[\"token\", \"count\"])\n",
    "econ_df = econ_df.sort_values(\"count\")\n",
    "econ_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "axes = econ_df.plot(kind=\"barh\")\n",
    "axes.set_yticklabels(econ_df[\"token\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Individual tokens or unigrams as they're sometimes called aren't the only thing we can tally. We can also look at pairs of words (bigrams), or more generally, n-grams."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "bigrams = nltk.bigrams(transcript_tokens)\n",
    "bigram_freq_dist = FreqDist(bigrams)\n",
    "bigram_freq_dist.most_common(25)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stemming\n",
    "The chart above shows that there are are a number of versions of the same basic idea, e.g. \"workers\", \"working\", and \"worked\" that all have the same underlying concept behind them. \n",
    "\n",
    "Stemming is a technique that is used remove affixes from a word. There are different stemming algorithms that can be employed, and several are built in to NLTK. We'll use the Porter algorithm. The advantage of stemming is that it allows for additional aggregation of words that represent a similar concept. At the same time, there is a loss of information due to this aggregation, and as you can see below, the stems that are left after the stemming process aren't always easy to read."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "porter = nltk.PorterStemmer()\n",
    "econ_transcript_tokens_stemmed = [porter.stem(t) for t in econ_transcript_tokens]\n",
    "econ_transcript_tokens_stemmed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "econ_transcript_stemmed_freq_dist = FreqDist(econ_transcript_tokens_stemmed)\n",
    "econ_transcript_stemmed_freq_dist\n",
    "# Create a simple list\n",
    "econ_data = []\n",
    "for item in econ_transcript_stemmed_freq_dist:\n",
    "    econ_data.append([item, econ_transcript_stemmed_freq_dist[item]])\n",
    "\n",
    "# Use the list to intitlize a pandas dataframe\n",
    "econ_df = pd.DataFrame(econ_data, columns=[\"token\", \"count\"])\n",
    "econ_df = econ_df.sort_values(\"count\")\n",
    "axes = econ_df.plot(kind=\"barh\")\n",
    "axes.set_yticklabels(econ_df[\"token\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sentiment Analysis\n",
    "Sentiment analysis is a general term used to describe how text can be analyzed to infer subjective qualities like positivity, negativity, or other types of emotions like happiness, sadness, or anger. There are many pitfalls to be aware of when you're trying to make something as subjective as human emotion in a quantifiable number. Simple approaches often do not account for even basic linguistic structures like negations (e.g. \"I'm *not* sad today\"). Take these results with a grain of salt, but also be aware that there often is *some* signal in all of the ambiguous noise of text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll look at sentiment analysis by examining a corpus of reviews written on Yelp. Yelp has an [academic dataset](https://www.yelp.com/academic_dataset) that it makes available. It includes information on individual reviews, users, and businesses. From that large dataset we've sampled down to only the reviews that respond to businesses in the state of Maryland, which is a reasonable number. Here is the [yelp_reviews_md.csv](https://www.dropbox.com/s/oc8dx9qp6xk4245/yelp_reviews_md.csv?dl=0) file. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "reviews_df = pd.read_csv(\"Data/yelp_reviews_md.csv\", encoding=\"utf-8\")\n",
    "reviews_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "reviews_df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A simple method to pursue sentiment analysis is using various linguistic resources like dictionaries of terms that have been constructed to correspond to various categories of human communication. One of most popular linguistic resources for doing this is Linguistic Inquiry and Word Count ([LIWC](http://liwc.wpengine.com/)). From that project there are lists of terms that related to sentiment categories like \"Positive Emotion\", \"Negative Emotion\", and \"Anger\" and well as more general types of categories like \"Seeing\", \"Insight\", \"Health/illness\" etc. Another useful project with many linguistic resources is the [General Inquirer](http://www.wjh.harvard.edu/~inquirer/).\n",
    "\n",
    "Here we've isolated the term lists from LIWC corresponding to [positive](https://www.dropbox.com/s/bo82rjzhcd4jy6g/positive_terms.txt?dl=0) and [negative](https://www.dropbox.com/s/9ecamxu1jbtwd6n/negative_terms.txt?dl=0) terms in two separate files (linked previously in this sentence). Note that these term lists have been *stemmed* which means we will also need to stem our input text in order to match it against these dictionaries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "positive_terms = []\n",
    "f = open('Data/positive_terms.txt', \"r\")\n",
    "positive_terms = f.read().splitlines()\n",
    "f.close()\n",
    "\n",
    "negative_terms = []\n",
    "f = open('Data/negative_terms.txt', \"r\")\n",
    "negative_terms = f.read().splitlines()\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print positive_terms\n",
    "print \"\"\n",
    "print negative_terms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The hypothesis that we're going to test with sentiment analysis is that reviews with a higher overall star rating should have less negative sentiment than reviews with a lower overall star rating. And conversely, reviews with a higher overall star rating should have more positive sentiment than reviews with a lower overall star rating.\n",
    "\n",
    "First we need to measure a positivity and negativity score for each review. To do that we're going to count the rate of use of terms from a review that come from the positive or negative dictionaries, respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# This function will take a string and lowercase it, remove punctuations and spaces, and then tokentize and stem those tokens\n",
    "def normalize_review_text(text):\n",
    "    text = text.lower()\n",
    "    text = remove_punctuation(text)\n",
    "    text = \" \".join(text.split())\n",
    "    text_tokens = tokenizer.tokenize(text)\n",
    "    text_tokens = [porter.stem(w) for w in text_tokens if w not in stopword_list]\n",
    "    return text_tokens\n",
    "\n",
    "# Apply the function above to the text column\n",
    "reviews_df[\"text\"] = reviews_df[\"text\"].apply(normalize_review_text)\n",
    "reviews_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def calculate_positivity(text):\n",
    "    num_tokens = len(text)\n",
    "    num_positive_tokens = 0\n",
    "    for t in text:\n",
    "        if t in positive_terms:\n",
    "            num_positive_tokens = num_positive_tokens + 1\n",
    "    # The positivity score is the fraction of tokens that were found in the positive dictionary\n",
    "    return float(num_positive_tokens) / float(num_tokens)\n",
    "\n",
    "reviews_df[\"positivity_score\"] = reviews_df[\"text\"].apply(calculate_positivity)\n",
    "\n",
    "def calculate_negativity(text):\n",
    "    num_tokens = len(text)\n",
    "    num_negative_tokens = 0\n",
    "    for t in text:\n",
    "        if t in negative_terms:\n",
    "            num_negative_tokens = num_negative_tokens + 1\n",
    "    # The positivity score is the fraction of tokens that were found in the positive dictionary\n",
    "    return float(num_negative_tokens) / float(num_tokens)\n",
    "\n",
    "reviews_df[\"negativity_score\"] = reviews_df[\"text\"].apply(calculate_negativity)\n",
    "reviews_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on our hypothesis we should see a positive correlation between `positivity_score` and `stars` and a negative correlation between `negativity_score` and `stars`. Let's see:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import scipy.stats as sp\n",
    "\n",
    "pos_score_stars_corr = sp.pearsonr(reviews_df[\"stars\"].values, reviews_df[\"positivity_score\"].values)\n",
    "pos_score_stars_corr\n",
    "print \"Between the positivity score and the star rating there is a correlation coefficient of %.4f which has a p-value of %.4E \" % (pos_score_stars_corr[0], pos_score_stars_corr[1])\n",
    "\n",
    "print \"\"\n",
    "\n",
    "neg_score_stars_corr = sp.pearsonr(reviews_df[\"stars\"].values, reviews_df[\"negativity_score\"].values)\n",
    "neg_score_stars_corr\n",
    "print \"Between the negativity score and the star rating there is a correlation coefficient of %.4f which has a p-value of %.4E \" % (neg_score_stars_corr[0], neg_score_stars_corr[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on our (albeit very simple) sentiment analysis we've shown that the sentiments we've computed relate to the actual star ratings of reviews quite well - more positive reviews have higher star ratings, more negative reviews have lower star ratings.  \n",
    "\n",
    "How could we use this in an investigation? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}