GCDRI_TextAnalysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Introduction to Text Analysis with Python\n",
    "<br><br>\n",
    "### GCDRI June 8, 2016\n",
    "### Michelle McSweeney \n",
    "### @MMcSweeney7\n",
    "### \\#GCDRI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Major steps in doing text or language analysis\n",
    "1. Getting the data\n",
    "2. **Turning the data into numbers**\n",
    "3. Analyzing the data "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Question\n",
    "\n",
    "What parts of language (spoken, written, texted) can you count? \n",
    "\n",
    "What numbers can you come up with beyond counting?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Text Editor Functions\n",
    "\n",
    "* Word counts\n",
    "* Character Counts\n",
    "* Word-in-context (text wrangler & sublime)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Moving Beyond the Text Editor with Python + NLTK\n",
    "\n",
    "The Natural Language ToolKit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "from nltk.book import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Concordance**: Words in their contexts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "text1.concordance(\"love\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Similar**: Words that appear in a similar environment as the target word\n",
    "\n",
    "Might do this with two different texts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "text1.similar(\"hope\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "text2.similar(\"hope\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Common Contexts**: Two words that appear in similar environments as each other"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "text1.common_contexts(['whale','monster'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "text1.dispersion_plot([\"whale\", \"monster\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The Dispersion Plot is usually **BEHIND** the browser!\n",
    "\n",
    "**CLOSE THE DISPERSION PLOT BEFORE WE MOVE ON**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now we are going to calculate a bunch of things related to my text without NLTK - just using Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Count a specific word - how many times does this sequence of characters occur in my document?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "text1.count(\"love\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "How many **tokens** are in my text?\n",
    "\n",
    "**tokens** are unique sequences, let's start with an example:\n",
    "\n",
    "\"love\", \"bowie\", \"Bowie\", \"!\" and \":)\" are all unique **tokens**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "len(text1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "How many **unique** words are in my text? \n",
    "\n",
    "* first make a set that groups all the \"words\" together (numbers, punctuation sequences, etc.) - this groups together **types**. \n",
    "* Token = instance\n",
    "* Type = more general (\"bowie\" and \"Bowie\" are different types - why?)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "set(text1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "I like to **sort** my **set** so I know what I have:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "sorted(set(text1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now count the number of items in that set to find the number of unique words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "len(set(text1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Lexical Density**: the number of unique tokens divided by the total number of words. \n",
    "\n",
    "This is a descriptive measure of language register or grade level approximations.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "len(set(text1))/len(text1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Frequency Distribution** is a probability object that Python deals with. We will use it to make a graph of the most common words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "my_dist = FreqDist(text1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Since nothing appears, I like to go check for it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "type(my_dist)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now let's **plot** the graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "my_dist.plot(50,cumulative=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "It may be a little easier to look at as a list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "my_dist.most_common(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "That actually doesn't tell us very much. \n",
    "\n",
    "**PREVIEW** <br>\n",
    "*We need to remove the **stopwords** to learn more.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "love_words = ['love', 'joy', 'hope', 'amor']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "my_list = []\n",
    "for word in love_words:\n",
    "    if word in text8:\n",
    "        my_list.append(word)\n",
    "    else:\n",
    "        pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "print(my_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's pull a book in from the Internet\n",
    "\n",
    "Project Gutenburg is a great source! www.gutenberg.org\n",
    "\n",
    "Text 37472 is Zanzibar Tales translated by George W. Bateman\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "To make a book into a Text NLTK can deal with, we have to:\n",
    "\n",
    "* open the file from a location\n",
    "* read it/decode it \n",
    "* tokenize it (go from a string to a list of word)\n",
    "* nltk.Text() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "#import the urlopen command\n",
    "from urllib.request import urlopen\n",
    "#set the url to a variable\n",
    "my_url = \"https://www.gutenberg.org/files/37472/37472.txt\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "#open the file from the url\n",
    "file = urlopen(my_url)\n",
    "#read the opened file\n",
    "raw = file.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "#specify which decoding to use. (usually utf-8)\n",
    "zt = raw.decode('utf-8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "#check the type to be sure it worked. I expect a string now.\n",
    "type(zt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "If this doesn't work, or you are dealing with messier text, check out the ftfy library\n",
    "Fixes Text For You   \n",
    "\n",
    "http://ftfy.readthedocs.io"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import ftfy.bad_codecs\n",
    "\n",
    "bad_zt = raw.decode('utf-8-variants')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "type(zt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "#split the string into words with word_tokenize\n",
    "zt_tok = nltk.word_tokenize(zt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "#check to make sure it worked\n",
    "type(zt_tok)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "#get an idea of how big the file is\n",
    "len(zt_tok)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "#look at the first 10 words to be sure its correct\n",
    "zt_tok[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Yuck! That just looks like metadata! \n",
    "\n",
    "Removing metadata involves using Regular Expressions\n",
    "\n",
    "Regular Expressions are saved for another day. They are powerful but complicated"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "#I want to get rid of that intro metadata!!\n",
    "\n",
    "#I found the number 177 by copying this into Text Wrangler\n",
    "#Text Wrangler counts words, characters, and spaces\n",
    "\n",
    "#To do this for many files, you need Regular Expressions\n",
    "\n",
    "zt_tok[177:188]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "#turn the list of words into a text nltk can recognize\n",
    "zt_text = nltk.Text(zt_tok[177:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "#check to make sure it worked\n",
    "type(zt_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "#get an idea of how big the file is\n",
    "len(zt_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "One step further! \n",
    "**Part-of-Speech Tagging**\n",
    "\n",
    "NLTK uses the Penn Tag Set. \n",
    "\n",
    "There are better options (i.e., tree tagger and polyglot), but this illustrates the idea."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "#make a new object that has all the words and tags in it\n",
    "zt_tagged = nltk.pos_tag(zt_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "print(zt_tagged[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "This doesn't look like a dictionary! What's going on??"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "type(zt_tagged)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We have a **list of tuples**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "I'm going to put it in a for-loop\n",
    "\n",
    "Have to deal with this in a special way \n",
    "<br>(a, b) in my_list\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "#function to determine what is the most common tag in Zanzibar Tales\n",
    "def commontag(taggedbook):\n",
    "#create an empty dictionary\n",
    "    tag_dict = {}\n",
    "#for every word/tag combo in my list, \n",
    "    for (word, tag) in taggedbook:\n",
    "        if tag in tag_dict: \n",
    "            tag_dict[tag]+=1\n",
    "        else:\n",
    "            tag_dict[tag] = 1\n",
    "    print(tag_dict)\n",
    "\n",
    "commontag(zt_tagged)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "I wish I would have made that return an ordered dictionary!!!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Let's add a line of code with OrderedDict in it'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from collections import OrderedDict\n",
    "\n",
    "def commontag(taggedbook):\n",
    "#create an empty dictionary\n",
    "    tag_dict = {}\n",
    "#for every word/tag combo in my list, \n",
    "    for (word, tag) in taggedbook:\n",
    "        if tag in tag_dict: \n",
    "            tag_dict[tag]+=1\n",
    "        else:\n",
    "            tag_dict[tag] = 1\n",
    "    tag_dict = OrderedDict(sorted(tag_dict.items(), key=lambda t: t[1]))\n",
    "    print(tag_dict)\n",
    "\n",
    "commontag(zt_tagged)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "How do you know to put all that other stuff in (i.e., sorted, lambda, etc)?!?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "*Read the docs*\n",
    "\n",
    "https://docs.python.org/3.1/whatsnew/3.1.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "So far, we have counted things in our texts by looking at \n",
    "\n",
    "* Concordance\n",
    "* Words in similar environments\n",
    "* Words in common contexts\n",
    "* Unique words \n",
    "* Length of words \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Then we performed some operations, but still counted things:\n",
    "\n",
    "* Frequency Distributions\n",
    "* Lexical Density \n",
    "* Found words from a list in a text\n",
    "* Part-of-Speech Tags"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now we will perform operations on the Text itself before doing those operations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "To get a better idea of the content of a text, it's usually best to exclude stopwords\n",
    "\n",
    "**Stopwords** perform grammatical functions, but have limited semantic content.\n",
    "\n",
    "(the, a, at, in, of, with, etc.)\n",
    "\n",
    "Two methods:\n",
    "\n",
    "* Use a pre-defined list from nltk\n",
    "* Make your own list and use a for-loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from nltk.corpus import stopwords\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "#have to tell NLTK that you want the English stopwords\n",
    "mystops = stopwords.words('english')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "nostop_text1 = []\n",
    "\n",
    "#remove stop words\n",
    "#go through all the words in text1 and save all the non-stopwords\n",
    "for word in text1:\n",
    "    if word not in mystops:\n",
    "        nostop_text1.append(word)\n",
    "    else:\n",
    "        pass\n",
    "    \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "print(nostop_text1[50:150])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now to:\n",
    "* remove all that punctuation \n",
    "* make everything lowercase"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "#remove punctuation\n",
    "#go through all the items in text1 (without stopwords), and save everything that is alphabetic\n",
    "nopunct_text1 = []\n",
    "for word in nostop_text1:\n",
    "    if word.isalpha():\n",
    "        nopunct_text1.append(word)\n",
    "    else:\n",
    "        pass\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "lower_text1 = []\n",
    "\n",
    "for w in nopunct_text1:\n",
    "    lower_text1.append(w.lower())\n",
    "    \n",
    "print(lower_text1[:50])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "There's another way to write this, if this is easier to understand:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "new_text1 = [w for w in text1 if w not in mystops]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now you have a nice clean text. \n",
    "\n",
    "Let's look at the lexical density of that text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "len(set(new_text1))/len(new_text1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "That's much higher!! \n",
    "\n",
    "This is the density of the whole book without the stop words, which better represents the variety of words used"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "What if I want to read in my OWN corpus?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "f = open(\"/Users/michellejohnson/Desktop/projects/Thailand.txt\", 'r')\n",
    "my_file = f.read()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "type(my_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Going Forward**\n",
    "\n",
    "* Use a text editor to write complete programs\n",
    "    * Run these in the terminal\n",
    "* Use Spyder to write complete programs\n",
    "* Often save the program you write in the same file as the file you will be working with to shorted the path."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How do I know where to go?!?\n",
    "* http://www.nltk.org/book_1ed\n",
    "* http://www.nltk.org/\n",
    "    \n",
    "* Come to office hours\n",
    "* Come to Python Users' Group\n",
    "* Play! \n",
    "    * http://techblog.about.com/post/140231383537/analyzing-the-language-of-the-presidential-debates\n",
    "    * http://andybromberg.com/sentiment-analysis-python/\n",
    "    * etc.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}