stopwords.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Stopwords and Filtering\n",
    "\n",
    "## Using stopwords\n",
    "\n",
    "In natural language processing, we frequently work with a list of stopwords, those words that occur most often in any given text in a language. We might want to exclude words from this list from our larger body of texts before analysis, add to this list, or use just those words from our stopword list as a part of our project. \n",
    "\n",
    "NLTK has a handy way for pulling all those stopwords into your program."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', 'her', 'hers', 'herself', 'it', 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', 'these', 'those', 'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so', 'than', 'too', 'very', 's', 't', 'can', 'will', 'just', 'don', 'should', 'now', 'd', 'll', 'm', 'o', 're', 've', 'y', 'ain', 'aren', 'couldn', 'didn', 'doesn', 'hadn', 'hasn', 'haven', 'isn', 'ma', 'mightn', 'mustn', 'needn', 'shan', 'shouldn', 'wasn', 'weren', 'won', 'wouldn']\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "print(nltk.corpus.stopwords.words('english'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The words here are probably unsurprising: prepositions, pronouns, and similar words that show up frequently across English-language corpora. In most cases you will want to supplement such a list with your own words to create a stopwords list that makes sense for your corpus. If you work with early modern texts, for example, you probably have a whole range of different words and phrases particular to your period that you would want to keep in mind. NLTK comes with a range of different lists for different languages, but there are a range of other options available [online ](https://github.com/Alir3z4/stop-words). [CLTK](http://cltk.org/), a variation of NLTK for working with ancient languages, comes with its own lists as well. \n",
    "\n",
    "It's common in NLP to use lists like these to create filters for your text. Let's take a piece of the first chunk of Jacob's room, using the get_chunks() method we developing in the unit on [dividing your text](dividing.ipynb):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['CHAPTER',\n",
       " 'ONE',\n",
       " \"''\",\n",
       " 'So',\n",
       " 'of',\n",
       " 'course',\n",
       " ',',\n",
       " \"''\",\n",
       " 'wrote',\n",
       " 'Betty',\n",
       " 'Flanders',\n",
       " ',',\n",
       " 'pressing',\n",
       " 'her',\n",
       " 'heels',\n",
       " 'rather',\n",
       " 'deeper',\n",
       " 'in',\n",
       " 'the',\n",
       " 'sand',\n",
       " ',',\n",
       " '``',\n",
       " 'there',\n",
       " 'was',\n",
       " 'nothing',\n",
       " 'for',\n",
       " 'it',\n",
       " 'but',\n",
       " 'to',\n",
       " 'leave',\n",
       " '.',\n",
       " \"''\",\n",
       " 'Slowly',\n",
       " 'welling',\n",
       " 'from',\n",
       " 'the',\n",
       " 'point',\n",
       " 'of',\n",
       " 'her',\n",
       " 'gold',\n",
       " 'nib',\n",
       " ',',\n",
       " 'pale',\n",
       " 'blue',\n",
       " 'ink',\n",
       " 'dissolved',\n",
       " 'the',\n",
       " 'full',\n",
       " 'stop',\n",
       " ';']"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import math\n",
    "import nltk\n",
    "\n",
    "def get_chunks(text, num_chunks):\n",
    "    text_length = len(text)\n",
    "    text_chunks = []\n",
    "    number_of_chunks = num_chunks\n",
    "    for i in range(number_of_chunks):\n",
    "        chunk_size = text_length/number_of_chunks\n",
    "        chunk_start = math.floor(chunk_size * i)\n",
    "        chunk_end = math.floor(chunk_size * (i +1))\n",
    "        text_chunks.append(text[chunk_start:chunk_end])\n",
    "    return text_chunks\n",
    "\n",
    "filename = 'corpus/woolf/1922_jacobs_room.txt'\n",
    "with open(filename, 'r') as fin:\n",
    "    raw_text = fin.read()\n",
    "\n",
    "chunked_text = get_chunks(raw_text, 100)\n",
    "tokenized_text = [nltk.word_tokenize(chunk) for chunk in chunked_text]\n",
    "tokenized_text[0][:50]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use our stopwords list from above to filter out common words. This code removes stopwords from the first chunk and prints the first 50 words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['CHAPTER', 'ONE', \"''\", 'So', 'course', ',', \"''\", 'wrote', 'Betty', 'Flanders', ',', 'pressing', 'heels', 'rather', 'deeper', 'sand', ',', '``', 'nothing', 'leave', '.', \"''\", 'Slowly', 'welling', 'point', 'gold', 'nib', ',', 'pale', 'blue', 'ink', 'dissolved', 'full', 'stop', ';', 'pen', 'stuck', ';', 'eyes', 'fixed', ',', 'tears', 'slowly', 'filled', '.', 'The', 'entire', 'bay', 'quivered', ';']\n"
     ]
    }
   ],
   "source": [
    "filtered_chunk = [token for token in tokenized_text[0] if token not in nltk.corpus.stopwords.words('english')]\n",
    "print(filtered_chunk[:50])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice the words that are gone now: 'of', 'her', 'in', 'the'. Filtering stopwords often gives us a sense of those words that are more likely to be meaningful for the particular text. Here, we get character names, as well as a sense of vocabulary that might be thought of as more particular to Woolf. Note how these words are particular to the content of her text - we get a sense of what she is writing about and how she is describing it. You could imagine comparing this list of vocabulary to that of another author and finding it to be quite different!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding to your stopwords list\n",
    "\n",
    "In the previous example, even though we filtered out common words, we still have quite a lot of unwanted characters - punctuation, for example. These grammatical markings are often filtered out in much the same way, using a different part of nltk to expand our stopwords list. There are a couple different ways to do this. The simplest draws upon Python's built-in string class, which has a list of punctuation in it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', 'her', 'hers', 'herself', 'it', 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', 'these', 'those', 'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so', 'than', 'too', 'very', 's', 't', 'can', 'will', 'just', 'don', 'should', 'now', 'd', 'll', 'm', 'o', 're', 've', 'y', 'ain', 'aren', 'couldn', 'didn', 'doesn', 'hadn', 'hasn', 'haven', 'isn', 'ma', 'mightn', 'mustn', 'needn', 'shan', 'shouldn', 'wasn', 'weren', 'won', 'wouldn', '!', '\"', '#', '$', '%', '&', \"'\", '(', ')', '*', '+', ',', '-', '.', '/', ':', ';', '<', '=', '>', '?', '@', '[', '\\\\', ']', '^', '_', '`', '{', '|', '}', '~']\n"
     ]
    }
   ],
   "source": [
    "import string\n",
    "string.punctuation\n",
    "\n",
    "# create an initial stopword list by loading in the one from nltk\n",
    "\n",
    "stopword_list = nltk.corpus.stopwords.words('english')\n",
    "\n",
    "# add punctuation to that list and print out our new list\n",
    "\n",
    "stopword_list.extend(string.punctuation)\n",
    "print(stopword_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use this expanded list to filter again and get a more refined list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['CHAPTER', 'ONE', \"''\", 'So', 'course', \"''\", 'wrote', 'Betty', 'Flanders', 'pressing', 'heels', 'rather', 'deeper', 'sand', '``', 'nothing', 'leave', \"''\", 'Slowly', 'welling', 'point', 'gold', 'nib', 'pale', 'blue', 'ink', 'dissolved', 'full', 'stop', 'pen', 'stuck', 'eyes', 'fixed', 'tears', 'slowly', 'filled', 'The', 'entire', 'bay', 'quivered', 'lighthouse', 'wobbled', 'illusion', 'mast', 'Mr.', 'Connor', \"'s\", 'little', 'yacht', 'bending']\n"
     ]
    }
   ],
   "source": [
    "filtered_chunk = [token for token in tokenized_text[0] if token not in stopword_list]\n",
    "print(filtered_chunk[:50])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That gets us closer - we pulled out a few punctuation marks. But notice that some still made it through - this text has individual punctuation marks not contained within that generalized list we used before. So we'll want to further extend our stopwords list based on the things that got left out. You can use the same approach to add corpus-specific stopwords if you need to do so. Your research questions will ultimately drive your decisions about what words to include or not. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['CHAPTER', 'ONE', 'So', 'course', 'wrote', 'Betty', 'Flanders', 'pressing', 'heels', 'rather', 'deeper', 'sand', 'nothing', 'leave', 'Slowly', 'welling', 'point', 'gold', 'nib', 'pale', 'blue', 'ink', 'dissolved', 'full', 'stop', 'pen', 'stuck', 'eyes', 'fixed', 'tears', 'slowly', 'filled', 'The', 'entire', 'bay', 'quivered', 'lighthouse', 'wobbled', 'illusion', 'mast', 'Mr.', 'Connor', \"'s\", 'little', 'yacht', 'bending', 'like', 'wax', 'candle', 'sun']\n"
     ]
    }
   ],
   "source": [
    "# make a custom stopwords list and add it to our general stopwords list.\n",
    "custom_stop_list = [\"''\", \"``\"]\n",
    "stopword_list.extend(custom_stop_list)\n",
    "filtered_chunk = [token for token in tokenized_text[0] if token not in stopword_list]\n",
    "print(filtered_chunk[:50])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That looks more like it. We've successfully massaged out all the punctuation in this initial chunk, though further situations like this might come up later on. The lesson here is that textual data is messy, and even the most sophisticated natural language processing setups require a good deal of massaging and careful modification in light of particular situations.  Each text presents its own set of problems, and only through familiarity with your objects of study will you know what exactly needs to be accounted for. In short, there is no substitute for knowing your corpus. But methods like this can help you control what is in your texts by the time you get to analyzing them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why you might want to leave stopwords in\n",
    "\n",
    "There are good reasons why you might actually want to leave those stopwords in for your analysis. If these words are statistically the most common in English-language texts, then they must serve as meaningful points of comparison among many different texts. Put another way, any given two authors might take different objects of study in their texts. Filtering out common words can give a better sense of their particular interests. But the common words that they share might give a good sense of the particulars of their own literary style, of the ways in which they write. And stopwords might be especially necessary for more advanced kinds of [machine learning](https://towardsdatascience.com/why-you-should-avoid-removing-stopwords-aa7a353d2a52). Let's use this principle to compare Jacob's Room with The Voyage Out. Even though both these texts are by Virginia Woolf, they have very different styles. Let's see if these are represented by word counts, with or without stopwords. In what follows, we'll compare the texts in two ways, first with stopwords taken out and second using _only_ the stopwords."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=====\n",
      "Comparison of Texts with Stopwords Excluded\n",
      "The Voyage Out\n",
      "[(\"'s\", 1007), ('said', 874), ('one', 801), ('rachel', 579), (\"n't\", 513), ('mrs.', 437), ('like', 392), ('helen', 392), ('could', 380), ('people', 379)]\n",
      "Jacob's Room\n",
      "[('said', 425), (\"'s\", 411), ('jacob', 390), ('one', 291), ('mrs.', 225), ('like', 165), ('would', 150), ('little', 137), ('mr.', 134), ('flanders', 128)]\n",
      "=====\n",
      "Comparison of Stopwords in the texts\n",
      "The Voyage Out\n",
      "[(',', 10332), ('the', 7209), ('.', 7193), ('and', 4478), ('of', 3723), ('to', 3640), ('a', 2980), (\"''\", 2866), ('``', 2590), ('she', 2548), ('in', 2249), ('was', 2113), ('her', 1979), ('he', 1722), ('that', 1667)]\n",
      "Jacob's Room\n",
      "[(',', 4604), ('the', 3920), ('.', 2964), ('and', 1816), ('of', 1330), ('a', 1144), ('to', 1016), ('in', 997), (\"''\", 939), ('``', 805), (';', 790), ('was', 686), ('her', 605), ('he', 582), ('it', 574)]\n"
     ]
    }
   ],
   "source": [
    "# store the filenames of both texts\n",
    "\n",
    "fn_one = 'corpus/woolf/1915_the_voyage_out.txt'\n",
    "fn_two = 'corpus/woolf/1922_jacobs_room.txt'\n",
    "\n",
    "# read in the text of the_voyage_out and assign it a variable text_voyage\n",
    "\n",
    "with open(fn_one, 'r') as fin:\n",
    "    text_voyage = fin.read()\n",
    "\n",
    "# read in the text of jacobs_room and assign it a variable text_jacob\n",
    "    \n",
    "with open(fn_two, 'r') as fin:\n",
    "    text_jacob = fin.read()\n",
    "\n",
    "# read in the English-language stopwords list provided by nltk and store it in \n",
    "# a variable stopword_list\n",
    "\n",
    "stopword_list = nltk.corpus.stopwords.words('english')\n",
    "stopword_list.extend(string.punctuation)\n",
    "custom_stop_list = [\"''\", \"``\", '``','--']\n",
    "stopword_list.extend(custom_stop_list)\n",
    "# use nltk's built-in tokenizer to get a list of the tokens in each text. store\n",
    "# these in variables.\n",
    "\n",
    "tokens_voyage = [token.lower() for token in nltk.word_tokenize(text_voyage)]\n",
    "tokens_jacob = [token.lower() for token in nltk.word_tokenize(text_jacob)]\n",
    "\n",
    "\n",
    "# look at the tokens we've got and filter out stopwords by comparing tokens against\n",
    "# the stored list of stopwords.\n",
    "\n",
    "stop_tokens_voyage = [token for token in tokens_voyage if token in stopword_list]\n",
    "stop_tokens_jacob = [token for token in tokens_jacob if token in stopword_list]\n",
    "\n",
    "# create frequency distributions of the top tokens in each text. then print out\n",
    "# the ten most common tokens. since we previously filtered, these lists will contain\n",
    "# stopwords only.\n",
    "\n",
    "stop_freq_voyage = nltk.FreqDist(stop_tokens_voyage)\n",
    "stop_freq_jacob = nltk.FreqDist(stop_tokens_jacob)\n",
    "\n",
    "\n",
    "tokens_voyage = [token for token in tokens_voyage if token not in stopword_list]\n",
    "tokens_jacob = [token for token in tokens_jacob if token not in stopword_list]\n",
    "\n",
    "freq_voyage = nltk.FreqDist(tokens_voyage)\n",
    "freq_jacob = nltk.FreqDist(tokens_jacob)\n",
    "\n",
    "\n",
    "print('=====')\n",
    "print('Comparison of Texts with Stopwords Excluded')\n",
    "print('The Voyage Out')\n",
    "print(freq_voyage.most_common(10))\n",
    "print(\"Jacob's Room\")\n",
    "print(freq_jacob.most_common(10))\n",
    "\n",
    "print('=====')\n",
    "print('Comparison of Stopwords in the texts')\n",
    "print('The Voyage Out')\n",
    "print(stop_freq_voyage.most_common(15))\n",
    "print(\"Jacob's Room\")\n",
    "print(stop_freq_jacob.most_common(15))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On first blush, for one, it's clear that The Voyage Out is much longer than Jacob's Room. You get a clue to that by looking at the word counts in general or even just the number of different punctuation marks. In the results that exclude stopwords, note that Jacob's Room has a male name in the top ten most common words, while The Voyage Out does not. On the one hand, this makes sense - Jacob's Room is at least nominally about a character named Jacob. But this character is presented in an elliptical way, primarily through the descriptions and recollections of other characters. Looking at the results that are solely based on stopwords, we can see this reflected to some degree. The stopword counts for Jacobs Room are a bit more equivalent in gendered pronouns, which give a different picture of gender prominence in the novel than by just noticing the frequent use of Jacob's name. You could use this information in conjunction with other metrics about the text to make larger arguments about the representation of character and gender. \n",
    "\n",
    "We could combine this sort of analysis with one that takes into account punctuation use to get a better sense of an author's particular style. But, on first read, there doesn't actually look to be that much difference in the texts when looking solely at stopwords. This might be interesting, as upon _reading_ them Jacob's Room feels like a significant departure in style. We might ask why that style difference is not represented at the level of the word. These counts can be difficult to compare, though, because it can be hard to compare two lists of counts that overlap as much as they do when considering stopwords. Some knowledge of statistics could help us to pursue this question further."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}