lectures/8 Parsing text/8c Tokenizing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "89e060b8",
   "metadata": {},
   "source": [
    "# Tokenizing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08ee7a19",
   "metadata": {},
   "source": [
    "## Lecture objectives\n",
    "1. Introduce stopwords and how to remove them\n",
    "2. Demonstrate how to tokenize (split) text into words and sentences\n",
    "3. Explore how to lemmatize words"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06e48689",
   "metadata": {},
   "source": [
    "Let's start by loading in the text file that we used in the previous lecture."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "687ac868",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('../scratch/eirtext.txt','r') as f:\n",
    "    eirtext = f.read()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8025cbb",
   "metadata": {},
   "source": [
    "## Stopwords\n",
    "Recall our word counts from the previous lecture. Many of the most common words – `and`, `of`, etc. - are not particularly informative. This type of analysis might be useful in some applications, but here, we really need to push further.\n",
    "\n",
    "Let's use the `nltk` library to get rid of these common words that don't have a substantive meaning. They are called *stop words* in natural language processing jargon. \n",
    "\n",
    "`nltk` is a mammoth library, and has lots of submodules. We'll use the tokenize functions (more on this in a moment) and `stopwords` submodules for now.\n",
    "\n",
    "The first time we use them, we have to download the \"corpus\". If you don't do this, you'll get a helpful error message reminding you of this. See http://www.nltk.org/nltk_data/ for all the corpora that you can download."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "235d26dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.tokenize import word_tokenize, sent_tokenize\n",
    "\n",
    "# We only need to do this once \n",
    "nltk.download('stopwords')\n",
    "nltk.download('punkt')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5db7ad98",
   "metadata": {},
   "source": [
    "Let's take a look at the stopwords. `stopwords.words` just gives us a list of words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "381ab022",
   "metadata": {},
   "outputs": [],
   "source": [
    "type(stopwords.words('english'))\n",
    "print(stopwords.words('english')[:20])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9590a961",
   "metadata": {},
   "source": [
    "In several languages, too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "44b4521c",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(stopwords.words('spanish')[:10])\n",
    "print(stopwords.words('arabic')[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be4f4f03",
   "metadata": {},
   "source": [
    "## Tokenizing \n",
    "Tokenizing functions are essentially splitting functions. (Often, you might be able to use the `split()` function as we did in the previous lecture, but the tokenizing functions are more robust.)\n",
    "\n",
    "For example, `sent_tokenize` splits into a list of sentences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f985c31e",
   "metadata": {},
   "outputs": [],
   "source": [
    "sentences = sent_tokenize(eirtext)\n",
    "print(sentences[2])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c7097ca",
   "metadata": {},
   "source": [
    "For our purposes, we want to split into words. We can use `word_tokenize`. This should give us similar results to the `split()` function earlier, but it's a bit more robust.\n",
    "\n",
    "Before we count the words, let's also use `regex` to drop the digits, punctuation, and other non-words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f27a932",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import re\n",
    "\n",
    "# this is exactly the same function from the previous lecture\n",
    "def countWords(wordlist):\n",
    "    counts = {} \n",
    "    \n",
    "    for word in wordlist:\n",
    "        lword = word.lower()\n",
    "        if lword in counts:\n",
    "            counts[lword] +=1\n",
    "        else:\n",
    "            counts[lword] = 1\n",
    "\n",
    "    df = pd.DataFrame.from_dict(counts, orient='index', columns=['word_count'])\n",
    "    df.sort_values('word_count', ascending=False, inplace=True)\n",
    "    df.index.name = 'word'\n",
    "    \n",
    "    return df\n",
    "\n",
    "wordlist = word_tokenize(re.sub(r\"[^A-z\\s]\", \"\", eirtext))\n",
    "df = countWords(wordlist)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61008831",
   "metadata": {},
   "source": [
    "Now let's drop the stopwords from our counts. \n",
    "\n",
    "Remember that `stopwords.words` gives a list of words. So let's use the pandas `drop()` function to drop all of those words from the index. \n",
    "\n",
    "We add the `errors='ignore'` argument because not all of our stopwords will be in our DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e1813cfb",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.drop(index=stopwords.words('english'), errors='ignore', inplace=True)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "284819ec",
   "metadata": {},
   "source": [
    "## Lemmatizing\n",
    "Finally, we might want to *lemmatize* the words. We saw that process used in the [Brinkley & Stahmer](https://journals.sagepub.com/doi/abs/10.1177/0739456X21995890) paper. Lemmatization groups words with the same stem, e.g. `highway` and `highways`, or `constructing` and `construction`, through reducing them to their *root*.\n",
    "\n",
    "`nltk` has a built-in function for that - `PorterStemmer`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "074863c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.stem import PorterStemmer\n",
    "ps = PorterStemmer()\n",
    "print(ps.stem('construction'))\n",
    "print(ps.stem('highways'))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8a74fa0",
   "metadata": {},
   "source": [
    "Even if it doesn't know the (made-up) word, the stemmer takes a decent guess."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4aaf2b24",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(ps.stem('housingelementifcation'))  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08bd64ac",
   "metadata": {},
   "source": [
    "Let's add this to our function, and call our new function `countStems`. It's just one extra line."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "81e14d56",
   "metadata": {},
   "outputs": [],
   "source": [
    "def countStems(wordlist):\n",
    "    counts = {} \n",
    "    \n",
    "    for word in wordlist:\n",
    "        lword = word.lower()\n",
    "        # This is the extra line\n",
    "        lword = ps.stem(lword)\n",
    "        \n",
    "        if lword in counts:\n",
    "            counts[lword] +=1\n",
    "        else:\n",
    "            counts[lword] = 1\n",
    "\n",
    "    df = pd.DataFrame.from_dict(counts, orient='index', columns=['word_count'])\n",
    "    df.sort_values('word_count', ascending=False, inplace=True)\n",
    "    df.index.name = 'word'\n",
    "    df.drop(index=stopwords.words('english'), errors='ignore', inplace=True)\n",
    "\n",
    "    return df\n",
    "\n",
    "df = countStems(wordlist)\n",
    "df.head(10)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4dc1d53",
   "metadata": {},
   "source": [
    "Whether the stems are more useful than the original words is obviously a matter for your specific task.\n",
    "\n",
    "So now we've got the tools to bring in some text to a useful form. In the next module, we'll interpret the text using topic modeling and sentiment analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b453f388",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<h3>Key Takeaways</h3>\n",
    "<ul>\n",
    "  <li>Before analyzing a text, you will probably need to do clean-up such as removing stopwords, converting to lower case, and possibly lemmatizing the words.</li>\n",
    "</ul>\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}