lectures/8 Parsing text/8a Reading PDFs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "89e060b8",
   "metadata": {},
   "source": [
    "# Module 8. Parsing text\n",
    "# Reading PDFs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08ee7a19",
   "metadata": {},
   "source": [
    "## Lecture objectives\n",
    "1. Demonstrate how to load in text data from PDFs\n",
    "2. Show how to clean and simplify text data using `regex` "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06e48689",
   "metadata": {},
   "source": [
    "## Getting text into Python\n",
    "Before we process any text, we need to take a step back and figure out how to get that text into Python. Typically, plans and other policy documents come as PDFs, which are a pain to read. There are dozens of PDF readers for Python, all of which are flawed in different ways. (See some discussions [here](https://towardsdatascience.com/how-to-extract-text-from-pdf-245482a96de7), [here](https://johannesfilter.com/python-and-pdf-a-review-of-existing-tools/) and [here](https://stackoverflow.com/questions/34837707/how-to-extract-text-from-a-pdf-file).) We'll use `pdfminer.six`, which is fairly robust and is easier to install than some alternatives. YMMV.\n",
    "\n",
    "Let's start with an adaptation of the [LA Times analysis of California High-Speed Rail](https://github.com/datadesk/hsr-document-analysis). If you look at their code, they use the `urllib` library to download files. You can do the same but with a couple of extra steps using `requests`. \n",
    "\n",
    "But for now, let's work with just one of their files: [the EIR section on air quality and climate change, for the Bakersfield to Palmdale segment](https://hsr.ca.gov/wp-content/uploads/docs/programs/bakersfield-palmdale/BP_Draft_EIRS_Vol_1_CH_3.3_Air_Quality_and_Global_Climate_Change.pdf). It's in your git repository.\n",
    "\n",
    "We'll read the text using `pdfminer.six`. Its simplest function is `extract_text`. Full documentation is [here]\n",
    "\n",
    "Note that you will often have to experiment with other PDF parsers if you get unintelligible results. `PyPDF2` is another commonly used package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "687ac868",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pdfminer.high_level import extract_text\n",
    "\n",
    "fn = '../data/BP_Draft_EIRS_Vol_1_CH_3.3_Air_Quality_and_Global_Climate_Change.pdf'\n",
    "eirtext = extract_text(fn)\n",
    "print('Text is {} characters long'.format(len(eirtext)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09ee237c",
   "metadata": {},
   "source": [
    "Let's look at a few random extracts. We read the file into a string, so we can use our standard string slicing syntax.\n",
    "\n",
    "For example, let's look at 1,000 characters starting at the 200,000th character."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6e79b317",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(eirtext[199999:200999])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b49730a5",
   "metadata": {},
   "source": [
    "And another slice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6af34f05",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(eirtext[400000:401000])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a1f2332",
   "metadata": {},
   "source": [
    "## Cleaning up the text\n",
    "So we've got a bunch of text in, but clearly the formatting leaves something to be desired. In particularly, there are a lot of random line breaks. Let's use `regex` to convert all whitespace (spaces, tabs (`\\t`), and newlines (`\\n` or `\\r\\n`) to a single space. \n",
    "\n",
    "`regex` is short for \"regular expression,\" and is essentially a pattern matching tool for text. Think of it as a souped-up version of `replace`. \n",
    "\n",
    "`regex` is extremely powerful and has an extremely unfriendly syntax. But there are thousands of examples online. [Here's a good place to start](https://regexone.com/) if you want to explore more. And [this website](https://regex101.com) helps you test and debug your expressions.\n",
    "\n",
    "Let's look at an example – `r\"\\s+\"`:\n",
    "- The `r` tells Python that what follows is a \"raw string,\" and thus the `\\` character should be interpreted literally\n",
    "- `\\s` matches whitespace\n",
    "- `+` matches multiple occurences\n",
    "\n",
    "So basically, we are matching all whitespace, however long.\n",
    "\n",
    "Let's then use `re.sub` to replace that whitespace. The second argument is what we replace our matched substrings with. The third argument is the string to apply the substitution to. Note that we have some spaces, some tabs (`\\t`), and some newlines (`\\n`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c1ece90",
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "print(re.sub(r\"\\s+\", \" \", \"HSR\\tis     an\\nexpensive    boondoogle\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9b2ca84-38ec-4845-ab32-d3639ec84ef1",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<strong>Exercise:</strong> What happens if you omit the <strong>+</strong> sign from <strong>r\"\\s+\"</strong>? How can you explain your results?\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc138192",
   "metadata": {},
   "source": [
    "If we omit the `+` and just specify `r\"\\s\"`, we don't match multiple occurences. So 4 spaces are replaced with 4 spaces, rather than a single space. But the tabs and newlines are still converted to spaces."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "03a03838",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(re.sub(r\"\\s\", \" \", \"HSR\\twill     \\ntransform     California\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c89c5a1",
   "metadata": {},
   "source": [
    "I won't pass judgment on the content of either of these claims.\n",
    "\n",
    "Let's apply the `regex` to our text that we pulled out of the EIR"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59d45429",
   "metadata": {},
   "outputs": [],
   "source": [
    "eirtext = re.sub(r\"\\s+\", \" \", eirtext)\n",
    "print(eirtext[200001:201001])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6ba76105",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(eirtext[400001:401001])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b1a66fa",
   "metadata": {},
   "source": [
    "We can also use `regex` to get rid of punctuation, digits, etc. \n",
    "\n",
    "Here:\n",
    "* `[]` means match anything within the brackets\n",
    "* `^` means not\n",
    "* `A-z` is any letter in any case\n",
    "* `\\s` is any whitespace (which is just spaces, since we converted other whitespace like tabs to spaces\n",
    "\n",
    "So `[^A-z\\s]` captures anything that is not a letter or whitespace. \n",
    "\n",
    "Since we might want the punctuation at a later date, let's assign our cleaned text to a new variable, `eirtext_wordsonly`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "73d0572e",
   "metadata": {},
   "outputs": [],
   "source": [
    "eirtext_wordsonly = re.sub(r\"[^A-z\\s]\", \"\", eirtext)\n",
    "eirtext_wordsonly[400001:401001]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94da52e7",
   "metadata": {},
   "source": [
    "Notice that removing some digits, etc. means that we now have extra spaces. For example, `Table 3.3-46 provides a summary` becomes `Table  provides a summary.`\n",
    "\n",
    "So let's use our same process from before to remove duplicate spaces."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e50b438",
   "metadata": {},
   "outputs": [],
   "source": [
    "eirtext_wordsonly = re.sub(r\"\\s+\", \" \", eirtext_wordsonly)\n",
    "eirtext_wordsonly[394890:395200]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9066cc5-a3b6-46b9-96e3-3705d58dd838",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "    <strong>Exercise:</strong> The regex above also removes numbers. Adapt <strong>r\"[^A-z\\s]\"</strong> so that it retains the digits 0 through 9. <em>Hint:</em> Look at the websites linked above for tips on the syntax.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "528504d1-f67b-433a-88b5-24b5aa8db475",
   "metadata": {},
   "source": [
    "Note that `0-9` represents any number. So we can add this to our list of characters that will be matched."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a590d2c-5d90-463b-a55f-acff9a696bfc",
   "metadata": {},
   "outputs": [],
   "source": [
    "eirtext_wordsonly = re.sub(r\"[^A-z0-9\\s]\", \"\", eirtext)\n",
    "# and remove duplicate spaces\n",
    "eirtext_wordsonly = re.sub(r\"\\s+\", \" \", eirtext_wordsonly)\n",
    "eirtext_wordsonly[400001:401001]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca56d529",
   "metadata": {},
   "source": [
    "This looks much better! We now have some clean text to analyze.\n",
    "\n",
    "Let's pause here. We'll save the text to a file, so that we can load it in at the start of the next lecture.\n",
    "\n",
    "Note here that `open` opens the file object `f`. We then write `eirtext` to the file. The `with` syntax helps because it automatically closes the file afterwards."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "46750056",
   "metadata": {},
   "outputs": [],
   "source": [
    "# the encoding keyword is needed on Windows, as it does not write as unicode by default\n",
    "with open('../scratch/eirtext.txt', 'w', encoding=\"utf-8\") as f:\n",
    "    f.write(eirtext)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b453f388",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<h3>Key Takeaways</h3>\n",
    "<ul>\n",
    "  <li>PDFs are difficult to work with. pdfminer is a good starting point, but make sure to inspect your output.</li>\n",
    "  <li>regex is a powerful tool to clean up text, e.g. removing whitespace and punctuation.</li>\n",
    "</ul>\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}