Parsing structured data solution.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Parsing structured data\n",
    "## Week 6\n",
    "Through Guttag chapter 13.\n",
    "\n",
    "XML and JSON are formats you'll use often when dealing with textual data. Many texts are encoded in XML or one of its specific flavors, such as TEI. XML is much the older of the two, and by far the more powerful. It has a long history in print and electronic publishing, and is still commonly used in the production of edited texts, complex websites, and certain computational projects. It's a descendant of SGML and a proper superset of HTML.\n",
    "\n",
    "JSON, on the other hand, is simpler, lighter weight, newer, and generally easier to compute with. It's the de facto format for data exchange on the web.\n",
    "\n",
    "This exercise asks you to ingest, parse, and work with structured data in both XML and JSON formats.\n",
    "\n",
    "### XML\n",
    "\n",
    "We'll begin with XML, the more difficult of the two. XML looks a lot like HTML, if you've seen that. The main difference is that, where HTML consists of a fixed set of allowable tags, XML tags can be defined arbitrarily according to a spec of the user's choosing. In practice, we won't do much with this, but you'll often find yourself dealing with other projects' arbitrary XML use. For our purposes, all that matters is that you can figure out -- mostly by human examination -- which tags are used to encode what information.\n",
    "\n",
    "Here's an example of some XML:\n",
    "\n",
    "```\n",
    "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n",
    "<TEI xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"\n",
    "     xmlns=\"http://www.tei-c.org/ns/1.0\"\n",
    "     xsi:schemaLocation=\"http://www.tei-c.org/ns/1.0 http://www.dlib.indiana.edu/lib/xml/tei/p5/general.xsd\"\n",
    "     xml:id=\"VAC5615\">\n",
    "   <teiHeader>\n",
    "      <fileDesc>\n",
    "         <titleStmt>\n",
    "            <title>Growler's Income Tax </title>\n",
    "            <author>Arthur, T. S. (1809-1885)</author>\n",
    "         </titleStmt>\n",
    "         ...\n",
    "      </fileDesc>\n",
    "   </teiHeader>\n",
    "</TEI>\n",
    "```\n",
    "\n",
    "Exciting, no? This is the beginning of the file you'll use in this exercise. You can see that it begins with information about the XML version and schema used (it's TEI, just so you know). Then there's a header, which contains a title statement, which contains title and author info, etc. This is how XML, HTML, and other structured markup languages work: each element contains one or more others. These are often referred to as parent and child elements. Every child has exactly one parent, but parents many have multiple children and may be children themselves. In the above example,  `<title>` is the child of `<titleStmt>`, which is in turn the child of `<fileDesc>`. `<title>` and `<author>` are siblings.\n",
    "\n",
    "#### Beautiful Soup\n",
    "\n",
    "Parsing XML in the general case is a pain. You absolutely *do not* want to build a general-purpose XML parser. But you don't need to; other people have done it. One of the most widely used XML parsers for Python is [Beautiful Soup](http://www.crummy.com/software/BeautifulSoup/). (OK, technically Beautiful Soup is a wrapper for other parsers, but whatever.)\n",
    "\n",
    "Review two sources of information about BeatutifulSoup: The [documentation](http://www.crummy.com/software/BeautifulSoup/bs4/doc/) and a [tutorial](http://programminghistorian.org/lessons/intro-to-beautiful-soup) from the Programming Historian. Note that both of these sources cover installation, which is unnecessary if you're using Anaconda (because BeautifulSoup is included therein). Note, too, that the Programming Historian tutorial is written in Python 2, so code used there won't transfer directly to Python 3. Everything you need to know to complete this portion of the exercise is contained in the \"Quick Start\" section of the BS docs.\n",
    "\n",
    "Then download a copy of [Growler's Income Tax](https://github.com/wilkens/course-exercises-f15/blob/master/growler.xml) by T.S. Arthur in TEI-XML format. Save this file (not the GitHub page linked here; just the raw file) in the same directory in which your iPython notebook is saved.\n",
    "\n",
    "There are a lot of things you can do with BeautifulSoup, all of which depend on parsing an XML or HTML document and then searching or modifying the parsed results. In the present case, you'll need to extract four pieces of information from the XML file in question:\n",
    "\n",
    "1. The full, library-style author record, in the form \"LastName, FirstName (born-died)\".\n",
    "2. The number of paragraphs in the body of the text.\n",
    "3. The number of words in the body of the text.\n",
    "4. The full, plain-text (no XML tags) content of the embedded epistle.\n",
    "\n",
    "To do this, you'll need to know how the XML file is structured. Spend some time looking at it, whether in the GitHub display or in a text editor (but probably not Word, which tends to mess with markup). Each piece of information above can be extracted with one line of code, once you've imported and parsed the XML, though you may certainly use more than one line for each if you prefer. You'll use the `find`, `find_all`, and `get_text` methods of BeautifulSoup, plus other things you've learned as necessary. Potential gotcha: BeautifulSoup converts all tag names to lowercase.\n",
    "\n",
    "Enter your code in the cell or cells below. I've given you the code to import BeuatifulSoup and load the XML file, though you may need to change the file name and location to match whatever you've used. Your code should print the four pieces of information listed above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "with open('growler.xml', 'r') as f:\n",
    "    soup = BeautifulSoup(f, \"xml\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Uncomment the lines below to print XML and plain-text versions of the text. No need to do this, but it can be useful to verify that the import worked correctly, as well as to see how you go about extracting information from the parsed data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "## Pretty-print the XML, to see that import worked\n",
    "# print(soup.prettify())\n",
    "\n",
    "## Print the full plain-text version\n",
    "# print(soup.body.get_text())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now extract the information required in the problem spec ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Author: Arthur, T. S. (1809-1885)\n",
      "Wordcount: 1670\n",
      "Body paragraphs: 40\n",
      "Letter content: \n",
      "\n",
      "\"Philadelphia, Sept., 1863.\n",
      "\n",
      "\"RICHARD GROWLER, ESQ.,\n",
      "\"To JOHN M. RILEY. Dr.\n",
      "\"Collector Internal Revenue for the 4th District of \n",
      "Pennsylvania. Office 427 Chestnut St.\n",
      "\n",
      "\n",
      "\"For Tax on Income, for the year 1862 as per return made to the Assessor of the District, $43,21.\n",
      "\n",
      "\"Rec'd payment,\n",
      "\"JOHN M. RILEY, Collector.\"\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Extract the required information\n",
    "author = soup.author.get_text()\n",
    "wc = len(soup.body.get_text().split())\n",
    "pars = len(soup.body.find_all('p'))\n",
    "letter = soup.find(type=\"letter\").get_text()\n",
    "print(\"Author:\", author)\n",
    "print(\"Wordcount:\", wc)\n",
    "print(\"Body paragraphs:\", pars)\n",
    "print(\"Letter content:\", letter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### JSON\n",
    "\n",
    "Here we'll use the HathiTrust Research Center's [extracted features dataset](https://sharc.hathitrust.org/features), which contains page-level, part-of-speech-tagged word counts from 4.8 million public domain volumes held by the HatiTrust digital library. We'll work with a single book, which happens to be volume 4 of Bret Harte's *Collected Works*. You can see (and search) the [full-text scanned copy](http://babel.hathitrust.org/cgi/pt?id=mdp.39076000600655) via HT's reading interface.\n",
    "\n",
    "Your task is to determine the most frequently occurring noun (singular or plural, common or proper) on the 116th page of this text. For reference, you may want to consult [the relevant page image](http://babel.hathitrust.org/cgi/pt?id=mdp.39076000600655;view=1up;seq=116) in the HT reader. Note that we're interested in the page with sequence number 116, which isn't the same thing as the one with printed page number 116.\n",
    "\n",
    "This is tricky. Or, more accurately, it's conceptually tedious. The JSON loader reads in the JSON data as a multiply nested dictionary of dictionaries, plus a list of dictionaries corresponding to each page in the volume. What you need to do is walk that list of page-level dictionaries, looking for page 116, then iterating over the tokens on that page, selecting the nouns, and keeping track of which one occurs most often.\n",
    "\n",
    "For reference, the structure of the data is as follows:\n",
    "\n",
    "    features\n",
    "        pages\n",
    "            header\n",
    "            footer\n",
    "            [some info about the page, including the 'seq' key for page sequence number]\n",
    "            body\n",
    "                tokenPosCount\n",
    "                    [actual token, i.e., an individual word form]\n",
    "                        [PoS tag, e.g., 'NNP' for proper noun]\n",
    "                            [count, e.g., 3]\n",
    "                            \n",
    "Each one of those levels is the key for a dictionary, which dictionary contains the keys for the level below it, etc. So, when you read in the JSON data, you can address it the way you would any other dictionary, using the relevant keys one after another,  like so:\n",
    "    \n",
    "    data['features']['pages']\n",
    "    \n",
    "... which will yield a list of dictionaries, each containing the feature counts for one page of the volume.\n",
    "\n",
    "Here's the algorithm, in English: Use a `for` loop to iterate over the page-level entries, looking for one with key `'seq'` and value `'00000116'` (note that the value is a string, not an integer). If you've found page 116, iterate over the `'tokenPosCount'` dictionary within the `'body'` dictionary for that page. For each token in that dictionary, examine the associated part of speech tag. If it's a noun (that is, the PoS tag is one of `NN`, `NNS`, `NNP`, or `NNPS`), then record the value associated with the PoS key, which is the count of occurrences of that word with that PoS tag on that page. If it's the largest count yet seen, record both the word and the count. When you've finished iterating over the whole volume, print the word and the count.\n",
    "\n",
    "You can download the [HTRC JSON data](https://raw.githubusercontent.com/wilkens/course-exercises-f15/master/harte.json) from GitHub. It's too long for GitHub to display prettily, so that link takes you directly to the raw JSON. Save the file to your iPython notebook working directory as above.\n",
    "\n",
    "I've given you some code to get started. FYI, my answer requires about 10 additional lines of code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# medium NN 1\n",
      "# discovery NN 2\n",
      "# paper NN 4\n",
      "Most common noun on page 116: \" paper \" occurs 4 times\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "with open('harte.json', 'r') as g:\n",
    "    data = json.load(g)  # Parse the JSON input\n",
    "    max_count = 0        # Keep track of largest seen noun count\n",
    "    answer = ''          # Keep track of most frequently occurring noun\n",
    "    for i in data['features']['pages']:  # i is a dictionary of page-level data\n",
    "        if i['seq'] == '00000116':\n",
    "            for j in i['body']['tokenPosCount']:  # j is a dictionary of words on a page\n",
    "                for key in i['body']['tokenPosCount'][j].keys():  # key is a PoS tag\n",
    "                    if key in ('NN', 'NNS', 'NNP', 'NNPS'):\n",
    "                        count = int(i['body']['tokenPosCount'][j][key])\n",
    "                        if count > max_count:\n",
    "                            max_count = count\n",
    "                            answer = j\n",
    "                            print('#', j, key, i['body']['tokenPosCount'][j][key])  # Print each successive maximum, FWIW\n",
    "print(\"Most common noun on page 116: \\\"\", answer, \"\\\" occurs\", max_count, \"times\")  # Print the answer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}