lab4.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:3cb56b9317872b9a39042efd82f3a2fe16bca549b8cf0764f764f7abd1f4c6f1"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Natural Language Parsing"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "In this lab, we'll explore acquisition of semi-structured data from a web service, browsing the data and then parsing free-text content. \n",
      "\n",
      "The data come from wikipedia which has a very simple REST API. While the focus is on natural language parsing, we'll also explore JSON and Wikipedia's own markup format. \n",
      "\n",
      "We'll be using network access for this assignment, so MAKE SURE YOUR LAPTOP NETWORK IS UP before starting the VM. \n",
      "\n",
      "Once your VM is up and running, you can download this notebook file by clicking on icon at the top right of this page. Create a directory ~/labs/lab4 to hold it. "
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Stanford Parser Setup"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "First we need to install some parsing tools. Download the Stanford parser from <a href=\"https://bcourses.berkeley.edu/courses/1267848/files/50829945/download?wrap=1\">here</a>. If your network is not working, download from a browser on your host machine and then use drag-and-drop.\n",
      "\n",
      "Either way, you can put the parser in your \"Downloads\" directory. Unpack it with\n",
      "<pre>\n",
      "tar xvzf stanfordparser.tar.gz\n",
      "</pre>\n",
      "\n",
      "and then move it to the /opt directory with \n",
      "<pre>\n",
      "sudo mv StanfordParser /opt\n",
      "</pre>\n",
      "\n",
      "It will be helpful to have links to the parser scripts from your bin directory. If you havent already, create a directory ~/bin. Then \n",
      "<pre>\n",
      "cd ~/bin\n",
      "ln -s /opt/StanfordParser/lexparser.sh lexparser.sh\n",
      "ln -s /opt/StanfordParser/lexparser-gui.sh lexparser-gui.sh\n",
      "ln -s /opt/StanfordParser/dependencyviewer/dependencyviewer.sh dependencyviewer.sh\n",
      "</pre>\n",
      "\n",
      "These files will be in your path the next time you login. You can logout from the start button at the top right of the VM window. Then log back in again. "
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Mediawiki Parser Setup"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The mediawiki parser is memorably named \"mwparserfromhell\". To install it with a working network, all you need to do is\n",
      "<pre>\n",
      "sudo pip install mwparserfromhell\n",
      "</pre>\n",
      "\n",
      "if your network is not working, copy the package source from <a href=\"https://bcourses.berkeley.edu/courses/1267848/files/51008623/download?wrap=1\">here</a>. Untar it, which gives a directory tree starting at \"usr\". Traverse the directories until you find the \"mwparserfromhell\" direcory. Copy that directory to your python modules directory with:\n",
      "<pre>\n",
      "sudo cp -r mwparserfromhell /usr/local/lib/python2.7/dist-packages\n",
      "</pre>"
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Accessing the Wikipedia Web API"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Start by looking over the <a href=\"http://www.mediawiki.org/wiki/API:Main_page\">Mediawiki API documentation</a> which describes Wikipedia's RESTful API. \n",
      "\n",
      "The code below implements an API call with options:\n",
      "* format=json to receive JSON data\n",
      "* action=query to query Wikipedia content\n",
      "* titles=string to specify a list of page titles to search for\n",
      "* prop=revision to return the revisions of the page\n",
      "* rvprop=content to return the full page content\n",
      "\n",
      "We'll start with the title search string 'parsing' to retrieve the page about parsing."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": true,
     "input": [
      "import requests\n",
      "title='parsing'\n",
      "response = requests.get(\"http://en.wikipedia.org/w/api.php?format=json&action=query&titles=\"+str(title)+\"&prop=revisions&rvprop=content\")\n",
      "response"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The response object is an HTTP GET response. It turns out the requests package contains a json interpreter, which we can invoke as:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": true,
     "input": [
      "jsondata = response.json()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "If you dont have a working network, copy the file <a href=\"https://bcourses.berkeley.edu/courses/1267848/files/51028371/download?wrap=1\">parsing.json</a> into your ~/labs/lab4 directory. Then you can load it with:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import json\n",
      "# fp=open('/home/datascience/labs/lab4/parsing.json','r')\n",
      "# jsondata=json.load(fp);\n",
      "# fp.close()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The JSON object is hierarchically structured. To view it, its helpful to define a couple of helper routines:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": true,
     "input": [
      "import json\n",
      "def pretty(jdata):\n",
      "    str = json.dumps(jdata, sort_keys=True, indent=4).decode('string_escape');\n",
      "    return str\n",
      "\n",
      "def saveas(sdata, fname):\n",
      "    f = open(fname,'w');\n",
      "    f.write(sdata);\n",
      "    f.close();"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The first routine converts the JSON to a carefully-formatted string. The second writes a string to a file. We can use them together to save the JSON data to a better format for viewing."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": true,
     "input": [
      "saveas(pretty(jsondata), '/home/datascience/labs/lab4/'+title+'.json')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now open the file '/home/datascience/labs/lab4/parsing.json' by right-clicking on it and using \"open-with\" with emacs or gvim. Note the structure.\n",
      "\n",
      "The JSON parser converts JSON data nodes and lists of nodes. The nodes are represented as Python \"Dict\" objects, and the lists are Python lists. Each Dict maps the names of the nodes children to their values. We can query the type of each node using the \"type\" function. For each Dict, we can enumerate the keys using the keys() method. In this way we can explore the JSON tree (although its much quicker to eyeball it from the JSON file we just saved). But anyway we can browse with:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": true,
     "input": [
      "type(jsondata)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": true,
     "input": [
      "jsondata.keys()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "which is a list of just one string (a unicode string, hence the \"u\" prefix). We can then extract that node with"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": true,
     "input": [
      "jsondata['query']"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "and continue exploring:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": true,
     "input": [
      "type(jsondata['query'])"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": true,
     "input": [
      "jsondata['query'].keys()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "From the pretty-printed file, we know we are looking for the 'pages' child, which has a page id number. We dont know what this number is, so we cant use it as a key. But instead we can use the 'values()' method on the dictionary to get a list of all the nodes below it. We only need one page, so we take the first of those."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": true,
     "input": [
      "jsondata['query']['pages'].values()[0]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Continue down the tree, next to the \"revisions\" node. This time, take the *last* revision in the list."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "> TODO: write code below to extract the actual content of the article from the last revision in the list, then uncomment and run it. "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": true,
     "input": [
      "# content = "
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": true,
     "input": [
      "content"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The content is now a text string in Mediawiki's own format. To make sense of it we can use the mwparserfromhell (MWPH for short)."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": true,
     "input": [
      "import mwparserfromhell as mwph\n",
      "wikicode = mwph.parse(content)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "MWPH supports a variety of methods to explore mediawiki content. The main class is the Wikicode class, which is the type returned by mwph.parse(). e.g. try"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": true,
     "input": [
      "wikicode.filter_comments()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": true,
     "input": [
      "wikicode.filter_headings()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": true,
     "input": [
      "wikicode.filter_wikilinks()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "But since we want to parse the english text from the article, we want to ignore all these metadata. MWPH has a method to do this:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": true,
     "input": [
      "text = wikicode.strip_code()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": true,
     "input": [
      "text"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "This data is clean enough now that we can save it for parsing:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": true,
     "input": [
      " saveas(pretty(text), '/home/datascience/labs/lab4/'+title+'.txt')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Running the Stanford Parser"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "From a terminal window, type\n",
      "<pre>\n",
      "lexparser-gui.sh\n",
      "</pre>\n",
      "\n",
      "This brings up a GUI interface to the Stanford parser. To use it, click on \"Load Parser\" which brings up a file selection dialog. Navigate to\n",
      "\n",
      "<pre>\n",
      "/opt/StanfordParser/stanford-parser-3.4.1-models.jar\n",
      "</pre>\n",
      "\n",
      "and open it.\n",
      "\n",
      "Then you will see a list of parsers to use. Select\n",
      "\n",
      "<pre>\n",
      "englishPCFG.ser.gz\n",
      "</pre>\n",
      "\n",
      "You're now ready to parse some text!\n",
      "\n",
      "Click on the \"Load File\" button, and browse to the lab4 directory and load the parsing.txt file. Click on \"Parse\" to parse the current sentence (highlighted in yellow). "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "> TODO: look at the very first sentence. Did it parse correctly? Explain."
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Content Analysis"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now lets try to analyze some content from Wikipedia. To make our lives simpler, we'll use a simplified english version of wikipedia. Change the URL in the first code box in this file to:\n",
      "<pre>\n",
      "simple.wikipedia.org\n",
      "</pre>\n",
      "and change the query title to 'cat'. Rerun all the cells above. This should produce a file \"cat.txt\" in the lab4 directory. Load that file into the parser, and parse some of the sentences. \n",
      "\n",
      "If you cant access the network, download the file <a href=\"https://bcourses.berkeley.edu/courses/1267848/files/51028369/download?wrap=1\">cat.json</a> into your lab4 directory, and repeat the commands used earlier to load the parsing.json file. \n",
      "\n",
      "We'll now convert the parser output to XML, so we can process it further. Find the script\n",
      "<pre> \n",
      "/opt/StanfordParser/lexparser.sh\n",
      "</pre>\n",
      "and edit it so that its outputFormat is:\n",
      "<pre>\n",
      "-outputFormat \"xmlTree\"\n",
      "</pre>\n",
      "and add a new option:\n",
      "<pre>\n",
      "-outputFormatOptions \"xml\"\n",
      "</pre>\n",
      "save the new script as \n",
      "<pre>\n",
      "parsetoxml.sh\n",
      "</pre>\n",
      "and create an alias to it in your ~/bin directory. Now run from your lab4 directory\n",
      "<pre>\n",
      "parsetoxml.sh cat.txt > cat.xml\n",
      "</pre>\n",
      "you're ready now to analyze the cat data. We'll use Python's builtin ElementTree parser. "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from lxml import etree\n",
      "parser = etree.XMLParser(recover=True)\n",
      "tree = etree.parse('/home/datascience/labs/lab4/cat.xml',parser)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We can examine the root of this tree:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "root=tree.getroot()\n",
      "root.tag"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "len(root)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "root[0].tag"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "i.e. we have found the first sentence. The xmlTree representation is a little tricky however, as POS tags are stored as attributes of nodes rather than node tags. To get to the actual root node, we need to dig a little deeper (and we'll use the second sentence which is a bit more conventional):\n"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "root[1][0][0].attrib['value']\n",
      "\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "going down one level gets us to the actual sentence node:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "s=root[6][0][0][0]\n",
      "s.attrib['value']"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "and to get its children we can do:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "s[:]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "This is not too helpful, because the node types are hidden in the value attribs of these nodes. To see them, we can use a python anonymous function and map it over the list. "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "map(lambda (x): x.attrib['value'], s[:])"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now lets see if we can find sentences starting with noun phrases containing a given noun. The final function supports a flexible syntax (similar to xpath) for locating elements of given type or attributes. A slash \"/\" is like a directory specifier, and defines a child node. A double slash \"//\" specifies *any* descendent, child, grandchild, great-grandchild etc. The \"node[@value='NP']\" specifies a node with the given attribute value."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "agent = s.findall(\"./node[@value='NP']//node[@value='NN']//leaf[@value='cat']\")\n",
      "agent"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "finds all the nodes starting with an 'NP' child of s, and having a 'NN' node above a leaf with 'cat' value. \n",
      "\n",
      "We can similarly look for a verb in a verb phrase under the root node:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "verb = s.findall(\"./node[@value='VP']//node[@value='VBZ']//leaf[@value='is']\")\n",
      "verb"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Putting these together, we can discover sentences containing a given pair of (agent,action) pairs:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "\n",
      "def printnode(node):\n",
      "    for i in node.findall(\".//leaf\"):\n",
      "        print(\" \" + i.attrib['value']),\n",
      "    print('')\n",
      "\n",
      "def testnode(node, agent, action):\n",
      "    aa = node.findall(\"./node[@value='NP']//node[@value='NN']//leaf[@value='\"+agent+\"']\")\n",
      "    bb = node.findall(\"./node[@value='VP']//leaf[@value='\"+action+\"']\")\n",
      "    if (len(aa) > 0 and len(bb) > 0):\n",
      "        printnode(node)    \n",
      "\n",
      "def agentact(node, agent, action):\n",
      "    testnode(node, agent, action)\n",
      "    snodes = node.findall(\".//node[@value='S']\")\n",
      "    for snode in snodes:\n",
      "        testnode(snode, agent, action)\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "agentact(s, title, 'is')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Next we can map the agentact function across all the sentences in the Wikipedia entry:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "map(lambda (nn): agentact(nn[0][0][0], title, 'is'), root)\n",
      "[]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "> TODO: Create a new testnode2 function that captures some more facts about cats. Put it in the cell below, and then evaluate it. "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": true,
     "input": [
      "def agentact2(node, agent, action):\n",
      "    testnode2(node, agent, action)\n",
      "    snodes = node.findall(\".//node[@value='S']\")\n",
      "    for snode in snodes:\n",
      "        testnode2(snode, agent, action)\n",
      "        \n",
      "map(lambda (nn): agentact2(nn[0][0][0], title, 'is'), root)\n",
      "[]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Lab Responses"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Remember to fill out the lab responses <a href=\"https://bcourses.berkeley.edu/courses/1267848/quizzes/1790249\">here</a>."
     ]
    }
   ],
   "metadata": {}
  }
 ]
}