Week-14.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Week 14: XPath\n",
    "\n",
    "So yes, we spent most of last week sorting our heads around how to make Python work with Xpath and discussing XML.  Your assigned readings for last week included the W3C School's XPath tutorials, along with an optional refresher on XML.\n",
    "\n",
    "Let's be clear: even if you have worked with XML before, maybe even taken the metadata class using XML, knowing the precise structure and **names** of the bits and bobs inside of XML will be necessary to wrap your head around what XPath is all about.  \n",
    "\n",
    "## Readings for this week\n",
    "\n",
    "I'm going to be doing some demos in this notebook, focusing more on how the Python works and leave a lot of the XPath narrative to the W3C School's XPath lesson:  https://www.w3schools.com/xml/xpath_intro.asp.  The terminology section is one of the most important, so that might be something worth printing out or taking notes on.  You'll need to know the names of things to understand the later lessons. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# XPath Basics\n",
    "\n",
    "XPath statements tend to look a little like URLs, because the core tree structure behind websites and XML documents is about the same.  Philosophically speaking.  Let's take a basic XML snippet:\n",
    "\n",
    "```XML\n",
    "<book>\n",
    "    <author>Human, A.</author>\n",
    "    <title>This is not a book</title>\n",
    "</book>\n",
    "```\n",
    "\n",
    "There are numerous ways to describe this structure.\n",
    "\n",
    "* `<book>` is the root element with two children:  `<author>` and `<title>`.\n",
    "* The `<author>` element is a child of `<book>` and sibling to `<title>`.  \n",
    "\n",
    "These descriptions are the basis for how XPath queries are constructed.  So, you don't say \"I want the title element wihtin the book element\", it's \"Find the book element anywhere in the tree, then get the child element called author.  We can express this statement as such:\n",
    "\n",
    "`'//book/author'`\n",
    "\n",
    "Yes, the narrative is much longer than the actual statement, but this is the basis for every advanced XPath query.  We at least think this is correct, but we haven't tested it.  So let's inject this into the Python pattern we saw last week.  This pattern will be a little different because we're working off of a string instead of a file.  There are separate functions to use when reading XML from a file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<Element author at 0x106374c08>]\n"
     ]
    }
   ],
   "source": [
    "from lxml import etree\n",
    "\n",
    "xml = \"\"\"\n",
    "<book>\n",
    "    <author>Human, A.</author>\n",
    "    <title>This is not a book</title>\n",
    "</book>\"\"\"\n",
    "\n",
    "tree = etree.fromstring(xml)\n",
    "print(tree.xpath('//book/author'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Good news:  it worked!\n",
    "\n",
    "Bad news:   WTF is this `<Element author...` crap?\n",
    "\n",
    "# The two parts to every Xpath statement\n",
    "\n",
    "What the `xpath` statement has returned back to us is an `Element` object.  This is a little bundle of processed XML and something that we can act on in smart ways.\n",
    "\n",
    "What we need to ask ourselves next is: what do we want to get out of that element?\n",
    "\n",
    "Each element should have:\n",
    "\n",
    "1. A piece to select the right element or elements\n",
    "2. A piece to extract the data that you want out those elements\n",
    "\n",
    "Meaning, why did we look for that element?  What was our purpose?  Did we want an attribute value or the element text?  We need to use additional XPath syntax to actually extract information out of the element.\n",
    "\n",
    "## getting text of out element values with /text() and //text()\n",
    "\n",
    "So this brings us to the second part of nearly every expath statement:  the data extraction piece.  We've got `'//book/author'`, which will find the element in question.  We need to add `'//text()'` to extract out the actual text.  The `'//`' part of that says \"anywhere in the tree.  I usually recommend it in case there's additional text in other elements.  We'll explore this later, just remember that when I use two `/`s that I'm doing so on purpose. \n",
    "\n",
    "Sometimes you want to leave these things separate, so you have a two stage query:  find all the author elements and then extract the information out of those elements.  But in many cases we can put everyting together in one statement.  We can do that now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Human, A.']\n"
     ]
    }
   ],
   "source": [
    "from lxml import etree\n",
    "\n",
    "xml = \"\"\"\n",
    "<book>\n",
    "    <author>Human, A.</author>\n",
    "    <title>This is not a book</title>\n",
    "</book>\"\"\"\n",
    "\n",
    "tree = etree.fromstring(xml)\n",
    "print(tree.xpath('//book/author//text()')) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yay!  We've gotten it.  Now let's explore why we want to do `//text()` instead of `/text()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Human, A.']\n",
      "['Human, A.']\n"
     ]
    }
   ],
   "source": [
    "print(tree.xpath('//book/author//text()'))\n",
    "print(tree.xpath('//book/author/text()'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It actually works just fine with our example XML because all the text that I want is under the `<author>` element.  But what if I had sub elements that I can't always expect?  Let's take a look at the title element and tweak it from there."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "xml = \"\"\"\n",
    "<book>\n",
    "    <author>Human, A.</author>\n",
    "    <title>This is <b>not</b> a book</title>\n",
    "</book>\"\"\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I've changed the `<title>` statement to now have an element bolded.  While we know that this is just a display markup and the bolding has no semantic real meaning with the data structure, XML doesn't care and sees it just the same as another element.  Meaning that when I try to get text out with just a single slash, `/text()`, that it'll skip over the bolded text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['This is ', ' a book']\n"
     ]
    }
   ],
   "source": [
    "tree = etree.fromstring(xml)\n",
    "print(tree.xpath('//book/title/text()')) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see two things here:\n",
    "\n",
    "* instead of having one string of text I have two inside of my list.\n",
    "* the text content inside of the `<b>` element has been omitted.\n",
    "\n",
    "This means that with `/` it only looks for text one level deep.  Literally, \"just the text directly inside of the `<title>` element and do not search any deeper within the children elements.\"\n",
    "\n",
    "But I can change this to `//` and let it look deeper."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['This is ', 'not', ' a book']\n"
     ]
    }
   ],
   "source": [
    "print(tree.xpath('//book/title//text()')) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that all the text is there, but now I've got three strings instead of one.  This is a good use case for our `.join` manouver.  I actively want all this data to be a single value in one of my cells, so I can join it all back together.  The next question being: what should the delimiter be?  I can see that all the white space the belongs in the title is retained, so I can just use my empty string join."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'This is not a book'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "title_list = tree.xpath('//book/title//text()')\n",
    "\n",
    "title = \"\".join(title_list)\n",
    "\n",
    "title"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## In sum about getting element text...\n",
    "\n",
    "To get text out of an element you need to:\n",
    "\n",
    "* have an xpath statement that is selecing the right element(s)\n",
    "* use additional XPath functions to extract the data you want out of that element.\n",
    "    * You may do this as part of a single xpath query:\n",
    "        * `//book/title//text()'\n",
    "    * Or as a separate query:\n",
    "        * ```titles_elem_list = tree.xpath('//book/title')\n",
    "        for title in titles_elem_list:\n",
    "            print(title.xpath('.//text()'))```\n",
    "        * The odd `.//` that you have to do is because you've split the queries apart and need to make your second query more specific.  In this case, `.` indicates \"start from this element\", then the `//` says \"Look into all descendents within this element\"\n",
    "* sometimes you'll have sub elements in text, particularly text that has HTML formatting within it, or text with additional tags.  You must decide if you want to:\n",
    "    * look one level deep for text (using `/text()`)\n",
    "    * or if you want to look at all descendents within that element for text (using `//text()`)\n",
    "        * this also means that that you may have a single string of text (that you as a human understand to be a single string of text) coming in as a list of many strings.  You can use `\"\".join(list_of_strings)` to concatenate all of those strings back into one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['This is ', 'not', ' a book']\n"
     ]
    }
   ],
   "source": [
    "titles_elem_list = tree.xpath('//book/title')\n",
    "for title in titles_elem_list:\n",
    "    print(title.xpath('.//text()'))\n",
    "    # this "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting text from attributes\n",
    "\n",
    "Attributes are little pieces of text inside of an element.  Sometimes these are uninteresting little pieces of metadata, but sometimes they're the super juicy bit of information that you want.  Let's start with a new example:\n",
    "\n",
    "```XML\n",
    "<book id = \"42\">\n",
    "    <author>Human, A.</author>\n",
    "    <title>This is not a book</title>\n",
    "</book>```\n",
    "\n",
    "Instead of having a separate element of `<book_id>`, the unique ID for this book is embedded within an attribute of the `<book>` element.  We can now use another XPath syntax tool to say that we want the text of a specific attribute's value.  We do have to know the name of the attribute and (usually) which element within it.  Where we might normally put `//text()` after an element's name to get the element text, we can now say `@attribute_name` to have it give us that value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['42']\n"
     ]
    }
   ],
   "source": [
    "xml = \"\"\"\n",
    "<book id = \"42\">\n",
    "    <author>Human, A.</author>\n",
    "    <title >This is not a book</title>\n",
    "</book>\"\"\"\n",
    "\n",
    "tree = etree.fromstring(xml)\n",
    "print(tree.xpath('@id')) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This example looks a little odd, and that's because the `id` attribute is within the root element.  Let's add some more attribute values to play with this syntax more."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['42']\n",
      "['42', 'HumanEntity-0003']\n",
      "['HumanEntity-0003']\n",
      "['en']\n"
     ]
    }
   ],
   "source": [
    "xml = \"\"\"\n",
    "<book id = \"42\">\n",
    "    <author id = \"HumanEntity-0003\">Human, A.</author>\n",
    "    <title lang = \"en\">This is not a book</title>\n",
    "</book>\"\"\"\n",
    "\n",
    "tree = etree.fromstring(xml)\n",
    "print(tree.xpath('@id')) # I'll get the book id\n",
    "print(tree.xpath('//@id')) # look in the entire tree for instances of an id attribute and give them all to me\n",
    "print(tree.xpath('author/@id')) # just the id attribute from the author element\n",
    "print(tree.xpath('title/@lang')) # lang attribute from the title"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using attributes to specify elements\n",
    "\n",
    "We need a more complex example here.\n",
    "\n",
    "```XML\n",
    "<book id = \"42\">\n",
    "    <author id = \"HumanEntity-0003\" role = \"primary\">Human, Not A.</author>\n",
    "    <author id = \"HumanEntity-0004\" role = \"other\">Popsicle, Meat</author>\n",
    "    <author id = \"HumanEntity-0005\" role = \"other\">Cardassian, Kim</author>\n",
    "    <title lang = \"en\">This is not a book</title>\n",
    "</book>\n",
    "```\n",
    "\n",
    "So now we have several `<author>` elements.  We could potentially want only the primary author, only the other authors, or specify the ID of the author."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['HumanEntity-0003', 'HumanEntity-0004', 'HumanEntity-0005', 'NonHuman-0006']\n",
      "['primary', 'other', 'other']\n"
     ]
    }
   ],
   "source": [
    "xml = \"\"\"<book id = \"42\">\n",
    "    <author id = \"HumanEntity-0003\" role = \"primary\">Human, Not A.</author>\n",
    "    <author id = \"HumanEntity-0004\" role = \"other\">Popsicle, Meat</author>\n",
    "    <author id = \"HumanEntity-0005\" role = \"other\">Cardassian, Kim</author>\n",
    "    <author id = \"NonHuman-0006\">Here, Y. Am. I.</author>\n",
    "    <title lang = \"en\">This is not a book</title>\n",
    "</book>\"\"\"\n",
    "\n",
    "tree = etree.fromstring(xml)\n",
    "print(tree.xpath('author/@id')) # so we can get all the IDs now\n",
    "print(tree.xpath('author/@role')) # and all the roles.\n",
    "# but we can see that not all authors have a role, maybe that has meaning to our schema?\n",
    "# this is why you need to know the schema you are working with"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can select specific elements based on their attribute values with the syntax `element[@attribute_name = \"value\"]` or we can select specific elements that have a certain attribute with `element[@attribute_name]`.  Note how we'll need to go back to using our `//text()` tool because this is only helping us select elements, not extract data.\n",
    "\n",
    "Now that we're getting into the meatier sections of XPath, we can look at some of the functions at our disposal.  We can see that there are two types of author ids:  one for \"HumanEntity\" and one for \"NonHuman\".  We can use `element[contains(@attribute_name, \"partial match text\"]` to select an element based off a partial match for that text.  We'd still need to use `attribute_name` or `//text()` to get the data we want out of it.\n",
    "\n",
    "Watch out! This requires quotes, so you need to be careful about how you're either using `'` to surround the xpath statement and `\"` to give the statement quotes, or escaping out your quotes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Human, Not A.']\n",
      "['Popsicle, Meat', 'Cardassian, Kim']\n",
      "['Human, Not A.', 'Popsicle, Meat', 'Cardassian, Kim']\n",
      "['NonHuman-0006']\n"
     ]
    }
   ],
   "source": [
    "print(tree.xpath('author[@role = \"primary\"]//text()')) # all with primary role\n",
    "print(tree.xpath('author[@role = \"other\"]//text()')) # all with other role\n",
    "\n",
    "# now doing a partial text match for id values that contain things...\n",
    "\n",
    "print(tree.xpath('author[contains(@id, \"HumanEntity\")]//text()')) #grab the element text\n",
    "print(tree.xpath('author[contains(@id, \"NonHuman\")]/@id')) # grab the id attribute value text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Namespaces\n",
    "\n",
    "So far we've only been working on pretty plain XML with no namespaces, this means that we've been able to use plain element names in our xpath statements.  However, many times our metadata will be coming in with specific named schemas, perhaps even multiple.  Namespaces are a complex topic, but we can talk about this in the context of our assignment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "```XML\n",
    "<book xmlns = \"http://ischool.illinois.edu/faketown\" id = \"42\">\n",
    "    <author id = \"HumanEntity-0003\" role = \"primary\">Human, Not A.</author>\n",
    "    <author id = \"HumanEntity-0004\" role = \"other\">Popsicle, Meat</author>\n",
    "    <author id = \"HumanEntity-0005\" role = \"other\">Cardassian, Kim</author>\n",
    "    <title lang = \"en\">This is not a book</title>\n",
    "</book>\n",
    "```\n",
    "\n",
    "If you look closely, you'll see that I added a (fake) namespace to our root book element.  This means that our previous paths will fail. The empty lists tell us that the XPath statements are failing, but the rest of the code is running just fine.  There are two parts to handling namespaces within the `xpath()` function.  \n",
    "\n",
    "Part 1:  create a dictionary with an alias for that namespace as the key and the URL that matches the URL in the `xmlns` attribute.  Save this dictionary to a variable.  In this case, we'd want `{'book': 'http://ischool.illinois.edu/faketown'}`  We can give it any name we want, but it should be something pretty readable.\n",
    "\n",
    "Part 2:  Pass that namespace dictionary to the `xpath()` function via this syntax:  `tree.xpath('alias:element/stuff()', namespace = namespace_dict_variable)`.\n",
    "\n",
    "You'll need to use the `alias:` thing before each and every element name in your xpath statement.  Note that I said element name, and not attribute name or xpath functions (e.g. `text()` can stay just that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[]\n",
      "[]\n"
     ]
    }
   ],
   "source": [
    "xml = \"\"\"<book xmlns = \"http://ischool.illinois.edu/faketown\" id = \"42\">\n",
    "    <author id = \"HumanEntity-0003\" role = \"primary\">Human, Not A.</author>\n",
    "    <author id = \"HumanEntity-0004\" role = \"other\">Popsicle, Meat</author>\n",
    "    <author id = \"HumanEntity-0005\" role = \"other\">Cardassian, Kim</author>\n",
    "    <author id = \"NonHuman-0006\">Here, Y. Am. I.</author>\n",
    "    <title lang = \"en\">This is not a book</title>\n",
    "</book>\"\"\"\n",
    "\n",
    "tree = etree.fromstring(xml)\n",
    "print(tree.xpath('author/@id'))\n",
    "print(tree.xpath('author/@role'))\n",
    "\n",
    "# ERROR: \n",
    "# Undefined namespace prefix\n",
    "# Because, this has a namespace declared in the XML, but I'm not using it within my xapth"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['HumanEntity-0003', 'HumanEntity-0004', 'HumanEntity-0005', 'NonHuman-0006']\n",
      "[]\n"
     ]
    }
   ],
   "source": [
    "ns = {'bk': 'http://ischool.illinois.edu/faketown'}\n",
    "\n",
    "tree = etree.fromstring(xml)\n",
    "\n",
    "print(tree.xpath('bk:author/@id', namespaces = ns))\n",
    "print(tree.xpath('author/@role', namespaces = ns))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So why is my second query about author failing?  Well, I forgot the alias piece in my xpath statement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['HumanEntity-0003', 'HumanEntity-0004', 'HumanEntity-0005', 'NonHuman-0006']\n",
      "['primary', 'other', 'other']\n"
     ]
    }
   ],
   "source": [
    "ns = {'bk': 'http://ischool.illinois.edu/faketown'}\n",
    "\n",
    "tree = etree.fromstring(xml)\n",
    "\n",
    "print(tree.xpath('bk:author/@id', namespaces = ns))\n",
    "print(tree.xpath('bk:author/@role', namespaces = ns))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}