Finish up tutorial 4.

timstaley · Dec 10, 2015 · 1a36fce · 1a36fce
1 parent b006a30
commit 1a36fce
Show file tree

Hide file tree

Showing 6 changed files with 251 additions and 43 deletions.
diff --git a/docs/requirements.txt b/docs/requirements.txt
@@ -1,3 +1,5 @@
 sphinx
 jupyter
-voevent-parse
+voevent-parse
+matplotlib
+networkx
diff --git a/docs/source/notebooks/tutorial1.ipynb b/docs/source/notebooks/tutorial1.ipynb
@@ -5,7 +5,8 @@
    "metadata": {},
    "source": [
     "# Getting started with *voeventdb.remote*\n",
-    "In this notebook, we will cover the basics of working with voeventdb.remote, and give a quick tour of the sorts of data you can retrieve."
+    "\n",
+    "#### In this notebook, we will cover the basics of working with voeventdb.remote, and give a quick tour of the sorts of data you can retrieve."
    ]
   },
   {
@@ -124,7 +125,7 @@
    "cell_type": "code",
    "execution_count": null,
    "metadata": {
-    "collapsed": true
+    "collapsed": false
    },
    "outputs": [],
    "source": [
@@ -136,7 +137,7 @@
    "cell_type": "code",
    "execution_count": null,
    "metadata": {
-    "collapsed": true
+    "collapsed": false
    },
    "outputs": [],
    "source": [
@@ -155,7 +156,7 @@
    "cell_type": "code",
    "execution_count": null,
    "metadata": {
-    "collapsed": true
+    "collapsed": false
    },
    "outputs": [],
    "source": [
@@ -194,7 +195,7 @@
    "cell_type": "code",
    "execution_count": null,
    "metadata": {
-    "collapsed": true
+    "collapsed": false
    },
    "outputs": [],
    "source": [
@@ -213,7 +214,7 @@
    "cell_type": "code",
    "execution_count": null,
    "metadata": {
-    "collapsed": true
+    "collapsed": false
    },
    "outputs": [],
    "source": [
@@ -255,7 +256,7 @@
    "cell_type": "code",
    "execution_count": null,
    "metadata": {
-    "collapsed": true
+    "collapsed": false
    },
    "outputs": [],
    "source": [
@@ -273,7 +274,7 @@
    "cell_type": "code",
    "execution_count": null,
    "metadata": {
-    "collapsed": true
+    "collapsed": false
    },
    "outputs": [],
    "source": [
@@ -292,7 +293,7 @@
    "cell_type": "code",
    "execution_count": null,
    "metadata": {
-    "collapsed": true
+    "collapsed": false
    },
    "outputs": [],
    "source": [

diff --git a/docs/source/notebooks/tutorial2.ipynb b/docs/source/notebooks/tutorial2.ipynb
@@ -7,7 +7,7 @@
     "# Narrowing your query and digging into your results\n",
     "We previously covered the basics of working with voeventdb.remote in tutorial 1. \n",
     "\n",
-    "In this notebook, we'll demonstrate the use of filters to narrow down your query, and demonstrate the 'helper classes' you can use to easily access the details of a particular packet."
+    "#### In this notebook, we'll demonstrate how to use combinations of filters to narrow down your query, and demonstrate the 'helper classes' you can use to easily access the details of a particular packet."
    ]
   },
   {
@@ -98,15 +98,17 @@
    "outputs": [],
    "source": [
     "## To see the list of filters, you can use tab-completion:\n",
+    "## (Uncomment the following line and try it for yourself)\n",
     "# FilterKeys.\n",
-    "## Or the ipython doc-lookup magic, by prefixing with ``??``:\n",
+    "## Or the ipython doc-lookup magic, by prefixing with ``??`` and running the cell:\n",
     "# ??FilterKeys"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
+    "### Filtering by role\n",
     "So: we were trying to filter out the test-packets. ``FilterKeys.role`` sounds promising. To apply a filter, or multiple filters, we simply define a dictionary with the filters we want to apply, and then pass it to the relevant query-function, like this:"
    ]
   },
@@ -136,6 +138,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
+    "### Filtering by date\n",
     "That results in a much shorter list, containing only scientifically interesting streams. Still, those numbers are pretty large (mainly for *Swift*). It might be useful to get a smaller representative sample. How many packets will we get if we limit our query to a single week?"
    ]
   },
@@ -173,6 +176,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
+    "### Filtering by stream\n",
     "Ok, so there's still a lot of *Swift* packets there. Let's take a look at a sample of those, and see if we can break them up further. First, lets add another filter to limit our query to just *Swift* packets."
    ]
   },
@@ -184,7 +188,7 @@
    },
    "outputs": [],
    "source": [
-    "my_filters[FilterKeys.stream]='nasa.gsfc.gcn/SWIFT'\n",
+    "my_filters[FilterKeys.stream] = 'nasa.gsfc.gcn/SWIFT'\n",
     "my_filters"
    ]
   },
@@ -210,6 +214,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
+    "### Filters can be used across different query-endpoints\n",
     "Not particularly helpful, but at least everything is working as expected. Now, the neat thing about the voeventdb filters is that they can be applied to **any** query-endpoint - we can just re-use the filter-dictionary with the ``apiv1.ivorn`` function to get back a list of IVORNs:"
    ]
   },
@@ -255,6 +260,8 @@
    "source": [
     "Now we're getting somewhere! We can clearly see the subcategories of Swift packets - BAT alerts, XRT positions, UVOT followup, etc. \n",
     "\n",
+    "### Filtering by IVORN substring\n",
+    "\n",
     "We can use this knowledge to refine our filters, by filtering on a *substring* of the IVORN, using the ``ivorn_contains`` filter. For example, we might want to filter to just those IVORNs containing XRT positions (**note this filter is case-sensitive**):"
    ]
   },
@@ -287,7 +294,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "As in tutorial 1, we can inspect the details of any given packet - we'll take a look at the first one. This packet makes a good example, as it includes details of the event co-ordinates and timestamp, and also references an earlier VOEvent:"
+    "As in tutorial 1, we can inspect the details of any given packet using the ``synopsis`` endpoint - we'll take a look at the first one. This packet makes a good example, as it includes details of the event co-ordinates and timestamp, and also references an earlier VOEvent:"
    ]
   },
   {
@@ -332,15 +339,15 @@
    "outputs": [],
    "source": [
     "xrt_synopsis = Synopsis(synopsis_dict)\n",
-    "# Prints with nicer formatting:\n",
+    "# Prints with nicer formatting, ordering of values:\n",
     "print(xrt_synopsis)"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "You can easily access the values (with the ever-handy IPython autocompletion):"
+    "Now we can easily access the values (with the ever-handy IPython autocompletion):"
    ]
   },
   {
@@ -399,7 +406,7 @@
    },
    "outputs": [],
    "source": [
-    "# List of 1, in this case. Grab the first (only) element:\n",
+    "# List of 1, in this case. Grab the first (and only) element:\n",
     "sky_event = xrt_synopsis.sky_events[0]"
    ]
   },
@@ -446,12 +453,43 @@
     "print(sky_event.position.ra.hms)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Advanced usage: specifying multiple values for the same filter\n",
+    "Before we move on, it's worth mentioning that some filters can take on multiple values. This is specified by defining the filter-value as a list - for example, to return all VOEvents with a role of 'observation' or 'utility' we can use the following:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "my_filters = {apiv1.FilterKeys.role: ['observation','utility']}\n",
+    "apiv1.stream_count(my_filters)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "How does this work? Well, we can think of each entry in the list defining a separate filter.\n",
+    "For the `role` value, these filters are combined in the logical 'OR' sense, so we get back combined counts for both 'observation' and 'utility' packets. You can check whether a filter accepts multiple values, and if they are combined via logical 'OR' or 'AND', by checking the \n",
+    "[filter-definitions page](http://voeventdb.readthedocs.org/en/latest/apiv1/queryfilters.html) and looking for the ``combinator`` attribute. \n",
+    "\n",
+    "(If this sounds confusing, don't worry - it can safely be ignored unless you're planning to use really tricky queries.)"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "## Coming next ...\n",
-    "We've seen how to narrow our search, locate packets of interest, and pull up the details. \n",
+    "We've seen how to narrow our search, locate packets of interest, and use helper-classes to easily access packet details. \n",
     "In the next tutorial, we'll cover different ways of finding *related* VOEvents."
    ]
   }

diff --git a/docs/source/notebooks/tutorial3.ipynb b/docs/source/notebooks/tutorial3.ipynb
@@ -7,7 +7,7 @@
     "# Spatial queries\n",
     "By now, you should have some idea of how to explore the various ``voeventdb.remote`` endpoints and apply filters. \n",
     "\n",
-    "In this notebook, we'll extend our filter-toolset to include cone-searches.\n"
+    "#### In this notebook, we'll extend our filter-toolset to include cone-searches.\n"
    ]
   },
   {
@@ -75,7 +75,7 @@
    "metadata": {},
    "source": [
     "## Filtering using a cone-search (AKA spatial queries)\n",
-    "Next, let's see if there's any other recorded events with associated positions nearby. To do so, we'll need to define a 'cone', a sky-position and circle around it to search in. For setting up a ``voeventdb.remote`` cone-filter, we can use a tuple of type \n",
+    "Next, let's see if there are any other recorded events with associated positions nearby. To do so, we'll need to define a 'cone', a sky-position and circle around it to search in. For setting up a ``voeventdb.remote`` cone-filter, we can use a tuple of type \n",
     "\n",
     "``(astropy.coordinates.SkyCoord, astropy.coordinates.Angle)``\n",
     "\n",
@@ -98,7 +98,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "However, the XRT position has a really tight error-circle, about 5 arcseconds. Note that the cone-search will only return VOEvents with a *best-estimate position* within the cone - **it does not take into account overlapping error-circles** (at least for version 1!). This means that we could have a related event with a large error-circle just outside the tiny XRT error-circle, and it wouldn't be returned - so we have to use some judgement here. We'll set the cone angular radius to half a degree, instead:"
+    "However, the XRT position has a really tight error-circle, about 5 arcseconds. Note that the cone-search will only return VOEvents with a *best-estimate position* within the cone - **it does not take into account overlapping error-circles** (at least for version 1!). This means that we could have a related event with a large error-circle, but which has a best-estimate position just outside the tiny XRT error-circle, and it wouldn't be returned - so we have to use some judgement here. We'll set the cone angular radius to half a degree, instead:"
    ]
   },
   {
@@ -167,7 +167,9 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "So, we have a bunch of packets related to the same event (ID 666352), some 'SubSubThresh' events, and a 'Known_Pos' event. It's worth noting that 'SubSubThresh' events are extremely common and show up all over:"
+    "### Investigating 'background' event rates\n",
+    "\n",
+    "So, we have a bunch of *Swift* packets related to the same event (ID 666352), some 'BAT_SubSubThresh' events, and a 'Known_Pos' event. It's worth noting that 'SubSubThresh' events are extremely common and show up all over; in fact they currently make up about half the packets in the database:"
    ]
   },
   {
@@ -178,21 +180,21 @@
    },
    "outputs": [],
    "source": [
-    "all_count = apiv1.count(filters={FilterKeys.role:'observation'})\n",
+    "all_events_count = apiv1.count(filters={FilterKeys.role:'observation'})\n",
     "ss_thresh_count = apiv1.count(filters={\n",
     "        FilterKeys.role:'observation', \n",
     "        FilterKeys.ivorn_contains:'BAT_SubSubThresh',\n",
     "        })\n",
     "print(\"Of {} observation packets in the database, {} are BAT_SubSubThresh packets\".format(\n",
-    "        all_count, ss_thresh_count))"
+    "        all_events_count, ss_thresh_count))"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "So it's perhaps not surprising that we'd encounter a few of them co-incidentally lying in the search-cone.\n",
-    "We can define a search-cone at arbitrary co-ordinates and see what we get back, for comparison:"
+    "We can define a search-cone of the same radius at arbitrary co-ordinates and see what we get back, for comparison:"
    ]
   },
   {
@@ -247,17 +249,64 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Result - looks like we get a similar number of these events where-ever we point our search-cone!"
+    "Result: looks like we get a similar number of these 'SubSubThresh' events wherever we point our search-cone!"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Restricting the time-period to a 60-day window\n",
+    "Fortunately, it's easy to narrow things down a bit further, *if* we expect related events to occur within a reasonably short time-interval. Let's modify our original filter-set a bit more:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "from datetime import timedelta\n",
+    "cone_filters[FilterKeys.authored_since] = sky_event.timestamp - timedelta(days=30)\n",
+    "cone_filters[FilterKeys.authored_until] = sky_event.timestamp + timedelta(days=30)\n",
+    "cone_filters"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "sorted(apiv1.ivorn(cone_filters))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "As a result of restricting our search to a window about the XRT-position timestamp, we've cut out all of those 'SubSubThresh' events that probably weren't related.\n",
+    "\n",
+    "## A note of warning\n",
+    "What hasn't been mentioned so far is that **a query with a cone-search filter will only return VOEvents which contain a position** in the \n",
+    "[``WhereWhen``](http://voevent.readthedocs.org/en/latest/reading.html#overall-structure)\n",
+    "section of the packet. This sounds obvious, but on the other hand it's worth being reminded that some *related and interesting* VOEvents may not contain any position information.\n",
+    "\n",
+    "Most packets relating to follow-up or simultaneous observations of an event *will* contain position data, but we can imagine circumstances where this wouldn't apply, e.g. an all-sky high-energy gamma-ray detector without good localization.  Alternatively a VOEvent author may simply decide not to repeat co-ordinates that have been given in an earlier packet (perhaps when giving refined estimates about flux / lightcurve /  spectral index, etc), but instead rely on the **citation** mechanism - see the next notebook!"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "## Coming next ...\n",
-    "In the search above for packets near to a Swift XRT position, many of the packets were clearly designated as relating to the same event - they all had the same ID number. Is that relation encoded into the packet-data? Can we just select the VOEvents which are already marked as being related to a particular packet?\n",
+    "In the search above for packets near to a Swift XRT position, many of the packets were clearly designated as relating to the same event - they all had the same ID number. Is that relation encoded into the packet-data? Can we just select the VOEvents which are *already marked* as being related to a particular packet?\n",
     "\n",
-    "The answer, of course, is yes - see the next notebook."
+    "The answer, of course, is yes - we make use of **citation** data."
    ]
   }
  ],