Docs: Add tutorial3, + tweak 1,2.

timstaley · Dec 9, 2015 · e1dc54b · e1dc54b
1 parent 78bcd1c
commit e1dc54b
Show file tree

Hide file tree

Showing 5 changed files with 185 additions and 28 deletions.
diff --git a/docs/source/notebooks/tutorial1.ipynb b/docs/source/notebooks/tutorial1.ipynb
@@ -54,7 +54,11 @@
    "source": [
     "----------\n",
     "### Aside: Autocompletion tricks\n",
-    "For those new to IPython / notebooks: note you can use tab-completion to help you explore / save on typing. For example, type ``apiv1.`` in a code-cell and then hit **Tab** to see your available options. Likewise, position your cursor on a function and hit **Shift+Tab** to see a summary of function arguments, etc.\n",
+    "For those new to IPython / notebooks: note you can use tab-completion to help you explore / save on typing (as long as you've already imported the relevant packages). For example, type:\n",
+    "\n",
+    "    apiv1.\n",
+    "\n",
+    "in a notebook code-cell or IPython terminal and then hit **Tab** to see your available options. Likewise, in notebooks you can position your cursor on a function and hit **Shift+Tab** to see a summary of function arguments, etc.\n",
     "\n",
     "------------"
    ]
@@ -70,7 +74,7 @@
    "cell_type": "code",
    "execution_count": null,
    "metadata": {
-    "collapsed": true
+    "collapsed": false
    },
    "outputs": [],
    "source": [
@@ -92,7 +96,7 @@
    "cell_type": "code",
    "execution_count": null,
    "metadata": {
-    "collapsed": true
+    "collapsed": false
    },
    "outputs": [],
    "source": [

diff --git a/docs/source/notebooks/tutorial2.ipynb b/docs/source/notebooks/tutorial2.ipynb
@@ -7,7 +7,7 @@
     "# Narrowing your query and digging into your results\n",
     "We previously covered the basics of working with voeventdb.remote in tutorial 1. \n",
     "\n",
-    "In this notebook, we'll demonstrate the use of filters to narrow down your query, and demonstrate the 'helper classes' you can use to easily access the details for a particular packet."
+    "In this notebook, we'll demonstrate the use of filters to narrow down your query, and demonstrate the 'helper classes' you can use to easily access the details of a particular packet."
    ]
   },
   {
@@ -70,12 +70,11 @@
     "\n",
     "Quite obviously, a number of those streams are 'junk', they contain only test-packets used to verify that the VOEvent infrastructure is up and working correctly. For scientific work, we'll want to filter those out. \n",
     "\n",
-    "Fortunately, we can ask the voeventdb server to do the filtering work for us. The voeventdb.remote library comes with an easy-to-use list of filters, stored as ``voeventdb.remote.apiv1.FilterKeys``. To see what's available, you can either\n",
-    "use the IPython interface as in the cell below, or simply check the \n",
-    "[relevant section](http://voeventdbremote.readthedocs.org/en/latest/reference/index.html#voeventdb.remote.apiv1.FilterKeys)\n",
-    "of the documentation. \n",
+    "Fortunately, we can ask the voeventdb server to do the filtering work for us. The voeventdb.remote library comes with an easy-to-use list of filters, stored as \n",
+    "[``voeventdb.remote.apiv1.FilterKeys``](http://voeventdbremote.readthedocs.org/en/latest/reference/index.html#voeventdb.remote.apiv0.FilterKeys). \n",
+    "To see what's available at a glance you can use the IPython tab-completion and doc-lookup tools, as in the cell below.\n",
     "\n",
-    "Full definitions of the filter-keys (and example filter-values) can be found [in the voeventdb server docs](http://voeventdb.readthedocs.org/en/latest/apiv1/queryfilters.html#apiv1-filters), but we'll cover most of them in this notebook - read on."
+    "Full definitions of the filter-keys (and example filter-values) can be found [in the voeventdb server docs](http://voeventdb.readthedocs.org/en/latest/apiv1/queryfilters.html#apiv1-filters), but we'll cover most of them in these tutorial notebooks - read on."
    ]
   },
   {
@@ -137,7 +136,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Great! That's a much shorter list, and it only contains scientifically interesting streams. Still, those numbers are pretty large (mainly for *Swift*). It might be useful to get a smaller representative sample. How many packets will we get if we limit our query to a single week?"
+    "That results in a much shorter list, containing only scientifically interesting streams. Still, those numbers are pretty large (mainly for *Swift*). It might be useful to get a smaller representative sample. How many packets will we get if we limit our query to a single week?"
    ]
   },
   {
@@ -149,7 +148,8 @@
    "outputs": [],
    "source": [
     "from datetime import datetime, timedelta\n",
-    "start_date = datetime(2015,12,1)\n",
+    "import pytz\n",
+    "start_date = datetime(2015,12,1,tzinfo=pytz.UTC)\n",
     "my_filters = { \n",
     "    FilterKeys.role: 'observation',\n",
     "    FilterKeys.authored_since: start_date,\n",
@@ -173,7 +173,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Ok, so there's still a lot of SWIFT packets there. Let's take a look at a sample of those, and see if we can break them up further:"
+    "Ok, so there's still a lot of *Swift* packets there. Let's take a look at a sample of those, and see if we can break them up further. First, lets add another filter to limit our query to just *Swift* packets."
    ]
   },
   {
@@ -192,7 +192,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "OK, so now if we apply the filters to ``stream_count``, we only get back one entry (the *Swift* stream):"
+    "So now if we apply the filters to ``stream_count``, we only get back one entry (the *Swift* stream):"
    ]
   },
   {
@@ -210,7 +210,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Not particularly helpful, but at least everything is working as expected. Now, the neat thing about the voeventdb filters is that they can be applied to **any** query-endpoint - so we can just re-use the filter-dictionary to get back a list of IVORNs."
+    "Not particularly helpful, but at least everything is working as expected. Now, the neat thing about the voeventdb filters is that they can be applied to **any** query-endpoint - we can just re-use the filter-dictionary with the ``apiv1.ivorn`` function to get back a list of IVORNs:"
    ]
   },
   {
@@ -231,7 +231,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "OK, there's clearly a pattern to how the Swift IVORNs are formatted. We'll use a little Python trickery \n",
+    "That's a long list, but there's clearly a pattern to how the Swift IVORNs are formatted. We'll use a little Python trickery \n",
     "(cf [set](https://docs.python.org/2/library/stdtypes.html#set), \n",
     "[str.rsplit](https://docs.python.org/2/library/string.html#string.rsplit))\n",
     "to chop off the trailing ID numbers and  sort them into sub-categories:"
@@ -287,7 +287,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "As in tutorial 1, we can inspect the details of any given packet - we'll take a look at the first one. This packet make a good example, as it includes details of the event co-ordinates and timestamp, and also references an earlier VOEvent:"
+    "As in tutorial 1, we can inspect the details of any given packet - we'll take a look at the first one. This packet makes a good example, as it includes details of the event co-ordinates and timestamp, and also references an earlier VOEvent:"
    ]
   },
   {
@@ -377,7 +377,7 @@
     " - timestamp of the observed event.\n",
     " \n",
     "The position coordinates and error-circle are represented by \n",
-    "[``astropy.coordinates``](http://astropy.readthedocs.org/en/stable/coordinates/index.html) classes, which come with a bunch of features covered in the astropy docs."
+    "[``astropy.coordinates``](http://astropy.readthedocs.org/en/stable/coordinates/index.html) classes, which come with a bunch of features related to formatting, distance calculations, frame-of-reference transformations, etc."
    ]
   },
   {
@@ -451,8 +451,8 @@
    "metadata": {},
    "source": [
     "## Coming next ...\n",
-    "We've seen how to narrow our search, identify packets of interest, and pull up the details. \n",
-    "In the next tutorial, we'll cover different ways of finding related packets."
+    "We've seen how to narrow our search, locate packets of interest, and pull up the details. \n",
+    "In the next tutorial, we'll cover different ways of finding *related* VOEvents."
    ]
   }
  ],

diff --git a/docs/source/notebooks/tutorial3.ipynb b/docs/source/notebooks/tutorial3.ipynb
@@ -4,8 +4,10 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Filtering using a cone-search (AKA spatial queries)\n",
-    "We've just inspected a Fermi event which gave us a position estimate and and error-circle. Next, let's see if there's any other recorded events which lie in the error circle:"
+    "# Spatial queries\n",
+    "By now, you should have some idea of how to explore the various ``voeventdb.remote`` endpoints and apply filters. \n",
+    "\n",
+    "In this notebook, we'll extend our filter-toolset to include cone-searches.\n"
    ]
   },
   {
@@ -33,7 +35,15 @@
     "import voeventdb.remote as vr\n",
     "import voeventdb.remote.apiv1 as apiv1\n",
     "from voeventdb.remote.apiv1 import FilterKeys\n",
-    "from voeventdb.remote.helpers import Synopsis"
+    "from voeventdb.remote.helpers import Synopsis\n",
+    "from astropy.coordinates import Angle, SkyCoord"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "In the last tutorial, we inspected a  *Swift* XRT event which gave us a position estimate and and error-circle. Let's grab those details again:"
    ]
   },
   {
@@ -55,14 +65,21 @@
    },
    "outputs": [],
    "source": [
-    "sky_event = xrt_synopsis.sky_events[0]"
+    "sky_event = xrt_synopsis.sky_events[0]\n",
+    "print(sky_event)\n",
+    "print(\"Error-circle radius in degrees, arcminutes and arcseconds:\", sky_event.position_error)"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Filtering using a cone-search (AKA spatial queries)"
+    "## Filtering using a cone-search (AKA spatial queries)\n",
+    "Next, let's see if there's any other recorded events with associated positions nearby. To do so, we'll need to define a 'cone', a sky-position and circle around it to search in. For setting up a ``voeventdb.remote`` cone-filter, we can use a tuple of type \n",
+    "\n",
+    "``(astropy.coordinates.SkyCoord, astropy.coordinates.Angle)``\n",
+    "\n",
+    "A natural candidate is the position and position-error from the XRT event; like so:"
    ]
   },
   {
@@ -73,23 +90,48 @@
    },
    "outputs": [],
    "source": [
-    "cone = (sky_event.position, sky_event.position_error*3)\n",
+    "cone = (sky_event.position, sky_event.position_error)\n",
     "cone"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "However, the XRT position has a really tight error-circle, about 5 arcseconds. Note that the cone-search will only return VOEvents with a *best-estimate position* within the cone - **it does not take into account overlapping error-circles** (at least for version 1!). This means that we could have a related event with a large error-circle just outside the tiny XRT error-circle, and it wouldn't be returned - so we have to use some judgement here. We'll set the cone angular radius to half a degree, instead:"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {
     "collapsed": false
    },
    "outputs": [],
+   "source": [
+    "cone = (sky_event.position, Angle(0.5, unit='deg'))\n",
+    "cone"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "OK, let's see how that works:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
    "source": [
     "cone_filters = {\n",
     "    FilterKeys.role: 'observation',\n",
     "    FilterKeys.cone: cone\n",
-    "    }\n",
-    "apiv1.count(cone_filters)"
+    "    }"
    ]
   },
   {
@@ -103,14 +145,120 @@
     "apiv1.stream_count(cone_filters)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "A reasonable number. Let's take a look:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "sorted(apiv1.ivorn(cone_filters))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "So, we have a bunch of packets related to the same event (ID 666352), some 'SubSubThresh' events, and a 'Known_Pos' event. It's worth noting that 'SubSubThresh' events are extremely common and show up all over:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "all_count = apiv1.count(filters={FilterKeys.role:'observation'})\n",
+    "ss_thresh_count = apiv1.count(filters={\n",
+    "        FilterKeys.role:'observation', \n",
+    "        FilterKeys.ivorn_contains:'BAT_SubSubThresh',\n",
+    "        })\n",
+    "print(\"Of {} observation packets in the database, {} are BAT_SubSubThresh packets\".format(\n",
+    "        all_count, ss_thresh_count))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "So it's perhaps not surprising that we'd encounter a few of them co-incidentally lying in the search-cone.\n",
+    "We can define a search-cone at arbitrary co-ordinates and see what we get back, for comparison:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "cone2 = (SkyCoord(ra=0, dec=35., unit='deg'), Angle(0.5, unit='deg'))\n",
+    "cone2"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {
     "collapsed": true
    },
    "outputs": [],
-   "source": []
+   "source": [
+    "cone2_filters = {\n",
+    "    FilterKeys.role: 'observation',\n",
+    "    FilterKeys.cone: cone2\n",
+    "    }"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "apiv1.stream_count(filters=cone2_filters)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "apiv1.ivorn(filters=cone2_filters)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Result - looks like we get a similar number of these events where-ever we point our search-cone!"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Coming next ...\n",
+    "In the search above for packets near to a Swift XRT position, many of the packets were clearly designated as relating to the same event - they all had the same ID number. Is that relation encoded into the packet-data? Can we just select the VOEvents which are already marked as being related to a particular packet?\n",
+    "\n",
+    "The answer, of course, is yes - see the next notebook."
+   ]
   }
  ],
  "metadata": {

diff --git a/docs/source/tutorial/index.rst b/docs/source/tutorial/index.rst
@@ -16,4 +16,5 @@ online or work through them interactively by downloading them from the
 
    tutorial1
    tutorial2
+   tutorial3
 
diff --git a/docs/source/tutorial/tutorial3.rst b/docs/source/tutorial/tutorial3.rst
@@ -0,0 +1,4 @@
+Tutorial 3
+=============
+.. raw:: html
+   :file: tutorial3.html