docs/notebooks/MCS Metadata Tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import print_function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All MCS0030 (SDF-based) observations will have associated metadata in the form of a metadata tarball.  The LWA1 Data Archive stores all of the metadata files that have been aquired so that observers can access the metadata.  This tutorial demonstrates the LSL interfaces to the metadata and how to extract various information.\n",
    "\n",
    "First, lets get a metadata tarball:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Metadata Size: 470.6 kB\n"
     ]
    }
   ],
   "source": [
    "# This may take a bit...\n",
    "import os\n",
    "try:\n",
    "    from urllib2 import urlopen\n",
    "except ImportError:\n",
    "    from urllib.request import urlopen\n",
    "from tempfile import mkdtemp\n",
    "\n",
    "TEMPDIR = mkdtemp()\n",
    "TGZ_FILE = os.path.join(TEMPDIR, 'LK003_0113.tgz')\n",
    "\n",
    "fh1 = urlopen('https://lda10g.alliance.unm.edu/metadata/lwa1/observation/130921_done/LK003_0113.tgz')\n",
    "fh2 = open(TGZ_FILE, 'wb')\n",
    "fh2.write(fh1.read())\n",
    "fh1.close()\n",
    "fh2.close()\n",
    "    \n",
    "print(\"Metadata Size: %.1f kB\" % (os.path.getsize(TGZ_FILE)/1024.))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The lsl.common.metabundle module provides access to the contents of the tarball.  To get the SDF included in the tarball run:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Project ID: LK003\n",
      "Session ID: 113\n"
     ]
    }
   ],
   "source": [
    "from lsl.common import metabundle \n",
    "\n",
    "project = metabundle.get_sdf(TGZ_FILE)\n",
    "print(\"Project ID: %s\" % project.id)\n",
    "print(\"Session ID: %i\" % project.sessions[0].id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The \"project\" variable contains all of the observations in the SDF as well:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Observation #1\n",
      "   Target: M31\n",
      "   Mode: TRK_RADEC\n",
      "   Start:\n",
      "    MJD: 56556\n",
      "    MPM: 13692273\n",
      "   Duration: 4:00:00\n"
     ]
    }
   ],
   "source": [
    "from datetime import datetime, timedelta\n",
    "\n",
    "nObs = len(project.sessions[0].observations)\n",
    "for i in range(nObs):\n",
    "    currDur = project.sessions[0].observations[i].dur\n",
    "    currDur = timedelta(seconds=int(currDur/1000), microseconds=(currDur*1000) % 1000000)\n",
    "    \n",
    "    print(\"  Observation #%i\" % (i+1,))\n",
    "    print(\"   Target: %s\" % project.sessions[0].observations[i].target)\n",
    "    print(\"   Mode: %s\" % project.sessions[0].observations[i].mode)\n",
    "    print(\"   Start:\")\n",
    "    print(\"    MJD: %i\" % project.sessions[0].observations[i].mjd)\n",
    "    print(\"    MPM: %i\" % project.sessions[0].observations[i].mpm)\n",
    "    print(\"   Duration: %s\" % currDur)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Working with MJD and MPM values isn't particularlly transparent but a MJD/MPM pair can be easily converted to a Python datetime object storing UTC time:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MJD: 56556\n",
      "MPM: 13692273\n",
      "-> 2013-09-21 03:48:12.273000\n"
     ]
    }
   ],
   "source": [
    "from lsl.astro import utcjd_to_unix, MJD_OFFSET\n",
    "\n",
    "mjd = project.sessions[0].observations[0].mjd\n",
    "mpm = project.sessions[0].observations[0].mpm\n",
    "tStart = utcjd_to_unix(mjd + MJD_OFFSET)\n",
    "tStart += mpm / 1000.0\n",
    "\n",
    "print(\"MJD: %i\" % mjd)\n",
    "print(\"MPM: %i\" % mpm)\n",
    "print(\"-> %s\" % datetime.utcfromtimestamp(tStart))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The metadata tarball also stores information about the filename and DRSU barcode that the data were recorded on though the getSessionMetadata function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " Obs. #1: 056556_000068631 on S15TCV13S0001\n"
     ]
    }
   ],
   "source": [
    "fileInfo = metabundle.get_session_metadata(TGZ_FILE)\n",
    "for obsID in fileInfo.keys():\n",
    "    print(\" Obs. #%i: %s on %s\" % (obsID, fileInfo[obsID]['tag'], fileInfo[obsID]['barcode']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, the metadata also contains the ASP settings before and after the observation so that observers can find out what ASP was set to during their observations.  This is accomplised through the getASPConfigurationSummary function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Beginning\n",
      "    filter: 0\n",
      "    at1: 8\n",
      "    at2: 6\n",
      "    atsplit: 15\n",
      "  End\n",
      "    filter: 0\n",
      "    at1: 8\n",
      "    at2: 6\n",
      "    atsplit: 15\n"
     ]
    }
   ],
   "source": [
    "for which in ('Beginning', 'End'):\n",
    "    aspConfig = metabundle.get_asp_configuration_summary(TGZ_FILE, which=which)\n",
    "    print('  %s' % which)\n",
    "    for k in aspConfig.keys():\n",
    "        print('    %s: %i' % (k, aspConfig[k]))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}