Merge pull request #278 from mapazarr/gen_obs_tables

Generate observation tables

docs/dataformats/file_formats.rst

@@ -145,6 +145,8 @@ INI files ... to use it in your code
 
 Unfortunately INI files are not standardised, so there's only conventions and tons of variants.
 
+.. _CSV_files:
+
 CSV
 +++
 
@@ -194,6 +196,9 @@ XML
 
 **XML** files (see `Wikipedia <http://en.wikipedia.org/wiki/Xml>`__)
 
+GammaLib / ctools uses an "observation definition" XML format described
+`here <http://cta.irap.omp.eu/gammalib-devel/user_manual/modules/obs.html#describing-observations-using-xml>`__.
+
 TODO: describe
 
 FITS

docs/dataformats/index.rst

@@ -19,6 +19,21 @@ we simply use general-purpose file formats define our own semantics as described
 in the :ref:`dataformats_file_formats` section. 
 
 
+.. _time_gammapy:
+
+***************
+Time in Gammapy
+***************
+
+Gammapy adopts the exact same time standard as the Fermi Science Tools as described on
+this page: `Cicerone: Data - Time in Fermi Data Analysis
+<http://fermi.gsfc.nasa.gov/ssc/data/analysis/documentation/Cicerone/Cicerone_Data/
+Time_in_ScienceTools.html>`_.
+Every time should be defined as a `~astropy.time.Time` object, representing UTC seconds
+after the "Mission elapsed time" [MET]_ origin. Each experiment may have a different
+[MET]_ origin that should be included in the header of the data files.
+
+
 .. toctree::
    :maxdepth: 1
 

docs/dataformats/observation_lists.rst

@@ -10,38 +10,90 @@ the information in the run list. These filenames can either be given directly
 or there can be naming conventions, e.g. the event list for run ``42`` could be stored
 at ``$TEV_DATA/RUN_000042/Events.fits``.
 
+A run list must have at least a column called ``OBS_ID``::
+ 
+   OBS_ID
+   42
+   43
 
-CSV format
-----------
+Usually it has many more columns with information about each observation. A list of
+Gammapy supported columns is:
 
-We use the `CSV <http://en.wikipedia.org/wiki/Comma-separated_values>`_ (comma-separated values) format for run lists.
-This has the advantage that everyone can work with whatever tool they like. Here's some good options:
+================  ===========================================================================  =========
+column name       description                                                                  required?
+================  ===========================================================================  =========
+OBS_ID            observation ID as an integer                                                 yes
+RA                pointing position right ascension in equatorial (ICRS) coordinates           yes?
+DEC               pointing position declination in equatorial (ICRS) coordinates               yes?
+AZ                average azimuth angle during the observation                                 no
+ALT               average altitude angle during the observation                                no
+MUON_EFFICIENCY   average muon efficiency of the telescopes                                    yes?
+TIME_START        start time of the observation stored as number of seconds after [MET]_       yes?
+TIME_STOP         end time of the observation in the same format as TIME_START                 no
+TIME_OBSERVATION  duration of the observation                                                  no
+TIME_LIVE         duration of the observation without dead time                                no
+TRIGGER_RATE      average trigger rate of the system                                           no
+MEAN_TEMPERATURE  average temperature of the atmosphere                                        no
+N_TELS            number of telescopes participating in the observation                        yes?
+TEL_LIST          string with a [CSV]_ list of telescope IDs participating in the observation  yes?
+QUALITY           data quality; recommended: "spectral" or "detection"                         no
+================  ===========================================================================  =========
 
-* `LibreOffice Calc <http://www.libreoffice.org/discover/calc/>`_ 
-* `TOPCAT <http://www.star.bris.ac.uk/~mbt/topcat/>`_ or `STILTS <http://www.star.bris.ac.uk/~mbt/stilts/>`_
-* `csvkit <https://csvkit.readthedocs.org/en/latest/>`_
-* Your own script using e.g the `Python standard library CSV module <http://docs.python.org/2/library/csv.html>`_ or `pandas <http://pandas.pydata.org>`_
+Extra user-defined columns are allowed; Gammapy will ignore them.
 
-A run list must have at least a column called ``Run``::
- 
-   Run
-   42
-   43
+In order for the extra columns to have full meaning the following is needed:
 
-Usually it has many more columns with information about each run::
- 
-   Run,Telescope_Pattern,Date,Duration,GLON,GLAT,Target,Zenith
-   1234,24,2013-03-22 14:32,1832,83.7,-5.2,Crab Nebula,32
+ * Extra row right after the column name, specifying the unit of the quantity listed on each column.
+ * A header with at least the following keywords:
+
+================  ===========================================================================  =========
+keyword           description                                                                  required?
+================  ===========================================================================  =========
+OBSERVATORY_NAME  name of the observatory where the observations were taken. This is           no
+                  important for instance for coordinate transformations between celestial
+                  (i.e. RA/dec) and terrestrial (i.e. az/alt) coordinate systems.
+MJDREFI           reference time: integer value in mean julian days; details in                yes?
+                  :ref:`time_gammapy`.
+MJDREFF           reference time: fraction of integer value defined in MJDREFI; details in     yes?
+                  :ref:`time_gammapy`.
+================  ===========================================================================  =========
+
+Extra user-defined header entries are allowed; Gammapy will ignore them.
+
+
+Example
+-------
+The tool `~gammapy.datasets.make_test_observation_table` can generate a `~gammapy.obs.ObservationTable`
+with dummy values.
 
-Special column names that the Gammapy analysis tools understand:
+Header (metadata)::
 
-* ``Run`` --- Run number (int)
-* ``Telescope_Pattern`` --- Binary pattern describing which telescopes participated in the run
-* ``Date`` --- Date of observation
-* ``RA``, ``DEC`` or ``GLON``, ``GLAT`` -- Pointing position in Equatorial (ICRS) or Galactic coordinates
+   {u'MJDREFI': 55197.0, u'OBSERVATORY_NAME': u'HESS', u'MJDREFF': 0.0}
 
-XML format
-----------
+Table:
 
-GammaLib / ctools uses an "observation definition" XML format described
-`here <http://gammalib.sourceforge.net/user_manual/modules/obs.html#describing-observations-using-xml>`__.
++------+----------------+---------+-------------+-------------+-------------+-------------+-------------+--------------+------+---------------+
+|OBS_ID|TIME_OBSERVATION|TIME_LIVE|  TIME_START |  TIME_STOP  |      AZ     |     ALT     |      RA     |     DEC      |N_TELS|MUON_EFFICIENCY|
++------+----------------+---------+-------------+-------------+-------------+-------------+-------------+--------------+------+---------------+
+|      |       s        |    s    |      s      |      s      |     deg     |     deg     |     deg     |     deg      |      |               |
++======+================+=========+=============+=============+=============+=============+=============+==============+======+===============+
+|     1|          1800.0|   1500.0|118890428.352|118892228.352|130.926874751|49.6209457026|96.3849089136|-43.6914197077|     3| 0.814535992712|
++------+----------------+---------+-------------+-------------+-------------+-------------+-------------+--------------+------+---------------+
+|     2|          1800.0|   1500.0|112242990.559|112244790.559|272.213179564|70.2673929472| 339.00128923|-21.1698098192|     3| 0.976469816749|
++------+----------------+---------+-------------+-------------+-------------+-------------+-------------+--------------+------+---------------+
+|     3|          1800.0|   1500.0| 66444741.854| 66446541.854|346.848818939| 46.138583188|162.086175054| 19.6398873974|     4| 0.920096961383|
++------+----------------+---------+-------------+-------------+-------------+-------------+-------------+--------------+------+---------------+
+|     4|          1800.0|   1500.0|22388716.9244|22390516.9244|300.850748052|55.1330124055|32.9474858892|-3.19910057294|     3| 0.678431411337|
++------+----------------+---------+-------------+-------------+-------------+-------------+-------------+--------------+------+---------------+
+|     5|          1800.0|   1500.0|135469063.888|135470863.888|355.160931662| 48.734744852|197.123663537| 17.9411145072|     4|  0.77879533822|
++------+----------------+---------+-------------+-------------+-------------+-------------+-------------+--------------+------+---------------+
+|     6|          1800.0|   1500.0|21857681.6916|21859481.6916|124.846967209| 78.859585347| 14.162859563|-29.3419432185|     4| 0.709642622408|
++------+----------------+---------+-------------+-------------+-------------+-------------+-------------+--------------+------+---------------+
+|     7|          1800.0|   1500.0| 57554741.106| 57556541.106|268.541714486|48.8489560299|64.8265458802|-18.2634404823|     3| 0.908426763354|
++------+----------------+---------+-------------+-------------+-------------+-------------+-------------+--------------+------+---------------+
+|     8|          1800.0|   1500.0|19181027.9045|19182827.9045|120.558129392| 49.663761361| 24.791511978|-37.1789681608|     4| 0.980162662473|
++------+----------------+---------+-------------+-------------+-------------+-------------+-------------+--------------+------+---------------+
+|     9|          1800.0|   1500.0|120447694.583|120449494.583| 132.10271454|78.7455993174|89.7950895353|-30.5128854184|     3| 0.807695978946|
++------+----------------+---------+-------------+-------------+-------------+-------------+-------------+--------------+------+---------------+
+|    10|          1800.0|   1500.0|144207430.361|144209230.361|323.562657045|45.4005803262|324.596045439| 13.6761217326|     3| 0.694201696626|
++------+----------------+---------+-------------+-------------+-------------+-------------+-------------+--------------+------+---------------+

docs/references.rst

@@ -3,6 +3,11 @@
 References
 ==========
 
+.. _publications:
+
+Publications
+------------
+
 This is the bibliography containing the literature references for the implemented methods
 referenced from the Gammapy docs.
 
@@ -58,3 +63,12 @@ Software references:
    
 .. [FSSC2013] `Fermi LAT Collaboration (2013) <http://fermi.gsfc.nasa.gov/ssc/data/analysis/scitools/overview.html>`_
    "Science Tools: LAT Data Analysis Tools"
+
+
+.. _glossary:
+
+Glossary
+--------
+
+.. [CSV] comma-separated values; see also :ref:`CSV_files`.
+.. [MET] mission elapsed time; see also :ref:`time_gammapy`.

docs/utils/index.rst

@@ -56,4 +56,6 @@ Reference/API
 
 .. automodapi:: gammapy.utils.testing
     :no-inheritance-diagram:
-
+
+.. automodapi:: gammapy.utils.time
+    :no-inheritance-diagram:

gammapy/data/event_list.py

@@ -13,6 +13,7 @@
 from ..image import wcs_histogram2d
 from ..data import GoodTimeIntervals, TelescopeArray
 from ..data import InvalidDataError
+from ..utils.time import time_ref_from_dict
 from . import utils
 
 __all__ = ['EventList',
@@ -78,7 +79,7 @@ def time(self):
         With 32-bit floats times will be incorrect by a few seconds
         when e.g. adding them to the reference time.
         """
-        met_ref = utils._time_ref_from_dict(self.meta)
+        met_ref = time_ref_from_dict(self.meta)
         met = Quantity(self['TIME'].astype('float64'), 'second')
         time = met_ref + met
         return time
@@ -619,7 +620,7 @@ def check_times(self):
         )
 
         telescope = self.dset.event_list.meta['TELESCOP']
-        met_ref = utils._time_ref_from_dict(self.dset.event_list.meta)
+        met_ref = time_ref_from_dict(self.dset.event_list.meta)
 
         if telescope in telescope_met_refs.keys():
             dt = (met_ref - telescope_met_refs[telescope])

gammapy/data/gti.py

@@ -4,6 +4,7 @@
 import numpy as np
 from astropy.units import Quantity
 from astropy.table import Table
+from ..utils.time import time_ref_from_dict
 from . import utils
 
 __all__ = ['GoodTimeIntervals']
@@ -44,13 +45,13 @@ def time_sum(self):
     @property
     def time_start(self):
         """GTI start times (`~astropy.time.Time`)."""
-        met_ref = utils._time_ref_from_dict(self.meta)
+        met_ref = time_ref_from_dict(self.meta)
         met = Quantity(self['START'].astype('float64'), 'second')
         return met_ref + met
 
     @property
     def time_stop(self):
         """GTI end times (`~astropy.time.Time`)."""
-        met_ref = utils._time_ref_from_dict(self.meta)
+        met_ref = time_ref_from_dict(self.meta)
         met = Quantity(self['STOP'].astype('float64'), 'second')
         return met_ref + met

gammapy/data/utils.py

@@ -2,7 +2,6 @@
 """Misc utility functions."""
 from __future__ import (absolute_import, division, print_function,
                         unicode_literals)
-from astropy.time import Time
 from astropy.coordinates import Angle, EarthLocation
 from astropy.units import Quantity
 
@@ -12,13 +11,3 @@ def _earth_location_from_dict(meta):
     lat = Angle(meta['GEOLAT'], 'deg')
     height = Quantity(meta['ALTITUDE'], 'meter')
     return EarthLocation(lon=lon, lat=lat, height=height)
-
-
-def _time_ref_from_dict(meta):
-    # Note: the `float` call here is to make sure we use 64-bit
-    mjd = float(meta['MJDREFI']) + float(meta['MJDREFF'])
-    # TODO: Is 'tt' a default we should put here?
-    scale = meta.get('TIMESYS', 'tt').lower()
-    # Note: we could call .copy('iso') or .replicate('iso')
-    # here if we prefer 'iso' over 'mjd' format in most places.
-    return Time(mjd, format='mjd', scale=scale)