Merge pull request #352 from cdeil/plot-irfs

Add data management tools

.continuous-integration/travis/setup_dependencies_common.sh

@@ -20,6 +20,10 @@ fi
 # CORE DEPENDENCIES
 conda install --yes pytest Cython jinja2 psutil
 
+# These dependencies are just needed for some functionality, they are not core.
+# But the pytest runner fails with an ImportError if we don't put it here
+conda install --yes click
+
 # NUMPY
 if [[ $NUMPY_VERSION == dev ]]
 then

CHANGES.rst

@@ -50,6 +50,7 @@ Pull requests
 - [#359] Add 1D spectrum analysis tool based on gammapy.hspec (Johannes King)
 - [#351] Rewrite most parts of EnergyDispersion class (Johannes King)
 - [#353] Add some scripts and examples (Christoph Deil)
+- [#352] Add data management tools (Christoph Deil)
 - [#348] Misc code cleanup (Christoph Deil)
 - [#347] Add background cube model comparison plot script (Manuel Paz Arribas)
 - [#342] Add gammapy-bin-image test (Christoph Deil)

docs/astro/source/index.rst

@@ -14,6 +14,7 @@ The `gammapy.astro.source` module contains classes of source models, which can b
 
 Source Models
 =============
+
 The following source models are available:
 
 .. toctree::

docs/background/index.rst

@@ -29,17 +29,19 @@ TODO
 
 Hello World.
 
-Reference/API
-=============
-
-.. automodapi:: gammapy.background
-    :no-inheritance-diagram:
+Using `gammapy.background`
+--------------------------
 
-Sub-pages
----------
+If you'd like to learn more about using `gammapy.background`, read the following sub-pages:
 
 .. toctree::
    :maxdepth: 1
 
    models
    make_models
+
+Reference/API
+=============
+
+.. automodapi:: gammapy.background
+    :no-inheritance-diagram:

docs/image/index.rst

@@ -34,6 +34,8 @@ as input and output:
 Using `gammapy.image`
 =====================
 
+If you'd like to learn more about using `gammapy.image`, read the following sub-pages:
+
 .. toctree::
    :maxdepth: 1
 

docs/install.rst

@@ -36,7 +36,8 @@ and then run these commands:
 
     conda config --add channels astropy --add channels sherpa
     conda install gammapy naima \
-        scipy matplotlib ipython-notebook cython
+        scipy matplotlib ipython-notebook \
+        cython click
 
 We strongly recommend that you install the optional dependencies of Gammapy to have the full
 functionality available:
@@ -168,7 +169,7 @@ The following packages are available:
     sudo apt-get install \
         python3-pip python3-scipy python3-matplotlib python3-skimage python3-sklearn \
         python3-pandas python3-h5py python3-yaml ipython3-notebook python3-uncertainties \
-        python3-astropy
+        python3-astropy python3-click
 
 The following packages have to be installed with pip:
 
@@ -209,7 +210,7 @@ The following packages are available via Macports:
     sudo port install \
         $PY-pip $PY-scipy $PY-matplotlib $PY-scikit-image $PY-scikit-learn \
         $PY-pandas $PY-emcee $PY-h5py $PY-yaml $PY-ipython $PY-uncertainties \
-        $PY-healpy $PY-astropy
+        $PY-healpy $PY-astropy $PY-click $PY-cython
 
 
 If you want some other Python version, use a different suffix (e.g. ``py27`` or ``py35``).
@@ -300,6 +301,13 @@ The required core dependencies of Gammapy are:
 * `Numpy`_ - the fundamental package for scientific computing with Python
 * `Astropy`_ - the core package for Astronomy in Python
 
+We're currently using
+
+* `click`_ for making command line tools
+* `PyYAML`_ for `YAML <http://en.wikipedia.org/wiki/YAML>`__ data handling (config and results files
+* `flask`_ and some Flask plugins for Gammapy web apps
+
+
 Currently optional dependencies that are being considered as core dependencies:
 
 * `Sherpa`_ for modeling / fitting (doesn't work with Python 3 yet)
@@ -325,7 +333,6 @@ Allowed optional dependencies:
 * `iminuit`_ for fitting by optimization
 * `emcee`_ for fitting by MCMC sampling
 * `h5py`_ for `HDF5 <http://en.wikipedia.org/wiki/Hierarchical_Data_Format>`__ data handling
-* `PyYAML`_ for `YAML <http://en.wikipedia.org/wiki/YAML>`__ data handling
 * `healpy`_ for `HEALPIX <http://healpix.jpl.nasa.gov/>`__ data handling
 
 

docs/irf/index.rst

@@ -62,8 +62,10 @@ Getting Started
 TODO: document
 
 
-Using `gammapy.image`
-=====================
+Using `gammapy.irf`
+===================
+
+If you'd like to learn more about using `gammapy.irf`, read the following sub-pages:
 
 .. toctree::
    :maxdepth: 1

docs/obs/index.rst

@@ -21,6 +21,70 @@ and some book-keeping is required when running the analysis.
 Getting Started
 ===============
 
+Gammapy contains command line tools to manage data and work with subsets of observations.
+This allows you to be up and running quickly and to focus on analysis.
+
+* ``gammapy-data-manage`` -- Manage data locally and on servers
+* ``gammapy-data-browse`` -- A web app to browse local data (stats and quick look plots)
+* ``gammapy-obs-select`` -- Select observations of interest for a given analysis (TODO: rename, currently called ``gammapy-find-obs``)
+* ``gammapy-obs-group`` -- Group observations (TODO: implement)
+
+Download data
+-------------
+
+List which data you have available locally:
+
+.. code-block:: bash
+
+   $ gammapy-data-manage status
+
+If you're a H.E.S.S. member you can download data like this:
+
+.. code-block:: bash
+
+   $ gammapy-data-manage status
+
+For information on how to distribute data via a data server, see :ref:`obs_server`.
+
+Simulate data
+-------------
+
+If not, you can simulate some data to have something to play around with:
+
+.. code-block:: bash
+
+   $ gammapy-data-manage simulate hess01
+
+Other relevant pages: :ref:`datasets_obssim`
+
+Browse data
+-----------
+
+We have a web app that lets you browse the local data via a graphical user interface (GUI) in your web browser:
+
+.. code-block:: bash
+
+   $ gammapy-data-browse
+
+This is mostly useful for data producers and experts, not so much for end users.
+We plan to add similar web apps for analysts to make it easy to browse analysis
+inputs and results for a given target.
+
+Select observations
+-------------------
+
+Once you have
+
+.. code-block:: bash
+
+   $ gammapy-data-manage status
+
+
+Observatory locations
+---------------------
+
+Gammapy contains the locations of gamma-ray telescopes:
+
 .. code-block:: python
 
    >>> from gammapy.obs import observatory_locations
@@ -29,14 +93,23 @@ Getting Started
    >>> print(observatory_locations.HESS.geodetic)
    (<Longitude 16.500222222222224 deg>, <Latitude -23.271777777772456 deg>, <Quantity 1835.0 km>)
 
+This can be convenient e.g. for observation planning, or to transform between Alt-Az and RA-DEC coordinates.
+
+TODO: We should probably update this to use the `astroplan.Observer` class,
+which contains a similar observatory lookup database via `astroplan.Observer.at_site`.
+
+
 Using `gammapy.obs`
-=====================
+===================
+
+If you'd like to learn more about using `gammapy.obs`, read the following sub-pages:
 
 .. toctree::
    :maxdepth: 1
 
    find_observations
    observation_grouping
+   server
 
 Reference/API
 =============

docs/obs/server.rst

@@ -0,0 +1,50 @@
+.. include:: ../references.txt
+
+.. _obs_server:
+
+Data Server
+===========
+
+The ``gammapy-data-manage`` tool expects that data is organised in a certain way,
+both locally for end users and on servers for data providers.
+
+This format and some background information is given on this page,
+i.e. this is mostly useful info if you'd like to distribute data.
+
+File and folder structure
+-------------------------
+
+The server must contain a ``data.yaml`` file, which declares which data
+it provides, where it's located and a little bit of info about the data
+that can help users decide if they want it.
+
+Here's an example ``data.yaml`` file:
+
+.. code-block:: yaml
+
+    TODO
+
+It is strongly recommended, but not absolutely required, that all data
+is stored in a single directory, with the ``data.yaml`` file at the top
+level. You can use symlinks if the files are actually distributed in
+several directories to make it appear as if they are in one directory.
+
+The main advantage of having everything in one folder is that it's
+easy to mirror all data using `rsync`_ to another machine (end-user or server):
+
+.. code-block:: bash
+
+ rsync -uvrl <USERNAME>@<HOSTNAME>:<SERVER_DATA_ROOT_DIR> .
+
+
+Config files
+------------
+
+TODO: probably need a local config file the user can edit, as well as an online
+config file with pre-defined remotes
+
+.. code-block:: yaml
+
+    TODO
+
+

docs/references.txt

@@ -35,6 +35,8 @@
 .. _emcee: http://dan.iel.fm/emcee/current/
 .. _Enrico: https://github.com/gammapy/enrico/
 .. _ctapipe: https://github.com/cta-observatory/ctapipe
+.. _click: http://click.pocoo.org/
+.. _flask: http://flask.pocoo.org/
 
 .. _MPIK Heidelberg: http://www.mpi-hd.mpg.de/mpi/en/start/
 .. _Python for gamma-ray astronomy 2015: http://gammapy.github.io/PyGamma15/
@@ -67,3 +69,4 @@
 
 .. _ctobssim: http://cta.irap.omp.eu/ctools-devel/reference_manual/ctobssim.html
 .. _gtpsf: http://fermi.gsfc.nasa.gov/ssc/data/analysis/scitools/help/gtpsf.txt
+.. _rsync: https://en.wikipedia.org/wiki/Rsync

examples/make_survey_map.py

@@ -167,7 +167,6 @@ def make_source_catalog():
 
     hdu = fits.open(TS_IMAGES)['sqrt_ts']
     header = fits.getheader(REF_IMAGE)
-    # import IPython; IPython.embed()
     wcs = WCS(header)
 
     print('Running find_peaks ...')