Skip to content

Commit

Permalink
Merge pull request #352 from cdeil/plot-irfs
Browse files Browse the repository at this point in the history
Add data management tools
  • Loading branch information
cdeil committed Oct 11, 2015
2 parents 19950dc + 41016e9 commit 6446a30
Show file tree
Hide file tree
Showing 49 changed files with 1,323 additions and 407 deletions.
4 changes: 4 additions & 0 deletions .continuous-integration/travis/setup_dependencies_common.sh
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,10 @@ fi
# CORE DEPENDENCIES
conda install --yes pytest Cython jinja2 psutil

# These dependencies are just needed for some functionality, they are not core.
# But the pytest runner fails with an ImportError if we don't put it here
conda install --yes click

# NUMPY
if [[ $NUMPY_VERSION == dev ]]
then
Expand Down
1 change: 1 addition & 0 deletions CHANGES.rst
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ Pull requests
- [#359] Add 1D spectrum analysis tool based on gammapy.hspec (Johannes King)
- [#351] Rewrite most parts of EnergyDispersion class (Johannes King)
- [#353] Add some scripts and examples (Christoph Deil)
- [#352] Add data management tools (Christoph Deil)
- [#348] Misc code cleanup (Christoph Deil)
- [#347] Add background cube model comparison plot script (Manuel Paz Arribas)
- [#342] Add gammapy-bin-image test (Christoph Deil)
Expand Down
1 change: 1 addition & 0 deletions docs/astro/source/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ The `gammapy.astro.source` module contains classes of source models, which can b

Source Models
=============

The following source models are available:

.. toctree::
Expand Down
16 changes: 9 additions & 7 deletions docs/background/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -29,17 +29,19 @@ TODO

Hello World.

Reference/API
=============

.. automodapi:: gammapy.background
:no-inheritance-diagram:
Using `gammapy.background`
--------------------------

Sub-pages
---------
If you'd like to learn more about using `gammapy.background`, read the following sub-pages:

.. toctree::
:maxdepth: 1

models
make_models

Reference/API
=============

.. automodapi:: gammapy.background
:no-inheritance-diagram:
2 changes: 2 additions & 0 deletions docs/image/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,8 @@ as input and output:
Using `gammapy.image`
=====================

If you'd like to learn more about using `gammapy.image`, read the following sub-pages:

.. toctree::
:maxdepth: 1

Expand Down
15 changes: 11 additions & 4 deletions docs/install.rst
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,8 @@ and then run these commands:
conda config --add channels astropy --add channels sherpa
conda install gammapy naima \
scipy matplotlib ipython-notebook cython
scipy matplotlib ipython-notebook \
cython click
We strongly recommend that you install the optional dependencies of Gammapy to have the full
functionality available:
Expand Down Expand Up @@ -168,7 +169,7 @@ The following packages are available:
sudo apt-get install \
python3-pip python3-scipy python3-matplotlib python3-skimage python3-sklearn \
python3-pandas python3-h5py python3-yaml ipython3-notebook python3-uncertainties \
python3-astropy
python3-astropy python3-click
The following packages have to be installed with pip:

Expand Down Expand Up @@ -209,7 +210,7 @@ The following packages are available via Macports:
sudo port install \
$PY-pip $PY-scipy $PY-matplotlib $PY-scikit-image $PY-scikit-learn \
$PY-pandas $PY-emcee $PY-h5py $PY-yaml $PY-ipython $PY-uncertainties \
$PY-healpy $PY-astropy
$PY-healpy $PY-astropy $PY-click $PY-cython
If you want some other Python version, use a different suffix (e.g. ``py27`` or ``py35``).
Expand Down Expand Up @@ -300,6 +301,13 @@ The required core dependencies of Gammapy are:
* `Numpy`_ - the fundamental package for scientific computing with Python
* `Astropy`_ - the core package for Astronomy in Python

We're currently using

* `click`_ for making command line tools
* `PyYAML`_ for `YAML <http://en.wikipedia.org/wiki/YAML>`__ data handling (config and results files
* `flask`_ and some Flask plugins for Gammapy web apps


Currently optional dependencies that are being considered as core dependencies:

* `Sherpa`_ for modeling / fitting (doesn't work with Python 3 yet)
Expand All @@ -325,7 +333,6 @@ Allowed optional dependencies:
* `iminuit`_ for fitting by optimization
* `emcee`_ for fitting by MCMC sampling
* `h5py`_ for `HDF5 <http://en.wikipedia.org/wiki/Hierarchical_Data_Format>`__ data handling
* `PyYAML`_ for `YAML <http://en.wikipedia.org/wiki/YAML>`__ data handling
* `healpy`_ for `HEALPIX <http://healpix.jpl.nasa.gov/>`__ data handling


Expand Down
6 changes: 4 additions & 2 deletions docs/irf/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -62,8 +62,10 @@ Getting Started
TODO: document


Using `gammapy.image`
=====================
Using `gammapy.irf`
===================

If you'd like to learn more about using `gammapy.irf`, read the following sub-pages:

.. toctree::
:maxdepth: 1
Expand Down
75 changes: 74 additions & 1 deletion docs/obs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,70 @@ and some book-keeping is required when running the analysis.
Getting Started
===============

Gammapy contains command line tools to manage data and work with subsets of observations.
This allows you to be up and running quickly and to focus on analysis.

* ``gammapy-data-manage`` -- Manage data locally and on servers
* ``gammapy-data-browse`` -- A web app to browse local data (stats and quick look plots)
* ``gammapy-obs-select`` -- Select observations of interest for a given analysis (TODO: rename, currently called ``gammapy-find-obs``)
* ``gammapy-obs-group`` -- Group observations (TODO: implement)

Download data
-------------

List which data you have available locally:

.. code-block:: bash
$ gammapy-data-manage status
If you're a H.E.S.S. member you can download data like this:

.. code-block:: bash
$ gammapy-data-manage status
For information on how to distribute data via a data server, see :ref:`obs_server`.

Simulate data
-------------

If not, you can simulate some data to have something to play around with:

.. code-block:: bash
$ gammapy-data-manage simulate hess01
Other relevant pages: :ref:`datasets_obssim`

Browse data
-----------

We have a web app that lets you browse the local data via a graphical user interface (GUI) in your web browser:

.. code-block:: bash
$ gammapy-data-browse
This is mostly useful for data producers and experts, not so much for end users.
We plan to add similar web apps for analysts to make it easy to browse analysis
inputs and results for a given target.

Select observations
-------------------

Once you have

.. code-block:: bash
$ gammapy-data-manage status
Observatory locations
---------------------

Gammapy contains the locations of gamma-ray telescopes:

.. code-block:: python
>>> from gammapy.obs import observatory_locations
Expand All @@ -29,14 +93,23 @@ Getting Started
>>> print(observatory_locations.HESS.geodetic)
(<Longitude 16.500222222222224 deg>, <Latitude -23.271777777772456 deg>, <Quantity 1835.0 km>)
This can be convenient e.g. for observation planning, or to transform between Alt-Az and RA-DEC coordinates.

TODO: We should probably update this to use the `astroplan.Observer` class,
which contains a similar observatory lookup database via `astroplan.Observer.at_site`.


Using `gammapy.obs`
=====================
===================

If you'd like to learn more about using `gammapy.obs`, read the following sub-pages:

.. toctree::
:maxdepth: 1

find_observations
observation_grouping
server

Reference/API
=============
Expand Down
50 changes: 50 additions & 0 deletions docs/obs/server.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
.. include:: ../references.txt

.. _obs_server:

Data Server
===========

The ``gammapy-data-manage`` tool expects that data is organised in a certain way,
both locally for end users and on servers for data providers.

This format and some background information is given on this page,
i.e. this is mostly useful info if you'd like to distribute data.

File and folder structure
-------------------------

The server must contain a ``data.yaml`` file, which declares which data
it provides, where it's located and a little bit of info about the data
that can help users decide if they want it.

Here's an example ``data.yaml`` file:

.. code-block:: yaml
TODO
It is strongly recommended, but not absolutely required, that all data
is stored in a single directory, with the ``data.yaml`` file at the top
level. You can use symlinks if the files are actually distributed in
several directories to make it appear as if they are in one directory.

The main advantage of having everything in one folder is that it's
easy to mirror all data using `rsync`_ to another machine (end-user or server):

.. code-block:: bash
rsync -uvrl <USERNAME>@<HOSTNAME>:<SERVER_DATA_ROOT_DIR> .
Config files
------------

TODO: probably need a local config file the user can edit, as well as an online
config file with pre-defined remotes

.. code-block:: yaml
TODO
3 changes: 3 additions & 0 deletions docs/references.txt
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@
.. _emcee: http://dan.iel.fm/emcee/current/
.. _Enrico: https://github.com/gammapy/enrico/
.. _ctapipe: https://github.com/cta-observatory/ctapipe
.. _click: http://click.pocoo.org/
.. _flask: http://flask.pocoo.org/

.. _MPIK Heidelberg: http://www.mpi-hd.mpg.de/mpi/en/start/
.. _Python for gamma-ray astronomy 2015: http://gammapy.github.io/PyGamma15/
Expand Down Expand Up @@ -67,3 +69,4 @@

.. _ctobssim: http://cta.irap.omp.eu/ctools-devel/reference_manual/ctobssim.html
.. _gtpsf: http://fermi.gsfc.nasa.gov/ssc/data/analysis/scitools/help/gtpsf.txt
.. _rsync: https://en.wikipedia.org/wiki/Rsync
1 change: 0 additions & 1 deletion examples/make_survey_map.py
Original file line number Diff line number Diff line change
Expand Up @@ -167,7 +167,6 @@ def make_source_catalog():

hdu = fits.open(TS_IMAGES)['sqrt_ts']
header = fits.getheader(REF_IMAGE)
# import IPython; IPython.embed()
wcs = WCS(header)

print('Running find_peaks ...')
Expand Down
Loading

0 comments on commit 6446a30

Please sign in to comment.