Merge branch 'master' into arraydir

.gitignore

@@ -24,6 +24,11 @@ MANIFEST
 docs/api
 docs/_build
 
+# Benchmarks
+benchmarks/DL1/plots*
+benchmarks/DL2/plots*
+benchmarks/DL3/plots*
+
 # Eclipse editor project files
 .project
 .pydevproject
@@ -76,7 +81,9 @@ distribute-*.tar.gz
 *.h5
 *.hdf5
 *.fits
-*.fits.gz
+irf.fits.gz
+table_best_cutoff.fits
+*.pkl.gz
 
 *.pstats
 target

.travis.yml

@@ -30,13 +30,13 @@ before_install:
     - conda info -a # Useful for debugging any issues with conda
 
 install:
-    - conda env create -f protopipe_environment.yml
+    - conda env create -f environment.yml
     - conda activate protopipe
     - pip install travis-sphinx codecov pytest-cov
     - python setup.py install
 
 script:
-    # - pytest --cov=protopipe
+    - pytest --cov=protopipe
     - travis-sphinx -v --outdir=docs/_build build -n --source=docs/
 
 after_script:

CHANGELOG.rst

@@ -0,0 +1,164 @@
+.. X.Y.Z (Mon DD, YYYY)
+.. -------------------
+..
+.. Summary
+.. +++++++
+..
+.. - Released Mon DD, YYYY
+.. - X contributor(s)
+.. - X pull requests
+..
+.. **Description**
+..
+.. bla bla.
+..
+.. **Contributors:**
+..
+.. In alphabetical order by first name:
+..
+.. - ...
+..
+.. Pull Requests
+.. +++++++++++++
+..
+.. - [#X] Title (Author)
+
+.. _protopipe_0p3_release:
+
+0.3 (Unreleased)
+-----------------
+
+Planned for early summer 2020.
+
+.. _gammapy_0p2p1_release:
+
+`0.2.1 <https://github.com/cta-observatory/protopipe/releases/tag/v0.2.1>`__ (Oct 28, 2019)
+-------------------------------------------------------------------------------------------
+
+Summary
++++++++
+
+- Released Oct 28, 2019
+- 1 contributor
+- 1 pull requests
+
+**Description**
+
+The ctapipe-based cleaning algorithm for the biggest cluster was crashing in
+case of cleaned images with no surviving pixel clusters.
+
+**Contributors:**
+
+In alphabetical order by first name:
+
+- Michele Peresano
+
+Pull Requests
++++++++++++++
+
+- [#16] Bugfix: Closes #15 (Michele Peresano)
+
+`0.2.0 <https://github.com/cta-observatory/protopipe/releases/tag/v0.2.0>`__ (Oct 24, 2019)
+-------------------------------------------------------------------------------------------
+
+Summary
++++++++
+
+- Released Oct 24, 2019
+- 3 contributor(s)
+- 7 pull requests
+
+**Description**
+
+*protopipe* 0.2 now fully supports the stable release of *ctapipe* 0.7.0. The main improvements involve the calibration process (high gain selected by default), the direction reconstruction and new camera-type labels.
+
+Code based on *pywi*/*pywi-cta* libraries, relevant for wavelet-based image cleaning, has been removed in favor of *ctapipe* or made completely optional where needed. Wavelet cleaning is still optional but will need those two libraries to be additionally installed. Tailcut-based cleaning is now faster.
+
+The README has been improved with installation, basic use, and developer instructions. Dependencies are listed in `protopipe_environment.yaml` and have been simplified.
+
+The auxiliary scripts `merge_tables.py` and `merge.sh` have been added to allow merging of DL1 and DL2 HDF5 tables.
+
+The `mars_cleaning_1st_pass` method is now imported from _ctapipe_. Novel code using the largest cluster of survived pixels (`number_of_islands` and `largest_island` methods in the `event_preparer` module) has been hardcoded in _protopipe_ and will disappear with the next release of _ctapipe_.
+
+Model estimators now load the camera types directly from the `analysis .yaml` configuration file.
+
+**Contributors:**
+
+In alphabetical order by first name:
+
+- Alice Donini
+- Michele Peresano
+- Thierry Stolarczyk
+
+Pull Requests
++++++++++++++
+
+This list is incomplete. Small improvements and bug fixes are not listed here.
+The complete list is found `here <https://github.com/gammapy/gammapy/pulls?q=is%3Apr+milestone%3A0.16+is%3Aclosed>`__.
+
+- [#9] Update image cleaning and make wavelet-based algorithms independent
+- [#8] Import CTA-MARS 1st pass cleaning from ctapipe
+
+`0.1.1 <https://github.com/cta-observatory/protopipe/releases/tag/v0.1.1>`__ (Oct 1, 2019)
+------------------------------------------------------------------------------------------
+
+Summary
++++++++
+
+- Released Oct 1, 2019
+- X contributor(s)
+- X pull request(s)
+
+**Description**
+
+The `write_dl1` and `write_dl2` tools can now save an additional file through the flag `--save-images` when applied to a single run. This file will contain the original and calibrated (after gain selection) photoelectron images per event.
+
+A new method `save_fig` has been introduced in the `utils` module, so that `model_diagnostic` can save images also in PNG format.
+
+Additional docstrings and PEP8 formatting have been added throughout the code.
+
+**Contributors:**
+
+In alphabetical order by first name:
+
+- ...
+
+Pull Requests
++++++++++++++
+
+The development of *protopipe* on GitHub started out directly in the master branch,
+so there are no pull request we can list here.
+
+`0.1.0 <https://github.com/cta-observatory/protopipe/releases/tag/v0.1.0>`__ (Sep 23, 2019)
+-------------------------------------------------------------------------------------------
+
+Summary
++++++++
+
+- Released Sep 23, 2019
+- 6 contributor(s)
+- 1 pull request(s)
+
+**Description**
+
+First version of *protopipe* to be publicly release on GitHub.
+This version is based on ctapipe 0.6.2 (conda package stable version).
+Its performance has been shown in a
+`presentation <https://indico.cta-observatory.org/event/1995/contributions/19991/attachments/15559/19825/CTAC_Lugano_2019_Peresano.pdf>`__
+at the CTAC meeting in Lugano 2019.
+
+**Contributors:**
+
+In alphabetical order by first name:
+
+- David Landriu
+- Julien Lefacheur
+- Karl Kosack
+- Michele Peresano
+- Thomas Vuillaume
+- Tino Michael
+
+Pull Requests
++++++++++++++
+
+- [#2] Custom arrays, example configs and aux scripts (M.Peresano)

MANIFEST.in

@@ -1,5 +1,6 @@
 include README.rst
 
+recursive-include protopipe/aux
 recursive-include protopipe/pipeline/*.py
 
-global-exclude *.pyc
+global-exclude *.pyc

README.md

@@ -1,10 +1,27 @@
 protopipe
 =========
 
-![Code Quality](https://api.codacy.com/project/badge/Grade/32f2fb2df3154fa1838c765d4f9110ba)
+[![Travis CI][image1]][hyperlink1]
+[![Code Quality][image2]][hyperlink2]
+[![Coverage][image3]][hyperlink3]
+<!---[![Zenodo][image4]][hyperlink4]--->
+
+[hyperlink1]: https://travis-ci.org/cta-observatory/protopipe
+[image1]: https://travis-ci.org/cta-observatory/protopipe.svg?branch=master
+[hyperlink2]: https://app.codacy.com/manual/HealthyPear/protopipe/dashboard
+[image2]: https://api.codacy.com/project/badge/Grade/32f2fb2df3154fa1838c765d4f9110ba
+[hyperlink3]: https://codecov.io/gh/cta-observatory/protopipe
+[image3]: https://codecov.io/gh/cta-observatory/protopipe/branch/master/graph/badge.svg
+<!---
+[hyperlink4]: https://travis-ci.org/cta-observatory/protopipe
+[image4]: https://travis-ci.org/cta-observatory/protopipe.svg?branch=master
+--->
 
 A pipeline prototype for the Cherenkov Telescope Array.
 
+* Code: https://github.com/cta-observatory/protopipe
+* Docs: https://cta-observatory.github.io/protopipe/
+
 Installation
 ------------
 
@@ -14,27 +31,31 @@ Get the source code and create the required basic conda environment:
 
           git clone https://github.com/cta-observatory/protopipe.git
           cd protopipe
-          conda env create -f protopipe_environment.yml
+          conda env create -f environment.yml
           conda activate protopipe
-          
-In case you have already an environment 'myenv' set up for that kind of development, use:
 
-          conda env update -n myenv -f protopipe_environment.yml
-          
+In case you are updating your environment from a previously existing one, named e.g. 'myenv', use:
+
+          conda env update -n myenv -f environment.yml
+
 This environment contains the bare minimum in order to run the scripts and build the documentation.
 
-It doesn't take into account any additional tool you could use later on (it is suggested to install _ipython_, _jupyter_ and _vitables_, especially if you want to contribute to the code).
+It doesn't take into account any additional tool you could use later on (it is suggested to install _jupyter_ and _vitables_, especially if you want to contribute to the code).
 
-Next you need to install _protopipe_ itself (_protopipe_ is the folder where the code was expanded. The name differs if you downloaded a released tagged version, it can be _protopipe-0.2_ for instance) :
+Next you need to install _protopipe_ itself (the main folder will be named _protopipe_ for the development version, or _protopipe-X.Y.Z_ for a released tagged version) :
 
           cd protopipe
-          python setup.py develop
 
-This will let you make changes to your local git repository without the need to update your environment every time.
-In the abscence of a _conda_ or _pip_ installation made avalaible, to install a released version (no further development), use :
+To install a _released_ version with no further development, use:
 
           python setup.py install
 
+To use and develop the latest _development_ version, use:
+
+          python setup.py develop
+
+The second command will let you make changes to your local git repository without the need to update your environment every time.
+
 Remember that the environment needs to be activated in order for _protopipe_ to work.
 This procedure has been successfully tested on macOS (10.10.5 & 10.14.6) and on Scientific Linux 7.
 
@@ -46,20 +67,18 @@ From the main folder go down to the documentation repository and create the docu
           cd docs
           make html
 
-You will probably get some harmless warnings.
+If you are (or plan to be) a developer and you get some warnings when you build the documention, before pushing a Pull Request please try to fix them.
+
 The initial page is stored in _ _build/html/index.html_, which you can open using your favorite internet browser.
 
 Test if it works
 -----------------
-Before starting to use _protopipe_, be sure to be inside the relevant environment (e.g. `conda activate protopipe`).
-Get a _simtel_ Monte Carlo file obtained with _prod3b_ with only _LSTCam_ and _NectarCam_ cameras.
-Any other kind of cameras in the file would lead to a crash, see issues #23. 
-Note that some generic La Palma files can contain _FlashCam_ cameras.
-The write_dl1.py script will prodcue a DL1 output from the _simtel_ data.
+Before starting to use _protopipe_, be sure to be inside the relevant conda environment (should be named `protopipe` by default) and try to print the help documentation for one of the scripts, e.g.,
 
           cd protopipe/scripts
           python write_dl1.py -h
 
+
 Analysis chain general description
 ----------------------------------
 The following instructions refer to local use of protopipe, which doesn't involve the use of the DIRAC computing grid (you can ignore the grid.yaml configuration file).