Merge pull request #54 from HealthyPear/documentation-grid_support

Update documentation for GRID support

docs/contribute/beforepushing.rst

@@ -7,14 +7,14 @@ Tests and unit-tests
 --------------------
 
 Being *protopipe* based on *ctapipe*, all the tools imported from the latter
-have been already tested and approved (remember that *protopipe* uses the
+have been already tested and approved (remember that *protopipe* uses always the
 latest version of *ctapipe* which has been released in the Anaconda framework).
 
 .. warning::
   This is not true for,
 
-  - hard-coded parts that protopipe had to modify from ctapipe,
-  - protopipe functions themselves (this is a big no-no and must be solved...)
+  - hard-coded parts that had to be modified from ctapipe,
+  - protopipe functions themselves (which will eventually migrate to ctapipe)
 
   Regarding the first point: given the difference in versions between the
   imported ctapipe and its development version, sometimes it's possible that, in
@@ -27,11 +27,10 @@ This test is in charge to detect if changes in
 `protopipe.pipeline.event_preparer` or `protopipe.scripts.write_dl1` produce any
 fatal behaviour or crash.
 
-It uses the `gamma_test_large` file of ctapipe (a CTA South array composed of
+It uses the `gamma_test_large` file of ctapipe (a Prod2 CTA South array composed of
 LSTCam, FlashCam and ASTRICam with about ~100 simulated showers).
 It is expected that an HDF5 file named `test_dl1.h5` is produced and is non-empty.
 
-
 The test can be executed directly from the main folder of protopipe by launching
 `pytest`. It is also automatically triggered by the continuous integration
 everytime a new pull-request is pushed to the repository, and its correct
@@ -49,36 +48,36 @@ Benchmarks
 
    benchmarks/benchmarks_DL1_calibration.ipynb
 
-The *protopipe* package contains the folder *benchmarks* (not a module)
-hosting the notebooks used for benchmarking protopipe.
+*protopipe* contains a folder named *benchmarks* (not a module)
+hosting the notebooks used for benchmarking.
 
-Their contents follows the development triggered by the ongoing
+Their contents follow the development triggered by the ongoing
 comparison between protopipe and CTA-MARS (see
 `this issue <https://github.com/cta-observatory/protopipe/issues/24>`__ and
 references therein for a summary).
 
 For this reason, the
 `MC sample <https://forge.in2p3.fr/attachments/download/63177/CTA-N_from_South.zip>`)
-to be used for these benchmarks needs to be the same.
+to be used for these benchmarks **needs** to be the same.
 
 The benchmarks are organised in 3 folders,
 
 - DL1
-  
+
   * `calibration <benchmarks/benchmarks_DL1_calibration.ipynb>`__ | *benchmarks_DL1_calibration.ipynb*
   * `image cleaning <benchmarks/benchmarks_DL1_image-cleaning.ipynb>`__ | *benchmarks_DL1_image-cleaning.ipynb*
-  
+
 - DL2
-  
+
   * `direction reconstruction <benchmarks/benchmarks_DL2_direction-reconstruction.ipynb>`__ | *benchmarks_DL2_direction-reconstruction.ipynb*
   * energy estimation
   * particle classification
-  
+
 - DL3
-  
+
   * cuts optimization
   * Instrument Response Functions
-  
+
 
 .. note::
    This part of protopipe is not meant to be kept here in the end, in order to

docs/contribute/gitrepo.rst

@@ -3,70 +3,67 @@
 The repository
 ==============
 
-It is the place from which both users and developers can monitor the
-status of the development of *protopipe*.
+Is useful for both basic users and developers to monitor the status of the
+development of *protopipe*.
 
-The best place where to start is the **projects** tab.
-Here, as of today, there are 3 open projects:
+Start from the **projects** tab, which currently consists of
 
 - *Next release*,
 - *Development of new algorithms*,
 - *Bugs*.
 
-All the projects do not point to specific deadlines, instead they are meant to
-give a continuous overview about the current activities.
+They don't come with specific deadlines, because they are meant to
+give a continuous overview regardless of versioning.
 
-If what you had in mind is already covered there, you can participate,
-otherwise you can open an issue yourself.
+Please, if what you have in mind is not already covered open a new issue.
 
 Next release
 ------------
 
-This project collects all open issues and pull-requests that are related to the
+This project collects all open issues and pull-requests related to the
 work needed for releasing a new version of the pipeline.
 It is organized in 4 sections:
 
-- *Summary issues*, these are not "issues" in the real sense of the word,
-  but rather GitHub issues that list enhancements which are all related to a particular subject
-- *To Do*, these are open issues that should trigger pull-requests (some can be as simple as a question),
+- *Summary issues*, lists of issues all related to a particular subject,
+- *To Do*, open issues that should trigger pull-requests (some can be as simple as a question),
 - *In progress*, pull-requests pushed by a user to the repository,
-- *Review in progress*, one or some of the mantainers have started reviewing
-  the pull-requests and/or discussing with the authors,
+- *Review in progress*, one or some of the maintainers started reviewing
+  the pull-request(s) and/or discussing with the authors,
 - *Reviewer approved*, the pull-request has been approved by the mantainers,
   but not yet merged into the master branch,
-- *Done*, the pull-request has been accepted and merged; any issue linked to it
-  (and likely appearing in the "To Do" section) will be automatically closed and will disappear.
+- *Done*, the pull-request has been accepted and merged; any linked issue
+  in the "To Do" column will automatically disappear.
 
-At any point, if an issue or pull-request gets re-opened (maybe because there was
-an error or an incompletness has been spotted) it will automatically reappear
-in the corresponding section of this project.
+At any point, if an issue or pull-request gets re-opened it will automatically
+reappear in the corresponding section of this project.
 
 Development of new algorithms
 -----------------------------
 
-By this we mean the development of new features that, even if not vital for the
-next release, are still needed by the collaboration/observatory and can be
+Features that, even if not vital for the
+next release, are still needed by some working groups and can be
 tested with this pipeline.
-An example of this is the support for the divergent pointing technique developed
-in ctapipe.
+An example of this is the support for the divergent pointing technique.
 
 This project has the same structure of the "Next release" project and works in
-the same way. In particular,
+the same way.
 
-- relevant issues and pull-requests should be labelled as **additional tool**,
-- the issues listed in the "Summary issues" column are expected to
-  be isolated from each other, so each issue is a algorithm/subject itself.
+In particular,
+
+- relevant issues and pull-requests should be labelled as ``additional tool``,
+- the "Summary issues" column are expected to
+  be isolated from each other (each one referring to a different algorithm).
 
 Bugs
 ----
 
-This is meant to be a tracker for bugs, but also for situations in which
-the code works (so technically not a bug), but either we discovered a limitation
-in performance or this has degraded for an unkown reason.
+A tracker for bugs, but also for situations in which
+the code works (so technically not a bug), but either a limitation or degradation
+in performance has been discovered.
 
 The project is divided in the following sections:
 
-- *Needs triage*, collects all open issues tagged either **bug** or **wrong behaviour**
+- *Needs triage*, collects all open issues labelled either ``bug`` or ``wrong behaviour``
   that have not been classified by priority,
 - *High priority*, open issues that previously needed triage, but that have been
   recognized to be fatal or urgent,

docs/contribute/instructions.rst

@@ -3,29 +3,28 @@
 Instructions
 ============
 
-These are some guidelines on how to contribute to *protopipe* through its
-repository in GitHub.
+These are some guidelines on how to contribute to *protopipe*.
 This of course makes sense only for the development branch, aka the *master*
 branch.
 
 This is usually done is 4 steps:
 
 1. you start using protopipe
 2. you find that either there is problem or *protopipe*
-   is missing a feature that important for your research
-3. you open an issue
-4. you open a pull-request
+   is missing a feature that is important for your research
+3. you open an issue (or pull-request, if you already have a solution)
 
 Open an issue
 -------------
 
-Technically you could open directly a pull-request, but is preferable to open an
-issue first, in order to warn others and trigger a possible discussion.
+It is always preferable to open an issue first, in order to warn others and
+trigger a discussion.
 This will be useful also to identify more precisely what needs to be done.
 
 If you are not able to do it, the administrators of the repository should **label
 your issue** depending on its nature.
-The labels most used in protopipe are quite self-explanatory:
+
+The labels normally used are quite self-explanatory:
 
 - bug
 - fix
@@ -35,55 +34,55 @@ The labels most used in protopipe are quite self-explanatory:
 - dependency update
 - summary
 
+An issue can have multiple labels.
 If you find that these are limited, you can propose new ones.
 
-Prepare your pull-request
--------------------------
+Prepare and open a pull-request
+-------------------------------
 
 This section assumes that you went through the installation for developers.
 
-When you want to fix a bug or develop something new:
-
-  1. update your **local** *master* branch with `git pull upstream master`
+  1. update your **local** *master* branch with ``git pull upstream master``
   2. create and move to a new **local** branch from your **local** *master* with
-     `git checkout -b your_branch`
+     ``git checkout -b your_branch``
   3. develop inside it
   4. push it to *origin*, thereby creating a copy of your branch also there
-  5. continue to develop and push until you feel ready
+  5. before pushing, please go through some checks (:ref:`beforepushing`)
   6. start a *pull request* using the web interface from *origin/your_branch*
      to *upstream/master*
 
     1. wait for an outcome
     2. if necessary, you can update or fix things in your branch because now
-       everything is traced
+       everything is traced!
        (**local/your_branch** --> **origin/your_branch** --> **pull request**)
 
+If your pull-request targets an issue, it should:
+
+- have the same labels of that issue,
+- if related to one ore more opened issues, its description should contain,
+
+ - the phrase `Closes #X #Y ...` where X is the number associated to the issue(s) if any,
+ - a reference to the issue, e.g. "as reported in #X ..." or similar
+
+This will keep things clean and organized, so when you or
+someone else land on the Projects page, the information is readily available
+and updated.
+
 .. Note::
 
-  If your developments take a relatively long time, consider to update periodically your **local** *master* branch.
+  If your developments take a relatively long time, consider to update
+  periodically your **local** *master* branch.
 
-  If in doing this you see that the files on which you are working on have been modified *upstream*,
+  If while doing this you see that the files on which you are working have been
+  modified *upstream*,
 
     * move into your **local** branch,
     * merge the new master into your branch ``git merge master``,
     * resolve eventual conflicts
     * push to origin
 
-  In this way, your pull request will be up-to-date with the master branch into which you want to merge your changes.
-  If your changes are relatively small and `you know what you are doing <https://www.atlassian.com/git/tutorials/merging-vs-rebasing>`_, you can use ``git rebase master``, instead of merging.
-
-Open your pull-request
-----------------------
-
-When you or someone else open a pull-request which targets this issue, he/she
-should:
-
-- mirror the labels,
-- if it's related to one ore more open issues, add in its description,
-
-  - the phrase `Closes #X #Y ...` where X is the number associated to the issue(s) if any,
-  - a reference to the issue, e.g. "as reported in #X" or similar
-
-This will keep things clean and organized, so when you or
-someone else land on the Projects page, the information is readily available
-and updated.
+  In this way, your pull request will be up-to-date with the master branch into
+  which you want to merge your changes.
+  If your changes are relatively small and
+  `you know what you are doing <https://www.atlassian.com/git/tutorials/merging-vs-rebasing>`_,
+  you can use ``git rebase master``, instead of merging.

docs/index.rst

@@ -14,8 +14,6 @@ but since Sep 23, 2019 is also open for development by other members of the CTA
 
 The code is hosted in a `GitHub repository <https://github.com/cta-observatory/protopipe>`__, to
 which this documentation is linked.
-The API documentation gets automatically updated as soon as a new contribution
-gets accepted and merged into the public code.
 
 .. warning::
   This is not yet stable code, so expect large and rapid changes.

docs/install/basic.rst

@@ -10,7 +10,7 @@ Steps for installation:
 
   1. uncompress the file which is always called *protopipe-X.Y.Z* depending on version,
   2. enter the folder ``cd protopipe-X.Y.Z``
-  3. create a dedicated environment with ``conda env create -f protopipe_environment.yml``
+  3. create a dedicated environment with ``conda env create -f environment.yml``
   4. activate it with ``conda activate protopipe``
   5. install *protopipe* itself with ``python setup.py install``.
 
@@ -20,4 +20,11 @@ Steps for installation:
 
   The development version could disrupt functionalities that were working for
   you, but the latest released version could lack some of those you need.
-  To know how to install the latest development version go to :ref:`install-developer`.
+
+  To install the latest development version go to :ref:`install-developer`.
+
+Next steps:
+
+ * get accustomed to the basic pipeline workflow (:ref:`use-pipeline`),
+ * then make your own complete analysis (:ref:`use-grid`),
+ * for bugs and new features, please contribute to the project (:ref:`contribute`).

docs/install/developers.rst

@@ -10,5 +10,11 @@ If you want to use *protopipe* and also contribute to its development, follow th
   3. execute points 3 and 4 in the instructions for :ref:`install-basic`.
   4. install *protopipe* itself in developer mode with ``python setup.py develop``
 
-In this way, When you change branch or modify the code, the scripts will always
-refer to the that version of the code and you will not need to install it again.
+In this way, you will always use the version of the source code on which you
+are working.
+
+Next steps:
+
+ * get accustomed to the basic pipeline workflow (:ref:`use-pipeline`),
+ * then make your own complete analysis (:ref:`use-grid`),
+ * for bugs and new features, please contribute to the project (:ref:`contribute`).

docs/install/grid.rst

@@ -0,0 +1,41 @@
+.. _install-grid:
+
+Grid environment
+================
+
+General requirements
+--------------------
+
+Regardless of what operating system you are using, in order to be able to use
+*protopipe* on the grid you will need to be able to interface with it.
+
+Go through the initial steps described
+`here <https://forge.in2p3.fr/projects/cta_dirac/wiki/CTA-DIRAC_Users_Guide>`__.
+
+In the following it is then assumed that you have:
+
+* a Grid identity and a CTA VO registration,
+* a certificate and a user key in the hidden folder ``$HOME/.globus``
+
+Installation instructions
+-------------------------
+
+The installation instructions are the following:
+
+* get the interface code (``git clone https://github.com/HealthyPear/protopipe-grid-interface.git``),
+* create and enter a folder where to work,
+* **(only Windows/macos users)** copy the ``VagrantFile`` from the parent folder of `protopipe-grid-interface`,
+* **(only Windows/macos users)** edit the first arguments of lines from 47 to 50 according to your local setup,
+* create the `data` shared folder to interact externally with the analysis products,
+* **(only Windows/macos users)** enter in the virtual environment (``vagrant up && vagrant ssh``),
+* get the Singularity container (``singularity pull --name CTADIRAC_with_protopipe.sif shub://HealthyPear/CTADIRAC``),
+* enter in it (``singularity shell CTADIRAC_with_protopipe.sif``).
+
+From here you should be able to activate the DIRAC environment (``dirac-proxy-init``).
+The container provides the latest version of CTADIRAC and the necessary python
+modules for the `protopipe-grid` code to run.
+Now you can proceed with the analysis workflow (:ref:`use-grid`).
+
+.. Note::
+  For Linux users, the host's $HOME coincides with the guests' one by default.
+  For macos/Windows users, this has been set when you edited the VagrantFile.