Upgrade calibration benchmarks

[HealthyPear]

Description:

• now we don't need protopipe for this part, we can use directly ctapipe-stage1-process with proper configuration to mimic CTA-MARS
• added a new method to find optimized cleaning thresholds which substitutes the old one
• all Prod3b cameras are now supported (but some, like e.g. FlashCam, not optimized)

TO DO

• complete tests
• fix Travis error finding config file
• Update documentation
• Merge Upgrade of data training #58

[review-notebook-app]

Check out this pull request on 

Review Jupyter notebook visual diffs & provide feedback on notebooks.

---

Powered by ReviewNB

[codecov]

Codecov Report

Merging #59 (d4d31bb) into master (3225536) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #59   +/-   ##
=======================================
  Coverage   31.66%   31.66%           
=======================================
  Files          20       20           
  Lines        2201     2201           
=======================================
  Hits          697      697           
  Misses       1504     1504           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3225536...d4d31bb. Read the comment docs.

[HealthyPear]

I might have mistakenly removed an image cleaning benchmark, but it was anyway an older version with respect the one contained in PR #76 .

Please, until PR #62 don't mind the readthedocs check as it is broken at this stage (and is anyway only a nice thing I wanted to add as a backup for the docs).

[review-notebook-app]

View / edit / reply to this conversation on ReviewNB

kosack commented on 2020-12-01T11:18:09Z
----------------------------------------------------------------

For accessing tables in DL1 files, you should now use read_table (which reads and returns an astropy Table directly, no need for extra conversion, and units are handled automatically:

from ctapipe.io import read_table

all_mc_images = read_table(filename, "/simulation/event/telescope/images")

For loading all the telescope tables, it's also easy, since you can load the subarray first:

subarray = SubarrayDescription.from_hdf(filename)

for tel_id in subarray.tel_ids:

tab = read_table(filename, f"/dl1/event/telescope/images/tel_{tel_id:03d}")

[kosack]

a few comments in general:

Binary files

should not really include binary files in the repo, rather download them from elsewhere. (like the ROOT files), not sure if those are allowed to be exposed publicly anyhow. Might be a good reason to move this to GitLab and not public.

In benchmarks_DL1_calibration.ipynb and elsewhere

The suffix is not used in this function, it's just returned as is, so redundant

def load_reset_simtel(indir = "./", fileName = "gamma_20deg_180deg_run100___cta-prod3-demo-2147m-LaPalma-baseline.simtel.gz", max_events=None, config="test"):
    """(Re)load the simtel file for all events and telescopes."""
    source = event_source(input_url=f"{indir}/{fileName}", max_events=max_events)
    suffix = config # all generated plots will have this as a suffix in their name
    return source, suffix

It would be better to drop this function and use with statements when you use the event source later, so that files are correctly closed. E.g.:

with event_source(f"{indir}/{filename}") as source:
    # do stuff...

even better and less error prone, store indir and filename as pathlib.Path objects, so you can do event_source(indir / filename) and it will work on both linux and windows. You can just turn them into paths using indir = pathlib.Path(indir).

tables.open is never closed

In many places you use tables.open, but never close the file after (leading to memory and file handle leaks). Better to use with again, or just use ctapipe.io.read_table to get tables, which handles it internally

from ctapipe.io import read_table
filepath = Path(indir) / filename
true_images = read_table(filepath, f"/simulation/event/telescope/images/{tel}")
reco_images = read_table(filepath, f"/dl1/event/telescope/images/{tel}")

# or if you don't want to open/close twice, you can do:
with tables.open_file(filepath) as h5file:
    true_images = read_table(h5file, f"/simulation/event/telescope/images/{tel}")
    reco_images = read_table(h5file, f"/dl1/event/telescope/images/{tel}")

Note you then get back Astropy QTables, so would need to change a bit the code using them, but in general it will simplify that code (reco_images['image'] instead of reco_images.col('image')), and no more need to do true_images = Table(trueimages[:]) since that is done for you already

Also, it may be easier to read to explicitly show the table name to tel_id relationship:

true_images = read_table(h5file, f"/simulation/event/telescope/images/tel_{tel_id:03d}")