Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement new DL1 data format from ctapipe #25

Open
HealthyPear opened this issue Nov 7, 2019 · 1 comment
Open

Implement new DL1 data format from ctapipe #25

HealthyPear opened this issue Nov 7, 2019 · 1 comment
Labels
enhancement New feature or request
Milestone

Comments

@HealthyPear
Copy link
Member

HealthyPear commented Nov 7, 2019

This will represent a major enhancement of the capabilities of protopipe and possibly other pipelines to interface with other groups and compare results between themselves.
It will also represent a major change to the code structure of protopipe (mainly write_dl1.py and the code portions called by it).

Further information in cta-observatory/ctapipe#1163.

Of course, we depend on ctapipe's own timescale.
It would be much preferable to wait for ctapipe 0.8 and import this feature straight from it (better if in sub-modules of the complete DL1 script, which currently doesn't have all protopipe's features) than hardcode it and then take it out once that release will be available.

This represents a requirement for protopipe 0.3.

@HealthyPear HealthyPear added this to the Release 0.3 milestone Nov 7, 2019
@HealthyPear HealthyPear added the enhancement New feature or request label Nov 7, 2019
@HealthyPear HealthyPear changed the title Implement new DL1 data format from _ctapipe_ Implement new DL1 data format from ctapipe Nov 7, 2019
@HealthyPear HealthyPear moved this from In progress to To do in Pipeline features and enhancements Jan 31, 2020
@HealthyPear HealthyPear mentioned this issue Sep 7, 2020
4 tasks
@HealthyPear
Copy link
Member Author

Update:

PR #58 takes already care of some stuff:

  • all DL1 variables related to event data (meaning aside from metadata and such) is written by protopipe.scripts.data_training with the same names
  • the HDF5 structure is still the old one, but very simple to use (and actually sufficient, since we do not work with metadata yet) - a single table

As a result, calibration (#59) and image cleaning benchmarks can now be performed using directly ctapipe-stage1-tool, in which the relevant code tested from protopipe.pipeline.event_preparer has been migrated.

ctapipe will soon have a DL1 EventSource (much like the SimtelEventSource we use to open simtel files) - see cta-observatory/ctapipe#1357.

This will be very useful to simplify things here, because:

  • I can finally call ctapipe-stage1-tool as whole instead of the entire protopipe.pipeline.event_preparer and write the DL1 files (for which the only difference with the current "training" files obtained from protopipe.scripts.data_training are the table structure and the exclusive presence of DL1 information)
  • I can read the DL1 file and perform the remaining part of protopipe.pipeline.event_preparer providing the shower's direction information in a separate DL2 file (so the DL1 file can be re-used for e.g. different reconstruction algorithms)
  • protopipe.scripts.write_dl2 will open such DL2 file and add the remaining information (estimated particle type and energy)

@HealthyPear HealthyPear removed this from the Release 0.3 milestone Oct 22, 2020
@HealthyPear HealthyPear added this to the v0.6.0 milestone Nov 10, 2020
@HealthyPear HealthyPear linked a pull request Mar 9, 2021 that will close this issue
9 tasks
@HealthyPear HealthyPear modified the milestones: v0.5.0, v0.6.0 May 10, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Development

Successfully merging a pull request may close this issue.

1 participant