Input DL2 format

[HealthyPear]

At the start of the project the DL2 input format was that of protopipe.
The final format to be read by pyirf should be the one provided by ctapipe.

For reference, here are some of the relevant issues/PR present there,

• DL2-like data: columns and format ctapipe#173
• DL2 data organization ctapipe#858
• Discussion on restructuring Event Container structure ctapipe#1165

Of course that work is still ongoing, so this is just an initial direction.

[HealthyPear]

Temporary input formats to be supported:

• lstchain (is it already synchronized with ctapipe @vuillaut ?)
• EventDisplay (in order to be able to crosscheck the basic first results - see EventDisplay root to DL2 hdf/fits converter #2 )
• CTA-MARS ?
• other single telescope data e.g. from NectarCam, ASTRI, ecc?

[kosack]

DL2 is fortunately much simpler than DL1, especially since you only need the DL2/Event/Subarray information (a single table with shower + discrimination parameters). It essentially will looks like the DL3 format, only in HDF5 and with possibly more than one shower reconstruction, so we'll need to think of some conventions for that.

I'd suggest designing the software similar to how IACT-Tools (or whatever the FACT one is called), where the columns you use as input are just a dict somewhere, so you can easily adapt to different formats.

One question from me: will this package do all of the parts of Stage 3 of the pipeline, or just the IRF parts? For that I mean, there is a first step to divide events into reconstruction classes and choose which reconstruction to use for each event (if multiple are included at once). It's not clear if that is part of this or not. If not, then what you really take as input is not DL2/Event, but really DL3/Event (where the final reconstruction has been chosen, event quality classification has been applied, and gamma/hadron discrimination has been applied)

[HealthyPear]

One question from me: will this package do all of the parts of Stage 3 of the pipeline, or just the IRF parts? For that I mean, there is a first step to divide events into reconstruction classes and choose which reconstruction to use for each event (if multiple are included at once). It's not clear if that is part of this or not. If not, then what you really take as input is not DL2/Event, but really DL3/Event (where the final reconstruction has been chosen, event quality classification has been applied, and gamma/hadron discrimination has been applied)

The package is meant to do what protopipe.perf does (even though in that case there are a lot of simplifications).

Then, if it has to be really an independent tool (even from ctapipe), I guess we can discuss about the part in which cuts are applied.
In my (maybe limited) view, the event classes or quality levels should be defined generally (like the definitions of the single IRFs), so it makes sense that this part is also done generally (if the purpose of pyirf is to be used also by other IACT facilities outside CTA).

[HealthyPear]

DL2 is fortunately much simpler than DL1, especially since you only need the DL2/Event/Subarray information (a single table with shower + discrimination parameters). It essentially will looks like the DL3 format, only in HDF5 and with possibly more than one shower reconstruction, so we'll need to think of some conventions for that.

If the purpose of pyirf is to be used by CTA and other related insturments, it is obvious that there will not be a sigle data format in the end.
Otherwise, the current plan remains unchanged.

I'd suggest designing the software similar to how IACT-Tools (or whatever the FACT one is called), where the columns you use as input are just a dict somewhere, so you can easily adapt to different formats.

Could you point this to me? I am afraid I am not familiar with it...

[maxnoe]

If the purpose of pyirf is to be used by CTA and other related insturments, it is obvious that there will not be a sigle data format in the end.

I don't think pyirf should concern itself to much with input file formats.

Provide library functions that take plain arrays of the needed quantities and some example scripts how to use these functions with file format X

Could you point this to me? I am afraid I am not familiar with it...

https://github.com/fact-project/aict-tools

[HealthyPear]

I don't think pyirf should concern itself to much with input file formats.

Provide library functions that take plain arrays of the needed quantities and some example scripts how to use these functions with file format X

But if the same variable in some other DL2 file (not produced with a ctapipe-based pipeline) has a different name, this has to be coded into pyirf no?

https://github.com/fact-project/aict-tools

Thank you!

[maxnoe]

But if the same variable in some other DL2 file (not produced with a ctapipe-based pipeline) has a different name, this has to be coded into pyirf no?

No, if you only provide library functions it is on the user side from where to read a given quantity.

calculate_sensitivity(
     e_est=my_df['gamma_energy_prediction'].to_numpy(),
     ...
)

If you want to provide command line utilities that take input files, the story is different.
Than we need to think about what to support and how flexible it should be.

But even in that case, we could get away with fits + hdf5 (maybe root with uproot) and some configuration values for column names.

[HealthyPear]

As per #36, now any specific input format translation to the internal data format of pyirf is left to the specific pipeline which will import pyirf functions.