DL2 data organization

[kbruegge]

This is related to #843 and #173. I'm writing this down to discuss about at the next meeting.

We need to find a way to organize our data within the produced output files.
Independent of whether we use fits or hdf5, we need to decide what groups/hdus we want in the file.

For storing the image parameters in the DL2 output we have some options:

• One group/hdu containing image
• One group/hdu per telescope type
• One group/hdu per actual telescope
• One group/hdu per subarray (whatever the exact definition of subarray might be).
• ...

For storing array-wide information (number of triggered tels per event, list of currently operating tels, true impact position, estimated impact position etc. )in the DL2 output we have the option :

• One group/hdu containing all per event monte carlo information and another one for trigger information
• One group/hdu containing all array-wide information on a per event basis

How do we organize monte carlo information in general:

• Separate monte carlo into a distinct group/hdu
• Prefix monte carlo information (mc_something) but keep them in the same group/hdu

[moralejo]

On Wed, 5 Dec 2018 at 10:35, Kai Brügge ***@***.***> wrote: This is related to #843 <#843> and #173 <#173>. I'm writing this down to discuss about at the next meeting. We need to find a way to organize our data within the produced output files. Independent of whether we use fits or hdf5, we need to decide what groups/hdus we want in the file. For storing the image parameters in the DL2 output we have some options: - One group/hdu containing image - One group/hdu per telescope type - One group/hdu per actual telescope - One group/hdu per subarray (whatever the exact definition of subarray might be). - ...

You meant DL1, right? DL2 should have no telescope-wise information. It is not clear to me what is the advantage of the grouping by telescope type or subarray. I assume all telescopes present in the file were performing the same observation and can (should) be combined in the subsequent steps of analysis. If sub-arrays are doing different observations the data streams will also be separate (I hope). "Per actual telescope" seems the best choice to me.

…

-- Abelardo Moralejo Olaizola Institut de Física d'Altes Energies Tel : +34 931641662 Fax: +34 935811938 Avís - Aviso - Legal Notice - (LOPD) - http://legal.ifae.es

-- Avís - Aviso - Legal Notice - (LOPD) - http://legal.ifae.es <http://legal.ifae.es/>

[maxnoe]

DL2 should have no telescope-wise information.

DL2 is image parameters, right? Image parameters are per telescope.

[jjlk]

DL2 is shower information, e.g. direction, impact parameter, energy, gamaness, etc. Here is CTA document discussing about this: https://jama.cta-observatory.org/attachment/4767/v/cta-high-level.pdf

[maxnoe]

Ah, sorry. The off-by-one error. I was thinking DL1

[maxnoe]

We will use the same layout as the DL1 output for now.