Support reading sim_telarray DST0 files

[ParsonsRD]

Hi all,
The reading of the DST0 files (essentially dl1 data level) produced by Konrad's sim_telarray is currently broken. When reading these files the pedestal subtraction is currently not performed correctly. This seems to be due to the following code in simteleventsource.py:

pedestal = cam_mon["pedestal"] / cam_mon["n_ped_slices"]
In this case cam_mon["n_ped_slices"] will have a value of whatever the number of slices was in the original simulation (typically something large like 40), but in the DST0 the pedestal stored is that of the integrated signal, i.e. a number of slices = 1. So will result in a subtraction of a far too low value. As far as I know there is no flag in the data to confirm that you are reading a DST0 file so a fix is not obvious. One option could be to use the n_samples length of the waveform from

n_gains, n_pixels, n_samples = adc_samples.shape

As this should always be the same as the pedestal samples for a sim_telarray output file, but one for a DST0 file. But there may be safer alternatives out there.

[kosack]

Do you have an example DST0 file (ideally small) we could add to the test suite?

Right now SimTelEventSource only supports raw simtel files, but it should be easy to implement handling the DST format ones, either in SimTelEventSource itself, or a derived version. At the very least it should fail with an error if a DST0 file is used, so our current is_compatible() function is not sufficient.

I guess DST0 files just have the traces already integrated? Or are there other differences?

[ParsonsRD]

Sure attached is my 10 event example file that I use for the template generation test functions. I found this issue when updating the template generation to v0.11.0.

gamma_HESS_example.simhess.gz

[maxnoe]

Sorry for only coming back to this now. The file you provided only contains ADCSums, not ADCSamples:

In the case where no adc samples are found in the SimTelEventSource, adc_samples is filled from adc_sums, upshaed to a 3d array.

From todays point of view, this is certainly wrong and should not be done, instead we should directly fill the DL1 images using the ADCSums, right?

ctapipe/ctapipe/io/simteleventsource.py

Lines 762 to 765 in 6217b85

	for tel_id, telescope_event in telescope_events.items():
	adc_samples = telescope_event.get("adc_samples")
	if adc_samples is None:
	adc_samples = telescope_event["adc_sums"][:, :, np.newaxis]