Setup integration tests pipeline

[HealthyPear]

These are NOT unit-tests.
These are NOT benchmarks.

These are tests checking that all protopipe scripts (aka the "steps" or "tools" of the pipeline) end correctly with proper output.

Current status

• protopipe-TRAINING
• protopipe-MODEL
• protopipe-DL2 (Implement DL2 integration tests #126)
• protopipe-DL3-EventDisplay (Add integration test for DL2-to-DL3 step #137)

Operations to perform

Each integration test should be contained in a template function that performs the following checks,

• the script doesn't crash
• the output file exists and it is not empty (provided that the input settings are correct)
• the output file has the correct format (probably this will be checked indirectly through the ctapipe and pyirf libraries)

Requirements

• an analysis configuration (the set of protopipe's config files)
  • DL1/DL2
  • Energy model
  • Classification model
  • DL3
• a server from which to store the simulated test dataset(s)
• setup the CI that steers these operations

Dataset(s)

We should have 2 datasets (CTA-N + CTA-S both baseline arrays) per production.

Each dataset could have the same composition as a normal analysis, so

• 10% gammas for energy training
• 10% gammas + 60% protons for classification training
• 80% gammas + 40% protons + 100% electrons for performance estimation

UPDATE: Actually it is not necessary, we can just use the same proportions between files and at the same time keeping the current pipeline workflow

They could be produced directly using simtel and not by cutting an already existing simtel file.
This to avoid situations in which a small test file doesn't contain enough triggered events.
So we could simulate e.g. 100 showers (=10% statistics) with E > 10TeV within a scatter radius of 150 meters.

UPDATE: Documentation has been updated accordingly (at least for Prod3b test files)

[HealthyPear]

Opened cta-observatory/ctapipe#1651 to test the download of testdata from the cc-in2p3 dataserver.

[HealthyPear]

after PR #111

• add proper diffuse test files for prod3b for both sites (see prod3b test diffuse file has no triggered events ctapipe-extra#34)
• add Prod5 files for both sites

so we would be able to

• test on diffuse data only
• test already on prod5