Comparison with CTA-MARS: Cuts optimization

[HealthyPear]

Context

Some of these steps are different between CTA-MARS and EventDisplay.
We should save as much information as possible in terms of what operations are missing in our API.

What CTA-MARS does

Paraphrasing from the IRF document:

In each energy bin, cuts in:

• multiplicity (scanned from >= 2 to >= 9 telescopes of any type),
• Hadronness (from 0.5 to 0.95 gamma efficiency)
• event direction (from 0.5 to 0.95 gamma efficiency)

are optimized in order to achieve the best possible (pointlike- source) gamma-ray flux sensitivity (against the background of cosmic-ray electrons and protons), according to the standard definition within CTA ( 5 sigma, >= 10 events, >= 5% of residual background).

Current situation
(to be updated as the discussion in this issue grows)

• Multiplicity cut ( from pyirf)

Questions for @moralejo :

• for each energy bin you choose the multiplicity value in the range [2,9] which maximizes sensitivity - correct?
• in which order/combination are the 3 cuts applied?
• what type of cuts are (namely are all 3 only best-sensitivity cuts or there also e.g. a theta 68% cut applied in combination with the others like EventDisplay does)

[moralejo]

Questions for @moralejo :

Apologies, I had overlooked this - while I had independently opened cta-observatory/pyirf#103 on the same topic - leading to cta-observatory/pyirf#104

• for each energy bin you choose the multiplicity value in the range [2,9] which maximizes sensitivity - correct?

Correct

• in which order/combination are the 3 cuts applied?

Note that the efficiencies of the hadronness cut and the theta2 cuts arbitrarily forced to be the same. Together with the relatively coarse efficiency steps, this means that the optimization on the test MC (the same used for performance calculations) will in general not pick on lucky fluctuations. Of course cut optimization on an independent sample should be preferred - one could even divide the test sample in, say, odd/even event id's, and use one half to optimize the cuts on the other and vice-versa, so that the whole test sample is used.
There is no order of the cuts, all combinations of multiplicity and (hadronness+theta2) efficiency (same for both) are considered. Hence it is just a 2D scan.

• what type of cuts are (namely are all 3 only best-sensitivity cuts or there also e.g. a theta 68% cut applied in combination with the others like EventDisplay does)

Best sensitivity, with the constraint of same efficiency for theta2 & hadronness cuts.

[HealthyPear]

As soon as cta-observatory/pyirf#123 is merged and a new release is out we can use it here and reproduce this approach.