Impact of asymmetric uncertainties on signal extraction

[TommyDESY]

Dear experts,

We're currently working on a B -> Xu l nu branching fraction measurement. The question we have is related to the discussions in #2251 and #2257.

When throwing toys with pyhf, the event per-bin rates are Poisson-varied and the auxiliary data (or global observables) are Gaussian-varied. However, for global observables with asymmetric uncertainties, the corresponding Gaussian will also be asymmetric (the actual form of the Gaussian is not straightforward to derive but this paper gives a good idea of what I mean here). In the end when one plots the bias for a given number of toys, a biased normal distribution is obtained (see plots shown in the above mentioned discussions).

So, if the toy bias appears only because of the asymmetric Gaussian variations then should this also happen in the signal extraction ? It seems that the asymmetry here impacts toys but will it also bias the extracted POI (the signal strength giving the branching fraction in our case) ? I would need some clarification on that point.

If the asymmetries don't impact the signal extraction but only the Asimov toy pull, how would one evaluate the stability of the fit ? I reckon one could do a linearity check (where the signal strength is scaled up and down in the Asimov data and the fit is expected to recover this scaled signal strength) but if someone has other ideas, please tell me. In the end our goal would be to avoid having to symmetrise all uncertainties and still prove the good quality of the fit.

Thank you for your help.

Tommy

[alexander-held]

Hi Tommy, I think it would be useful to me if you could describe at a higher level what concerns you are looking to address. Are you looking into toys because you cannot rely on the asymptotic approximation for discovery significance / limits, or are you investigating something else? You mention stability and quality of the fit:

• Stability sounds like potential concerns about convergence?
• "Quality" might refer to goodness-of-fit? If so, cabinetry implements this via the saturated model in cabinetry.fit (code), exposed via the cabinetry.fit.fit API.

Neither of these two things necessarily require toys, but it depends on the details.

[TommyDESY]

Hi Alexander,

I'll rephrase my question to make it a bit clearer.

I produce toys in order to check whether the POI extraction is biased or not and whether its uncertainty is computed correctly. There are more details in #2257 but basically the pull is calculated as (mu - mu_initial) / uncertainty, with mu and uncertainty being the bestfit and uncertainty from the cabinetry fit respectively and mu_initial the suggested_init from the pyhf model. Since the event rates are Poisson-varied, for a sufficiently high number of toys the pull distribution should be a Gaussian normal distribution.

However, the distribution we obtain is shifted (again see #2257 for an example). As explained above, at this point we understand where this bias comes from but what we are not sure about is whether this bias observed in the pull distribution translates to a bias in the extracted signal strength. From a simple linearity check I did not observe any bias but I would like to have your opinion regarding this bias. Suggestions for other potential bias tests are welcome. I must emphasise that with symmetric uncertainties the pull distribution shift disappears.

I hope this clarifies my question.