Adding uncertainties on data?

[ldcorpe]

Hello,

I want to use pyhf to re-interpret an unfolded measurement.

Unlike detector-level data, unfolded measurements in a given bin may have stat errors which are difference from the typical Poisson(n). Indeed, there can be significant correlations between bins even for stat uncertainties!

So my question is twofold:
a) can one add extra uncertainties on data ? as far as I can see the histfitter spec defines a pdf which only allows modifications to the prediction.
b) is there a way to load a covariance matrix into pyhf instead of using Poisson(n) ?

best wishes and thanks for any advice!

Louie

[kratsg]

Sorry for not responding faster... this fell through.

a) Extra uncertainties on data - nope. Typically in HEP, there are no uncertainties on data (observations). The uncertainties are all on the monte-carlo/expected rates.
b) No. This is not supported by HistFactory generally.

[ldcorpe]

Hi Giordon

actually we have the case in a) for any and all unfolded measurements : the unfolding procedure transfers experimental uncertainties onto the unfolded data. So then if you are doing a fit on unfolded data , you need this (this is exactly the use case I had in mind)

b) shame ! What if the data stat uncertainties are correlated ? For example fitting two searches which are not orthogonal but where you know the correlation

anyway, we did both those things by hand with Minuit, but would be cool if central tools supported these not so crazy use cases !

Louie

[ldcorpe]

I should also say that I understand personnpower is at a premium so not expecting you to implement these non-trivial things ! Just curiosities and I guess highlighting they can be useful

[kratsg]

a) For unfolding, this needs to defer to unfolding experts (e.g. Tomas Dado) but my understanding is that unfolded data is not data. Perhaps this could be a longer discussion, but what would be the data/observations you fit to here? If your data has uncertainties, then you're not fitting to it, per-se.

b) Yes, but this isn't handled by HistFactory.