pyhfcorr

Treating correlated uncertainties with pyhf

pyhf is based on the HistFactory statistical model. This is a very general tool for statistical inference of binned data.

One feature that is not included in pyhf is the correct treatment of arbitrarily correlated uncertainties. The current implementation features only fully (de)correlated uncertainties.

The use of this package is to add the option for arbitrarily correlated uncertainties, by a simple pre-processing step of the pyhf model. The mathematical background used is simple singular value decomposition (SVD) (also see below).

To account for correlations between parameters, one simply adds a correlation field to the pyhf model. Here we specify a name, which will be the new modifier name, the correlated variables vars, and the correlation matrix corr:

spec = {
    "channels" : ..., 
    "correlations": [
        {
            "name": "corr_1_2",
            "vars": ["unc1", "unc2"],
            "corr": [[1.0, 0.5], [0.5, 1.]],
        }
    ]
}

By pre-processing the model specification, we obtain a new specification, which is now pyhf compatible and has the correlation correctly implemented:

new_spec = pyhfcorr.decorrelate.decorrelate(spec)

Singular value decomposition

We can decompose a correlation matrix $C$ as

$$ C = U S^2 U^H = USSU^H = (US)(US)^H,$$

where $U$ is a unitary transformation matrix, $UU^H=1$ and $S$ is the diagonal matrix of standard deviations, $S=S^H$.

The geometrical interpretation of this is, that by applying the transformation $(US)^{-1}$ to correlated data results in an uncorrelated data-set with unity standard deviation.

The rotation $U^{-1} = U^H$ rotates points to a new coordinate system, where correlations between the dimensions vanish. In this rotated coordinate system, $S^{-1}$ scales the dimensions accordingly.

An illustration for a 2-dimensional random multivariate dataset $x$ with correlation coefficient $\rho=0.8$ is shown here: