Debugging Impact plots

[ssaha1234]

Dear pyhf community,
I am following the code here to produce some impact plots for my analysis - https://github.com/scikit-hep/pyhf/blob/master/docs/examples/notebooks/ImpactPlot.ipynb

I have histosys and shapefactor modifiers, but when I add staterror, the code crashes, with an error like this

  idx = model.config.par_slice(labels[i]).start
  File "/afs/cern.ch/user/s/ssaha/.local/lib/python3.8/site-packages/pyhf/pdf.py", line 320, in par_slice
    return self.par_map[name]['slice']
KeyError: 'AR1_TTZ_e_MCstat[00]'

AR1_TTZ_e_MCstat is the name of my staterror modifier. I am not able to understand the exact issue, any hints would be much appreciated !

Best,
Shreya

[matthewfeickert]

This is probably a duplicate of Issue #822

[ssaha1234]

Dear @alexander-held and @matthewfeickert ,
I just have a quick follow-up question about this new value error I am getting, after changing the impact plot to process the staterrors.
I understand that I can get around this, by changing the bounds in my likehood json file, but the fit was converging previously, with the same bounds and same values (the json file is exactly identical). Ideally, I would like to keep the bounds fixed to what I have presently, but the script crashes then.
I am not sure why I see this error now, and not previously.

ValueError: fit initialization parameter (index: 46, value: -0.2161005298973337) lies outside of its bounds: [0, 1]
To correct this adjust the initialization parameter values in the model spec or those given
as arguments to pyhf.infer.fit. If this value is intended, adjust the range of the parameter
bounds.

Thanks a lot for the clarification !
Best,
Shreya

[alexander-held]

What is "previously" here? Are you comparing a normal maximum likelihood estimate with this impact calculation, or before/after the width, initv[i] fix? When performing the impact calculation, parameters are held fixed at values of +/- 1 sigma around the nominal / best-fit values. If the nominal fit works fine, it is still quite possible that one of the fits in the impact calculation fail, since the initial conditions differ.

More generally I would suggest to increase the bounds: if your estimate is close (within uncertainty) to the bounds, then the estimate of the uncertainty can be trusted less. Is there a particular reason you need to stick with [0, 1] here?

[ssaha1234]

Hi @alexander-held ,
Yes exactly ! By previously, I mean after the width, initv[i] fix, for getting the staterrors. I can change the values of the bound, this is the bound of the POI, and I am not sure I understand why it should get a negative value. But yes, I can try it out with [-1,1].
Best,
Shreya

[alexander-held]

I am surprised it is the POI, I thought the notebook skips the calculation for the POI since it does not make sense to calculate the impact of the POI on itself. In general for normalization factors a negative value may not make a lot of physical sense, but with finite sensitivity it is not unusual that the 1 sigma uncertainty reaches down to negative values. It is quite common to allow for negative values in this scenario to avoid introducing boundaries close to the best-fit values.

[ssaha1234]

Hi @alexander-held ,
Thanks for the detailed answer, yes I agree that there can be boundary effects factoring in. I get a similar behavious for the ttZ normalisation. My current bounds are [0,3] and I get a negative value where the fit fails. I will relax the boundaries and check the plots, to see if it makes sense.
Thanks !
Best,
Shreya