Analytic optimization of staterror parameters

[alexander-held]

Summary

It would be great to add analytic staterror parameter optimization, as available in Combine: https://cms-analysis.github.io/HiggsAnalysis-CombinedLimit/part2/bin-wise-stats/#analytic-minimisation.

Technically this would imply hiding such parameters from the minimizer and doing the optimization within the NLL function evaluation. There are probably a number of ways to try this out relatively easily (even without changes to pyhf itself). One could replace model.logpdf by manually evaluated Poisson + constraint terms, where the model prediction model.main_model.expected_data going into the Poisson term is performed with the optimal staterror parameter values. These parameters could be set to constant, but would just need to automatically get updated at each function evaluation. This implies no need for changes in the minimizer themselves.

cc @nsmith-

Additional Information

n/a

Code of Conduct

• I agree to follow the Code of Conduct

[matthewfeickert]

As a note to future us, I'm in favor of this. Thank you @alexander-held and @nsmith- for bringing this up.

[alexander-held]

I gave this a try, here's an implementation: https://gist.github.com/alexander-held/b020445d089c3ef79abe93d63543c5e7. Seems to be working, but I have not tested it extensively. There are a few to-do items and some cleanup needed. In addition, this will not be working correctly with (partially) pruned staterror configurations, so that needs some thought.