hypotest documentation not clear on return types depending on test statistic chosen

[kratsg]

Summary

If we have return_tail_probs=True, then for test_stat in ['qmu', 'qtilde'], we return [CLs, CLsb, CLb]; while for test_stat=q0 we return [CLsb, CLb] only.

Documentation Page Link

https://scikit-hep.org/pyhf/_generated/pyhf.infer.hypotest.html#pyhf.infer.hypotest

Code of Conduct

• I agree to follow the Code of Conduct

[kratsg]

To summarize briefly from offline with @phinate and @alexander-held (some bullet points):

• https://github.com/scikit-hep/cabinetry/blob/59b49fbb5e3caffb76b12df5277d71912e909712/src/cabinetry/fit/__init__.py#L1106-L1115 gives p0 for reference
• https://github.com/gradhep/relaxed/blob/2bff24096f06068eba9b6ec75fdb0ab69a1243bc/src/relaxed/infer.py#L137
• conventionally, we don't typically report CLs for discovery, only $p_0$ and $p_\mu$
• likely $\mathrm{CL}_{sb} \equiv p_0$ in pyhf's code base but this needs to be clarified a lot
• CL naming is just bad i think everyone agrees on this

[kratsg]

One more thing, CLx is making the implicit assumption (for x=s, sb, b) that the test stat involve is either qmu or qmu-tilde.

[matthewfeickert]

One more thing, CLx is making the implicit assumption (for x=s, sb, b) that the test stat involve is either qmu or qmu-tilde.

We do state this in the pyhf.infer.hypotest docs

pyhf/src/pyhf/infer/__init__.py

Lines 108 to 112 in 852a224

	for signal strength :math:`\mu` and model hypothesis signal strength
	:math:`\mu'`, where the cumulative density functions
	:math:`F\left(q(\mu)\,\middle|\mu'\right)` are given by Equations (57)
	and (65) of :xref:`arXiv:1007.1727` for upper-limit-like test
	statistic :math:`q \in \{q_{\mu}, \tilde{q}_{\mu}\}`.

with a special note as well

pyhf/src/pyhf/infer/__init__.py

Lines 115 to 122 in 852a224

	.. note::
	The definitions of the :math:`\mathrm{CL}_{s+b}` and
	:math:`\mathrm{CL}_{b}` used are based on profile likelihood
	ratio test statistics.
	This procedure is common in the LHC-era, but differs from
	procedures used in the LEP and Tevatron eras, as briefly
	discussed in :math:`\S` 3.8 of :xref:`arXiv:1007.1727`.

Although the place that I've found it to be the most clear (and I should update this as well in the docs) is in Appendix A.1.2 and Table 11 in the Procedure for the LHC Higgs boson search combination in Summer 2011 — which actually describes what ATLAS and CMS did for the Higgs discovery.

(This reference document has been quite useful before, as we used it as a reference check in PR #1610).

[matthewfeickert]

CL naming is just bad i think everyone agrees on this

@cranmer would agree with you too, I think, given Issue #965. :)