Is there/could there be an "official" Model._modifications(pars) implementation?

[lhenkelm]

Model._modifications (here) is an undocumented method that seems to be intended to return modifier effects given parameter values.

This sounds very useful to me, e.g. with a use case like in get_mc_counts in this example. If generalized, this would make it possible to do generic pre-fit and post-fit plots of distributions, purely from the data in a workspace or model specification. The only data necessary to inject from the outside would be info about the bin edges.

However, when I tried using Model._modifications like that, I ran into some issues.
Apart from the leading underscore being scary, it seems it does not work on all model specs:

>>> import pyhf
>>> assert pyhf.__version__ == '0.6.1'
>>> pdf = pyhf.simplemodels.hepdata_like(
...    signal_data=[12.0, 11.0],
...    bkg_data=[50.0, 52.0],
...    bkg_uncerts=[3.0, 7.0],
... )
>>> pdf._modifications(pdf.config.suggested_init())
TypeError                                 Traceback (most recent call last)
...
TypeError: only integer scalar arrays can be converted to a scalar index
>>> # it really is due to shapesys.py:168
>>> pdf.spec['channels'][0]['samples'][1]['modifiers'].pop()
{'name': 'uncorr_bkguncrt', 'type': 'shapesys', 'data': [3.0, 7.0]}
>>> pdf = pyhf.Model(pdf.spec)
>>> pdf._modifications(pdf.config.suggested_init())
([],
 [array([[[[1., 1.]],
  
          [[1., 1.]]]])])

So the question is: Is there already a better way to do this?
And if not, the request would be to fix _modifications and document it/make it official.

[alexander-held]

Hi, if you are looking for pre- and post-fit distributions, the Model.main_model.expected_data API is suitable (I think this is not documented in the API since _MainModel is not public?):

pyhf/src/pyhf/pdf.py

Lines 479 to 505 in 671ed56

	def expected_data(self, pars, return_by_sample=False):
	"""
	Compute the expected rates for given values of parameters.
	For a single channel single sample, we compute:
	Pois(d | fac(pars) * (delta(pars) + nom) ) * Gaus(a | pars[is_gaus], sigmas) * Pois(a * cfac | pars[is_poi] * cfac)
	where:
	- delta(pars) is the result of an apply(pars) of combined modifiers
	with 'addition' op_code
	- factor(pars) is the result of apply(pars) of combined modifiers
	with 'multiplication' op_code
	- pars[is_gaus] are the subset of parameters that are constrained by
	gauss (with sigmas accordingly, some of which are computed by
	modifiers)
	- pars[is_pois] are the poissons and their rates (they come with
	their own additional factors unrelated to factor(pars) which are
	also computed by the finalize() of the modifier)
	So in the end we only make 3 calls to pdfs
	1. The pdf of data and modified rates
	2. All Gaussian constraint as one call
	3. All Poisson constraints as one call
	"""

You might also be interested in cabinetry, which provides an API for pre- and post-fit plots (with uncertainties), using only a model and the data you want to overlay: cabinetry.visualize.data_MC (bin edge information can be provided additionally).

[lukasheinrich]

we had a pretty long discussion on thiss here

#1377

wherer we make it possible to pass custom axes, and return Artist objects so you can channge the styling after the fact

[matthewfeickert]

where the API usually returns a matplotlib Axes or Figure object,

I think it is best in general to stick to the matplotlib API decisions of returning artists and not axes. I like seaborn and what it does, but the basic object that its API returns and operates on is a much higher level object: seaborn.FacetGrid

I think it is much better to not try to reinvent another plotting API. If you want a seaborn like API experience then take seaborn as a dependency and then use that as your visualization library that you build on for plotting functions. However, this will probably work poorly for most of our goals in HEP visualization and a better approach is to stick to the matplotlib API.

[lhenkelm]

I think it is best in general to stick to the matplotlib API decisions of returning artists and not axes.

With great respect, I disagree. Unless you are trying to extend matplotlib (as in, make things matplotlib cannot be used to plot plottable),
you are not trying to solve the kind of problem mpl is trying to solve. They solve the problem of "make any plot".
A HEP plotting library is not trying to make any kind of plot you cannot make using mpl (as far as I am aware, at least).
Rather such a library is trying to provide useful "abbreviations" in the "language" already defined by matplotlib.
So a higher level seems appropriate.

I (it seems you disagree, and that is of course OK) do not think Artists are the best abstraction to phrase the abbreviations in.
People using matplotlib use it in terms of axes, figures, and pyplot (bah).
So any student or researcher who has ever done any kind of "Intro to scientific computing in python" will have seen axes and figures,
and if they only have seen pyplot, it takes at most 5 minutes to google for the OO equivalent.
So the simplest thing would be to just provide a few functions, that either produce or modify axes.

axes have 62k hits on SO
artists have 4k hits on SO

[matthewfeickert]

Yeah, these choices are hard in general. I'm also not a visualization expert, so I'm more parroting back what I've learned/heard from people who are. I think that we're also talking about different things, though.

A HEP plotting library

a visualization module in any of the libraries that are focused on statistical analysis should not be viewed as plotting libraries. Ideally these will all be factored out in the future, just as pyhf hopes that the pyhf.infer module eventually will be as well. They should be focused on integrating with and using the existing visualization libraries that are being developed, and that requires that there is a consistent API that can be used between libraries, or at least a clearly understood hierarchy.

These discussions have been ongoing for quite some time already between the groups in Scikit-HEP and IRIS-HEP that are already working on visualization. Since you have differing views from what seems to be converging on I would encourage you to both check out the discussions and work on things like mplhep and hist, to see why these decisions are being made (with a lot of help and advice from people who have actually built full visualization libraries before). But I would also encourage you to make your views heard in the GitHub Discussions there as well.

You're views are obviously shared by many and are also arrived at through lots of time and work. They're not wrong. But I think it helps to make sure that the scope and use cases are fully understood by everyone when these discussions are being had, as otherwise it is way too easy to be talking past each other on the internet when people have different idea.

So I hope that we see a lot more discussions with you coming up in the future with @henryiii and @andrzejnovak involved as well. I think it would be great to have you in the mix. :)

[lukasheinrich]

Hi all,

i'm not sure whether there actually is any disagreementt is.. I think we all agree that it'd be great to have a HEP-specific viz library, much like seaborn is a viz library somewhat specialized for stats.

Whetther the API is

ax = mplhep.some_plot(...) #seabornstyle

(what I find lacking here is that its harder to navigate back to the artists if e.g. you want to change faceoclors)

artists_tuple = mplhep.some_plot(...)  #mplstyle

(here MPL is a bit lacking in that it's harder to navigate to the axes

container = mplhep.some_plot(...)  #mplstyle
container.axes
container.some_artist.set_fillcolor('g')

(I prefer this, I still maintain having ax axes attr is useful )

or something else is prob. a detail (the current version inn pyhf just requires passing an ax argument so we circumvent this, while in the new PR we either use a passed ax or use plt.gca() (we do seem to all share a distastet of plt ;) ).

As @matthewfeickert said hopefully we will factor this out to mplehep which I hope becomes basically seaborn for HEP
I agree with @lhenkelm original point that it'd be great if cabinetry would automatically save figures but allow modification first.