pyhf 0.7.0 slower than v0.5.4

[efemberg11]

Hello!
Is there any particular differences, between v0.7.0 and 0.5.4, which slowing down fit calculation? If yes, how to modify my script, to get faser calculation (as v0.5.4) in v0.7.0? =)
I'm running same script (https://github.com/efemberg11/pyhf-Trexfitter_problem/blob/f269c7ba3f75ce48cedfed272516ca137c1740fd/problem/impct.py) for both versions, and there is ranking procedure ~0.5 sec (v0.5.4) vs ~1 sec (v0.7.0) on small dataset. And when I do ranking whith another my bigger roostats, there is ~160sec (v0.5.4) and ~350 sec (v0.7.0).
I suspect, that different optimizers (or vesions) are being used, is it possible? because, there is logging different verbosities.
With regards, Ramdas

[matthewfeickert]

Is there any particular differences, between v0.7.0 and 0.5.4

Please use the latest release, v0.7.0. Please check the release notes for the multitude of things that have changed since v0.5.4.

I'm running same script (https://github.com/efemberg11/pyhf-Trexfitter_problem/blob/f269c7ba3f75ce48cedfed272516ca137c1740fd/problem/impct.py) for both versions, and there is ranking procedure
...
how to modify my script, to get faster calculation

I haven't had time to look at your actual code yet, but in general if you want to understand where you're running into slowdowns with code you should profile it. if you're not familiar with using profilers here's an example that demonstrates how to use cProfile and snakeviz to profile a simple example and inspect the output: https://gist.github.com/matthewfeickert/dd6e7a5fda1ed1e3498219dafe5f9ea1

[matthewfeickert]

I'll also note that if you're trying to do direct comparisons between pyhf and TRexFitter that you might have a more straightforward time using cabinetry's higher level APIs.

[efemberg11]

As I understand, since v0.6.0 pyhf uses iminuit v2x and it works slower, than minuit v1.x in pyhf v0.5.4.
I used cabinetry, and there is also slower fitting.

[alexander-held]

I'm spontaneously not aware of anything in iminuit v2 that would cause us to expect slower fitting times there. How many parameters are in your ranking? If the whole ranking takes 0.5 - 1 second, perhaps the difference in execution time is not actually the fitting itself but some of the associated overhead. I can also not think of anything in cabinetry that could explain the difference, but pyhf 0.5.4 is from a while back so I would have to look more closely through the changes.

If you have a single fit that got substantially slower, that might be a good thing to look at in order to try and understand better the behavior.

[efemberg11]

Sorry for my late answer.
Here I have upload one of my roostats https://github.com/efemberg11/pyhf-0.7.0-vs-0.5.4
There is 65 NPs, and pyhf 0.5.4 for ranking takes ~796 sec. When I use the same script, pyhf 0.70 takes ~1809 sec. Ranking with cabinetry takes ~1682 sec.

[kratsg]

Can you use scipy optimizer in both cases to isolate if it's an optimizer-dependent issue?

[efemberg11]

I used an another workspace with more stats and got the following results by adjusting my scripts.

I think there were some mistakes in scripts or workspace and I corrected it. And now I guess that this liitle slowing down of latest pyhf version is not significant. Scipy optimizer doesnt calculate errors, thereby its faster (I guess).

[efemberg11]

About scipy, Im not interested about this optimizer, I just tried to compare particularities between v0.7 and 0.54. Fortunately I find my own mistakes. There was some anomaly in fit, when I started to fitting with pyhf 0.70 and compare with trex fitter, the calculation time became about the same. But it wasnt when I used pyhf v0.54, there was faster calcs. My mistake was that I got confused in my workspaces.
About TrexFitter, I used f-option only, without minos, and without prunings. As I understand, now there is problems in pyhf (issue662) with pruning, threshold settings etc.

[alexander-held]

Yes, the models being fitted will not be the exact same with the gamma treatment, but that should (if anything) give you less parameters to fit in TRExFitter, which should result in a faster fit. If you are not pruning anything in TRExFitter anyway, this should not matter at all.

[efemberg11]

Yes, the models being fitted will not be the exact same with the gamma treatment, but that should (if anything) give you less parameters to fit in TRExFitter, which should result in a faster fit. If you are not pruning anything in TRExFitter anyway, this should not matter at all.

Yes, with prunings and thresholds TrexFitter provides faster fit. But I cant compare it with pyhf(

[alexander-held]

Faster than the ~6 minutes in the table above, or faster even than pyhf? Maybe I am misreading the table, but what surprises me is that it is this much slower for what I thought is a setup without any pruning for any of the columns you are comparing.

[efemberg11]

Faster than the ~6 minutes in the table above, or faster even than pyhf? Maybe I am misreading the table, but what surprises me is that it is this much slower for what I thought is a setup without any pruning for any of the columns you are comparing.

I have produced roostats xml files with TrexFitter with some params in config:
SystPruningShape: 0.00
SystPruningNorm: 0.000
CorrelationThreshold: 0.000
MCstatThreshold: 0.00

and I used it for all fits in table. With that config I have 124 NPs. If I apply for example MCstatThreshold 0.05 and increase little bit another pruning, trexfitter gives me another fitresults with ~88 NPs, and runtime will be little bit faster, but not faster than pyhf with "all" data.