pyhf version installation >=0.6.0

[ssaha1234]

Dear pyhf community,

I am trying to install the latest version of pyhf using python -m pip install pyhf, but the version I get is 0.5.4. Unfortunately, the minuit_optimizer is crashing as "tolerance" is not included in this minuit version (1.5.4). Should I be using the git installation

python -m pip install --upgrade "git+https://github.com/scikit-hep/pyhf.git#egg=pyhf"

?

Thank you very much!

Best,
Shreya

[matthewfeickert]

@ssaha1234 It sounds like there's a few things happening here:

First, I assume from what you're saying that your Python version is 3.6. pyhf dropped support for Python 3.6 back in v0.6.0 (2021-02-15) after we announced the plan to do so in January 2021 on the pyhf-announcements mailing list and didn't get any responses asking us to hold off. (We'd recommend joining the mailing list if you aren't on it already! :) )

Our recommendation is that you upgrade your Python runtime to 3.7 or 3.8. This should be easy to do on your personal laptop and any cluster that has CVMFS on it has LCG views that support Python 3.7+. If you have questions on this though we can try to point you to resources.

Second,

Unfortunately, the minuit_optimizer is crashing as "tolerance" is not included in this minuit version (1.5.4)

sounds like you're trying to use APIs from newer versions of pyhf with older releases.

If you look back at the pyhf v0.5.4 API docs for pyhf.optimize.opt_minuit.minuit_optimizer you'll note that the API for the supported versions of iminuit (v1.5.3 and v1.5.4) in that range do not have a tolerance kwarg as that was added in pyhf v0.6.1 to use the iminuit v2.x API as is noted in the release notes.

So if you're going to use pyhf v0.5.4 you're going to have make sure that you're also using the correct API. It is also worth reading the iminuit Changelog docs to follow the API changes there as well.

As pyhf is a library that users need to integrate with their own code, and not an application with locked dependencies, we can't make too many decisions about what versions of other libraries users choose to run with or we'd overly restrict people and make installation of pyhf impossible in some cases. We can only state what are compatible releases with our APIs and then ensure that we provide lower bound limits in our installation that satisfy those requirements.

[matthewfeickert]

@ssaha1234 You are running into a multitude of problems that are brought on by the many problems of CVMFS not respecting local Python virtual environments (without great pain and PATH surgery) (c.f. dguest/pandamonium#48). In LCG_100 the CVFMS team have decided to include pyhf v0.5.4 in the view

$ lsetup 'views LCG_100 x86_64-centos7-gcc9-opt'
************************************************************************
Requested:  views ... 
 Setting up views LCG_100:x86_64-centos7-gcc9-opt ... 
>>>>>>>>>>>>>>>>>>>>>>>>> Information for user <<<<<<<<<<<<<<<<<<<<<<<<<
************************************************************************
$ python --version --version
Python 3.8.6 (default, Jan 27 2021, 21:24:33) 
[GCC 9.2.0]
$ python -m pip list | grep pyhf
pyhf                               0.5.4

and that means that it will take precedence by default. If you try to just do

$ python -m pip install --upgrade pyhf

this will unfortunately not help you as you do not have write access to the LCG view's python's site-packages directory

$ command -v python
/cvmfs/sft.cern.ch/lcg/views/LCG_100/x86_64-centos7-gcc9-opt/bin/python
$ python -m pip show pyhf
Name: pyhf
Version: 0.5.4
Summary: pure-Python HistFactory implementation with tensors and autodiff
Home-page: https://github.com/scikit-hep/pyhf
Author: Lukas Heinrich, Matthew Feickert, Giordon Stark
Author-email: lukas.heinrich@cern.ch, matthew.feickert@cern.ch, gstark@cern.ch
License: Apache
Location: /cvmfs/sft.cern.ch/lcg/views/LCG_100/x86_64-centos7-gcc9-opt/lib/python3.8/site-packages
Requires: scipy, click, tqdm, jsonschema, jsonpatch, pyyaml
Required-by: 

and so running python -m pip install --upgrade will attempt a local user install under $HOME/.local/lib/python3.8/site-packages and $HOME/.local/bin which is probably not on PYTHONPATH or PATH for you.

You can of course manually force this

$ export PYTHONPATH="${HOME}/.local/lib/python3.8/site-packages:${PYTHONPATH}"
$ export PATH="${HOME}/.local/bin:${PATH}"

which then places things correctly for use

$ python -m pip list | grep pyhf
pyhf                               0.6.2
$ python -m pip show pyhf
Name: pyhf
Version: 0.6.2
Summary: pure-Python HistFactory implementation with tensors and autodiff
Home-page: https://github.com/scikit-hep/pyhf
Author: Lukas Heinrich, Matthew Feickert, Giordon Stark
Author-email: lukas.heinrich@cern.ch, matthew.feickert@cern.ch, gstark@cern.ch
License: Apache
Location: /home/feickert/.local/lib/python3.8/site-packages
Requires: jsonpatch, click, pyyaml, jsonschema, tqdm, scipy
Required-by: 
$ which pyhf
~/.local/bin/pyhf

but you can see this is pretty hacky and is a terrible idea in general.

The method that I describe in the linked pandamonium Issue above is about as good as I have found, but I think @kratsg, @henryiii, and @kpedro88 might have all found workaround methods that might be safer then what I did.

On a related note, you could also be running into issues with what python you think pip is linked to not being the same python. In the above code snippet you try to use pip 3 different ways: pip, python -m pip, pip3.

I very strongly urge you to not do this and to only use python -m pip as that is the only way you can be sure that the pip you're invoking the same one that your python runtime knows about.

Given everything that we just discussed here, if you want a pretty hacky broken way of getting things working though you can just do the following which should work

$ cat requirements.txt  # Write this file such that it has the following
pyhf[minuit,contrib]~=0.6.2
mplhep
$ lsetup 'views LCG_100 x86_64-centos7-gcc9-opt'
$ export PYTHONPATH="${HOME}/.local/lib/python3.8/site-packages:${PYTHONPATH}"
$ export PATH="${HOME}/.local/bin:${PATH}"
$ python -m pip install --upgrade -r requirements.txt

then you should get the following

$ python -m pip list | grep 'pyhf\|iminuit\|mplhep'
iminuit                            2.7.0
mplhep                             0.3.9
mplhep-data                        0.0.2
pyhf                               0.6.2

[kpedro88]

Just to chime in about cvmfs + pip: it's indeed very irritating to try to update packages when the base filesystem is read-only. A few points to consider:

1. pip install --user should usually be avoided by physicists by default. We tend to have multiple software releases / computing environments deployed simultaneously for different projects. Dumping everything into ~/.local will lead to incompatibilities and very weird, hard-to-debug errors. (Python3 helpfully includes ~/.local in the Python path even if you don't tell it to do that.)
2. pip install --prefix [dir] can in principle be used instead, to create a .local area in a specified directory that is not $HOME. However, I've occasionally encountered cases where this doesn't work properly, for unclear reasons. (It also doesn't work at all in really old versions of pip, but we should be trying to stop using such old versions.) A more reliable option is: HOME=$PWD pip install --user, which tricks pip into thinking the current directory is $HOME. (Just to be clear, this works because --user is more reliable than --prefix; it's still not advisable to use --user without this hack, as described in item 1).
3. pip install --ignore-installed can be used to try to update packages when starting from an LCG environment (combined with item 2 above). It may reinstall some dependencies also, depending on the requirements of the package. Sometimes this leads to reinstalling everything, but the other options (virtualenv, conda, etc.) definitely require reinstalling everything, so it can be worth trying. If it doesn't work well, best to give up and just create a virtualenv.

[ssaha1234]

Dear @matthewfeickert and @kpedro88 ,
Thank you so much for the very detail and clear answer, I understand the issues much more clearly now ! It is very helpful to understand all the commands we are using.

I have managed to make it run with the most recent pyhf and minuit version :))
Thanks a lot again !

Best,
Shreya

[matthewfeickert]

@ssaha1234 Great! If you fee like things are answered as much as they can be for now, can you please mark this question as Answered? (This helps other users who are scrolling through the Discussions know which Discussions to look at first for help if they have similar questions.)

[ssaha1234]

Done !
Thanks a lot again,
Best,
Shreya