Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hepmc3: 3.2.0 -> 3.2.2 #92078

Merged
merged 2 commits into from Aug 9, 2020
Merged

hepmc3: 3.2.0 -> 3.2.2 #92078

merged 2 commits into from Aug 9, 2020

Conversation

@jonringer
Copy link
Contributor

jonringer commented Jul 2, 2020

Motivation for this change

noticed it was failing on #91886

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.
https://github.com/NixOS/nixpkgs/pull/92078
7 packages built:
fastnlo hepmc3 python27Packages.hepmc3 python37Packages.hepmc3 python38Packages.hepmc3 root yoda-with-root
@jonringer
Copy link
Contributor Author

jonringer commented Jul 2, 2020

@veprbl I had to stop the python.pkgs.numpy from being propagated, as this would potentially export a different python interpreter than the hepmc3 python modules.

@veprbl
Copy link
Member

veprbl commented Jul 2, 2020

Thanks for the bump! I have a comment about numpy:

@veprbl I had to stop the python.pkgs.numpy from being propagated, as this would potentially export a different python interpreter than the hepmc3 python modules.

I assume you mean a different numpy package?

IIRC, ROOT links against the numpy library, so using it with any other numpy library could break. Python libraries in nixpkgs usually use propagatedBuildInputs for their dependency on numpy (or any other python library), but we are not changing those. Some other options include: removing numpy support or instantiating pythonPackages.root.

@jonringer jonringer force-pushed the jonringer:bump-hepmc3 branch from 15e8c07 to 0399ab8 Jul 2, 2020
@jonringer
Copy link
Contributor Author

jonringer commented Jul 2, 2020

IIRC, ROOT links against the numpy library, so using it with any other numpy library could break. Python libraries in nixpkgs usually use propagatedBuildInputs for their dependency on numpy (or any other python library), but we are not changing those. Some other options include: removing numpy support or instantiating pythonPackages.root.

then we should probably use something like passthru to do this. As propagating python.pkgs.numpy will also introduce the python38 interpreter into the user's environment which is strange, if a user also wanted to use python37, but now has a python38 on his/her path.

cc @FRidh

@veprbl
Copy link
Member

veprbl commented Jul 2, 2020

As propagating python.pkgs.numpy will also introduce the python38 interpreter into the user's environment which is strange

Must admit, I did not know about this:

# nix-shell -p python37 -p python38Packages.numpy --pure --run "python3 --version"
Python 3.7.6
# nix-shell -p python38Packages.numpy --pure --run "python3 --version"
Python 3.8.2
# nix-shell -p python37Packages.numpy --pure --run "python3 --version"
Python 3.7.6
# nix-shell -p python37Packages.pytest --pure --run "python3 --version"
Python 3.7.6

I still don't see how this is a problem of this specific expression. If you change it, you might as well change it everywhere in pkgs/development/python-modules.

@jonringer jonringer force-pushed the jonringer:bump-hepmc3 branch 2 times, most recently from de87ef7 to 73524da Jul 2, 2020
@jonringer
Copy link
Contributor Author

jonringer commented Jul 2, 2020

I altered hepmc3 to override the python interpreter passed to root, so they should be coherent. But not a very elegant solution

I still don't see how this is a problem of this specific expression. If you change it, you might as well change it everywhere in pkgs/development/python-modules.

because root is defined in all-packages.nix and hepmc3 is defined in python-modules. with exception of when they both get passed the same python version, it will be incoherent.

For example, the python37Packages.hepmc3 on master fails due to compilation issues, but with the bump it will fail because it's trying to install for a different python interpreter

[13:21:19] jon@nixos ~/projects/nixpkgs (master)
$ nix-build -A python37Packages.hepmc3
...
-- Found Python3: /nix/store/f87w21b91cws0wbsvyfn5vnlyv491czi-python3-3.8.3/bin/python3.8 (found suitable version "3.8.3", minimum required is "3") found components:  Development Interpreter
-- HepMC3 python: Python verson 3.X found in /nix/store/f87w21b91cws0wbsvyfn5vnlyv491czi-python3-3.8.3/bin/python3.8. Python bindings generation is possible.
-- HepMC3 python: Tweek HEPMC3_Python_SITEARCH38 option to set the installation path for the python38 bindings.
-- HepMC3 python: HEPMC3_Python_SITEARCH38 defaults to /nix/store/f87w21b91cws0wbsvyfn5vnlyv491czi-python3-3.8.3/lib/python3.8/site-package
...
@jonringer
Copy link
Contributor Author

jonringer commented Jul 2, 2020

https://github.com/NixOS/nixpkgs/pull/92078
7 packages built:
fastnlo hepmc3 python27Packages.hepmc3 python37Packages.hepmc3 python38Packages.hepmc3 root yoda-with-root
@jonringer jonringer force-pushed the jonringer:bump-hepmc3 branch from 73524da to 123c332 Jul 2, 2020
@jonringer
Copy link
Contributor Author

jonringer commented Jul 2, 2020

I also added an installcheckphase, just to make sure it's able to be imported

https://github.com/NixOS/nixpkgs/pull/92078
4 packages built:
fastnlo python27Packages.hepmc3 python37Packages.hepmc3 python38Packages.hepmc3
Copy link
Member

veprbl left a comment

Thanks for the test in hepmc3.

Regarding your "root: don't propagated python interpreter" commit. You remove the explicit python dependency, which means that you treat interpreter propagation as a useful feature that we should rely on, but then you remove pythonPackages.numpy from propagatedBuildInputs and motivate that by saying that the users will not expect interpreter to be propagated. I would really like some clarification on what is our stance on interpreter propagation before starting changing things back and forth.

@jonringer jonringer force-pushed the jonringer:bump-hepmc3 branch from 123c332 to 982878f Jul 2, 2020
@jonringer
Copy link
Contributor Author

jonringer commented Jul 2, 2020

You remove the explicit python dependency, which means that you treat interpreter propagation as a useful feature that we should rely on, but then you remove pythonPackages.numpy from propagatedBuildInputs and motivate that by saying that the users will not expect interpreter to be propagated. I would really like some clarification on what is our stance on interpreter propagation before starting changing things back and forth.

I'm coming from the user's perspective. It's surprising to have the interpreter I didn't ask for in my nix-shell environment, but for the build environment, i feel like it's much less surprising to have the interpreter present.

If I had the following shell.nix:

with import <nixpkgs> {};

pkgs.mkShell {
  buildInputs = [
    python37
    root
  ];
}

I would expect to have python37 as my python interpreter, and for root to not bleed into my environment.

[15:50:52] jon@nixos ~/projects/nixpkgs (master)
$ nix-shell --pure -p "with import ./. {}; [ root python37 ]"

[nix-shell:~/projects/nixpkgs]$ python --version
Python 3.8.3
...
[15:51:20] jon@nixos ~/projects/nixpkgs (bump-hepmc3)
$ nix-shell --pure -p "with import ./. {}; [ root python37 ]"

[nix-shell:~/projects/nixpkgs]$ python --version
Python 3.7.7
@ofborg ofborg bot requested a review from veprbl Jul 2, 2020
doInstallCheck = withPython;
# prevent nix from trying to dereference a null python
installCheckPhase = stdenv.lib.optionalString withPython ''
PYTHONPATH=${placeholder "out"}/${python.sitePackages} python -c 'import pyHepMC3'

This comment has been minimized.

@FRidh

FRidh Jul 3, 2020 Member

${python.interpreter}

@FRidh
Copy link
Member

FRidh commented Jul 3, 2020

I would really like some clarification on what is our stance on interpreter propagation before starting changing things back and forth.

You can never expect nix-shell -p python37Packages.numpy to work because Python packages don't necessarily propagate the interpreter. In a build it means explicitly adding python37 to buildInputsor, in case of nix-shell, adding the corresponding -p python37. Having said that, that's still an unsupported way of working.

About the propagation one can say it's bad either way. If we propagate, packages may get numpy when they don't want it (hence we should avoid propagation altogether), however, if we don't, we break Python packages that depend on this package requiring numpy. Clearly, this issue should be fixed in a proper way and it's on the todo list (the passthru idea with a custom propagation logic).

If I had the following shell.nix:
....
I would expect to have python37 as my python interpreter, and for root to not bleed into my environment.

That is something you cannot expect because nix-shell and mkShell run setup hooks, it's a tool for reproducing a build environment after all. I don't think we should be optimizing for that type of use, despite it being very common.

Now I don't know how these tools use numpy, but from what I understood I think it makes sense to ensure they both get build with the same interpreter version. But not propagating numpy is something I have a mixed opinion on.

@jonringer
Copy link
Contributor Author

jonringer commented Jul 3, 2020

That is something you cannot expect because nix-shell and mkShell run setup hooks, it's a tool for reproducing a build environment after all. I don't think we should be optimizing for that type of use, despite it being very common.

It was also causing a build failure for python37Packages.hepmc3 as the python38 interpreter was the one that was found.

@veprbl
Copy link
Member

veprbl commented Jul 4, 2020

Let me summarize my understanding of the situation.

The python interpreter propagation originally added 6 years ago in ab423e6 and its original purpose for introducing it was for convenience of use in nix-shell. It is currently implemented as:

propagatedBuildInputs = propagatedBuildInputs ++ [ python ];

As we can see, it affects only python packages defined using the buildPythonPackage, so, for example, python3Packages.numpy will propagate python3, but python3Packages.yoda will not. Hence:

You can never expect nix-shell -p python37Packages.numpy to work because Python packages don't necessarily propagate the interpreter. In a build it means explicitly adding python37 to buildInputsor, in case of nix-shell, adding the corresponding -p python37. Having said that, that's still an unsupported way of working.

Then, I think, the @jonringer 's original concern was that one does not expect a package referred to as root to propagate a specific python package version (unlike a hypothetical python3Packages.root which would at least clearly indicate a version).

About the propagation one can say it's bad either way. If we propagate, packages may get numpy when they don't want it (hence we should avoid propagation altogether), however, if we don't, we break Python packages that depend on this package requiring numpy.

I actually was not worried too much because if someone is using a ROOT API which returns a numpy array (like ROOT.RDataFrame(tree).AsNumpy()), they are very likely to do some sort of processing for it, at that point including the python3Packages.numpy explicitly in the environment becomes a reasonable thing to do. What I had in mind was that one wants to have the same numpy as was used by ROOT, but perhaps this would be better achieved by adding inherit (python.pkgs) numpy to the expression for ROOT (I suppose, this is what @jonringer was suggesting in #92078 (comment)) and the using it as buildInputs = [ root root.numpy ];.

Clearly, this issue should be fixed in a proper way and it's on the todo list (the passthru idea with a custom propagation logic).

I don't know, but I am curious what other kind of propagation we could do.

Overall it doesn't seem like the interpreter propagation feature is doing much good for us. It doesn't seem to be mentioned in the manual (which, contrary to the common practice, is heavily biased towards using python.withPackages), and I'm not aware of anyone actually advertising or using it. It works, but not always. Perhaps we should consider deprecating it.

I also would like to mention the case when the python interpreter binaries are not needed at all. For example, Blender is linking to the interpreter directly and needs to import and depend on python packages like numpy.

@veprbl
veprbl approved these changes Jul 4, 2020
Copy link
Member

veprbl left a comment

I don't mind disabling the numpy propagation.

buildInputs = [ gl2ps pcre python zlib libxml2 lz4 lzma gsl xxHash ]
++ stdenv.lib.optionals (!stdenv.isDarwin) [ libX11 libXpm libXft libXext libGLU libGL ]
++ stdenv.lib.optionals (stdenv.isDarwin) [ Cocoa OpenGL ]
buildInputs = [ gl2ps pcre zlib libxml2 lz4 lzma gsl xxHash python.pkgs.numpy ]

This comment has been minimized.

@veprbl

veprbl Jul 4, 2020 Member

I would prefer to keep python dependency explicitly. One of the reasons being that ROOT can be built without numpy but with python.

@@ -3,19 +3,21 @@
let
pythonVersion = with stdenv.lib.versions; "${major python.version}${minor python.version}";
withPython = python != null;
# ensure that root is built with the same python interpreter, as it links against numpy
root_py = if withPython then root.override { inherit python; } else root;

This comment has been minimized.

@veprbl

veprbl Jul 4, 2020 Member

I don't think this makes a lot of sense to do this as HepMC doesn't interact with ROOT via any python API.

This comment has been minimized.

@jonringer

jonringer Jul 4, 2020 Author Contributor

it felt bad writing it, I'm just not aware of a different way to ensure that they both use the same interpreter version (if it's even important).

This comment has been minimized.

@veprbl

veprbl Jul 4, 2020 Member

Well it is an idiomatic code to ensure matching interpreter versions, but that is not needed in this particular case.

This comment has been minimized.

@jonringer

jonringer Jul 4, 2020 Author Contributor

right but just saying root doesn't give an indication what python interpreter will be used. As opposed to python37Packages.hepmc3

This comment has been minimized.

@veprbl

veprbl Jul 4, 2020 Member

I suppose HepMC3 links only to libRIO, which is not affected by python in any way. And root's setupHook will set LD_LIBRARY_PATH, so hepmc might end up working with a different build of ROOT in some cases.

@jonringer
Copy link
Contributor Author

jonringer commented Aug 9, 2020

merging to fix build, if additional improvements are needed, they can be done in another PR

@jonringer jonringer merged commit 23d6cd3 into NixOS:master Aug 9, 2020
17 checks passed
17 checks passed
hepmc3, hepmc3.passthru.tests, root, root.passthru.tests on x86_64-darwin Timed out, unknown build status
Details
Evaluation Performance Report Evaluator Performance Report
Details
grahamcofborg-eval ^.^!
Details
grahamcofborg-eval-check-maintainers matching changed paths to changed attrs...
Details
grahamcofborg-eval-check-meta config.nix: checkMeta = true
Details
grahamcofborg-eval-darwin nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="982878f"; rev="982878fb08917933492cbbb52fb2d3122c9a183d"; } ./pkgs/t
Details
grahamcofborg-eval-lib-tests nix-build --arg pkgs import ./. {} ./lib/tests/release.nix
Details
grahamcofborg-eval-nixos nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="982878f"; rev="982878fb08917933492cbbb52fb2d3122c9a183d"; } ./nixos/
Details
grahamcofborg-eval-nixos-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="982878f"; rev="982878fb08917933492cbbb52fb2d3122c9a183d"; } ./nixos/
Details
grahamcofborg-eval-nixos-options nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="982878f"; rev="982878fb08917933492cbbb52fb2d3122c9a183d"; } ./nixos/
Details
grahamcofborg-eval-nixpkgs-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="982878f"; rev="982878fb08917933492cbbb52fb2d3122c9a183d"; } ./pkgs/t
Details
grahamcofborg-eval-nixpkgs-tarball nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="982878f"; rev="982878fb08917933492cbbb52fb2d3122c9a183d"; } ./pkgs/t
Details
grahamcofborg-eval-nixpkgs-unstable-jobset nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="982878f"; rev="982878fb08917933492cbbb52fb2d3122c9a183d"; } ./pkgs/t
Details
grahamcofborg-eval-package-list nix-env -qa --json --file .
Details
grahamcofborg-eval-package-list-no-aliases nix-env -qa --json --file . --arg config { allowAliases = false; }
Details
hepmc3, hepmc3.passthru.tests, root, root.passthru.tests on aarch64-linux Success
Details
hepmc3, hepmc3.passthru.tests, root, root.passthru.tests on x86_64-linux Success
Details
@jonringer jonringer deleted the jonringer:bump-hepmc3 branch Aug 9, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants
You can’t perform that action at this time.