maturinBuildHook: specify python interpreter #116001
Conversation
ofborg
bot
added
6.topic: rust
10.rebuild-darwin: 11-100
10.rebuild-linux: 11-100
labels
|
(the qiskit failures reported by r-rmcgibbo are pre-existing, despite what the comment says. e.g. https://hydra.nixos.org/job/nixpkgs/trunk/python38Packages.cvxpy.aarch64-linux) |
| @@ -19,6 +19,7 @@ maturinBuildHook() { | |||
| --target @rustTargetPlatformSpec@ \ | |||
| --manylinux off \ | |||
| --strip \ | |||
| --interpreter $(command -v python) \ | |||
There was a problem hiding this comment.
There was a problem hiding this comment.
strictDeps is used, so its the interpreter of the build platform.
There was a problem hiding this comment.
However, its probably better to hardcode the interpreter with a substitution, because there are alternative Python interpreters, such as pypy.
There was a problem hiding this comment.
However, its probably better to hardcode the interpreter with a substitution, because there are alternative Python interpreters, such as
pypy.
Yeah, that would be even better.
There are some variables that can be set for providing the correct header and library paths:
https://pyo3.rs/v0.13.2/building_and_distribution.html#cross-compiling
But that would only work for bindings that use PyO3. However, since PyO3 is the most commonly used crate for binding Python, I guess the hook could detect if a package is using PyO3 and then set these variables in the expected way.
There was a problem hiding this comment.
strictDepsis used, so its the interpreter of the build platform.
maturin does run the interpreter in order to extract some metadata about it, so if it's the build platform's rather than the host platform's python, I'm not sure that's going to work without emulation...
However, its probably better to hardcode the interpreter with a substitution, because there are alternative Python interpreters, such as pypy.
This was my first thought as well, but it wasn't clear to me how to accomplish it within the hook / nix code.
There was a problem hiding this comment.
...however, I am not sure that that would be good either, isn't the Python interpreter metadata supposed to be for the platform you are building for?
A separate build interpreter is built. This interpreter adds the host metadata to sysconfig.
There was a problem hiding this comment.
In CPython cross-compilation does not really get much attention, so many downstreams such as Nixpkgs and Yocto have their patches.
There was a problem hiding this comment.
So, this will get messy if we use a substitution, because then the hook will depend on Python, and we need to rebuild the hook for each Python package set as well, resulting in a matrix of hooks. Not very nice.
There was a problem hiding this comment.
So, this will get messy if we use a substitution, because then the hook will depend on Python, and we need to rebuild the hook for each Python package set as well, resulting in a matrix of hooks. Not very nice.
Agreed. It seems like it's going to be tricky to both:
(a) have this work for pypy and other alternative python interpreters, and;
(b) not need to rebuild the hook for each Python package set.
Unfortunately my PR against maturin (PyO3/maturin#471) did not get a lot of traction, and filing a bug against rustlang or qemu (it's probably a qemu bug, but I'm just guessing) is daunting.
I suppose I can just change this PR to something ugly like
get_python() {
for name in "pypy" "python"; do
path=$(command -v "$name" || echo "")
if [ -f "$path" ]; then
echo $path;
break
fi
done
}
...
maturin ... \
--interpreter $(get_python)
There was a problem hiding this comment.
maybe the interpreters should export a setup hook setting an alias python.
|
I marked this as stale due to inactivity. → More info |
stale
bot
added
the
2.status: stale
|
Hello, I came to this issue while cross-compiling from host otherwise I was getting this: |
Motivation for this change
Pass absolute path to python interpreter to maturin, to work around a cross compilation issue under qemu user-mode emulation, fixes #115601
#115601 (comment)
Can be tested by running something like:
from an x86 machine with
boot.binfmt.emulatedSystems = [ "aarch64-linux" ];cc @danieldk
Things done
sandboxinnix.confon non-NixOS linux)nix-shell -p nixpkgs-review --run "nixpkgs-review wip"./result/bin/)nix path-info -Sbefore and after)