Can’t build matplotlib as a flake (something about certifi) #280
Comments
|
Maybe related to #271? |
|
Related: #42 (comment) This should be fixed with this override: matplotlib = super.matplotlib.overridePythonAttrs (
old: {
propagatedBuildInputs = (old.propagatedBuildInputs or [ ]) ++ [ self.certifi ];
}
);The build still fails for matplotlib >= 3.4.0 because it now tries to download Qhull during the build. It is possible to use a system version, but the current nixpkgs package is at version 2016.1, which is too old. We need at least version 2020.2. |
|
Thanks a lot for the pointers! I got it to build, here is how:
Would it be ok to somehow override system libraries in this repository? E.g. provide |
|
Thanks for working on this. I would recommend that you first update After that I don't know what the poetry2nix policy on depending on specific nixpkgs versions is. Maybe we can just assume that we have the latest nixpkgs and then release a new poetry2nix version. Maybe adding an override is the preferred approach. I think that decision would be up to @adisbladis. |
|
In case others wonder how exactly to put in place the solution given by @kirelagin : let
qhullOverlay = self: super:
{
# Update Qhull to a recent version because Nixpkgs only has the 2016 version
qhull = super.qhull.overrideAttrs (old: {
src = super.fetchFromGitHub {
owner = "qhull";
repo = "qhull";
rev = "d0e8064088e0d51f7df9b380b22d66a3c6ba0c9a";
# If you don't know the hash, the first time, set:
# sha256 = "0000000000000000000000000000000000000000000000000000";
# then nix will fail the build with such an error message:
# hash mismatch in fixed-output derivation '/nix/store/m1ga09c0z1a6n7rj8ky3s31dpgalsn0n-source':
# wanted: sha256:0000000000000000000000000000000000000000000000000000
# got: sha256:173gxk0ymiw94glyjzjizp8bv8g72gwkjhacigd1an09jshdrjb4
sha256 = "0g009w5mzidxgvj6p90gd7rrhp325cslmgfd5b64vrxqjswcpydk";
};
});
};
in
{ pkgs ? import <nixpkgs> { overlays = [ qhullOverlay ]; }
}:
pythonEnv = poetry2nix.mkPoetryEnv {
projectDir = ./path/to/your/python/project;
overrides =
poetry2nix.overrides.withDefaults
(
self: super: {
matplotlib = super.matplotlib.overrideAttrs (old: {
# Add the Python `certifi` dep and the system `qhull` dep
propagatedBuildInputs = (old.propagatedBuildInputs or [ ]) ++ [ self.certifi pkgs.qhull ];
# Tell Matplotlib to use the system Qhull
postPatch = ''
cat > setup.cfg <<EOF
[libs]
system_freetype = True
system_qhull = True
EOF
'';
});
}
);
};There may be cleaner/better solutions but hey, at least it works 😄 |
|
yeah we should have a bot that updates the flake to track unstable, like it is going to be done in home-manager. I've submitted a PR for qhull. As for fixing poetry2nix, it's a matter of updating the flake and nixpkgs if we want matplotlib > 0.3.4 |
|
Thank you for working on this @teto :) This issue was closed by a keyword in the qhull update commit message. I think it should remain open for now since the override isn't added to the default overrides yet and the nixpkgs version situation is not resolved. We should at least wait unti the qhull update is in |
|
yeah i was surprised to see it closed but that's automation for your :) agreed. |
|
Dunno why it got closed and why I got subscribed to this ticket. I just pushed the most recent nixpkgs commits to my own nixpkgs fork. I'll reopen. |
|
Thanks for reopening. I think this happened because the commit contained the "fix" keyword which triggered the close once somebody with the necessary permission (you) pushed it to a repository. |
|
I've tried @kirelagin's solution but get a really weird error: nix logSomehow nuke-references is destroying all the library paths? And qhull's path is not even included in the compiler invocation... I'm using the following override: matplotlib = prev.matplotlib.overrideAttrs (old: {
# Add the Python `certifi` dep and the system `qhull` dep
propagatedBuildInputs = (old.propagatedBuildInputs or [ ]) ++ [ final.certifi qhull ];
# Tell Matplotlib to use the system Qhull
postPatch = ''
cat > setup.cfg <<EOF
[libs]
system_freetype = True
system_qhull = True
EOF
'';
});
|
|
I was wondering why I had matplotlib failing even with (and nixpkgs unstable builds matplotlib 3.4.1 with the certifi/qhull fixes) but the poetry2nix overrides overwrite the nix package so it needs to be overriden too The poetry2nix flake is getting old though so I wouldn't mind a bump there. |
|
It seems to me that this issue is fixed in all supported nixpkgs versions. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Please, see kirelagin/p2n-matplotlib for a reproduction – it is a fresh Poetry project depending only on
matplotlib+ a flake usingpoetry2nixmaster.When I try to
nix buildit, I get:Full build log
Sourcing python-remove-tests-dir-hook Sourcing python-catch-conflicts-hook.sh Sourcing python-remove-bin-bytecode-hook.sh Sourcing pip-build-hook Using pipBuildPhase Using pipShellHook Sourcing pip-install-hook Using pipInstallPhase Sourcing python-imports-check-hook.sh Using pythonImportsCheckPhase Sourcing python-namespaces-hook @nix { "action": "setPhase", "phase": "unpackPhase" } unpacking sources unpacking source archive /nix/store/d8k6dfkifz4hm9253ziv0ln9j8fiip5i-matplotlib-3.4.1.tar.gz source root is matplotlib-3.4.1 setting SOURCE_DATE_EPOCH to timestamp 1617177269 of file matplotlib-3.4.1/lib/matplotlib/_version.py @nix { "action": "setPhase", "phase": "patchPhase" } patching sources @nix { "action": "setPhase", "phase": "configurePhase" } configuring no configure script, doing nothing @nix { "action": "setPhase", "phase": "buildPhase" } building Executing pipBuildPhase Creating a wheel... �[33mWARNING: The directory '/homeless-shelter/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.�[0m Processing /build/matplotlib-3.4.1 �[31m ERROR: Command errored out with exit status 1: command: /nix/store/5470xw15wnn972ap0c4f7q642z4nvh6f-python3-3.8.8/bin/python3.8 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/build/pip-req-build-yyfnzujn/setup.py'"'"'; __file__='"'"'/build/pip-req-build-yyfnzujn/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /build/pip-pip-egg-info-xwl37da4 cwd: /build/pip-req-build-yyfnzujn/ Complete output (48 lines): WARNING: The directory '/homeless-shelter/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag. WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno -2] Name or service not known')': /simple/certifi/ WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno -2] Name or service not known')': /simple/certifi/ WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno -2] Name or service not known')': /simple/certifi/ WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno -2] Name or service not known')': /simple/certifi/ WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError(': Failed to establish a new connection: [Errno -2] Name or service not known')': /simple/certifi/ ERROR: Could not find a version that satisfies the requirement certifi>=2020.06.20 (from versions: none) ERROR: No matching distribution found for certifi>=2020.06.20 Traceback (most recent call last): File "/nix/store/fzzzxfag55c9fm7xjma1qh6jzgn7nd14-python3.8-setuptools-50.3.1/lib/python3.8/site-packages/setuptools/installer.py", line 126, in fetch_build_egg subprocess.check_call(cmd) File "/nix/store/5470xw15wnn972ap0c4f7q642z4nvh6f-python3-3.8.8/lib/python3.8/subprocess.py", line 364, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['/nix/store/5470xw15wnn972ap0c4f7q642z4nvh6f-python3-3.8.8/bin/python3.8', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/build/tmpbiqvfb5t', '--quiet', 'certifi>=2020.06.20']' returned non-zero exit status 1. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "", line 1, in File "/build/pip-req-build-yyfnzujn/setup.py", line 258, in setup( # Finally, pass this all along to distutils to do the heavy lifting. File "/nix/store/fzzzxfag55c9fm7xjma1qh6jzgn7nd14-python3.8-setuptools-50.3.1/lib/python3.8/site-packages/setuptools/__init__.py", line 152, in setup _install_setup_requires(attrs) File "/nix/store/fzzzxfag55c9fm7xjma1qh6jzgn7nd14-python3.8-setuptools-50.3.1/lib/python3.8/site-packages/setuptools/__init__.py", line 147, in _install_setup_requires dist.fetch_build_eggs(dist.setup_requires) File "/nix/store/fzzzxfag55c9fm7xjma1qh6jzgn7nd14-python3.8-setuptools-50.3.1/lib/python3.8/site-packages/setuptools/dist.py", line 673, in fetch_build_eggs resolved_dists = pkg_resources.working_set.resolve( File "/nix/store/fzzzxfag55c9fm7xjma1qh6jzgn7nd14-python3.8-setuptools-50.3.1/lib/python3.8/site-packages/pkg_resources/__init__.py", line 764, in resolve dist = best[req.key] = env.best_match( File "/nix/store/fzzzxfag55c9fm7xjma1qh6jzgn7nd14-python3.8-setuptools-50.3.1/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1049, in best_match return self.obtain(req, installer) File "/nix/store/fzzzxfag55c9fm7xjma1qh6jzgn7nd14-python3.8-setuptools-50.3.1/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1061, in obtain return installer(requirement) File "/nix/store/fzzzxfag55c9fm7xjma1qh6jzgn7nd14-python3.8-setuptools-50.3.1/lib/python3.8/site-packages/setuptools/dist.py", line 732, in fetch_build_egg return fetch_build_egg(self, req) File "/nix/store/fzzzxfag55c9fm7xjma1qh6jzgn7nd14-python3.8-setuptools-50.3.1/lib/python3.8/site-packages/setuptools/installer.py", line 128, in fetch_build_egg raise DistutilsError(str(e)) from e distutils.errors.DistutilsError: Command '['/nix/store/5470xw15wnn972ap0c4f7q642z4nvh6f-python3-3.8.8/bin/python3.8', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/build/tmpbiqvfb5t', '--quiet', 'certifi>=2020.06.20']' returned non-zero exit status 1. Edit setup.cfg to change the build options; suppress output with --quiet. BUILDING MATPLOTLIB matplotlib: yes [3.4.1] python: yes [3.8.8 (default, Feb 19 2021, 11:04:50) [GCC 10.2.0]] platform: yes [linux] tests: no [skipping due to configuration] macosx: no [Mac OS-X only] ----------------------------------------�[0m �[31mERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.�[0mI don’t understand what it means, but I tried adding
certifito the dependencies and doingnix build --impure– still the same.The text was updated successfully, but these errors were encountered: