Python packages installing absolute paths to $out/lib/python-xxx/site-packages

[bennofs]

There's some python packages in nixpkgs that fail due to the use of the wheel python build system. Here's one case that I manually fixed: capstone-engine/capstone#783.

The reason for this seems to be that with wheels, paths in in data_files in the setup.py are always interpreted relative to the SITE_PACKAGES directory, even if they start with a /. Some packages, however, try to manually construct an absolute path, which is how we end up with double absolute paths.

Here's a list of affected packages (nix-locate 'site-packages/nix/store'):

• yc9vgl2bjvd42qlxpydwwk56819qjk59-electrum-dash-2.4.1/lib/python2.7/site-packages/nix /cc @np
• kslpi5ih76qqy6ciiam5r1aa4xjgnml4-getmail-4.53.0/lib/python2.7/site-packages/nix /cc @domenkozar @7c6f434c
• r9myky0b01yjhk6fl1yfxpwcz8i38ila-xpra-0.17.6/lib/python2.7/site-packages/nix /cc @offlinehacker
• fr598z84y99c0rfxvlakj50rldvjmxqc-python3.5-qnotero-1.0.0/lib/python3.5/site-packages/nix/cc @nico202
• idyrkzskcm0wsrr54q51c05r0dk5wkbl-cloud-init-0.7.9/lib/python2.7/site-packages/nix /cc @madjar
• kmgrq1a2rhdd0mww18mhqlrfaz72xqni-rubber-1.4/lib/python2.7/site-packages/nix /cc @ttuegel @peterhoeg
• bpr1321mk0v53k06fj8cby4pzmqlky6n-knockknock-rbf14bbff/lib/python2.7/site-packages/nix @copumpkin
• v8r2cw2jp19v3gl9lxk3q6i3h7f8i86q-pithos-1.1.2/lib/python3.5/site-packages/nix /cc @obadz @jgeerds
• rvhnn46nxf6v57ins74qfvvnddpxv4wr-ulauncher-4.4.0.r1/lib/python2.7/site-packages/nix @aaronjanse @worldofpeace
• zm1fflgbmw3d6g7xr4i3ab8xgvm93qmb-setools-2017-11-10/lib/python3.7/site-packages/nix (no maintainer listed)
• 4f0qs8089zc3wnx1nd58w4lrd74d5pba-pypolicyd-spf-2.0.2/lib/python3.7/site-packages/nix @abbradar
• qv7ixwb3slpcba2lc9fbvnkw77lg6ffy-nicotine-plus-1.4.1/lib/python2.7/site-packages/nix @klntsky
• la2nk7cjmkmrz3q4h67flb8089jd2ns1-nagstamon-3.2.1/lib/python3.7/site-packages/nix @pSub
• iizf6hlz6r1sxbsc980xw9y39sif5vwa-gshogi-0.5.1/lib/python3.7/site-packages/nix @ciil
• 6a8vpv1g9xmxch20wqq56chhlgy0v84v-electrum-ltc-3.1.3.1/lib/python3.7/site-packages/nix @asymmetric

I haven't checked it this is a problem for the function of the packages (it could still be harmless). Perhaps we should fix these before the release, like #23253? (/cc @globin)

[nico202]

Don't know why so much noise, however let me know: 1) if it's really fixed and 2) if you prefer not to use realpath

[FRidh]

Let me guess, they determine whether we're installing in a "virtualenv" and want to keep the data_files outside of it?

That seems to be the case in Cloud Init https://git.launchpad.net/cloud-init/tree/setup.py?id=0.7.9#n97

[FRidh]

Would it be "bad" if we fix sys.real_prefix to return the same value as sys.prefix? Of course, that would "break" the test these packages use to determine whether they'e in a virtualenv. Note that sys.real_prefix was removed in Python 3.x

[bennofs]

@FRidh I don't think this only happens with virtualenv. Look at the setup.py of the pkg I linked (capstone) for example

[bennofs]

@FRidh I'm not even sure if there is a way to write packages such that they behave the same under wheel and non-wheel method, without explicitly checking if building a wheel.

[FRidh]

I don't seem to understand the issue here. If there are differences between wheel and non-wheel, then that is something for wheel, pip or setuptools to fix. If packages do something weird in their setup.py, then that's a bug on their side.

[bennofs]

@FRdih yes I agree with that. Notifying upstream about the issue is a valid solution here, but as it stands, such packages are often subtly broken. Should we distribute broken packages in nixpkgs?

[bennofs]

I pinged the maintainers of the packages to give you a chance to check if this impacts your package and then take appropriate actions. I'm not using all of these packages, but in my experience, this particular issue often leads to strange problems that packages cannot find their data files later. So feel free to ignore this if everything is fine.

[7c6f434c]

Great. For getmail it was only about documentation. setup.py spends some effort to build the absolute path, and the comments complain that you cannot install using a relative path…

Short-circuited the simplest-to-kill part of all that. Thanks for noticing.

[mmahut]

Any news on this issue?

[bennofs]

@mmahut this is still an issue, I updated the first post with the current list of affected packages

[FRidh]

This is actually a duplicate of #4968.

[FRidh]

Actually, let's use this one instead as it is more up to date.

[worldofpeace]

Huh, I could've swore we fixed this for ulauncher. I'll look into it.

[ciil]

It's on my radar. I'll look into it over the weekend.

[seqizz]

nix-shell -p 'python3.withPackages(ps : with ps; [ uvicorn fastapi ])'   
these derivations will be built:
  /nix/store/b0x3ghq2p0hhpdnv5a21c0vmrq73krxd-python3-3.7.5-env.drv
building '/nix/store/b0x3ghq2p0hhpdnv5a21c0vmrq73krxd-python3-3.7.5-env.drv'...
collision between `/nix/store/m8nps45w28fpzzvh7jqcm5mmq6njgi0y-python3.7-starlette-0.12.4/LICENSE.md' and `/nix/store/wvv320lnlcv2yxzw3khpjsaxkmnzrgbr-python3.7-uvicorn-0.9.0/LICENSE.md'
builder for '/nix/store/b0x3ghq2p0hhpdnv5a21c0vmrq73krxd-python3-3.7.5-env.drv' failed with exit code 25
error: build of '/nix/store/b0x3ghq2p0hhpdnv5a21c0vmrq73krxd-python3-3.7.5-env.drv' failed

Adding two more packages 😕

[jtojnar]

Posting the upstream issue here as well to make it easier to find: pypa/setuptools#130

[emilazy]

It's quite easy to accidentally package something missing its runtime dependencies due to its setup.py misusing sys.prefix in this way; I ran into this with #86683 (fixed upstream in nicotine-plus/nicotine-plus#166). Perhaps the Python packaging machinery should have a simple check that you haven't created a nix directory in sys.prefix so that this error would at least result in a noisy failure rather than silent breakage?

[stale]

I marked this as stale due to inactivity. → More info

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/errno-13-when-trying-to-build-a-python-application/10741/8

[stale]

I marked this as stale due to inactivity. → More info

[emilazy]

still bad

[stale]

I marked this as stale due to inactivity. → More info

[bennofs]

still bad

[FRidh]

Yes, it's still an upstream issue. There is no spec for dealing with these files when using wheels.
https://discuss.python.org/t/should-there-be-a-new-standard-for-installing-arbitrary-data-files/7853

[alexvorobiev]

Adding two more

$ nix-shell -p 'python3.withPackages(ps : with ps; [ requirements-parser cyclonedx-python-lib ])'

building '/nix/store/6mbwfc4578cyplh3lb8g4fg9vh19hch4-python3-3.9.12-env.drv'...
error: collision between `/nix/store/fc4frxp2mwa5xcr8hrkcmn8pk8knwilf-python3.9-cyclonedx-python-lib-2.3.0/lib/python3.9/site-packages/LICENSE' and `/nix/store/c34666c339bgnqparpcn0ssppgjprcmk-python3.9-requirements-parser-0.5.0/lib/python3.9/site-packages/LICENSE'
error: builder for '/nix/store/6mbwfc4578cyplh3lb8g4fg9vh19hch4-python3-3.9.12-env.drv' failed with exit code 25;
       last 1 log lines:
       > error: collision between `/nix/store/fc4frxp2mwa5xcr8hrkcmn8pk8knwilf-python3.9-cyclonedx-python-lib-2.3.0/lib/python3.9/site-packages/LICENSE' and `/nix/store/c34666c339bgnqparpcn0ssppgjprcmk-python3.9-requirements-parser-0.5.0/lib/python3.9/site-packages/LICENSE'
       For full logs, run 'nix log /nix/store/6mbwfc4578cyplh3lb8g4fg9vh19hch4-python3-3.9.12-env.drv'.

[senorsmile]

3 more packages that seem to have the same issue:

• docker
• numpy
• toolz

Background:
I was trying to install the pypi docker package and running into the issue that it wasn't found when trying to import. Eventually ended up here: https://github.com/NixOS/nixpkgs/blob/master/doc/languages-frameworks/python.section.md#environment-defined-in-etcnixosconfigurationnix-environment-defined-in-etcnixosconfigurationnix. Following those instructions, found that numpy and toolz weren't being found either, but finally noticed the collision message, which led me to this page's issue.

As an not so convenient work around, I am able to run the python containing the right environment from:

warning: collision between `/nix/store/wp58vaq76dahy2af1hnbqqld1cif7pkj-python3-3.10.8-env/bin/python' and `/nix/store/zdba9frlxj2ba8ca095win3nphsiiqhb-python3-3.10.8/bin/python'

Which leads to success:

# /nix/store/wp58vaq76dahy2af1hnbqqld1cif7pkj-python3-3.10.8-env/bin/python
Python 3.10.8 (main, Oct 11 2022, 11:35:05) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> import docker

[senorsmile]

I believe that the issue I'm seeing is different than this issue, although all of the other documentation led me to believe otherwise. I determine that the above work when I remove the explicit installation of python3. So, all three of the packages appear fine.