pip should respect /usr/local

[FFY00]

What's the problem this feature will solve?

pip breaking system dependencies on Linux. Right now, users who want to respect the os dependencies need to install packages as user or need to use separated environments. This is obviously not optimal. We should be able to run pip without breaking the system out of the box.

Describe the solution you'd like

The solution is quite simple, just make pip respect /usr/local. pip shouldn't touch /usr/lib since that directory is supposed to be used by the distribution. pip should install its packages to /usr/local/lib.

However, I don't know what would be the best way to add /usr/local/lib/... to the PYTHONPATH. I think there's a section in python config that allows us to do that. If not, we could try to submit a PEP.

Alternative Solutions
As I mentioned before, those would be installing as user (--user) or using a pyenv.

[pradyunsg]

Same as #1668.

[FFY00]

No. #1668 suggest that pip should default to --user, that's not what I am proposing. --user installs to ~/.local not /usr/local. The issue here is that /usr/lib, specifically, breaks the distribution's packages, using this path is wrong as it's supposed to be only touched by the distribution, /usr/local/lib is the correct path.

[pradyunsg]

Ah, Sorry. I linked the wrong issue: #5605.

[FFY00]

Also not the same. This is a solution to that issue but it's not a duplicate. #5605 says we shouldn't upgrade the packages installed by distro's package manager, this issue says we shouldn't write to /usr/lib at all. Implement a function to see if the package was installed by pip doesn't solve issues like installing python modules that have already been installed by pip. /usr/lib isn't meant to be touched by anything else than the distribution itself.

Here's a really boiled down overview of the Filesystem Hierarchy Standard (FHS):
/lib is for Linux
/usr/lib is for the distribution
/usr/local/lib is for the user (user tools, this includes pip)

[pradyunsg]

IIUC, distributions can use pip itself when performing the packaging for their package manager, so being able to write to /usr/lib is something we'd want to allow doing -- it shouldn't be the default (which it is currently, both the above issues work toward changing that and improving behavior in this regard).

None the less, I'm a little short on time to look further into this right now. Sorry!

[pfmoore]

(From memory, I'm also short of time right now) What we write to is basically what Python tells us (via sysconfig) the "right location" is. So if a distribution doesn't want us to write to /usr/lib it should be patching its version of Python to tell us that. Or if you think it's a general enough issue, maybe you should raise an issue against core Python.

Pip never explicitly refers to /usr/lib in its codebase at the moment, and I don't think we should start doing so. We rely on standard libraries that provide filesystem layout data, and if those libraries are incorrect, then that's the place to fix this issue.

[FFY00]

@pradyunsg yes they can but is extremely discouraged, at least in Arch Linux. I don't see any harm in allowing this if it's explicitly indicated.

@pfmoore I looked into it and according to the documentation prefix is set when building python. We can't change that because it would change the entire location of our python installation, we don't want that, what we want is an alternative module directory. If you doesn't want to implement that, maybe we could try submitting a PEP so that a variable like prefix-user would be created but I doubt something like that would be merged.

[pfmoore]

@FFY00 Yes, basically this is largely around "how Python works". I don't honestly know the history, but it's probably based on the original design of distutils, if not even older than that. There's a single site-packages location, which is (somewhat) special, even though you can put other directories on sys.path. It will indeed be difficult to change something that's been around for so long, but conversely it's been around long enough that most people appear to be able to live with, or work around, it. Debian, for instance, uses a different install location (site-packages vs dist-packages I believe) and their solution mostly works (although it does trigger some issues with tools like pip, which expose the rough edges occasionally). You might want to look at how they solved this problem.

prefix is set when building python

Yes, but tools (like pip) typically get the install scheme via sysconfig (and/or distutils.sysconfig, there's a bit of historical confusion here) so the distribution could patch that. Or patch Python's startup code to change sys.prefix rather than using the build-defined value. Once you're customising Python, there are probably lots of options :-)

[FFY00]

I am a packager for Arch Linux. In Arch we try to patch software the less possible and this, however an issue, it's not that critical.

Arch strives to keep its packages as close to the original upstream software as possible.
In issues like this, we usually work with the upstream to fix them. This approach would benefit everyone having this problem, like Debian. IMO, there should be an additional alternative path for user modules and pip should use it by default. For this to happen, we would need to submit a PEP to python's upstream.

I will draft an email to send to the mailing list. If someone has any proposal on how to handle this issue, please let me know.

If anyone wants to contact me they can do so at lains@archlinux.org.

[pfmoore]

Cool - thanks for clarifying the context (I suspected you might be asking on behalf of a distribution, but wasn't sure). I'm not a Linux user, so I don't know the constraints/options involved, but I'd certainly support having a more flexible approach to the whole question of where things get installed. I do think it's important in a wider context than just pip, so your approach sounds good. Maybe starting on distutils-sig (which is where most of the people with a direct interest in packaging hang out) would be better than going straight to python-dev - having consensus from that list would likely make it much easier to get agreement on any changes needed in core Python (as well as giving people a chance to explore options that might help with existing Python versions - anything python-dev agree to is unlikely to affect Python versions prior to 3.8).

[FFY00]

Well, I am not officially asking on behalf of the distribution, I am just interested in getting this fixed 😄. Thank you for your suggestion, I'll contact distutils-sig first.

[dstufft]

Debian has 2 patches of relevance, there's the dist-packages patch, which just changes the name from site-packages to dist-packages, and allows Debian Python to live side by side with a manually installed Python. There's also the one they have that makes pip and co install to /usr/local, which is more relevant here I think.

At some point I'd love to get that patch upstreamed, but I haven't found the time for it.

[pfmoore]

@dstufft so you're in favour of pip special-casing a return of /usr/lib from sysconfig and replacing it with /usr/local/lib, much as @FFY00 originally proposed? If so, I apologise, @FFY00 I may have misled you on pip's position here (I'd defer to the other pip maintainers over how we behave on Unix, as that's not my platform).

[dstufft]

No, I agree with what you said. Pip asks Python to tell it where to install, if we want to change that we should change Python itself (I'm ~fine with backporting shims to try and get more consistent behavior between 3.NEW and old versions). I was just clarifying that Debian has two patches to Python itself, one for the dist-packages and one for /usr/local, and it was the 2nd one that actually matters for this issue.

[FFY00]

@dstufft where can I find the patches? If the /usr/local patch just overwrites the prefix that might not be the best approach. I mean, it's fine for Debian, but not for the upstream.

[dstufft]

https://bazaar.launchpad.net/~doko/python/pkg3.5-debian/view/head:/patches/distutils-install-layout.diff This one I think, although there might be other patches in that directory that are required as well.

[FFY00]

Thank you. I am not sure on out to approach this issue. We could patch install.py directly or we could add an alternative prefix (prefix-local?) that would point to the machine specific path. The last approach would not depend on the base prefix and would not hardcode /local making it easier to change the paths if we wanted. What do you think?

[chrahunt]

@FFY00, did you arrive at a solution for this?

[pradyunsg]

We're going to take a different direction with PEP 668 (https://python.github.io/peps/pep-0668/). Is this still relevant with that?

[FFY00]

It's not. In fact, I no longer think that using /usr/local is a correct solution as it hijacks that prefix, which is meant for local installations of Python, not for local installations of Python packages. LFS is not very explicitly on this, but I think this would be the correct interpretation.