[RFC] mark python3 site-packages as externally managed

[classabbyamp]

https://peps.python.org/pep-0668/

By adding a file /usr/lib/python3.X/EXTERNALLY_MANAGED, pip will not let users install python modules with pip outside of virtual environments

Pros

• will prevent people from breaking xbps-installed python modules by using pip outside a venv

Cons

• breaks pip install --user too (can be solved)
• may break some void-based containers (or similar things) that install things with pip
  • solutions:
    • pip --break-system-packages
    • doas pip config set install.break-system-packages True
    • noextract on the EXTERNALLY-MANAGED file (could be done by default in void's official containers)

Prior Art

• gentoo implements this, see here and here
• ubuntu apparently does this, but I can't find their source packages to check
• Alpine implemented this for ~1 day until people complained that it broke their containers (this is probably less of a concern for void, and could be mitigated)
• arch has not done this, from what I can tell
• debian and fedora put distro-packaged python modules in a different directory, according to the PEP

cc @void-linux/pkg-committers

[ahesford]

I think I'm OK with this. Having to configure pip for user installs is a little annoying but worth the protections for system packages. Still, I'm heavily dependent on pip per-user installation for my regular work and would want to do a bit of testing to see how painful this is in practice.

As for containers... when I deploy custom Python in containers, I think it's better to wrap things in a venv anyway (sometimes with --system-site-packages) to make installation of packaged stuff easier. If --system-site-packages is compatible with EXTERNALLY-MANAGED, maybe that's good enough; otherwise, defaulting to removing that file or setting the system-wide pip config to allow breaking system packages in a container seems fine. (We could even let container builds break and force users to take this action if we really wanted to.)

[chrysos349]

ubuntu apparently does this, but I can't find their source packages to check

yes, it does starting with 23.04. see below:

https://git.launchpad.net/ubuntu/+source/python3.11/tree/debian/rules?h=ubuntu/lunar#n1241
https://git.launchpad.net/ubuntu/+source/python3.11/tree/debian/EXTERNALLY-MANAGED.in?h=ubuntu/lunar

[icp1994]

The relevant Debian NEWS file

[CtrlC-Root]

As a heavy Python user I'll add my two cents. I think this is generally a great idea as long as pip install --user remains accessible. I already use virtual environments for all project packages in order to avoid breaking system packages. However I've noticed there are lots of scripts or instructions (which ask to be copy and pasted) on the internet that include sudo pip install in them. As a user it's certainly possible to carefully inspect these before running them but you can still make a mistake and miss something. By the time you notice the system wide packages are likely contaminated or broken in a way that's annoying to fix. So requiring an opt-in mechanism makes a lot of sense to me.

I do find pip install --user necessary for non-project specific tools though (ex. virtualfish, powerline-status). The alternative would be to create system packages for these tools but in my opinion that would be a higher burden all around.

EDIT: Perhaps if this was implemented it would be a good idea to add a new section to the handbook for Python that mentions the workarounds above for various use cases.

[icp1994]

I recommend using pipx instead. I also plan to package rye when it's a bit more mature.

[classabbyamp]

pip --user can cause the same issues as sudo pip because it will be added to Python's module path ahead of the system modules dirs, so the EXTERNALLY-MANAGED file also breaks pip --user

[CtrlC-Root]

pip --user can cause the same issues as sudo pip because it will be added to Python's module path ahead of the system modules dirs, so the EXTERNALLY-MANAGED file also breaks pip --user

Indeed but it's relatively easy to fix by removing the appropriate site-packages folder under .local/lib. Especially if you only use it for a few system wide tools and keep everything else in virtual environments. I think this is something people who use it regularly (on a rolling distro at least) would be familiar with since every time the python3 package is updated to a new major version you need to reinstall these packages anyways.

[0x5c]

I do find pip install --user necessary for non-project specific tools though (ex. virtualfish, powerline-status). The alternative would be to create system packages for these tools but in my opinion that would be a higher burden all around.

I have had tools like that in my workflow in the past, but never would it have been impossible to simply pass --break-system-packages like the error message suggests. Also as icp1994 mentioned earlier, tools like pipx are designed for this specific purpose of installing python-based tools in venvs and exposing them in the path.

Indeed but it's relatively easy to fix by removing the appropriate site-packages folder under .local/lib. Especially if you only use it for a few system wide tools and keep everything else in virtual environments. I think this is something people who use it regularly (on a rolling distro at least) would be familiar with since every time the python3 package is updated to a new major version you need to reinstall these packages anyways.

While easy to fix, this kind of bug has the potential to be very difficult to diagnose properly and take much valuable time, including in the distro's support places (bug tracker, IRC, etc).
This is the main reason the "externally managed" feature exists in the first place.

[CtrlC-Root]

I do find pip install --user necessary for non-project specific tools though (ex. virtualfish, powerline-status). The alternative would be to create system packages for these tools but in my opinion that would be a higher burden all around.

I have had tools like that in my workflow in the past, but never would it have been impossible to simply pass --break-system-packages like the error message suggests. Also as icp1994 mentioned earlier, tools like pipx are designed for this specific purpose of installing python-based tools in venvs and exposing them in the path.

Indeed but it's relatively easy to fix by removing the appropriate site-packages folder under .local/lib. Especially if you only use it for a few system wide tools and keep everything else in virtual environments. I think this is something people who use it regularly (on a rolling distro at least) would be familiar with since every time the python3 package is updated to a new major version you need to reinstall these packages anyways.

While easy to fix, this kind of bug has the potential to be very difficult to diagnose properly and take much valuable time, including in the distro's support places (bug tracker, IRC, etc). This is the main reason the "externally managed" feature exists in the first place.

I think we are actually in complete agreement here. I think this change is good. I think advanced users who know what they are doing can use the escape hatch / workarounds listed above to continue using pip install --user. Some users may choose an alternative like pipx if it suits them. The only thing I would add is to consider documenting these options in the handbook so they are more discoverable.

[classabbyamp]

if you look at the PR I made for this, the error message pip gives mentions everything already

[ahesford]

Adopted with Python 3.12.