ensurepip has no --break-system-packages option

[bjourne]

Bug report

Bug description:

$ python3 -m ensurepip --upgrade                                                                                                                                                                                   
error: externally-managed-environment
...
You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
$ python3 -m ensurepip --upgrade --break-system-packages
python -m ensurepip: error: unrecognized arguments: --break-system-package

So it needs the --break-system-packages option or the error message should not tell me to use it.

CPython versions tested on:

3.12

Operating systems tested on:

Linux

[terryjreedy]

I suspect ensurepip should not pass this message on from pip.

@pfmoore Are you involved with ensurepip?

[pfmoore]

I think the situation is a little more complex than that, unfortunately. Unless you pass the --break-system-packages flag to pip, it simply won't install - and ensurepip doesn't have a way to set that option as @bjourne pointed out.

However, I'm a little unclear about the scenario here. Normally, ensurepip is intended for use with a Python installation that you built yourself, so unless you marked the Python environment as externally managed, everything will work fine. So this looks as if the OP is running ensurepip on their system Python - and that's exactly the situation that should be blocked, because OS vendors don't want pip messing with the software they manage. If the user wants to install or upgrade pip, they should be using the OS supplied package that contains pip (assuming for some reason that their OS doesn't install pip when installing Python).

So I think I'd want to know more about why the user is running python3 -m ensurepip --upgrade in the first place. @bjourne can you clarify?

[bjourne]

It's beside the point - regardless of how I use Python the error message shouldn't tell me to do something I can't do.

Since I'm unhappy with how my OS vendor has decided to package Python, Haskell, and other language's runtimes I want to manage them myself. This I do by installing pip in ~/.local/bin which works perfectly fine (or at least used to). "We're all consenting adults here" and I want to continue with my workflow.

[pfmoore]

You can use pip (with the --break-system-packages flag). Why do you specifically need to use ensurepip? It's designed for preinstalling pip as part of building Python, not for general use. For your use case I'd suggest downloading get-pip.py and using that to install pip wherever you want.

I agree that the message is a bit misleading - if someone was motivated, they could look at the possibility of creating a PR that suppressed the one line "You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages." from the pip output, simply failing without suggesting that option. But to avoid this becoming a maintenance burden, it would need to be done in a way that didn't require updating whenever pip changed the wording of that message - and that's likely to be more work than it's worth, given that the benefit is relatively minor.

[bjourne]

I'm getting a new error message from python -m ensurepip --upgrade now:

error: externally-managed-environment
...Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/lib/python3.12/ensurepip/__main__.py", line 5, in <module>
    sys.exit(ensurepip._main())
             ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/ensurepip/__init__.py", line 284, in _main
    return _bootstrap(
           ^^^^^^^^^^^
  File "/usr/lib/python3.12/ensurepip/__init__.py", line 200, in _bootstrap
    return _run_pip([*args, *_PACKAGE_NAMES], additional_paths)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/ensurepip/__init__.py", line 101, in _run_pip
    return subprocess.run(cmd, check=True).returncode
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/python', '-W', 'ignore::DeprecationWarning', '-c', '\nimport runpy\nimport sys\nsys.path = [\'/tmp/tmpbtsaraow/pip-24.0-py3-none-any.whl\'] + sys.path\nsys.argv[1:] = [\'install\', \'--no-cache-dir\', \'--no-index\', \'--find-links\', \'/tmp/tmpbtsaraow\', \'--upgrade\', \'pip\']\nrunpy.run_module("pip", run_name="__main__", alter_sys=True)\n']' returned non-zero exit status 1.

I used ensurepip because the documentation recommends it: "The ensurepip package provides support for bootstrapping the pip installer into an existing Python installation or virtual environment. This bootstrapping approach reflects the fact that pip is an independent project with its own release cycle, and the latest available stable version is bundled with maintenance and feature releases of the CPython reference interpreter.

In most cases, end users of Python shouldn’t need to invoke this module directly (as pip should be bootstrapped by default), but it may be needed if installing pip was skipped when installing Python (or when creating a virtual environment) or after explicitly uninstalling pip." I didn't know about get-pip.py.

[pfmoore]

I asked on the PEP 668 (externally managed environments) discussion thread what the PEP authors thought about this situation: https://discuss.python.org/t/pep-668-marking-python-base-environments-as-externally-managed/10302/117

[bjourne]

Wow, thanks for the work you do!

[eli-schwartz]

You can use pip (with the --break-system-packages flag). Why do you specifically need to use ensurepip? It's designed for preinstalling pip as part of building Python, not for general use. For your use case I'd suggest downloading get-pip.py and using that to install pip wherever you want.

get-pip.py requires downloading and running some code from the internet which isn't verified or trivially verifiable. And pip doesn't have to be used with pypi.org (or indeed for downloading wheels from the internet) so the fact that it can't cryptographically verify the authenticity of a pypi package before installing it, isn't necessarily a reason to say that it's no different from get-pip.py

Besides: ensurepip is a public, and publicly documented, module. It's not obvious to me that it isn't for general use. It's being used here for precisely its documented purpose.

• If it was just used for preinstalling pip as part of building python, it would be part of the CPython build system, not an installable module, so perhaps in Tools/.

• If it was just for venv -- which also uses ensurepip -- then it would be a subpackage of venv, e.g. hidden as venv._ensurepip. This would even be additionally compatible with preinstalling pip, as cpython's own build system can rely on private implementation details of underscore modules.

But neither of these are the case. In fact, https://pip.pypa.io/en/stable/installation/ goes out of its way to list ensurepip as one of the ways to "officially" install pip, in addition to get-pip.py. In other words, pip directly encourages end users to run ensurepip, as an end user tool.

...

I don't think it makes sense to declare ensurepip as something not for general use. And if it's for general use, then it should support the flag it tells you to use to make sure it works.

[eli-schwartz]

Additionally, ensurepip supports the --user flag, which is clearly not intended for use by the preinstallation process, nor is it intended for use by venvs.

Since I'm unhappy with how my OS vendor has decided to package Python, Haskell, and other language's runtimes I want to manage them myself. This I do by installing pip in ~/.local/bin which works perfectly fine (or at least used to). "We're all consenting adults here" and I want to continue with my workflow.

Its sole use is to do precisely as the OP wants: disregard the choices of the OS vendor, and install pip to ~/.local/bin / ~/.local/lib/python3.XX/site-packages.

[pfmoore]

Note that the requirement to use --break-system-packages is also a choice made by the OS vendor, and the OP can override that by deleting the EXTERNALLY-MANAGED file that the vendor added (assuming they understand, and are comfortable with, the implications of doing so).

To be 100% clear, the OP hasn’t said how they built their copy of Python. It seems to me very odd that a user-built Python installed into ~/.local/bin should have a `site-packages’ that points to a system-controlled directory. So honestly, I think the OP has either misconfigured their build, or made some choices without fully understanding the implications. But they have options to do what they want to do, so that should be sufficient.

[eli-schwartz]

I'm not sure how you concluded that python is installed to ~/.local/bin just because the OP is attempting to install pip there.

The OP was very clear that it is not a user-built python.

Since I'm unhappy with how my OS vendor has decided to package Python,

This I do by installing pip in ~/.local/bin

[pfmoore]

Apologies, I misread the comment as saying that they had installed python in ~/.local/bin.

But I'm now even more confused, as I don't see how ensurepip would install pip in the user's ~/.local/bin directory if Python was installed in a system directory, without --user. And I don't see why pip would give the "you must use --break-system-packages" message if installing itself to ~/.local.

The OP originally said:

python3 -m ensurepip --upgrade
So it needs the --break-system-packages option or the error message should not tell me to use it.

My response was yes, the message is not ideal and could be changed, but it's non-trivial to do so. If someone wants to raise a PR for that, we can thrash out the details in that PR.

I do not think ensurepip needs the --break-system-packages option, because if the OS vendor has marked the location where pip is installed as system managed (using the EXTERNALLY-MANAGED file) then python3 -m ensurepip --upgrade should fail. If a user wants it to work, they have to override the OS vendor's choice, which they can do by removing the EXTERNALLY-MANAGED file (or by simply using pip install --upgrade --break-system-packages pip). They need to understand the consequences of what they are doing, obviously. We don't support people blindly ignoring their OS vendor's restrictions.

@eli-schwartz do you have an actual, reproducible issue where you are seeing this error, or are you simply arguing principles based on the OP's original report? Because I got the impression that the OP was happy with the original response (@bjourne if you do still have an issue here, please feel free to say so - I don't want the current discussion to distract from making sure you have a solution you are happy with).

[eli-schwartz]

Apologies, I misread the comment as saying that they had installed python in ~/.local/bin.

But I'm now even more confused, as I don't see how ensurepip would install pip in the user's ~/.local/bin directory if Python was installed in a system directory, without --user.

Ensurepip bundles a copy of pip, and injects itself into the path so that it can import pip and use pip to install a pip wheel. Pip itself contains the functionality for automatically assuming --user when you run its installation procedures.

So running /usr/bin/python -m ensurepip as a local user has always installed pip to ~/.local, for the same reason that /usr/bin/python -m pip install -U pip has done so.

This is an important and often forgotten-about safety feature of pip. My personal opinion is that it was prescient, and rendered EXTERNALLY-MANAGED unnecessary, but clearly not everyone agrees...

And I don't see why pip would give the "you must use --break-system-packages" message if installing itself to ~/.local.

Because pip has another implemented functionality, to require --break-system-packages (a flag that claims you're breaking the packages in the global site-packages) even when used with --user to operate on the user site-packages. I think this functionality is unhelpful, but I don't make the rules of pip.

I do not think ensurepip needs the --break-system-packages option, because if the OS vendor has marked the location where pip is installed as system managed (using the EXTERNALLY-MANAGED file) then python3 -m ensurepip --upgrade should fail. If a user wants it to work, they have to override the OS vendor's choice, which they can do by removing the EXTERNALLY-MANAGED file (or by simply using pip install --upgrade --break-system-packages pip). They need to understand the consequences of what they are doing, obviously. We don't support people blindly ignoring their OS vendor's restrictions.

s/pip is installed/pip should get installed/

That argument equally applies to pip. If the OS vendor has marked the location (that is to say, has marked ~/.local/bin by placing the EXTERNALLY-MANAGED file in /usr/lib on the same machine) where pip should be installed, python -m pip install --user requests should fail. If a user wants it to work, they have to override the OS vendor's choice.

So why does pip have an option to override the vendor's choice, but ensurepip doesn't?

@eli-schwartz do you have an actual, reproducible issue where you are seeing this error, or are you simply arguing principles based on the OP's original report?

I'm seeing the exact same error as the OP.

I tried running python -m ensurepip --user and got the "externally managed" error, and the suggestion to use --break-system-packages.

Yes, I went out of my way to verify it again with --user passed to ensurepip, even though it's the default pip functionality.

Because I got the impression that the OP was happy with the original response (@bjourne if you do still have an issue here, please feel free to say so - I don't want the current discussion to distract from making sure you have a solution you are happy with).

The OP responded to you saying you've asked on the python help forum, what others think -- by saying thanks for all the work you do.

I don't see anything there to suggest "happy with the original response" -- quite the contrary, my implicit understanding was that the OP is happy with your willingness to reconsider the matter by asking for further input from the PEP authors, and is awaiting a status update.

[bjourne]

Apologies, I misread the comment as saying that they had installed python in ~/.local/bin.

I have not.

But I'm now even more confused, as I don't see how ensurepip would install pip in the user's ~/.local/bin directory if Python was installed in a system directory, without --user. And I don't see why pip would give the "you must use --break-system-packages" message if installing itself to ~/.local.

For pip, --user is not required to install in ~/.local so I didn't think it would be required for ensurepip either. Regardless --user doesn't work either.

I do not think ensurepip needs the --break-system-packages option, because if the OS vendor has marked the location where pip is installed as system managed (using the EXTERNALLY-MANAGED file) then python3 -m ensurepip --upgrade should fail. If a user wants it to work, they have to override the OS vendor's choice, which they can do by removing the EXTERNALLY-MANAGED file (or by simply using pip install --upgrade --break-system-packages pip).

pip install --upgrade --break-system-packages pip <- impossible since pip is not installed. I didn't know about the EXTERNALLY-MANAGED file, but I don't want to remove it since Arch Linux's package manager will recreate it sooner or later and if I hadn't root I couldn't remove the file.

They need to understand the consequences of what they are doing, obviously. We don't support people blindly ignoring their OS vendor's restrictions.

Fair but --break-system-packages is explicit. I'm not blindingly doing it.

@bjourne if you do still have an issue here, please feel free to say so

No, get-pip.py solved it.

Note: I prefer it like this because for students with no root access and limited home directories installing python packages in ~/.local is the easiest. If they run into issues they can safely rm -rf ~/.local/lib/python3.12 and try again.