ensurepip has no --break-system-packages option

[bjourne]

Bug report

Bug description:

$ python3 -m ensurepip --upgrade                                                                                                                                                                                   
error: externally-managed-environment
...
You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
$ python3 -m ensurepip --upgrade --break-system-packages
python -m ensurepip: error: unrecognized arguments: --break-system-package

So it needs the --break-system-packages option or the error message should not tell me to use it.

CPython versions tested on:

3.12

Operating systems tested on:

Linux

[terryjreedy]

I suspect ensurepip should not pass this message on from pip.

@pfmoore Are you involved with ensurepip?

[pfmoore]

I think the situation is a little more complex than that, unfortunately. Unless you pass the --break-system-packages flag to pip, it simply won't install - and ensurepip doesn't have a way to set that option as @bjourne pointed out.

However, I'm a little unclear about the scenario here. Normally, ensurepip is intended for use with a Python installation that you built yourself, so unless you marked the Python environment as externally managed, everything will work fine. So this looks as if the OP is running ensurepip on their system Python - and that's exactly the situation that should be blocked, because OS vendors don't want pip messing with the software they manage. If the user wants to install or upgrade pip, they should be using the OS supplied package that contains pip (assuming for some reason that their OS doesn't install pip when installing Python).

So I think I'd want to know more about why the user is running python3 -m ensurepip --upgrade in the first place. @bjourne can you clarify?

[bjourne]

It's beside the point - regardless of how I use Python the error message shouldn't tell me to do something I can't do.

Since I'm unhappy with how my OS vendor has decided to package Python, Haskell, and other language's runtimes I want to manage them myself. This I do by installing pip in ~/.local/bin which works perfectly fine (or at least used to). "We're all consenting adults here" and I want to continue with my workflow.

[pfmoore]

You can use pip (with the --break-system-packages flag). Why do you specifically need to use ensurepip? It's designed for preinstalling pip as part of building Python, not for general use. For your use case I'd suggest downloading get-pip.py and using that to install pip wherever you want.

I agree that the message is a bit misleading - if someone was motivated, they could look at the possibility of creating a PR that suppressed the one line "You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages." from the pip output, simply failing without suggesting that option. But to avoid this becoming a maintenance burden, it would need to be done in a way that didn't require updating whenever pip changed the wording of that message - and that's likely to be more work than it's worth, given that the benefit is relatively minor.

[bjourne]

I'm getting a new error message from python -m ensurepip --upgrade now:

error: externally-managed-environment
...Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/lib/python3.12/ensurepip/__main__.py", line 5, in <module>
    sys.exit(ensurepip._main())
             ^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/ensurepip/__init__.py", line 284, in _main
    return _bootstrap(
           ^^^^^^^^^^^
  File "/usr/lib/python3.12/ensurepip/__init__.py", line 200, in _bootstrap
    return _run_pip([*args, *_PACKAGE_NAMES], additional_paths)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/ensurepip/__init__.py", line 101, in _run_pip
    return subprocess.run(cmd, check=True).returncode
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/python', '-W', 'ignore::DeprecationWarning', '-c', '\nimport runpy\nimport sys\nsys.path = [\'/tmp/tmpbtsaraow/pip-24.0-py3-none-any.whl\'] + sys.path\nsys.argv[1:] = [\'install\', \'--no-cache-dir\', \'--no-index\', \'--find-links\', \'/tmp/tmpbtsaraow\', \'--upgrade\', \'pip\']\nrunpy.run_module("pip", run_name="__main__", alter_sys=True)\n']' returned non-zero exit status 1.

I used ensurepip because the documentation recommends it: "The ensurepip package provides support for bootstrapping the pip installer into an existing Python installation or virtual environment. This bootstrapping approach reflects the fact that pip is an independent project with its own release cycle, and the latest available stable version is bundled with maintenance and feature releases of the CPython reference interpreter.

In most cases, end users of Python shouldn’t need to invoke this module directly (as pip should be bootstrapped by default), but it may be needed if installing pip was skipped when installing Python (or when creating a virtual environment) or after explicitly uninstalling pip." I didn't know about get-pip.py.

[pfmoore]

I asked on the PEP 668 (externally managed environments) discussion thread what the PEP authors thought about this situation: https://discuss.python.org/t/pep-668-marking-python-base-environments-as-externally-managed/10302/117

[bjourne]

Wow, thanks for the work you do!