ensurepip --upgrade doesn't change the version of pip used by venv

[cjw296]

BPO	37107
Nosy	@warsaw, @doko42, @pfmoore, @ncoghlan, @cjw296, @ned-deily, @encukou, @zooba, @dstufft

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-05-31.06:14:55.269>
labels = ['type-feature', 'library', '3.9']
title = "ensurepip --upgrade doesn't change the version of pip used by venv"
updated_at = <Date 2019-06-10.17:23:32.441>
user = 'https://github.com/cjw296'

bugs.python.org fields:

activity = <Date 2019-06-10.17:23:32.441>
actor = 'steve.dower'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2019-05-31.06:14:55.269>
creator = 'cjw296'
dependencies = []
files = []
hgrepos = []
issue_num = 37107
keywords = []
message_count = 6.0
messages = ['344029', '345069', '345070', '345082', '345113', '345136']
nosy_count = 9.0
nosy_names = ['barry', 'doko', 'paul.moore', 'ncoghlan', 'cjw296', 'ned.deily', 'petr.viktorin', 'steve.dower', 'dstufft']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue37107'
versions = ['Python 3.9']

[cjw296]

$ python3.7 -m ensurepip --upgrade
Looking in links: /var/folders/m6/tsd59qsj7pd_lldh4mhwh6kh0000gn/T/tmpqk_vncev
Requirement already up-to-date: setuptools in /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (39.0.1)
Requirement already up-to-date: pip in /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (18.1)

But:

$ python3.7 -m venv /tmp/test_venv
$ /tmp/test_venv/bin/pip --version
pip 10.0.1 from /private/tmp/test_venv/lib/python3.7/site-packages/pip (python 3.7)

[ncoghlan]

(Added packaging, Linux distro, and Windows and macOS installer folks to the cc list)

Chris and I were discussing this behaviour, and it turns out even I had forgotten how we had specified this feature in PEP-453: ensurepip --upgrade ensures that an older pip is brought up to date with the bundled version, but it does not check PyPI for the latest version the way that python3 -m pip install --upgrade pip does.

We both expected the ensurepip option to behave the same way as the pip option, since they share a name.

If I had the time machine keys, I'd use a more verbose name for the ensurepip flag (e.g. --upgrade-to-match-bundle) to help make it clearer that it does something different from the corresponding pip flag.

As it is though, for Python 3.9, I think we should change the behaviour of --upgrade to imply python -m pip install --upgrade pip, and then add a separate --network-upgrade/--no-network-upgrade option that allows folks to opt out of the PyPI part of the version check.

The make file would presumably be updated to pass the --no-network-upgrade flag, and I guess the macOS and Windows installers would as well (I'm not sure what the platform policies are around installers making random additional requests to external network services)

[ncoghlan]

Addressing the other part of Chris's initial post: there's also no --upgrade-pip option on venv itself.

Instead, there's only an --upgrade option that is intended for Python version upgrades, and restructures the internal layout of the venv to switch the Python major version number.

Unless you're using a Linux distro Python that has been patched to inject the external pip installation with rewheel or dirtbike, getting a venv that uses the externally updated version of pip requires running python3 - m venv --system-site-packages --without-pip ....

So my suggestion there would be to:

• rename "venv --upgrade" to "venv --set-interpreter" (keeping --upgrade as a legacy alias)
• default to running ensurepip --upgrade with its new behaviour
• add --network-upgrade/--no-network-upgrade options which get passed straight through to the updated ensurepip

[cjw296]

I don't suppose there's any chance we can treat the misnaming of these options as the bugs they feel like, so so fix them for 3.7+, rather than having people battle on with the confusion for another 3+ years until 3.9 is mainstream?

[encukou]

Please don't forget that it is possible to use venv without PyPI.

With my Fedora hat on: as a distro, we don't trust PyPI as a package delivery infrastructure. Lots of users do, and we allow the users to easily but explicitly opt in to trusting it by running "pip install". (In the same way, we don't trust rubygems, npm, CPAN, hackage, etc. -- there are too many, and PyPI is not special.)

Changing venv's existing --upgrade option to start installing from PyPI is problematic. We'd probably need to patch it out, creating an inconsistency between the distro and upstream: it would not do what python3 -m pip install --upgrade pip does.

ISTM, the proposed semantics aren't consistent: "venv --upgrade" would not match what "pip --upgrade" does: it would do what "pip install --upgrade pip" does. The latter needs an extra argument, explicitly saying *what* to upgrade. Making "pip" implicit makes sense for "ensurepip", but not for "venv".

Also, in my view, "network" should not be used as a synonym for PyPI (outside pip).

Could we instead make venv --upgrade print out a warning, saying that it upgrades Python, and to upgrade pip you should python -m pip install --upgrade pip instead?

---

Unless you're using a Linux distro Python that has been patched
to inject the external pip installation with rewheel or dirtbike,
getting a venv that uses the externally updated version of pip
requires running
python3 - m venv --system-site-packages --without-pip ....

FWIW, "rewheel" is no more: Fedora now distributes the wheels themselves, and patches system ensurepip to look in /usr/share/python-wheels/.

[zooba]

I'm with Petr here. ensurepip/venv should only go as far as we support, which is the version of the wheels included in our bundle. We should continue having the stdlib pretend that PyPI doesn't exist.

External network access would justify the longer option, in my opinion: "--upgrade-pip-from-pypi"

[XChikuX]

Hey guys,
Do we have any support for this feature?
We should either re-think if this issue can be solved.
Or close it permanently saying: Won't fix

[cjw296]

The issue still exists, I support @ncoghlan's approach, and I have as much faith in pypi as I do in Fedora's distribution mechanisms, if not more ;-)

[zooba]

I have as much faith in pypi as I do in Fedora's distribution mechanisms, if not more ;-)

It's not the mechanisms, but the validation. Fedora will have tested the version of pip you're getting from them in their context, whereas the PyPI release has only been tested in other contexts.

I totally understand Fedora not wanting to support users when they're not using their packages, and so not wanting them to easily get into an unsupported scenario. If there is any change made here, it should be easily patched out.

Possibly a more consistent solution is to more clearly document that it's for upgrading old venvs, and possibly add output indicating that it's only upgrading to the "default" versions. If this is still confusing, then we could display "Use 'python -m pip --upgrade pip' to upgrade from public internet sources".

But this doesn't really affect me. So if those who are impacted here decide it's best to just go straight to the newest release of pip, go for it.

[XChikuX]

The important patch result I'm expecting is that, we remain consistent with the latest environments outside and inside a venv.
There have been breaking changes to how packages are handled during their installing among pip versions. I do not want to deal with broken environments in python ever.
Not using PyPI is fine as long as there is an update mechanism for the defaults both outside and inside a venv.

[cjw296]

It's quite clear this issue is going nowhere. It pretty frustrating that we have such a confusing situation, but I can't see consensus ever being reached so we'll just all have to live with this confusing and broken behaviour, but hey, it's Python Packaging, so SNAFU, right?