Ubunutu 16.04 + system python-virtualenv = borked virtualenv which doesn't work.

[tzickel]

Thank you for providing feedback on Python packaging!

To help us help you, please fill out as much of the following as you can. If a question is not relevant, feel free to skip it.

1. What is your operating system and version?

Ubuntu 16.04 (Works ok on Ubunutu 18.04)

2. What is your Python version?

Python 2.7.12 (Ubuntu 16.04 system)

3. What version of pip do you have?

None (inside the borked virtualenv it's the latest pip version, currently 20.0.2)

To bootstrap python-virtualenv uses an Ubuntu modified pip 8.1.1 version....

5. Could you describe your issue in as much detail as possible?

setuptools 45.0.0 (in pypi) and above bork environments which are based on ubuntu 16.04 and the system python-virtualenv package.

$ docker run -it --rm ubuntu:16.04

root@742a36b0cb97:/# apt update && apt install -y python-virtualenv
...
Reading state information... Done
17 packages can be upgraded. Run 'apt list --upgradable' to see them.
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  ca-certificates dh-python file libexpat1 libffi6 libmagic1 libmpdec2 libpython-stdlib libpython2.7-minimal libpython2.7-stdlib libpython3-stdlib libpython3.5-minimal libpython3.5-stdlib libsqlite3-0 libssl1.0.0 mime-support openssl python python-minimal python-pip-whl
  python-pkg-resources python2.7 python2.7-minimal python3 python3-minimal python3-pkg-resources python3-virtualenv python3.5 python3.5-minimal virtualenv
Suggested packages:
  libdpkg-perl python-doc python-tk python-setuptools python2.7-doc binutils binfmt-support python3-doc python3-tk python3-venv python3-setuptools python3.5-venv python3.5-doc
The following NEW packages will be installed:
  ca-certificates dh-python file libexpat1 libffi6 libmagic1 libmpdec2 libpython-stdlib libpython2.7-minimal libpython2.7-stdlib libpython3-stdlib libpython3.5-minimal libpython3.5-stdlib libsqlite3-0 libssl1.0.0 mime-support openssl python python-minimal python-pip-whl
  python-pkg-resources python-virtualenv python2.7 python2.7-minimal python3 python3-minimal python3-pkg-resources python3-virtualenv python3.5 python3.5-minimal virtualenv
0 upgraded, 31 newly installed, 0 to remove and 17 not upgraded.
Need to get 12.4 MB of archives.
After this operation, 53.7 MB of additional disk space will be used.
...
done.

root@742a36b0cb97:/# virtualenv venv
Running virtualenv with interpreter /usr/bin/python2
New python executable in /venv/bin/python2
Also creating executable in /venv/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.

root@742a36b0cb97:/# source venv/bin/activate

(venv) root@742a36b0cb97:/# pip install setuptools
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Requirement already satisfied: setuptools in /venv/lib/python2.7/site-packages (45.0.0)
ERROR: Package 'setuptools' requires a different Python: 2.7.12 not in '>=3.5'

(venv) root@742a36b0cb97:/# 

As you can see, because python-virtualenv uses the modified ubuntu 16.04 pip wheel which is version 8.1.1 to bootstrap the environment, it downloads the latest setputools from pypi (currently 45.2.0), and then any package which depends on setuptools (besides having the wrong version) will hard fail with this error:

ERROR: Package 'setuptools' requires a different Python: 2.7.12 not in '>=3.5'

This seems like a pretty popular and major workflow which simply stopped working...

[mattip]

xref pypa/setuptools#2004 which I think is the same thing

[pganssle]

To bootstrap python-virtualenv uses an Ubuntu modified pip 8.1.1 version....

I assume that this is the root of the problem, pip versions less than 9 do not support python_requires, so you'll have to find some way to pin your setuptools. It seems like you should be able to do pip install "setuptools<45", though maybe also you need to create the virtualenv without setuptools (I think there's a --no-setuptools option).

[ncoghlan]

Debian 8 has a similar problem (if you try to update pip in a virtual environment, it will happily install one that won't actually run on Python 3.4, as the system pip is too old to respect Python-Requires).

My workaround was to make the environment without pip, then install "pip < 19.2" (19.2 was the first version that required 3.5).

After that, pip in the venv is new enough to respect Python-Requires, so everything sorts itself out.

(Debian 9 and Ubuntu 18.04 have new enough system pip versions that this problem doesn't come up)

[stefanor]

it downloads the latest setputools from pypi (currently 45.2.0),

I can't reproduce that, I get 45.0.0.

In a clean Ubuntu 16.04 (Xenial) chroot:

#  apt install python-virtualenv
...
(xenial-amd64)stefanor@elgar:/$ cd /tmp/
(xenial-amd64)stefanor@elgar:/tmp$ virtualenv venv
Running virtualenv with interpreter /usr/bin/python2
New python executable in /tmp/venv/bin/python2
Also creating executable in /tmp/venv/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.
(xenial-amd64)stefanor@elgar:/tmp$ venv/bin/python -c 'import setuptools; print setuptools.__version__'
45.0.0

Can you describe more about what workflow this breaks?

[uranusjr]

@stefanor OP tries to upgrade setuptools with pip after creating the virtualenv. You didn’t.

[stefanor]

Dare I say "don't do that, then?"

The latest setuptools isn't compatible with Python 2.7. There is no newer version that is compatible.

If you try to upgrade it, it's telling you that there's nothing useful it can do:

(xenial-amd64)stefanor@elgar:/tmp$ venv/bin/python -m pip install setuptools -U
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
WARNING: The directory '/home/stefanor/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Requirement already up-to-date: setuptools in ./venv/lib/python2.7/site-packages (45.0.0)
ERROR: Package 'setuptools' requires a different Python: 2.7.12 not in '>=3.5'

I don't know what behaviour you're desiring?

[uranusjr]

Dare I say "don't do that, then?"

Uh… sure? I mean, that’s basically what comments above yours were trying to explain (without using the exact wording).

[stefanor]

Oh, I misread the changelog 45.0.0 is not supposed to be compatible with Python 2.7, either.
So, virtualenv is installing too new of a version. Not sure why it selected that rather than 45.2.0. But it looks like we should probably patch it to select < 45.

@ncoghlan: In Debian 8 (jessie), you get an older pip and setuptools, bundled in python-pip-whl. It looks like the mechanism for using that wasn't working in Ubuntu Xenial, so you get closer to the latest versions, there. (Probably more accurately: It wasn't doing an upgrade after using the bundled pip).

But upgrading pip and then setuptools seems to do the right thing, as long as you do it in this order. I don't think you need to make the environment without pip:

(jessie-amd64)stefanor@elgar:/tmp$ virtualenv venv
Running virtualenv with interpreter /usr/bin/python2
New python executable in venv/bin/python2
Also creating executable in venv/bin/python
Installing setuptools, pip...done.
(jessie-amd64)stefanor@elgar:/tmp$ venv/bin/python -m pip install -U pip
Downloading/unpacking pip from https://files.pythonhosted.org/packages/54/0c/d01aa759fdc501a58f431eb594a17495f15b88da142ce14b5845662c13f3/pip-20.0.2-py2.py3-none-any.whl#sha256=4ae14a42d8adba3205ebeb38aa68cfc0b6c346e1ae2e699a0b3bad4da19cef5c
  Downloading pip-20.0.2-py2.py3-none-any.whl (1.4MB): 1.4MB downloaded
Installing collected packages: pip
  Found existing installation: pip 1.5.6
    Uninstalling pip:
      Successfully uninstalled pip
Successfully installed pip
Cleaning up...
(jessie-amd64)stefanor@elgar:/tmp$ venv/bin/python -m pip install -U setuptools
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
WARNING: The directory '/home/stefanor/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting setuptools
  Downloading setuptools-44.0.0-py2.py3-none-any.whl (583 kB)
     |################################| 583 kB 7.1 MB/s 
Installing collected packages: setuptools
  Attempting uninstall: setuptools
    Found existing installation: setuptools 5.5.1
    Uninstalling setuptools-5.5.1:
      Successfully uninstalled setuptools-5.5.1
Successfully installed setuptools-44.0.0

[tzickel]

Ok, I might have been wrong in my previous assessment of the bug, or confused with some other installation I've done.

Anyhow, now I think this is an PYPA bug not an Ubunutu/Debian, one, setuptools 45.0.0 which does NOT support python2 (it has inside requierments >=3.5), has the filename in pypi:
setuptools-45.0.0-py2.py3-none-any.whl
and this causes pip 8 to install it.

PYPA should remove the setuptools 45.0.0 from pypi and put a version which isn't marked as py2, and this will fix all the problems I think.

[stefanor]

Aha, well spotted.

Yeah, removing that (and replacing with a py3 non-universal version) would be the simplest.
Otherwise, we can SRU virtualenv in Ubuntu Xenial, to work-around this. But it'll take at least a week to get through the process.

[tzickel]

OK, now I understand why I'm confused, on Ubuntu 16.04 with python-virtualenv:

# virtualenv venv1
# venv1/bin/python -c "import pip, setuptools; print(pip.__version__, setuptools.__version__)"
('20.0.2', '45.0.0')
# virtualenv venv2 --no-download
# venv2/bin/python -c "import pip, setuptools; print(pip.__version__, setuptools.__version__)"
('8.1.1', '20.7.0')
# venv2/bin/python -m pip install setuptools -U
# venv2/bin/python -c "import pip, setuptools; print(pip.__version__, setuptools.__version__)"
('8.1.1', '45.2.0')

I cannot understand why in venv1 it goes to 45.0.0 and in venv 2 it goes to 45.2.0 (both are using the same debian/ubuntu vendored pip 8.1.1 for the task.

There might be a few issues at play here, but one of them is that pypi is currently distributing a wrong version of setuptools 45.0.0 which is marked as compatible with python 2 while it is not.

Since the most common use case where people create a virtualenv is like venv1, replacing the pypi file setuptools-45.0.0-py2.py3-none-any.whl with setuptools-45.0.0-py3-none-any.whl would hopefully solve most of the problems people are currently having.

That will also get them an updated pip 20.0.2 which will take care of the rest.

As for why the --no-download virtualenv get's to 45.2.0 it might be a different bug (or ont), but then people will still having the old buggy pip and it's up on them to first upgrade to a saner pip before setuptools (altough it's an issue as well).

In conclusion, can somoene from setuptools / pypa fix the wrong setuptools 45.0.0 package on pypi ? (or someone check that that will fix the main issue) ?

[tzickel]

@stefanor as for SRU virtualenv, there is another case of people who use "python-pip" to pip install packages on their --user dir (they are still using pip 8.1.1 if they didn't pip upgrade in their user dir), but maybe you can't save everyone.

[di]

After chatting with @pganssle, I think the right approach would be to remove setuptools-45.0.0-py2.py3-none-any.whl and upload it with the exact same file, named setuptools-45.0.0-py3-none-any.whl.

Since the files will have the same hash, this will prevent any disruption to users that have (correctly) pinned their hashes for this version, and should make pip avoid this release and instead prefer an older, Python 2-compatible release.

[stefanor]

The virtualenv patch could look something like:

--- a/virtualenv.py
+++ b/virtualenv.py
@@ -971,7 +971,10 @@
             fp.write(whl)
 
     if not no_setuptools:
-        to_install.append('setuptools')
+        if sys.version_info[0] < 3:
+            to_install.append('setuptools<45.0.0')
+        else:
+            to_install.append('setuptools')
         to_install.append('pkg_resources')

But I think fixing Xenial's (non-virtualenv) pip package would require backporting support for Requires-Python. That's probably not a trivial patch.

[pganssle]

After chatting with @pganssle, I think the right approach would be to remove setuptools-45.0.0-py2.py3-none-any.whl and upload it with the exact same file, named setuptools-45.0.0-py3-none-any.whl.

Since the files will have the same hash, this will prevent any disruption to users that have (correctly) pinned their hashes for this version, and should make pip avoid this release and instead prefer an older, Python 2-compatible release.

I am somewhat skeptical of this solution. I think it's a possibility if we know it won't be disruptive, but I am not entirely sure what workflows are out there that may use the filename in their lockfiles (or be relying on the existence of the file on PyPI, like a direct link).

Also, I'm not entirely sure if or why it would work. I would expect pip to fall back to using the sdist for the latest setuptools and I'm somewhat puzzled as to why it wouldn't do that. It doesn't seem like a good idea to do something possibly disruptive based on what could be some sort of weird cache or a package that Ubuntu is shipping.

Can anyone explain why this is falling back to version 45.0.0 instead of hitting either 45.2.0 or 44.0.0?

[mattip]

Looking at the index https://pypi.org/simple/setuptools/, 45.0.0 is the last to be labelled with py2.py3-none-any.whl, which is problematic since it does not support python2.

[stefanor]

Can anyone explain why this is falling back to version 45.0.0 instead of hitting either 45.2.0 or 44.0.0?

I can reproduce finding 45.0.0 with PIP_ONLY_BINARY=':all:' (which virtualenv sets), so wheel's filename really does look like the culprit.

[pganssle]

I can reproduce finding 45.0.0 with PIP_ONLY_BINARY=':all:' (which virtualenv sets), so wheel's filename really does look like the culprit.

Sure, but if you're setting options then that doesn't help anything, right? There are tons of workaround for people willing to pass flags to their virtual environments or whatever, and they are documented clearly in this thread so anyone can find them.

The only value for taking the potentially disruptive action of modifying an existing setuptools release would be if it would make this work by default for common workflows.

Edit: Oops, I didn't see that you said virtualenv sets PIP_ONLY_BINARY=':all:'. I didn't get that when I tried to reproduce this with an old version of virtualenv on Python 2, though maybe I'm wrong there?

[stefanor]

setuptools 45.0.0 is getting installed by virtualenv from here https://github.com/pypa/virtualenv/blob/15.0.1/virtualenv.py#L829-L900

args = ['install', '--ignore-installed', 'setuptools', 'pkg_resources', 'pip', 'wheel']

At that point, these wheels are installed:

CacheControl-0.11.5-py2.py3-none-any.whl
chardet-2.3.0-py2.py3-none-any.whl
colorama-0.3.7-py2.py3-none-any.whl
distlib-0.2.2-py2.py3-none-any.whl
html5lib-0.999-py2.py3-none-any.whl
ipaddress-0.0.0-py2.py3-none-any.whl
lockfile-0.12.2-py2.py3-none-any.whl
packaging-16.6-py2.py3-none-any.whl
pip-8.1.1-py2.py3-none-any.whl
pkg_resources-0.0.0-py2.py3-none-any.whl
progress-1.2-py2.py3-none-any.whl
pyparsing-2.0.3-py2.py3-none-any.whl
requests-2.9.1-py2.py3-none-any.whl
retrying-1.3.3-py2.py3-none-any.whl
setuptools-20.7.0-py2.py3-none-any.whl
six-1.10.0-py2.py3-none-any.whl
urllib3-1.13.1-py2.py3-none-any.whl
wheel-0.29.0-py2.py3-none-any.whl

Edit: Debian has some patches to virtualenv, so https://salsa.debian.org/python-team/modules/python-virtualenv/-/blob/patched/15.0.1+ds-3/virtualenv.py#L849-921 is probably better to look at.

[stefanor]

Actually, backporting Requires-Python support (pypa/pip#3877) was trivial and will fix standalone pip as well as virtualenv, for Xenial.

So, started the SRU process: https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1864766

Edit: Uploaded, waiting for the SRU team to approve it. Then it gets a week of ageing and testing, before releasing.