Require Python 3.5 or later, dropping support for Python 2.

[jaraco]

This change does not yet remove any of the compatibility for Python 2, but only aims to declare the dropped support.

Closes #1458

[jaraco]

In this latest push, I've added a new module, pkg_resources.py2_warn, which when imported emits an obnoxious warning about dropped support. This warning will allow this version of setuptools to continue to run on Python 2, just with that warning. Here's what it looks like:

setuptools drop-py2 $ .tox/py27/bin/python                                                                                                                                                                   

WARNING: Python 2.7 is not recommended. 
This version is included in macOS for compatibility with legacy software. 
Future versions of macOS will not include Python 2.7. 
Instead, it is recommended that you transition to using 'python3' from within Terminal.

Python 2.7.16 (default, Nov  9 2019, 05:55:08) 
[GCC 4.2.1 Compatible Apple LLVM 11.0.0 (clang-1100.0.32.4) (-macos10.15-objc-s on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import setuptools
pkg_resources/py2_warn.py:19: UserWarning: ************************************************************
You are running Setuptools on Python 2, which is no longer
supported by this version of Setuptools and
>>> WILL STOP WORKING <<<
in a subsequent release. Please ensure you are installing
Setuptools using pip 9.x or later or pin to `setuptools<45`
in your environment.
If you have done those things and are still encountering
this message, please comment in
https://github.com/pypa/setuptools/issues/1458
about the steps that led to this unsupported combination.
************************************************************
  sys.version_info < (3,) and warnings.warn("*" * 60 + msg + "*" * 60)

I believe by dropping support for Python 2 and emitting this warning, this project should be able to suss out any use-cases that are rapidly and unexpectedly running the unsupported version on Python 2, allowing this pull request to be accepted today and for Python 2 support to be removed in short order (weeks, not months).

[pganssle]

@jaraco Looking at this, it seems like this actually did drop support for Python 2. I thought we were just going to emit an annoying warning?

I am sort of on the fence about even emitting a warning, but I don't really think we should drop Python 2 support without coordinating with other PyPA packages. Especially pip, who vendors us and is planning to be one of the last packages to drop Python 2 support.

I think maybe what we should do is to get this on the agenda for the Packaging Summit and discuss overall strategy there. I don't think an extra 4 months of Python 2 support will be a huge problem for us.

[jaraco]

Right - I did drop declared support for Python 2, without dropping the compatibility layers. This means that installers using pip will get setuptools<45 on Python 2 and setuptools otherwise (similar to getting setuptools<44 on Python 3.4. I did, however, not drop the compatibility layer, so if this code does get installed on Python 2, it will continue to run for some time. This change should enable most environments to get a compatible version of setuptools through python_requires without any effort, give others time to pin to an old version of setuptools (if using easy_install or some other old installation mechanism), but not break any environments.

It will enable the project to solicit feedback on use-cases that may not be supported, but also set the stage to eliminate the compatibility layer. If pip needs Python 2 support, they can pin to setuptools<45 if necessary. They're already pinned to an older release, so that's probably not an issue. And surely if a fix is needed for pip, we'll back-port that to setuptools 44.x.

My hope is this change will be mostly automatic, and there will be few or no issues and we can simply move forward to the future.

I don't think an extra 4 months of Python 2 support will be a huge problem for us.

I spent several hours recently working through issues due to Python 2 for an issue that had a straightforward implementation on Python 3. Is that a huge problem? Perhaps not, but by the same measure, retaining Python 2 support indefinitely wouldn't be a huge problem, just a perpetual annoyance that's a drag on the project and limits innovation.

If there's significant harm in this change, we can revert it.

[pganssle]

Right - I did drop declared support for Python 2, without dropping the compatibility layers. This means that installers using pip will get setuptools<45 on Python 2 and setuptools otherwise (similar to getting setuptools<44 on Python 3.4. I did, however, not drop the compatibility layer, so if this code does get installed on Python 2, it will continue to run for some time. This change should enable most environments to get a compatible version of setuptools through python_requires without any effort, give others time to pin to an old version of setuptools (if using easy_install or some other old installation mechanism), but not break any environments.

Without testing for compatibility, compatibility will degrade by random bit-rot.

Also, this PR does not address my major concern with dropping Python 2 which is that I don't think we should be encouraging people to pin setuptools, because the standardization efforts and efforts to migrate people away from things like setup.py install require us to be able to make changes to versions that are widely adopted.

I spent several hours recently working through issues due to Python 2 for an issue that had a straightforward implementation on Python 3. Is that a huge problem? Perhaps not, but by the same measure, retaining Python 2 support indefinitely wouldn't be a huge problem, just a perpetual annoyance that's a drag on the project and limits innovation.

If there's significant harm in this change, we can revert it.

I get that it's annoying to continue supporting Python 2, but as I've said in several places, I am worried that this is going to hurt adoption of pyproject.toml and other important changes to the ecosystem. I don't think we should support Python 2 indefinitely, but we should probably coordinate with other Packaging projects and we should definitely give more advance notice than this.

I vote for reverting it.

[jaraco]

Without testing for compatibility, compatibility will degrade by random bit-rot.

True. I've been thinking about this concern. I'm tempted to create a test environment that tests Python 2 compatibility, but I thought I'd defer that effort until it's needed. I think there's a not insignificant chance that it won't be needed. The fact that any testing is not part of CI now is an issue and a risk, I agree.

I don't think we should be encouraging people to pin setuptools

I agree. And I don't think this change necessarily encourages pinning. The baseline expectation is that with pip 9 or later, no pinning is necessary and everything continues to work.

standardization efforts and efforts to migrate people away from things like setup.py install require us to be able to make changes to versions that are widely adopted.

Acknowledged. And it's possible to selectively backport those changes to 44.x, while at the same time strengthening the incentive to move away from setup.py install.

I say we revert it.

I don't yet see a compelling reason to revert it, other than the risk of bit rot. On the other hand, reverting this now would create a lot of noise. My intention was to give this a trial run and see how it goes; should we not at least gather some data about the impact of this change?

[pganssle]

I agree. And I don't think this change necessarily encourages pinning. The baseline expectation is that with pip 9 or later, no pinning is necessary and everything continues to work.

python_requires effectively pins setuptools, and importantly anyone who still maintains 2/3 compatibility will also need to pin setuptools and do a "feature freeze" where they don't use any modern features of setuptools.

Acknowledged. And it's possible to selectively backport those changes to 44.x, while at the same time strengthening the incentive to move away from setup.py install.

We can't make any breaking changes to the 44.x branch and keep with semver, for example, which means no hard removals of any of our deprecated commands. Admittedly, that's less important to me than the other stuff, though.

That said, it is way more work to maintain a system of backports than to continue maintaining compatibility.

On the other hand, reverting this now would create a lot of noise.

Yeah, I don't think you should have merged this and I definitely don't think you should have released it. I am aware that I have been slow on reviews and stuff, but I've tried to communicate that I was uncomfortable with moving quickly on this.

My intention was to give this a trial run and see how it goes; should we not at least gather some data about the impact of this change?

It seems unlikely that there will be immediate problems, but I also don't think a data-gathering period will be all that helpful for the problems I'm concerned about. No amount of a bunch of people pinned to a working version of setuptools is going to tell us whether this is going to cause problems when we want to improve PEP 517 support in the future.

In any case, what's done is done. I think this is going to cause problems in the long run no matter what we do now, because we'll probably cause confusion about when we are and are not dropping Python 2 support if we flip-flop back and forth. Might as well leave it like this for at least a week or two. I'd also like to hear more feedback from other PyPA projects as well. Maybe my concerns are overblown.

[gaborbernat]

As owner of virtualenv I'm committed to support it on cpython 2 for the next 18 months or so, and as I'm aware pip similarly 🤷‍♂️tox probably will drop with next major version is 6 months.

[jaraco]

I am aware that I have been slow on reviews and stuff, but I've tried to communicate that I was uncomfortable with moving quickly on this.

Please don't feel bad; I'm trying to be fearless and push things forward with a reasonable level of caution. If it was disrespectful of me to launch this without another ping, I apologize. I probably am much less patient with the system, given that I got into distribute/setuptools maintenance over a decade ago in order to support Python 3 (and supersede Python 2).

I also don't think a data-gathering period will be all that helpful for the problems I'm concerned about.

:/

If those problems emerge (including lots of people pinning to Setuptools < 45), we can readily release a new version of Setuptools with a clear message that we're bringing back support for Python 2.

No amount of a bunch of people pinned to a working version of setuptools is going to tell us whether this is going to cause problems when we want to improve PEP 517 support in the future.

I'm hoping that most pinning is selective (on Python 2 only) if at all. I get what you say about the implicit pin.

That said, it is way more work to maintain a system of backports than to continue maintaining compatibility.

I'm less sure about this concern. Much like CPython did, the Setuptools project can now selectively backport only those features that the team deems necessary for long-term support, and can sooner than later remove the compatibility layers in master. If the backporting gets too onerous, we can readily restore Python 2 support in master.

[dstufft]

For whatever it's worth, I do think that this is premature. Obviously I'm not a setuptools maintainer so take my opinion here with a grain of salt. I've generally been a big fan of data based decisions, so looking at what data I have available to me, in the last month, 58% of setuptools downloads from PyPI were for Python 2.7.

So what this is effectively saying is that for let's say roughly half of setuptools users, they cannot use any new feature of the packaging ecosystem if they use setuptools for their build system. This has larger implications across the entire ecosystem, it's also telling every project out there that uses setuptools for it's build system that they have to also drop support for Python 2.7, or they have to pin to a version that does support Python 2.7, or they have to play the same kind of game we've done with Python 2/3 and figure out what subset of options works the same across the 2.7 supporting and the 3.x supporting versions of setuptools.

All in all, I think for a project like setuptools (and pip, etc) it does far more harm to our users to drop support for 2.7 than the cost of keeping compatibility warrants.

[asottile]

there's an additional Python :: 3 :: Only classifier that can be useful setup-cfg-fmt can add it automatically based on python_requires

[kdelee]

I am aware that I have been slow on reviews and stuff, but I've tried to communicate that I was uncomfortable with moving quickly on this.

Please don't feel bad; I'm trying to be fearless and push things forward with a reasonable level of caution. If it was disrespectful of me to launch this without another ping, I apologize. I probably am much less patient with the system, given that I got into distribute/setuptools maintenance over a decade ago in order to support Python 3 (and supersede Python 2).

I also don't think a data-gathering period will be all that helpful for the problems I'm concerned about.

:/

If those problems emerge (including lots of people pinning to Setuptools < 45), we can readily release a new version of Setuptools with a clear message that we're bringing back support for Python 2.

Current changes are breaking some of our builds with the following error (not warning):

[2020-01-13T03:25:13.237Z]       Processing ./OURAPPNAME/requirements/vendor/setuptools-45.0.0.zip

[2020-01-13T03:25:13.237Z]       ERROR: Package 'setuptools' requires a different Python: 2.7.5 not in '>=3.5'

Only resolution we have for this is to pin to <45 AFAIK

FWIW our app actually uses python3, but we are building for a platform that uses python2 as the system python.

No amount of a bunch of people pinned to a working version of setuptools is going to tell us whether this is going to cause problems when we want to improve PEP 517 support in the future.

I'm hoping that most pinning is selective (on Python 2 only) if at all. I get what you say about the implicit pin.

That said, it is way more work to maintain a system of backports than to continue maintaining compatibility.

I'm less sure about this concern. Much like CPython did, the Setuptools project can now selectively backport only those features that the team deems necessary for long-term support, and can sooner than later remove the compatibility layers in master. If the backporting gets too onerous, we can readily restore Python 2 support in master.

[jaraco]

Current changes are breaking some of our builds

@kdelee First, sorry for the breakage; we were aiming for this release not to break things in the general case. It looks like you've run into an issue where you're using some one tool that doesn't honor Python-Requires to resolve/download setuptools, but then using pip to install it, so Python-Requires isn't honored for the first step, but then is enforced in the second step. I think you have several options.

• During install, pass --ignore-requires-python. That will likely allow the install of setuptools 45 on Python 2. However, I would recommend against that, as that will only push the problem down the road.
• Pin the requirement to <45. Is pinning a straightforward adjustment for you to make?
• Honor the Python-Requires metadata in the package when downloading, either by using pip 9 or later to resolve an unpinned "setuptools" requirement (for the python you're targeting) or otherwise honoring the metadata. You'll probably want to do this anyway as more and more packages are dropping support for Python 2 and so you'll need to do this for other packages.

Please help me understand the impact - does Setuptools provide the necessary tools for you to navigate this transition? In particular, does this affect new releases or does it also affect existing releases of your app? I had hoped for this change to require minimum intervention, so let me know if you don't have a straightforward way to address the concern, and in particular, what steps you run that lead you to this consequence.

[kdelee]

short story: we are going to pin on <45 and this is in an RPM build environment for rhel7, where python2 is the default python.

long story: trying to get one of our more build-focused devs (I'm just the one who found out the build was broken) to try and give you a more detailed response.

[exarkun]

This change broke Tahoe-LAFS CI for several platforms with older virtualenv/pip. I've put up a PR for Tahoe-LAFS to pin setuptools 44 - tahoe-lafs/tahoe-lafs#676.

[jaraco]

This change broke Tahoe-LAFS CI for several platforms with older virtualenv/pip. I've put up a PR for Tahoe-LAFS to pin setuptools 44 - tahoe-lafs/tahoe-lafs#676.

I looked into this, and here's what is happening:

• The project uses find-links to resolve packages.
• (I presume) pip doesn't honor Requires-Python in find-links, so grabs Setuptools 45 even on Python 2.
• If you have a late version of pip (I presume 9+), it will refuse to install Setuptools 45 on Python 2 and the breakage occurs.

Your suggestion to pin setuptools is probably the most expeditious at this point. Since you're using wheelhouse, you could also avoid putting setuptools >= 45 in the wheelhouse.