Drop Python 2.7 & 3.5

[mayeut]

Per #580 (comment)

Drop Python 3.5 support. (CPython 3.5 has reached EOL on 2020-09-30).
Drop Python 2.7 support. (CPython 2.7 has reached EOL on 2020-01-01).

Close #438

[YannickJadoul]

This is quite a bit more cleanup than I had expected. We had a lot more 2.7 workarounds/special cases than I thought!

[henryiii]

<rant>

I feel like I have to repeat myself here, but the key argument I had was that we should exactly support what manylinux supports. If they drop something, we drop it, but if they don't, we should not either. I view cibuildwheel on linux like a tool to build anything manylinux supports. If you can talk manylinux into dropping 2.7 support and PyPy 2 support in general, then I think we should drop it. But if you can't, then we should drop it (and manylinux1) when manylinux1 is dropped this summer. NumPy is planning on dropping manylinux1 this summer too, which would help.

People use cibuildwheel because it is easier than writing all the custom code themselves. Actually, most users start trying to build CI wheels themselves, thinking that they'd rather be in full control, but then eventually run into enough of the things that we have already solved for them that we win them over. If there are a bunch of things you can do by hand that cibuildwheel can't do, like CPython 2.7, PyPy 2.7, and CPython 3.5, then those could be reasons to drop cibuildwheel; building by hand is more powerful/better, then. Now when manylinux1 drops support or is dropped, then that's different; building by hand is not supported upstream. Basically, it's upstream's fault, not ours.

cibuildwheel is not a PyPA project; we have to convince people to use us. Manylinux1 is a PyPA project; they are viewed as the "official" thing to use to build wheels; if manylinux1 drops Python 2, some projects using it will drop Python 2. We are not as likely to have that luxury.

</rant>

Okay, practically, we are quickly coming up on the end of reasonable Python 2 support, and Python 2 users are going to have to pin everything, including cibuildwheel. And 2.0 is a perfect time to do it. Also, I don't see any pressing need to push on to a 2.0 release right away, so maybe master could drop 2.7 (and 3.5?), but have a release slated for Summer. We could make a 1.x branch, and make a docs build for it, and keep it alive until the 2.0 release (or after for critical bug fixes or CI changes).

PyPy2 is the sad loser here, but without NumPy or anything else, it's going to be pretty limited in its need for wheels, I think. Though I'd still take input on whether to drop PyPy2 from the PyPy team, and follow manylinux.

Also, keep in mind, just like dropping 3.5, keeping 2.7 alive for now is not a high cost (in fact, #536 made it more elegant to keep both). Making the 2010 -> 2014 like change, where suddenly 2010 stops building Python 2 wheels, is actually a rather big change (1.10 worthy, perhaps?), but one we can blame manylinux for; it's not our fault and users can pin.

[joerick]

My feeling is still to drop 2.7. I don't like keeping 2.7 support in manylinux1, because manylinux2010 is our default, so we get a messy situation where it's built on mac, windows, but linux requires setting manylinux_image... plus it feels like we've already given people plenty of time.

Regarding matching the manylinux policy, I think that's up for debate, but this is a wider conversation than manylinux, because we're talking about cibuildwheel's policy as a whole, which affects macOS and Windows too. Ultimately, we as cibuildwheel maintainers make the decision (although admittedly the manylinux change has given us the impetus to do so).

And yes, I agree that dropping Pypy2 is an unintended consequence of the CPython2 EOL. But that's true across the Python ecosystem, all library authors want to drop the Python 2 language for maintenance reasons. So I don't think we should worry too much about that - it reflects a broader trend in the ecosystem.

[henryiii]

For our project list, these are the packages that still produce Python 2 wheels:

• twisted
• psutil
• numexpr
• bx-python
• jq
• rtree
• dependency-injector
• ddtrace
• pyinstrument_cext
• autopy
• pyjet
• numpythia
• boost-histogram

I've just released the final 2.7 release of boost-histogram, and numpythia/pyjet are also Scikit-HEP projects, so I wouldn't worry about them. That's about 25% or less, but there are still some major packages there. I doubt a package like twisted or psutil would drop Python 2 support because cibuildwheel does (they might if manylinux drops it, or maybe wheel; they are more likely to drop us instead). I really thought MyPy would be listed above, but it only checks Python 2 code, it doesn't run on Python 2. :) It does get listed along with most of the above packages if you check for Python 3.5 wheels. I believe all of the above packages ship manylinux1 wheels (pretty much everyone does; the fact that Ubuntu 18.04 can't access manylinux2010 with Python 3.6 or even Python 3.8 is a killer)! Roughly 75% our our example users ship a manylinux1 wheel. And this is compared to the total, including packages that don't ship linux at all.

So I think it's getting small enough that we could call 1.x a LTS release, make a branch for maintenance and docs, then move ahead with 2.0. The two big possible features to be added (cross-compile and better Limited API support) both are only Python 3 anyway. It does double our maintenance for a while, so I would not suggest this just to "simplify" our code base; it does not, it complicates it for a while, but it would reduce pressure on trying to workaround the manylinux2010 image now missing 2.7 and that being our default. I don't want to lose the support of the packages above, and unlike NumPy or other libraries, cibuildwheel has to adapt to changes in CI, new manylinux1 releases, etc, so we can't completely drop the 1.x branch until this summer. This will also be the only way to build for PyPy 2, I expect.

If we want to keep this in a single codebase without supporting two branches for a while, we could query the image, and only try to build Pythons that exist in the image. Or, we could just throw an error if someone tries to build Python 2.7 on manylinux2010; very few of our projects do that (2-3, I think), and it should be an error anyway, to tell people to stop using it. This would be a "softer" error than dropping it completely, and would hopefully move users away from 2.7 by the time it is dropped this summer.

PS: I'm assuming the worst and that 1.9.x will take a lot of maintenance due to drastic CI changes or something; it might actually take very little.

[henryiii]

I think we should probably go 3.6+ only for 2.0, dropping lots of 3.5 workarounds too, and making Big Sur simpler to support (though maybe checking with mypy first to see when they plan to drop 3.5; I think they are the largest (and almost only) user to support 3.5 but not 2.7).

[henryiii]

Also, if anyone wants to do similar analyses, I'll dump my script into a discussion for now.

[joerick]

Thank you for putting that list together, that's very useful :) Looking at it I note that the main twisted library is actually python_requires = >=3.5.2, the CPython 2.7 wheel would be redundant there. psutil support actually goes back to 2.6 and Python 3.4, but they only do 2.7 and 3.6+ wheels, so I guess they'd be okay with the change. But yes, the rest would need the pipx cibuildwheel==1.9.x ... workaround if they wanted to keep support. I also see that pyinstrument_cext is in there, I can definitively say that one mostly only still has support because I never bothered to remove it :)

On releases, I think it would be worth a minor release for #595, so it might be 1.10.x that is the LTS, we'll see. Dropping Python 2.7 and 3.5 and calling that a 2.0 release makes sense to me. And there probably is no rush for it either. We'll have to figure out something sexy to add in there too, perhaps the limited-api support can serve as the shiny new feature :)

[henryiii]

If we can get cross-compiling, that would be a good 2.0 feature. :) Like Limited API, it's Python 3 only, too. :)

If we have any cleanup to do, like Blacking a bit more or #593, that should also go in before we make a v1 branch. If we keep unimportant divergences (like styling) to a minimum between the branches, it will make it easier to backport for a period if need be.

I noticed some weirdness about Python-Requires and twisted; I feel like they are playing with dropping support before fully dropping it? I'm also guessing that MyPy will drop 3.5 in the near future, maybe April?

Unrelated, I feel killing off manylinux1 while Ubuntu 18.04 exists is going to be a real (but necessary) headache....

[henryiii]

Well, I was actually thinking we'd drop 3.5, then split for 1.x and 2.x, then drop 2.7, as I don't really have interest in supporting 3.5 after manylinux drops it even in an LTS version. Though admittedly, projects tend to drop 2.7 and 3.5 together, and we are stuck on a manylinux 2010 image for 1.x, so maybe keeping 3.5 on 1.x is best (though especially if it requires a bit of maintenance, I'd be happy to drop 2.7 on manylinux2010 and only support it on manylinux1; most of our users set manylinux1 anyway, probably nearly all that support 2.7 do.)

Happy to help rebase (on any of these PRs, actually) after the black merge.

[henryiii]

I'd at least like the order of the two commits to be reversed, if that's not too hard. Again, if I ask for something to be done, I usually willing to do it myself if needed. ;)

[henryiii]

Would it be hard to flip the order of the two commits? Just in case we need to punt and drop 3.5 from the 1.x branch in the future, might make it easier to cherry-pick.

[mayeut]

Would it be hard to flip the order of the two commits? Just in case we need to punt and drop 3.5 from the 1.x branch in the future, might make it easier to cherry-pick.

It's ready. I did this in 2 times. 1st rebase then flip the order. I'll wait for a green status (I have a backup branch with this order just in case there's something wrong with the flip) & I'll force push again.

[mayeut]

Now ordered "drop 3.5" then "drop 2.7"

[henryiii]

Reminder: needs rebase and merge, not squash and merge!

[henryiii]

Once we make a v1 branch, I think this is good to rebase and merge. I think we should also add links to the docs for Python 2.7 / PyPy2 that point at the v1 docs, but we can wait till we have v1 docs to do that.

[joerick]

Beautiful. A tear in my eye. So many hacks and bodges removed. 😆

I've created the 1.x branch so we are good-to-go on merging this!

[henryiii]

I've added a second set of 2.7 drops and a few doc links in 249b1cd - please check my doc links to make sure they look somewhat reasonable, then we'll need to merge the 2.7 commits together if everything passes.

[henryiii]

One AppVeyor job stalled, probably not caused by this? Can someone restart it?