Python 2 deprecation schedule

[pganssle]

I've been giving a lot of thought as to the Python 2 deprecation schedule for dateutil, and I think I have come up with a tentative plan. Before I go ahead and put it on Python 3 statement, I'd appreciate any comments if people think this is unrealistic or could be done better.

Here is my current plan and timeline:

1. August-September 2018: 3.0 release - Supporting Python 2.7, 3.4+
   We have a lot of deprecated behavior in the 2.x branch that I'd like to remove, and a few small breaks to backwards compatibility that I'd like to make that aren't really something that can be deprecated (e.g. the behavior of dayfirst and yearfirst that has so many people pinned on 2.4.x). Ideally, we'd also have the new parser up and running before then.
2. January 2019: 4.0 release - Supporting only Python 3.4+. Version 3.0 becomes LTS, bugfixes only.
   This release should make basically no breaking changes other than dropping support for Python 2, so that people can write Python 2/3 compatible code pinned to python-dateutil < 5.0.0, rather than doing more complicated backports.
3. July 2019 - Final bugfix release for version 3.0. Zoneinfo support continues indefinitely.

After the 4.0 release, I think we'll want to make at least a 6 month deprecation period before any more backwards-compatibility breaking changes are made.

Somewhere in here I'm also planning to put in a compiled backend, but I think that will be opt-in for at least the 2.x branch.

@Carreau gave a talk about how best to drop support for Python 2 that I basically agree with at PyBay 2017. You can find the video here.

CC @jbrockmendel @jdufresne @jreback

[mgedmin]

I think you meant "pinned to python-dateutil < 4.0.0".

[pganssle]

@mgedmin No, I meant pinned to python-dateutil<5.0.0 (though you're right, I didn't mean <=). The point would be that code written for 3.0 would still work with code written for 4.0, except that the 4.x branch doesn't work with Python 2. If you pin to python-dateutil<5.0.0 on your app that supports both Python 2 and 3, your Python 2 users will get something from the 3.x line and your Python 3 users would get something from the 4.x line, assuming they use a pip that supports Python-Requires.

If you have an app or library that is Python 2-only, you'd pin to python-dateutil<4.0.0.

[mgedmin]

Ah, I see! I'm still not used to Python-Requires (which needs a very recent pip, IIRC).

[Zac-HD]

The overall plan sounds good, but I'd explicitly stop any free support for Python 2 at 2020-01-01, which would mean dropping the reference to 'indefinite' zoneinfo support. Paid support is a different story, but that distinction is IMO important for both the 2/3 transition and OSS sustainability communities.

[pganssle]

@Zac-HD I'm not really in a position to accept any payment for support, nor am I particularly interested in that. By "indefinite" I mean, "so long as it does not significantly inconvenience me to do so", but given that before the 3.0 release I'm planning on splitting the zoneinfo package into a separate data-only package, it may well come free with providing zoneinfo updates to Python 3 (since it really just involves publishing a package containing a tarball).

[Carreau]

Ah, I see! I'm still not used to Python-Requires (which needs a very recent pip, IIRC).

Yes, Dec 2016 (pip 8+). So mostly old systems with system pip will install old-incompatible version by missing the Python-Requires field.

There is a short recap of the talk here if you prefer to read. TLDR being to explicitly fail in setup.py if you detect python2 to avoid installing an incompatible version.

Exited to see more people joining the Py3 statement.

As for paying support, as far as I understand nothing prevent a company to provide a python 2 compatible fork with support. The only question is about the trademark/usage of the name – which could be arrange by a contract between the trademark/name owner and an interested company.

[mahmoud]

Generally, I haven't found maintaining 2/3-compatible code all that difficult, and I think it's worth it for a lot of less-vocal users. Granted, maybe there are changes with fold and other aspects of datetimes that make this undesirable. I'd like to see 2 support continue indefinitely, because for an unfortunate number of projects, ending 2 spells ending their usage of Python.

Anyways, aside from that general sentiment, I mostly came here to let @Carreau know about http://collabmark.org/ (if he didn't already). Open-source trademark sharing doesn't necessarily require a bespoke license/contract! :)

[Carreau]

Anyways, aside from that general sentiment, I mostly came here to let @Carreau know about http://collabmark.org/ (if he didn't already). Open-source trademark sharing doesn't necessarily require a bespoke license/contract! :)

Thanks ! I didn't knew and will share that with the Jupyter team.

Python 2/3 simultaneous support can be easy for some project and hard for others. We dropped for IPython because some area were truly impossible to maintain in both.
For many python 2 users they actually don't need new version of their software, so dropping is no an issue.

Sidenote, for Py2/3 compat we maintain an open backport bot (used by Jupyter/IPython/matplotlib) (https://github.com/MeeseeksBox/MeeseeksDev), hosted on heroku, you don't need to host your own. So PRs that trivially can be backported are just one mention away.

You just need to register you project with the GitHub integration (see readme), and either mention the bot or modify your milestone description to auto backport on merge.

That had a huge impact on our willingness to backport minor changes.

[pganssle]

@mahmoud If people are willing to use a 10+ year old, unmaintained version of Python, I don't see why a 2-year-old unmaintained version of the (quite stable) dateutil library would somehow be a dealbreaker for them.

If you're using only Python 2 and Python 2 features, then yeah it's easy to maintain 2/3 compatibility, but there are a ton of useful features in Python 3 that can't be trivially backported, for any project, type annotation and f-strings, for a few examples.

Additionally, as more and more libraries drop Python 2 support, maintaining polyglot libraries will get harder and harder. I recently dropped Python 3.2 support not because I found it difficult to maintain, but because a whole bunch of libraries in the testing stack (pytest, tox, e.g.) dropped it, and there isn't a version of pip that supports Requires-Python for Python 3.2, making it decidedly non-trivial to cobble together a testing stack.

I think it's important to do this now, with plenty of notice and forethought and a decent period of overlap where the same interface is supported for both versions so that we aren't creating more work for people looking to transition from 2 to 3 in a seamless way in the future, or needlessly breaking a bunch of 2-only code that already "just works".

@Carreau Thanks for mentioning the backport bot - you told me about that at PyBay but I totally forgot about it - even when I was looking for something exactly like that just a week or two ago when trying to work out the best workflow for parallel release branches.

[Carreau]

Thanks for mentioning the backport bot - you told me about that at PyBay but I totally forgot about it - even when I was looking for something exactly like that just a week or two ago when trying to work out the best workflow for parallel release branches.

Let me know if it works/if you want more informations. Right now it requires push access but I have plan to avoid that. Just need to write the code.

[mahmoud]

@Carreau Let me know what they think about CollabMark!

@pganssle It's not really fair to call 2.7 10+ years old, or if one does, to use that as a criticism. Almost all of Python-the-language's best features are even older than that. It's also not fair to compare Python 2.7.x to anything other than Python 3 tip. The IRL adoption (and real-world testing) have long been night and day.

I've always been a little surprised at the preemptive drop. Frankly I don't know what 2020 will bring. I see a lot of legacy projects saying they're gonna drop 3, but I also see a lot of parties stepping forward to say they'll continue support. At the very least, it would be considerate to document the hard technical reasons why py2 support would overburden a project to the point of dropping it before PSF support ends.

[pganssle]

It's not really fair to call 2.7 10+ years old, or if one does, to use that as a criticism.

I don't mean it as a criticism, I just mean that people using Python 2 probably are more concerned with stability than with getting the latest features. dateutil's core feature set is quite stable, just like Python 2's. When we drop support, it doesn't mean that users from Python 2 can't use dateutil anymore, it just means they can only use the perfectly good 3.x branch, or they have to write their own backports of new features.

At the very least, it would be considerate to document the hard technical reasons why py2 support would overburden a project to the point of dropping it before PSF support ends.

Honestly, the "hard technical reasons" aren't that important. At some point Python 2 support will end. I think many of the people participating in the Python 3 statement are looking to make that transition easier by acting as canaries in the coal mine. For example, IPython's early dropping of Python 2.x support in the 6.x branch has been a relatively harmless demonstration of a lot of weakpoints in systems that don't support the Requires-Python metadata, allowing them to be fixed ahead of the onslaught of packages dropping Python 2 support. When matplotlib drops support for 2.x in their 3.0 release this summer, I think we'll see a lot more of these kinds of bugs cropping up as well. I think it's important to give the ecosystem time to cope with this and develop idioms around the transition, rather than everyone coordinating on January 1, 2020 as the "we can drop Python 2" day.

[Carreau]

Let's not (re)start a discussion on if python 2 support should be dropped or when, at least not here, that would be a slippery slope. This has been hashed and rehashed many times.

Let focus on making sure dateutil have a nice and planned transition that disturb users the least.

[mahmoud]

@Carreau Not sure what made you think that's what was happening here, my core suggestion was to document what, if any, Python 3 features or Python 2 misfeatures were leading to an earlier drop in support.

The Python 3 statement isn't news to me, though its list of scientific packages has grown a bit since I last looked. Hopefully I'm not being too on the nose by pointing out that -- aside from the almost-non-sequitur Zulip, Kivy, and mitmproxy -- dateutil is the least scientific name on the list. Even the most basic blog has good reason for using dateutil. So, some additional documentation/justification for the wider audience may be in order.

[mahmoud]

Also, probably well-known by the current thread participants, but libraries.io gives a pretty good sense of just how many non-scientific projects depend on dateutil. Granted that's mostly FOSS, and the caution I'm trying to articulate is mostly motivated from industrial and personal use. :)

[pganssle]

The fact that dateutil hits a wider swathe of libraries than the ones already dropping Python 3 support is an argument in favor of doing it, not against. As I've mentioned before, "dropping support" sounds much more dramatic than it actually is. Python 2 stopped getting new features years ago, so basically everything that applies to dateutil applies to Python 2 itself, and the only people actually affected by this are people still using Python 2 despite its support being dropped. The only thing that actually likely does need to be continually updated is the zoneinfo files, and I have no plans to drop Python 2 support for that any time soon.

Given that "using an old version of dateutil" is really not going to be that big a deal, if it's actually true that this will be the first library with a 2.x LTS branch and a 3.x-only branch that most people see, I think it's a good thing to spur people on to the reality of dealing with these problems before everything else has broken, so that the transition can happen in an orderly manner.

One thing I haven't mentioned so far because it isn't really relevant is that if there were a security-critical bug (something that I think has not happened yet in the history of dateutil) and someone submits a PR for it, I'd be willing to cut a new release on the 3.x branch beyond the cutoff deadline. Given that that has never happened and the fact that stable libraries like dateutil and pyyaml have gone years between releases without anyone noticing or really caring, I honestly think the biggest problem that will be caused here is making sure everyone has a package manager that respects Requires-Python, and that's already a problem (from when I dropped Python 2.6: problems in botocore, problems installing in Python 2.6) that's only going to get worse.

[tsellers-r7]

@pganssle - Was a decision made on this? I looked at the site but didn't find a definitive schedule or published statement.

Thanks much.

[ffe4]

@pganssle, I would also like to know whether there is anything new. dateutil was supposed to have already stopped supporting python 2, but this discussion seems to have ended almost two years ago.

Are there any plans to work something out in the near future? Is this issue stale enough to be closed?

[pganssle]

So yeah, this obviously got way away from me and did not go according to plan. I was hoping I would have had more time by now to work on all the major breaking changes that I would need to do, but alas.

Given that people have been happy with the minimal major updates to this package so far, I propose a revision to the plan:

1. Wait until PEP 615 is accepted (hopefully no more than a week or so) and take a dependency on tzdata to solve issue Move zoneinfo data to a separate package #701.
2. Merge the few minor breaking changes I've been putting off for 3.0 (see 3.0 milestone).
3. Maintain this 3.0 version for no less than 3 months and until no later than January 1, 2021.
4. Cut a 4.0 release dropping old Python version support.

[ffe4]

Definitely sounds good to me. I haven't really looked at the milestone yet but I'll see if there is something in there I can help with. I have also just closed #8, which made me think that we should probably also think about deprecations. Actually removing them could probably wait until 4.0, but I wanted to make sure that there are no features left that we might want to deprecate with 3.0 and that are not on the milestone.

Also, just to be clear, will we be dropping version support for all old Python versions including 3.5 if the 4.0 date lands after 2020-09-13 (Py3.5 EOL)?

[Safihre]

Wondering if removing 2.7 support is still on the radar?

[mariocj89]

It is indeed, but we are waiting for #1130 to land first, so we can split tzdata from this package and allow users to continue taking new tzdata releases without having to reinstall dateutil.

[walles]

@mariocj89 is #1130 still being waited for? Should it?

@Jorricks said in #1130 (comment):

The code is very complex and you are only doing this for python 2 users. Given that reached end of life years ago, I'm not to enthusiastic to work on this anyway.

[mariocj89]

Yeah it is sadly waiting for the PR, if @pganssle is happy to drop it altogether we can also get this moving.

[kytta]

Well, it didn't go according to plan, and it's alright. I propose a way more radical solution, which is not that bad, since Python 2 should be dead by this point. Here's a (proposed) checklist that would finally move this issue forward:

• abandon Backport zoneinfo logic into dateutil.tz.tzfile #1130, as it tries to fill the Python 2 users' needs, which doesn't make sense in 2023
• remove all Python 2 code, six, and everything else. https://github.com/asottile/pyupgrade might help.
• remove vendored tzdata in favour of https://github.com/pganssle/zoneinfo
• release a new version that is Python 3 only. The only changes are the required Python version and zone info.
  • since the package will be specified as py3-only, there's no need to make this a major release IMO, because it would be impossible to install it on py2 anyway (pip won't allow this)
• release a patch version (2.8.3, I suppose) with deprecation warnings that are shown when people use Python 2
• Postpone every other bugfix/feature to the next milestone

Python 2 has been dead for three years now, and I'm tired of seeing six in my dependency tree :)

[pganssle]

I think we'll just go ahead with a modified form of the existing plan. #1130 still needs to be merged, and the only part that still needs to be finished is the one part of it that you suggested keeping, which is removing the vendored tzdata in favor of... I'm guessing you meant PyPI's tzdata rather than zoneinfo?

[pganssle]

By the way, #1130 is really not that hard for a motivated individual to finish, I just have approximately 0 time for any of this stuff. I've said there's no problem if someone wants to take it over.

[kytta]

[...] which is removing the vendored tzdata in favor of... I'm guessing you meant PyPI's tzdata rather than zoneinfo?

I actually meant zoneinfo, because it tries to load the zone data from the OS first, but I guess in your case we can simply go with tzdata, yes