Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: Propose standard policy for dropping support of old Python versions #14086

Merged
merged 36 commits into from Aug 19, 2019

Conversation

@tacaswell
Copy link
Contributor

commented Jul 23, 2019

This NEP is a proposing a standard policy for the community to determine when we age-out support for old versions of Python. This came out of in-person discussions at SciPy earlier in July and scattered discussion across github. This is being proposed by maintainers from Matplotlib, scikit-learn,
IPython, Jupyter, yt, SciPy, NumPy, and scikit-image.

TL;DR:

We propose only supporting versions of Python initially released in the preceding 42 months of a major or minor release of any of our projects.

tacaswell and others added some commits Jul 19, 2019

@tacaswell

This comment has been minimized.

Copy link
Contributor Author

commented Jul 23, 2019

@charris charris changed the title Propose standard policy for dropping support Cpython ENH: Propose standard policy for dropping support Cpython Jul 23, 2019

@charris

This comment has been minimized.

Copy link
Member

commented Jul 23, 2019

RuntimeError: Title for NEP 28 does not start with "NEP 28 — " (note that — here is a special, enlongated dash)

@charris

This comment has been minimized.

Copy link
Member

commented Jul 23, 2019

Should send a note to the mailing list. I can do that if you don't want to subscribe.

@Carreau

This comment has been minimized.

Copy link
Contributor

commented Jul 23, 2019

👍 support nep 20 badge.

release in Dec19 should support py36 and newer, and a project
with a minor release in Nov20 should support py37 and newer.

The current CPython release cadence is 18 months so a 42 month window

This comment has been minimized.

Copy link
@willingc

willingc Jul 23, 2019

Contributor

Just an FYI. It is highly likely that the release cadence will become shorter than 18 months for CPython.

This comment has been minimized.

Copy link
@tacaswell

tacaswell Jul 23, 2019

Author Contributor

All the more reason for this!

doc/neps/nep-0028-deprecation_policy.rst Outdated Show resolved Hide resolved
doc/neps/nep-0028-deprecation_policy.rst Outdated Show resolved Hide resolved
doc/neps/nep-0028-deprecation_policy.rst Outdated Show resolved Hide resolved
doc/neps/nep-0028-deprecation_policy.rst Outdated Show resolved Hide resolved

- support minor versions of ``CPython`` initially released
42 months prior to our planned release date
- always support at least the 2 latest versions of ``CPython``

This comment has been minimized.

Copy link
@mdhaber

mdhaber Jul 23, 2019

Contributor

Does this mean Project X Version Y will support all minor version of CPython released in the 42 months prior to the release of X vY or the two most recent minor versions of CPython released before X vY, whichever is "greater"?
(I don't mean to suggest this is a clearer way of writing it; just checking understanding.)

This comment has been minimized.

Copy link
@tacaswell

tacaswell Jul 23, 2019

Author Contributor

Yes, as commented below, this is hedging in the case that the Python release cycle slows down significantly (worst case, we don't have a Python minor release for 42 months and the rule says we support no Python ;) ) .


As the first bug fix release is typically a few months after the
initial release, you can achieve the same name effect by making the
window longer which is easier to explain.

This comment has been minimized.

Copy link
@mdhaber

mdhaber Jul 23, 2019

Contributor

Is "same name effect" a typo?
"X.Y.1 release" means "first bug fix release"?

You can achieve the same name effect by making the window longer

here, "window" refers to the window relative to the minor release (rather than "window" relative to the first bug fix release),

which is easier to explain.

It is easier to explain a window relative to a minor release than it is to explain a window relative to a first bug fix release?

Maybe the distinction between the two is not so easy to explain : ) I'm afraid I had to read this several times before I understood.

This comment has been minimized.

Copy link
@tacaswell

tacaswell Jul 23, 2019

Author Contributor

Tried to re-word this, hopefully it is better now?

doc/neps/nep-0028-deprecation_policy.rst Outdated Show resolved Hide resolved
Given the current release cadence of the CPython, the proposed time
(42 months) is roughly equivalent to "the last two" CPython minor
versions. However, if CPython changes their release cadence, any rule
based on the number of minor releases will need to be changed.

This comment has been minimized.

Copy link
@mdhaber

mdhaber Jul 23, 2019

Contributor

But isn't part of the rule being suggested based on the number of minor releases?

always support at least the 2 latest versions of CPython

By version, do you not mean "minor release" e.g. 3.6 and 3.7?
https://devguide.python.org/devcycle/

Sorry if I'm misunderstanding - but if I do, maybe others would?

This comment has been minimized.

Copy link
@Carreau

Carreau Jul 23, 2019

Contributor

The rule is based on time (42 month, 3 and a half year), which happen to be 2 minor version of (C)Python with current release cadence. As pointed by Carol, the release cadence of Python is likely to go up so that would be 3 or minor releases if the new PEP is accepted.

This comment has been minimized.

Copy link
@mdhaber

mdhaber Jul 23, 2019

Contributor

The rule is based on time (42 month, 3 and a half year), which happen to be 2 minor version

That's what I thought, as the document is pretty explicit about where the 42 month number comes from. In that case, I would suggest "always support at least the 2 latest versions of CPython" should be dropped from the recommended project development guidelines, as it is redundant. If we want to leave it in there for clarification, maybe it should be explained below the bullets so it does not look like an independent rule.

This comment has been minimized.

Copy link
@tacaswell

tacaswell Jul 23, 2019

Author Contributor

This is mostly as a hedge in case anything goes sideways upstream and we don't have a Python release for 42 months. In that case we may have bigger problems.....

tacaswell and others added some commits Jul 23, 2019

DOC: Apply wording changes from @mdhaber
Co-Authored-By: Matt Haberland <mdhaber@mit.edu>

@tacaswell tacaswell changed the title ENH: Propose standard policy for dropping support Cpython ENH: Propose standard policy for dropping support of old Python versions Jul 23, 2019

@tacaswell

This comment has been minimized.

Copy link
Contributor Author

commented Jul 23, 2019

Thank you everyone for the copy editing, I swear I actually read it before I opened the PR....

@adrinjalali

This comment has been minimized.

Copy link

commented Aug 1, 2019

If this proposal is making decisions under a certain assumption regarding CPython's release cycle, euther that cycle's duration should probably be included in the formula (if decisions are made based on time), or a number of releases should be included in it. That's why I think @rth 's suggestion of having something along the lines of

Or maybe a rule about supporting CPython X number of month, but for no more than 3 minor Python versions, to address maintenance concerns?

is not a bad idea.

@willingc

This comment has been minimized.

Copy link
Contributor

commented Aug 1, 2019

@charris Can you please remove the meme? It's not relevant to the discussion. Thanks!

@charris

This comment has been minimized.

Copy link
Member

commented Aug 1, 2019

@willingc Sure. Humor on the internet is always dangerous territory.

@willingc

This comment has been minimized.

Copy link
Contributor

commented Aug 1, 2019

Thanks @charris. I appreciate it. ☀️

@Carreau

This comment has been minimized.

Copy link
Contributor

commented Aug 13, 2019

Hi all, were there any discussion about this on the NumPy call ?

I believe the usual process with Pep is when there is a significant number of people with opposite views, they can submit a competing Pep.

Can I suggest we go this route instead of trying to have @tacaswell try to figure out how to update this document ? It will be easier to trac people and projects that are would adopt it as-is vs those who won't.

@Carreau

This comment has been minimized.

Copy link
Contributor

commented Aug 13, 2019

Now with my hat of maintainer of IPython; I'm planning on adopting this schedule.

I also want to note that we do not enforce it on the community, but are just a group of project that agreed this is was a good compromise, and we can still decide to adhere to NEP29, and decide 6 month later it was a mistake. It would be sad and confusing but it's a possibility.

@seberg

This comment has been minimized.

Copy link
Member

commented Aug 13, 2019

There was some discussion yes, it was likely posted here before by Matthew, but:
https://github.com/matthew-brett/pypi-by-distro/blob/master/analyze_pypi_downloads.ipynb

was data Matthew gathered. The issue is that it is hard to say how much of the users downloading new numpy versions on LTS systems is due to testing. I know a few people from previous work who would use such ancient LTS systems, with arguments such as "I will not update until I finished my PhD". But many might not even notice if they get an older numpy version (OTOH, bug fixes will not hit that).

That means the discussion was, that if we extend (for NumPy), the logic to encompass about 3 Python versions, we basically cover more (most?) of the full LTS live span. Of course the counter argument is that installing a newer python version is not all that much of a hassle nowadays.

The question is what we want for NumPy. If we tend towards supporting a bit more, I think all that is needed is one paragraph saying something like: "This is a guideline and in exceptional cases projects with a large dependency base may choose to have longer support. At this time NumPy will aim to support 3 Python versions leading to an XX month long support."

I do actually believe there is a userbase out there that sticks to LTS until end of life, e.g. in sciences, and they may occasionally need newer versions. But it is very hard to tell if the intersection of "need new version" and "on ancient system" is large enough to worry about.

@rgommers

This comment has been minimized.

Copy link
Member

commented Aug 13, 2019

To add to what @seberg said, a conclusion from the call was: while these older Linux distros are not critical, the PyPy lag and other more niche systems (Raspbian et al.) may be. At least those are interesting to consider in more detail (more so than LTS Linux distros) - while our user base on those platforms is not that large, we shouldn't make the lives of the maintainers of those platforms unnecessarily hard.

@charris

This comment has been minimized.

Copy link
Member

commented Aug 13, 2019

I suppose the word "support" should be clarified. When I use the word, it means only that the release can be compiled and executed with the supported python versions.

@GaelVaroquaux

This comment has been minimized.

Copy link
Contributor

commented Aug 14, 2019

@rgommers

This comment has been minimized.

Copy link
Member

commented Aug 14, 2019

In my anecdotal experience, this user base is huge.

@matthew-brett did some research based on PyPI download stats, and his notebook (linked above) gives an answer: roughly 6+-3% or so. The vast majority of those would be fine with version N-1 (where N is newest). So: yes there's a user base, but it's not the most important issue here.

@charris

This comment has been minimized.

Copy link
Member

commented Aug 14, 2019

@tacaswell

This comment has been minimized.

Copy link
Contributor Author

commented Aug 14, 2019

I think the key idea in this NEP is that we all adopt clear, calendar based, policies for dropping support for outdated versions of Python.

The problem with any policy based on number of Python releases is that it makes it extremely hard to plan. We are looking 3-4 years out and in that time things can happen (like the BDFL stepping down, the release cadence changing, etc.) that are impossible to predict.

Even 6 months out, it is hard to predict if upstream is going to run into something critical in the RC process and delay the final release by a few months. If you are targeting "the last 3 Pythons" and 6 months out start using new features and then CPython's release gets delayed past your expected release date do you hold your release until CPython releases or do you temporarily break your rules?

On the flip side, if you promised X months worth of feature updates to a user when they installed a given version of Python and then Python minor versions come out faster than you expected. If you stick to the "last N" rule you will be cutting off feature support to that user sooner than you initially promised. If we end up supporting too many versions of Python then we should shorten the window going forward.

On the other hand, setting a minimum number of Python to support is ok because worst case is we over-deliver on the initial promise to the users. To invert the paragraph that @seberg suggested above:

"This is a guideline and in exceptional cases projects with a large dependency base may choose to have longer support. At this time NumPy will have at least a XX month long support window and support at least the last N Python minor versions."

No matter what happens you will meet or exceed what we promised the users.


I think we are all in rough conceptually agreement, but disagree about the size of the time window.

The minimum viable support window is 2x the expected CPython release cadence (to make sure we always support at least 2 pythons for the reasons @Carreau noted above) and the maximum viable support window is the bug fix period of CPython (5 years, we should not support a version of Python once upstream no longer supports it). At a 18 month cadence the range is [36, 60], at 12mo cadence the range is [24, 60] and at 24mo the range is [48, 60].

The currently proposed window (42 months) is about in the middle for both the 18 and 12 month cadences. I think strikes a good balance between letting the projects move to the future and make use of new language features while providing a stable base for and to give people who have to install python at-scale a reasonable timeline to manage their upgrades.

It also gives alternative implementations of python ample window to catch up with CPython, if they have fallen 2 minor versions (and 3.5 years) behind of CPython it is not clear to me that they are ever going to get support for newer versions of Python.

Rasbian Buster (aka debian 10) was released in the middle of July and comes with py37, I think that takes concerns about rasbian off the table?


The intersection that this policy is problematic for is "needs to be on LTS" and "needs new versions of scipy stack" and "insists on using system python" which makes the pool even smaller and is easily addressed by user-space Python.

tacaswell added some commits Aug 14, 2019

@rgommers

This comment has been minimized.

Copy link
Member

commented Aug 14, 2019

nice summary, thanks @tacaswell.

It also gives alternative implementations of python ample window to catch up with CPython, if they have fallen 2 minor versions (and 3.5 years) behind of CPython it is not clear to me that they are ever going to get support for newer versions of Python.

perhaps, PyPy seems to be moving forward continuously, just a lot slower than everyone would like.

@mattip could you give your opinion on the current proposal and 42 month window on behalf of the PyPy team, or ping someone else to do so?

@charris

This comment has been minimized.

Copy link
Member

commented Aug 14, 2019

I think the key idea in this NEP is that we all adopt clear, calendar based, policies for dropping support for outdated versions of Python.

My main objections are that it is calender based and that the period is a bit short for NumPy. I prefer released based myself and the defacto three Python releases standard that NumPy currently follows, ignoring the Python 2.7 anomaly, seems about right to me. If Python went to a longer release cycle, I'd happily go to two releases and if they go to a one year release cycle I stick with three releases. In calender terms, about 50 months might work.

@mattip

This comment has been minimized.

Copy link
Member

commented Aug 14, 2019

PyPy is now trying to finish 3.6 and 2.7 compatibility, there are some vague plans to start a 3.7 version sometime later this year which would be after the release of CPython 3.8. PyPy is unlikely to begin work on a new version before its final release, so they would always be implementing at least one minor version behind. Three versions should be fine, two might be tight.

@tacaswell

This comment has been minimized.

Copy link
Contributor Author

commented Aug 14, 2019

@charris In my day-job my group manages deploying Python environments for running data collection and analysis across something like ~150 computers including the standard stack, in-house software, and domain specific stuff from the xray community [1]. In that context, or in the context of a cluster manager who is going to provide a python module (in the HPC sense, not the python sense), having a fixed date when any given Python is going to no longer be supported by the newest version of the stack is super useful for planning.

A "N versions of python" policy requires the reader to know the Python release plans, the proposed time-window based approach directly tells you when support of a given release version of Python will sun set.

[1] For reference, we just rolled the experimental floor to py37 from py36 and have had no issues (due to changing the python version).

DOC: clarify that these are minimum support guide lines
Projects that want to support longer time windows can, but this NEP is
setting a minimum floor for the support window.
@tacaswell

This comment has been minimized.

Copy link
Contributor Author

commented Aug 14, 2019

On the weekly call we agreed to add language that indicates that these are minimum support windows (42mo / 2 minor versions for python, 24mo / 3 minor versions for numpy). This gives a minimum support window for down-stream projects and institutions to plan against, but lets projects that want to have longer support windows to still use the language but just take their time about actually dropping support.

This NEP is also about signaling to down-down-down stream domain specific packages what sort of support windows they should be having for the grad-student + post-doc projects used by a handful of labs.

NumPy its self will likely go with (42-50mo / 3 python minors minimums), but that is a bit out of scope for this NEP which is meant to be guidance to down stream projects, not policy for NumPy.


On the call we did discuss dropping the minimum Python window to 36 months, but I think the logic for the buffer holds and 3.5 years feels "right" to me (just under an under-grad degree, just over a post-doc, and about half of an (American) PhD). However, it would be very easy to persuade me that it should be lower.

@hmaarrfk hmaarrfk referenced this pull request Aug 17, 2019
@mattip

This comment has been minimized.

Copy link
Member

commented Aug 18, 2019

LGTM, I think we can merge this. It has "draft" status, pending further discussion, merging it will make it more discoverable.

@hmaarrfk hmaarrfk referenced this pull request Aug 18, 2019
0 of 9 tasks complete
@rgommers

This comment has been minimized.

Copy link
Member

commented Aug 19, 2019

LGTM, I think we can merge this. It has "draft" status, pending further discussion, merging it will make it more discoverable.

agreed, merging.

Let's continue the discussion though. After the feedback/concerns (from e.g. @charris, @rth, @GaelVaroquaux) about the 3.5 years being too fast for some packages, the text is changed to a minimum support window. Does that address everyone's concerns?

@rgommers rgommers merged commit e2b5f60 into numpy:master Aug 19, 2019

13 of 16 checks passed

LGTM analysis: C/C++ No code changes detected
Details
LGTM analysis: JavaScript No code changes detected
Details
LGTM analysis: Python No code changes detected
Details
Shippable Run 5596 status is SUCCESS.
Details
azure-pipeline numpy.numpy Build #20190814.17 succeeded
Details
azure-pipeline numpy.numpy (Linux_PyPy3) Linux_PyPy3 succeeded
Details
azure-pipeline numpy.numpy (Linux_Python_36_32bit_full_with_asserts) Linux_Python_36_32bit_full_with_asserts succeeded
Details
azure-pipeline numpy.numpy (Windows Python35-64bit-full) Windows Python35-64bit-full succeeded
Details
azure-pipeline numpy.numpy (Windows Python36-32bit-fast) Windows Python36-32bit-fast succeeded
Details
azure-pipeline numpy.numpy (Windows Python36-64bit-full) Windows Python36-64bit-full succeeded
Details
azure-pipeline numpy.numpy (Windows Python37-32bit-fast) Windows Python37-32bit-fast succeeded
Details
azure-pipeline numpy.numpy (Windows Python37-64bit-full) Windows Python37-64bit-full succeeded
Details
azure-pipeline numpy.numpy (macOS) macOS succeeded
Details
ci/circleci Your tests passed on CircleCI!
Details
codecov/project 82.67% (target 1%)
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@tacaswell tacaswell deleted the tacaswell:cpython_support_nep branch Aug 22, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.