diff --git a/pep-0598.rst b/pep-0598.rst new file mode 100644 index 00000000000..fd8da4370a8 --- /dev/null +++ b/pep-0598.rst @@ -0,0 +1,527 @@ +PEP: 598 +Title: Introducing minor feature releases +Version: $Revision$ +Last-Modified: $Date$ +Author: Nick Coghlan +Discussions-To: https://discuss.python.org/t/pep-596-python-3-9-release-schedule-doubling-the-release-cadence/1828 +Status: Draft +Type: Informational +Content-Type: text/x-rst +Created: 15-Jun-2019 +Python-Version: 3.9 + + +Abstract +======== + +PEP 596 has proposed reducing the feature delivery latency for the Python +standard library and CPython reference interpreter by increasing the frequency +of major CPython releases from every 18-24 months to instead occur every 9-12 +months. + +Adopting such an approach has several significant practical downsides, as a +CPython major release carries certain expectations (most notably, a 5 year +maintenance lifecycle, support for parallel installation with the previous +major release, and the possibility of breaking chnages to the CPython-specific +ABI, requiring recompilation of all extension modules) that mean faster major +releases would significantly increase the burden of maintaining 3rd party +Python libraries and applications across all actively supported CPython +releases. + +This PEP presents a competing proposal to instead *slow down* the frequency of +parallel installable major feature releases that change the filesystem layout +and CPython ABI to a consistent 24 month cycle, but to compensate for this by +introducing the notion of build compatible minor feature releases, and then +deferring the full feature freeze of a given major release series from the +initial X.Y.0 release to an X.Y.Z minor feature release that occurs ~12 months +after that initial release. + +The existing cycle and the new cycle would be synchronised on their feature +freeze release dates, so the full Python 3.9.x feature freeze would occur in +October 2021, 24 months after the Python 3.8.0 feature release, with the initial +Python 3.9.0 release taking place in October 2020. + + +Example Future Release Schedules +================================ + +Under this proposal, Python 3.9.0a1 would be released in November 2019, shortly +after the Python 3.8.0 feature freeze release in October 2019. + +The 3.9.0b1 release would then follow 6 months later in May 2020, with 3.9.0 +itself being released in October 2020. + +Assuming minor releases of 3.9.x were to occur quarterly, then the overall +release timeline would look like: + +* 2019-11: 3.9.0a1 +* ... additional alpha releases as determined by the release manager +* 2020-05: 3.9.0b1 +* ... additional beta releases as determined by the release manager +* 2020-08: 3.9.0bX (final beta release that locks ABI compatibility) +* 2020-09: 3.9.0rc1 +* ... additional release candidates as determined by the release manager +* 2020-10: 3.9.0 +* 2021-01: 3.9.1 +* 2021-04: 3.9.2 +* 2021-07: 3.9.3 +* 2021-10: 3.9.4 (final minor feature release) +* 2022-01: 3.9.5 +* 2022-04: 3.9.6 +* 2022-07: 3.9.7 +* 2022-10: 3.9.8 (final regular maintenance release) +* ... additional security fix only releases as needed +* 2025-10: 3.9.x branch closed + +The Python 3.10 release series would start being published the month after the +final Python 3.9 minor feature release, in parallel with the final 12 months +of routine Python 3.9 maintenance releases: + +* 2021-11: 3.10.0a1 +* ... additional alpha releases as determined by the release manager +* 2022-05: 3.10.0b1 +* ... additional beta releases as determined by the release manager +* 2022-08: 3.10.0bX (final beta release that locks ABI compatibility) +* 2022-09: 3.10.0rc1 +* ... additional release candidates as determined by the release manager +* 2022-10: 3.10.0 +* 2023-01: 3.10.1 +* 2023-04: 3.10.2 +* 2023-07: 3.10.3 +* 2023-10: 3.10.4 (final minor feature release) +* 2024-01: 3.10.5 +* 2024-04: 3.10.6 +* 2024-07: 3.10.7 +* 2024-10: 3.10.8 (final regular maintenance release) +* ... additional security fix only releases as needed +* 2027-10: 3.10.x branch closed + +In this model, there are always two or three active branches: + +* 2019-04 -> 2019-10: 3.9.0 pre-alpha, 3.8.0 pre-release, 3.7.x maintenance +* 2019-10 -> 2020-05: 3.9.0 pre-beta, 3.8.x maintenance +* 2020-05 -> 2020-10: 3.10.0 pre-alpha, 3.9.0 pre-release, 3.8.x maintenance +* 2020-10 -> 2021-10: 3.10.0 pre-alpha, 3.9.x feature addition, 3.8.x maintenance +* 2021-10 -> 2022-05: 3.10.0 pre-beta, 3.9.x maintenance +* 2022-05 -> 2022-10: 3.11.0 pre-alpha, 3.10.0 pre-release, 3.9.x maintenance +* 2022-10 -> 2023-10: 3.11.0 pre-alpha, 3.10.x feature addition, 3.9.x maintenance +* 2023-10 -> 2024-05: 3.11.0 pre-beta, 3.10.x maintenance +* 2024-05 -> 2024-10: 3.12.0 pre-alpha, 3.11.0 pre-release, 3.10.x maintenance +* ... etc + +(Pre-alpha and pre-beta development occurs on the main git branch, all other +development occurs on a release specific branch with changes typically +backported from the main git branch) + +TODO: this really needs a diagram to help explain it, so I'll add a picture +once I have one to add. + +This is quite similar to the status quo, but with a more consistent cadence, +alternating between major feature release years (2020, 2022, etc) that focus +on the alpha and beta cycle for a new major feature release (while continuing +to publish maintenance releases for the previous major release series), and +minor feature release years (2021, 2023, etc), that focus on making smaller +improvements to the current major release series (while making plans for the +next major release the following year). + + +Proposal +======== + +Excluding alpha and beta releases, CPython currently has 3 different kinds +of release increment: + +* Feature release (i.e. ``X.Y.0`` releases) +* Maintenance release (``X.Y.Z`` releases within ~2 years of ``X.Y.0``) +* Source-only security release (subsequent ``X.Y.Z`` releases) + +Feature freeze takes place at the time of the ``X.Y.0b1`` release. +Build compatibility freeze now takes place at the time of the last beta release +(providing time for projects to upload wheel archives to PyPI prior to the +first release candidate). + +This then creates the following periods in the lifecycle of a release series: + +* Pre-beta (release series is the CPython development branch) +* Beta (release enters maintenance mode, ABI compatibility mostly locked) +* Maintenance (ABI locked, only bug fixes & docs enhancements accepted) +* Security fix only (no further binary releases, only security fixes accepted) +* End of life (no further releases of any kind) + +The proposal in this PEP is that the "Feature release" category be split up into +two different kinds of feature release: + +* Major feature release (``X.Y.0`` releases) +* Minor feature release (``X.Y.Z`` releases within ~1 year of ``X.Y.0``) +* Maintenance release (``X.Y.Z`` releases within ~1-2 years of of ``X.Y.0``) +* Source-only security release (subsequent ``X.Y.Z`` releases) + +This would then introduce a new "Feature additions" phase in the release series +lifecycle: + +* Pre-beta (release series is the CPython development branch) +* Beta (release enters feature additions mode, ABI compatibility not yet locked) +* Feature additions (ABI locked, API additions accepted) +* Maintenance only (ABI locked, only bug fixes & docs enhancements accepted) +* Security fix only (no further binary releases, only security fixes accepted) +* End of life (no further releases of any kind) + +The pre-release beta period would be relaxed to use the minor feature release +policy for changes, rather than the stricter maintenance-only release policy. + +The existing "Maintenance" release series state would be renamed to +"Maintenance only", as "Feature addition" releases are also maintenance +releases (just as maintenance releases can also be security fix releases). + +For governance purposes, major feature releases are the only releases that +would qualify as a "feature release" in the PEP 13 sense (minor feature releases +wouldn't count). + + +Major feature releases +---------------------- + +Major feature releases are just the existing feature releases, given a new name +to help distinguish them from the new minor feature releases. + +Major feature releases would continue to define the following language, build, +and installation compatibility constraints: + +- Python language grammar +- ``ast`` module AST format +- CPython interpreter opcode format +- ``pyc`` file magic number and filename compatibility tags +- extension module filename compatibility tags +- wheel archive compatibility tags +- default package and module import directories +- default installation filename and directories + +Major feature releases would also continue to be the only releases where: + +- new deprecations, pending deprecations, and other warnings can be introduced +- existing pending deprecations can be converted to full deprecations +- existing warnings can be converted to errors +- other changes requiring "Porting to Python X.Y" entries in the What's New + document can be introduced + +Key characteristics of major feature releases: + +- a major feature release installation does not conflict with installations of + other major feature releases (i.e. they can be installed in parallel) +- a major feature release installation can be updated to a later minor release + without requiring reinstallation or any other changes to previously installed + components +- major feature releases may contain higher risk changes to the language and + interpreter, such as grammar modifications, major refactoring of interpreter + and standard library internals, or potentially invasive feature additions with + a risk of unintended side effects on other existing functionality + +Key expectations around major feature releases and release series: + +- most public projects will only actively test against the *most recent* + minor release within a release series +- many (most?) public projects will only add a new release series to their test + matrix *after* the initial feature release has already been published, which + can make it difficult to resolve issues that require providing new flags or + APIs to explicitly opt-in to old behaviour after a default behaviour changed +- private projects with known target environments will test against whichever + minor release version they're actually using +- most private projects will also only consider migrating to a new release + series *after* the initial feature release has already been published, again + posing a problem if the resolution of their problems requires an API addition + + +The key motivation of the proposal in this PEP is that the public and private +project behaviours described above aren't *new* expectations: they're +descriptions of the way CPython release series are already handled by the wider +community today. As such, the PEP represents an attempt to adjust our release +policies and processes to better match the way the wider community already +handles them, rather than changing our processes in a way that then means the +wider community needs to adjust to us rather than the other way around. + + +Minor feature releases +---------------------- + +Minor feature releases are the key new process addition being proposed by this +PEP. They are subject to the same strict runtime compatibility requirements +as the existing maintenance releases, but would have the following more +relaxed policies around API additions and enhancements: + +* new public APIs can be added to any standard library module (including builtins) +* new optional arguments can be added to existing APIs (including builtins) +* new public APIs can be added to the stable C ABI (with appropriate version guards) +* new public APIs can be added to the CPython C API +* with the approval of the release manager, backwards compatible semantic + improvements can be made to existing APIs and syntactic constructs +* with the approval of the release manager, performance improvements can be + incorporated for existing APIs and syntactic constructs + +The intent of this change in policy is to allow usability improvements for new +(and existing!) language features to be delivered in a more timely fashion, +rather than requiring users to incur the inherent delay and costs of waiting for +and then upgrading to the next major release series. + +Key expectations around minor feature releases: + +- "don't break existing installations on upgrade" remains a key requirement + for minor releases, even with the more permissive change inclusion policy +- more intrusive changes should still be deferred to the next major feature + release +- public Python projects that start relying on features added in a minor feature + release should set their ``Python-Requires`` metadata appropriately (projects + already do this when necessary - e.g. ``aiohttp`` specifically requires + 3.5.3 or later due to an issue with ``asyncio.get_event_loop()`` in earlier + versions) + + +Motivation +========== + +The motivation for change in this PEP is essentially the same as the motivation +for change in PEP 596: the current 18-24 month gap between feature releases has +a lot of undesirable consequences, especially for the standard library (see +PEP 596 for further articulation of the details). + +The concern with the specific proposal in PEP 596 is that it doubles the number +of actively supported Python branches, increasing the complexity of +compatibility testing matrices for the entire Python community, increasing the +number of binary Python wheels to be uploaded to PyPI when not using the stable +ABI, and just generally having a high chance of inflicting a relatively high +level of additional cost across the entire Python ecosystem. + +The view taken in this PEP is that there's an alternative approach that provides +most of the benefits of a faster major release without actually incurring the +associated costs: we can split the current X.Y.0 "feature freeze" into two parts, +such that X.Y.0 only imposes a "runtime compatibility freeze", and the full +feature freeze is deferred until later in the release series lifecycle. + + +Caveats and Limitations +======================= + +This proposal does NOT retroactively apply to Python 3.8 - it is being proposed +for Python 3.9 and later releases only. + +Actual release dates may be adjusted up to a month earlier or later at +the discretion of the release manager, based on release team availability, and +the timing of other events (e.g. PyCon US, or the annual core development +sprints). However, part of the goal of this proposal is to provide a consistent +annual cadence for both contributors and end users, so adjustments ideally would +be rare. + +This PEP does not dictate a specific cadence for minor releases within a release +series - it just specifies the timelines for transitions between the release +series lifecycle phases (pre-alpha, alpha, beta, feature additions, +bug fixes, security fixes). The number of minor releases within each phase is +determined by the release manager for that series based on how frequently they +and the rest of the release team for that series are prepared to undertake the +associated work. + +However, for the sake of the example timelines, the PEP assumes quarterly +minor releases (the cadence used for Python 3.6 and 3.7, splitting the +difference between the twice yearly cadence used for some historical release +series, and the monthly cadence expected for Python 3.8 and 3.9). + + +Design Discussion +================= + +Why this proposal over simply doing more frequent major releases? +----------------------------------------------------------------- + +The filesystem layout changes and other inherently incompatible changes involved +in a major version update create additional work for large sections of the +wider Python community. + +Decoupling those layout changes from the Python version numbering scheme is also +something that in and of itself involves making backwards incompatible changes. + +We also don't have a straightforward means to communicate to the community "Only +support major version X.Y until X.Y+1 is out, but support X.Z until X.Z+2 is +out". + +So this PEP takes as its starting assumption that the vast majority of Python +users simply *shouldn't need to care* that we're changing our release policy, +and the only folks that will be affected will be those that are eagerly waiting +for standard library improvements, and other backwards compatible interpreter +enhancements. + + +Duration of the feature additions period +---------------------------------------- + +This PEP proposes that feature additions be limited to 12 months after the +initial major feature release. + +The primary motivation for that is specifically to sync up with the Ubuntu LTS +timing, such that the final minor feature release for Python 3.9.x gets +published in October 2021, ready for inclusion in the Ubuntu 22.04 release. +(other LTS Linux distributions like RHEL, SLES, and Debian don't have a fixed +publishing cadence, so they can more easily tweak their LTS timing a bit to +align with stable versions of their inputs. Canonical deliberately haven't +given themselves that flexibility with their own release cycle). + +The 12 month feature addition period then arises from splitting the time +from the 2019-10 release of Python 3.8.0 and a final Python 3.9.x minor feature +release in 2021-10 evenly between pre-release development and subsequent +minor feature releases. + +This is an area where this PEP could adopt part of the proposal in PEP 596, +by instead making that split ~9 months of pre-release development, and ~15 +months of minor feature releases: + +* 2019-11: 3.9.0a1 +* ... additional alpha releases as determined by the release manager +* 2020-03: 3.9.0b1 +* 2020-04: 3.9.0b2 +* 2020-05: 3.9.0b3 (final beta release that locks ABI compatibility) +* 2020-06: 3.9.0rc1 +* ... additional release candidates as determined by the release manager +* 2020-07: 3.9.0 +* 2020-10: 3.9.1 +* 2021-01: 3.9.2 +* 2021-04: 3.9.3 +* 2021-07: 3.9.4 +* 2021-10: 3.9.5 (final minor feature release) +* 2022-01: 3.9.6 +* 2022-04: 3.9.7 +* 2022-07: 3.9.8 +* 2022-10: 3.9.9 (final regular maintenance release) +* ... additional security fix only releases as needed +* 2025-10: 3.9.x branch closed + +This approach would mean there were still always two or three active branches, +it's just that proportionally more time would be spent with a branch in the +"feature additions" phase, as compared to the "pre-alpha", "pre-beta", and +"pre-release" phases: + +* 2019-04 -> 2019-10: 3.9.0 pre-alpha, 3.8.0 pre-release, 3.7.x maintenance +* 2019-10 -> 2020-03: 3.9.0 pre-beta, 3.8.x maintenance +* 2020-03 -> 2020-07: 3.10.0 pre-alpha, 3.9.0 pre-release, 3.8.x maintenance +* 2020-07 -> 2021-10: 3.10.0 pre-alpha, 3.9.x feature addition, 3.8.x maintenance +* 2021-10 -> 2022-03: 3.10.0 pre-beta, 3.9.x maintenance +* 2022-03 -> 2022-07: 3.11.0 pre-alpha, 3.10.0 pre-release, 3.9.x maintenance +* 2022-07 -> 2023-10: 3.11.0 pre-alpha, 3.10.x feature addition, 3.9.x maintenance +* 2023-10 -> 2024-03: 3.11.0 pre-beta, 3.10.x maintenance +* 2024-03 -> 2024-07: 3.12.0 pre-alpha, 3.11.0 pre-release, 3.10.x maintenance +* ... etc + + +Duration of the unreleased pre-alpha period +------------------------------------------- + +In the baseline proposal in this PEP, the proposed timelines still include +periods where we go for 18 months without making a release from the main git +branch (e.g. 3.9.0b1 would branch off in 2020-05, and 3.10.0a1 wouldn't be +published until 2021-11). They just allow for a wider variety of changes to +be backported to the most recent maintenance branch for 12 of those months. + +The variant of the proposal that moves the beta branch point earlier in the +release series lifecycle would increase that period of no direct releases to +21 months - the only period where releases were made directly from the main +branch would be during the relatively short window between the last minor +feature release of the previous release series, and the beta branch point a +few months later. + +While alternating the annual cadence between "big foundational enhancements" +and "targeted low risk API usability improvements" is a deliberate feature of +this proposal, it still seems strange to wait that long for feedback in the +event that changes *are* made shortly after the previous major version is +branched. + +An alternative way of handling this would be to start publishing alpha releases +for the next major feature release during the feature addition period (similar +to the way that PEP 596 proposes to starting publishing Python 3.9.0 alpha +releases during the Python 3.8.0 release candidate period). + +However, rather than setting specific timelines for that at a policy level, +it may make sense to leave that decision to invididual release managers, based +on the specific changes that are being proposed for the release they're +managing. + + +Why not switch directly to full semantic versioning? +---------------------------------------------------- + +If this were a versioning design document for a new language, it *would* use +semantic versioning: the policies described above for major feature releases +would be applied to X.0.0 releases, the policies for minor feature releases +would be applied to X.Y.0 releases, and the policies for maintenance only +releases would be applied to X.Y.Z releases. + +The problem for Python specifically is that all the policies and properties for +parallel installation support and ABI compatibility definitions are currently +associated with the first *two* fields of the version number, and it has been +that way for the better part of thirty years. + +As a result, it makes sense to split out the policy question of introducing +minor feature releases in the first place from the technical question of making +the version numbering scheme better match the semantics of the different release +types. + +If the proposal in this PEP were to be accepted by the Steering Council for +Python 3.9, then a better time to tackle that technical question would be for +the subsequent October 2022 major feature release, as there are already inherent +compatibility risks associated with the choice of either "Python 4.0" (erroneous +checks for the major version being exactly 3 rather than 3 or greater), or +"Python 3.10" (code incorrectly assuming that the minor version will always +contain exactly one decimal digit). + +There are complex pros and cons on both sides of that future choice, and this +PEP would merely add one more potential pro in favour of choosing the +"Python 4.0" option, with the caveat that we would also need to amend the +affected installation layout and compatibility markers to only consider the +major version number (rather than both the major and minor version). + +If such a change were to be proposed and accepted, then the example 3.10.x +timeline given above would instead become the following 4.x series timeline: + +* 2021-11: 4.0.0a1 +* ... additional alpha releases as determined by the release manager +* 2022-05: 4.0.0b1 +* ... additional beta releases as determined by the release manager +* 2022-08: 4.0.0bX (final beta release that locks ABI compatibility) +* 2022-09: 4.0.0rc1 +* ... additional release candidates as determined by the release manager +* 2022-10: 4.0.0 +* 2023-01: 4.1.0 +* 2023-04: 4.2.0 +* 2023-07: 4.3.0 +* 2023-10: 4.4.0 (final minor feature release) +* 2024-01: 4.4.1 +* 2024-04: 4.4.2 +* 2024-07: 4.4.3 +* 2024-10: 4.4.4 (final regular maintenance release) +* ... additional security fix only releases as needed +* 2027-10: 4.x branch closed + +And the 5 year schedule forecast would look like: + +* 2019-04 -> 2019-10: 3.9.0 pre-alpha, 3.8.0 pre-release, 3.7.x maintenance +* 2019-10 -> 2020-05: 3.9.0 pre-beta, 3.8.x maintenance +* 2020-05 -> 2020-10: 4.0.0 pre-alpha, 3.9.0 pre-release, 3.8.x maintenance +* 2020-10 -> 2021-10: 4.0.0 pre-alpha, 3.9.x feature addition, 3.8.x maintenance +* 2021-10 -> 2022-05: 4.0.0 pre-beta, 3.9.x maintenance +* 2022-05 -> 2022-10: 5.0.0 pre-alpha, 4.0.0 pre-release, 3.9.x maintenance +* 2022-10 -> 2023-10: 5.0.0 pre-alpha, 4.x.0 feature addition, 3.9.x maintenance +* 2023-10 -> 2024-05: 5.0.0 pre-beta, 4.x.y maintenance +* 2024-05 -> 2024-10: 6.0.0 pre-alpha, 5.0.0 pre-release, 4.x.y maintenance +* ... etc + + + +Copyright +========= + +This document has been placed in the public domain. + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 80 + coding: utf-8 + End: