Add clarifying policy on release blockers/milestones

[asmeurer]

Right now it isn't really clear what should go in a release milestone. So a lot ends up there that probably shouldn't be, and it ends up delaying the release. So we should work out a clear policy of what should and shouldn't be a blocker for a release. Once we finalize it, let's add it to the wiki.

Here's a start, but this is by no means final, so please discuss it.

I'm using RFC 2119 terminology.

Things that MUST block a release. These things absolutely must be fixed or worked around before a release can happen:

• Test failures on master
• Anything that prevents the release script from working

Things that SHOULD block a release. These things should ideally be fixed, but there can be exceptions, e.g., if no one is able to fix it on time for the release. In extreme cases, a change can be reverted in the release branch so as to minimize regressions or API breaks in released versions.

• Breaking changes to APIs that were introduced since the previous release (so as to minimize future deprecations)
• Major performance regressions.
• Major behavioral regressions. Wrong result regressions are particularly of high priority.
• New deprecations (see https://github.com/sympy/sympy/wiki/Deprecating-policy).

Things that SHOULD NOT block a release. These can always make it into the next release. Ideally releases will happen often enough that this isn't a major concern:

• New features
• Non-critical bug fixes

Once a release branch is created, only release blockers should be merged into it, ideally only by the release manager (this can be enforced by branch protection if the release manager so choses). All other changes should continue to be merged into master. That way the release branch has minimal churn.

The ultimate decision of what constitutes a release blocker is up to the release manager for a given release.

Whether something in the SHOULD group that cannot be fixed should be postponed or just be removed from the milestone completely will be decided on a case-by-case basis. The longer an issue has been present (in terms of number of released versions), the less it should be considered as a blocker:

I also think we need some sort of policy that says that if someone adds an issue to a milestone, they must provide a motivation as to why it should be a release blocker (unless it's obvious, e.g., with #17884).

[oscarbenjamin]

I also think we need some sort of policy that says that if someone adds an issue to a milestone, they must provide a motivation as to why it should be a release blocker

This is probably the most important point.

Sometimes the milestone is added because someone wants to remind themself to complete some work before the release which I think is fine but that isn't a release blocker. If they make it clear when adding the milestone that that is all it means then it is easy for the release manager to remove the milestone.

The other side is that someone adding a milestone needs to do a bit of work to ensure that it is a regression etc and requiring a clear statement about why it is a release blocker should help with that.

[asmeurer]

Maybe we could have some way of enforcing this with the bot.