Use priority labels

[mossheim]

Hey all,

I suggest we start using priority labels and milestones to organize issues and PRs. I've heard multiple people involved with the project express a desire for more and clearer prioritization, and I think these would help a lot. We previously used milestones around the time of the 3.10.z releases, and that seemed to be generally useful. To my knowledge we haven't used priority labels yet though. (This builds off some of the conversation in #4940)

For milestones, we used to do this:

• 'next patch' (3.12.1)
• 'next minor' (3.12.0)
• 'future patch' (3.12.x)
• 'future'

But the priority labels should mostly cover the need for a 'future' milestone, so I think just the first 3 are needed. The milestones would be named with the actual version numbers, and we would update them on every release.

For priority labels, I'm proposing a set of 4 or 5 levels (tentatively P1-P5) that give a general sense of how to prioritize among open issues and PRs -- in other words, what developers should be working on next. This could be a combination of severity, number of people affected, whether a workaround exists, and/or developers' desire to work on it. These labels would be for use by anyone who can add labels. Almost every open issue and PR (with the possible exception of organizational/question issues like this one) would have a priority label. This would take a while at the start, but with the work shared among several people it wouldn't take more than a few weeks, I think.

One obvious question is how to decide what the different levels mean, or where the boundaries between them are. The boundaries are always going to be subjective and depend on the person, but I would bet there is some shared ground. I would suggest we find it by having multiple people rank the same set of issues, and discuss the results. We should be able to come to at least a few heuristics which we can then document. This would be a good opportunity to understand each others' perspectives. :)

Of course, people are free to work on whatever they want. These labels would only be there for people who want to understand the overall priorities of the project and how they can help with that.

I was originally thinking we would only need to have priority labels for issues, but @jamshark70 raised a good point that not every PR is tied to an issue (and seeing prioritization among PRs is also convenient). So I would tentatively say that all PRs would get a priority label, even the ones with associated issues. Even though it's a tiny bit redundant, that seems simpler and easier to remember process-wise.

[mossheim]

update: i just created a 3.12.0 milestone and added everything from #5247 to it.

[mossheim]

the milestones seem to be working ok, so i'm going to consider that "implemented" and change this to just "Use priority labels".