Propose replacing github label priority/failing-test with kind/failing-test

[spiffxp]

As CI Signal Lead for 1.9 I often found myself filing issues with priority/critical-urgent priority/failing-test. Having two priorities assigned to an issue seems... weird. I think the idea was to encourage folks to prioritize failing tests as the most important thing. I'm not sure how well that has worked in practice.

It would make more sense to me if we called it kind/failing-test, to separate from kind/bug or kind/flake. I could then say that kubernetes/test-infra#6095 holds true, that an issue should only ever have one priority label applied to it.

I went digging through devstats and maybe it hurts my case. It does look like priority/failing-test issues get closed a little more quickly than priority/critical-urgent issues. That said, I would argue it comes from effectively triaging failing tests into a bucket, not so much that the bucket has the word "priority" in it.

• issues age|repo:kubernetes priority/failing-test
• issues age|repo:kubernetes priority/critical-urgent
• issues age|repo:kubernetes
• issues age|repo:kubernetes kind/bug priority/failing-test
• issues age|repo:kubernetes kind/bug priority/crticial-urgent
• issues age|repo:kubernetes kind/bug

ref: trying to make github issue labels more sane kubernetes/kubernetes#57911

WDYT? @kubernetes/sig-contributor-experience-misc-use-only-as-a-last-resort

[tpepper]

While one might argue simply that all tests must pass and any failure is gating, failing test and criticality are orthogonal to me here especially in light of your discussion in the SIG-testing session at KubeCon: if there are highly fail-likely scenarios that go unaddressed for a long time, those will dilute or confuse critical prioritization. The proposal makes sense.

Do you see kind/flake as probably a second level triage assessment? For example a kind/failing-test is probably the first determination, and then with some more understanding of variability of the test's outcome it gets further classified as a flake? I suppose there's utility in having both.

[spiffxp]

Personally I would probably use kind/failing-test when the test has failed N runs in a row, vs. kind/flake when the test is flipping back and forth between passing/failing. I had some sleep human-induced hysteresis when it came to opening issues against test failures.

[cblecker]

Opened kubernetes/test-infra#6642 to actually do this.

[jberkus]

@cblecker you opened a PR for this, then closed it. How were you planning to proceed from here? I'd like to get this settled.

[spiffxp]

Note to self: none of the devstats links work anymore, I think the dashboard got renamed

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[jberkus]

/remove-lifecycle stale

[spiffxp]

/assign
I'm interested in reducing CI Signal toil, I'll aim to have a rollout proposal out next week

[jberkus]

Ah, I was going to write one up today, but am happy to wait for you.

[spiffxp]

Opening PR's to change docs per https://github.com/search?q=org%3Akubernetes+priority%2Ffailing-test&type=Code

[spiffxp]

kubernetes-dev thread asking for comment: https://groups.google.com/d/msg/kubernetes-dev/W_LfpE1iGlE/o8-M_J5xBgAJ

Will roll out sometime after 3pm PT Wed 7/25 if I hear no objections

[bgrant0607]

The reason I proposed priority/failing-test originally was because the auto-filing bot was DoSing other priority labels, and the issues were piling up without action -- clearly they weren't being treated as urgent, and many weren't actually urgent.

If the issues are being filed by humans and are actually treated as urgent, it's fine to use other priorities.

[spiffxp]

I've removed holds from the linked PR's. Will e-mail out once I see the label_sync has happened

[spiffxp]

Migration complete:

• org:kubernetes label:priority/failing-test - 0 results
• org:kubernetes label:kind/failing-test - 25 open

/close