Automatically open issues for failing tests on `main`

[gjoseph92]

With recent work towards cleaning up CI, we've gotten a lot closer to all-green builds. After a couple more tests are fixed, hopefully the short test report will be empty.

In order to keep it empty, I'd like to add automation to help enforce a cultural practice of not ignoring flaky tests, and catching them quickly. I'd be happy to discuss other ways of doing this, but here's one simple thing that comes to mind.

Can we add a GitHub action that:

• opens an issue every time a new test fails on main
• comments on the existing issue when it fails again with a link to the new failure
• links to the first PR after which it failed
• tags the author of that PR, and core devs

Obviously, sometimes the author-tagging will be wrong (especially at the beginning, when we still have plenty of pre-existing flaky tests that may pop up randomly). Maybe we don't actually want it at first. But to me, it's initially more about a way of signaling a new attitude towards flaky tests, and increasing visibility. (Statistically, most PR authors are probably also core devs.)

Also, I opened this here, but it could probably happen on dask/dask too. @jrbourbeau want to keep this issue here, or move it to dask/community?

cc @hendrikmakait @martindurant @quasiben @fjetter @ian-r-rose @jsignell @crusaderky

[fjetter]

I like the proposal but would suggest to start with something very simple. To tag and link all the PRs as you suggest we'd need to either maintain a database or would require awkward parsing of github issues.

How about, as a first iteration, we start by opening a ticket automatically that includes the current failing commit? This merge commit includes all necessary information to get the offending PR.

tags the author of that PR, and core devs

I don't like the idea of tagging that many people. I suggest to start without at first and see how it goes.

[jrbourbeau]

+1 for automating the flaky test issue opening process, thanks for bringing it up @gjoseph92

I agree with @fjetter that starting with something simple seems reasonable.

comments on the existing issue when it fails again with a link to the new failure

Everyone who watches distributed will get a notification for this. I realize the point here is to increase visibility into failures, but there's also a balance here with getting too many notifications. I don't have a good sense for how often folks would get a ping for this. Maybe it's okay if it's only a couple of notifications a day, but I wouldn't want to send out 10s of notifications for this every day.

[gjoseph92]

awkward parsing of github issues

Squash commits from GitHub automatically include the PR number at the end of the subject line unless you manually delete it, which nobody does. I'd be okay relying on this as a simple first iteration:

PR=$(git log --format=%s -n 1 | sed -r "s:^.*\(#([0-9]+)\)$:\1:")
AUTHOR=$(gh pr view $PR --json author --jq ".author.login")

I wouldn't want to send out 10s of notifications for this every day

If a test is failing on main 10s of times per day, to me that's a serious enough issue that it deserves that kind of noise. Still, I agree that tagging wouldn't have to be in the first iteration.

If you're watching distributed though, I think you've signed up for a lot of noise. It's either create a new issue every time, or comment on an existing one, so I think the noise level would be the same regardless of how it's implemented. (Simply ignoring tests that have failed once already doesn't feel right to me.)