Track test failure statistics across all PRs

[gjoseph92]

The flaky test dashboard has been invaluable both in improving the stability of our CI, and as a PR author, in identifying which test failures are real and which can be ignored since they're flaky.

However, the test dashboard only reflects builds on main. Main is only a small percentage of all the builds we run. There are plenty of tests that are flaky but haven't failed on main recently.

Could we record outcomes of every test run for PRs (maybe into a separate database)? Write concurrency to sqlite could be tricky but hopefully we can figure something out.

This would increase the amount of data we have on test failures by an order of magnitude. More data is very helpful when trying to measure something with nondeterministic behavior (like flaky tests).

Obviously, many test failures in branches are just due to changes in that branch, so the data within a single branch wouldn't be very meaningful. But I have a feeling that common failures across branches are an important signal.

With more data, we could:

1. better identify the failure rate of some tests, and the priority of fixing them (how frequently do they fail? how widely do they fail? what's the chance of a given PR encountering this failure?)
2. better visualize test failures: this failure rate data would make for a great dashboard, especially with a toggle between "only main" and "all PRs"
3. better identify when a particular test started failing (if a test fails on main, search for the first build on any branch where it failed—that's probably closest to the commit that actually broke it). This could help a lot in trying to identify the responsible PR in Automatically open issues for failing tests on main #6969.
4. better identify problematic tests: if the same test fails in multiple (say >3) different PRs within a time window (1 week?), it's probably flaky (versus related to the changes made in those PRs), even if it hasn't failed on main yet. It could then also qualify for getting an issue opened: Automatically open issues for failing tests on main #6969.

cc @fjetter @hendrikmakait @ian-r-rose

[fjetter]

Could we record outcomes of every test run for PRs (maybe into a separate database)? Write concurrency to sqlite could be tricky but hopefully we can figure something out.

We would still use the same job but the job would not only look at the main branch but at PRs as well. That should be a reasonably straight forward change.

Obviously, many test failures in branches are just due to changes in that branch, so the data within a single branch wouldn't be very meaningful. But I have a feeling that common failures across branches are an important signal.

How would we distinguish this?

---

I'm worried about

• Rate limiting of the API
• Data quality
• Complexity of this solution

---

Our method to collect more data in the past was the "CI stress test". We triggered >=10 runs and collected the samples. I typically run this on my fork to not interfere with

distributed/.github/workflows/tests.yaml

Lines 38 to 44 in 10cf945

	# Uncomment to stress-test the test suite for random failures.
	# Must also change env.TEST_ID below.
	# This will take a LONG time and delay all PRs across the whole github.com/dask!
	# To avoid hamstringing other people, change 'on: [push, pull_request]' above
	# to just 'on: [push]'; this way the stress test will run exclusively in your
	# branch (https://github.com/<your name>/distributed/actions).
	# run: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]

on this branch https://github.com/fjetter/distributed/tree/stress_ci I I am running this frequently and am posting the results on #6625
The automation of this process would be much easier than collecting data from all branches