cmd/go: add support for dealing with flaky tests

[bradfitz]

Background

First off, flaky tests (a test that usually passes but sometimes fails) are the worst and you should not write them and fix them immediately.

That said, the larger teams & projects & tests get, the more likely they get introduced. And sometimes it's technically and/or politically hard to get them fixed. (e.g. the person who wrote it originally left the company or works on a new team) You're then left deciding whether to skip/delete the entire test (which might be otherwise super useful), or teach your team to become immune to and start ignoring test failures, which is the worst, when teams start submitting when CI's red, becoming blind to real test failures.

Google doesn't have this problem internally because Google's build system supports detecting & annotating flaky tests: https://bazel.build/reference/be/common-definitions

Tailscale has its own test wrapper that retries tests annotated with flakytest.Mark as flaky. We can't use go test -exec=... unfortunately, as that prevents caching (#27207). So instead we need a separate tool wraps cmd/go (kinda awkwardly, as it turns out).

Go internally also uses these flaky test markers 79 times:

• https://pkg.go.dev/internal/testenv#SkipFlaky
• https://pkg.go.dev/internal/testenv#SkipFlakyNet

So, clearly flaky tests are a problem. And cmd/go isn't helping.

Proposal

I propose that Go:

• let users annotate known flaky tests (ala tb.MarkFlaky(urlOrMessage string))
• let cmd/go support re-running marked flaky tests up to N times (default 3 like Bazel?). Maybe add -retrycount or -maxcount aside the existing -count flag?
• output machine readable output from cmd/go (that test2json recognizes) to say that a test was flaky and failed but eventually passed, that users can plug into their dashboards/analytics, to find tests that are no longer flaky
• optional: support re-running even tests not annotated as flaky after they fail, to discover that they're flaky. (perhaps it can still fail the test if not annotated, but it can fail with a different failure type, so the users learn it's flaky and can go fix or annotate)

FAQ

Won't this encourage writing flaky tests? No. Everybody hates flaky tests.

Won't this discourage fixing flaky tests? No. We're already retrying flaking tests. We're just fighting cmd/go to do so.

Why do you have flaky tests? Unit tests for pure functions aren't flaking. The tests usually flaking are big and involve timeouts and the network and subprocesses, even hitting localhost servers started by the tests. Many CI systems are super slow & oversubscribed. Even super generous network timeouts can fail.

Wasn't there already a proposal about this? Yes, in 2018: #27181. The summary was to not write flaky tests. I'm trying again.

[bcmills]

I've investigated an awful lot of test flakes in the Go project over the past few years. Some observations:

• Many test flakes are due to hard-coded timeouts (in the test or in the code under test).

  • Removing the arbitrary timeouts is often sufficient to deflake the test.
  • For tests that verify time-measurement functions, the fix is usually to compute upper and lower bounds based on the actual time elapsed in the test.
  • For tests that specifically check timeout behavior, the fix is usually to retry the test with exponentially longer timeouts until it succeeds.

• Many test flakes manifest as deadlocks or hangs.

  • Retrying a test that fails in this way can increase the end-to-end time of the test by an arbitrarily large factor.
  • Reducing that factor implies running the individual test with a much shorter timeout, but a much shorter timeout increases the risk of a flake due to the timeout being too short.
    • Which leads us back to the pattern of retrying the test with exponentially longer timeouts.

• Test flakes that are not in one of the above categories generally have a systematic (but undiagnosed) failure mode.

  • This is what allows us to use tools like watchflakes to classify errors.
  • These tests should only be retried if the failure mode matches the known pattern; otherwise, the flakiness annotation can mask real regressions in the code under test.

The common elements of these points are that flaky tests often need to be retried at a fine granularity and with a progressively increasing timeout, rather than retried a fixed number of times in the same configuration.

To me, that suggests an API addition to the testing package rather than a solution in cmd/go.

[bitfield]

Everybody hates flaky tests

I'm sure we all agree on that, Brad, but it doesn't follow that we should add special functionality to Go to make it easier for developers to live with flaky tests.

The summary was to not write flaky tests. I'm trying again.

I'm not sure why you expect a different answer this time. Frankly, if having flaky tests annoys you, or is inconvenient, I'd say that's a good thing.

sometimes it's technically and/or politically hard to get them fixed

Agreed. The solution is to delete them instead, not to mask the flakiness with automatic retries.

[painhardcore]

I wholeheartedly support this proposal. Addressing flaky tests is a real-world necessity and providing a structured way to manage them in the standard library is a step in the right direction.
This not only facilitates better tracking and management of these tests but also synergizes well with other tools like linters, by enforcing the addition of comments with Issue numbers or JIRA tickets, thus ensuring that the issues are properly documented and tracked.
Moreover, this proposal strikes a good balance by acknowledging the existence of flaky tests without encouraging their creation or ignoring their need for eventual resolution. It's a pragmatic approach to a problem that, despite our best efforts, persists in many large-scale projects.

[vearutop]

These tests should only be retried if the failure mode matches the known pattern; otherwise, the flakiness annotation can mask real regressions in the code under test.

To me, that suggests an API addition to the testing package rather than a solution in cmd/go.

I agree, I'd rather have a conservative opt-in way to retry test failures, so that flakyness "policy" is defined per test and is in full control of a developer.

	t.RunRetry("name", func(t *testing.T, attempt int) {
		if attempt > 3 {
			// Give up and forgive.
			t.Skip()
			
			// Or give up and fail hard.
			// t.FailNow()
		}

		// Iterate with increasing tolerance depending on attempt.

		t.Fail()
	})

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[dnephin]

the larger teams & projects & tests get, the more likely [flaky tests] get introduced. And sometimes it's technically and/or politically hard to get them fixed. (e.g. the person who wrote it originally left the company or works on a new team)

This is a really important point that I think is worth elaborating. The impact of flaky tests on the contributors to a software project is often directly correlated to the constraints imposed by the organization responsible for the project.

Many software projects won't suffer from flaky tests. Maybe because the code has minimal concurrency, the scope of the problem is small, or because the code isn't critical enough to require comprehensive end-to-end testing of all the components. Other projects with significant concurrency, large scope, and large end-to-end tests may encounter the occasional flaky test, but also don't suffer from flaky tests because the team has sufficient time and resources to deal with them. The Go project has some flaky tests, but appears to have significant infrastructure in place to identify, and triage the flaky tests, and also to mitigate the impact of those flakes on other contributors. Not every project is so fortunate. If your experience is working only on projects that are in these two categories it can be easy to dismiss the problem of flaky tests as not significant. Maybe the problem of flaky tests is not significant to the majority of projects, but it can be very real for some.

A few years ago I joined a team that was working on a well known open source project. The project has thousands of tests, many of them could flake. The team had attempted to deal with the flaky tests for years, but could never get them into a reasonable state. The project had been around for 6 years already, had hundreds of contributors who no longer worked on it (many from the open source community), and most of the flaky tests were written by someone who was no longer involved with the project. The cost of fixing a flaky test was high enough that by the time one or two were fixed new ones had appeared. Contributors were forced to either click "rerun job" in CI multiple times (and wait hours to get a green run), or assume the test was flaky and merge with a failing CI check.

Multiple attempts were made to manually identify flaky tests by creating github issues to track them. This helped a bit, but by itself didn't really address the problem. Creating github issues for every CI run was still taking time from every contributor, and it was often easier to click re-run and not create a github issue. Github issues on their own did not help prioritize the most flaky of tests. This manual approach was not enough.

This experience is what motivated me to write the --rerun-fails feature in gotestsum. It doesn't use test markers, but otherwise seems to implement the proposal described in the original post. Roughly:

• if more than x (default 10) tests fail in the initial run, do not re-run anything. This is to avoid re-running tests when there is an infrastructure problem, or a real problem in a core piece of code.
• for each test that failed in the initial run, re-run it up to n times (default 2 more attempts)
• show all the test failures in the output, so that someone looking at the run knows that some tests had to be retried
• --rerun-fails-report=filename will output the list of tests that were re-run. This can get posted to the Github PR as a comment, so that the author of the PR is still aware of any potentially new flaky tests in their CI run.

This feature allowed the team to make real progress against fixing flaky tests. Instead of manually ticketing and re-running failures we could use the test2json output to find the most common flakes, and focus our efforts on those tests.

I've seen some interest in this feature from the community. This sourcegraph search shows at least a few recognizable projects using it. I imagine there are others that are not indexed by source graph, not open source, or are using the flag in a script that doesn't match the search query.

[dnephin]

Given my experience with flaky tests above, I have a few questions about the proposal.

Why only re-run marked tests? My rational for re-running flaky tests has generally been that an existing flaky test should not introduce friction for other contributors. If a contributor has to create a github issue and edit a test they are not familiar with, that is non-zero friction and they may be tempted to just re-run the CI job instead. I would generally prefer to have an automated system notice the flakes by reading the test2json output and seeing that a test both failed and passed in the same run. That same system should be able to sort the flakes to find the most common ones.

Some indication of re-run tests is important, but given the nature of a flaky test, it seems unlikely that the person adding the mark is going to be the same person that introduce the flake.

Is the marking of a test more for the developer reading the test code, so that they understand it's a flaky test?

Do we need new output to show "that a test was flaky and failed but eventually passed" ? If a test appears multiple times in the test2json with both fail and pass doesn't that indicate a flake?

[bcmills]

I think @vearutop is on the right track with RunRetry, but I think it's also important to be able to distinguish an outright Fail (meaning “this test had an unexpected failure mode”) from a “flake” (meaning “this test failed to produce a useful signal”).

I suggest the following additions to testing.T. The Retry method is analogous to Fail, and the Restart methods are analogous to the Skip methods.

// Restart is equivalent to Retry followed by Skip.
func (*T) Restart(args ...any)

// Restartf is equivalent to Retry followed by Skipf.
func (*T) Restartf(format string, args ...any)

// Retry marks the function as unable to produce a useful signal,
// but continues execution.
// If the test runs to completion or is skipped, and is not marked as
// having failed, it will be re-run.
func (*T) Retry()

// Retries reports the number of times the current test or subtest has been
// re-run due to a call to Retry. The first run of each test reports zero retries.
func (*T) Retries() int64

The same methods could also be added to testing.B and testing.F if desired, but note that this would not allow for flaky Example functions.

The idea is that when a test flakes, it calls Restart or Restartf with a suitable message; the re-run can then use the Retries method to compute whatever new timeouts it needs. If for some reason a test wants to limit the number of retries it attempts, it could also use the Retries counter to trigger a call to Fatalf or similar.

[bcmills]

@dnephin

Is the marking of a test more for the developer reading the test code, so that they understand it's a flaky test?

The marking of a test is so that the flake-retry mechanism doesn't mask actual regressions — such as data races or logical races — that may manifest as low-but-nonzero-probability failures in tests that were previously not flaky at all.

[willfaught]

Why return int64 instead of int?