testing: add -failfast to stop after first test failure

[cespare]

I propose we add a testing flag to cause go test to exit after the first test failure.

There are two cases where I've wanted such a flag:

• When I'm debugging a flaky test, I want to run it until failure. I often use -count 10000 for this purpose, but it will keep running the test after it fails once, so if I'm not looking at my terminal while it's running I need to go hunt through my scrollback to locate the failure. (Alternatively, I can use go test -c and re-run the test until failure using my shell, but this is slower, particularly if there's some kind of setup in a TestMain.)
• When I'm debugging a test that has many failures, I want to focus on and fix a single test at a time. Today I can pick one of the failures and use -run to select it, but then once I've fixed that failure I want to focus the next one so I need to run all the tests again and repeat the process. It would be more convenient to run go test -exit1 (need a better flag name).

In the presence of t.Parallel this gets a little trickier. I suggest handling it by collecting the parallel results as they arrive and then exiting after the first failure, printing only the results that came in before the failure and then the failure last, but not printing results that arrived after the failure. The main subtlety for the user is that they might see output logged from other parallel tests which don't have success/failure printouts, but that seems like a minor concern.

[AlexRouSg]

You could have a single test function and use subtests and then check if one failed and don't run the rest.

func TestFoo(t *testing.T) {
    failed := false

    if !failed {
        t.Run("A=1", func(t *testing.T) { failed = true })
    }

    if !failed {
        t.Run("A=2", func(t *testing.T) { failed = true })
    }

    if !failed {
        t.Run("A=3", func(t *testing.T) { failed = true })
    }
}

[cespare]

@AlexRouSg sure, or I can comment out all the other tests. This proposal is about a way to improve the use cases I mentioned without restructuring the code.

[rsc]

This seems OK but we have to come up with a decent flag name.

[spf13]

In a few other similar tools the flag used is "--failfast"

• Django
• Rspec

[mvdan]

Others that come to mind are -failstop and -firstfail.

[rsc]

Let's use -(test.)failfast. If anyone wants to send a CL, please go ahead. (Remember to write a test and run mkalldocs.sh.) Thanks.

[gopherbot]

Change https://golang.org/cl/74450 mentions this issue: testing: add -failfast to stop after first test failure

[inancgumus]

I completed my work on this. Currently, it tries to stop the tests from running. However, with parallel tests, this isn't easy to do right now (I guess).