proposal: testing: flag to continue testing after -failfast failure

[sparr]

Problem

After using go test -failfast to find the first failing test, then fixing the failure, there is no straightforward way to resume the tests from the point of failure. I have to either run all the earlier tests again, or enumerate the list of tests that come after the failing test.

Goal

Reduce the total time to resolve multiple bugs that each affect one or more tests but not most or all tests. Reduce the number of times each test is run.

Proposal

Add a flag such that -run TestFoo will not only run all tests matching TestFoo but all tests after the first such match.

Example

$ go test -failfast
--- PASS: TestA
--- PASS: TestB
--- FAIL: TestC
$ patch -i fix_test_c.patch
$ go test -run TestC
--- PASS: TestC
$ go test -failfast
--- PASS: TestA
--- PASS: TestB
--- PASS: TestC
--- PASS: TestD
--- FAIL: TestE

I didn't want to run TestA or TestB again here, or TestC an extra time. I would rather have done something like

go test -run TestC -andlatertests -failfast

(I acknowledge that's not a good name for the flag)

[cespare]

This seems like it is only relevant in packages where the tests are very slow. In most of the code I work on, running all the tests again would be quicker than typing out all the right flags to selectively re-run the tests that weren't run the first time. In addition, if you're making edits to the code to fix the first failure, it's possible to introduce new failures, so it seems to me that I'd want to run all the tests again anyway (even if they're slow).

I also think this is a fairly special-purpose flag that wouldn't be used much but would add to the already considerable weight of documentation for the testing package, go help testflag, and so on.

[sparr]

Yes, this change is aimed at packages with slow tests. Integration tests for the package I'm working on right now take 15-20 minutes to run in the environments we generally test in.

typing out all the right flags to selectively re-run the tests that weren't run the first time

The point here is to reduce this to a single flag.

it seems to me that I'd want to run all the tests again anyway (even if they're slow)

Running all the tests again once makes sense. But that's not what happens in the situations this is aimed at; the early tests end up getting run far more times than the later tests.

would add to the already considerable weight of documentation

This could be documented like -failfast, only appearing in some docs and not others. go help testflag would get longer, but not https://pkg.go.dev/testing or other high-visibility documentation.

[cespare]

The point here is to reduce this to a single flag.

In addition to -andlatertests you also need to provide an appropriate test name for -run. So I count two additional flags I'd type out in order to use this feature.

I will also say that the fact that this proposed flag is intertwined with the -run flag (it both mandates -run and changes its meaning) is a fairly confusing aspect of it.

Running all the tests again once makes sense. But that's not what happens in the situations this is aimed at; the early tests end up getting run far more times than the later tests.

In this workflow:

$ go test -failfast
# ... fix one failure ...
$ go test -failfast
# ... fix another failure ...
# etc

you run the tests once each time you change something, which is generally the best approach IMO. Each edit is an opportunity to break a previously-passing test.

If your tests take many minutes to run, then I could see why you'd want to avoid re-running the tests after every change. Maybe you want to re-test only the just-fixed test and then only do one big re-run at the end or something like that. But that workflow seems fairly specialized and not something we need to support specially with go test flags.

If I were facing a similar situation some options I'd think about would include:

• Just manually invoking the tests I want to run
• Using the -skip flag to skip some slow tests that I've already fixed or don't want to re-run
• Inserting a few temporary t.Skips if there are a few particularly slow tests that I don't want to keep re-testing
  • A more permanent version of this could be to have -short mode where certain very slow tests are skipped
• Writing a tool to implement the described workflow by running go test for me with -run and/or -skip and taking advantage of the -json flag to understand the output as well as go test -list
• Making the tests run faster

[sparr]

In addition to -andlatertests you also need to provide an appropriate test name for -run. So I count two additional flags I'd type out in order to use this feature.

This feature isn't aimed at you. For people who are already -runing the failing test (which, to be frank, should be everyone, but that's not a hill I care to die on today) it's just one additional flag.

I will also say that the fact that this proposed flag is intertwined with the -run flag (it both mandates -run and changes its meaning) is a fairly confusing aspect of it.

I agree this might be confusing. Another approach could be to have a new -runstartat Foo that does what -run Foo -andlatertests did in my original example.

Maybe you want to re-test only the just-fixed test and then only do one big re-run at the end or something like that.

Yes. Exactly that.

But that workflow seems fairly specialized

I propose that part of that specialization is because we don't have this feature. People's testing habits are based on the features available to them.

Just manually invoking the tests I want to run

It's a long list. And yes, that's what I'm doing now. My current plan is to put together a shell script that does a bunch of go test -list, finds the test I stopped on, then runs all the remaining tests. My first attempt ran into unexpected complexity for assembling the equivalent list of tests to a normal test run, which prompted me to write this proposal instead.

... [slow tests] ...

My integration tests are mostly of similar length. There are no slow tests to omit or t.Skip. I'd be doing it for every test before the now-passing one, which just brings us back to the original problem.

Also, this isn't just about the project I'm working on right now. The same problem exists in half the closest dozen related projects (mostly written by substantially different people / teams / etc).

[sparr]

Also, a lot of the suggested alternatives become much more tedious/difficult when go test is buried deep in a Makefile or other build process.

[ianlancetaylor]

How should handle this handle tests that call t.Parallel ?

[sparr]

I don't think making a test parallelizable affects the order in which tests are launched, and it's that order that this option would respect. Do you have a particular scenario in mind?

[bcmills]

t.Parallel matters because the current behavior of -failfast with parallel tests is somewhat ambiguous.

There is also an analogous interaction with package patterns. (go test can test many packages at a time, not just one.)

See:

• cmd/go: go test -failfast doesn't stop immediately with t.Parallel() #30522
• cmd/go: -failfast flag does not apply across packages #33038

[sparr]

For parallel tests, there could be some overlap. If TestA, TestB, TestC are run in parallel and TestB fails, then starting over at TestB would run TestC again.

Running tests across packages, as long as -failfast doesn't stop other packages from testing, would also produce some repeat tests.

However, both of these cases would still produce fewer repeat tests than running all tests again.

[ianlancetaylor]

Right now parallel tests work by starting the test and then immediately suspending it. Once all the non-parallel tests have completed, the parallel tests are started, with the number of tests run concurrently controlled by the -test.parallel option (default is GOMAXPROCS). So they don't fit cleanly into a paradigm of continuing the tests from a specific point. And it's even less clear if the test that failed was itself a parallel test.

[sparr]

Thanks. That wasn't clear to me. You're right that this would not fit cleanly here, but it would still reduce the number of tests run in most such cases.

[rsc]

The motivation here is unclear, or at least unconvincing. If you want to see all the other test failures, why use -failfast in the first place?

[sparr]

I've added a "Goal" section to the original post to clarify motivation, as follows:

Reduce the total time to resolve multiple bugs that each affect one or more tests but not most or all tests. Reduce the number of times each test is run.

My desired workflow, which I believe would be the fastest path to success in the scenarios I'm concerned with, is as follows:

1. Run all tests until one fails
2. Fix the bug that caused that test to fail
3. Re-run that single test to confirm the test no longer fails (repeat 2 otherwise)
4. Run the remaining tests until one fails
5. Repeat from 2 until there are no failures
6. Repeat from 1 until there are no failures

In a perfect world, 6 wouldn't re-run the tests that passed in the last iteration of 5, but that would be more complex and less valuable.