Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testing: add -failfast to stop after first test failure #21700

Closed
cespare opened this issue Aug 30, 2017 · 8 comments

Comments

Projects
None yet
7 participants
@cespare
Copy link
Contributor

commented Aug 30, 2017

I propose we add a testing flag to cause go test to exit after the first test failure.

There are two cases where I've wanted such a flag:

  • When I'm debugging a flaky test, I want to run it until failure. I often use -count 10000 for this purpose, but it will keep running the test after it fails once, so if I'm not looking at my terminal while it's running I need to go hunt through my scrollback to locate the failure. (Alternatively, I can use go test -c and re-run the test until failure using my shell, but this is slower, particularly if there's some kind of setup in a TestMain.)
  • When I'm debugging a test that has many failures, I want to focus on and fix a single test at a time. Today I can pick one of the failures and use -run to select it, but then once I've fixed that failure I want to focus the next one so I need to run all the tests again and repeat the process. It would be more convenient to run go test -exit1 (need a better flag name).

In the presence of t.Parallel this gets a little trickier. I suggest handling it by collecting the parallel results as they arrive and then exiting after the first failure, printing only the results that came in before the failure and then the failure last, but not printing results that arrived after the failure. The main subtlety for the user is that they might see output logged from other parallel tests which don't have success/failure printouts, but that seems like a minor concern.

@cespare cespare added the Proposal label Aug 30, 2017

@gopherbot gopherbot added this to the Proposal milestone Aug 30, 2017

@AlexRouSg

This comment has been minimized.

Copy link
Contributor

commented Aug 31, 2017

You could have a single test function and use subtests and then check if one failed and don't run the rest.

func TestFoo(t *testing.T) {
    failed := false

    if !failed {
        t.Run("A=1", func(t *testing.T) { failed = true })
    }

    if !failed {
        t.Run("A=2", func(t *testing.T) { failed = true })
    }

    if !failed {
        t.Run("A=3", func(t *testing.T) { failed = true })
    }
}
@cespare

This comment has been minimized.

Copy link
Contributor Author

commented Sep 6, 2017

@AlexRouSg sure, or I can comment out all the other tests. This proposal is about a way to improve the use cases I mentioned without restructuring the code.

@rsc

This comment has been minimized.

Copy link
Contributor

commented Oct 16, 2017

This seems OK but we have to come up with a decent flag name.

@spf13

This comment has been minimized.

Copy link
Contributor

commented Oct 16, 2017

In a few other similar tools the flag used is "--failfast"

  • Django
  • Rspec
@mvdan

This comment has been minimized.

Copy link
Member

commented Oct 20, 2017

Others that come to mind are -failstop and -firstfail.

@rsc

This comment has been minimized.

Copy link
Contributor

commented Oct 20, 2017

Let's use -(test.)failfast. If anyone wants to send a CL, please go ahead. (Remember to write a test and run mkalldocs.sh.) Thanks.

@rsc rsc modified the milestones: Proposal, Go1.10 Oct 20, 2017

@rsc rsc changed the title proposal: testing: add a flag to exit after the first test failure testing: add -failfast to stop after first test failure Oct 20, 2017

@gopherbot

This comment has been minimized.

Copy link

commented Oct 30, 2017

Change https://golang.org/cl/74450 mentions this issue: testing: add -failfast to stop after first test failure

@gopherbot gopherbot closed this in 153e409 Nov 29, 2017

@inancgumus

This comment has been minimized.

Copy link
Contributor

commented Nov 29, 2017

I completed my work on this. Currently, it tries to stop the tests from running. However, with parallel tests, this isn't easy to do right now (I guess).

@golang golang locked and limited conversation to collaborators Nov 29, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.