Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testing: TestMain should not require os.Exit #34129

Open
abuchanan-nr opened this issue Sep 5, 2019 · 9 comments

Comments

@abuchanan-nr
Copy link

@abuchanan-nr abuchanan-nr commented Sep 5, 2019

TestMain is expected to call os.Exit itself. This leads to subtle issues with defer and panic that leads to much headscratching. I'm proposing that if TestMain doesn't call os.Exit, the testing framework will call it for you after TestMain returns.

Let's say you start here:

func TestMain(m *testing.M) {
  setup()
  defer teardown()
  os.Exit(m.Run())
}

Subtle bug #1: your teardown isn't running; you're making a mess of your test environment. Try this:

func TestMain(m *testing.M) {
  var exitCode int
  defer os.Exit(exitCode)

  setup()
  defer teardown()
  exitCode = m.Run()
}

Oops, setup has a panic call. Now when you run go test, it's silent..

go test .
ok  	_/Users/abuchanan/scratch/testmain_bug	0.010s

Uh, headscratch. "Why isn't go running my tests?", you ask. Oh, os.Exit is hiding my panic call. Ok, finally:

func TestMain(m *testing.M) {
  // testMain wrapper is needed to support defers and panics.
  // os.Exit will ignore those and exit silently.
  os.Exit(testMain(m))
}

func testMain(m *testing.M) int {
  setup()
  defer teardown()
  return m.Run()
}

Let's save people some headscratching (and time and effort and silly wrapper code). The testing framework probably has enough information to call os.Exit appropriately for you after TestMain returns, right? Why require users to call it?

TestMain is really useful for writing end-to-end tests, where a single setup/teardown for all the tests is important. If I break those tests into packages for organization, it would be nice not to copy that TestMain wrapper code everywhere.

@gopherbot gopherbot added this to the Proposal milestone Sep 5, 2019
@gopherbot gopherbot added the Proposal label Sep 5, 2019
@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Sep 5, 2019

Related: #9917.

One possibility, that I thought was discussed before but I can't find an issue, is to change the behavior depending on the result of TestMain. If TestMain does not return a value, assume it calls os.Exit. If TestMain returns an int, then pass that int to os.Exit.

@abuchanan-nr

This comment has been minimized.

Copy link
Author

@abuchanan-nr abuchanan-nr commented Sep 5, 2019

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Sep 5, 2019

Ah, thanks.

@rsc

This comment has been minimized.

Copy link
Contributor

@rsc rsc commented Sep 12, 2019

This mistake does happen a lot, and the testing package that called TestMain and prepared the m can certainly record the result of m.Run for use in its own outer os.Exit.

Letting TestMain simply call m.Run instead of calling os.Exit(m.Run) seems OK and makes defer more useful.

I'm not as confident about alternate TestMain signatures; that seems unnecessary to solve this specific problem.

/cc @mpvl @robpike

@rsc

This comment has been minimized.

Copy link
Contributor

@rsc rsc commented Sep 25, 2019

@robpike and @mpvl, any thoughts?

@robpike

This comment has been minimized.

Copy link
Contributor

@robpike robpike commented Sep 26, 2019

No real insight from me. The topic seems covered by @ianlancetaylor's and @rsc's points.

@rsc

This comment has been minimized.

Copy link
Contributor

@rsc rsc commented Oct 2, 2019

It seems like if we do the "automatic os.Exit" that will fix quite a few buggy tests, and it basically eliminates the need for a TestMain that returns int. We could always revisit that and add it later, if needed, but let's start with the automatic os.Exit.

To be clear, the proposal is this. Right now, the overall test func main calls TestMain(m), and the TestMain function is expected to do two things:

  1. Call m.Run() and record the result.
  2. Call os.Exit with the result of m.Run.

The proposal is that:

  • m.Run records its own result in an unexported field of m, and then
  • if TestMain(m) returns, the outer test func main harness calls os.Exit with that code.

This would mean that implementations of TestMain that forget to call os.Exit will have it called for them. And implementations that want to explicitly avoid os.Exit in order to make defer useful can do that.

If TestMain does not call m.Run (presumably for a good reason) and returns, then the outer func main should os.Exit(0) as it does today.

Does anyone object to this plan or think it would not address the necessary use cases?

@rsc

This comment has been minimized.

Copy link
Contributor

@rsc rsc commented Oct 9, 2019

This seems like a likely accept.

Leaving open a week for final comments.

@bradfitz

This comment has been minimized.

Copy link
Member

@bradfitz bradfitz commented Oct 23, 2019

No new comments in the past week, so marking as accepted.

@bradfitz bradfitz changed the title proposal: TestMain should not require os.Exit testing: TestMain should not require os.Exit Oct 23, 2019
@bradfitz bradfitz modified the milestones: Proposal, Backlog Oct 23, 2019
@bradfitz bradfitz added the NeedsFix label Oct 23, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.