testing: add M.Cleanup

[matttproud]

This is a feature request/proposal for testing.M. I was doing a Readability Review today of user code. The code for one reason or another was using a custom main entrypoint. I noticed how bare the support in testing.M is for users to report and handle setup/teardown failures as well as conduct cleanups. The setup, execution, and teardown of the custom test main needed to do both of these tasks. It dawned on me that there no advice is given in the documentation about where setup error messages should go and how to correctly abort (e.g., what exit value).

I would like to propose that for similar reasons of testing.T having (*testing.T).Cleanup and (*testing.T).Fatal, testing.M should have the same.

Between the two of them, the cleanup is less important. But how to report errors and setup failures looks very relevant.

For instance, if a test main fails, should the user call log.Fatal or one of its variants, hand emit to stderr or stdout? I have seen the whole zoo of behavior before. Is any non-zero exit value OK or only some? I get that custom test mains are an advanced feature and should be seldom used, but it seems potentially like we are not providing the users an adequately documented journey. Maybe this is intentional (e.g., Go tests to support custom test harness and executor requirement contracts)? It does deserve some Schrift at the very least.

[ianlancetaylor]

It seems to me that testing.M.Cleanup can be implemented simply using defer. This isn't true for testing.T.Cleanup, because of parallel subtests, which can continue to run after the test function completes. So I don't see a strong argument for adding testing.M.Cleanup.

[matttproud]

I agree with that, but I came to suspect a Cleanup facility may be needed if a formal Fatal and exit facility is added to testing.M. One of the triggers for my concern and filing this issue was seeing log.Fatal calls combined with load-bearing defer blocks that wouldn’t run (defer block to reconcile external state that the test mutated in its main entry point and tried to revert). It seems either better documentation or API could ameliorate this since keeping the nuance and semantics in mind as tribal or institutional knowledge simply isn’t sustainable. It also felt wrong to maintain in-house documentation on this as well. I trust your folks judgment. :-)

[rsc]

It sounds like we don't need to worry about an M.Cleanup until/unless we have M.Fatal etc.
And I don't know why we would have M.Fatal - is that a separate proposal?
M is intentionally low-level. It's not supposed to be something with a rich API.

[matttproud]

Over recent quarters, I have seen folks carelessly gravitate toward using TestMain when it isn’t required (as in, there are better least-mechanism alternatives like the builtin testing facilities for the task at-hand). I think it would be useful for TestMain to have some documentation to make the low-level point you made more clear. Here is a tracer shot: “TestMain is an advanced feature and not meant for casual use. Regular testing functions will suffice for most cases.” I don’t mean to bring a point too orthogonal to the discussion, but perhaps illuminating the intended path of the user journey in documentation would provide an escape valve alternative versus expanding the API. What do you think?

[rsc]

It's certainly easy to add a sentence or two to testing.M.

[rsc]

Based on the discussion above, this proposal seems like a likely decline.
— rsc for the proposal review group

[gopherbot]

Change https://golang.org/cl/334649 mentions this issue: testing: clarify in docs that TestMain is advanced

[rsc]

No change in consensus, so declined.
— rsc for the proposal review group

[AndrewDK]

Reviving an old issue, the use case here is that deferred functions that are called in TestMain do not run if there's a panic in m.Run, even if there's a deferred recover on the stack in TestMain. Any processes started in TestMain don't get cleaned up when the test exits in this case.

[sunny-b]

Adding my hat to this. The documentation on testing.M is a bit confusing and contradictory to being able to cleanup after functions. The main example is that m.Run returns an exit code that is meant to be used in os.Exit.

It even says so in the documentation

However, were you to actually use os.Exit, it would bypass any defer statements you made in order to cleanup.

The best way I found to get around this was to only call os.Exit if the code was not a successful exit (i.e. not 0). But that's a bit wonky IMO. The documentation should probably be a bit more clear about that.

retCode := m.Run()

if retCode != 0 {
  os.Exit(retCode)
}