Repository navigation
testing: add M.Cleanup #42161
Description
Activity
It seems to me that
testing.M.Cleanupcan be implemented simply usingdefer. This isn't true fortesting.T.Cleanup, because of parallel subtests, which can continue to run after the test function completes. So I don't see a strong argument for addingtesting.M.Cleanup.- I agree with that, but I came to suspect a Cleanup facility may be needed if a formal Fatal and exit facility is added to testing.M. One of the triggers for my concern and filing this issue was seeing log.Fatal calls combined with load-bearing defer blocks that wouldn’t run (defer block to reconcile external state that the test mutated in its main entry point and tried to revert). It seems either better documentation or API could ameliorate this since keeping the nuance and semantics in mind as tribal or institutional knowledge simply isn’t sustainable. It also felt wrong to maintain in-house documentation on this as well. I trust your folks judgment. :-)
It sounds like we don't need to worry about an M.Cleanup until/unless we have M.Fatal etc.
And I don't know why we would have M.Fatal - is that a separate proposal?
M is intentionally low-level. It's not supposed to be something with a rich API.- changed the title
[-]Proposal: Give testing.M both a Cleanup and Failure Logging Function[/-][+]testing: add M.Cleanup[/+]on Jul 14, 2021 - Over recent quarters, I have seen folks carelessly gravitate toward using TestMain when it isn’t required (as in, there are better least-mechanism alternatives like the builtin testing facilities for the task at-hand). I think it would be useful for TestMain to have some documentation to make the low-level point you made more clear. Here is a tracer shot: “TestMain is an advanced feature and not meant for casual use. Regular testing functions will suffice for most cases.” I don’t mean to bring a point too orthogonal to the discussion, but perhaps illuminating the intended path of the user journey in documentation would provide an escape valve alternative versus expanding the API. What do you think?
It's certainly easy to add a sentence or two to testing.M.
Based on the discussion above, this proposal seems like a likely decline.
— rsc for the proposal review groupChange https://golang.org/cl/334649 mentions this issue:
testing: clarify in docs that TestMain is advancedReacted by Oleg KovalovReviving an old issue, the use case here is that deferred functions that are called in TestMain do not run if there's a panic in m.Run, even if there's a deferred recover on the stack in TestMain. Any processes started in TestMain don't get cleaned up when the test exits in this case.
Reacted by Martin Desrumaux, minikomi, Nissim Natanov (Outreach) and Josh GilesAdding my hat to this. The documentation on
testing.Mis a bit confusing and contradictory to being able to cleanup after functions. The main example is thatm.Runreturns an exit code that is meant to be used inos.Exit.It even says so in the documentation
However, were you to actually use
os.Exit, it would bypass anydeferstatements you made in order to cleanup.The best way I found to get around this was to only call
os.Exitif the code was not a successful exit (i.e. not0). But that's a bit wonky IMO. The documentation should probably be a bit more clear about that.retCode := m.Run() if retCode != 0 { os.Exit(retCode) }
- locked and limited conversation to collaborators
on Oct 14, 2023 - added a commit that references this issue
on Apr 29, 2024
This is a feature request/proposal for
testing.M. I was doing a Readability Review today of user code. The code for one reason or another was using a custom main entrypoint. I noticed how bare the support intesting.Mis for users to report and handle setup/teardown failures as well as conduct cleanups. The setup, execution, and teardown of the custom test main needed to do both of these tasks. It dawned on me that there no advice is given in the documentation about where setup error messages should go and how to correctly abort (e.g., what exit value).I would like to propose that for similar reasons of
testing.Thaving(*testing.T).Cleanupand(*testing.T).Fatal,testing.Mshould have the same.Between the two of them, the cleanup is less important. But how to report errors and setup failures looks very relevant.
For instance, if a test main fails, should the user call
log.Fatalor one of its variants, hand emit tostderrorstdout? I have seen the whole zoo of behavior before. Is any non-zero exit value OK or only some? I get that custom test mains are an advanced feature and should be seldom used, but it seems potentially like we are not providing the users an adequately documented journey. Maybe this is intentional (e.g., Go tests to support custom test harness and executor requirement contracts)? It does deserve some Schrift at the very least.