testing: clarify when TestMain is appropriate and not

[matttproud]

We had an open discussion in a longer design review thread about when user-authored custom TestMain (debuted in release 1.4) entrypoints are necessary or not compared to func init. We were able to offer at best fuzzy criteria for this. It would be beneficial for the newcomer's perspective if some basic litmus tests, advice, or criteria were provided. I have no opinion on where but rather that they are central and findable.

I don't want to bike shed the reasoning, but I will enumerate some of the tradeoffs I heard:

1. More individuals are familiar with func init semantics, so that speaks in favor of func init for setup. One counterpoint is that a setup routine that requires all package initializers to have been run by the time it is executed could become flaky or error-prone.
2. Not all tests strictly need what is commonly setup, making the test setup process more expensive than it needs to be (e.g., when a test execution filter -test.run=pattern is used). This also speaks against func init above.
3. From a code health standpoint, central setup in func init or func TestMain, could imply mutated global state. Unless the tests use whatever is setup in a hermetic way individually, it could accidentally encode test ordering assumptions between test functions. Might be better to encourage tests to locally set up what they need versus not and to enforce cleanup, especially now that we have (*testing.T).Cleanup.

One final ancillary point I heard in an earlier discussion related to user-authored TestMain that a documentation treatment should also cover:

1. what is the preferred way to handle errors in TestMain? panic, os.Exit(1), log.Fatal?
2. where should text output be written: stdout or stderr?

Before stating that the answers to the above is obvious, it may be obvious to a seasoned practitioner but not a newcomer.

[ianlancetaylor]

The main argument for TestMain, per #8202, is the ability to run something on shut down, after all the tests have been run. A secondary argument is that TestMain, unlike init functions, can examine flags passed to the program. If all you need is something to run at program startup that doesn't need to look at flags, then an init function is fine.

Personally I would say that errors in TestMain should be reported on os.Stderr and that TestMain should call either os.Exit or log.Fatal depending on personal preference.

I'm not sure any of this needs to go into the docs, but a blog post would be fine.

[matttproud]

Would an example code body or similar suffice? We're going to take care of this, I expect, in our internal documentation, but I'd hate to leave folks wondering. I've seen a zoo of practices around use over the years, some of which have been wrong. There are nice soft ways that counsel above could be phrased without requiring folks to look at previous issue IDs. What do you think?

[ianlancetaylor]

I'm not sure how to write an example for TestMain.

Do you not like the idea of a blog post? It might be similar to the one for contexts (https://blog.golang.org/context), in the sense of a blog post describing the right way to use a library feature.

[matttproud]

Don't get me wrong; I like the idea of blog posts. I've grown preference-wise very attached to the native API documentation as a place to find canonical information and patterns. I trust in terms of your recommendation on a blog post so won't push more. I will talk to Jean de Klerk about this one since he has experience with writing them. It is probably relatively easy to explain this advice as a blog post since they often take on a rhetorical-storytelling form. I'll ruminate further.

…

[bcmills]

IMO an init function is usually not the right choice no matter what: if tests expect some shared setup, a sync.Once is usually a better fit. If every test necessarily requires some kind of setup (like stubbing out functions or injecting dependencies), that's usually a symptom of a deeper problem with the package (like too much global state, or an API that can test itself but can't be integration-tested by downstream users), and fixing that deeper problem brings the package back to where a sync.Once will suffice.

[gopherbot]

Change https://golang.org/cl/334649 mentions this issue: testing: clarify in docs that TestMain is advanced

[bluelamar]

There is also the issue whereby a singleton object is required by the tests.