proposal: testing: per-test setup and teardown support

[dfawley]

There are some checks that are helpful to perform after every test runs, e.g. monitoring for leaked goroutines. However, remembering to add these checks to every single test in a package is difficult and cumbersome. The testing package should provide a way to allow setup and cleanup code to be run globally for every test in a package, which has access to the testing.T for each test case. Note that TestMain does not allow you to run code between test functions; only at the very beginning and end of the whole package.

[ianlancetaylor]

As a proposal, it would be helpful to suggest exactly how this should work, so we can see what new API would be required.

[dfawley]

There are many different ways this could work, and I don't want to prescribe one for fear of flailing wildly and having it be rejected when other ways of implementing the same feature would work. A few ideas:

1. An interceptor-style function with the signature func(t *testing.T, f func(*testing.T)) could be registered in an init(). The interceptor is given the t and the test function f, and can perform setup, call f, and then perform cleanup.

2. A specially-named function (e.g. TestInterceptor(*testing.I) where I is a new interceptor type that contains t and f, as above.

3. New functionality in testing.M used via TestMain.

The last one sounds the most approachable to me, so I will flesh it out a bit.

How I would imagine the usage would look:

package my_test

func TestMain(m *testing.M) {
	for _, t := range m.Tests {
		setup()
		t.Run()  // invokes the current test function
		if err := cleanup(); err != nil {
			t.Error("error cleaning up test:", err)
		}
	}
}

Implementation:

package testing

type M struct {
	...
	Tests []TestRunner
}

type TestRunner struct {
	*T  // embed the testing.T that will be used for this test.
	// other unexported fields as needed
}

func (TestRunner) Run()  // runs this test with this TestRunner's *T

I haven't looked into the implementation of the testing package much; it's possible an iterator instead of a slice would be easier to implement, since creating all the testing.Ts at initialization might be difficult/impossible -- or an initializer on the TestRunner to create the T could be required instead. Also, a TestRunner.Done method may be required for implementation detail reasons. (Again, an iterator might be better, since it could automatically perform any necessary cleanup after the previous test.)

[rsc]

/cc @mpvl
This seems too intrusive to me, fwiw.

[rsc]

Various of us who have had to write code like this have found it helpful to write it explicitly instead of having an implicit thing that applies to every test in the package. Sometimes only 9 of 10 tests need a particular setup/teardown. Another common pattern is to put setup/teardown in one Test and then put the tests that need that setup/teardown into subtests. Or write a helper that does the setup/teardown and pass in the 'body' to run in between. There are lots of ways. We shouldn't hard-code one.

[dfawley]

Per-test setup and teardown is a common feature in most testing frameworks, and is supported in Java, C++, and Python, in OSS and at Google. I know there are other ways to implement this, but they are all cumbersome or error-prone or both. Something included in the language's primary testing framework would make it both easy to use and reliable.

[jeanbza]

+1. This is something extremely common in test frameworks. @rsc, please reconsider. This might not be something y'all use, but it is something the community uses:

• (stdlib) Python's unittest package has setUp (and tearDown).
• (stdlib) JUnit has @Before (and @ After). (I think older versions also have implicitly named setup and teardown functions?)
• (stdlib) PHP has setUp (and tearDown).
• (stdlib) C# has TestInitialize (and TestCleanup). (actually, I don't know anything about C#, so I might have interpreted the docs incorrectly on this one... but, I think I read it correctly)
• (third party) Node's Jasmine has beforeEach (and afterEach).
• (third party) Node's Mocha has before (and after).
• (third party) Go's Ginkgo has BeforeEach (and AfterEach).
• (third party) Groovy's Spock has setup (and cleanup).

[ianlancetaylor]

Why not just use a single test function with subtests? Why is that any harder to use than the suggestions in this issue?

[dfawley]

@ianlancetaylor What would that look like for a package with 10~100 tests? Something like this is what I was imagining:

func testFoo(t *testing.T) { ... }
func testBar(t *testing.T) { ... }
func testBaz(t *testing.T) { ... }

func TestEverything(t *testing.T) {
	// Maintaining this map is error-prone and cumbersome (note the subtle bug):
	fs := map[string]func(*testing.T){"testFoo": testFoo, "testBar": testBar, "testBaz": testBar}
	// You may be able to use the `importer` package to enumerate tests instead,
	// but that starts getting complicated.
	for name, f := range fs {
		setup()
		t.Run(name, f)
		teardown()
	}
}

Is there a better way to do this?

[ianlancetaylor]

var tests = []func(t *testing.T){
    func(t *testing.T) {
        ...
    },
    func(t *testing.T) {
        ...
    },
    ...
}

func TestEverything(t *testing.T) {
    for i, fn := range tests {
        setup()
        t.Run(strconv.Itoa(i), fn)
        teardown()
    }
}

Yes, it's more awkward, but it seems manageable, and there are cases where a bit of awkwardness is preferable to hidden magic.

[dfawley]

There are several undesirable aspects to your variant:

• The tests are not named. That makes it difficult to individually run them, and when one fails, you don't really know what failed. With any more than 2-3 tests, it's hard to even find the test case you're looking for since the numbers are implicit. There's also the issue of re-numbering due to inserting/deleting tests.
• The tests all must be in the same _test.go file. This approach doesn't work for larger packages with tests in multiple files.
• The tests themselves are not top-level functions. That adds an extra level of indent everywhere. This is bad for complex tests.

[ianlancetaylor]

There are obvious mitigations for those comments, but I said up front that the approach is awkward.

On the other hand, implicit test setup and teardown adds actions that are not clearly visible when looking at a large testsuite. And a single package will often have different kinds of tests, some of which need a particular kind of setup and teardown and some of which need a different kind, so over time the complexity of the setup and teardown tend to increase.

I don't think there is an obvious win here so we are choosing to be explicit and simple, as is the common pattern for the Go language.